Masked Path Modeling for Vision-and-Language Navigation
Authors: Zi-Yi Dou
Feng Gao
Nanyun Peng
Venue: EMNLP 2023 Findings (2023)
Keywords: Vision-and-Language Navigation
Masked Data Modeling
TL;DR: A scalable pretraining objective for vision-and-language navigation that uses a rich amount of self-collected data to connect vision and action modalities.

Abstract: Vision-and-language navigation (VLN) agents are trained to navigate in real-world environments based on natural language instructions. A major challenge in VLN is the limited available training data, which hinders the models' ability to generalize effectively. Previous approaches have attempted to alleviate this issue by using external tools to generate pseudo-labeled data or integrating web-scaled image-text pairs during training. However, these methods often rely on automatically-generated or out-of-domain data, leading to challenges such as suboptimal data quality and domain mismatch. In this paper, we introduce a masked path modeling (MPM) objective. MPM pretrains an agent using self-collected data for subsequent navigation tasks, eliminating the need for external tools. Specifically, our method allows the agent to explore navigation environments and record the paths it traverses alongside the corresponding agent actions. Subsequently, we train the agent on this collected data to reconstruct the original action sequence when given a randomly masked subsequence of the original path. This approach enables the agent to accumulate a diverse and substantial dataset, facilitating the connection between visual observations of paths and the agent's actions, which is the foundation of the VLN task. Importantly, the collected data are in-domain, and the training process avoids synthetic data with uncertain quality, addressing previous issues. We conduct experiments on various VLN datasets and demonstrate the applications of MPM across different levels of instruction complexity. Our results exhibit significant improvements in success rates, with enhancements of 1.3\%, 1.1\%, and 1.2\% on the val-unseen split of the Room-to-Room, Room-for-Room, and Room-across-Room datasets, respectively. Additionally, we underscore the adaptability of MPM as well as the potential for additional improvements when the agent is allowed to explore unseen environments prior to testing.

Masked Path Modeling for Vision-and-Languugg Navigation
Zi-Yi Dou', Feng Gao², Nanyun Peng
!University of California, Los Angeles #Amazon Alexa Al (zdou, violetpeng Jecs. ucla. edu
Abstract
Vake-aad Larganrr Navigataa
Pr
navigation (VLN) agents are trained to navigate real-world environ- ments based natural language instructions. A major challenge in VLN is the limited avail- able training data, which hinders the models' ability to generalize effectively. Previous ap- proaches have attempted to alleviate this is- sue by using external tools generate pseudo- labeled data or integrating web-scaled image- text pairs during training. However, these meth- ods often rely automaticalll--gneeaaii out-of-domain data, leading to challenges such as suboptimal data quality and domain mis- match. In this paper, introduce masked path modeling (MPM) objective. MPM pre- trains an agent using self-collected data for sub- sequent navigation tasks, eliminating the need for external tools. Specifically, our method allows the agent explore navigation environ- ments and record the paths it traverses along- side the corresponding agent actions. Subse- quently, we train the agent this collected data to reconstruct the original action sequence when given a randomly masked subsequence of the original path. This approach enables the agent to accumulate a diverse and substan- tial dataset, facilitating the connection between visual observations of paths and the agent's ac- tions, which the foundation of the VLN task. Importantly, the collected data in-domain, and the training process avoids synthetic data with uncertain quality, addressing previous is- sues. We conduct experiments various VLN datasets and demonstrate the applications of MPM across different levels instruction com- plexity. Our results exhibit significant improve- ments in success rates, with enhancements of 1.3%, 1.1% and 1.2% on the val-unseen split of the Room-to teeett---ooee Ro-Room, Room-for- foom and Koom-across-Roood datasets, respectively, Ad- ditionally, underscore the adaptability of MPM as well as the potential for additional improvements when the agent is allowed to ex- plore unseen environments prior to testing.'
Macked Modeltig
Figure 1:
language navigation agents are trained to predict actions in real-world environment given natural language instruction. We incorporate our proposed masked path modeling objective into training, where the agent is trained to reconstruct the original action sequence when given randomly masked sub- sequence of the visual inputs of the original path. The figure is adapted from Anderson et al. (2018b),
Vision-and aaroon-nnd-aanguan
or
Introduction
A seiio-anddlanguusd navigation (VLN) agent is trained to follow natural language instructions and navigate within an environment to achieve a spec- ified goal. This task requires the agent to pos- sess several sophisticated abilities, including under- standing and grounding language phrases to visual objects, as well as planning and executing actions in real-world setting.
The pretraining-then siininggtte--iiit finettiinn paradigm (Pe- ters et al., 2018; Devlin al., 2019; Chen et al., 2021b; He et al., 2022) has proven to be effec- tive in addressing these challenges in the field VLN. By utilizing various supervision signals and proposing pretraining objectives, significant im- provements have been demonstrated across VLN tasks. Prior works have explored the use of internet- scale image-text datasets to acquire grounded vi- sion and language representations. Notable contri- butions in this area include the works of Majumdar et al. (2020) and Guhur et al. (2021), who lever- age web-scraped image-caption corpora to learn
'Code is available at httpp://ggitee com/
PlusLabNLP/mpm.

---

general vision-language representations and then finetune the models using in-domain VLN data to adapt the representations specifically for VLN tasks. Similarly, Shen et al. (2022) and Khandelwal et al. (2022) employ the CLIP vision encoder (Rad- ford et al., 2021) pretrained with image-text con- trastive loss and showcase the application of such models in various embodied tasks.
the collected path and action information for training, eliminating the need for synthesizing additional signals.
It explicitly focuses on conditional action gen- eration. Concretely, as shown in Figure 1, to construct the MPM objective, we randomly mask certain viewpoints in the self-collected paths and train the agent to reconstruct the original paths based on the masked ones. MPM similar to the VLN objective because the agent is trained to output a sequence of actions given specific instructions, with the distinction that the instructions are presented masked paths rather than natural language instructions. Consequently, MPM effectively prepares the agent for VLN tasks that require conditioned action generation skills.
While the vision-language pretrained represen- tations improve the alignment between vision and language modalities, it is important to note that there is a domain gap between the pretraining data and VLN data. In addition, these models are un- aware of how to connect the learned representa- tions to actions because they are not explicitly trained for generating actions, which is criti- cal skill for VLN tasks. To address the issue and integrate action generation into pretraining, prior works such as PREVALENT (Hao et al., 2020) HAMT (Chen et al., 2021a) use a single-step action prediction objective based on human-annotated and synthetic image-text-action triplets. However, the scalability of the pretraining objective is limited due to the scarcity of annotated instruction-actio action pairs Specifically, they require training on pairs of natural language instruction and action sequences which are costly to obtain in a scalable way. There is also a line of work that utilizes synthetic data for training, either obtaining pseudo-instructions from sampled paths (Fried et al., 2018b; Tan et al., 2019; Hao et al., 2020; Wang et al., 2022b) or gen- erating both the visual environments as well as the language instructions (Kamath et al., 2023; Wang al., 2023). However, the autommatcallly-eeeraaedd data cannot be perfect and the noise during data generation can impact the model performance. In this paper, we present an approach to pretrain VLN models with masked path modeling (MPM), which leverages in-domain path data collected by agent for supe-super learning. The proposed objective targets addressing the two major limita- tions of previous work:
As a result, our pretraining objective is scalable and well-suited for addressing the VLN task.
We evaluate the proposed method on various VLN datasets with different types of instructions, including Room-to-Room (Anderson et al., 2018c), Room-for-Room (Jain et al.. 2019), and Room- across Room (Ku et al., 2020). Experimental re- sults demonstrate that MPM can achieve significant improvements in both seen and unseen environ- ments compared with strong baselines. For exam- plc, we achieve improvements of 1.32%, 1.05%. and 1.19% on the val-unseen split of the Room-to- Room, Room- Room-for-Bel formRoom, and Room- acrossscos---- across-Room datasets, respectively. In addition, we demonstrate the MPM can be flexibly integrated into different types of models and achieve improvements. Fur- thermore, an analysis reveals the potential for addi- tional improvements when the agent is allowed to explore unseen environments prior to testing.
Methods
In this section, we first introduce the basic settings and model architecture of vision-and-langun language nav- igation, then illustrate the details of each compo- nent of our proposed approach.
It collects scalable in-domain pretraining data without data synthesis. During the pretraining phase, the VLN agent explores the environ- ment randomly and gathers navigation paths along with its actions, which are then used for pretraining the agent. Because the agent actively explores different environments, we can collect a rich amount of diverse paths. In addition, the MPM objective only requires
2.1 Background
Training Data. The training data 'D of VLN con- sists of parallel instruction-action pairs {(ft a²)) from different environments. However, it is hard to manually annotate a large amount of instruction-action data for VLN. Therefore, re- searchers have proposed various data augmenta- tion strategies (Fried et al., 2018b; Tan et al.,

---

VIN
Action Prediction
MPM
Action Prediction
Cros-Modal Trunsformer
Coos-Modal Transformrr
Transformer
Hirrschiaal Vision Trannfurmer
Transfomer
Illeraciicl Visioa Transformer
Vision Transformer
Linear Trundforation
Oistright sentar reah vilerii terilifiaad
History
Path
History
We follow Chen et al. (2021a) to design the base model architecture. In VLN, the model separately encodes the given instruction with a text encoder, its past history with a hierarchical visual encoder, and its current observations with another vision encoder; the encoded representations transformer to predict the final action. In MPM, we directly feed masked subpath to the cross- moosll transformer instead of language instruction and the model is trained to predict the original action given the masked subpath. All the parameters between VLN and MPM are shared.
Figure 2:
then fed into joint cross-modal
2019; Li et al., 2022; Kamath et al., 2023) to pro- vide additional supervision for VLN models. Int this work, we follow a common practice to use speaker model (Fried et al., 2018b) to generate lan- guage instructions given randomly sampled paths and enrich the training data with the generated instruction-action pairs. Specifically, we incorpo- rate the PREVALENT-generated data (Hao al., 2020) into training.
Text Features. Following previous work (Chen et al., 2021a), we use pretrained text encoder to encode the language instructions. We use the standard BERT model (Devlin et al., 2019) to en- code the English instructions for the R2R and R4R datasets (Anderson et al., 2018c; Jain al., 2019) and the XLM-R model (Conneau et al., 2020) encode the multilingual instructions for the RxR datasct (Ku et al., 2020).
Base Settings. Using the parallel instruction- action pairs D, VLN agent is trained follow given language instruction and generate a sequence actions to reach the final destination in a photo- realistic environment. Formally, in a given environ- ment e, the navigation agent parameterized by 0 learns to model the distribution P(ali, e; 0), where and denote instruction and action variables, re- spectively.
At each step, the agent is given panoramic observation of its current position in the environment, denoted {u:}k For each view in the panoramic observation, its vision fea- ture is first extracted using a pretrained vision en- coder. While many of the previous methods (Ander- son et al., 2018;; Chen et al., 2021a) use vision en- coders pretrained on ImageNet (Fei-Fei et 2009) for image classification, we find that CLIP vision encoder (Radford et al., 2021) achieves stronger performance, which is consistent with the findings of Shen et al. (2022). Therefore, use CLIP to first extract vision features and then the CLIP features are fed into the transformers for history and observation encoding.
Vision Features.
Model Architecture. In this paper, we employ a history-aware multimodal transformer architecture design following Chen et al. (2021a) as it achieves strong VLN performance across datasets, although it should be noted that our approach is compati- ble with most existing model architectures in VLN, Overall, as shown Figure 2, at each action pre- diction step, we have a transformer text encoder to encode the given language instruction, a hier- archical vision transformer to encode all the past observations of the agent, and another vision trans- former to encode the agent panoramic observation of the current step; then, the three types of represen- tations will be concatenated together and fed into a cross-modal transformer for joint encoding, and the final pooled representation is used for action prediction.
choose
For the current observations, in addition the CLIP features, with the relative angle of each view ia in the panoramic observation, represented as REL(w;) = (sinO;, cos0,, sind,, cosó;) where 0, and Φi are the relative heading and elevation angle to the agent's orientation, respectively. The combined representa- tions (CLIP((;:: REL(2;) [} are then fed into a transformer to obtain K processed representations.
also feed the model
History Features. The model also keeps track of its past observations with a hierarchical vision

---

transformer, where the panoramic observation at each step is first encoded by a single vector with a vision transformer, and all the panoramic repre- sentations are jointly encoded with another trans- former along the temporal dimension. We refer to Chen et al. (2021a) for details.
path modeling, however, the agent can actively col- lect a great amount of data given an environment for training. During the data collection period, we ask the agent to randomly choose the next view- point with equal probabilities at each step. Also, we keep track of all the visited viewpoints, and the agent is not allowed to visit the same viewpoint twice. To control the length of the paths, we uti- lize pre-computed statistics from the training data regarding agent paths. We then randomly select paths, ensuring that each path length sampled according to the distribution of path lengths ob- served in the training data. More sophisticated path collection techniques such as using techniques to encourage the diversity of sampled paths may also be used but here we leave it as future work. During path masking, we ensure that the last viewpoint is not masked so that the agent is always aware of the goal viewpoint.
Cross-Modal Interactions. The history features and the current observation features concate- nated as the vision modality, and dual-stream cross-modal fusion architecture used to encode both the vision and text modalities and allow for cross cross-moda modal information exchange. At each layer, we have a self-attention block for inter-modal inter- actions and a cross-attention block for vision-text interactions.
2.2 Masked Path Modeling
In this part, we illustrate the main idea of our proposed masked path modeling method. We go through the details of the active data collection, model architecture, and training strategies.
Model Architecture for MPM. As in Figure 2, the main difference between masked path mod- eling and the VLN objective is that the input of masked path modeling is a sequence of visual ob- servations instead of a natural language instruction. Therefore, we employ the same architecture as the original HAMT model except that we perform a linear transformation on the CLIP-encoded visual features as match the input and model dimen- sions, and then directly feed transformed features to the crossmodal transformer module. While col- lecting the visual features along a masked path, we do not use the panoramic view but only the view that the agent currently faces so as to make the pre- training task harder. All the module parameters are shared between the masked path modeling and VLN objectives.
General Framework. Masked path modeling is inspired by the masked data modeling pretrain- ing methods in the language and vision commu- nities (Devlin al., 2019; He et al., 2022), where the general idea is that the model is trained to reconstruct an original input (e.g.. a sentence or an image) given parts of the input masked. In VLN, we propose to first ask an agent to perform a sequence of actions and collect a path consist- ing of several viewpoints - -P1, P2. .Pn), then we mask x of the viewpoints in this path and feed the observations along the masked path Pm (Pm Pmai )Pme) the agent and the agent trained to perform the same sequence of actions as before to reconstruct the original path P given Pm- Note that different from the common masked data modeling methods, the impul and out- put modalities are different in MPM. Specifically, the model inputs are visual observations of the en- vironment while the model outputs are the agent actions. This design can explicitly train the model connect vision inputs and action outputs, which forms the foundation of the VLN task.
Training Strategies. We include our masked path modeling objective into the pretraining and finetuning stages of HAMT (Chen et al., 2021a). Concretely, during pretraining, the agent is jointly pretrained with masked path modeling and stan- dard objectives including masked language model- ing and instruction trajectory matching. We also include single-step action prediction and regres- sion (SAP/SAR), and spatial relationship predic- tion (SPREL) objectives as in Chen et al. (2021a).4
Data Collection. One of the major bottlenecks of VLN is the lack of training data and it is hard to col- lect large-scale in-domain data for VLN. In masked
'In the decoding side, the agent still receives panoramic views as in previous work (Fried et al., 2018b; Chen et al., 2021a).
We set x=25 in this paper and perform sensitivity analysis in the experiment section.
"We do not use the masked region modeling objective in HAMT because it requires distilling the knowledge of an

---

Validation Seen
Validation Unseen
Test Unseen
Model
TL
NEJ
SRt
SPL
TL
NE,
SR
SPL
TL
NEI
SR*
SPL
HAMT (Chen al., 2021a)
11.15
2.51
76
72
11.46
2.29
66
61
12.27
3.93
65
60
HAMT w/ Li and Bansal (2023)
62
65
60
HAMT+
11.11
2.65
75.02
71.75
11.93
3.34
67.05
61.69
12.70
3.57
67.19
61.94
HAMT+ w/ MPM
10.86
2.43
76.30
72.85
11.99
3.44
68.37
62.59
12.54
3.47
67.79
62.54
Table Results on the Room-to-Rowe too--omm-oooooooo dataset (Anderson al., 20)
We incorporate MPM into a strong baseline (HAMT+) and achieve significant improvements across settings. The hest scores are in bold.
Validation Seen
Validation Unseen
Model
NEJ
SR
CLSt nDTW1
sDTW1
NEG
SR1
CLSt
nDTW:
sDTW1
HAMT (Chen et al., 2021a)
6.09
44.6
57.7
50.3
31.8
HAMT+
4.62
57.29
67.97
61.01
41.96
5,90
44.75
61.84
54.18
33.89
HAMT+ w/ MPM
4.29
59.13
70.50
64.88
48.28
5.65
46.88
62.76
55.23
35.50
Table Results on the Koom-for-Room dataset (Jain et al., 2019). MPM can also improve the model performance in this setting across all the evaluation metrics. The best scores are in bold.
The SAP and SAR objectives ask the model to predict the next action based on instruction, his-
3.1 Settings
We go through the experimental settings in this part, including our used datasets, evaluation metrics, and implementation details.
tory from the ground-truth demonstration, and the current observation with imitation learning, where SAP formulates the task as a classification task while SAR trains the model to regress the action heading and elevation angles. The SPREL objec- tive trains the model predict the relative spatial position of two views in a panorama based on only visual features, angle features, or both types of fea- tures. We refer readers to Chen et al. (2021a) for more details.
3.1.1 Datasets
We evaluate the models on different types of VLN datasets, including the Room-to-Room (R2R) (An- derson et al., 2018c), Room-for- (R4R) (Jain
et al., 2019) and Room-across-Room (RxR) (Ku et al., 2020) datasets.
Then, during finetuning, the model is jointly trained with both masked path modeling and the VLN objective with equal loss weights.5 We com- bine the Asynchronous Advantage Actor-Critio (A3C) reinforcement learning objective (Mnih et al., 2016) and imitation learning objective for the VLN objective following previous work (Tan et al., 2019; Chen et al., 2021b), but only use the imitation learning objective for masked path mod- eling because it is stable and also it is non-trivial to design step-wise rewards in this setting.
R2R. The R2R dataset is built based on Matter- port3D (Chang et al., 2017) and has 7, 7,189 paths, with each path paired with 3 different English in- structions and the average length of all the paths is 29. R2R is split into training, validation, and test sets; the validation set consists of two splits: 1) val-seen, where all the paths are sampled from environments that are also seen the training set, and 2) val-unseen, where paths are sampled from environments that do not appear in the training set so as to test the generalization ability of agents, The paths in the test set are from new environments unseen in the training and validation sets.
3 Experiments
The R4R dataset is an algorithmically pro- duced extension of R2R that concatenates two adja- cent tail-to-head paths in R2R as well as their cor- responding instructions to form a new instruction- path pair. With this extension, R4R has longer paths and instructions, and the paths are not always the shorted path from the starting point to the goal, making the dataset less biased than R2R.
R4R.
In this section, we present our experimental results with the proposed masked path modeling objective.
image classification model while our vision encoder is CLIP. Also, the CLIP vision encoder frozen during pretraining to save computation time.
*We did not see significant performance differences when tuning the loss weights in our preliminary studies.

---

Validation Seen
Validation Unseen
Test Unseen
Model
SR1
SPL nDTWt sDTWt
SRT
SPLt nDTW sDTWt
SR1
SPL1 nDTWt sDTWt
HAMT (Chen al., 2021a)
59.4
58.9
65.3
50.9
56.5
56.0
63.1
48.3
53.12
46.62
59.94
45.19
HAMT+
63.93
59.93
68.59
55.47
62.00
58.05
67.52
53.87
HAMT+ w/ MPM
67.73
63.89
71.02
58.86
63.51
59.24
67.71
54.53
60.00
52.52
63.97
51.13
Table 3: Results on the Room-across across- Room dataset (Ku et al., 2020). The best scores are in bold.
R2R Validation Unseen
R4R Validation Unseen
Model
TL
N
SR
SPL1
NET
SR1
CLSt
nDTW
sDTW1
HAMT+ MPM
11.99
3.44
68.37
62.59
5.65
46.88
62.76
55.23 57.95
35,50 39.06
HAMT+ PPM--Pexpl
11.37
3.33
69.60
64.69
5.13
51.34
63.72
Table 4: Pre-exploring the test environments with MPM can further improve the model performance.
The RxR dataset follows the same environ- ment division as that in the R2R dataset. Different from R2R, RxR is a larger dataset that has 16,522 paths in total. In addition, the instructions are mul- tilingual and in three languages, including English, Hindi, and Telugu. The lengths of the instructions in RxR are also much larger than that in R2R (av- erage length: 78 vs. 29).
RxR.
use CLIP-ViT (Radford et al., 2021) pretrained vision encoder and is not finetuned during train- ing. Specifically, our model consists of a 9-layer text transformer, a 2-layer panoramic transformer for encoding history information, and a 4-layer transformer for cross-modal encoding. In each panoramic observation, there are K = 36 views of images and we use LLIP-VVT-L-336614 to encode the input images. We denote the HAMT baselinc with CLIP- VI--L-33--333/11 as HAMT+
3.1.2 Evaluation Metries
We adopt the standard evaluation metrics in VLN (Anderson et al., 20) to evaluate models. Specifically, we evaluate models with 1) trajectory lengths (TL.): the length of the agent path measured in meters; 2) navigation error (NE): the average dis- tance between the final position of agents and the goal position measured in meters; 3) success rate (SR): the proportion of agents whose final position is within three meters of the target; 4) success rate weighted by normalized inverse path length (SPL)): success rate normalized by the ratio between the length of the shortest path and the predicted path.
Pretraining. During pretraining, we randomly select proxy tasks including masked path model- ing for each mini-batch with a predefined ratio as in Chen et al. (2021a). Different from Chen et al. (2021a), the CLIP-ViT is frozen instead of finetuned during both pretraining and finetuning in order to save computational costs. We train the model for 200k iterations with the Adam opti- mizer (Loshchilov and Hutter, 2018) and the learn- ing rate is set to 5e-5 and the batch size is set to 64, take around 1 day to finish training on 4 NVIDIA Tesla V100 GPUs.
Because the above metrics are heavily biased towards whether or not the agent can reach the goal position while ignoring the specific path the agents take, Jain et al. (2019) propose the cover- age weighted by length score (CIS) metric that measures the path fidelity between the predicted path and target path for the R4R dataset. Simi- larly, Ku al. (2020) propose normalized dynamic time warping (nDTW) and success rate weighted by dynamic time warping (sDTW) (Magalhães et al., 2019) for RxR.
Finetuning. During finetuning, the model is jointly finetuned with the II+RR. and masked path modeling objectives with equal weights. The model is fine-tuned for 300k iterations with learn- ing rate of le-5 and batch size of on a single V100 GPU, taking around 2.5 days to finish. 6 The best model selected according to performance on the val unseen split. We use the same augmented data as Hong et al. (2021) following previous work for the R2R dataset, while no augmented data is used for other datasets. Greedy search is applied in inference following the single-run setting. In
3.1.3 Implementation Details
Model Architecture. We build our models upon the HAMT model (Chen et al., 2021a) and fol- low all of its parameter settings except that we
"Following the hyper-parameters in https://github:
s2..sh.

---

R2R Validation Unseen
R4R Validation Unseen
Path Design
NE,
SR1
SPL
NE,
SR
CLSt
nDTW
sDTWt
MPM w/ R2R Paths
11.99
3.44
68.37
62.59
5.74
45.39
61.42
54.42 55.23
33.92
MPM w/ R4R Paths
11.97
3.38
68.20
62.30
5.65
46.88
62.76
35.50
Table 5: MPM performs best when its collected paths resemble the paths of test environments. Here we only control the lengths of the paths to be similar the paths of either R2R R4R.
MPM
R2R
Validation Unseen
R4R Validation Unseen
Model
PT
FT
TL
NEL
SR1
SPI.t
NEJ
SRt
CLSt
nDTWf
sDTW
HAMT+
X
X
11.93
3.34
67.05
61.69
5.90
44.75
61.84
54.18
33.89
HAMT+
X
V
11.84
3,40
67.65
61.74
5.83
46.35
63.66
56.73
35.87
HAMT+
11.99
3.44
68.37
62.59
5.65
46.88
62.76
55.23
35.50
Table Including MPM during both pretraining (PT) and finetuning (FT) can generally lead to the best performance.
both pretraining and finetuning, the agent samples a batch of paths for MPM from the available envi- ronments in Matterport3D and we do not allow the agents to explore test environments.
baseline in success rates on the R2R, R4R, and RxR datasets respectively on the validation set, demonstrating the effectiveness of our approach. We attribute these improvements to the fact that our objective allows the model to be trained on variety of paths and can thus improve the generalization ability of the model in unseen environments.
3.2 Main Results
The main results of the baselines and our model are listed in Table 2, and 3. We report both the numbers in the HAMT paper (Chen al., 20) and our reproduced performance.
3.3 Analysis
In this part, we perform several analyses to gain insights into MPM. We leave more analysis results in Appendix.
First, it should be noted that because we use the strong CLIP vision encoder, our reproduced HAMT+ baseline can achieve better performance than the original HAMT paper across settings, even surpassing the state-of-the-arr end-to-end trained model (Li and Bansal, 2023). Especially, the RxR datasets, our HAMT+ outperforms HAMT by 4.53% and 4.57 sDTW on the validation seen split and 5.5% success rate and 4.42 sDTW on the validation unseen split.
Exploring Unseen Environments. Because we allow the agents to autonomously acquire data and learn without the need for annotated data, train the agents in scalable way. We hypothe- size that when trained with masked path modeling, the agent can be familiarized with the explored environments and thus improve its navigation per- formance, even though it not explicitly trained with the VLN objective. To verify this, we train the model with masked path modeling on unseen environments in the validation sets with the same before and test its VLN perfor- mance. As shown in Table 4, performing masked path modeling on unseen environments can signifi- cantly improve the model navigation performance, demonstrating the potential using the objective in a large-scale setting. Especially, exploring un- seen environments can bring 4.46% and 3.56 im- provements in SR and sDTW on the R4R validation unseen respectively.
Built upon a strong baseline, our model can still outperform it across settings. Notably, the perfor- mance gains are pronounced when measured with path fidelity metries (i.e., CLS, nDTW, sDTW) the long-horizon VLN datasets R4R and RxR in the seen enviromments, indicating that the masked path modeling objective can encourage the models to faithfully follow the natural language instructions and complete the paths accordingly. This intu- itive as the VLN training objectives can optimize the models towards taking the shortest path to reach the final goal, whereas during masked path model- ing, the model is trained to reconstruct the original paths, thus the model can follow the instructions more faithfully.
hyper-parametent
Path Design. When collecting the paths, we make the lengths of the sampled paths follow the distribution of that in the training data so that the paths are similar to the navigation paths. As shown
In unseen environments, our model achieves 1.32%, 2.13%, and 1.51 151% improvements over the

---

R2R Validation Unseen
REVERIE Validation Unseen
Model
TL
NET
SR
SPLt
OSR
SR1
CLS1
RGSt
RGSPL1
DUET (Chen al., 2022)
13.94
3.31
72
62
22.11 11
51.07
46.98
33.73
32.15
23.03
DUET MPM
13.37
3.05
72.84
62.09
21.35
52.51
49.11
35.64
32.52
23.48
Table MPM can flexibly applied to various models and datasets.
in Table 5, if we switch the path designs between R2R and R4R, the performance gains will drop marginally, indicating that making the paths be- tween MPM and VLN similar can best utilize the MPM objective for VLN. We leave more sophisti- cated path designs as future work.
vision-and-language pretraining with focus on its applications in VLN, as well as pretraining for control and embodied learning.
Vision-and-Languagg Navigation. Building -ang-uuuge navigation models has received increasing attention recent years (An- derson et al., 2018b; Fried et al., 201 8a; Wang et al., 2018; Li et al., 2019b; Zhu et al., 2020b; Kurita and Cho, 2021) and various benchmarks have been proposed to evaluate the ability of embodied agents to follow instructions and accomplish specified tasks (Kolve al., 2017; Anderson al., 2018a; Savva al., 2019; Anderson et al., 2018c; Chen et al., 2019; Ku al., 2020; Shridhar et al., 2020; Padmakumar et al., 2022). this line of research, representative works include Fried et al. (2018b) who propose panoramic action space and use speaker follow to synthesize additional training data. In addition, Tan et al. (2019) propose to mix imitation learning and A3C (Mnih et al., 2016) and increase the diversity of the synthesized data by adding noise into the environments during data generation. To utilize additional training signals, Ma et al. (2019) propose the self-monitoring agent that improves vision-language alignment with co-grounding module and progress monitor; Zhu et al. (2020a) propose four self-supervised auxiliary tasks that are beneficial for the task of VLN. There are also works on designing better architectures (Chen et al., 2021a, 2022) for VLN. In terms of data augmentation, addition to methods (Fried et al., 2018b; Tan et al., 2019; Hao et al., 2020; Dou and Peng, 2022) that synthetic language instructions are generated from randomly sampled paths, there are also works on synthesizing both the environments and the instructions (Wang et al., 2022b; Kamath et al., 2023). However, most of these works require external models to generate imperfect data, which can potentially harm the performance of VLN models.
MPM during Pretraining. During the pretrain- ing stage, we follow HAMT to train the models with masked language modeling, instruction tra- jectory matching, single-step action prediction and regression, and spatial relationship prediction tasks. We also choose to include the masked path model- to mitigate
ing objective during pretraining the difference between pretraining and finetuning. As shown in Table we can see that including masked path modeling important as it can well prepare the models for the finetuming stage, al- though only doing masked path modeling during finetuning can also bring marginal improvements. Notably, not including MPM during pretraining seems to achieve comparable even better perfor- mance than including it on R4R. One possible ex- planation is that during pretraining the path lengths are similar those of R2R, thus the pretrained agent may be more suitable for R2R than R4R.
Applications in Other Models and Datasets. Previously, we mainly experiment with the end- to-end HAMT model on navigation tasks with fine- grained language instructions. In this part, we im- plement MPM upon the strong DUET model (Chen et al., 2022). As shown in Table 7, MPM can still improve the model performance with this map- based model on both R2R and REVERIE, demon- strating the flexibilty of MPM. It is worth noting that REVERIE is a remote object grounding task while MPM mainly concerns with navigating to a goal viewpoint or image, thus the object grounding performance is not improved signficantly, and we leave this as future work.
4 Related Work
We overview three lines related research, in- cluding vision-and dadion-and-lanaae and-language navigation in general,
Vision-and-Langunn Pretraining. Pretraining models on large web corpora have proven to be

---

highly effective in natural language processing (Pe- al., 2018; Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020), and similar techniques have been applied in computer vision (Chen et al., 2020a, b02 He et al., 2022; Bao et al., 2022) and vision-language communities (Li et al., 2019a; Chen et al., 2020b; Radford et al., 2021; Kim et al.. 2021; Li et al., 2021; Dou et al., 2022). In the field of VIN,, researchers have tried multimodally pretrained language or vision repre- sentations (Anderson al., 201 8b; Li et al., 2019b). Notably, the CLIP vision encoder (Radford ct al., 2021) pretrained on large image-caption data has proven to be generally effective for vision-and- language embodied tasks (Khandelwal et al., 2022; Shen et al., 2022). To jointly learn transferrable vision and language representations for VLN, Ma- jumdar et al. (2020) and Guhur et al. (2021) pro- pose to first pretrain models on large image-caption data such as Conceptual Captions (Sharma al., 2018) and then adapt the representations for VLN tasks by finetuning the model on in-domain VLN data. While the pretrained representations can be useful, the pretraining process does not explicitly connect the learned representations to output ac- tions. To further integrate action generation into VLN pretraining, researchers have attempted to di- rectly use VLN data for pretraining. For example, PREVALENT (Hao et al., 2020) is pretrained on image-text-action triplets with a single-step action prediction objective and masked instruction model- ing objective; Chen et al. (2021a) further propose a single-step regression and spatial relationship pre- diction objective that introduces more supervisions. However, the pretraining data is limited by the size of VLN data and thus is difficult apply their approaches large-scale settings; Li and Bansal (2023) propose three proxy tasks, one of which is to reconstruct the trajectory semantic information given masked inputs, but they fail to connect vision and language modalities and train on self-collected data.
states; Nair al. (2022) pretrain representations on human ego-centric video data and adapt the representations on robotic tasks. In terms of us- ing self-collected data for training. Chaplot et al. (2021) propose to build 3D semantic maps and use them to improve action and perception skills, Similar to our work. Wu et al. (2023) propose to pretrain models by reconstructing full trajectory or given random subset of the same trajectory, and the pretrained models can be used for differ- ent downstream purposes like inverse dynamics, forward dynamics, imitation learning, offline RL, and representation learning. Different from these works, our method explicitly optimizes models for conditioned action generation and the agent can self-collect a rich amount of data for pretraining.
unimodally or
5 Conclusion
In this paper. we propose masked path modeling, a pretraining objective designed for vision-and- language navigation. The objective can utilize scal- able data explored from different environments to improve the agent's 's conditioned action generation ability. We incorporate the objective into a strong baseline and demonstrate improvements across dif- ferent settings. An analysis further reveals the po- tential for scaling the objective to large-scale set- tings. Future directions include designing better exploration strategies as well as investigating appli- cations in more fields.
Limitations
MPM is designed to connect vision and action modalities, while focusing less on the language modality. addition, we constrain the agent to collect paths from the available environments in Matterport3D, which can limit the diversity of the paths to some degree. We also focus solely on VLN settings and do not investigate its applications in other embodied instruction following tasks.
Acknowledgment
Pretraining for Control and Embodied Learn- ing. The general idea of pretraining on large- scale data has been adopted in embodied tasks, including data collection (Burda et al., 2019), rep- resentation learning (Yang and Nachum, 2021). and world model learning (Sco al., 2023). For example, Pathak al. (2017) present curiosity- driven supevvuee data collection approach that encourages agents to explore unfamiliarized
We thank anonymous reviewers for their insightful feedback. We also thank Cheng-Fu Fu Yang, Sidi Lu and other members from the UCLA NLP group for their helpful feedback and discussions. Zi- Yi is supported by the Amazon Fellowship and DARPA Machine Common Sense (MCS) program under Cooperative Agreement N66001-19-2-4032 and NIH RO1HL152270.

---

References
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi. Cordelia Schmid, and Ivan Laptev. 2022. Think global, act local: Dual-scale graph transformer for assion-and-languzee navigation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Peter Anderson, Aneel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. 2018a. 8a. On evaluation of cm- bodied navigation agents. arXiv preprint.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018b. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the Conference Com- puter Vision and Pattern Recognition (CVPR).
Ting Chen, Simon Kornblith, Mohammad Norozzi, and Geoffrey Hinton. 2020a. A simple framework for contrastive learning of visual representations. In Pro- ceedings of the International Conference on Machine Learning (ICML).
Xinlei Chen, Saining Xie, and Kaiming He. 2021b. An empirical study of training self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV).
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018c. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the Conference on Com- puter Vision and Pattern Recognition (CVPR).
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. UNITER: Universal image-text representation learning. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV).
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022. BEÎT: Bert pre-training of image transformers. Proceedings of the International Conference on Learning Representations (ICLR).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyamov. 2020. Unsupervised cross- cross-lingua lingual representation learning at scale. In Pro- ceedings of the Anmual Meeting of the Association for Computational Linguistics (ACL).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics (NAACL).
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. 2019. Exploration by random network dis- tillation. In Proceedings of the International Confer- ence on Learning Representations (ICLR).
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matter- port3d: Learning from RGB-D data in indoor envi- ronments. In Proceedings of the International Con- ference 3D Vision (3DV).
Zi-Yi Dou and Nanyun Peng. 2022. FOAM: A follower- aware speaker model for vision-and- lannnuagee naviga- tion. In Proceedings of the Conference of the North American Chapter of the Association for Computa- tional Linguistics (NAACL).
Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Russ R Salakhutdinov. 2021. SEAL: Self-supervised embodied active learn- ing using exploration and 3d consistency. In Pro- ceedings of the Conference on Neural Information Processing Systems (NeurIPS).
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang, Lijuan Wang. Chenguang Zhu. Pengchuan Zhang. Lu Yuan, Nanyun Peng. Zicheng Liu, and Michael Zeng. 2022. An empirical study of training mod-to-end vision-ande languag transform- ers. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Li Fei-Fei, Jia Deng, and Kai Li. 2009. ImageNet: Constructing large-scale image database. In Pro- ceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021a. History aware multimodal transformer for Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
Daniel Fried, Jacob Andreas, and Dan Klein, 2018a. Unified pragmatic models for generating and follow- ing instructions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
navigation. In

---

Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-PPillipe Morency, Taylor Kirg-KKikpattick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018b. Speaker-follower models for vision-and- and--angu navigation. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
Alexander Ku, Peter Anderson, Roma Patel, Eugene le. and Jason Baldridge. 2020. Room- across-room Mul- tilingual vision-and-languuge navigation with dense spatiotemporal grounding. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP).
Shuhei Kurita and Kyunghyun Cho. 2021. Generative language-grounde grounded policy in vision-and- and-languae navigation with bayes' rule. In Proceedings of the In- ternational Conference on Learning Representations (ICLR).
Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, and Cordelia Schmid. 2021. Airbert: In- domain pretraining for vision-and and- land-lag naviga- tion. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Jialu Li and Mohit Bansal. 2023. Improving vision-and- language navigation by generating future-view image semantics. In Proceedings of the Conference on Com- puter Vision and Pattern Recognition (CVPR).
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning generic agent for vision-and-lare ann--ang navigation via pre- training. In Proceedings of the Conference on Com- puler Vision and Pattern Recognition (CVPR).
Jialu Li, Hao Tan, and Mohit Bansal. 2022. EnvEdit: Environment editing for rision-and-language naviga- tion. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Dollár, and Ross Girshick. 2022. Masked autoen- coders are scalable vision learners. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Junnan Li, Ramprasaath Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Proceedings of the Conference on Neural Informa- tion Processing Systems (NeurIPS).
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez- Opazo, and Stephen Gould, 2021. A recurrent vision- and-language bert for navigation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019a. VisualBERT: A simple and performant baseline for vision and lan- guage. arXiv preprint.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene le, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in vision-and- language navigation. In Proceedings of the Annual Meeting of the Association for Computational Lin- guistics (ACL).
Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao. Noah A Smith, and Yejin Choi. 2019b. Robust navigation with language pretraining and stochastic sampling, In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Aishwarya Kamath, Peter Anderson, Su Wang. Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Ja- son Baldridge. and Zarana Parekh. 2023. A new path: Scaling dasion-and-laguual synthetic instructions and imitation learning. In Pro- ceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint.
navigation with
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Simple but effec- tive: CLIP embeddings for embodied AI. In Pro- ceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Ilya Loshchilov and Frank Hutter, 2018. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representa- tions (ICLR).
Chih- Yao Ma, Jiasen La, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019. Self-monitoring navigation agent via auxil- iary progress estimation. Proceedings of the In- ternational Conference on Learning Representations (ICLR).
Wonjae Kim, Bokyung Son, and IIdoo Kim. 2021. ViLT: Vision and language transformer without con- volution or region supervision. In Proceedings of the International Conference on Machine Learning (ICML).
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Van- derBilt, Luca Weihs, Alvaro Herrasti, Malt Deitke, Kiana Ebsani, Daniel Gordon, Yuke Zhu, et al. 2017. AI2-THOR: An interactive 3D environment for vi- sual AI. arXiv preprint.
Gabriel Magalhães, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. General evaluation for instruction conditioned navigation using dynamic time warping. In Workshop on Visually Grounded Interaction and Language (ViGIL).

---

Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. 2020. Im- proving vision-and- lannauag navigation with image- text pairs from the web. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV).
Sheng Shen, Liumian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. 2022. How much can CLIP benefit vision-and-languagg tasks? In Proceedings of the In- ternational Conference on Learning Reprexentations (ICIRR).
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves. Timothy Lillicrap, Tim Harley. David Silver, and Koray Kavukcuoglu. 2016. Asyn- chronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML).
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozheh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. 2022. R3M: universal visual representation for robot manipula- tion. In Proceedings of the Conference Robot Learning (CoRL).
Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learn- ing to navigate unseen environments: Back transla- tion with environmental dropout. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Narayan-Chen, Span- dana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022. TEACh: Task-driven em- bodied agents that chat. In Proceedings of the AAA! Conference on Artificial Intelligence (AAAI).
Hanging Wang. Wei Liang, Lue V Gool, and Wenguan Wang. 2022a. Towards versatile embodied naviga- tion. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the International Conference on Machine Learning (ICML).
Su Wang, Ceslee Montgomery, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, and Pe- ter Anderson. 2022b. Less is more: Generating grounded navigation instructions from landmarks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. 2018. Look before you leap: Bridging model-free and model-based reinforcement learning for planned- ahead vision-and and- languae nav- igation. In Proceedings of the European Conference on Computer Vision (ECCV).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference Machine Learning
Zun Wang, Jialu Li, Yicong Hong. Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. 2023. Scaling data generation in vision-and aooon-nd--lar lan-uugg navigation. In Proceedings s of the International Con- ference on Computer Vision (ICCV).
(ICML).
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019. Habitat: A platform for embodied AI research. In Proceedings of the International Conference Computer Vision (ICCV).
Philipp Wu, Arjun Majumdar, Kevin Stone, Yixin Lin, Igor Mordatch, Pieter Abbeel, and Aravind Ra- jeswaran. 2023. Masked trajectory models for predic- tion, representation, and control. In Proceedings of the International Conference on Machine Learning (ICML).
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. 2023. Masked world models for visual control. In Proceedings of the Conference on Robot Learning (CoRL).
Mengjiao Yang and Ofir Nachum. 2021. Representation matters: offline pretraining for sequential decision making. In Proceedings of the International Confer- ence on Machine Learning (ICML).
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the Annual Meet- ing of the Association for Computational Linguistics (ACL).
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020a. Vision-language navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the Conference on Computer Vision and Pateer Recognition (CVPR).

---

Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng. Vihan Jain, Eugene le, and Fei Sha. 2020b. Baby- walk: Going farther in vision-and-languuaee naviga- tion by taking baby steps. In Proceedings of the An- nual Meeting of the Association for Computational Linguistics (ACL).
A Additional Analysis Results
In this section, we present more analysis results to provide further insights into our proposed method.
(%6
74
Validation Seen
Validation Unseen
68
40
60
Mask Ratio (%)
Figure 3: Effect of the mask ratio of the MPM objective. The model generally robust to this with 25% achieving the best performance unseen environments.
Impact of the Mask Ratio. The mask ratio is hyperparameter in masked path modeling and investigate the optimal ratio in this paragraph. As shown in Figure 3, the objective is generally robust to the mask ratio, with 25% leading to the best performance and <=50% bringing improve- ments on the baseline. Therefore, we choose to randomly mask 25% of the viewpoints along path in this paper. It is worth noting that because we always keep the goal viewpoint mentioned in the method section, masking 100% of the paths is equivalent to an image-goal navigation task. Be- cause the model performance can be improved even with 100% viewpoints removed, the improvements of MPM can be partially attribute to multi-task learning, which has proven to be effective in Wang et al. (2022a).
Performance on Instructions of Different Lengths. We evaluate the performance both the baseline model and our proposed approach on instructions of varying lengths on the R2R valida- tion set. Figure 4 illustrates the results, indicating that the benefits of employing MPM are particu- larly significant when instructions are lengthy. We attribute this observation the fact that the sam- pled paths occasionally involve intricate navigation patterns. Consequently, the integration of MPM enables the model to effectively learn how to nav- igate and follow complex instructions. Similarly,

---

100
our
Bass
80
40
Lengths
Figure 4: Performance of models instructions of dif- ferent lengths on the R2R validation set. Our model can significantly improve the model performance, especially long instructions.
our method can achieve significant improvements when the instructions are relatively short possibly because the sampled paths are sometimes simple.
Qualitative Examples. We also sample some ex- amples from the R2R validation set with both the baseline model and model. As shown in Fig- ure and 6, our method can encourage the model to closely follow the language instructions even they are rather lengthy or concise. On the other hand, the baseline model fails to do so: can generate incorrect paths when the instructions are short and false and unnecessarily complicated paths when the instructions are long.

---

(a) Ground-truth
(b) Base
(c) Ours
Figure 5: Qualitative results for the instruction "Walk across patio, stop at hanging basket chair."
(a) Ground-truth
(b) Base
(c) Ours
Figure 6: Qualitative results for the instruction "Turn and walk towards the open brown wooden door that leads to office with a large desk. Exit room through door. Walk around the left side double open doors that leads to hallway. Walk out into the hallway until you reach the first door the right. Tur tight and take two steps into the room, stopping in the doorway to the room next to the carpet."
table and go through the


Reviews:
[{"review_id": "tAxff9xRbF", "reviewer": "Reviewer_rViw", "summary": "", "questions": "My questions are listed in the weakness of the paper.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback! We are encouraged to know that you find our paper strongly motivated and our idea interesting. Below we address your concerns:\n\n> More tasks applying MPM like REVERIE and more “X+MPM” comparison experiments like “DUET+MPM”.\n\nWe agree that MPM is a general paradigm and following your suggestions, we perform additional experiments on the REVERIE and R2R tasks with DUET as baseline with minimal parameter tuning. We list the model performance on the val-unseen set below. We can see that MPM can achieve good navigation improvements over the strong baseline on these two tasks, demonstrating the effectiveness of MPM. We will include the results in the revised version. Note that currently we only ask the model to reconstruct the masked path and do not ask the model to output object regions, thus the model grounding performance is not improved and we leave incorporating object supervision into MPM as a future direction.\n| **Model** | **TL** |  **NE** |  **SR** | **SPL**  |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| DUET | 13.94 | 3.31 | 72 | 60 |\n| DUET+MPM | 13.61 | **2.90** | **72.80** | **63.64** |\n\n| **Model** | **TL** |  **OSR** |  **SR** | **SPL**  | **RGS** |  **RGSPL** |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| DUET | 22.11 | 51.07  | 46.98 | 33.73 | **32.15** | **23.03** |\n| DUET+MPM | 22.54 |  **51.41** | **47.57** | **33.91** | 31.87 | 22.67 |\n\n>Confusing words/claims\n\nThank you for your suggestions! We will revise these words and claims carefully in the revised version. In the paper, “path dynamics” mainly refers to the ability to map a given path to a sequence of actions, which is different from previous works such as PREVALENT that generate pseudo instruction-pair data for data augmentation.\n\n>100\\% masking still improves the model performance\n\nThank you for pointing this out! To clarify, we always keep the last viewpoint of the collected path in all the settings. Therefore, the agent is still conditioned on the goal viewpoint in the 100\\% masking setting. We will make this clear in the revised version.\n\n\nWe hope the additional experiments and explanations can address your concerns and we are happy to conduct more experiments per your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is strongly motivated and well-written.\n2. The main idea is to directly use masked paths instead of instructions to guide agent navigation as paths are easier to get than instructions. This idea is interesting because path is a very different modality from natural language instructions.\n3. They achieve strong results with MPM on multiple VLN datasets and the authors provide some interesting analysis of the method.", "reasons_to_reject": "1. MPM is a very general objective for training navigation agents but the author only applies MPM to fine-grained instruction-following tasks. I'd like to see more tasks applying MPM like REVERIE.\n2. The improvements reported in the paper are mainly related to the data/training and not to the model, so I expected more \"X+MPM\" comparison experiments. Like DUET+MPM, EnvDrop+MPM to support the claim in the paper.\n3. Some words/claims in the paper are confusing for me. E.g. \"Path dynamics\" mentioned in the abstract isn't explained well in the main paper (even occurs). The widely-used VLN augmentation data Prevalent also collects \"diverse and substantial amount of data/paths\" as it samples all the possible paths from MP3D environments.\n4. Some experiment results are confusing. E.g. In Figure 3, 100% mask improves ~1% SR (from ~67% to ~68%). But in this case, the agent got no signals from the masked paths, which should introduce noise to the training process. I don't know why it can still improve agent's performance.\n\nWhile the paper presents some positive results of MPM compared to the baseline, I think the reason behind the observed improvements appears to need some exploration.", "questions_to_authors": "My questions are listed in the weakness of the paper.", "paper_topic_and_main_contributions": "This paper presents a new objective for VLN training: Mask Path Modeling (MPM), which is to predict the complete path given a randomly masked subpath, to address the data-scarcity problem for training VLN agents. MPM is applied to a history-aware multimodal transformer model in both pre-training and fine-tuning and gets a large performance gain, with +1.3%, +1.1%, and +1.2% SR on the val-unseen split of the R2R, R4R, and RxR datasets, respectively.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U3PfniQiQe", "reviewer": "Reviewer_Sdnd", "summary": "", "questions": "Please see Reasons to Reject", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback! We are glad that you find our paper well-written and our framework is simple and effective. Below we address your concerns:\n\n>Difference between Hao et al. (2020) and our work\n\nThank you for mentioning the relevant work! We would like to emphasize that while one of their objectives is to predict missing actions, their agent is **conditioned on both the instruction and paths** while our MPM objective only requires the model to be conditioned on **self-collected paths.** This design leads to significant differences: their model can only be trained on a limited number of parallel instruction-path pairs, whereas our MPM allows the model to utilize large-scale self-collected data. We have this discussion in the introduction (line 74-80) and related work sections (line 577-586) and we will expand the paragraphs.\n\n> Cross-modal perception in MPM\n\nWe agree that cross-modal perception is crucial in navigation. In MPM, the main objective is to connect the vision representations to the output actions with large-scale self-collected data and it is not focused on improving the vision-language interactions. **We believe that improving cross-modal interactions is orthogonal to our work and we leave it as our future work.**\n\nThat being said, our vision encoder is initialized with the CLIP model, which is trained with aligned image-text pairs, ensuring a good match between the model vision and language representations. **In addition, we did try adding a contrastive loss between the vision and language encoders.** Specifically, we use the CLIP-style contrastive losses to explicitly align the vision representations of the collected paths and the language representations of the input instructions. However, we did not see improvements with such an objective as shown below. We will include these results in the revised version as a reference for future work.\n\n\n| **Model** | **TL** |  **NE** |  **SR** | **SPL**  |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| HAMT+MPM | 11.99 | 3.44  | 68.37 | 62.59 |\n| HAMT+MPM+CLIP | 11.59  | 3.35 | 67.18 | 61.97 |\n\n\n\n> Extending to large-scale settings\n\nWe would first like to clarify our claim that previous methods are limited by the size of VLN data. Most of the previous works have to use human-annotated or machine-generated **instruction-path pairs**. Human-annotated data is hard to acquire and is therefore limited in quantity; machine-generated data can sometimes be of low quality. On the other hand, our method only requires the model to predict a self-collected path from the environment given some of its viewpoints masked. Therefore, the agent can collect **a large number of paths from the environment without using an external model** to generate data of questionable quality. We will explain this in detail in the revised version. **In 3.3, we see performance gains when the agent is allowed to explore unseen environments, demonstrating the potential of MPM given more downstream environments without the need to have human-annotated instruction-path pairs.**\n\nWe hope the additional experiments and explanations can address your concerns and we are happy to conduct more experiments per your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This paper is well-written and easy to follow.\n+ This MPM framework is simple yet effective and seems to be model agnostic with better scalability.", "reasons_to_reject": "+ The idea of mask-then-reconstruct for VLN has first been proposed by (Hao et al., 2020), where the difference is to predict missing language tokens or actions of the navigation. I am wondering if this achieves the EMNLP novelty bar.\n+ For MPM learning, it only considers the visual observation of the navigation path. How to improve the crucial cross-modal perception in this process? Does MPM just understand the action transformation but not the vision-and-language interaction?\n+ In L584, they mention that previous methods are limited by the size of VLN data. However, MPM also only collects randomly-sampled paths in the downstream environment. How does MPM extend to a large-scale setting as they claim?", "questions_to_authors": "Please see Reasons to Reject", "paper_topic_and_main_contributions": "This paper introduces masked path modeling (MPM) for vision-and-language navigation (VLN) to address the data scarcity limitation. They first explore and collect navigation paths in the downstream environments. MPM then learns to reconstruct the executed actions of the masked subpath. Experimental results show that MPM can effectively improve the navigator performance across all R2R, R4R, and RxR datasets.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[ECCV'20] Counterfactual Vision-and-Language Navigation via Adversarial Path Sampler, which also considers data augmentation for VLN.\n\n"}, "extra_text": {}}, {"review_id": "PhGcGR7H3h", "reviewer": "Reviewer_otNW", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We appreciate that you acknowledge the good performance of our model and that our paper is well-written. Below we address your concerns:\n\n>Difference between PREVALENT and ours\n\nThank you for pointing out the previous work! We would like to note that **PREVALENT requires an external model to generate pseudo language instructions** which can sometimes be of low quality. On the other hand, our method only requires the model to predict a self-collected path from the environment given some of its viewpoints masked. Therefore, the agent can collect a large number of paths from the environment without using an external model, ensuring the quality of the data. At each training step, we will sample a batch of paths for MPM and the number of the total sampled paths is 8*300K=2.4M. In addition, our pipeline is more flexible as we can collect paths in varying lengths (3.3 Path Design) whereas an external model can perform badly on out-of-domain paths (e.g. too long or short instructions).\n\nIn addition, **our work is complementary to PREVALENT** because following previous work, we also use the PREVALENT-generated data (line 414-415) and we can still see performance improvements across settings, demonstrating that our work is indeed complementary to PREVALENT and both of them can improve the model performance.\n\nWe will include this discussion in the revised version.\n\n>Masked Trajectory Modeling (MTM) in Li & Bansal (2023)\n\nThank you for mentioning the work! We will cite this paper and compare with it in the revised version, and we would like to note that:\n\n- We respectfully disagree with summarizing our work as “simply adding a pretraining/finetuning objective”. In the introduction part, we well-motivated our work as pointed out by Reviewer rViw and emphasized that previous works relied on limited VLN data whereas our work can utilize large-scale self-collected data on conditional action generation (line 85-111), which none of the previous works have proposed before. \n\n- Similar to PREVALENT that also employs an MTM-like objective, their agent is **conditioned on both the instruction and paths** while our MPM objective only requires the model to be conditioned on **self-collected paths.** This design leads to significant differences: their model can only be trained on parallel instruction-path data which is limited, whereas our MPM allows the model to utilize large-scale self-collected data. We have this discussion in the Intro (line 74-80) and related work sections (line 577-586) and we will expand the paragraphs.\n\n- **Our model can outperform Li & Bansal (2023).** Following your suggestions, we perform additional experiments with DUET as the base model and below is the model performance on the R2R val-unseen set with minimal parameter tuning. We can see that our model outperforms Li & Bansal (2023). Considering that Li & Bansal (2023) have multiple pretraining objectives included (i.e., MTM, MPM, APIG), the performance gains demonstrated the advantages of MPM over previous work\n\n| **Model** | **TL** |  **NE** |  **SR** | **SPL**  |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| DUET | 13.94 | 3.31 | 72 | 60 |\n| DUET + Li & Bansal (2023) | - | - | 72| 62 |\n| DUET+MPM | 13.61 | **2.90** | **72.80** | **63.64** |\n\n\n- We will cite and discuss the paper in detail but per the *ACL policies (https://www.aclweb.org/adminwiki/index.php?title=ACL_Policies_for_Submission,_Review_and_Citation), **papers appearing less than 3 months before the submission deadline should be considered contemporaneous to the submission**, and Li & Bansal (2023) is posted ~2 months before the submission date. \n\n> Stronger Baselines\n\nThank you for your suggestions! We would like to note that:\n\n- Our work is based on an improved version of HAMT (referred to as HAMT+ in the paper, line 428-435). As reported in the paper, **compared with SoTA models without using the map information (e.g. Li & Bansal (2023)), HAMT+ can achieve comparable or even better performance.** Therefore, **our baseline is indeed strong** and improvements over this strong baseline should be convincing.\n\n| **Model** |  **Val Unseen SR** | **Val Unseen SPL**  | **Test Unseen SR** | **Test Unseen SPL**  |\n| :--------: | :--------: | :--------: | :--------: | :--------: | \n| HAMT| 66 | 61 | 65 | 60 |\n| Li & Bansal (2023)|  **68** | **62** | 65 |60 |\n| HAMT+ | 67 | **62** | **67** | **62** |\n\n- Following suggestions, **we conduct additional experiments on R2R and REVERIE with DUET as the base model.** We list the model performance on the val-unseen set below. We can see navigation performance improvements of MPM over the strong baseline on these two tasks, demonstrating the effectiveness of MPM. We will include the results in the revised version. Note that currently we only ask the model to reconstruct the masked path and do not ask the model to output object regions, thus the model grounding performance is not improved and we leave incorporating object supervision into MPM as a future direction.\n\n| **Model** | **TL** |  **NE** |  **SR** | **SPL**  |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| DUET | 13.94 | 3.31 | 72 | 60 |\n| DUET+MPM | 13.61 | **2.90** | **72.80** | **63.64** |\n\n| **Model** | **TL** |  **OSR** |  **SR** | **SPL**  | **RGS** |  **RGSPL** |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| DUET | 22.11 | 51.07  | 46.98 | 33.73 | **32.15** | **23.03** |\n| DUET+MPM | 22.54 |  **51.41** | **47.57** | **33.91** | 31.87 | 22.67 |\n\n\nWe hope the additional experiments and explanations can address your concerns and we are happy to conduct more experiments per your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good performance on multiple VLN datasets. \n2. The paper is well-written and easy to follow.\n3. Reasonable approach to improve the performance.", "reasons_to_reject": "1. The data collection pipeline is not new. In PREVALENT, they traverse through all the possible paths (with length ranging from 3-6) in the seen environments. Furthermore, PREVALENT generates synthetic instructions for all those unannotated paths. Given the data collection process mentioned in the paper, I don't think any new paths are actually created, thus I'm not sure why MPM helps reduce the data scarcity problem in VLN. Besides, no stats about the collected paths are given in the paper. \n2. Simply adding a pre-training/fine-tuning objective is not novel enough. Similar ideas that optimize the self-supervised image-level loss are explored in \"Improving Vision-and-Language Navigation by Generating Future-View Image Semantics\", where they propose Masked Trajectory Modeling (MTM) that masks out views in a trajectory. \n3. HAMT is a sota approach in 2021. To demonstrate the effectiveness of the proposed approach (i.e., could generalize to different model architecture/training strategy), at least MPM should be applied to \"Think Global, Act Local: Dual-Scale Graph Transformer for Vision-and-Language Navigation\" (CVPR 2022).", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper proposes Masked Path Modeling to improve VLN performance. Specifically, this paper uses masked paths to replace the instructions and optimize the single action prediction training objective. Empirically, on R2R, R4R, and RxR datasets, the proposed approach improves the baseline method by around 1-2% in success rate.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new auxiliary task for vision-and-language navigation, and achieves effective improvement over the baseline. I especially appreciate the honesty in both the paper (e.g., using a stronger baseline with latest CLIP model) and the discussions (e.g., acknowledge that 100% masking = img-goal navigation). Overall, most of the reviewer's concern are resolved during the discussion. 

P.s., I encourage the authors to incorporate the proposed changes in their final summary, and also resolve the confusion raised by reviewers. Some confusions come from writing (not from results). For these, I did not see a plan of changes in the final summary but I think that these clarifications are important. A paper is born for readers to learn, thus clear writing is also important as the strong results.
Decision: Accept-Findings