Venue: NAACL 2022

# Diagnosing Vision-and-Language Navigation: What Really Matters

Wanrong Zhu\({}^{\sharp}\), Yuankai Qi\({}^{\lx@sectionsign}\), Pradyumna Narayana\({}^{*}\), Kazoo Sone\({}^{*}\), Sugato Basu\({}^{*}\), Eric Xin Wang\({}^{\ddagger}\), Qi Wu\({}^{\lx@sectionsign}\), Miguel Eckstein\({}^{\sharp}\), William Yang Wang\({}^{\sharp}\)UC Santa Barbara, \({}^{\lx@sectionsign}\)University of Adelaide, \({}^{*}\)Google, \({}^{\ddagger}\)UC Santa Cruz

{wanrongzhu,migueleckstein,wangwilliamyang}@ucsb.edu, qykhsr@gmail.com

{pradyn,song,sugato}@google.com, xwang366@ucsc.edu, qi.wu01@adelaide.edu.au

###### Abstract

Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines' performance and human benchmarks. Moreover, the agents' inner mechanisms for navigation decisions remain unclear. To the best of our knowledge, how the agents perceive the multimodal input is under-studied and needs investigation. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation. Results show that indoor navigation agents refer to both object and direction tokens when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. Transformer-based agents acquire a better cross-modal understanding of objects and display strong numerical reasoning ability than non-Transformer-based agents. When it comes to vision-and-language alignments, many models claim that they can align object tokens with specific visual targets. We find unbalanced attention on the vision and text input and doubt the reliability of such cross-modal alignments.1

Footnote 1: Code and data used in this study are available at [https://github.com/NegB/Diagnose_VLN](https://github.com/NegB/Diagnose_VLN).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5981 - 5993

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

A key challenge for Artificial Intelligence (AI) research is to move beyond Independent and Identically Distributed (i.i.d.) data analysis: We need to teach AI agents to understand multimodal input data, and jointly learn to reason and perform incremental and dynamic decision-making with the help from humans. Vision-and-Language Navigation (VLN) has received much attention due to its active perception and multimodal grounding setting, dynamic decision-making nature, rich applications, and accurate evaluation of agents' performances in language-guided visual grounding. As the AI research community gradually shifts its attention from the static empirical analysis of datasets to more challenging settings that require incremental decision-making processes, the interactive task of VLN deserves a more in-depth analysis of why it works and how it works.

Various setups have been proposed to address to the VLN task. Researchers generate visual trajectories and collect human-annotated instructions for indoor Anderson et al. (2018); Jain et al. (2019a); Ku et al. (2020a); Chen et al. (2021) and outdoor environment Chen et al. (2019); Mehta et al. (2020); Mirowski et al. (2018). There are also interactive VLN settings based on dialogues Nguyen et al. (2019); Nguyen and III (2019); Zhu et al. (2020c), and task that navigates agents to localize a remote object Qi et al. (2020c). However, few studies ask the _Why_ and _How_ questions: Why do these agents work (or do not work)? How do agents make decisions in different setups?

Through the years, agents with different model architectures and training mechanisms have been proposed for indoor VLN Anderson et al. (2018); Fried et al. (2018); Hao et al. (2020); Hong et al. (2020a, b); Huang et al. (2019); Ke et al. (2019); Li et al. (2019); Ma et al. (2019); Qi et al. (2020b); Tan

\begin{table}
\begin{tabular}{l r r r} \hline  & **R2R** & **RxR** & **Touchdown** \\ \hline Human Performance & 86 & 94 & 92 \\ SoTA Model Performance & 78 & 53 & 17 \\ \hline \end{tabular}
\end{table}
Table 1: There exists salient gaps between machines’ vision-and-language navigation (VLN) performance and human benchmarks. Navigation success rates are reported on the R2R Anderson et al. (2018) and the RxR dataset Ku et al. (2020b) for indoor VLN and the Touchdown dataset Chen et al. (2019) for outdoor VLN.2

et al., 2019; Wang et al., 2020a, 2019, 2018, 2020b; Zhu et al., 2020a) and outdoor VLN Chen et al. (2019); Ma et al. (2019); Mirowski et al. (2018); Xia et al. (2020); Xiang et al. (2020); Zhu et al. (2020b). Back-translation eases the urgent problem of data scarcity Fried et al. (2018). Imitation learning and reinforcement learning enhance agents' generalization ability Wang et al. (2019, 2018). With the rise of BERT-based models, researchers also apply Transformer and pre-training to further improve navigation performance Hao et al. (2020); Hong et al. (2020); Zhu et al. (2020b). While applying new techniques to the navigation agents might boost their performance, we still know little about how agents make each turning decision. Treatment of the agents' processing of instructions and perception of the visual environment as a black box might hinder the design of a generic model that fully understands visual and textual input regardless of VLN setups. Table 1 shows non-negligible performance gaps between neural agents and humans on both indoor and outdoor VLN tasks.

Therefore, we focus on analyzing how the navigation agents understand the multimodal input data in this work. We conduct our investigation from the perspectives of natural language instruction, visual environment, and the interpretation of vision-language alignment. We create counterfactual interventions to alter the instructions and the visual environment in the validation dataset, focusing on variables related to objects, directions and numerics. More specifically, we modify the instruction by removing or replacing the object/direction/numeric tokens, and we adjust the environment by masking out visual instances or horizontally flipping the viewpoint images. Subsequently, we examine the interventions' treatment effects on agents' evaluation performance while keeping other variables unchanged. We set up experiments on the R2R Anderson et al. (2018) and the RxR dataset Ku et al. (2020) for indoor VLN and the Touchdown dataset Chen et al. (2019) for outdoor VLN. We examine nine VLN agents on the three datasets with quantitative ablation diagnostics on the text and visual inputs.

In summary, our key findings include:

1. Indoor navigation agents refer to both objects and directions in the instruction when making decisions. In contrast, outdoor navigation agents heavily rely on directions and poorly understand visual objects. (Section 4)
2. Instead of merely staring at surrounding objects, indoor navigation agents are able to set their sights on objects further from the current viewpoint. (Section 5)
3. Transformer-based agents display stronger numerical reasoning ability (Section 4), and acquire better cross-modal understanding of objects, compared to non-Transformer-based agents. (Section 6)
4. Indoor agents can align object tokens to certain targets in the visual environment to a certain extent, but display in-balanced attention on text and visual input. (Section 6)

We hope these findings reveal opportunities and obstacles of current VLN models and lead to new research directions.

## 2 Related Work

Instruction Followingis a long-standing topic in AI studies that ask an agent to follow natural language instructions and accomplish target tasks, which can be dated back to the SHRLDU Winograd (1971). Efforts made to tackle this classic problem spans from defining templates Klingspor et al. (1997); Antoniol et al. (2011), designing hard-encoded concepts to ground visual attributes and spatial relations Steels and Vogt (1997); Roy (2002); Guadarrama et al. (2013); Kollar et al. (2013); Matuszek et al. (2014), to constructing varies datasets and learning environments Anderson et al. (1991); Bisk et al. (2016); Misra et al. (2018). Many methods have been proposed to map the instructions into sequence of actions, such as reinforcement learning Branavan et al. (2009, 2010); Vogel and Jurafsky (2010); Misra et al. (2017), semantic parsing Chen and Mooney (2011); Artzi and Zettlemoyer (2013), alignment-based model Andreas Klein (2015), and neural networks Bisk et al. (2016); Mei et al. (2016); Tan and Bansal (2018).

Vision-and-Language Navigationis a task where an agent comprehends the natural language instructions and reasons through the visual environment. Many studies aim at improving VLN agents' performance in one way or another. To enrich training data, a line of work Fried et al. (2018); Zhu et al. (2020b) use back-translation to generate augmented instructions. To enforce cross-modal grounding, RPA and RCM Wang et al. (2018, 2019) use reinforcement learning, SMNA Ma et al. (2019) uses a visual-textual co-grounding module to improve cross-modal alignment, Rel 

Graph Hong et al. (2020) uses graphs for task formulation. To address the generalizability problem to unseen environment, PRESS Li et al. (2019) introduces a stochastic sampling scheme, EnvDrop Tan et al. (2019) proposes environment dropout. To utilize visual information from the environment, AuxRN Zhu et al. (2020) uses auxiliary tasks to assist semantic information extraction, VLN-HAMT Chen et al. (2021) incorporates panorama history with a hierarchical vision transformer. FAST Ke et al. (2019) makes use of asynchronous search and allows the agent to backtrack if it discerns a mistake after attending to global and local knowledge. With the success of BERT-related models in NLP, researchers also start to build Transformer-based navigation agents and add a pre-training process before fine-tuning on the downstream VLN task Hao et al. (2020); Hong et al. (2020); Zhu et al. (2020); Chen et al. (2021).

Model Behavior AnalysisAs multimodal studies gain more and more attention, there are lines of works that focus on explaining models' behaviors to better understand and handle the tasks. Some generate textual explanations by training another model to mimic human explanations Hendricks et al. (2016); Park et al. (2018); Wu and Mooney (2019). Others generate visual explanations with the help of attention mechanism Lu et al. (2016) or gradient analysis Selvaraju et al. (2017). There are also attempts to provide multimodal explanations, e.g., Li et al. (2018) breaks up the end-to-end VQA process and examines the intermediate results by extracting attributes from the visual instances. Another line of work examines model performance by conducting ablation studies on input data. Recent analyses on language modelling O'Connor and Andreas (2021), machine translation Fernandes et al. (2021), and instruction following Dan et al. (2021) ablate/perturb both training and validation data. A study on multimodal models Frank et al. (2021) only applies ablation during evaluation, which is the same as our settings.

## 3 Background and Research Questions

We first bring in the task of Vision-and-Language Navigation and introduce the datasets and agents used for comparison. Then we list out the research questions to study in this work.

### Vision-and-Language Navigation

In the vision-and-language navigation task, the navigation agent is asked to find the path to reach the target location following the instructions \({\cal X}\). The navigation procedure can be viewed as a sequential decision-making process. At each time step \(t\), the visual environment presents an image view \(\mathbf{v}_{t}\). With reference to the instruction \({\cal X}\) and the visual view \(\mathbf{v}_{t}\), the agent is expected to choose an action \(a_{t}\) such as _turn left_ or _stop_.

DatasetsWe conduct indoor navigation experiments on the Room-to-Room R2R dataset Anderson et al. (2018) and the Room-across-Room RxRdataset Ku et al. (2020), and test outdoor VLN on Touchdown Chen et al. (2019). R2R and RxR are built upon real estate layouts and contain separate graphs for each apartment/house. Unlike R2R, which shoots for the shortest path, RxR has longer and more variable paths. R2R only contains English instructions, while RxR also includes instructions in Hindi and Telugu. In this study, we only cover the English subset for RxR, and will refer to it as RxR-en in the following sections. Navigation in Touchdown occurs in the urban environment, where the viewpoints form a huge connected graph. Compared to indoor environments, Touchdown has more complicated visual environments and a more extensive search space. The evaluation results are reported on the validation unseen sets for R2R and RxR-en and on the test set for Touchdown.

ModelsTable 2 lists out the models covered in our study. We use the code and trained checkpoints shared by the authors in the following experiments.

For indoor navigation on R2R, we study a widely adopted base model Envdrop Tan et al. (2019), a backtracking framework for self-correction FAST Ke et al. (2019), and two SoTA models VLN \(\circlearrow\) BERT Hong et al. (2020) and PREVA

\begin{table}
\begin{tabular}{l l c c} \hline \multicolumn{1}{c}{**Dataset**} & **Model** & \multicolumn{1}{c}{**Trans?**} & **Visual Feature** \\ \hline \multirow{4}{*}{R2R} & EnvDrop Tan et al. (2019) & \(\times\) & \multirow{4}{*}{ResNet-152} \\  & FAST Lee et al. (2019) & \(\times\) & & \\  & VLN + BERT Hong et al. (2020) & ✓ & \\ \cline{1-1}  & PREVALENT Han et al. (2020) & ✓ & \\ \hline \multirow{2}{*}{RuR-en} & CLIP-ViM Shen et al. (2021) & \(\times\) & \multirow{2}{*}{CLIP-VNT} \\  & VLA-HAMT Chen et al. (2021) & ✓ & \\ \hline \multirow{4}{*}{Touchdown} & RONCAT Chen et al. (2019) & \(\times\) & \multirow{4}{*}{ResNet-18} \\  & ARC Xiang et al. (2020) & \(\times\) & \\ \cline{1-1}  & VLN-Transformer Zhu et al. (2020) & ✓ & \\ \hline \end{tabular}
\end{table}
Table 2: The VLN datasets and models covered in this study. We record whether the model structure is Transformer-based, and the pre-trained feature extractor used to encode visual environment.

 

LENT Hao et al. (2020). The Envdrop introduces environment dropout on top of the Speaker-Follower Fried et al. (2018) model, FAST conducts an asynchronous search for backtracking, PREVALENT, and VLN BERT are Transformer-based agents with pre-trained models.

For navigation on RxR-en, we examine CLIP-ViL Shen et al. (2021) and VLN-HAMT Chen et al. (2021). CLIP-ViL shares the same model structure with EnvDrop. The only difference is that CLIP-ViL uses CLIP-ViT Radford et al. (2021) to extract visual features, while EnvDrop uses ImageNet ResNet Szegedy et al. (2017) features. VLN-HAMT incorporates a long-horizon history into decision-making by encoding all the past panoramic observations via a hierarchical vision Transformer.

For outdoor navigation on Touchdown, we consider the common baseline RCONCAT Chen et al. (2019), and two SoTA models ARC Xiang et al. (2020) and VLN-Transfomer Zhu et al. (2020b). RCONCAT encodes the trajectory and the instruction in an LSTM-based manner. ARC improves RCONCAT by paying special attention to the stop signals. VLN-Transfomer is a Transformer-based agent that applies pre-training on an external dataset for outdoor navigation in urban areas.

MetricsIn the following experiments, we evaluate navigation performance with Success Rate (SR) for indoor agents and Task Completion (TC) rate for outdoor agents. Both SR and TC measure the accuracy of completing the navigation task, reflecting the agents' overall ability to finish navigation correctly. An indoor navigation task is considered complete if the agent's final position locates within 3 meters of the target location. For outdoor navigation, the task is considered complete if the agent stops at the target location or one of its adjacent nodes in the environment graph.

### Research Questions

Current VLN studies have reached their bottleneck as only minor performance improvements have been achieved recently, while a significant gap still exists between machine and human performance. This motivates us to find the reasons.

To better understand how VLN agents make decisions during navigation, we conduct a series of experiments on indoor and outdoor VLN tasks, aiming to answer the following questions that might help us locate the deficiencies of current model designs and explore future research directions:

1. _What can the agents learn from the instructions? Do they pay more attention to object tokens or directions tokens? Do they have the ability to count? (Section 4)_
2. _What do agents see in the visual environment? Are they staring at the closely surrounded objects or also browsing further layout? Do they focus on individual visual instances or perceive the overall outline? (Section 5)_
3. _Can agents match textual tokens to visual entities? How reliable are such connections? (Section 6)_

## 4 Analysis on Instruction Understanding

This section examines whether and to what extent the agent understands VLN instructions. We focus on how the agent perceives object-related tokens, direction-related tokens, and numeric tokens, and their effects on final navigation performance. Table 3 shows exemplar instructions of the three

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Dataset** & **\#Data** & **\(\overline{L_{\text{sent}}}\)** & **\(\overline{L_{\text{sent}}}\)** & **\#Object** & \(p(\textbf{tok}_{\text{init}})\) & \(p(\textbf{tok}_{\text{free}})\) \\ \hline R2R & 2.3k & 6.0 & 29.3 & 0.6k & 19.8\% & 7.3\% \\ RvR-en & 4.6k & 8.5 & 111.3 & 1.4k & 16.1\% & 6.5\% \\ Touchdown & 1.4k & 34.4 & 92.5 & 1.0k & 16.8\% & 6.8\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Statistics of R2R, RxR-en and Touchdown datasets. #Data is the dataset size used for evaluation in this study. \(\overline{L_{\text{path}}}\) is the average path length, which is the number of viewpoints covered in the trajectory. \(\overline{L_{\text{instr}}}\) is the average instruction length. #Object denotes the number of unique objects mentioned in the instructions. \(p(\texttt{tok}_{\text{obj}})\) and \(p(\texttt{tok}_{\text{direct}})\) indicates the percentage of object/direction tokens per instruction.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Dataset** & **Instruction** \\ \hline \multirow{3}{*}{R2R} & Walk through the door by the sink into the middle of the next room. Turn right and walk down the hallway and enter the third door on your right. \\ \cline{2-2}  & We’re facing towards a small picture that’s attached to the wall, turns slightly to the left, after endure the hallway but’s in front of you, turn to the left, take it steps further.. On your right there are a few glass doors and on the left there’s a living room, walk towards the living room, turn slightly to the left. On the right there are four chairs and a beautiful coffee table in the middle, on the left there’s a console table with a vase with flowers on top, walk part the conable table towards the back of the chair that’s in microfactly out of you...We now close various days a lamp, and on the right there’s a smoke console table with decorations on top, and thus vary ‘constantization. \\ \hline \multirow{3}{*}{Torchdown} & Orient yourself so that you are moving in the same direction as traffic. Go straight through 3 intersections: Keep moving, forward, after the slot intersection, you should act as drag for a store to win a whole background and red dots as well as a red and white bullwge target. Cointing single straight puts litters and at the next intersection, turn left go through these intersection and drop just after the wall on your left with the purple zig mag patterns.

 

datasets covered in our study. As shown in Table 4, Touchdown's trajectory length is significantly longer than the other two indoor datasets. RxR-en and Touchdown have longer instructions than R2R. The ratios of object and direction tokens in all three datasets are comparable, involving about two times more object tokens than direction tokens.

### The Effect of Object-related Tokens

We first create counterfactual interventions on instructions by masking out the object tokens. We use Stanza Qi et al. (2020) part-of-speech (POS) tagger to locate object-related tokens. A token will be regarded as an object token if its POS tag is _NOUN_ or _PROPN_. During masking, we replace the object token with a specified mask token _[MASK]_. Then we examine the average treatment effects of the intervention on agents' performance, while keeping other variables unchanged.

Noticeably, when we mask out the object tokens, the number of visible tokens in the provided instruction also decreases, which is a coherent factor with masking object tokens and might interfere with our analysis. To eliminate the effect of reducing visible tokens, we add a controlled trial in which we randomly mask out the same amount of tokens. Table 5 gives an example of masking object tokens (#2) and its corresponding controlled trial (#4).

We follow each agent's original experiment setting for all the experiments in this study and train it on the original train set. Then we apply masking to object tokens in the validation set, and report agents' relative performance changes under each setting. We conduct five repetitive experiments and report the average scores for settings that involve random masking or replacing.

Table 6 presents how the agents' navigation performance change when object tokens are masked out (#2 & #3). Intuitively, not knowing what objects are mentioned in the instruction lowers all models' performance. Comparing the masking ablations with the controlled trial for indoor VLN, we notice that masking out the object tokens result in a more drastic decrease in success rate than masking out random tokens. This holds for all indoor agents, which verifies that indoor agents depend on object tokens more than other tokens. However, when we compare results on the Touchdown for outdoor VLN, we notice in surprise that masking out the object tokens has a weaker impact on task completion rate than masking out random tokens. This suggests that current outdoor navigation agents do not fully take object tokens into consideration when making decisions. This may be caused by the weak visual recognition module in current outdoor agents. As addressed in Table 2, all three outdoor

\begin{table}
\begin{tabular}{c|l|c c c c c|c c c c} \hline \multirow{2}{*}{**\(\ast\) Ablation**} & \multirow{2}{*}{**Setting**} & \multicolumn{8}{c|}{**SR \(\uparrow\) on RxR**} & \multicolumn{2}{c|}{**SR \(\uparrow\) on RxR-en**} & \multicolumn{2}{c}{**TC \(\uparrow\) on Touchdown**} \\ \cline{3-11}  & & & EnvDrop & FAST & VLN\textbackslash{BERT} & PREVALENT & CLIPVIL & HAMT & ROONCAT & ARC & VLNTrans \\ \hline
1 & – & Vanilla & 49.77 & 63.90 & 53.30 & 57.13 & 40.21 & 52.52 & 11.78 & 15.19 & 16.11 \\ \hline
2 & \multirow{2}{*}{Object} & Mask Controlled Trial & -36\% & -38\% & -21\% & -20\% & -48\% & -32\% & -34\% & -36\% & -6\% \\  & & Controlled Trial & -31\% & -26\% & -8\% & -8\% & -35\% & -23\% & -44\% & -55\% & -15\% \\ \hline
4 & \multirow{2}{*}{Direction} & Mask & -23\% & -23\% & -15\% & -11\% & -39\% & -28\% & -73\% & -90\% & -45\% \\  & & Controlled Trial & -11\% & -12\% & -4\% & -3\% & -14\% & -9\% & -22\% & -23\% & -8\% \\ \hline \end{tabular}
\end{table}
Table 6: The navigation performance for indoor and outdoor agents on object-token and direction-token ablations. We record the validation score in the “vanilla” setting, and report the relative performance change for each ablation setting. For object-token ablations, the “mask” setting masks out all the object-tokens, while the controlled trial masks out the same amount of random tokens. The same applies to direction-token ablations.

\begin{table}
\begin{tabular}{p{34.1pt} p{34.1pt}} \multicolumn{2}{p{34.1pt}}{**Instruction**} \\ \hline
1 & Vanilla & Go _left_ down the **hallway** toward the **exit sign.** Go into the **door** on the _left_ and _stop_ by the **table**. \\ \hline
2 & Mask Object Tokens & Go left down the [MASK] towards the [MASK]. To introduce the [MASK] on the left and stop by the [MASK]. \\
3 & Replace Object Tokens & Go let down the **portrait** toward the **gda Mountains**. Go into the **football** on the left and stop by the **heads. \\
4 & Controlled Trial & Go [MASK] down the hallway [MASK] the exit sign. To into the door on [MASK] left and [MASK] by [MASK] table. \\
5 & Mask Direction Tokens & Go _[MASK]_ down the hallway toward the exit sign. Go into the door on the _right_ and _/_MASK_ by the table. \\
6 & Replace Direction Tokens & Go _Right_ down the hallway toward the exit sign. Going into the door on the _right_ and _/_forward_ by the table. \\
7 & Controlled Trial & Go left down the _[MASK]_/_[MASK]_ the exit sign. Go into the door on the left and _/_MASK_ by the table. \\ \end{tabular}
\end{table}
Table 5: Example of instruction modification. In the original instruction, there are five **object-related tokens**, and three _direction-related tokens_. In the object token ablations, we mask out the object tokens, or replace them with randomly sampled object tokens. The controlled trial randomly masked out five tokens from the instruction for a fair comparison. Likewise the direction tokens.

 

agents rely on visual features extracted by ResNet-18, which may not be powerful enough to fully incorporate the complicated urban environments.

### The Effect of Direction-related Tokens

We regard the following tokens as direction-related tokens: _left, right, back, front, forward, stop_. Similar to how we ablate the object tokens, we mask out direction tokens from the instruction and examine the impact on agents' navigation performance. Table 5 provides examples of direction tokens masking (#5), and its controlled trial (#7) where the same amount of random tokens are masked out. Table 6 shows agents' performance under various direction tokens ablation settings (#4 & #5).

For indoor agents, masking out the direction tokens cause a sharper drop in success rate compared to masking out random tokens, which means the indoor navigation agents do consider the direction tokens during navigation. We also notice that agents are more sensitive to the loss of direction guidance on RxR-en than on R2R. Such difference may be caused by the way these two datasets are designed. R2R's ground-truth trajectories are the shortest path from start to goal. Previous studies have noted that R2R has the danger of exposing structural bias and leaking hidden shortcuts Thomason et al. (2019), and that such design encourages goal-seeking over path adherence Jain et al. (2019b). RxR is crafted to include longer and more variable paths to avoid such biases. Naturally, agents on RxR-en would pay more attention to direction tokens since they may approach their goal indirectly.

For outdoor navigation agents, masking out direction tokens leads to a drastic decline in task completion rate, compared to random masking. This indicates that current outdoor navigation agents heavily rely on the direction tokens when making decisions. Given the complicated visual environments and instructions in the outdoor navigation task, current agents fail to fully use the instructions, especially ignoring the rich object-related information. The ARC model shows the most salient performance decline of 90% to the instructions ablated by direction token masking. Aside from the classifier that predicts the next direction to take, ARC also uses a stop indicator to decide whether to stop at each step or not. Its unique mechanism for detecting stop signals might explain why it is more sensitive to the existence of direction tokens.

### The Effect of Numeric Tokens

We conduct ablation studies on agents' understanding of numeric tokens on RxR-en for indoor agents and Touchdown for outdoor agents. We select a subset of examples whose instructions contain numeric tokens,3 and construct ablated instructions on top. Table 7 provides the statistics of the instructions for numeric ablations. Table 8 lists out the results. The VLN-HAMT on RxR-en and VLN-Transformer on Touchdown have comparable performance when masking numeric tokens over random tokens, and have worse performance when replacing numeric tokens. This suggests that these two Transformer-based agents have the ability to conduct numerical reasoning to some extent. In contrast, other non-Transformer-based agents have less salient performance drops when replacing numeric tokens. For RCONCAT and ARC, replacing numeric tokens even leads to higher task completion rates. This implies the insufficient counting ability of the non-Transformer-based agents.

Footnote 3: We consider instructions that contain cardinal numbers from 1 to 20, and ordinal numbers from 1st to 20th.

## 5 Analysis on Visual Environment

This section investigates what the agent perceives in the visual environment. We set an eye on inspecting the agent's understanding of the surrounding objects and direction-related information.

### Effect of Objects in the Environment

Built upon the Matterport dataset Chang et al. (2017), R2R and RxR obtain detailed object instance annotations and serve as an excellent source

\begin{table}
\begin{tabular}{l|r r r} \hline \hline  & **\#Data** & \multicolumn{1}{c}{\(\overline{E_{\text{instr}}}\)} & \multicolumn{1}{c}{\(\overline{p(\text{tok}_{\text{num}})}\)} \\ \hline RxR-en & 2.0k & 135.0 & 1.4\% \\ Touchdown & 1.0k & 100.1 & 2.0\% \\ \hline \end{tabular}
\end{table}
Table 7: Statistics of RxR-en and Touchdown data samples with numeric tokens examined for evaluation. \(\overline{L_{\text{instr}}}\) is the average instruction length, and \(\overline{p(\text{tok}_{\text{num}})}\) denotes the percentage of numeric tokens per instruction.

\begin{table}
\begin{tabular}{l|r r|r r r} \hline \multirow{2}{*}{**Setting**} & \multicolumn{3}{c|}{**SR \(\uparrow\) on RxR-en**} & \multicolumn{3}{c}{**TC \(\uparrow\) on Touchdown**} \\ \cline{2-6}  & CLIPVIL & HAMT & RECONCAT & ARC & VLNTrans \\ \hline Vanilla & 36.05 & 47.38 & 11.76 & 14.24 & 16.31 \\ \hline Mask Number & -4\% & -5\% & 3\% & -7\% & -3\% \\ Replace Number & -3\% & -6\% & 1\% & 2\% & -11\% \\ Controlled Trial & -6\% & -3\% & -5\% & -4\% & -6\% \\ \hline \hline \end{tabular}
\end{table}
Table 8: Navigation performance on different numeric-token ablations settings.

 

for our visual object studies. Touchdown is based on Google Street View and does not acquire object-related annotations. Thus, we conduct experiments on the indoor VLN environment.

We designed several ablation settings for visual objects. The "mask all visible" setting applies masking to all the visible visual objects in the environment (except for wall/ceiling/floor). The "mask foreground" setting ablates the visual objects within 3 meters of the camera viewpoint, which we refer to as the foreground area. The region beyond 3 meters from the camera viewpoint is regarded as the background area. Figure 1 shows an example for comparison. We choose 3 meters as the boundary because the bounding box annotations for objects within 3 meters are provided in REVERIE Qi et al. (2020c). We denote the number of visual objects within 3 meters as \(k\), and add a controlled trial that masks out \(k\) random visual objects from all the visible objects at the current viewpoint, regardless of their depth.

Table 9 compares the number of visual objects under various ablation settings. We mask out the objects in each view by filling the corresponding bounding boxes with the mean color of the surrounding. Then we follow original experiment settings and use ResNet-152 He et al. (2016) CNN to extract image features for R2R agents, and use CLIP-ViT-B/32 Radford et al. (2021) to extract visual features for RxR-en agents.

Results for visual object ablations are shown in Table 10. We examine the influence of masking out different quantities of visual objects by comparing the "mask all visible" setting with the controlled trial (#2 vs. #4). It comes naturally that masking out all the visible objects has a more salient impact on the success rate for all the listed indoor agents. We study the influence of masking visual objects at different depths by comparing the "mask foreground" setting with the controlled trial (#3 vs. #4). Noted here that the number of foreground objects is limited. Thus only a few objects are being masked out in both settings. Still, all listed indoor agents display worse performance on the controlled trial. Such results state that masking out further visual instances in the background, even only a tiny amount, will hurt navigation performance. This indicates that the tested agents consider all the objects in the visual environment during navigation, instead of merely staring at the closely surrounding objects.

Notice that the agent designs, the dataset domains, and the visual feature extractors are three coherent factors that may result in performance differences. We further justify this by adding another set of ablation studies, where we apply ImageNet ResNet-152 and CLIP-ViT to extract visual features for R2R and RxR-en, and evaluate with the same agent model EnvDrop. Results are shown in Table 11. The trend of different masking settings aligns with our previous findings in Table 10, and verifies that the background information is also crucial in the visual features.

Figure 1: Accessible objects within different ranges.

\begin{table}
\begin{tabular}{l|r r r r r|r r r} \hline \multirow{2}{*}{**\(\bullet\)**} & \multirow{2}{*}{**Ablation**} & \multicolumn{4}{c|}{**SR + en Right-en**} \\ \cline{3-8}  & & & ExpDrop & FAST & Recur & PVLT & CLIP-VIL & HAMT \\ \hline
1 & - & Vanilla & 49.77 & 63.90 & 53.30 & 57.13 & 40.21 & 53.52 \\ \hline
2 & MAV & -346 & -67u & -37u & -47u & -30u & -43e \\
3 & Object & MFG & -3sfe & -6sfe & -1sfe & -8sfe & -2sfe \\
4 & CT & -5sfe & -10wg & -6sfe & -9sfe & -3sfe & -5sfe \\ \hline
5 & Direction & Flip & -41sfe & -30u & -48sfe & -59sfe & -36sfe & -47sfe \\ \hline \end{tabular}
\end{table}
Table 10: Indoor navigation performance on various ablation settings on the visual environment. We compare three masking settings on the visual objects: mask all visible objects (MAV), mask foreground objects (MFG), and the controlled trial (CT). We horizontally flip the viewpoint to ablate direction-related visual information. Recur: VLN \(\odot\) BERT. PVLT: PREVALENT.

 



### Effect of Directions in the Environment

In this ablation setting, we randomly flip some of the viewpoints horizontally. The objects' relative positions at the flipped viewpoints will be reversed. Presumably, suppose the agent can follow the instruction and find the corresponding direction to approach. In that case, the flipped viewpoints will misguide the agent in the opposite direction and lower the navigation success rate. As shown in Table 10, flipping the viewpoints leads to drastic declines in the success rate for all listed indoor agents. This verifies our previous finding that indoor agents can understand directions in the instruction. We notice that FAST is the only listed model that is less affected by the direction flipping ablation than by the object masking ablation (#2 vs. #5). This suggests that FAST's asynchronous backtracking search is able to adjust and recover from errors that occur when choosing directions to some extent.

## 6 Analysis on Vision-Language Alignment

This section examines the agents' ability to learn vision-language alignment when executing the navigation. We focus on whether the agents can understand the objects mentioned in the instruction and align them to the correct visual instance in the environment, which is crucial to completing this multimodal task. To verify the existence of vision-language alignment, we add perturbations to the visual and textual input, and check how they affect agents' performance.

### Instruction Side Perturbation

We add noise to the textual input by randomly replacing object tokens with random object tokens in the instruction. Table 5 shows an example (#3). This experiment aims to verify whether the agent can line the object tokens up to certain visual targets. The assumption is that if the agent can correctly align objects mentioned in the instruction to some targets in the visual environment, then replacing the object token will confuse and misguide the agent. Examining Figure 2, we notice that for all three datasets, the Transformer-based models have worse performance when replacing the object tokens, compared to simple masking. This indicates that Transformer-based models have a better cross-modal understanding of objects, and can align object tokens to the visual targets. Such superior performance may result from the fact that the Transformer-based models are often pre-trained on multimodal resources, thus displaying a slightly more vital ability to form alignment.

### Environment Side Perturbation

We add noise to the visual input by conducting the following ablations. In the "dynamic mask" setting, we dynamically mask out the visual object regions mentioned in the instruction. We randomly mask out the same amount of visual objects at each viewpoint in its controlled trial. We also compare with the "mask tokens" setting, where we mask out all the object tokens in the instruction, while leaving the visual environment untouched. This

\begin{table}
\begin{tabular}{l|c c c c|c c} \hline \multirow{2}{*}{**Setting**} & \multicolumn{4}{c|}{**SR \uparrow on R2R**} & \multicolumn{2}{c}{**SR \uparrow on RxR-en**} \\ \cline{2-7}  & EnvDrop & FAST & Recur & PVLT & CLIPVIL & HAMT \\ \hline Vanilla & 49.77 & 63.90 & 53.30 & 57.13 & 40.21 & 52.52 \\ \hline Dynamic Mask Controlled Trial & -8\% & -24\% & -5\% & -13\% & -19\% & -23\% \\ \hline Mask Tokens & -36\% & -38\% & -21\% & -20\% & -48\% & -32\% \\ \hline \end{tabular}
\end{table}
Table 12: Indoor navigation performance when dynamically masking out the visual objects mentioned in the instructions. Recur: VLN \(\circlearrow\) BERT. PVLT: PREVALENT.

Figure 2: Performance gap between masking and replacing object tokens from instructions. If \(\Delta>0\), then replacing object tokens leads to worse navigation performance, which suggests that the agent has a better understanding of the object tokens.



experiment aims to determine if the agent aligns the textual object tokens to the correct visual target. The assumption is that if the agent builds proper vision-language alignment and we mask out visual objects mentioned in the instruction, then the agent may get confused since it can not find the counterpart in the visual environment.

Results are shown in Table 12. The success rate witnesses a decline when dynamically masking out the visual objects. However, we notice in surprise that when all visual objects mentioned in the instruction are masked out, the agents can still reach a success rate higher than 44% on R2R and higher than 32% on RxR-en. This contradicts the previous assumption and casts doubt on the reliability of the navigation agents' vision-language alignment.

Comparing "dynamic mask" with the "mask tokens" setting, we notice that the visual object ablation has much smaller impact on navigation performance than the text object ablations, which suggests that current models have unbalanced attention on vision and text for the VLN task. Recent studies on pre-trained vision-and-language models Frank et al. (2021) reveal that such asymmetry is also witnessed in other multimodal tasks. Future studies may follow the line of constructing a more balanced VLN agent.

## 7 Conclusion

In this paper, we inspect how the navigation agents understand the multimodal information by conducting ablation diagnostics input data. We find out that indoor navigation agents refer to both object tokens and direction tokens in the instruction when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. When it comes to vision-and-language alignments, we witness unbalanced attention on text and vision, and doubt the reliability of cross-modal alignments. We hope this work encourages more investigation and research into understanding neural VLN agents' black-box and improves the task setups and navigation agents' capacity for future studies.

## Acknowledgement

We would like to show our gratitude to the anonymous reviewers for their thought-provoking comments. We would like to thank the Robert N. Nyyce Trust for their generous gift to the University of California via the Noyce Initiative. The UCSB authors were also sponsored by the U.S. Army Research Office, and this work was accomplished under Contract Number W911NF19-D-0001 for the Institute for Collaborative Biotechnologies. The views and conclusions contained in this research are those of the authors and should not be interpreted as representing the sponsors or the U.S. government's official policy, expressed or inferred. Regardless of any copyright notation herein, the United States government is authorized to reproduce and distribute reprints for government purposes.

## References

* Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Vanderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 3674-3683. IEEE Computer Society.
* Andreas and Klein (2015) Jacob Andreas and Dan Klein. 2015. Alignment-based compositional semantics for instruction following. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1165-1174, Lisbon, Portugal. Association for Computational Linguistics.
* Antoniol et al. (2011) Giuliano Antoniol, Roldano Cattoni, and Mauro Cettolo. 2011. Robust speech understanding for robot telecontrol.
* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association for Computational Linguistics_, 1:49-62.
* Bisk et al. (2016a) Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016a. Natural language communication with robots. In _NAACL_.
* Bisk et al. (2016b) Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016b. Natural language communication with robots. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 751-761, San Diego, California. Association for Computational Linguistics.
* Branavan et al. (2009) S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In _Proceedings of 

the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 82-90, Suntec, Singapore. Association for Computational Linguistics.
* Branavan et al. (2010) S.R.K. Branavan, Luke Zettlemoyer, and Regina Barzilay. 2010. Reading between the lines: Learning to map high-level instructions to commands. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 1268-1277, Uppsala, Sweden. Association for Computational Linguistics.
* Chang et al. (2017) Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from RGB-D data in indoor environments. In _2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 2017_, pages 667-676. IEEE Computer Society.
* Chen and Mooney (2011) David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In _AAAI 2011_.
* Chen et al. (2019) Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. TOUCHDOWN: natural language navigation and spatial reasoning in visual street environments. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 12538-12547. Computer Vision Foundation / IEEE.
* Chen et al. (2021) Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021. History aware multimodal transformer for vision-and-language navigation. In _NeurIPS_.
* Dan et al. (2021) Soham Dan, Michael Zhou, and Dan Roth. 2021. Generalization in instruction following systems. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 976-981, Online. Association for Computational Linguistics.
* Fernandes et al. (2021) Patrick Fernandes, Kayo Yin, Graham Neubig, and Andre F. T. Martins. 2021. Measuring and increasing context usage in context-aware machine translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6467-6478, Online. Association for Computational Linguistics.
* Frank et al. (2021) Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers. In _EMNLP_.
* Fried et al. (2018) Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada_, pages 3318-3329.
* Guadarrama et al. (2013) Sergio Guadarrama, Lorenzo Riano, David Hamilton Golland, Daniel Goehring, Yangqing Jia, Dan Klein, P. Abbeel, and Trevor Darrell. 2013. Grounding spatial relations for human-robot interaction. _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_, pages 1640-1647.
* Hao et al. (2020) Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pretraining. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_, pages 13134-13143. IEEE.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_, pages 770-778. IEEE Computer Society.
* ECCV 2016
- 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV_, volume 9908 of _Lecture Notes in Computer Science_, pages 3-19. Springer.
* Hong et al. (2020a) Yicong Hong, Cristian Rodriguez Opazo, Yuankai Qi, Qi Wu, and Stephen Gould. 2020a. Language and visual entity relationship graph for agent navigation. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Hong et al. (2020b) Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez Opazo, and Stephen Gould. 2020b. A recurrent vision-and-language BERT for navigation. _CoRR_, abs/2011.13922.
* November 2, 2019_, pages 7403-7412. IEEE.
* Jain et al. (2019a) Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019a. Stay on the path: Instruction fidelity in vision-and-language navigation. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 1862-1872. Association for Computational Linguistics 

Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019b. Stay on the path: Instruction fidelity in vision-and-language navigation. _ArXiv_, abs/1905.12255.
* Ke et al. (2019) Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha S. Srinivasa. 2019. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 6741-6749. Computer Vision Foundation / IEEE.
* Klingspor et al. (1997) Volker Klingspor, John Demiris, and Michael Kaiser. 1997. Human-robot-communication and machine learning. _APPLIED ARTIFICIAL INTELLIGENCE JOURNAL_, 11(11):719-746.
* Kollar et al. (2013) Thomas Kollar, Jayant Krishnamurthy, and Grant P. Strimel. 2013. Toward interactive grounded language acquisition. In _Robotics: Science and Systems_.
* Ku et al. (2020a) Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020a. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 4392-4412. Association for Computational Linguistics.
* Ku et al. (2020b) Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020b. Room-Across-Room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In _Conference on Empirical Methods for Natural Language Processing (EMNLP)._.
* November 4, 2018_, pages 1338-1346. Association for Computational Linguistics.
* Li et al. (2019) Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. 2019. Robust navigation with language pretraining and stochastic sampling. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 1494-1499. Association for Computational Linguistics.
* Lu et al. (2016)Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_, pages 289-297.
* Ma et al. (2019) Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019a. Self-monitoring navigation agent via auxiliary progress estimation. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Ma et al. (2019b) Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. 2019b. The regretful agent: Heuristic-aided navigation through progress estimation. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 6732-6740. Computer Vision Foundation / IEEE.
* Matuszek et al. (2014) Cynthia Matuszek, Liefeng Bo, Luke Zettlemoyer, and Dieter Fox. 2014. Learning from unscripted deictic gesture and language for human-robot interactions. In _AAAI_.
* Mehta et al. (2020) Harsh Mehta, Yoav Artzi, Jason Baldridge, Eugene Ie, and Piotr Mirowski. 2020. Retouchdown: Adding touchdown to streetlearn as a shareable resource for language grounding tasks in street view. _CoRR_, abs/2001.03671.
* Mei et al. (2016) Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In _AAAI_.
* Mirowski et al. (2018) Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. 2018. Learning to navigate in cities without a map. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada_, pages 2424-2435.
* Misra et al. (2018) Dipendra Misra, Andrew Bennett, Valts Blukis, Eyynid Niklasson, Max Shattkin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environments with visual goal prediction. In _EMNLP_.
* Misra et al. (2017) Dipendra Kumar Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. In _EMNLP_.
* Nguyen et al. (2019) Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. 2019. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 12527-12537. Computer Vision Foundation / IEEE.
* Nguyen and Daume III (2019) Khanh Nguyen and Hal Daume III. 2019. Help, annai! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 

2019, Hong Kong, China, November 3-7, 2019_, pages 684-695. Association for Computational Linguistics.
* O'Connor and Andreas (2021) Joe O'Connor and Jacob Andreas. 2021. What context features can transformer language models use? In _ACL/IJCNLP_.
* Park et al. (2018) Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 8779-8788. IEEE Computer Society.
* Qi et al. (2020a) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020a. Stanza: A python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020_, pages 101-108. Association for Computational Linguistics.
* ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X_, volume 12355 of _Lecture Notes in Computer Science_, pages 303-317. Springer.
* Qi et al. (2020c) Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020c. REVERIE: remote embodied visual referring expression in real indoor environments. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_, pages 9979-9988. IEEE.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhnii Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In _ICML_.
* Roy (2002) Deb K. Roy. 2002. Learning visually grounded words and syntax for a scene description task. _Comput. Speech Lang._, 16:353-385.
* Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhashek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_, pages 618-626. IEEE Computer Society.
* Shen et al. (2021) Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2021. How much can clip benefit vision-and-language tasks? _ArXiv_, abs/2107.06383.
* Steels and Vogt (1997) Luc L. Steels and Paul Vogt. 1997. Grounding adaptive language games in robotic agents.
* Szegedy et al. (2017) Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Amir Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In _AAAI_.
* Tan and Bansal (2018) Hao Tan and Mohit Bansal. 2018. Source-target inference models for spatial instruction understanding. In _AAAI_.
* Tan et al. (2019) Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 2610-2621. Association for Computational Linguistics.
* Thomason et al. (2019) Jesse Thomason, Daniel Gordon, and Yonatan Bisk. 2019. Shifting the baseline: Single modality performance on visual navigation & qa. In _NAACL_.
* Vogel and Jurafsky (2010) Adam Vogel and Daniel Jurafsky. 2010. Learning to follow navigational directions. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 806-814, Uppsala, Sweden. Association for Computational Linguistics.
* ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX_, volume 12354 of _Lecture Notes in Computer Science_, pages 126-141. Springer.
* Wang et al. (2019) Xin Wang, Qiuyuan Huang, Asli Celikylmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 6629-6638. Computer Vision Foundation / IEEE.
* ECCV 2018
- 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XVI_, volume 11220 of _Lecture Notes in Computer Science_, pages 38-55. Springer.
* ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV_, volume 12369 of _Lecture Notes in Computer Science_, pages 413-430. Springer
 

* Winograd (1971) Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language.
* Wu and Mooney (2019) Jialin Wu and Raymond Mooney. 2019. Faithful multimodal explanation for visual question answering. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 103-112, Florence, Italy. Association for Computational Linguistics.
* Xia et al. (2020) Qiaolin Xia, Xiujun Li, Chunyuan Li, Yonatan Bisk, Zhifang Sui, Jianfeng Gao, Yejin Choi, and Noah A. Smith. 2020. Multi-view learning for vision-and-language navigation. _CoRR_, abs/2003.00857.
* Xiang et al. (2020) Jiannan Xiang, Xin Wang, and William Yang Wang. 2020. Learning to stop: A simple yet effective approach to urban vision-language navigation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_, pages 699-707. Association for Computational Linguistics.
* Zhu et al. (2020a) Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020a. Vision-language navigation with self-supervised auxiliary reasoning tasks. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_, pages 10009-10019. IEEE.
* Zhu et al. (2020b) Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, and William Yang Wang. 2020b. Multimodal text style transfer for outdoor vision-and-language navigation. _CoRR_, abs/2007.00229.
* Zhu et al. (2020c) Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, and Xiaodan Liang. 2020c. Vision-dialog navigation by exploring crossmodal memory. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_, pages 10727-10736. IEEE.