SlowGuess/NAACL_2022_OCR huggingface dataset:
Wanrong Zhu $^{\text{II}}$ , Yuankai Qi $^{\text{S}}$ , Pradyumna Narayana $^{\text{A}}$ , Kazoo Sone $^{\text{A}}$ , Sugato Basu $^{\text{A}}$ , Eric Xin Wang $^{\text{S}}$ , Qi Wu $^{\text{S}}$ , Miguel Eckstein $^{\text{I}}$ , William Yang Wang $^{\text{I}}$
$^{\mathbb{Q}}$ UC Santa Barbara,  $^\S$ University of Adelaide, \*Google,  $^\ddagger$ UC Santa Cruz
{wanrongzhu, miqueleckstein, wangwilliamyang}@ucsb.edu, qykshr@gmail.com  
{pradyn, sone, sugato} @google.com, xwang366@ucsc.edu, qi.wu01@adelaide.edu.au
Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines' performance and human benchmarks. Moreover, the agents' inner mechanisms for navigation decisions remain unclear. To the best of our knowledge, how the agents perceive the multimodal input is under-studied and needs investigation. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation. Results show that indoor navigation agents refer to both object and direction tokens when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. Transformer-based agents acquire a better cross-modal understanding of objects and display strong numerical reasoning ability than non-Transformer-based agents. When it comes to vision-and-language alignments, many models claim that they can align object tokens with specific visual targets. We find unbalanced attention on the vision and text input and doubt the reliability of such cross-modal alignments.<sup>1</sup>
A key challenge for Artificial Intelligence (AI) research is to move beyond Independent and Identically Distributed (i.i.d.) data analysis: We need to teach AI agents to understand multimodal input data, and jointly learn to reason and perform incremental and dynamic decision-making with the help from humans. Vision-and-Language Navigation (VLN) has received much attention due to its active perception and multimodal grounding setting, dynamic decision-making nature, rich applications,
R2R	RxR	Touchdown
Human Performance	86	94	92
SoTA Model Performance	78	53	17
Table 1: There exists salient gaps between machines' vision-and-language navigation (VLN) performance and human benchmarks. Navigation success rates are reported on the R2R (Anderson et al., 2018) and the RxR dataset (Ku et al., 2020b) for indoor VLN and the Touchdown dataset (Chen et al., 2019) for outdoor VLN.
and accurate evaluation of agents' performances in language-guided visual grounding. As the AI research community gradually shifts its attention from the static empirical analysis of datasets to more challenging settings that require incremental decision-making processes, the interactive task of VLN deserves a more in-depth analysis of why it works and how it works.
Various setups have been proposed to address to the VLN task. Researchers generate visual trajectories and collect human-annotated instructions for indoor (Anderson et al., 2018; Jain et al., 2019a; Ku et al., 2020a; Chen et al., 2021) and outdoor environment (Chen et al., 2019; Mehta et al., 2020; Mirowski et al., 2018). There are also interactive VLN settings based on dialogues (Nguyen et al., 2019; Nguyen and III, 2019; Zhu et al., 2020c), and task that navigates agents to localize a remote object (Qi et al., 2020c). However, few studies ask the Why and How questions: Why do these agents work (or do not work)? How do agents make decisions in different setups?
Through the years, agents with different model architectures and training mechanisms have been proposed for indoor VLN (Anderson et al., 2018; Fried et al., 2018; Hao et al., 2020; Hong et al., 2020a,b; Huang et al., 2019; Ke et al., 2019; Li et al., 2019; Ma et al., 2019a; Qi et al., 2020b; Tan
$^{1}$ Code and data used in this study are available at https://github.com/VegB/Diagnose_VLN.
2We record the published state-of-the-art performance on R2R, RxR and Touchdown leaderboards on Dec.15th, 2021.
Human Language Technologies, pages 5981 - 5993
et al., 2019; Wang et al., 2020a, 2019, 2018, 2020b; Zhu et al., 2020a) and outdoor VLN (Chen et al., 2019; Ma et al., 2019b; Mirowski et al., 2018; Xia et al., 2020; Xiang et al., 2020; Zhu et al., 2020b). Back-translation eases the urgent problem of data scarcity (Fried et al., 2018). Imitation learning and reinforcement learning enhance agents' generalization ability (Wang et al., 2019, 2018). With the rise of BERT-based models, researchers also apply Transformer and pre-training to further improve navigation performance (Hao et al., 2020; Hong et al., 2020b; Zhu et al., 2020b). While applying new techniques to the navigation agents might boost their performance, we still know little about how agents make each turning decision. Treatment of the agents' processing of instructions and perception of the visual environment as a black box might hinder the design of a generic model that fully understands visual and textual input regardless of VLN setups. Table 1 shows non-negligible performance gaps between neural agents and humans on both indoor and outdoor VLN tasks.
Therefore, we focus on analyzing how the navigation agents understand the multimodal input data in this work. We conduct our investigation from the perspectives of natural language instruction, visual environment, and the interpretation of vision-language alignment. We create counterfactual interventions to alter the instructions and the visual environment in the validation dataset, focusing on variables related to objects, directions and numerics. More specifically, we modify the instruction by removing or replacing the object/direction/numeric tokens, and we adjust the environment by masking out visual instances or horizontally flipping the viewpoint images. Subsequently, we examine the interventions' treatment effects on agents' evaluation performance while keeping other variables unchanged. We set up experiments on the R2R (Anderson et al., 2018) and the RxR dataset (Ku et al., 2020b) for indoor VLN and the Touchdown dataset (Chen et al., 2019) for outdoor VLN. We examine nine VLN agents on the three datasets with quantitative ablation diagnostics on the text and visual inputs.
In summary, our key findings include:
1. Indoor navigation agents refer to both objects and directions in the instruction when making decisions. In contrast, outdoor navigation agents heavily rely on directions and poorly understand visual objects. (Section 4)
2. Instead of merely staring at surrounding objects, indoor navigation agents are able to set their sights on objects further from the current viewpoint. (Section 5)
3. Transformer-based agents display stronger numerical reasoning ability (Section 4), and acquire better cross-modal understanding of objects, compared to non-Transformer-based agents. (Section 6)
4. Indoor agents can align object tokens to certain targets in the visual environment to a certain extent, but display in-balanced attention on text and visual input. (Section 6)
We hope these findings reveal opportunities and obstacles of current VLN models and lead to new research directions.
Instruction Following is a long-standing topic in AI studies that ask an agent to follow natural language instructions and accomplish target tasks, which can be dated back to the SHRLDU (Winograd, 1971). Efforts made to tackle this classic problem spans from defining templates (Klingspor et al., 1997; Antoniol et al., 2011), designing hard-encoded concepts to ground visual attributes and spatial relations (Steels and Vogt, 1997; Roy, 2002; Guadarrama et al., 2013; Kollar et al., 2013; Matuszek et al., 2014), to constructing varies datasets and learning environments (Anderson et al., 1991; Bisk et al., 2016a; Misra et al., 2018). Many methods have been proposed to map the instructions into sequence of actions, such as reinforcement learning (Branavan et al., 2009, 2010; Vogel and Jurafsky, 2010; Misra et al., 2017), semantic parsing (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013), alignment-based model (Andreas and Klein, 2015), and neural networks (Bisk et al., 2016b; Mei et al., 2016; Tan and Bansal, 2018).
Vision-and-Language Navigation is a task where an agent comprehends the natural language instructions and reasons through the visual environment. Many studies aim at improving VLN agents' performance in one way or another. To enrich training data, a line of work (Fried et al., 2018; Zhu et al., 2020b) use back-translation to generate augmented instructions. To enforce cross-modal grounding, RPA and RCM (Wang et al., 2018, 2019) use reinforcement learning, SMNA (Ma et al., 2019a) uses a visual-textual co-grounding module to improve cross-modal alignment, Rel-
Graph (Hong et al., 2020a) uses graphs for task formulation. To address the generalizability problem to unseen environment, PRESS (Li et al., 2019) introduces a stochastic sampling scheme, EnvDrop (Tan et al., 2019) proposes environment dropout. To utilize visual information from the environment, AuxRN (Zhu et al., 2020a) uses auxiliary tasks to assist semantic information extraction, VLN-HAMT (Chen et al., 2021) incorporates panorama history with a hierarchical vision transformer. FAST (Ke et al., 2019) makes use of asynchronous search and allows the agent to backtrack if it discerns a mistake after attending to global and local knowledge. With the success of BERT-related models in NLP, researchers also start to build Transformer-based navigation agents and add a pre-training process before fine-tuning on the downstream VLN task (Hao et al., 2020; Hong et al., 2020b; Zhu et al., 2020b; Chen et al., 2021).
Model Behavior Analysis As multimodal studies gain more and more attention, there are lines of works that focus on explaining models' behaviors to better understand and handle the tasks. Some generate textual explanations by training another model to mimic human explanations (Hendricks et al., 2016; Park et al., 2018; Wu and Mooney, 2019). Others generate visual explanations with the help of attention mechanism (Lu et al., 2016) or gradient analysis (Selvaraju et al., 2017). There are also attempts to provide multimodal explanations, e.g., Li et al. (2018) breaks up the end-to-end VQA process and examines the intermediate results by extracting attributes from the visual instances. Another line of work examines model performance by conducting ablation studies on input data. Recent analyses on language modelling (O'Connor and Andreas, 2021), machine translation (Fernandes et al., 2021), and instruction following (Dan et al., 2021) ablate/perturb both training and validation data. A study on multimodal models (Frank et al., 2021) only applies ablation during evaluation, which is the same as our settings.
3 Background and Research Questions
We first bring in the task of Vision-and-Language Navigation and introduce the datasets and agents used for comparison. Then we list out the research questions to study in this work.
Dataset	Model	Trans?	Visual Feature
R2R	EnvDrop (Tan et al., 2019)	×	ResNet-152
FAST (Ke et al., 2019)	×
VLN ⊙ BERT (Hong et al., 2020b)	✓
PREVALENT (Hao et al., 2020)	✓
RxR-en	CLIP-ViL (Shen et al., 2021)	×	CLIP-ViT
VLN-HAMT (Chen et al., 2021)	✓
Touchdown	RCONCAT (Chen et al., 2019)	×	ResNet-18
ARC (Xiang et al., 2020)	×
VLN-Transformer (Zhu et al., 2020b)	✓
Table 2: The VLN datasets and models covered in this study. We record whether the model structure is Transformer-based, and the pre-trained feature extractor used to encode visual environment.
3.1 Vision-and-Language Navigation
In the vision-and-language navigation task, the navigation agent is asked to find the path to reach the target location following the instructions  $\mathcal{X}$ . The navigation procedure can be viewed as a sequential decision-making process. At each time step  $t$ , the visual environment presents an image view  $\pmb{v}_t$ . With reference to the instruction  $\mathcal{X}$  and the visual view  $\pmb{v}_t$ , the agent is expected to choose an action  $a_t$  such as turn left or stop.
Datasets We conduct indoor navigation experiments on the Room-to-Room (R2R) dataset (Anderson et al., 2018) and the Room-across-Room (RxR) dataset (Ku et al., 2020b), and test outdoor VLN on Touchdown (Chen et al., 2019). R2R and RxR are built upon real estate layouts and contain separate graphs for each apartment/house. Unlike R2R, which shoots for the shortest path, RxR has longer and more variable paths. R2R only contains English instructions, while RxR also includes instructions in Hindi and Telugu. In this study, we only cover the English subset for RxR, and will refer to it as RxR-en in the following sections. Navigation in Touchdown occurs in the urban environment, where the viewpoints form a huge connected graph. Compared to indoor environments, Touchdown has more complicated visual environments and a more extensive search space. The evaluation results are reported on the validation unseen sets for R2R and RxR-en and on the test set for Touchdown.
Models Table 2 lists out the models covered in our study. We use the code and trained checkpoints shared by the authors in the following experiments.
For indoor navigation on R2R, we study a widely adopted base model Envdrop (Tan et al., 2019), a backtracking framework for self-correction FAST (Ke et al., 2019), and two SoTA models VLN  $\odot$  BERT (Hong et al., 2020b) and PREVA
LENT (Hao et al., 2020). The Envdrop introduces environment dropout on top of the Speaker-Follower (Fried et al., 2018) model, FAST conducts an asynchronous search for backtracking, PREVALENT, and VLN  $\bigcirc$  BERT are Transformer-based agents with pre-trained models.
For navigation on RxR-en, we examine CLIP-ViL (Shen et al., 2021) and VLN-HAMT (Chen et al., 2021). CLIP-ViL shares the same model structure with EnvDrop. The only difference is that CLIP-ViL uses CLIP-ViT (Radford et al., 2021) to extract visual features, while EnvDrop uses ImageNet ResNet (Szegedy et al., 2017) features. VLN-HAMT incorporates a long-horizon history into decision-making by encoding all the past panoramic observations via a hierarchical vision Transformer.
For outdoor navigation on Touchdown, we consider the common baseline RCONCAT (Chen et al., 2019), and two SoTA models ARC (Xiang et al., 2020) and VLN-Transformer (Zhu et al., 2020b). RCONCAT encodes the trajectory and the instruction in an LSTM-based manner. ARC improves RCONCAT by paying special attention to the stop signals. VLN-Transformer is a Transformer-based agent that applies pre-training on an external dataset for outdoor navigation in urban areas.
Metrics In the following experiments, we evaluate navigation performance with Success Rate (SR) for indoor agents and Task Completion (TC) rate for outdoor agents. Both SR and TC measure the accuracy of completing the navigation task, reflecting the agents' overall ability to finish navigation correctly. An indoor navigation task is considered complete if the agent's final position locates within 3 meters of the target location. For outdoor navigation, the task is considered complete if the agent stops at the target location or one of its adjacent nodes in the environment graph.
Current VLN studies have reached their bottleneck as only minor performance improvements have been achieved recently, while a significant gap still exists between machine and human performance. This motivates us to find the reasons.
To better understand how VLN agents make decisions during navigation, we conduct a series of experiments on indoor and outdoor VLN tasks, aiming to answer the following questions that might help us locate the deficiencies of current model
Dataset	Instruction
R2R	Walk through the door by the sink into the middle of the next room. Turn right and walk down the hallway and enter the third door on your right.
RxR-en	We're facing towards a small picture that's attached to the wall, turn slightly to the right, and enter the hallway that's in front of you, turn to the left, take five steps further... On your right there are a few glass doors and on the left there's a living room, walk towards the living room, turn slightly to the left... On the right there are four chairs and a beautiful coffee table in the middle, on the left there's a console table with a vase with flowers on top, walk past the console table towards the back of the chair that's in direfront of you... We're now facing towards a lamp, and on the right there's a marble console table with decorations on top, and that's your destination.
Touchdown	Orient yourself so that you are moving in the same direction as traffic. Go straight through 3 intersections. Keep moving forward, after the 3rd intersection, you should see a signs for a store with a white background and red dots as well as a red and white bullseye target. Continue going straight past this store and at the next intersection, turn left. Go through one intersection and stop just after the wall on your left with the purple zig zag patterns.
Table 3: Instructions from R2R, RxR-en and Touchdown with object-tokens, direction-tokens and numeric-tokens highlighted.
Dataset	#Data	\( \overline{\mathbf{L}}_{\text{path}} \)	\( \overline{\mathbf{L}}_{\text{instr}} \)	#Object	\( \overline{p(\mathbf{tok}_{\mathbf{obj}})} \)	\( \overline{p(\mathbf{tok}_{\mathbf{direct}})} \)
R2R	2.3k	6.0	29.3	0.6k	19.8%	7.3%
RxR-en	4.6k	8.5	111.3	1.4k	16.1%	6.5%
Touchdown	1.4k	34.4	92.5	1.0k	16.8%	6.8%
Table 4: Statistics of R2R, RxR-en and Touchdown datasets. #Data is the dataset size used for evaluation in this study.  $\overline{L_{\text{path}}}$  is the average path length, which is the number of viewpoints covered in the trajectory.  $\overline{L_{\text{instr}}}$  is the average instruction length. #Object denotes the number of unique objects mentioned in the instructions.  $\overline{p(\mathrm{tok}_{\mathrm{obj}})}$  and  $\overline{p(\mathrm{tok}_{\mathrm{direc}})}$  indicates the percentage of object/direction tokens per instruction.
designs and explore future research directions:
1. What can the agents learn from the instructions? Do they pay more attention to object tokens or directions tokens? Do they have the ability to count? (Section 4)
2. What do agents see in the visual environment? Are they staring at the closely surrounded objects or also browsing further layout? Do they focus on individual visual instances or perceive the overall outline? (Section 5)
3. Can agents match textual tokens to visual entities? How reliable are such connections? (Section 6)
4 Analysis on Instruction Understanding
This section examines whether and to what extent the agent understands VLN instructions. We focus on how the agent perceives object-related tokens, direction-related tokens, and numeric tokens, and their effects on final navigation performance. Table 3 shows exemplar instructions of the three
*	Setting	Instruction
1	Vanilla	Go left down the hallway toward the exit sign. Go into the door on the left and stop by the table.
2	Mask Object Tokens	Go left down the [MASK] toward the [MASK] [MASK]. To into the [MASK] on the left and stop by the [MASK].
3	Replace Object Tokens	Go left down the portrait toward the sofa fountains. Go into the football on the left and stop by the boats.
4	Controlled Trial	Go [MASK] down the hallway [MASK] the exit sign. To into the door on [MASK] left and [MASK] by [MASK] table.
5	Mask Direction Tokens	Go [MASK] down the hallway toward the exit sign. Go into the door on the [MASK] and [MASK] by the table.
6	Replace Direction Tokens	Go right down the hallway toward the exit sign. Go into the door on the right and forward by the table.
7	Controlled Trial	Go left down the [MASK] [MASK] the exit sign. Go into the door on the left and [MASK] by the table.
Table 5: Example of instruction modification. In the original instruction, there are five object-related tokens, and three direction-related tokens. In the object token ablations, we mask out the object tokens, or replace them with randomly sampled object tokens. The controlled trial randomly masked out five tokens from the instruction for a fair comparison. Likewise the direction tokens.
*	Ablation	Setting	SR ↑ on R2R	SR ↑ on RxR-en	TC ↑ on Touchdown
EnvDrop	FAST	VLN○BERT	PREVALENT	CLIPViL	HAMT	RCONCAT	ARC	VLNTrans
1	-	Vanilla	49.77	63.90	53.30	57.13	40.21	52.52	11.78	15.19	16.11
2	Object	Mask	-36%	-38%	-21%	-20%	-48%	-32%	-34%	-36%	-6%
3	Controlled Trial	-31%	-26%	-8%	-8%	-35%	-23%	-44%	-55%	-15%
4	Direction	Mask	-23%	-23%	-15%	-11%	-39%	-28%	-73%	-90%	-45%
5	Controlled Trial	-11%	-12%	-4%	-3%	-14%	-9%	-22%	-23%	-8%
Table 6: The navigation performance for indoor and outdoor agents on object-token and direction-token ablations. We record the validation score in the "vanilla" setting, and report the relative performance change for each ablation setting. For object-token ablations, the "mask" setting masks out all the object-tokens, while the controlled trial masks out the same amount of random tokens. The same applies to direction-token ablations.
datasets covered in our study. As shown in Table 4, Touchdown's trajectory length is significantly longer than the other two indoor datasets. RxR-en and Touchdown have longer instructions than R2R. The ratios of object and direction tokens in all three datasets are comparable, involving about two times more object tokens than direction tokens.
4.1 The Effect of Object-related Tokens
We first create counterfactual interventions on instructions by masking out the object tokens. We use Stanza (Qi et al., 2020a) part-of-speech (POS) tagger to locate object-related tokens. A token will be regarded as an object token if its POS tag is NOUN or PROPN. During masking, we replace the object token with a specified mask token [MASK]. Then we examine the average treatment effects of the intervention on agents' performance, while keeping other variables unchanged.
Noticeably, when we mask out the object tokens, the number of visible tokens in the provided instruction also decreases, which is a coherent factor with masking object tokens and might interfere with our analysis. To eliminate the effect of reducing visible tokens, we add a controlled trial in which we randomly mask out the same amount of tokens. Table 5 gives an example of masking object tokens (#2) and its corresponding controlled trial (#4).
We follow each agent's original experiment setting for all the experiments in this study and train it on the original train set. Then we apply masking to object tokens in the validation set, and report agents' relative performance changes under each setting. We conduct five repetitive experiments and report the average scores for settings that involve random masking or replacing.
Table 6 presents how the agents' navigation performance change when object tokens are masked out (#2 & #3). Intuitively, not knowing what objects are mentioned in the instruction lowers all models' performance. Comparing the masking ablations with the controlled trial for indoor VLN, we notice that masking out the object tokens result in a more drastic decrease in success rate than masking out random tokens. This holds for all indoor agents, which verifies that indoor agents depend on object tokens more than other tokens. However, when we compare results on the Touchdown for outdoor VLN, we notice in surprise that masking out the object tokens has a weaker impact on task completion rate than masking out random tokens. This suggests that current outdoor navigation agents do not fully take object tokens into consideration when making decisions. This may be caused by the weak visual recognition module in current outdoor agents. As addressed in Table 2, all three outdoor
agents rely on visual features extracted by ResNet-18, which may not be powerful enough to fully incorporate the complicated urban environments.
4.2 The Effect of Direction-related Tokens