SlowGuess/NAACL_2022_OCR huggingface dataset:
We regard the following tokens as direction-related tokens: left, right, back, front, forward, stop. Similar to how we ablate the object tokens, we mask out direction tokens from the instruction and examine the impact on agents' navigation performance. Table 5 provides examples of direction tokens masking (#5), and its controlled trial (#7) where the same amount of random tokens are masked out. Table 6 shows agents' performance under various direction tokens ablation settings (#4 & #5).
For indoor agents, masking out the direction tokens cause a sharper drop in success rate compared to masking out random tokens, which means the indoor navigation agents do consider the direction tokens during navigation. We also notice that agents are more sensitive to the loss of direction guidance on RxR-en than on R2R. Such difference may be caused by the way these two datasets are designed. R2R's ground-truth trajectories are the shortest path from start to goal. Previous studies have noted that R2R has the danger of exposing structural bias and leaking hidden shortcuts (Thomason et al., 2019), and that such design encourages goal-seeking over path adherence (Jain et al., 2019b). RxR is crafted to include longer and more variable paths to avoid such biases. Naturally, agents on RxR-en would pay more attention to direction tokens since they may approach their goal indirectly.
For outdoor navigation agents, masking out direction tokens leads to a drastic decline in task completion rate, compared to random masking. This indicates that current outdoor navigation agents heavily rely on the direction tokens when making decisions. Given the complicated visual environments and instructions in the outdoor navigation task, current agents fail to fully use the instructions, especially ignoring the rich object-related information. The ARC model shows the most salient performance decline of  $90\%$  to the instructions ablated by direction token masking. Aside from the classifier that predicts the next direction to take, ARC also uses a stop indicator to decide whether to stop at each step or not. Its unique mechanism for detecting stop signals might explain why it is more sensitive to the existence of direction tokens.
#Data	\( \overline{\mathbf{L}_{\text{instr}}} \)	\( p(\mathbf{tok}_{\text{num}}) \)
RxR-en	2.0k	135.0	1.4%
Touchdown	1.0k	100.1	2.0%
Table 7: Statistics of RxR-en and Touchdown data samples with numeric tokens examined for evaluation.  $\overline{\mathbf{L}_{\mathrm{instr}}}$  is the average instruction length, and  $p(\mathrm{tok_{num}})$  denotes the percentage of numeric tokens per instruction.
Setting	SR ↑ on RxR-en	TC ↑ on Touchdown
CLIPViL	HAMT	RCONCAT	ARC	VLNTrans
Vanilla	36.05	47.38	11.76	14.24	16.31
Mask Number	-4%	-5%	3%	-7%	-3%
Replace Number	-3%	-6%	1%	2%	-11%
Controlled Trial	-6%	-3%	-5%	-4%	-6%
Table 8: Navigation performance on different numeric-token ablations settings.
4.3 The Effect of Numeric Tokens
We conduct ablation studies on agents' understanding of numeric tokens on RxR-en for indoor agents and Touchdown for outdoor agents. We select a subset of examples whose instructions contain numeric tokens, and construct ablated instructions on top. Table 7 provides the statistics of the instructions for numeric ablations. Table 8 lists out the results. The VLN-HAMT on RxR-en and VLN-Transformer on Touchdown have comparable performance when masking numeric tokens over random tokens, and have worse performance when replacing numeric tokens. This suggests that these two Transformer-based agents have the ability to conduct numerical reasoning to some extent. In contrast, other non-Transformer-based agents have less salient performance drops when replacing numeric tokens. For RCONCAT and ARC, replacing numeric tokens even leads to higher task completion rates. This implies the insufficient counting ability of the non-Transformer-based agents.
5 Analysis on Visual Environment
This section investigates what the agent perceives in the visual environment. We set an eye on inspecting the agent's understanding of the surrounding objects and direction-related information.
5.1 Effect of Objects in the Environment
Built upon the Matterport dataset (Chang et al., 2017), R2R and RxR obtain detailed object instance annotations and serve as an excellent source
3We consider instructions that contain cardinal numbers from 1 to 20, and ordinal numbers from 1st to 20th.
(a) Foreground Objects
(b) All Visible Objects
Figure 1: Accessible objects within different ranges.
Setting	R2R	RxR-en
All Visible Objects (except wall/floor/ceiling)	35.1	35.5
Foreground Objects	3.2	3.9
Objects Mentioned in Instruction	5.3	11.8
for our visual object studies. Touchdown is based on Google Street View and does not acquire object-related annotations. Thus, we conduct experiments on the indoor VLN environment.
We designed several ablation settings for visual objects. The "mask all visible" setting applies masking to all the visible visual objects in the environment (except for wall/ceiling/floor). The "mask foreground" setting ablates the visual objects within 3 meters of the camera viewpoint, which we refer to as the foreground area. The region beyond 3 meters from the camera viewpoint is regarded as the background area. Figure 1 shows an example for comparison. We choose 3 meters as the boundary because the bounding box annotations for objects within 3 meters are provided in REVERIE (Qi et al., 2020c). We denote the number of visual objects within 3 meters as  $k$ , and add a controlled trial that masks out  $k$  random visual objects from all the visible objects at the current viewpoint, regardless of their depth.
Table 9 compares the number of visual objects under various ablation settings. We mask out the objects in each view by filling the corresponding bounding boxes with the mean color of the surrounding. Then we follow original experiment settings and use ResNet-152 (He et al., 2016) CNN to extract image features for R2R agents, and use CLIP-ViT-B/32 (Radford et al., 2021) to extract visual features for RxR-en agents.
Results for visual object ablations are shown in Table 10. We examine the influence of masking out different quantities of visual objects by comparing the "mask all visible" setting with the controlled trial (#2 vs. #4). It comes naturally that masking
Table 9: The average number of visual objects in the panorama at each viewpoint under different settings.
*	Ablation	Setting	SR ↑ on R2R	SR ↑ on RxR-en
EnvDrop	FAST	Recur	PVLT	CLIPViL	HAMT
1	-	Vanilla	49.77	63.90	53.30	57.13	40.21	52.52
2	Object	MAV	-34%	-67%	-37%	-47%	-30%	-43%
3	MFG	-3%	-6%	-1%	-8%	-2%	-2%
4	CT	-5%	-10%	-6%	-9%	-3%	-5%
5	Direction	Flip	-41%	-30%	-48%	-59%	-36%	-47%
Table 10: Indoor navigation performance on various ablation settings on the visual environment. We compare three masking settings on the visual objects: mask all visible objects (MAV), mask foreground objects (MFG), and the controlled trial (CT). We horizontally flip the viewpoint to ablate direction-related visual information. Recur: VLN  $\odot$  BERT. PVLT: PREVALENT.
Dataset	Visual Feature	Vanilla	MAV	MFG	CT
R2R	ResNet-152	49.77	-34%	-3%	-5%
CLIP-ViT	56.36	-34%	-2%	-5%
RxR-en	ResNet-152	35.27	-42%	-1%	-3%
CLIP-ViT	40.21	-30%	-2%	-3%
Table 11: EnvDrop's navigation performance on R2R and RxR-en with different visual object ablation settings when using different visual features. MAV: mask all visible objects. MFG: mask foreground objects. CT: controlled trial.
out all the visible objects has a more salient impact on the success rate for all the listed indoor agents. We study the influence of masking visual objects at different depths by comparing the "mask foreground" setting with the controlled trial (#3 vs. #4). Noted here that the number of foreground objects is limited. Thus only a few objects are being masked out in both settings. Still, all listed indoor agents display worse performance on the controlled trial. Such results state that masking out further visual instances in the background, even only a tiny amount, will hurt navigation performance. This indicates that the tested agents consider all the objects in the visual environment during navigation, instead of merely staring at the closely surrounding objects.
Notice that the agent designs, the dataset domains, and the visual feature extractors are three coherent factors that may result in performance differences. We further justify this by adding another set of ablation studies, where we apply ImageNet ResNet-152 and CLIP-ViT to extract visual features for R2R and RxR-en, and evaluate with the same agent model EnvDrop. Results are shown in Table 11. The trend of different masking settings aligns with our previous findings in Table 10, and verifies that the background information is also crucial in the visual features.
5.2 Effect of Directions in the Environment
In this ablation setting, we randomly flip some of the viewpoints horizontally. The objects' relative positions at the flipped viewpoints will be reversed. Presumably, suppose the agent can follow the instruction and find the corresponding direction to approach. In that case, the flipped viewpoints will misguide the agent in the opposite direction and lower the navigation success rate. As shown in Table 10, flipping the viewpoints leads to drastic declines in the success rate for all listed indoor agents. This verifies our previous finding that indoor agents can understand directions in the instruction. We notice that FAST is the only listed model that is less affected by the direction flipping ablation than by the object masking ablation (#2 vs. #5). This suggests that FAST's asynchronous backtracking search is able to adjust and recover from errors that occur when choosing directions to some extent.
6 Analysis on Vision-Language Alignment
This section examines the agents' ability to learn vision-language alignment when executing the navigation. We focus on whether the agents can understand the objects mentioned in the instruction and align them to the correct visual instance in the environment, which is crucial to completing this multimodal task. To verify the existence of vision-language alignment, we add perturbations to the visual and textual input, and check how they affect agents' performance.
6.1 Instruction Side Perturbation
We add noise to the textual input by randomly replacing object tokens with random object tokens in the instruction. Table 5 shows an example (#3). This experiment aims to verify whether the agent can line the object tokens up to certain visual targets. The assumption is that if the agent can correctly align objects mentioned in the instruction to some targets in the visual environment, then replacing the object token will confuse and misguide the agent. Examining Figure 2, we notice that for all three datasets, the Transformer-based models have worse performance when replacing the object tokens, compared to simple masking. This indicates that Transformer-based models have a better cross-modal understanding of objects, and can align object tokens to the visual targets. Such superior performance may result from the fact that
(a) R2R
(b) RxR-en
Figure 2: Performance gap between masking and replacing object tokens from instructions. If  $\Delta > 0$ , then replacing object tokens leads to worse navigation performance, which suggests that the agent has a better understanding of the object tokens.
Setting	SR ↑ on R2R	SR ↑ on RxR-en
EnvDrop	FAST	Recur	PVLT	CLIPViL	HAMT
Vanilla	49.77	63.90	53.30	57.13	40.21	52.52
Dynamic Mask	-10%	-24%	-5%	-13%	-19%	-23%
Controlled Trial	-8%	-19%	-3%	-11%	-16%	-20%
Mask Tokens	-36%	-38%	-21%	-20%	-48%	-32%
Table 12: Indoor navigation performance when dynamically masking out the visual objects mentioned in the instructions. Recur: VLN  $\bigcirc$  BERT. PVLT: PREVALENT.
the Transformer-based models are often pre-trained on multimodal resources, thus displaying a slightly more vital ability to form alignment.
6.2 Environment Side Perturbation
We add noise to the visual input by conducting the following ablations. In the "dynamic mask" setting, we dynamically mask out the visual object regions mentioned in the instruction. We randomly mask out the same amount of visual objects at each viewpoint in its controlled trial. We also compare with the "mask tokens" setting, where we mask out all the object tokens in the instruction, while leaving the visual environment untouched. This
experiment aims to determine if the agent aligns the textual object tokens to the correct visual target. The assumption is that if the agent builds proper vision-language alignment and we mask out visual objects mentioned in the instruction, then the agent may get confused since it can not find the counterpart in the visual environment.
Results are shown in Table 12. The success rate witnesses a decline when dynamically masking out the visual objects. However, we notice in surprise that when all visual objects mentioned in the instruction are masked out, the agents can still reach a success rate higher than  $44\%$  on R2R and higher than  $32\%$  on RxR-en. This contradicts the previous assumption and casts doubt on the reliability of the navigation agents' vision-language alignment.
Comparing "dynamic mask" with the "mask tokens" setting, we notice that the visual object ablation has much smaller impact on navigation performance than the text object ablations, which suggests that current models have unbalanced attention on vision and text for the VLN task. Recent studies on pre-trained vision- and-language models (Frank et al., 2021) reveal that such asymmetry is also witnessed in other multimodal tasks. Future studies may follow the line of constructing a more balanced VLN agent.
In this paper, we inspect how the navigation agents understand the multimodal information by conducting ablation diagnostics input data. We find out that indoor navigation agents refer to both object tokens and direction tokens in the instruction when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. When it comes to vision-and-language alignments, we witness unbalanced attention on text and vision, and doubt the reliability of cross-modal alignments. We hope this work encourages more investigation and research into understanding neural VLN agents' black-box and improves the task setups and navigation agents' capacity for future studies.
We would like to show our gratitude to the anonymous reviewers for their thought-provoking comments. We would like to thank the Robert N. Noyce Trust for their generous gift to the University of California via the Noyce Initiative. The UCSB authors
were also sponsored by the U.S. Army Research Office, and this work was accomplished under Contract Number W911NF19-D-0001 for the Institute for Collaborative Biotechnologies. The views and conclusions contained in this research are those of the authors and should not be interpreted as representing the sponsors or the U.S. government's official policy, expressed or inferred. Regardless of any copyright notation herein, the United States government is authorized to reproduce and distribute reprints for government purposes.
Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen D. Isard, Jacqueline C. Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, Henry S. Thompson, and Regina Weinert. 1991. The hcrc map task corpus. Language and Speech, 34:351 - 366.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3674-3683. IEEE Computer Society.
Jacob Andreas and Dan Klein. 2015. Alignment-based compositional semantics for instruction following. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1165-1174, Lisbon, Portugal. Association for Computational Linguistics.
Giuliano Antoniol, Roldano Cattoni, and Mauro Cettolo. 2011. Robust speech understanding for robot telecontrol.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62.
Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016a. Natural language communication with robots. In NAACL.
Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016b. Natural language communication with robots. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 751-761, San Diego, California. Association for Computational Linguistics.
S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of
the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 82-90, Suntec, Singapore. Association for Computational Linguistics.
S.R.K. Branavan, Luke Zettlemoyer, and Regina Barzilay. 2010. Reading between the lines: Learning to map high-level instructions to commands. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1268-1277, Uppsala, Sweden. Association for Computational Linguistics.
Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from RGB-D data in indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 2017, pages 667-676. IEEE Computer Society.
David L. Chen and Raymond J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In AAAI 2011.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. TOUCHDOWN: natural language navigation and spatial reasoning in visual street environments. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 12538-12547. Computer Vision Foundation / IEEE.
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021. History aware multimodal transformer for vision-and-language navigation. In NeurIPS.
Soham Dan, Michael Zhou, and Dan Roth. 2021. Generalization in instruction following systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 976-981, Online. Association for Computational Linguistics.
Patrick Fernandes, Kayo Yin, Graham Neubig, and André F. T. Martins. 2021. Measuring and increasing context usage in context-aware machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6467-6478, Online. Association for Computational Linguistics.
Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers. In EMNLP.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein,
and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 3318-3329.
Sergio Guadarrama, Lorenzo Riano, David Hamilton Golland, Daniel Goehring, Yangqing Jia, Dan Klein, P. Abbeel, and Trevor Darrell. 2013. Grounding spatial relations for human-robot interaction. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1640-1647.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pretraining. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13134-13143. IEEE.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016. Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 3-19. Springer.
Yicong Hong, Cristian Rodriguez Opazo, Yuankai Qi, Qi Wu, and Stephen Gould. 2020a. Language and visual entity relationship graph for agent navigation. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez Opazo, and Stephen Gould. 2020b. A recurrent vision-and-language BERT for navigation. CoRR, abs/2011.13922.
Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhães, Jason Baldridge, and Eugene Ie. 2019. Transferable representation learning in vision-and-language navigation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 7403-7412. IEEE.
Vihan Jain, Gabriel Magalhães, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019a. Stay on the path: Instruction fidelity in vision-and-language navigation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1862-1872. Association for Computational Linguistics.