SlowGuess/NAACL_2022_OCR huggingface dataset:
In our experiments, we learn the shared cross-lingual representation based on cased multilingual  $\mathrm{BERT}_{\mathrm{BASE}}$ . For the pre-trained vision model, we compare performance between image features extracted from ImageNet-pre-trained (Russakovsky et al., 2015) ResNet-152 (He et al., 2016) and CLIP-pre-trained (Radford et al., 2021) vision transformer (ViT-B/32) (Dosovitskiy et al., 2021) (abbreviated as 'CLIP feature' later). More details
about representation learning and navigation training can be found in Appendix.
5.1 Test Set Results
We compare our final agent model with results on the Room-Across-Room (RxR) leaderboard. Our agent is a multilingual model that learn three languages in the same model. Compared with monolingual agents that learn instructions in three languages separately, a multilingual agent performs worse due to high-resource languages degradation (Ku et al., 2020; Aharoni et al., 2019; Pratap et al., 2020). Our agent is tested under the single-run setup. In the single-run setting, the agent only navigates once and does not pre-exlore the test environment. As shown in Table 1, our CLEAR model with CLIP features is  $16.88\%$  higher in nDTW score than the baseline mono-lingual model (Ku et al., 2020) ('RxR') that utilizes ResNet features and other base navigation model. Furthermore, our model is  $2.59\%$  higher in nDTW score than the mono-lingual model (Shen et al., 2022) ('CLIP') that utilizes CLIP features and the same base navigation model as ours.
5.2 Ablation Results
We demonstrate the effectiveness of our learned visual and language representations with ablation studies. The baseline model (annotated as 'ResNet' in Table 2) uses multilingual BERT and pre-trained ResNet to encode instructions and images without the representation learning stage. Our CLEAR-ResNet ('ResNet+both' in Table 2) outperforms its baseline models in all evaluation metrics on average. Specifically, it improves the baseline model by  $5.3\%$  in success rate (SR) and  $4.3\%$  in nDTW score on average over three languages. These results demonstrate that our CLEAR agent is not only more capable of reaching the target, but also follows the ground-truth path better.
We then show that both the cross-lingual language representation and environment-agnostic visual representation contribute to the overall improvement. When the cross-lingual language representation is added ('+text'), we see consistent improvement on the averaged metrics and observe that Hindi benefits most from the cross-lingual language representation. When adding the environment-agnostic visual representation ('+visual'), the nDTW score improves by  $2.6\%$ . These
Models	SR↑	SPL↑	NDTW↑	SDTW↑
avg	en	hi	te	avg	en	hi	te	avg	en	hi	te	avg	en	hi	te
RxR	22.8	22.2	23.0	23.1	20.4	19.8	20.7	20.7	38.9	38.6	39.2	38.8	18.2	17.8	18.3	18.4
ResNet	35.1	35.4	36.4	33.4	31.6	31.6	33.0	30.4	51.1	50.7	52.3	50.3	30.1	30.1	31.4	28.7
+text	36.0	36.1	37.6	34.3	31.7	31.7	33.2	30.3	52.0	52.3	53.4	50.2	30.5	30.5	32.0	29.1
+visual	35.6	35.8	36.9	33.9	32.5	32.6	33.9	31.0	53.7	53.6	55.1	52.5	30.5	30.5	31.7	29.1
+both	40.4	41.5	42.2	37.6	36.5	36.7	38.5	34.3	55.4	54.4	57.8	54.1	34.6	35.1	36.4	32.2
CLIP	41.7	42.5	44.0	38.6	37.1	37.2	39.2	34.8	55.8	55.6	57.3	54.5	35.6	36.3	37.6	33.3
+both	44.4	46.0	46.0	41.1	39.3	40.1	41.0	36.9	57.0	57.2	58.1	55.7	37.8	38.7	39.3	35.3
Table 2: Ablation study of our model with ResNet features and CLIP features on validation unseen sets. 'avg' is the agent's average performance on English, Hindi, and Telugu instructions.
Methods	SR↑	SPL↑	NDTW↑	SDTW↑
m-BERT	35.1	31.6	51.1	30.1
Mono	32.9	30.4	51.4	28.0
Multi	36.0	31.7	52.0	30.5
Table 3: Comparison between language representation trained with mono-lingual instruction pairs ('Mono') and multi-lingual instruction pairs ('Multi') on validation unseen sets. 'm-BERT' is the method that uses original multilingual BERT as language representation.
improvements validate the effectiveness of our learned language and visual representations.
Moreover, we show that our CLEAR approach could generalize to other pre-trained visual features. We implement another model (annotated as 'CLIP' in Table 2) that uses CLIP to encode images, which is a stronger baseline compared with the ResNet baseline ('ResNet' in Table 2). Our CLEAR-CLIP model ('CLIP+both' in Table 2) also shows  $2.7\%$  improvement in success rate (SR) and  $1.2\%$  improvement in nDTW score on average over three languages. This demonstrates the effectiveness of our CLEAR approach over different pre-trained visual features.
6.1 Effectiveness of Cross-Lingual Representations
In this section, we show the effectiveness of our language representation learning method described in Sec. 3.1. We first show the effectiveness of using paired multilingual instructions instead of monolingual instructions in the language representation learning stage. Then, we show that our learned cross-lingual language representation captures the visual concepts behind the instruction better than the original multilingual BERT representation.
Multilingual vs. monolingual. To show that the multilingual instruction pairs are crucial for our cross-lingual language representation learning, we experiment with fine-tuning multilingual
BERT with instruction pairs in same language only ('Mono' in Table 3). We observe that compared with the agent with cross-lingual representation ('Multi'), the success rate decreases by  $3.1\%$  and sDTW score decreases by  $2.5\%$ . Furthermore, compared with the baseline model that uses the original multilingual-BERT ('m-BERT'), the success rate drops  $2.2\%$  and the sDTW score drops  $2.1\%$ . This result indicates that instruction representations in one language cannot benefit from learning representation in other languages if the multi-lingual representation is only supervised by contrastive loss between mono-lingual instruction pairs.
Capturing visual concepts. Our cross-lingual language representation can ground to the visual environment more easily by capturing the visual concepts in the instruction. We demonstrate that shared visual concepts in different paths are captured by our language representation. We first encode the instruction as in Eqn. 2 with cross-lingual representation and original multilingual BERT separately. For every instruction, we retrieve another instruction with the highest cosine similarity under the constraints that two instructions don't correspond to the same path and equal path length. As shown in Figure 3, the second row is the query instruction and the first row is its corresponding path. The following four rows correspond to the instruction-path picked with cross-lingual representation and multilingual-BERT representation. First, we observe in Figure 3 that our cross-lingual representation retrieves a Hindi instruction while the multilingual-BERT picks an English instruction. This indicates that our cross-lingual representation learns to encode instructions with similar semantics in different languages closer to each other. Besides, we observe that in all three paths, the agent passes tables and chairs, but only in the query path and the cross-lingual paired path, the agent stops at places similar to "bar stools". This demonstrates that the visual objects in the cross-lingual picked path are
Figure 3: Comparison of the most similar instruction picked with cross-lingual representation and multilingual BERT. Our cross-lingual picked instruction mentions more visual object as in the query instruction. Besides, the path corresponding to the cross-lingual picked instruction contains more accurate visual objects as in the query path.
Similarity	SR↑	SPL↑	NDTW↑	SDTW↑
mono	32.9	30.4	51.4	28.0
en+hi	32.0	28.3	48.4	26.9
en+te	30.0	27.2	48.8	25.4
hi+te	27.8	25.0	46.1	23.2
multi	36.0	31.7	52.0	30.5
Table 4: Comparison between language representation trained with different instruction pairs. 'mono' indicates representation trained with mono-lingual instruction pairs, 'multi' indicates representation trained with multi-lingual instruction pairs in all three languages, and 'en+hi' indicates representation trained with multilingual instruction pairs in English and Hindi only.
more similar to the objects in the query path.
6.2 Effectiveness of Optimizing Similarity between Three Languages
In this section, we further show that only optimizing the similarity between a subset of languages (i.e., two out of three languages) will hurt the performance. Specifically, we train the language representation that optimizes similarity between only English and Hindi ('en+hi'), only English and Telugu ('en+te'), only Hindi and Telugu ('hi+te'), and only single language ('mono'). Given paired language instructions in English and Hindi in unseen set, the average distance is 0.61 for our language representation (i.e., optimizes similarity between all three languages), 0.43 for  $en+te$ , 1.67 for  $hi+te$ , and 1.55 for same language only, indicating that explicitly optimizing the similarity between  $en+hi$  helps reduce the distance between  $en+hi$  most. Adding  $te$  in optimization will make  $en+hi$  farther from each other, but still much better than
only optimizing  $hi + te$ , and could also make the distance between all three languages to be closer to each other. We further show the performance of training the navigation agent with these language representations in Table 4. We observe that both the success rate and the nDTW score drop significantly when only training on a subset of languages. This result shows that it's crucial to train the language representation with instruction pairs in all three languages.
6.3 Decreasing Gap between Seen and Unseen Environments
Most previous navigation models (Wang et al., 2019a; Ma et al., 2019a; Majumdar et al., 2020) suffer from a large performance drop when moving from seen validation to unseen validation because the visual encoder overfits the low-level appearance features (Zhang et al., 2020). Our environment agnostic visual representation can decrease the performance gap between validation seen and unseen environments. As shown in Table 5, the nDTW gap is decreased from 1.6 to 1.0 compared with baseline model. It is also lower than the gap of multi-lingual agent in Ku et al. (2020).
6.4 Comparison with Other Contrastive Learning Approaches
In this section, we compare with SimCSE (Gao et al., 2021), an effective contrastive learning approach for text representation learning. We use SimCSE on our visual representation learning, where we use dropout as positives in contrastive learning. Using SimCSE to train the visual representation gets 34.8/53.0 (SR/nDTW),
Models	seen	unseen	|Δ|
SR	SPL	NDTW	SDTW	SR	SPL	NDTW	SDTW	SR	SPL	NDTW	SDTW
Ku et al. (2020)	25.2	-	42.2	20.7	22.8	-	38.9	18.2	2.4	-	3.3	2.5
ResNet	38.4	34.1	52.7	32.6	35.1	31.6	51.1	30.1	3.3	2.5	1.6	2.5
+visual	34.1	31.1	52.7	28.8	35.6	32.5	53.7	30.5	1.5	1.4	1.0	1.7
Table 5: The results of adding our learned visual representation on validation seen environments and validation unseen environments.  $|\Delta|$  indicates absolute performance difference between seen and unseen environments.
Models	SR↑	SPL↑	NDTW↑	SDTW↑
ResNet	49.1	44.7	58.8	42.0
+text	49.0	45.2	59.5	42.3
+visual	50.4	46.3	60.3	43.4
CLEAR	50.5	46.4	60.6	43.3
ResNet-zero	30.9	27.9	49.0	26.3
CLEAR-zero	35.4	30.1	49.0	28.2
Table 6: Results on R2R validation unseen environments. "CLEAR" (based on ResNet) transfers the language and visual representation from RxR dataset, and "ResNet" is the baseline model that uses multilingual BERT and pre-trained ResNet. "ResNet-zero" and "CLEAR-zero" are zero-shot performance of baseline and our approach on R2R dataset.
which is lower than our visual representation (35.6/53.7). Furthermore, we experiment with using both dropout as positives and our identified path pairs as positives. The performance decreases in nDTW score (52.4) compared with only using our identified path pairs as positives (53.7).
6.5 Generalization to Other VLN Tasks
We further evaluate our CLEAR approach's generalizability on Room-to-Room (R2R) dataset (Anderson et al., 2018b) and Cooperative Vision-and-Dialog Navigation (CVDN) dataset (Thomason et al., 2019), in which we directly transfer our CLEAR approach and train on the navigation task on R2R and CVDN. R2R and CVDN follows the same training, validation seen, and validation unseen split of environments as Room-Across-Room dataset. The main difference is that the language instructions in R2R and CVDN is monolingual (i.e., English). Besides, instructions in CVDN are multi-round dialogues between navigator and the oracle. Our baseline model uses multilingual BERT to encode instructions and the ResNet pretrained on ImageNet to extract image features. The cross-lingual language representation and environment-agnostic visual representation is trained on RxR dataset (as in Sec. 3.1 and Sec. 3.2). We then train the navigation agent on R2R dataset and CVDN dataset with the language and visual encoder initialized from our CLEAR representation.
As shown in Table 6, on R2R dataset, our learned
representation outperforms the baseline by  $1.4\%$  in success rate and  $1.8\%$  in nDTW. Furthermore, we show that the zero-shot performance of our approach improves the baseline by  $4.5\%$  in success rate and  $2.2\%$  in SPL on R2R dataset. On CVDN dataset, our learned representation outperforms the baseline by 0.74 in Goal Progress (4.05 vs. 3.31) after training on CVDN dataset, and outperforms the baseline by 0.42 in Goal Progress (0.92 vs. 0.50) in the zero-shot setting. Goal Progress measures the progress made towards the target location and is the main evaluation metric in CVDN. This result demonstrates that our learned cross-lingual and environment agnostic representation could generalize to other tasks.
6.6 Generalization to Other VLN Agents
We further evaluate our CLEAR approach's generalizability to other VLN agents. Specifically, we adapt CLEAR to SotA VLN agent HAMT (Chen et al., 2021). With the pre-trained weights released in HAMT, we further learn the text representation and visual representation with our approach. Adapting CLEAR to HAMT achieves  $57.2\%$  in success rate and  $65.6\%$  in nDTW score, which is  $0.7\%$  higher than HAMT in success rate and  $2.5\%$  higher than HAMT in nDTW score on RxR validation unseen set, demonstrating the effectiveness of our proposed approach over SotA VLN models.
In this paper, we presented the CLEAR method that learns a cross-lingual and environment-agnostic representation. We demonstrated that our cross-lingual language representation captures more visual semantics and our environment-agnostic representation generalizes better to unseen environments. Our experiments on Room-Across-Room dataset suggest that our CLEAR method improved the performance in all evaluation metrics over a strong baseline. Furthermore, we qualitatively and quantitatively analyze the effectiveness of every component of our CLEAR approach and its generalizability to other tasks and base VLN agents.
In this paper, we presented a method to learn cross-lingual and environment-agnostic representations for Vision-and-Language Navigation. Vision-and-Language Navigation task can be used in many real-world applications, for example, a home service robot can bring things to the owner based on natural language instructions, making people's life easier. Our learned representations enable the agent to understand multi-lingual instructions and improve agents' generalizability to unseen environments. However, currently we learn our cross-lingual representation from three languages (i.e., English, Hindi, and Telugu) only due to dataset availability, which might limit its generalization to other languages. Besides, similar to other instruction-following agent, our agent might fail to reach the target given some instructions, which requires further human assistance.
We thank the reviewers for their helpful comments. This work was supported by ARO W911NF2110220, ONR N000141812871, DARPA KAIROS FA8750-19-2-1004, Google Focused Award. The views contained in this article are those of the authors and not of the funding agency.
Roee Aharoni, Melvin Johnson, and Orhan First. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.
Dong An, Yuankai Qi, Yan Huang, Qi Wu, Liang Wang, and Tieniu Tan. 2021. Neighbor-view enhanced model for vision and language navigation. In MM '21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 5101-5109. ACM.
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. 2018a. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen
Gould, and Anton van den Hengel. 2018b. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683.
Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A Plummer. 2020. Learning to scale multilingual representations for vision-language tasks. In European Conference on Computer Vision, pages 197-213. Springer.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, pages 12538-12547.
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021. History aware multimodal transformer for vision-and-language navigation. Advances in Neural Information Processing Systems, 34.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*, pages 4171–4186.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In Advances in Neural Information Processing Systems, pages 3314-3325.
Tsu-Jui Fu, Xin Eric Wang, Matthew F Peterson, Scott T Grafton, Miguel P Eckstein, and William Yang Wang. 2020. Counterfactual vision-and-language navigation via adversarial path sampler. In European Conference on Computer Vision, pages 71-86. Springer.
Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? dataset and methods for multilingual image question answering. Neurips.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In EMNLP.
Jiuxiang Gu, Shafiq Joty, Jianfei Cai, and Gang Wang. 2018. Unpaired image captioning by language pivoting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 503-519.
Agrim Gupta, Piotr Dollar, and Ross Girshick. 2019. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5356-5364.
Deepak Gupta, Pabitra Lenka, Asif Ekbal, and Pushpak Bhattacharyya. 2020. A unified framework for multilingual and code-mixed visual question answering. In ACL, pages 900-913.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137-13146.
Kaiming He, Georgia Gkioxari, Piotr Dólar, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8).
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. 2021. A recurrent vision-and-language bert for navigation. In CVPR.
Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, and E. Ie. 2019a. Multi-modal discriminative model for vision-and-language navigation. SpLU-RoboNLP Workshop at NAACL, abs/1905.13358.
Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, and Eugene Ie. 2019b. Transferable representation learning in vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7404-7413.
Po-Yao Huang, Junjie Hu, Xiaojun Chang, and Alexander Hauptmann. 2020. Unsupervised multimodal neural machine translation with pseudo visual pivoting. ACL.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in vision-and-language navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1862-1872, Florence, Italy. Association for Computational Linguistics.
Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. 2019. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In CVPR, pages 6741-6749.
Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, and Bryan Plummer. 2020. Mule: Multimodal universal language embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11254-11261.
Hyounghun Kim, Jialu Li, and Mohit Bansal. 2021. Ndh-full: Learning and evaluating navigational agents on full-length dialogue. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6432-6442.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. 2020. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In ECCV, pages 104-120.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In EMNLP, pages 4392-4412.
Weiyu Lan, Xirong Li, and Jianfeng Dong. 2017. Fluency-guided cross-lingual image captioning. In Proceedings of the 25th ACM international conference on Multimedia, pages 1549-1557.
Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, and Rita Cucchiara. 2021. Multimodal attention networks for low-level vision-and-language navigation. Computer Vision and Image Understanding.
Jialu Li, Hao Tan, and Mohit Bansal. 2021. Improving cross-modal alignment in vision language navigation via syntactic information. In ACL, pages 1041-1050.
Jialu Li, Hao Tan, and Mohit Bansal. 2022. Envedit: Environment editing for vision-and-language navigation. In CVPR.
Xiujun Li, C. Li, Qiaolin Xia, Yonatan Bisk, A. Celikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. 2019. Robust navigation with language pretraining and stochastic sampling. In EMNLP/IJCNLP.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. *ICLR*.
Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019a. Self-monitoring navigation agent via auxiliary progress estimation. In Proceedings of the International Conference on Learning Representations (ICLR).
Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. 2019b. The regretful agent: Heuristic-aided navigation through progress estimation. In CVPR, pages 6732-6740.