SlowGuess/NAACL_2021_OCR huggingface dataset:
Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, and E. Ie. 2019a. Multi-modal discriminative model for vision-and-language navigation. ArXiv, abs/1905.13358.
Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, and Eugene Ie. 2019b. Transferable representation learning in vision-and-language navigation. In Proceedings of the IEEE International Conference on Computer Vision, pages 7404-7413.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in vision-and-language navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1862-1872.
Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. 2019. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6741-6749.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954.
Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, and Rita Cucchiara. 2019. Perceive, transform, and act: Multi-modal attention networks for vision-and-language navigation. arXiv preprint arXiv:1911.12377.
Xiujun Li, C. Li, Qiaolin Xia, Yonatan Bisk, A. Celikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. 2019. Robust navigation with language pretraining and stochastic sampling. In EMNLP/IJCNLP.
Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan Al-Regib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019a. Self-monitoring navigation agent via auxiliary progress estimation. In Proceedings of the International Conference on Learning Representations (ICLR).
Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. 2019b. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6732-6740.
Gabriel Magalhaes, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. Effective and general evaluation for instruction conditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. 2020. Improving vision-and-language navigation with image-text pairs from the web. arXiv preprint arXiv:2004.14973.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55-60.
Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environments with visual goal prediction. In EMNLP.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105-1116.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937.
Khanh Nguyen and Hal Daume III. 2019. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 684-695.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020a. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. 2020b. Object-and-action aware model for visual language navigation. arXiv preprint arXiv:2007.14626.
Jinfeng Rao, Kartikeya Upasani, Anusha Balakrishnan, Michael White, Anuj Kumar, and Rajen Subba. 2019. A tree-to-sequence model for neural nlg in task-oriented dialog. In Proceedings of the 12th International Conference on Natural Language Generation, pages 95-100.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge.
International Journal of Computer Vision (IJCV), 115(3):211-252.
Chao Su, Heyan Huang, Shumin Shi, Ping Jian, and Xuewen Shi. 2020. Neural machine translation with gumbel tree-lstm based encoder. Journal of Visual Communication and Image Representation, page 102811.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1556-1566.
Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2610-2621.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In Conference on Robot Learning, pages 394-406.
Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. 2020a. Active visual information gathering for vision-language navigation. arXiv preprint arXiv:2007.08037.
Hu Wang, Qi Wu, and Chunhua Shen. 2020b. Soft expert reward learning for vision-and-language navigation. arXiv preprint arXiv:2007.10835.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6629-6638.
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. 2018. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 37-53.
Qiaolin Xia, Xiujun Li, Chunyuan Li, Yonatan Bisk, Zhifang Sui, Jianfeng Gao, Yejin Choi, and Noah A Smith. 2020. Multi-view learning for vision-and-language navigation. arXiv preprint arXiv:2003.00857.
Jiannan Xiang, Xin Eric Wang, and William Yang Wang. 2020. Learning to stop: A simple yet effective approach to urban vision-language navigation. arXiv preprint arXiv:2009.13112.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020a. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10012-10022.
Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, and Fei Sha. 2020b. Babywalk: Going farther in vision-and-language navigation by taking baby steps. In ACL, pages 2539-2556.
Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, and Xiaodan Liang. 2020c. Vision-dialog navigation by exploring cross-modal memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Vision-Language navigation requires an agent to navigate through a 3D environment to a target location based on a given natural language instruction. Formally, the natural language instruction is a sequence of words  $\{w_{i}\}_{i = 1}^{l}$ , where  $l$  is the length of the instruction and  $w_{i}$  is  $ith$  word in the sequence. At each time step, the agent perceives a panoramic view of the current viewpoint. Besides, the agent has access to a set of navigable locations  $\{l_{t,k}\}_{k = 1}^{K}$ , where  $K$  is the total number of reachable locations from the current viewpoint. The agent needs to select an action  $a_{t}$  from the list of navigable viewpoints  $\{l_{t,k}\}_{k = 1}^{K}$  based on the given instruction, navigation history and current panoramic views. If the viewpoint selected from the list is the same as the current viewpoint, the agent predicts a "STOP" action.
A.2 Model Details
Visual Encoder. Same as previous work in VLN, at time step  $t$ , we discretize the panoramic view into 36 single views  $\{o_{t,p}\}_{p=1}^{36}$ . Each single view is a RGB image, annotated with its angles of heading and elevation  $(\theta_{t,p}, \phi_{t,p})$ . Each RGB image is encoded with a pre-trained ResNet-152 (He et al., 2016) on ImageNet (Russakovsky et al., 2015). A four dimension orientation feature  $(\cos \theta_{t,p}, \sin \theta_{t,p}, \cos \phi_{t,p}, \sin \phi_{t,p})$  is concatenated with the ResNet feature to form the final representation for the view  $p$  of a panoramic  $\{f_{t,p}\}_{p=1}^{36}$ . Similarly, we get the visual representation  $\{g_{t,k}\}_{k=1}^{K}$  for  $K$  navigable locations  $\{l_{t,k}\}_{k=1}^{K}$  at time step  $t$ .
We use a mixture of imitation learning and reinforcement learning to train the agent.
Imitation Learning. During training, instead of navigating to the predicted action at each time step, teacher-forcing is used to determine which navigable viewpoint to pick. Given the shortest path between the start point and target point, at each time step  $t$ , the agent tries to imitate the teacher action  $a_{t}^{\star}$  by minimizing the negative log probability:
$$
L _ {I L} = \sum_ {t} - a _ {t} ^ {\star} \operatorname {l o g p} _ {t} \tag {11}
$$
Reinforcement Learning. We combine imitation learning with reinforcement learning to learn a more generalizable agent. Since the teacher path is the shortest path between the start point and the target point, there is no guarantee that the teacher path is the same as indicated by the given instruction. Thus, reinforcement learning is applied for better instruction following and state exploration. At each time step  $t$ , the agent samples an action  $a_{t}$  from the predicted distribution  $p_{t}(a_{t})$ . At each time step, if the agent moves closer to the target viewpoint, a positive reward  $+1$  is given, otherwise the agent receives a negative reward -1. When the agent predicts the "STOP" action, the agent will receive a  $+3/-3$  reward based on whether the agent is within 3m from the target viewpoint. We use Actor-Critic (Mnih et al., 2016) to train the agent. The loss of reinforcement learning is:
$$
L _ {R L} = \sum_ {t} \left(R _ {t} - R _ {b _ {t}}\right) \log p _ {t} \left(a _ {t}\right) + \eta H \left(p _ {t} \left(a _ {t}\right)\right) \tag {12}
$$
where  $R_{t}$  is the discounted future cumulative rewards at time step  $t$ ,  $R_{b_t}$  is the expected cumulative rewards (baseline) approximated by the value function  $V$ ,  $H(p_{t}(a_{t}))$  is the entropy term for regularization. Specifically,
$$
R _ {t} = r _ {t} + \sum_ {i = 1} ^ {T - t} \gamma^ {i} r _ {t + i} \tag {13}
$$
$$
R _ {b _ {t}} = V (h _ {t}) = W _ {v _ {2}} \sigma \left(W _ {v _ {1}} h _ {t}\right) \tag {14}
$$
where  $\sigma$  is the ReLU activation function, and  $r_t$  is the immediate reward we defined earlier. The value function is trained with L2 loss:
$$
L _ {V} = \frac {1}{2} \left(R _ {t} - R _ {b _ {t}}\right) ^ {2} \tag {15}
$$
We optimize a mixture loss of imitation learning and reinforcement learning:
$$
L _ {M I X} = \left(L _ {R L} + L _ {V}\right) + \lambda L _ {I L} \tag {16}
$$
R2R Dataset. The R2R dataset contains 21567 human annotated instructions with an average instruction lengths of 29. The training set contains 14025 instructions in 61 environments. The seen validation set contains 1020 instructions in the same 61 environments as the training set. The unseen validation set contains 2349 instructions in 11 environment which is not included in the training set. The test set contains 4173 instructions in 18 environments.
RxR Dataset. The RxR dataset is an extension to the R2R dataset, where the instructions are longer and in languages other than English (Hindi and Telugu). It contains 126069 instructions with an average instruction length of 78. Besides, different from R2R that only contains guide path (i.e., the shortest path between the start point and the target point), RxR pairs each guide path with a human-annotated follower path (i.e., the path that human generates following the instruction). We only use the guide path to train the agent in this paper. The seen and unseen environment split is the same as in the R2R dataset. There are 16522 paths in total, and each path is annotated in 3 languages. The training set contains 11089 paths, the seen validation set contains 1232 paths, the unseen validation contains 1517 paths, and the test set contains 2684 paths.
A.5 Evaluation Metrics
To evaluate the performance of our model, we use the following evaluation metrics: (1) Success Rate (SR): If the agent stops less than  $3\mathrm{m}$  from the target location, we consider the navigation as a success. (2) Success Rate Weighted by Path Length (SPL) (Anderson et al., 2018a): This metric penalizes long trajectories (e.g., find the target using beam search over the environment graph). (3) normalized Dynamic Time Warping (nDTW) (Magalhaes et al., 2019): This metric penalizes deviations from the reference path. (4) success rate weighted by Dynamic Time Warping (sDTW) (Magalhaes et al., 2019): This metric constraints nDTW only to successful navigation and considers path fidelity and agent success. (5) Coverage weighted by Length
Score (CLS) (Jain et al., 2019): Similar to nDTW, this metric also encourages path fidelity.
A.6 Implementation Details
We generate the dependency parse tree for English using Stanford CoreNLP (Manning et al., 2014). We use Stanza Toolkit (Qi et al., 2020a) to generate the dependency parse tree for Hindi and Telugu. For both baseline model and our syntax-aware model, the learned word embedding size is 256 and the dimension of the action embedding is 128. We set the hidden size for the bi-directional LSTM to be 256. For syntax-aware model, the hidden size for the bi-LSTM and Tree-LSTM in the language encoder is 512. We use ResNet-152 (He et al., 2016) to extract the 2048 dimensional image feature. We use RMSProp (Hinton et al., 2012) as the optimizer with learning rate 1e-4 and batch size 64. The weight  $\lambda$  we use to combine imitation learning loss and reinforcement learning loss is set to be 0.2. In reinforcement learning, the discount factor  $\gamma$  is 0.9 and the entropy weight  $\eta$  is 0.01. During training, we set the max action length to be 35 for R2R dataset and 70 for RxR dataset. We train both agents on R2R for 80,000 iterations. We train both agents on RxR for 200,000 iterations. The baseline model contains approximate 6 million parameters. Our syntax-aware model contain approximate 8 million parameters.