Venue: NAACL 2021

# Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Jialu Li Hao Tan Mohit Bansal

UNC Chapel Hill

{jialuli, airsplay, mbansal}@cs.unc.edu

###### Abstract

Vision language navigation is the task that requires an agent to navigate through a 3D environment based on natural language instructions. One key challenge in this task is to ground instructions with the current visual information that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction required for the next action. However, different words have different functions in a sentence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like dependencies and phrase structures can aid the agent to locate important parts of the instruction. Hence, in this paper, we propose a navigation agent that utilizes syntax information derived from a dependency tree to enhance alignment between the instruction and the current visual scenes. Empirically, our agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the current visual information via qualitative visualizations.1

Footnote 1: Code and models: [https://github.com/jiailuli-luka/SyntaxVLN](https://github.com/jiailuli-luka/SyntaxVLN)

## 1 Introduction

Vision-Language Navigation defines the task of requiring an agent to navigate through a visual environment based on natural language instructions Anderson et al. (2018); Misra et al. (2018); Chen et al. (2019); Jain et al. (2019); Nguyen and Daume III (2019); Thomason et al. (2020). This task poses several challenges. To complete this task, an embodied agent needs to perceive the surrounding environment, understand the given natural language instructions, and most importantly, ground (or align) the instruction in the visual scenes. In this paper, we aim to make one step towards grounding natural language instructions with visual environment via syntax-enriched alignment.

Recently, several approaches were proposed to solve the Vision-Language Navigation task with better interactions between natural language instructions and visual scenes Fried et al. (2018); Wang et al. (2019); Landi et al. (2019); Wang et al. (2020); Huang et al. (2019); Hu et al. (2019); Majumdar et al. (2020); Ma et al. (2019); Qi et al. (2020); Zhu et al. (2020),c). Some approaches utilize soft attention over individual words for better cross-modal grounding, while others improve co-grounding with better language and vision representation and additional alignment module.

Although these models achieve significant improvement in performance, they do not explicitly consider syntactic linguistic information in their alignment and decision-making. We argue that

Figure 1: An example in the Room-to-Room task. We generate the dependency parse tree for the instruction. The words are grouped by the head node (highlighted in the tree). Each sub-instruction (i.e., grouped words) corresponds to one step in the navigation with the same color. Modifiers in red boxes can be easily identified from the tree structure.



the syntactic information (e.g., phrases, word functions, modifiers) captured by dependency parse trees is crucial for accurate alignment between the instructions and the environment. As shown in Figure 1, for the instruction _"Walk forward then turn right at the stairs then go down the stairs."_, the dependency parse tree effectively aggregates syntactically-close words together for the agent (e.g., groups phrase information _"Walk forward"_ at node _"Walk"_), and each phrase here corresponds to one navigation action. Besides, the dependency tree structure also helps identify modifiers like _"forward"_ and _"right"_. This syntactic information helps the agent identify important words, locate phrases (e.g., sub-instructions), and learn a better alignment between the instruction and the visual environment.

Therefore, in this paper, we propose an encoder module that can incorporate simple but important syntactic information from parse trees for the vision-language-navigation task. Our proposed encoder utilizes the ChildSum Tree-LSTM Tai et al. (2015) over a dependency tree to achieve a syntax-aware representation of the instruction, enabling the agent to focus on more syntactically important words and align not only words but also phrases with the visual scenes.

We conduct experiments on both the Room-to-Room (R2R) dataset Anderson et al. (2018) and the Room-across-Room (RxR) dataset Ku et al. (2020). Empirical results show that our proposed approach significantly improves the performance over the baseline model on success rate and achieves the new state-of-the-art (at the time of submission) on the RxR dataset, which contains instruction in three languages English, Hindi, and Telugu (catalog). Moreover, by using structured information from syntax, we are also able to avoid word-level shallow overfitting of the model and hence achieve better generalization in the unseen environment. Our analysis further shows that our syntax-aware agent has better interpretability and learns better cross-modality matching.

## 2 Related Work

Visual and Textual Grounding in VLN.In vision-language navigation tasks, visual and textual co-grounding aims to learn the relationship between natural language instructions and the visual environments. A main line of research in VLN utilizes soft attention over individual words for cross-modal grounding in both the natural language instruction and the visual scene Wang et al. (2019); Tan et al. (2019); Landi et al. (2019); Xia et al. (2020); Wang et al. (2020); Ma et al. (2020); Zhu et al. (2020). Other works improve vision and language representations Hu et al. (2019); Li et al. (2019); Huang et al. (2019); Hao et al. (2020); Majumdar et al. (2020) and propose an additional progress monitor module Ma et al. (2019); Ba et al. (2019) and object and action aware modules Qi et al. (2020) that aid co-grounding.

The closest work to ours is from Hong et al. (2020), where they use the dependency tree to generate pre-divided sub-instructions, and then propose a shifting module to select and attend to a single sub-instruction at each time-step, which implicitly captures some syntax information. Compared with them, we directly use the dependency tree to explicitly incorporate syntactic information and get syntax-aware instruction representations, hence achieving substantial improvement in generalizing to the unseen environment.

Tree-based Language Representations.Dependency tree provides essential syntactic information for understanding a sentence. Tree-LSTM Tai et al. (2015) has been widely used to encode parsed tree information and shown improvement over multiple tasks, such as relation extraction Miwa and Bansal (2016); Geng et al. (2020), machine translation Su et al. (2020); Choi et al. (2017); Eriguchi et al. (2016), dialogue Rao et al. (2019), and language inference Chen et al. (2017). We are novel in incorporating a dependency tree into the vision-language navigation task via a Tree-LSTM for better phrase-level alignment between the visual environment and language instructions.

## 3 Method

As illustrated in Figure 2, our base model follows the sequence-to-sequence architecture of previous VLN agents. Our tree-based encoder module is built on top of the strong Environment Drop Agent Tan et al. (2019). The main difference is that we employ a tree-based language encoder to encode dependency tree information to allow better language grounding. At each time step, we ground all the encoded nodes (i.e., syntax-aware representations) in the dependency tree with the visual information to get the attended textual representation 



#### Generating Dependency Tree Representation with Tree-LSTM.

We first generate the dependency parse tree with Stanford CoreNLP Manning et al. (2014) for English instructions and Stanza Toolkit Qi et al. (2020) for Hindi and Telugu instructions. Since the tree-structure is invariant to the children's order (i.e., switching the order of the children of a node doesn't change the tree structure), directly using a Tree-LSTM over an embedding layer may lose important word order information in the instruction. Thus, here we use a bidirectional LSTM with an embedding layer to generate word representations that preserve sequential information of the instruction. Specifically, given an instruction \({\left\{w_{i}\right\}^{l}_{i=1}}\), we generate syntax-aware representation \({\left\{\hat{u}_{dep_{i}}\right\}^{l}_{i=1}}\) as:

\[\hat{w}_{i} =\mathrm{Embedding}(w_{i}) \tag{1}\] \[u_{1},u_{2},...,u_{l} =\text{Bi-LSTM}(\hat{w}_{1},\hat{w}_{2},...,\hat{w}_{l})\] (2) \[{\left\{\hat{u}_{dep_{i}}\right\}^{l}_{i=1}} =\mathrm{Tree}\text{-LSTM}({\left\{u_{i}\right\}^{l}_{i=1}}) \tag{3}\]

#### Visual Encoder and Navigation Decoder.

Given panoramic features \({\left\{f_{t,p}\right\}^{36}_{p=1}}\) and visual representation \({\left\{g_{t,k}\right\}^{K}_{k=1}}\) for \(K\) navigable locations at time step \(t\)2, we picks the next viewpoint from \(K\) navigable locations as:

Footnote 2: Details for generating these features are in Appendix.

\[p_{t}(a_{t}=k)=\mathrm{Softmax}_{k}(g_{t,k}^{T}W_{G}\widetilde{h}_{t}) \tag{4}\]

where \(\widetilde{h}_{t}\) is the context aware hidden states, and \(W_{G}\) is learned weight parameter. Specifically, we compute the \(\widetilde{h}_{t}\) as:

\[\beta_{t,p} =\mathrm{softmax}_{p}(f_{t,p}^{T}W_{F}\widetilde{h}_{t-1}) \tag{5}\] \[\widetilde{f}_{t} =\sum_{p}\beta_{t,p}f_{t,p}\] (6) \[h_{t} =\mathrm{LSTM}([\widetilde{f}_{t};\widetilde{a}_{t-1}], \widetilde{h}_{t-1})\] (7) \[\gamma_{t,i} =\mathrm{softmax}_{i}(\tilde{u}_{dep_{i}}^{T}W_{U}h_{t})\] (8) \[\widetilde{u}_{t} =\sum_{i}\gamma_{t,i}\tilde{u}_{dep_{i}}\] (9) \[\widetilde{h}_{t} =\tanh(W_{M}[\widetilde{u}_{t};h_{t}]) \tag{10}\]

where \(\widetilde{a}_{t-1}\) is the previous action embedding, \(\widetilde{f}_{t}\) is the attended panoramic representation, and \(h_{t}\) is the decoder hidden state. \(W_{F}\), \(W_{U}\), \(W_{M}\) are learned weight parameters. We compute the attended language representation over all dependency node representations which are aware of syntax information.

We use a mixture of imitation learning and reinforcement learning to train the agent. Details can be found in Appendix.

## 4 Experimental Setup

### Datasets

We evaluate our agent on Room-to-Room R2R dataset Anderson et al. (2018) and Room-Across-Room RxR) dataset Ku et al. (2020). Both datasets are built on Matterport3D simulator Anderson et al. (2018). The R2R dataset contains 21567 human-annotated instructions with an average instruction length of 29. The dataset is divided into training set, seen validation set, unseen validation set, and test set. The RxR dataset is an extension of the R2R dataset. The instructions are longer (with an average instruction length of 78), and the instructions are in three languages (i.e., English, Hindi, and Telugu). The RxR dataset follows the same division as the R2R dataset. Details can be found in the Appendix.

### Evaluation Metrics

To evaluate the performance of our model, we use the following evaluation metrics: Success Rate (SR), Success Rate Weighted by Path Length (SPL) Anderson et al. (2018), normalized Dynamic Time Warping (nDTW) Magalhaes et al. (2019), success rate weighted by Dynamic Time Warping Magalhaes et al. (2019) and Coverage weighted by Length Score (CLS). Detailed description for each metric can be found in Appendix.

Figure 2: Architecture for our syntax-aware agent.





## 5 Results and Analysis

### Room-to-Room Dataset

We compare our agent with the baseline agent Tan et al. (2019)3 on the R2R test leaderboard. Our syntax-aware agent achieves 47.8% in success rate and 0.45 in SPL, improving the baseline model by 2.1% in success rate and 2% in SPL.

Footnote 3: The baseline is the re-implementation of their model without back translation based on code: [https://github.com/airsplay/R2R-EnvDrop](https://github.com/airsplay/R2R-EnvDrop)

Besides, as shown in Table 1, our syntax-aware agent achieves 3.7% improvement in success rate over the baseline model in validation unseen environment, indicating that explicitly incorporating syntax information can better guide the agent to ground to visual scenes.

We further experiment on R2R dataset to see whether the increase in performance comes from more model parameters. Compared with the model that uses a 2-layer LSTM (+LSTM), we can see that our syntax-aware model still achieves 2.6% increase in success rate in validation unseen environment. This result validates the effectiveness of incorporating syntax information.

### Room-Across-Room Dataset

We first compare our agent with the baseline agent in RxR paper Ku et al. (2020) on the RxR test leaderboard. Our syntax-aware agent ("SAA (mono)" on the leaderboard) outperforms the baseline in all metrics, improving the nDTW score by 5.73% and success rate by 9.98%. Moreover, We compare our agent with the baseline on RxR validation set. As shown in Table 2, in all three languages, our syntax-aware agent achieves higher success rate than the baseline agent in all metrics on validation unseen set. Specifically, our model gets 2.6% improvement in Hindi instructions in terms of success rate. For English and Telugu instructions, our model gets smaller improvement - 0.7% and 0.1% respectively. One reason for these results could be the correlation with the quality of the dependency parser in that language. Besides, compared with the baseline in Ku et al. (2020), our agent achieves the new state-of-the-art on RxR dataset (at the time of submission).

### Qualitative Analysis

As shown in Fig 3 and Fig 4, we illustrate a qualitative example to show that our agent learns better alignment between instruction and visual scenes. As shown in Fig 4, given instruction _"Walk past the shelves and out of the garage. Stop in front of the opening to the wine cellar."_, the baseline agent tends to focus on the same phrases (e.g., "past the shelves", "wine cellar") during navigation. This suggests that the baseline agent is not able to learn a good alignment between the instruction and the current visual scene. However, as shown in Figure 3, our dependency-based agent can successfully identify the correct parts of the instruction that are correlated with the current visual scenes, and picks the next action with fidelity. At the beginning of navigation, our agent focuses on the first sub-instruction "walk past the shelves". Then "out of the garage" gradually becomes the most important phrase, indicating that the agent should go out of the garage after passing the shelves. When

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \multicolumn{2}{c}{**Models**} & \multicolumn{6}{c}{**Val Seen**} & \multicolumn{6}{c}{**Val Unseen**} \\ \cline{2-11}  & SR(\%) & SPL & nDTW & sDTW & CLS & SR(\%) & SPL & nDTW & sDTW & CLS \\ \hline
**EnvDrop** & 58.3 & 0.55 & 0.67 & 0.52 & 0.67 & 45.3 & 0.42 & 0.58 & 0.39 & 0.58 \\
**+LSTM** & 60.3 & 0.57 & 0.69 & 0.55 & 0.69 & 46.4 & 0.43 & 0.58 & 0.40 & 0.58 \\
**+syntax** & **62.6** & **0.60** & **0.70** & **0.56** & **0.70** & **49.0** & **0.45** & **0.59** & **0.42** & **0.59** \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of the model with and without our tree-based encoder on the seen validation set and unseen validation set of R2R dataset.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \multicolumn{2}{c}{**Models**} & \multicolumn{6}{c}{**Val Seen**} & \multicolumn{6}{c}{**Val Unseen**} \\ \hline  & SR(\%) & SPL & nDTW & sDTW & CLS & SR(\%) & SPL & nDTW & sDTW & CLS \\ \hline
**EnvDrop (en)** & \(48.1\) & **0.44** & 0.57 & 0.40 & 0.61 & 38.5 & 0.34 & 0.51 & 0.32 & 0.54 \\
**+syntax (en)** & **48.1** & \(0.44\) & **0.58** & **0.40** & **0.61** & **39.2** & **0.35** & **0.52** & **0.32** & **0.56** \\
**EnvDrop (hi)** & \(49.6\) & \(0.45\) & \(0.57\) & \(0.41\) & \(0.61\) & 39.9 & 0.35 & 0.49 & 0.32 & 0.53 \\
**+syntax (hi)** & **55.2** & **0.52** & **0.61** & **0.46** & **0.64** & **42.5** & **0.38** & **0.54** & **0.35** & **0.58** \\
**EnvDrop (te)** & \(45.8\) & \(0.42\) & \(0.56\) & \(0.38\) & \(0.60\) & 38.3 & 0.34 & 0.50 & 0.31 & 0.54 \\
**+syntax (te)** & **49.1** & **0.46** & **0.59** & **0.41** & **0.63** & **38.4** & **0.35** & **0.52** & **0.32** & **0.56** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of the model with the tree-based encoder and without the tree-based encoder on the seen validation set and unseen validation set of RxR dataset.



the agent sees the wine sculpture, it infers that the opening is for wine cellar and stops in front of the opening near the wine sculpture.

### Implementation Variants

Since the goal of our paper is to explore the role of syntax information in vision-language navigation, we try several implementation variants to include syntax information. First, we try to use mean-pooling instead of Tree-LSTM to encode the dependency tree structure. This implementation variant decreases the performance by around 3% in terms of success rate on validation unseen environments. Besides, we explore whether syntax information from a constituency tree can also help with the instruction following and navigation. Similar to how we incorporate dependency tree information, we use a Tree-LSTM to encode the constituency tree information. However, the performance decreases around 2% in terms of success rate in validation unseen environments, indicating that syntax information extracted from dependency tree is more beneficial for the vision-and-language navigation task.

## 6 Conclusion

In this paper, we presented a tree-based encoder module that incorporates phrase-level information from parse trees. We demonstrated that syntax information can help the agent learn a better alignment between instruction and visual scenes, and generalize better to unseen environments. Our experiments on Room-to-Room dataset and Roomm Across-Room dataset both suggest that incorporating syntax information encoded by our tree-based encoder module can significantly improve the performance over baseline VLN models.

## 7 Ethical Considerations

Vision-Language Navigation is the task that requires an agent to navigate through a 3D environment based on given natural language instructions. An agent that can interact with the environment based on instructions can be used in many real-world applications, for example, a home service robot can bring things to the owner based on instruction, making people's life easier. However,

Figure 4: The weights for the grounded instructions for baseline model.

Figure 3: The weights for the grounded instructions for our syntax-aware model.

 

when deployed in the real world, even if the agent can navigate successfully based on the instruction, it might still need further human assistance to keep working successfully (e.g., a home-cleaning robot might be stuck in the corner of the room and cannot get out by itself).

The performance on the validation set with the unseen environment is much lower than the seen environment. Change in environment will significantly influence the performance of the agent. When the agent is deployed in an unseen environment, it will have a higher probability of failure, wasting energy and time. A further pre-exploration of the environment will be needed for better performance of the agent when deployed to real-world applications. Moreover, our agent relies on the quality of the dependency parser to some extent. Though we achieve improvement in all three languages when using the dependency tree information, the agent in Hindi and English benefit most from the syntax information because of the best available parser for these languages.

## 8 Acknowledgement

We thank the reviewers for their helpful discussions. This work was supported by ARO-YIP Award W911NF-18-1-0336, DARPA MCS Grant N66001-19-2-4031, a Google Focused Research Award, and a Bloomberg Data Science Ph.D. Fellowship. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.

## References

* Anderson et al. (2018a) Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. 2018a. On evaluation of embodied navigation agents. _arXiv preprint arXiv:1807.06757_.
* Anderson et al. (2018b) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 3674-3683.
* Chen et al. (2019) Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 12538-12547.
* Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1657-1668.
* Choi et al. (2017) Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2017. Learning to compose task-specific tree structures. _arXiv preprint arXiv:1707.02786_.
* Eriguchi et al. (2016) Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 823-833.
* Fried et al. (2018) Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In _Advances in Neural Information Processing Systems_, pages 3314-3325.
* Geng et al. (2020) ZhiQiang Geng, GuoFei Chen, YongMing Han, Gang Lu, and Fang Li. 2020. Semantic relation extraction using sequential and tree-structured lstm with attention. _Information Sciences_, 509:183-192.
* Hao et al. (2020) Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pretraining. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 13137-13146.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Hinton et al. (2012) Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. _Cited on_, 14(8).
* Hong et al. (2020) Yicong Hong, Cristian Rodriguez, Qi Wu, and Stephen Gould. 2020. Sub-instruction aware vision-and-language navigation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3360-3376.
* Hu et al. (2019) Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. 2019. Are you looking? grounding to multiple modalities in vision-and-language navigation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6551-6557 

* Huang et al. (2019a) Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, and E. le. 2019a. Multi-modal discriminative model for vision-and-language navigation. _ArXiv_, abs/1905.13588.
* Huang et al. (2019b) Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, and Eugene Ie. 2019b. Transferable representation learning in vision-and-language navigation. In _Proceedings of the IEEE International Conference on Computer Vision_, pages 7404-7413.
* Jain et al. (2019) Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in vision-and-language navigation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1862-1872.
* Ke et al. (2019) Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. 2019. Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 6741-6749.
* Ku et al. (2020) Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. _arXiv preprint arXiv:2010.07954_.
* Landi et al. (2019) Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, and Rita Cucchiara. 2019. Perceive, transform, and act: Multi-modal attention networks for vision-and-language navigation. _arXiv preprint arXiv:1911.12377_.
* Li et al. (2019) Xiujun Li, C. Li, Qiaolin Xia, Yonatan Bisk, A. Celikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. 2019. Robust navigation with language pretraining and stochastic sampling. In _EMNLP/IJCNLP_.
* Ma et al. (2019a) Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019a. Self-monitoring navigation agent via auxiliary progress estimation. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Ma et al. (2019b) Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. 2019b. The regretful agent: Heuristic-aided navigation through progress estimation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 6732-6740.
* Magalhaes et al. (2019) Gabriel Magalhaes, Vianh Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. Effective and general evaluation for instruction conditioned navigation using dynamic time warping. _arXiv preprint arXiv:1907.05446_.
* Majumdar et al. (2020) Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. 2020. Improving vision-and-language navigation with image-text pairs from the web. _arXiv preprint arXiv:2004.14973_.
* Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stantford cernap natural language processing toolkit. In _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_, pages 55-60.
* Misra et al. (2018) Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environments with visual goal prediction. In _EMNLP_.
* Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using LSTMs on sequences and tree structures. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1105-1116.
* Mnih et al. (2016) Volodymyr Mnih, Adria Puigdonnech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In _International conference on machine learning_, pages 1928-1937.
* Nguyen and Daume III (2019) Khanh Nguyen and Hal Daume III. 2019. Help, annai! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 684-695.
* Qi et al. (2020a) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020a. Stanza: A python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 101-108, Online. Association for Computational Linguistics.
* Qi et al. (2020b) Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, and Qi Wu. 2020b. Object-and-action aware model for visual language navigation. _arXiv preprint arXiv:2007.14626_.
* Rao et al. (2019)Jinfeng Rao, Kartikeya Upasani, Anusha Balakrishnan, Michael White, Anuj Kumar, and Rajen Subba. 2019. A tree-to-sequence model for neural nlg in task-oriented dialog. In _Proceedings of the 12th International Conference on Natural Language Generation_, pages 95-100.
* Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge.

 

International Journal of Computer Vision (IJCV)_, 115(3):211-252.
* Su et al. (2020) Chao Su, Heyan Huang, Shumin Shi, Ping Jian, and Xuewen Shi. 2020. Neural machine translation with gumbel tree-lstm based encoder. _Journal of Visual Communication and Image Representation_, page 102811.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_, pages 1556-1566.
* Tan et al. (2019) Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2610-2621.
* Thomason et al. (2020) Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In _Conference on Robot Learning_, pages 394-406.
* Wang et al. (2020a) Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. 2020a. Active visual information gathering for vision-language navigation. _arXiv preprint arXiv:2007.08037_.
* Wang et al. (2020b) Hu Wang, Qi Wu, and Chunhua Shen. 2020b. Soft expert reward learning for vision-and-language navigation. _arXiv preprint arXiv:2007.10835_.
* Wang et al. (2019) Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 6629-6638.
* Wang et al. (2018) Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. 2018. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 37-53.
* Xia et al. (2020) Qiaolin Xia, Xiujun Li, Chunyuan Li, Yonatan Bisk, Zhifang Sui, Jianfeng Gao, Yejin Choi, and Noah A Smith. 2020. Multi-view learning for vision-and-language navigation. _arXiv preprint arXiv:2003.00857_.
* Xiang et al. (2020) Jiannan Xiang, Xin Eric Wang, and William Yang Wang. 2020. Learning to stop: A simple yet effective approach to urban vision-language navigation. _arXiv preprint arXiv:2009.13112_.
* Zhu et al. (2020a) Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020a. Vision-language navigation with self-supervised auxiliary reasoning tasks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 10012-10022.
* Zhu et al. (2020b) Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, and Fei Sha. 2020b. Babywalk: Going farther in vision-and-language navigation by taking baby steps. In _ACL_, pages 2539-2556.
* Zhu et al. (2020c) Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, and Xiaodan Liang. 2020c. Vision-dialog navigation by exploring cross-modal memory. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.

## Appendix A Appendix

### Problem Setup

Vision-Language navigation requires an agent to navigate through a 3D environment to a target location based on a given natural language instruction. Formally, the natural language instruction is a sequence of words \(\{w_{i}\}^{l}_{i=1}\), where \(l\) is the length of the instruction and \(w_{i}\) is \(ith\) word in the sequence. At each time step, the agent perceives a panoramic view of the current viewpoint. Besides, the agent has access to a set of navigable locations \(\{l_{t,k}\}^{K}_{k=1}\), where \(K\) is the total number of reachable locations from the current viewpoint. The agent needs to select an action \(a_{t}\) from the list of navigable viewpoints \(\{l_{t,k}\}^{K}_{k=1}\) based on the given instruction, navigation history and current panoramic views. If the viewpoint selected from the list is the same as the current viewpoint, the agent predicts a "STOP" action.

### Model Details

**Visual Encoder.** Same as previous work in VLN, at time step \(t\), we discretize the panoramic view into 36 single views \(\{o_{t,p}\}^{36}_{p=1}\). Each single view is a RGB image, annotated with its angles of heading and elevation \((\theta_{t,p},\phi_{t,p})\). Each RGB image is encoded with a pre-trained ResNet-152 He et al. (2016) on ImageNet Russakovsky et al. (2015). A four dimension orientation feature \((cos\theta_{t,p},sin\theta_{t,p},cos\phi_{t,p},sin\phi_{t,p})\) is concatenated with the ResNet feature to form the final representation for the view \(p\) of a panoramic \(\{f_{t,p}\}^{36}_{p=1}\). Similarly, we get the visual representation \(\{g_{t,k}\}^{K}_{k=1}\) for \(K\) navigable locations \(\{l_{t,k}\}^{K}_{k=1}\) at time step \(t\) 



### Training

We use a mixture of imitation learning and reinforcement learning to train the agent.

**Imitation Learning.** During training, instead of navigating to the predicted action at each time step, teacher-forcing is used to determine which navigable viewpoint to pick. Given the shortest path between the start point and target point, at each time step \(t\), the agent tries to imitate the teacher action \(a_{t}^{*}\) by minimizing the negative log probability:

\[L_{IL}=\sum_{t}-a_{t}^{*}logp_{t} \tag{11}\]

**Reinforcement Learning.** We combine imitation learning with reinforcement learning to learn a more generalizable agent. Since the teacher path is the shortest path between the start point and the target point, there is no guarantee that the teacher path is the same as indicated by the given instruction. Thus, reinforcement learning is applied for better instruction following and state exploration. At each time step \(t\), the agent samples an action \(a_{t}\) from the predicted distribution \(p_{t}(a_{t})\). At each time step, if the agent moves closer to the target viewpoint, a positive reward +1 is given, otherwise the agent receives a negative reward -1. When the agent predicts the "STOP" action, the agent will receive a +3/-3 reward based on whether the agent is within 3m from the target viewpoint. We use Actor-Critic Mnih et al. (2016) to train the agent. The loss of reinforcement learning is:

\[L_{RL}=\sum_{t}{(R_{t}-R_{b_{t}})logp_{t}(a_{t})}+\eta H(p_{t}(a_{ t})) \tag{12}\]

where \(R_{t}\) is the discounted future cumulative rewards at time step \(t\), \(R_{b_{t}}\) is the expected cumulative rewards (baseline) approximated by the value function \(V\), \(H(p_{t}(a_{t}))\) is the entropy term for regularization. Specifically,

\[R_{t} =r_{t}+\sum_{i=1}^{T-t}\gamma^{i}r_{t+i} \tag{13}\] \[R_{b_{t}} =V(h_{t})=W_{v_{2}}\sigma(W_{v_{1}}h_{t}) \tag{14}\]

where \(\sigma\) is the ReLU activation function, and \(r_{t}\) is the immediate reward we defined earlier. The value function is trained with L2 loss:

\[L_{V}=\frac{1}{2}(R_{t}-R_{b_{t}})^{2} \tag{15}\]

We optimize a mixture loss of imitation learning and reinforcement learning:

\[L_{MIX}=(L_{RL}+L_{V})+\lambda L_{IL} \tag{16}\]

### Dataset

**R2R Dataset.** The R2R dataset contains 21567 human annotated instructions with an average instruction lengths of 29. The training set contains 14025 instructions in 61 environments. The seen validation set contains 1020 instructions in the same 61 environments as the training set. The unseen validation set contains 2349 instructions in 11 environment which is not included in the training set. The test set contains 4173 instructions in 18 environments.

**RxR Dataset.** The RxR dataset is an extension to the R2R dataset, where the instructions are longer and in languages other than English Hindi and Telugu). It contains 126069 instructions with an average instruction length of 78. Besides, different from R2R that only contains guide path (i.e., the shortest path between the start point and the target point), RxR pairs each guide path with a human-annotated follower path (i.e., the path that human generates following the instruction). We only use the guide path to train the agent in this paper. The seen and unseen environment split is the same as in the R2R dataset. There are 16522 paths in total, and each path is annotated in 3 languages. The training set contains 11089 paths, the seen validation set contains 1232 paths, the unseen validation contains 1517 paths, and the test set contains 2684 paths.

### Evaluation Metrics

To evaluate the performance of our model, we use the following evaluation metrics: (1) Success Rate (SR): If the agent stops less than 3m from the target location, we consider the navigation as a success. (2) Success Rate Weighted by Path Length SPL Anderson et al. (2018): This metric penalizes long trajectories (e.g., find the target using beam search over the environment graph). (3) normalized Dynamic Time Warping nDTW Magalhaes et al. (2019): This metric penalizes deviations from the reference path. (4) success rate weighted by Dynamic Time Warping sDTW Magalhaes et al. (2019): This metric constraints nDTW only to successful navigation and considers path fidelity and agent success. (5) Coverage weighted by Length

Score (CLS) Jain et al. (2019): Similar to nDTW, this metric also encourages path fidelity.

### Implementation Details

We generate the dependency parse tree for English using Stanford CoreNLP Manning et al. (2014). We use Stanza Toolkit Qi et al. (2020) to generate the dependency parse tree for Hindi and Telugu. For both baseline model and our syntax-aware model, the learned word embedding size is 256 and the dimension of the action embedding is 128. We set the hidden size for the bi-directional LSTM to be 256. For syntax-aware model, the hidden size for the bi-LSTM and Tree-LSTM in the language encoder is 512. We use ResNet-152 He et al. (2016) to extract the 2048 dimensional image feature. We use RMSProp Hinton et al. (2012) as the optimizer with learning rate 1e-4 and batch size 64. The weight \(\lambda\) we use to combine imitation learning loss and reinforcement learning loss is set to be 0.2. In reinforcement learning, the discount factor \(\gamma\) is 0.9 and the entropy weight \(\eta\) is 0.01. During training, we set the max action length to be 35 for R2R dataset and 70 for RxR dataset. We train both agents on R2R for 80,000 iterations. We train both agents on RxR for 200,000 iterations. The baseline model contains approximate 6 million parameters. Our syntax-aware model contain approximate 8 million parameters.