SlowGuess/NAACL_2022_OCR huggingface dataset:
Gabriel Ilharco Magalhaes, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. General evaluation for instruction conditioned navigation using dynamic time warping. In NeurIPS Visually Grounded Interaction and Language (ViGIL) Workshop.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. 2020. Improving vision-and-language navigation with imagetext pairs from the web. In ECCV.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937.
Khanh Nguyen and Hal Daumé III. 2019. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. In EMNLP-IJCNLP, pages 684-695.
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. 2020. Massively multilingual asr: 50 languages, 1 model, 1 billion parameters. arXiv preprint arXiv:2007.03001.
Yuankai Qi, Zizheng Pan, S. Zhang, A. V. D. Hengel, and Qi Wu. 2020a. Object-and-action aware model for visual language navigation. In ECCV.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020b. Reverie: Remote embodied visual referring expression in real indoor environments. In CVPR, pages 9982-9991.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. Image, 2:T2.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2022. How much can clip benefit vision-and-language tasks? ICLR.
Nobuyuki Shimizu, Na Rong, and Takashi Miyazaki. 2018. Visual question answering dataset for bilingual image understanding: A study of cross-lingual transfer using attention maps. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1918-1928.
Gunnar A Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh, Lucas Smaira, Mateusz Malinowski, João Carreira, Phil Blunsom, and Andrew Zisserman. 2020. Visual grounding in video for unsupervised word translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10850-10859.
Didac Surís, Dave Epstein, and Carl Vondrick. 2022. Globetrotter: Unsupervised multilingual translation from visual alignment. CVPR.
Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout. In NAACL, pages 2610-2621.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2019. Vision-and-dialog navigation. In arXiv.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In Conference on Robot Learning, pages 394-406. PMLR.
Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen. 2020a. Active visual information gathering for vision-language navigation. In ECCV.
Hu Wang, Qi Wu, and Chunhua Shen. 2020b. Soft expert reward learning for vision-and-language navigation. ECCV.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019a. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6629-6638.
Xin Wang, Vihan Jain, Eugene Ie, William Yang Wang, Zornitsa Kozareva, and Sujith Ravi. 2020c. Environment-agnostic multitask learning for natural language grounded navigation. ECCV.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019b. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581-4591.
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Wang. 2018. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In ECCV.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. 2019. Detector2. https://github.com/facebookresearch/detectron2.
Qiaolin Xia, Xiujun Li, C. Li, Yonatan Bisk, Zhifang Sui, Yejin Choi, and N. A. Smith. 2020. Multi-view learning for vision-and-language navigation. ArXiv, abs/2003.00857.
Yubo Zhang, Hao Tan, and Mohit Bansal. 2020. Diagnosing the environment bias in vision-and-language navigation. In *IJCAI* 2020, pages 890-897. ijcai.org.
Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. 2021. Soon: Scenario oriented object navigation with graph-based exploration. In CVPR, pages 12689-12699.
Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, and Fei Sha. 2020. Babywalk: Going farther in vision-and-language navigation by taking baby steps. In ACL, pages 2539-2556.
In this supplementary, we provide a detailed description of our navigation model structure (Sec. B), representation learning and navigation learning objective (Sec. C), dataset (Sec. D), evaluation metrics (Sec. E), implementation details (Sec. F), and additional analysis in the last four sections. In this analysis, we first show that using object-matching as constraints during visual representation learning improves the nDTW score (Sec. G). Then we show that our CLEAR approach decreases the performance variance among different environments (Sec. H) and learn better alignment between the instruction and the environment (Sec. J). We further analyze whether the word representation from our learned cross-lingual representation also learn the visual/spatial information (Sec. I). Moreover, we investigate the effect of filtering out low-quality paths (Sec. K). Lastly, we show the high correspondence between instruction similarity and path pair alignment in Sec. L.
B Navigation Model
Our navigation agent follows the decoder structure as Tan et al. (2019).
At each time step  $t$ , the agent perceives a panoramic view of the current location. The panoramic view is discretized into 36 single views  $\{o_{t,m}\}_{m=1}^{36}$  (12 angles and 3 camera poses per angle). Given the visual representation for each view  $\widehat{v}_{t,m}$ , we concatenate it with the orientation feature to get the view features  $\{f_{t,m}\}_{m=1}^{36}$ :
$$
f _ {t, m} = \left[ \widehat {v} _ {t, m}; \left(\cos \theta_ {t, m}, \sin \theta_ {t, m}, \cos \phi_ {t, m}, \sin \phi_ {t, m}\right) \right] \tag {11}
$$
where  $\theta_{t,m}$  and  $\phi_{t,m}$  the heading and elevation of view  $o_{t,m}$ .
As a reaction to the input, the agent needs to select one of the  $K$  navigable locations as an action. The action is represented as the orientation features (heading and elevation) between the current viewpoint and the chosen navigable viewpoint. The navigation decoder takes the attended visual feature  $\widehat{f}_t$  of the current viewpoint and the previous action embedding  $a_{t-1}$  as input, and updates its environment-aware context vector  $h_t$ :
$$
\gamma_ {t, m} = \operatorname {S o f t m a x} _ {m} \left(f _ {t, m} ^ {T} W _ {f} \widehat {h} _ {t - 1}\right) \tag {12}
$$
$$
\widehat {f} _ {t} = \sum_ {m} \gamma_ {t, m} f _ {t, m} \tag {13}
$$
$$
h _ {t} = \operatorname {L S T M} \left(\left[ \widehat {f _ {t}}; a _ {t - 1} \right], \widehat {h} _ {t - 1}\right) \tag {14}
$$
where  $a_{t - 1}$  is represented as the orientation features  $(\cos \theta_{t - 1,k^{\star}},\sin \theta_{t - 1,k^{\star}},\cos \phi_{t - 1,k^{\star}},\sin \phi_{t - 1,k^{\star}})$  of the chosen navigable viewpoint  $k^{\star}$  at time step  $t - 1$  and  $\widehat{h}_{t - 1}$  is the instruction-aware context vector that incorporates the attended instruction information. The navigator calculates the probability of moving to the  $k$  -th navigable location based on the alignment between the visual feature  $g_{t,k}$  of that navigable location and the instruction-aware context vector  $\widehat{h}_t$  ..
$$
\rho_ {t, j} = \operatorname {S o f t m a x} _ {j} \left(\widehat {w} _ {j} ^ {T} W _ {l} h _ {t}\right) \tag {15}
$$
$$
u _ {t} = \sum_ {j} \rho_ {t, j} \widehat {w} _ {j} \tag {16}
$$
$$
\widehat {h} _ {t} = \tanh  \left(W _ {m} \left[ u _ {t}; h _ {t} \right]\right) \tag {17}
$$
$$
p \left(a _ {t} = k\right) = \operatorname {S o f t m a x} _ {k} \left(g _ {t, k} ^ {T} W _ {a} \widehat {h} _ {t}\right) \tag {18}
$$
where  $g_{t,k}$  is constructed similarly as  $f_{t,i}$  in Eqn. 11, and  $\widehat{w}_j$  is the language representation.
In the representation learning stage, given a pair of instructions that correspond to the same navigation path, we train the shared multilingual encoder to generate representations of paired instructions close to each other by optimizing a contrastive loss  $L_{lang}$ . Furthermore, we train the visual encoder to learn the connections between paths with similar instructions by optimizing the contrastive loss  $L_{visual}$ . The representation learning process transfers the language representation to domain-specific
language representation and adapts the visual representation to learn the correlation underlying the navigation environments.
In the navigation learning stage, we use a mixture of imitation learning and reinforcement learning to train the agent on the navigation task as in Tan et al. (2019).
In imitation learning, we use teacher-forcing to determine the next navigable viewpoint. Different from previous methods (Hong et al., 2021; Tan et al., 2019; Huang et al., 2019b) that takes the shortest path as the teacher action, our teacher action  $a_{t}^{\star}$  at each time step  $t$  is picked based on the given ground truth path between the start point and target point. The agent tries to imitate the teacher action  $a_{t}^{\star}$  by minimizing the negative log probability:
$$
L _ {I L} = \sum_ {t} - a _ {t} ^ {\star} \log p _ {t} \tag {19}
$$
We combine reinforcement learning with imitation learning to learn a more generalizable agent. At each time step  $t$ , the agent samples an action  $a_{t}$  from the predicted distribution  $p_{t}(a_{t})$ . We follow (Hong et al., 2021) to do the reward shaping. The immediate reward at each time step  $t$  consists of three parts. First, if the agent moves closer to the target viewpoint, a positive reward  $+1$  is given, otherwise the agent receives a negative reward -1. Second, to encourage instruction following, we include normalized Dynamic Time Warping (nDTW) score in the reward. The agent gets a positive reward if the nDTW score for the navigated path increases. Lastly, the agent receives a negative reward if it misses the target. When the agent predicts the "STOP" action, the agent will receive a  $+3/ -3$  reward based on whether the agent is within 3 meters from the target viewpoint. We use Advantage Actor-Critic (Mnih et al., 2016) to train the agent.
The navigation loss  $L_{\text{nav}}$  is a weighted combination of imitation learning loss and reinforcement learning loss.
$$
L _ {\text {n a v}} = L _ {R L} + \lambda L _ {I L} \tag {20}
$$
We evaluate our agent on the Room-Across-Room (RxR) dataset (Ku et al., 2020). The dataset is built on the Matterport3D simulator (Anderson et al., 2018b). It contains 126,069 human-annotated instructions with an average instruction length of 78.
Methods	SR↑	SPL↑	NDTW↑	SDTW↑
+visual	35.6	32.5	53.7	30.5
-sample	37.8	33.7	53.0	32.1
-object	36.6	33.0	52.4	30.9
Table 7: Comparison between visual representation trained with objects constraints ('+visual'), without sampling strategy ('-sample') and without object constraints ('-object') on validation unseen sets. nDTW is the main metric for Room-Across-Room (RxR) dataset.
The dataset is split into training set, seen validation set, unseen validation set, and test set. In the unseen validation set and test set, the environments do not appear in the training set. Thus the performance on these two sets show the model's generalizability to new environments. There are 16,522 paths in total, and each path is annotated in 3 languages (and 3 instructions per language on average). The training set contains 11,089 paths, the seen validation set contains 1,232 paths, the unseen validation contains 1,517 paths, and the test set contains 2,684 paths.
To evaluate the performance of our model, we follow the metrics used in the Room-Across-Room paper (Ku et al., 2020). The metrics include: (1) Success Rate (SR): We consider a success for navigation if the agent stops less than  $3\mathrm{m}$  from the target location. (2) Success rate weighted by Path Length (SPL) (Anderson et al., 2018a): This metric penalizes the navigation with long paths (i.e., when both navigations reach the target, the navigation with shorter path length has a higher SPL score). (3) normalized Dynamic Time Warping (nDTW) (Magalhaes et al., 2019): This metric measures the path fidelity by penalizing deviations from the reference path. The agent navigates to the target through the shortest path instead of instruction following will be penalized. (4) success rate weighted by Dynamic Time Warping (sDTW) (Magalhaes et al., 2019): This metric only considers nDTW of successful navigation and ignores failed navigation. Normalized Dynamic Time Warping (nDTW) is the main metrics for RxR and Success Rate (SR) and Success rate weighted by Path Length (SPL) are the main metrics for R2R.
In our experiments, we learn the shared multilingual representation based on cased multilingual  $\mathrm{BERT}_{\mathrm{BASE}}$ . The instruction is truncated from the end with a maximum sequence length of 160. For the pre-trained vision model, we compare performance between image features extracted from ImageNet-pre-trained (Russakovsky et al., 2015) ResNet-152 (He et al., 2016) and CLIP-pre-trained (Radford et al., 2021) vision transformer (ViT-B/32) (Dosovitskiy et al., 2021) (abbreviated as 'CLIP feature' later). The 27 object classes are: 'drawing', 'faucet', 'cabinet', 'hinge', 'cushion', 'sofa', 'chair', 'pillow', 'armchair', 'lamp', 'vase', 'knob', 'curtain', 'statue(sculpture)', 'doorknob', 'vent', 'lightbulb', 'flowerpot', 'book', 'pipe', 'painting', 'wall socket', 'bed', 'mirror', 'television set', 'flower arrangement', 'chandelier'. The navigation decoder's hidden size is 768 and the action embedding size is 128. The language encoder is optimized with AdamW (Loshchilov and Hutter, 2019) with linear-decayed learning rate. The peak learning rate is 4e-5 for both the representation learning and the navigation agent learning stage. The visual encoder, the navigation decoder, and the discriminator are optimized with RMSProp (Hinton et al., 2012) with learning rate 1e-4. The weight  $\lambda$  we use to combine loss is set to be 0.4 for the ResNet-based full model and 0.2 for the CLIP-based full model. The batch size for training ResNet features and CLIP features are 12 and 16, respectively. During training, CLIP model is around 1.5 times faster than ResNet model in this setting since CLIP features are 512 dimensions while ResNet features are 2048 dimensions. To keep roughly the same amount of training time, we train the agent with ResNet features for 100K iterations, while we train model CLIP-ViT features for 150K iterations.
G Analysis: Effectiveness of Object-Matching Constraints
Our visual representation learning optimizes the similarity between panoramic views at each step of the semantically-aligned path pairs. Since paths are not fully-aligned, we use object-matching as a constraints to filter out panoramic view pairs that don't contain same objects. As shown in Table 7, the visual representation trained with fixed object classes as constraints ('-sample') improve the nDTW score (the main metric for RxR dataset) by  $0.6\%$  com
pared with the visual representation trained without object-matching constraints ('-object'), suggesting that using object-matching as constraints help learn a better visual representation. Besides, the sampling strategy (i.e., randomly sample 10 object classes from 27 object classes during each iteration) also helps the visual representation learning ('+visual'), further improving the nDTW score by  $0.7\%$  compared with the visual representation learned with fixed 27 object classes ('-sample'). In total, our object-matching constraints and sampling strategy ('+visual') improves the performance by  $1.3\%$  in nDTW score compared with learning without object constraints ('-object').
H Analysis: Performance Variance Reduction among Different Environments
We demonstrate that our CLEAR approach could decrease the performance variance (i.e., performance's standard deviation) among different environments. Intuitively, we hope the agent to perform equally well in different environments instead of getting high performance by only learning to navigate through several easy environments. We show the results for 11 environments in validation unseen set in Table 8. Our CLEAR approach ('+both' as in Table 2 in the main paper) outperforms the baseline model ('ResNet' as in Table 2 in the main paper) in most of the environments. Moreover, the weighted standard deviation (weighted by # Data in Table 8) of our CLEAR approach is lower than the baseline model. Specifically, the standard deviation of nDTW score for our CLEAR approach is 9.24 while the standard deviation of nDTW score for the baseline model is 10.01, suggesting that our CLEAR approach decreases the performance variance between different environments.
I Analysis: Word Representation from Cross-Lingual Representation
The visual semantics are injected during learning the cross-lingual language representation by maximizing the similarity between full instruction sentences (representation of 'CLS' token). However, it's unclear that whether the word-level representation also learned such visual information. In this section, we investigate whether the learning encodes spatially close words/objects closer to each other. As shown in Table 9, we check the top-5 close words to 'kitchen', and 'fire' from a vocabu
Environment	# Data	ResNet	CLEAR
SR	SPL	NDTW	SDTW	SR	SPL	NDTW	SDTW
1206	32.4	26.7	49.8	26.5	35.9	30.2	50.0	28.0
2177	27.0	23.8	41.3	22.3	28.6	26.0	47.5	24.4
567	38.1	34.3	56.9	31.8	48.0	44.8	64.4	40.4
1692	38.3	34.8	56.1	33.2	39.5	36.0	57.6	33.4
153	57.5	54.7	72.1	53.1	64.1	60.0	74.5	57.3
1404	42.7	38.7	58.3	37.8	41.3	39.1	61.1	36.6
900	52.0	49.5	67.9	46.3	45.7	44.2	65.9	41.0
2223	40.8	36.9	57.7	35.5	44.0	39.2	60.3	38.2
18	38.9	32.7	59.4	34.0	50.0	45.1	70.8	47.1
1152	42.4	38.6	54.6	36.1	38.1	35.2	55.1	33.3
2160	18.1	16.4	34.6	14.7	15.9	13.1	37.1	13.1
Table 8: The results of our CLEAR method and ResNet baseline on different environments in validation unseen set. # Data means the number of instruction-path pairs for each environment.
Figure 4: The attention weights for the grounded instruction for our CLEAR model, ResNet based baseline model, and ground truth from RxR dataset.
lary of 2754 English tokens. We see that our cross-lingual representation puts words that appear spatially near each other close (e.g. 'kitchen' and 'island'/'dinning', 'fire' and 'chair'/'fireplace') while m-BERT representation fails (e.g. 'kitchen' and 'room'/'house', 'fire' and 'family'/'study').
J Analysis: Alignment between Instructions and Environments
The Room-Across-Room dataset provides groundtruth alignment between instructions and navigation paths. To demonstrate that our CLEAR approach learns a good alignment between instructions and paths, we not only compare our CLEAR approach with the baseline approach, but also compare it with the ground truth alignment provided in the RxR dataset. The attention weights for grounded instruction for CLEAR, Baseline, and
Ground Truth are shown in Figure 4. We observe that our CLEAR model successfully attends to sub-instructions "turn right", "move towards the open door to your right and exit the room through the door", "slightly turn left", "move towards and stand in front of the sofa" sequentially. Although the baseline model also successfully executes the first two sub-instructions "turn right" and "move towards the open door", yet the baseline agent gets lost in the later navigation. Furthermore, the alignment learned by our CLEAR approach matches better with the ground truth alignment provided in the RxR dataset.
K Analysis: Filtering out Low Quality Path Pairs
We investigate whether filtering out low-quality path pairs during visual representation learning
Word	Top-5
kitchen	‘island’, ‘counter’, ‘maker’, ‘din’, ‘#id ing’
‘living’, ‘counter’, ‘room’, ‘table’, ‘house’
fire	‘##place’, ‘over’, ‘place’, ‘chair’, ‘#fas’
‘display’, ‘study’, ‘family’, ‘living’, ‘coffee’
Table 9: Top-5 closest tokens for 'Kitchen' and 'fire'. Top-row: tokens picked by our cross-lingual representation. Bottom-row: tokens picked by multi-lingual BERT baseline.
Similarity	SR↑	SPL↑	NDTW↑	SDTW↑
0.00	35.6	32.5	53.7	30.5
0.90	36.2	32.2	51.9	30.7
0.95	38.6	34.3	53.3	33.0
0.98	38.6	34.3	52.9	32.9
0.99	37.8	33.5	52.6	32.0
1.00	30.9	28.0	49.7	26.1
Table 10: Performance in validation unseen environment when filtering out different percentages of data in training our visual representation. 0.90 means filter out data with similarity score less than 0.90.
could further improve the performance. Since our identified path pairs are retrieved based on the similarity between instructions, we hypothesize that the path pair is aligned better if having a higher instruction similarity score. Thus, we experiment with filtering out instruction pairs that have a cosine similarity score less than 0.90, 0.95, 0.98, and 0.99, and then train the visual representation with filtered data and object-matching constraints. The proportion of filtered-out data is  $1\%$ ,  $6\%$ ,  $28\%$  and  $58\%$  respectively. We also experiment with filtering out  $0\%$  and  $100\%$ . Filtering out  $0\%$  of the data is the same to our proposed environment-agnostic visual representation ('+visual' in Table 2) and filtering out  $100\%$  of the data is analogous to randomly initialize the visual encoder<sup>3</sup>. We then train our environment-agnostic representation (in Sec. 3.2) based on the remaining data and show its performance on the validation unseen environments. As shown in Table 10, though the success rate improves when filtering out some path pairs
with lower quality, not filtering out any path pairs achieve the highest nDTW score. This demonstrates that using object-matching constraints without filtering out path pairs with low instruction similarity is enough for learning a good visual representation. Furthermore, we see a significant performance drop when not fine-tuning the visual representation on any data, which indicates that training the visual encoder with semantically-aligned path pairs is important for agent performance.
L Analysis: Correspondence between Instruction Similarity and Path Pair Alignment
In this section, we show that instruction pairs that have high similarity have similar BLEU score and ROUGE score to the instruction pairs that corresponding to the same path. Specifically, the BLEU-1 and ROUGE-L score for instruction pairs that have high similarity are 0.42 and 0.320, and the BLEU-1 and ROUGE-L score for the instruction pairs that corresponding to the same path are 0.41 and 0.323. Randomly picking gets 0.37 BLEU-1 score and 0.295 ROUGE-L score. These results indicate that high similarity instruction pairs may be of competitive quality as the instruction pairs that corresponding to the same path, and can be used to pick the semantically-aligned path pairs.
Note that filtering out  $100\%$  of the data is not the same as the baseline model ('ResNet' in Table 2). The baseline model does not have the visual encoder we introduced in Sec. 3.2