SlowGuess/NAACL_2022_OCR huggingface dataset:
Jialu Li Hao Tan Mohit Bansal
{jialuli,airsplay，mbansal}@cs.unc.edu
Vision-and-Language Navigation (VLN) tasks require an agent to navigate through the environment based on language instructions. In this paper, we aim to solve two key challenges in this task: utilizing multilingual instructions for improved instruction-path grounding and navigating through new environments that are unseen during training. To address these challenges, first, our agent learns a shared and visually-aligned cross-lingual language representation for the three languages (English, Hindi and Telugu) in the Room-Across-Room dataset. Our language representation learning is guided by text pairs that are aligned by visual information. Second, our agent learns an environment-agnostic visual representation by maximizing the similarity between semantically-aligned image pairs (with constraints on object-matching) from different environments. Our environmentagnostic visual representation can mitigate the environment bias induced by low-level visual information. Empirically, on the Room-Across-Room dataset, we show that our multi-lingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation. Furthermore, we show that our learned language and visual representations can be successfully transferred to the Room-to-Room and Cooperative Vision-and-Dialogue Navigation task, and present detailed qualitative and quantitative generalization and grounding analysis. $^{1}$
The Vision-and-Language Navigation task requires an agent to navigate through the environment based on language instructions. This task has two unsolved challenges. First, directly introducing pretrained linguistic and visual representations into
English Instruction
You are facing towards the closed door right now, turn back, you can find a washroom in front of you, enter into the washroom, stand in between the bath tub and the wash basin, that would be your end point.
Telugu Instruction
200 200   
200   
200   
200   
200   
200   
200   
200
Hindi Instruction
piotepalotekkradvukarek
Path A
Figure 1: Motivation for cross-lingual and environment-agnostic visual representations: The English instruction, Telugu instruction, Hindi instruction on the left all correspond to the same path - Path A. The words in red correspond to the same visual object "wash basin". Path A and Path B are similar paths (i.e., the instruction for these two paths are semantically similar) in different environments.
Path B
these agents suffers from domain shift (i.e., pretrained linguistic and visual representation might not generalize to VLN task) (Huang et al., 2019b). Learning the instruction representation while also learning how to navigate based on the instruction is even more challenging for a multi-lingual agent, since more language variance is injected via multi-lingual instructions. At the same time, it also poses the important question that whether we can utilize multi-lingual instructions to learn a better cross-lingual representation and improve instruction-path grounding and referencing. Second, previous works (Fried et al., 2018; Wang et al., 2019a; Landi et al., 2021; Wang et al., 2020a; Huang et al., 2019a; Ma et al., 2019a; Majumdar et al., 2020; Qi et al., 2020a) on vision-language navigation have seen that agents tend to perform substantially worse in environments that are unseen during training, indicating the lack of generalizability of the navigation agent. In this paper, we propose to address these two challenges via cross-lingual and environment agnostic representations.
1Code and model are available at https://github. com/jialuli-luka/CLEAR.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 633 - 649
Although some initial progress (Huang et al., 2019b; Majumdar et al., 2020; Hong et al., 2021; Chen et al., 2021) has been made towards introducing pre-trained linguistic representations into vision-language navigation agents, how to understand and utilize paired multilingual instructions to transfer the pre-trained linguistic representation to multilingual navigation agent still remains unexplored. We argue that for a multilingual agent, the linguistic representation can capture more visual concepts from learning the similarity between paired multilingual instructions. As shown in Figure 1, though the three instructions shown here are in different languages and vary in length and level of detail<sup>2</sup>, all of them correspond to the sample path - Path A. Hence, by learning the similarity between these paired instructions, the cross-lingual language representation of the same visual concept mentioned in these paired instructions (e.g., the red words correspond to the same visual object "wash basin") will be close to each other, making it easier for the agent to comprehend. Furthermore, the cross-lingual language representation will benefit from the complementary information from instructions in different languages since they elicit more references to visible entities. For example, in Figure 1, the target room environment "washroom" is only mentioned in English instructions. Hindi and Telugu instructions could benefit from learning the connection between "washroom" and "wash basin" through learning from the English instruction.
Moreover, many methods have been proposed to encourage agent generalization to unseen environments during training (Tan et al., 2019; Wang et al., 2020c; Fu et al., 2020; Zhang et al., 2020). Zhang et al. (2020) has shown that it is the low-level appearance information that causes the environment bias. To mitigate this bias, previous works only consider one single environment when learning the visual representation for a given path. We instead learn an environment-agnostic visual representation by exploring the connections between multiple environments. For the example shown in Figure 1, Path A and Path B are two semantically aligned paths in different environments. In both cases, the
agent needs to head into the washroom and stop beside the wash basin. Learning the relationship between these paired paths helps the agent comprehend concepts like "bath tub", and not be distracted by the low-level appearance of the objects in unseen environments.
Overall, in this paper, we propose 'CLEAR: Cross-Lingual and Environment-Agnostic Representations' to address the two challenges above. First, we define a visually-aligned instruction pair as two instructions that correspond to the same navigation path. Given the instruction pairs, we transfer the pre-trained multilingual BERT (Devlin et al., 2019) to the Vision-Language Navigation task by encouraging these paired instructions to be embedded close to each other. Second, we identify semantically-aligned path pairs based on the similarity between instructions. Intuitively, if the similarity between the two instructions is high, then their corresponding navigation path will be semantically similar (i.e., mentioning the same objects like "wash basin"). We further filter out image pairs (a pair of paths will contain multiple image pairs) that do not contain the same objects, for higher path pair similarity. Then, we train an environment agnostic visual representation that learns the connection between these semantically-aligned path pairs.
We conduct experiments on the Room-Across-Room (RxR) dataset (Ku et al., 2020), which contains instruction in three languages (English, Hindi, and Telugu). Empirical results show that our proposed representations significantly improves the performance over the mono-lingual model (Shen et al., 2022) by  $2.59\%$  in nDTW score on RxR test leaderboard. We further show that our CLEAR approach outperforms our baseline that utilizes ResNet (He et al., 2016) to extract image features by  $5.3\%$  in success rate and  $4.3\%$  in nDTW score (and it also outperforms a stronger baseline that utilizes the recent CLIP (Radford et al., 2021) method to extract image features). Moreover, our CLEAR approach shows better generalizability when transferred to Room-to-Room (R2R) dataset (Anderson et al., 2018b) and Cooperative Vision-and-Dialogue Navigation dataset (Thomason et al., 2019), and adapted to other SotA VLN Agent (Chen et al., 2021). We also demonstrate the advantage of optimizing similarity between all the three languages in RxR dataset for language representation learning and the effectiveness of the
2We translate Telugu instruction and Hindi instruction into English instruction with Google Translation for reference here (the translated instructions are not used in representation learning or navigation learning). Telugu: Return to the left from where you are standing, enter the door on the opposite side, and go to the side of the wash basin on the left and wait. Hindi: Turn back and go inside the door directly, come to the right side of the sink and stop.
way we generate positive path pairs for visual representation learning. Lastly, we demonstrate that our cross-lingual language representation captures visual semantics underlying the instructions, and our environment-agnostic visual representation generalizes better to the unseen environment with both qualitative and quantitative analysis.
Vision-and-language navigation. Vision-and-Language Navigation (VLN) requires an agent to find the routes to the desired target based on instructions (Jain et al., 2019; Thomason et al., 2020; Nguyen and Daumé III, 2019; Qi et al., 2020b; Chen et al., 2019; Krantz et al., 2020). Specifically, there are two key challenges in VLN: grounding the natural language instruction to visual environments and generalizing to unseen environments. To address the first challenge, one line of research in VLN utilizes carefully designed cross-modal attention modules (Wang et al., 2018, 2019a; Tan et al., 2019; Landi et al., 2021; Xia et al., 2020; Wang et al., 2020b,a; Zhu et al., 2020; Li et al., 2021; Zhu et al., 2021; An et al., 2021; Kim et al., 2021), progress monitor modules (Ma et al., 2019b,a; Ke et al., 2019), and object-action aware modules (Qi et al., 2020a). Another line of research improves vision and language co-grounding by improving vision and language representations with pre-training techniques (Li et al., 2019; Huang et al., 2019b; Hao et al., 2020; Majumdar et al., 2020; Hong et al., 2021). Li et al. (2019) directly adopts pre-trained BERT for encoding instructions, Hao et al. (2020) and Hong et al. (2021) learn from a large amount of image-text-action triplets, Majumdar et al. (2020) learns from large amount of text-image pairs from the web, and Huang et al. (2019b) transfers language and visual representation to in-domain representation with auxiliary tasks. Different from them, we utilize the visually-aligned multilingual instructions to learn a cross-lingual language representation that inherently captures visual semantics underlying the instruction.
Multiple methods have been proposed to encourage generalization to unseen environments during training (Zhang et al., 2020; Tan et al., 2019; Wang et al., 2020c; Fu et al., 2020; Li et al., 2022). Zhang et al. (2020) demonstrates that it is the low-level appearance information that causes the large performance gap between seen and unseen environments. Tan et al. (2019) proposes to use environ-
ment dropout on visual features to create new environments and Fu et al. (2020) utilizes adversarial path sampling to encourage generalization. However, both of these methods rely on a speaker module to generate synthetic training data and can be considered as data augmentation methods, which are complementary to our proposed environment-agnostic visual representation. The closest work to ours is Wang et al. (2020c), where they proposes to pair an environment classifier with gradient reversal layer to learn an environment-agnostic representation. However, they only consider one single environment when learning the visual representation for a given path (i.e., given one path and predict its environment). In our environment-agnostic representation learning, we explore the connections between multiple environments (i.e., maximize the similarity between paths from different environments).
Vision-and-language with multilinguality. There has been growing interest in combining vision and language for tasks such as visual-guided machine translation (Sigurdsson et al., 2020; Surís et al., 2022; Huang et al., 2020), multi-lingual visual question answering (Gao et al., 2015; Gupta et al., 2020; Shimizu et al., 2018), multi-lingual image captioning (Gu et al., 2018; Lan et al., 2017), multilingual video captioning (Wang et al., 2019b), and multi-lingual image-sentence retrieval (Kim et al., 2020; Burns et al., 2020). In this paper, we work on multi-lingual vision-and-language navigation. We use vision (i.e., navigation path) as a bridge between multi-lingual instructions and learn a cross-lingual representation that captures visual concepts. Moreover, our method also use language as a bridge between different visual environments to learn an environment-agnostic visual representation.
In this section, we present our CLEAR method that learns cross-lingual language representations and environment-agnostic visual representations. Given these learned language and visual representations, we then train the agent on the vision-and-language navigation task with imitation learning and reinforcement learning. The overall representation learning and navigation agent training processes are illustrated in Figure 2. We next describe our representation learning methods in Sec. 3.1 and Sec. 3.2. The navigation model (Tan et al., 2019) and training process are detailed in Appendix.
Figure 2: Left: the agent learns a cross-lingual language representation and an environment-agnostic visual representation via maximizing the similarity between positive pairs (connected with blue line) and minimizing the similarity between negative pairs (connected with red dashed line). For simplicity, we use 3 as batch size when illustrating the positive pairs and negative pairs. Right: then the agent is trained on the vision-and-language navigation task based on these learned representations.
3.1 Language Representation Learning
The goal of our language representation learning approach is to learn a cross-lingual language representation that can mitigate the natural ambiguity and variance in multilingual instructions and improve the path-instruction alignment by capturing the shared and salient visual concepts underlying the instructions. We define visually-aligned instruction pairs as instructions that correspond to the same navigation path. Since these instruction pairs refer to the same navigation path, the visual concepts underlying these instructions (e.g., visual objects mentioned in the instruction) are shared. Thus, we could train the language representation to emphasize these visual concepts by learning the connection between these visually-aligned instruction pairs.
For each navigation path, the Room-Across-Room (RxR) dataset (Ku et al., 2020) provides 9 corresponding language instructions in 3 languages (English, Hindi, and Telugu). During training, for each navigation path, we randomly sample two instructions out of the nine corresponding instructions as the visually-aligned instruction pairs. The two instructions can be in different languages, which helps the agent learn a cross-lingual language representation. Exclusively learning connections between instructions in the same language will lose crucial information across languages, and we quantitatively illustrate this result in Sec. 6.1.
Given the instruction  $\{w_{i}\}_{i = 0}^{m}$  with  $m$  words, we use feature of the [CLS] token (i.e.,  $w_{0}$ ) in the pre-trained multilingual BERT (Devlin et al., 2019)
outputs as the sentence representation  $\tilde{w}$ :
$$
\left\{\widehat {w} _ {i} \right\} _ {i = 0} ^ {m} = \mathrm {m - B E R T} \left(\left\{w _ {i} \right\} _ {i = 0} ^ {m}\right) \tag {1}
$$
$$
\widetilde {w} = \widehat {w} _ {0} \tag {2}
$$
In a batch of size  $N$ , we have  $N$  positive pairs of instructions with representations  $(\widetilde{w}_j, \widetilde{u}_j)_{j=1}^N$  from Eqn. 2. Each positive pair is matched with  $2(N-1)$  negatives in the batch (i.e.,  $\{\widetilde{w}_k\}_{k \neq i}$  and  $\{\widetilde{u}_k\}_{k \neq j}$ ). Our goal is to learn a representation that maps instructions for the same path closer to each other in the representation space, regardless of the language and the natural variance in human-generated instructions. We learn the representation by optimizing a contrastive loss:
$$
L _ {l a n g} = - \sum_ {i = 1} ^ {N} \log \frac {\exp \left(\alpha_ {i , i} / \tau\right)}{\sum_ {k = 1} ^ {2 N} \mathbb {1} _ {k \neq i} \exp \left(\alpha_ {i , k} / \tau\right)} \tag {3}
$$
$$
\alpha_ {i, j} = \frac {\widetilde {w} _ {i} ^ {T} \widetilde {u} _ {j}}{\| \widetilde {w} _ {i} \| \| \widetilde {u} _ {j} \|} \tag {4}
$$
where  $\alpha_{i,j}$  is the similarity between the instruction  $\widetilde{w}_i$  and  $\widetilde{u}_j$ , and  $\tau$  is the temperature hyperparameter.
3.2 Visual Representation Learning
Our goal in visual representation learning is to learn an environment-agnostic visual representation that can mitigate the environment bias caused by objects' low-level appearance, such that it could generalize better to unseen environments. Intuitively, the agent would learn the general concept of objects instead of the low-level appearance if the agent can identify the same objects in two images
in different environment. Thus, we train the agent to learn the connected visual semantics between the semantically-aligned navigation paths (i.e., paths that mention the same objects or mention similar actions in different environments).
Identifying semantically-aligned path pairs: Although the appearance of the path varies a lot in different environments, the instructions that describe the similar paths are more consistent across environments. Based on this intuition, we use language as the bridge between paths in multiple visual environments. Specifically, we propose to use instruction similarity as a direct measurement of how semantically similar two paths are. For each instruction-path pairs  $(I,P)$  given in the Room-Across-Room (RxR) dataset, we first represent each instruction  $I$  as in Eqn. 2. Then, we compute the cosine similarity between the representation of instruction  $I$  and all the other instructions in the training set. We pick the instruction  $\widehat{I}$  that is most similar to  $I$  and also constraints that  $\widehat{I}$ 's corresponding path  $\widehat{P}$  has the same path length as  $P$ . Thus, we group  $P$  and  $\widehat{P}$  as the semantically-similar path pair.
Constraint on object-matching: In a batch of size  $N$ , we have  $N$  positive semantically-aligned path pairs  $(P_k, Q_k)_{k=1}^N$ . We represent the positive path pair  $(P_k, Q_k)$  as sequences of panoramic views  $(\{p_{k,t}\}_{t=1}^{L_k}, \{q_{k,t}\}_{t=1}^{L_k})$  with length  $L_k$ . Since paths might not be fully aligned (i.e., correspondence between image pairs  $\{p_{k,t}\}$  and  $\{q_{k,t}\}$  might not hold), we use object-matching to filter out image pairs that don't contain the same objects. Specifically, we use Mask-RCNN (He et al., 2017) model trained on LVIS dataset (Gupta et al., 2019) in detection2 (Wu et al., 2019) to detect objects in the 36 discretized views of the panoramic view. We filter out object classes that appear less than  $1\%$  of the time in all panoramic views. 27 object classes left, including objects like 'cabinet', 'chair', and 'sofa'. All object classes can be found in Appendix. During training, we randomly sample 10 out of 27 object classes in each iteration and filter out image pairs that don't contain same objects of the sampled 10 object classes. Our object-matching constraint ensures that the corresponding image pairs  $\{p_{k,t}\}$  and  $\{q_{k,t}\}$  also have a high semantic similarity.
Visual encoder: The panoramic view of time step  $t$  is discretized into 36 single views  $\{o_{t,i}\}_{i=1}^{36}$ . We
encode the visual representation for each view as:
$$
\widehat {o} _ {t, i} = \text {p r e - t r a i n e d m o d e l} \left(o _ {t, i}\right) \tag {5}
$$
$$
v _ {t, i} = W _ {v 1} \operatorname {R e L U} \left(W _ {v 2} \widehat {o} _ {t, i}\right) \tag {6}
$$
$$
\widehat {v} _ {t, i} = \operatorname {L a y e r N o r m} \left(v _ {t, i} + \widehat {o} _ {t, i}\right) \tag {7}
$$
We first encode images with pre-trained vision models. Then the encoded view features are passed through two fully-connected layers with ReLU as activation function. Layer normalization and residual connection are applied on top of the fully-connected layer.
Learning visual representation: Given the N positive semantically-aligned path pairs  $(P_{k},Q_{k})_{k = 1}^{N}$ , at each time step  $t$ , we have  $N_{p}$  panoramic views (computed as the average of 36 single views as in Eqn. 10) that have a positive pair (i.e., the paired view contain at least one same object). For each view  $p_{k,t}$  that has a positive pair, the visual encoder is trained to predict which of the  $N$  possible panoramic views  $\{q_{k,t}\}_{k = 1}^{N}$  contain similar semantic information. Specifically, we train the visual encoder to maximize the cosine similarity of the  $N_{p}$  positive image pairs in the batch while minimizing the cosine similarity of the  $N*N_{p} - N_{p}$  negative image pairs (i.e. each view has  $N - 1$  negatives). We optimize the contrastive loss as:
$$
L _ {\text {v i s u a l}} = - \sum_ {k = 1} ^ {N _ {p}} \sum_ {t = 1} ^ {L _ {k}} \log \left(\operatorname {S o f t m a x} _ {k} \left(\beta_ {k, t} / \tau\right)\right) \tag {8}
$$
$$
\beta_ {k, t} = \frac {p _ {k , t} ^ {T} q _ {k , t}}{\| p _ {k , t} \| \| q _ {k , t} \|} \tag {9}
$$
where  $\beta_{k,t}$  is the similarity between positive panoramic view pair  $p_{k,t}$  and  $q_{k,t}$ , and  $\tau$  is the temperature hyperparameter. We compute the panoramic view representation as the average of 36 single views:
$$
p _ {k, t} = \frac {1}{3 6} \sum_ {i = 1} ^ {3 6} \widehat {v} _ {p, k, t, i} \tag {10}
$$
where  $\widehat{v}_{p,k,t,i}$  is the output representation from the visual encoder.  $q_{k,t}$  is computed similarly.
Our CLEAR agent has two stages of learning: representation learning and navigation learning.
In the representation learning stage, we train the multilingual encoder and visual encoder by optimizing the contrastive loss  $L_{lang}$  in Eqn. 3 and
Models	SR↑	SPL↑	NDTW↑	SDTW↑
RxR	20.98	18.55	36.81	16.88
CLIP	38.34	35.17	51.10	32.42
Our	40.29	36.57	53.69	34.86
Table 1: Test leaderboard results under single run setup. RxR is the mono-lingual baseline in Ku et al. (2020), CLIP is the mono-lingual agent in Shen et al. (2022)
$L_{\text{visual}}$  in Eqn. 8 respectively. The representation learning process transfers the language representation to domain-specific language representation and adapts the visual representation to learn the correlation underlying the navigation environments.
In the navigation learning stage, we use a mixture of imitation learning and reinforcement learning to train the agent on the navigation task as in Tan et al. (2019). Details can be found in Appendix.
We evaluate our agent on the Room-Across-Room (RxR) dataset (Ku et al., 2020). The dataset is split into training set, seen and unseen validation set, and test set. In the unseen validation set and test set, the environments are not appeared in training set. Thus the performance on these two sets show the model's generalizability to new environments. More details can be found in Appendix.
To evaluate the performance of our model, we follow the metrics used in the Room-Across-Room paper (Ku et al., 2020) (details in Appendix): Success Rate (SR), Success rate weighted by Path Length (SPL) (Anderson et al., 2018a), normalized Dynamic Time Warping (nDTW) (Magalhaes et al., 2019), and success rate weighted by Dynamic Time Warping (sDTW) (Magalhaes et al., 2019). nDTW and sDTW are the main metrics for RxR and SR and SPL are the main metrics for R2R.