SlowGuess/NAACL_2022_OCR huggingface dataset:
Peerat Limkonchotiwat†, Wuttikorn Ponwitayarat†, Can Udomcharoenchaikit†, Ekapol Chuangsuwanich‡, Sarana Nutanong†
†School of Information Science and Technology, VISTEC, Thailand  
‡Department of Computer Engineering, Chulalongkorn University, Thailand  
{peerat.l_s19, wuttikornp_pro, canu_pro, snutanon} @vistec.ac.th, ekapolc@cp.eng.chula.ac.th
Cross-Lingual Retrieval Question Answering (CL-ReQA) is concerned with retrieving answer documents or passages to a question written in a different language. A common approach to CL-ReQA is to create a multilingual sentence embedding space such that question-answer pairs across different languages are close to each other. In this paper, we propose a novel CL-ReQA method utilizing the concept of language knowledge transfer and a new cross-lingual consistency training technique to create a multilingual embedding space for ReQA. To assess the effectiveness of our work, we conducted comprehensive experiments on CL-ReQA and a downstream task, machine reading QA. We compared our proposed method with the current state-of-the-art solutions across three public CL-ReQA corpora. Our method outperforms competitors in 19 out of 21 settings of CL-ReQA. When used with a downstream machine reading QA task, our method outperforms the best existing language-model-based method by  $10\%$  in F1 while being 10 times faster in sentence embedding computation. The code and models are available at https://github.com/mrpeerat/CL-ReLKT.
Cross-lingual question answering allows a question posed in one language to be answered using materials written in a different language. As exemplified in Figure 1, one may ask, "Who was the first king of Hongsawadee?" and have their answer retrieved from a collection of historical documents in Burmese or other languages. To support the given example application, we require a retrieval system that can handle documents and questions in multiple languages at the same time. That is, we want to map questions and answers from multiple languages into the same space for easy retrieval. This functionality is also known as Cross-Linguial Retrieval Question Answering (CL-ReQA).
Figure 1: Overview of CL-ReQA. A user wishes to retrieve the answer to the question "Who was the first king of Hongsawadee?" from a collection of multilingual documents.
1.1 Existing Methods
One prominent approach to CL-ReQA is multilingual sentence embedding, i.e., creating an embedding space that can handle questions and answers from different languages. This approach can be further categorized into (i) LM-Based: finetuning a language model (LM), e.g., mBERT and XLM-R; (ii) USE-Based: finetuning the Universal Sentence Encoder (USE) for QA.
LM-based. Devlin et al. (2019) and Conneau et al. (2020) proposed a pretrained large-scale language model (LM) with multiple languages (100+ languages) called mBERT and XLM-R, respectively. Both solutions rely on finetuning the LM part to the target task. Reimers and Gurevych (2020) showed an accuracy improvement from  $11.6\%$  to  $88.6\%$  after finetuning with a bilingual text mining task. Finetuning LMs has been explored by many recent works, e.g., triplet loss with various supervised learning tasks (Reimers and Gurevych, 2019), knowledge distillation (Reimers and Gurevych, 2020), dense network QA encoder (Karpukhin et al., 2020), and providing initial word embeddings for the translation task (Feng et al., 2020). Nonetheless, finetuning these models requires a large number of training samples (more than 100,000 sentences in some cases (Reimers and Gurevych, 2020; Zhang et al., 2021; Wang et al.,
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2141 - 2155 July 10-15, 2022 ©2022 Association for Computational Linguistics
2021)) to give the best performance in multilingual settings. On the other hand, cross-lingual QA training corpora are usually smaller with only 1,000 to 1,500 questions per language. We need a method that can operate with a limited amount of data.
Multilingual Universal Sentence Encoding (mUSE). Based on the Universal Sentence Encoder (USE) architecture (Cer et al., 2018), Yang et al. (2020) proposed a training method utilizing a multilingual corpus with 16 different languages and multiple training objectives. They call their pretrained network multilingual USE or mUSE.
Experimental results from Trijakwanich et al. (2021) show that mUSE provides superior performance over the LM-based methods. However, this method performs poorly on languages outside the mUSE training corpus, i.e., unsupported languages. This limitation hinders the adoption of mUSE on limited-resource languages.
1.2 Our Work
Proposed Method. In this paper, our goal is to improve the robustness of multilingual sentence embedding that works with a wide range of languages, including those with a limited amount of training data. Leveraging the generalizability of language knowledge transfer (LKT), we propose a Cross-Lingual Retrieval Language Knowledge Transfer (CL-ReLKT) framework. Figure 1 illustrates how cross-lingual retrieval can be conducted through a multilingual embedding function  $h(\cdot)$ . Given a question-document pair  $(q,d)$  in any language,  $h(d)$  is closer to  $h(q)$  than any other documents using any similarity measure, e.g., cosine similarity.
Figure 2: QA vector representations before and after performing the CL-ReLKT framework. The main goal of our framework is to improve the consistencies between document-question pairs from different languages in the embedding space so that they can be correctly retrieved.
Learning Objective. As shown in Figure 2, the proposed CL-ReLKT framework is designed to improve the embedding space by making crosslingual question-answer pairs closer to each other. The crux of our proposed framework lies in the
following two parts. First, we formulate a LKT process to create a language-generalized student. In particular, we leverage the fact that there is likely to be one language in a large multilingual corpus that dominates all others. We use that language to help improve the embedding quality of other languages. Second, we formulate a new loss function designed to improve the cross-lingual consistency between question-answer pairs in a multilingual environment. We aim to improve the consistencies between the teacher (dominant language) and student (other languages) for the following teacher-student output pairs: question-question, document-document, and document-question.
Experimental Studies. To determine the effectiveness of our approach, we compared the proposed methods with the current best practices (discussed in Section 1.1) on the CL-ReQA task across three datasets in 15 languages. Experimental results show that the CL-ReLKT framework outperformed all competitive methods on languages supported by mUSE in all cases. The results on unsupported languages, i.e., languages outside of the mUSE training corpus, show that the CL-ReLKT framework improved the performance of the mUSE encoder significantly  $(p < 0.05)$  in all cases. Moreover, on a downstream task of machine reading QA (MR-QA), our method obtained better F1 and exact match scores than those of the best existing LM-based method in seven out of eight cases. Last but not least, our method is also 10 times faster than the state-of-the-art LM-based competitor in sentence embedding computational cost.
Summary of Contributions.
- We propose a new language knowledge transfer method called Cross-Linguual Retrieval Language Knowledge Transfer (CL-ReLKT) to transfer knowledge from the dominant language to non-dominant languages and build a language-generalized encoder.
- We design a new loss function to enforce crosslingual consistency between dominant and non-dominant language vector representations.
- To assess the performance and efficiency of the models, we conducted an extensive set of experimental studies involving 2 tasks, 15 languages, and 8 competitors. Experimental results show the benefits of our proposed CL-ReLKT framework. Moreover, we found that retrieving answers at the document level yields a significant improvement over the passage-level methods.
2.1 Dominant Language
In a multilingual dataset, the distribution of languages tends to be imbalanced. As shown in Figure 3, the number of sentences in English is approximately  $50\%$  of all sentences in the corpus used to construct mUSE (Yang et al., 2020).
Figure 3: The distribution of QA training data used by mUSE (Yang et al., 2020) $^1$ .
Due to the stated language imbalance, the model performance in languages with a large amount of data tend to be substantially better than that in other languages (Arivazhagan et al., 2019; Wang et al., 2020). This issue can be problematic when we want the model performance to be consistent across multiple languages.
For the case of mUSE, as shown in Figure 3, we can see that English is the dominant language in terms of training data available. Hence, the English-to-English retrieval performance tends to be better than all other language pairs. To verify this performance gap, we conducted a CL-ReQA experimental study using questions in non-English and answer documents in English; mUSE was used to encode the questions and documents. Experimental results show a significant performance improvement when the questions are translated into English instead of using the original non-English questions, i.e., translated questions from Russian to English improving the precision-at-1 from  $43.3\%$  to  $52.8\%$ . For the full results, see Appendix A.3.
2.2 Language Knowledge Transfer (LKT)
There are many techniques to boost a model performance on low-resource languages using the structure obtained from rich-resource ones. Transfer and multitask learning have been popular paradigms for
leveraging rich-resource languages. These methods usually rely on the shared-encoder strategy so that the language pattern learned in one language can be shared across all other languages using the same model (Lin et al., 2019; Nooralahzadeh et al., 2020; Zoph et al., 2016; Schwenk and Douze, 2017; Neubig and Hu, 2018; Yang et al., 2020; Feng et al., 2020). These classes of techniques are commonly known as Language Knowledge Transfer (LKT).
With a shared encoder, improvements on one language tend to benefit other languages as well. Let us consider a scenario where we have a large number of question-answer pairs in English and a significantly smaller number of pairs in other languages, e.g., Russian, French, and German. By letting other languages share the same encoder as we update the encoder weights while training with English data, we can also improve the general encoding performance of the model in other languages.
In this section, we formulate our proposed methods by leveraging the two concepts discussed in the previous section, dominant language and language knowledge transfer. In particular, we perform language knowledge transfer to transfer the knowledge from the dominant languages to other languages. Our proposed method consists of two stages: teacher model preparation and Cross-Lingual Retrieval Language Knowledge Transfer (CL-ReLKT), which are described as follows.
3.1 Stage 1: Teacher Model Preparation
The purpose of this stage is to create a strong teacher for language knowledge transfer in the next stage. For a base model, we use  $\mathrm{mUSE}_{\mathrm{small}}$  for efficiency and performance reasons<sup>2</sup>.
There are several techniques that can be used to create the teacher model,  $\mathrm{mUSE}_{\mathrm{teacher}}$ . The results from our ablation study (Appendix A.5) suggest that triplet loss is a reasonable choice. As shown in Equation 1, triplet loss  $\mathcal{L}_{\mathrm{tp}}$  is a training objective that maximizes the cosine similarity  $\cos (\cdot)$  between anchor-positive pairs  $(a,p)$  and makes similarity between anchor-negative pairs  $(a,n)$  smaller than a given threshold  $\alpha$  for all the training data  $M$ .
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {t p}} = \sum_ {i = 0} ^ {| M |} \left[ \max  \left(\left(1 - \cos \left(h \left(a _ {i}\right), h \left(p _ {i}\right)\right)\right) - \right. \right. \tag {1} \\ \left.\left. (1 - \cos \left(h \left(a _ {i}\right), h \left(n _ {i}\right)\right)\right) + \alpha , 0\right)\left. \right] \\ \end{array}
$$
<sup>1</sup>For brevity, we use the ISO-639 standard to refer to the languages used in this paper.
See Sections 5.2 and 5.4 for further details.
While the anchors  $a$  can be randomly sampled from the questions, we need the CL-ReQA model to help select positives  $p$  and negatives  $n$ . For negative sample categorization, we consider two options. First, we can directly use the original  $\mathrm{mUSE}_{\mathrm{small}}$  model to categorize the negative samples according to the current embedding space (online fashion) (Kaya and Bilge, 2019). Second, we can apply the method proposed by Karpukhin et al. (2020), which utilizes BM25 (Trotman et al., 2014) to produce textual similarity scores. From the ablation study given in Appendix A.5.1, the results show that the first three epochs use the initial strategy for triplet mining (Kaya and Bilge, 2019) before proceeding to online mining (Kaya and Bilge, 2019) for five epochs yield the best performance.
3.2 Stage 2: CL-ReLKT
We now describe our method to improve the general CL-ReQA performance using the concept of language knowledge transfer (LKT). As we mentioned in Section 2.2, LKT is a technique that the language pattern learned in one language can be shared across all other languages using the same model. Applying the same concept to our problem, we can set the LKT process to improve the embedding consistency between the dominant language and other languages. In particular, we setup the LKT environment as follows: (i) the teacher operates in the dominant language, i.e., English; (ii) the student operates in non-dominant languages; (iii) the student tries to mimic the embedding outputs of the teacher. In what follows, we describe the teacher and student models, inputs, and the loss function for the training process.
Figure 4: The training process of Cross-lingual Retrieval Language Knowledge Transfer (CL-ReLKT) comprising (i) a teacher model  $(T)$ ,  $\mathrm{mUSE}_{\text {teacher }}$ ; (ii) a student model  $(S)$ ,  $\mathrm{mUSE}_{\text {cl-relkt }}$ ; and (iii) three training objectives, Obj 1-Obj 3
Teacher and Student Models. As illustrated in Figure 4, the Cross-Linguual Retrieval Language Knowledge Transfer (CL-RelK T) process consists of a teacher, student, and loss function. Initially, the
student's parameters are initialized to the same values as those of the teacher trained in Stage 1. During the training process, the teacher's parameters are fixed; we only adjust the student's parameters according to the loss function.<sup>3</sup>
Inputs. Let us now consider input questions and answer documents of the training process. As illustrated in Figure 4, both teacher and student models accept the same document input  $d$ . However, there are two different versions for each question, English  $q^{\mathrm{en}}$  and non-English  $q^{\mathrm{ne}}$ . The English question  $q^{\mathrm{en}}$  is a translation of the original one  $q^{\mathrm{ne}}$ . This gives us a question pair  $(q^{\mathrm{ne}}, q^{\mathrm{en}})$  for language knowledge transfer between different languages. For simplicity, we use Google Neural Machine Translation (GNMT) to translate  $q^{\mathrm{ne}}$  into  $q^{\mathrm{en}}$ . Note that if available, one may also use human-translated parallel questions.
The teacher model  $T()$  accepts  $q^{\mathrm{en}}$  as input, while the student model  $S()$  accepts  $q^{\mathrm{ne}}$  as input. In other words,  $q^{\mathrm{en}}$  functions as the "reference" of the LKT process. According to our assessment (Appendix A.5.3), English provides the best performance and hence is chosen as the dominant language for the training process. Note that this finding also conforms with the data distribution shown in Figure 3.
Loss Function. The goal of our CL-ReLKT loss function  $\mathcal{L}_{\mathrm{CL - ReLKT}}$  is to let the student mimic the teacher's knowledge from the dominant language to the student's target language. As shown in Figure 5, our loss function  $\mathcal{L}_{\mathrm{CL - ReLKT}}$  has three consistency objectives, namely, question-question, document-document, and document-question. We describe them as follows.
- Obj 1: Question-Question. The first objective is to enforce the consistency between  $S()$  and  $T()$  when encoding the same question expressed in English  $q^{\mathrm{en}}$  and non-English  $q^{\mathrm{ne}}$ , respectively.
- Obj 2: Document-Document. While adjusting the student  $S()$  for the first objective, we also want to keep its answer document encoding unchanged. Hence, we want to maintain the consistency between  $T(d)$  and  $S(d)$ .
- Obj 3: Document-Question. To accommodate the lookup process, the embedding space should also keep question-answer pairs consistent with each other. As our third objective, we minimize
Note that the student model can be of any architecture and can be initialized using any method. In this work, we choose the self-model for simplicity. See Section 5.5 and Appendix A.4 for more information.
the discrepancy between the student's question vector  $S(q^{\mathrm{ne}})$  and the teacher's document vector  $T(d)$ .
Figure 5: Illustration three objectives of Cross-Lingual Language Knowledge Transfer (CL-ReLKT) loss function.
We formulate the loss function  $\mathcal{L}_{\mathrm{CL - ReLKT}}$  as a linear combination of these three consistency objectives. Using the squared L2 norm as the discrepancy measure, we obtain the following loss function:
$$
\mathcal {L} _ {\mathrm {C L - R e L K T}} = \frac {\gamma}{| M |} \sum_ {i = 0} ^ {| M |} [ \beta | | T (q _ {i} ^ {\mathrm {e n}}) - S (q _ {i} ^ {\mathrm {n e}}) | | ^ {2} + \tag {2}
$$
$$
\lambda | | T (d _ {i}) - S (d _ {i}) | | ^ {2} + \omega | | T (d _ {i}) - S (q _ {i} ^ {\mathrm {n e}}) | | ^ {2} ],
$$
where  $M$  is the set of training samples used in a given batch, and  $\beta$ ,  $\lambda$ , and  $\omega$  are the weighting coefficients.
Discussion. As stated earlier, the goal of the loss function is to transfer the teacher's knowledge to the student operating in target languages. Since the performance of the teacher's dominant language is generalized, after the LKT process, other languages will have the same properties. The experimental results show that the student can better handle unsupported languages and improve the performance of supported languages than the teacher model. This improvement comes from the cross-lingual consistency objectives, Obj 1 and Obj 3, in the loss function, while Obj 2 maintains the monolingual consistency. Moreover,  $\mathcal{L}_{\mathrm{CL - ReLKT}}$  does not require the teacher and student models to be of the same architecture; it can be applied to any pre-trained models. (For more information, see Appendix A.4)
To evaluate the effectiveness of our method, we conduct our experiments on three well-known CL-ReQA corpora: XORQA, XQuAD, and MLQA. All experiments were done by XX  $\rightarrow$  EN where XX is the question language (15 languages), and EN is answer passages or documents.
XORQA (Asai et al., 2021a) is a benchmark dataset for multilingual open-retrieval question answering. The dataset contains questions in a diverse set of seven non-English languages and answer documents in English. We use the Gold Paragraph part of the corpus, which contains 12,895 documents and 8,949 question-answer pairs. The authors, however, did not provide a public test dataset. Thus, we divide the samples into train/dev/test  $(0.7 / 0.1 / 0.2)$ .
XQuAD (Artetxe et al., 2020) is a dataset for evaluating cross-lingual question answering performance. XQuAD comprises 48 documents and 13,090 question-answer pairs obtained from the development set of SQuAD v1.1 (Rajpurkar et al., 2016) with 11 languages. Since XQuAD is too small for model training, we used it for testing only. Translated questions from SQuAD v1.1 (training set) were used instead for training the models (the same setup as XQuAD (Artetxe et al., 2020)).
MLQA (Lewis et al., 2020) is also a dataset for evaluating cross-lingual question answering performance. The dataset contains 15,806 documents and 33,706 question-answer pairs in seven different languages. However, the authors did not provide any training dataset. As a result, we combined the development and test datasets and divided them into train/dev/test (0.7/0.1/0.2).
We compare the performance of our method with two groups of competitive methods as follows:
LM-based. As discussed in Section 1, one approach to CL-ReQA is to use an embedding space based on some language model. In the experimental studies, we compare our methods to the following LM-based competitors.
- XLM-R-nli-stsb: A RoBERTa-based crosslingual model trained using the NLI and STS benchmark datasets (Reimers and Gurevych, 2019).
- mBERT-triplet: A BERT-based multilingual model finetuned with a QA dataset using triplet loss (Reimers and Gurevych, 2019).
- XLM-R←SBERT: A XLM-RoBERTa model trained by distilling from the sentence BERT model (Reimers and Gurevych, 2020).
- DPR: A dense-network solution using the multilingual BERT model to provide the cross-lingual QA capability (Karpukhin et al., 2020).
- CORA: An adaptation of DPR on multilingual Wikipedia QA data (Asai et al., 2021b).
- LaBSE: A multilingual sentence encoder using mBERT to provide initial word embedding vectors (Feng et al., 2020).
We retrained mBERT-triplet, XLM-R←SBERT and DPR with the CL-ReQA training set following previous work (Reimers and Gurevych, 2019; Asai et al., 2021b; Zhang et al., 2021).
Multilingual Universal Sentence Encoding (mUSE). As an alternative to LM, we can also construct a QA embedding space from a well-known multilingual sentence encoder, mUSE. In particular, we consider the following mUSE variants.
-  $mUSE_{small}$ : The  $mUSE_{small}$  encoder was based on Convolution Neural Network (Kim, 2014).
-  $mUSE_{large}$ : The  $mUSE_{large}$  encoder was on the transformer architecture (Vaswani et al., 2017). Note that although there exists a QA variant of  $mUSE$ ,  $mUSE_{qa}$ , we found that this QA variant does not provide any performance improvement over  $mUSE_{large}$ . As a result, we omit  $mUSE_{qa}$  from our study.
Our proposed methods. As previously discussed, we construct our proposed methods based on  $\mathrm{mUSE}_{\mathrm{small}}$ . The first method,  $\mathrm{mUSE}_{\mathrm{teacher}}$ , is constructed from triplet loss where each triplet consists of a question, its corresponding answer document, and a non-answer document. The second method,  $\mathrm{mUSE}_{\mathrm{cl-relkt}}$ , is constructed from the process of Cross-Linguial Retrieval Language Knowledge Transfer (with all languages in the training datasets).
4.3 Hyperparameter and Evaluation Settings
Hyperparameter. In these experiments, we use grid search on the following hyperparameters: learning rates, triplet loss margin  $(\alpha)$ ,  $\mathcal{L}_{\mathrm{CL - ReLKT}}$ ’s coefficients  $(\gamma ,\beta ,\omega)$ , and the number of negative samples for triplet loss. The hyper-parameter configurations are given in Table 1.
Hyperparameters	Values for grid search
Learning Rates	1e-3,5e-4, 1e-4, 1e-5, 1e-6
α	0.1-1 (0.01/steps)
γ	[100, 1000, 10000]
β	1,1e-1, 1e-2, 1e-3, 1e-4
ω	1,1e-1, 1e-2, 1e-3, 1e-4
#negative samples	[1, 2, 3, 5, 10]
For the Cross-Linguial Retrieval Language Knowledge Transfer settings, we use a batch size of 8 with a total number of 10 epochs. Since the
student model receives the initial weights from the teacher, the CL-ReLKT's loss value ranges between  $[10^{-3}, 10^{-5}]$  and the loss value of Obj 3 is lower than those of other objectives. To prevent the CL-ReLKT's loss value from being too small, we multiply the value by  $\gamma$  and set the coefficient of Obj 2,  $\lambda$ , to 1. In addition, we evaluate the precision score on the development set every 100 steps. If the precision score does not improve, the learning rate is halved.