SlowGuess/NAACL_2022_OCR huggingface dataset:
Evaluation. We use precision at  $k$  where we set  $k$  to 1 (P@1) which is a common practice for the CL-ReQA task (Ahmad et al., 2019; Yang et al., 2020; Guo et al., 2021) and cross-lingual retrieval tasks (Reimers and Gurevych, 2020; Feng et al., 2020). We also provide precision at 5 and 10 results in Appendix A.2. Furthermore, we used McNemar's test as the significant statistical measurement  $(p < 0.05)$  for all experiments.
5.1 Passage- vs Document-bases on Machine Reading QA (MR-QA)
To determine the best answer retrieval unit for MRQA, we compare two scenarios. (i) Passage-based: Retrieving answers as passages; (ii) document-based: Retrieving answers as documents. For conciseness, we chose DPR, which is the state-of-the-art LM-based competitor, for comparison. For testing, we chose XORQA, which is the newest MR-QA benchmark. For all test cases, we used the same machine reading comprehension model constructed from XLM-R. In particular, we finetuned XLM-R using the same training portion of XORQA described in Section 4.1.
Table 1: Hyperparameter configurations.
Model	XORQA
RU	KO	JA	FI	AVG
F1	EM	F1	EM	F1	EM	F1	EM	F1
Passage-based
DPR	17.4	12.9	6.1	0.3	14.6	10.4	21.7	15.3	15.0
mUSEcl-relkt	21.1	16.8	27.8	20.4	26.1	20.2	20.7	16.1	23.9
Document-based
DPR	16.8	13.0	6.2	0.3	15.3	11.5	22.5	16.4	15.2
mUSEcl-relkt	24.3	18.8	28.5	21.7	26.3	20.6	21.2	16.7	25.1
Table 2: F1 and EM scores on the cross-lingual machine reading QA.
Results. Table 2 displays the MR-QA scores as F1 and exact match (EM) and provides a comparison between the two MR-QA input options: passage-based and document-based. In all cases, the document-based option improves over the passage-based one. We can also see that our method signif
Model	XORQA	XQuAD	MLQA
RU	KO	JA	AR	DE	ES	RU	TH	ZH	TR	AR	DE	ES	ZH
LM-based
mBERT-triplet	41.0	19.0	46.4	23.5	53.4	52.1	44.1	6.7	36.6	43.3	10.1	28.9	32.8	14.7
XLM-R-nli-stsb	28.7	26.6	28.0	39.1	43.3	45.8	42.9	41.2	44.1	44.5	27.9	34.0	31.2	29.4
XLM-R←SBERT	21.5	19.8	20.7	39.5	39.9	41.6	41.6	40.8	40.3	43.3	24.8	33.4	30.8	34.7
DPR	33.8	2.0	26.9	38.7	51.3	58.0	52.1	10.9	29.2	41.2	35.5	56.6	59.0	55.0
CORA	18.9	11.5	10.4	21.0	39.9	36.1	34.5	4.6	20.6	24.8	18.2	31.2	35.6	19.4
LaBSE	29.8	26.7	33.2	41.2	43.7	47.1	42.4	13.0	44.5	40.8	33.8	35.4	38.4	40.3
Multilingual Universal Sentence Encoding (mUSE)
mUSEsmall	43.3	35.5	41.2	64.7	73.1	75.6	66.8	72.7	71.0	70.2	44.5	60.4	57.0	53.2
mUSElarge	52.1	41.1	47.7	57.1	65.1	68.5	59.7	63.4	62.2	61.8	35.6	42.3	39.5	31.8
Our proposed methods
mUSEteacher	54.2	44.5	47.7	68.5	79.8	82.4	72.3	75.2	82.3	72.7	49.1	64.8	62.8	57.1
mUSEcl-relkt	58.2	47.7	49.5	79.4	83.2	84.0	83.6	86.1	82.4	80.3	48.5	64.8	62.8	57.9
Table 3: Precision at 1 (P@1) on the CL-ReQA task in supported languages
icantly outperformed DPR for both input options on average.
Discussion. The document-based representation has advantages and drawbacks in comparison to the passage-based one. In particular, by grouping passages associated with the same document together, the retrieval unit becomes larger, making it harder to miss an answer. However, operating at the document level also means that we have to handle a larger input. That is, for the three corpora used in the experimental studies, the model has to handle 128 tokens on average when the input is a passage, while the input size can be up to 1,996 tokens when the input is a document. The results provide empirical evidence that the benefits outweigh the drawback. We believe that as machine reading models improve over time, longer input passages will provide even better results. For more information about the retriever's performance, see Appendix A.1.
5.2 CL-ReQA: Supported Languages
Previously, we have shown that document-based yield the best MR-QA results. Thus, we will use this setting for the rest of the embeddings experiments. In this experiment, we report the effectiveness of our proposed methods on mUSE's supported languages where the answer retrieval unit is document-based. We evaluated our methods against the competitors discussed in Section 4.2.
Results. As shown in Table 3, our proposed models  $mUSE_{teacher}$  and  $mUSE_{cl - relkt}$  provide significant improvements from the base model,  $\mathrm{mUSE}_{\mathrm{small}}$ . Moreover, our models also outperformed the largest pre-trained variant of mUSE,  $\mathrm{mUSE}_{\mathrm{large}}$ . All of our proposed models also significantly performed better than the LM-based competitors. The
results also show that our consistency enhancement method, CL-ReLKT, were effective in all cases except AR for the MLQA dataset.
Discussion. Experimental results verify that for languages supported by mUSE, our approach based on the language knowledge transfer concept (Section 2.2) can provide significant improvements over the teacher model. However, the improvements were less significant than ours when the language knowledge transfer concept is applied to an LM to create mBERT-triplet and XLM-R-nli-stsb from mBERT and XLM-R, respectively.
Notice that methods based on mBERT performed poorly in Thai (TH). We can also see that finetuning mBERT with triplet loss (mBERT-triplet) and multilingual dense retrieval (CORA) did not provide any improvements on Thai. This is because Thai was not included in the construction process of mBERT (uncased-version), and the amount of the training data is insufficient to improve the model.
5.3 CL-ReQA:Unsupported Languages
Let us consider how well our proposed models performed when used with languages not supported by the base model,  $\mathrm{mUSE}_{\mathrm{small}}$ , i.e., FI, RO, EL, HI, and VI. Similar to the study presented in the previous subsection, we used XORQA, XQuAD, and MLQA as our test corpora.
Results. Table 4 presents the P@1 scores of our methods and the competitors. We can see that the original mUSE models,  $\mathrm{mUSE}_{\mathrm{small}}$  and  $\mathrm{mUSE}_{\mathrm{large}}$ , did not perform well in these languages. As expected,  $\mathrm{mUSE}_{\mathrm{teacher}}$  had a tendency to provide some improvements over  $\mathrm{mUSE}_{\mathrm{small}}$ . This is because these languages were not included in the original training process, and the amount of
data is insufficient to improve the performance of these languages. In contrast, we obtained significant improvements through the CL-ReLKT methods,  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$ . For five out of seven cases,  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$  were the best performer compared to other models. Two LM-based methods, XLMR←SBERT and LaBSE, were the best performer in HI with the test corpora of XQuAD and MLQA, respectively.
As an alternative to cross-lingual retrieval, one can convert the problem to a monolingual retrieval one using a machine translation (MT) model. We found that using an MT model (i.e., GNMT, MBART) with DPR following Asai et al. (2021a) helps improve the performance of LM-based models. However, the performance decreases in some languages, i.e., in Finish (XORQA), DPR's performance is dropped from 39.1 to 32.0 and dropped to 36.4 when GNMT and MBart were applied to the DPR, respectively. For the full discussion and results see Appendix A.3.
Model	XORQA	XQuAD	MLQA
FI	RO	EL	HI	VI	HI	VI
LM-based
mBERT-triplet	18.7	48.3	30.3	23.9	39.9	7.1	24.1
XLM-R-nli-stsb	30.7	45.4	44.5	39.1	40.3	31.4	28.6
XLM-R←SBERT	25.3	42.4	42.9	40.3	40.3	29.4	28.4
DPR	39.1	52.9	36.1	15.5	10.1	26.2	32.1
CORA	15.1	27.7	26.9	18.5	28.6	15.8	23.9
LaBSE	40.6	42.0	42.9	37.8	39.9	31.8	27.2
Multilingual Universal Sentence Encoding (mUSE)
mUSEsmall	18.3	41.6	10.9	4.2	25.6	2.0	25.4
mUSElarge	27.2	49.6	13.0	3.4	25.6	1.4	16.8
Our proposed methods
mUSEteacher	25.0	42.4	13.4	4.2	29.4	2.0	27.4
mUSEcl-relkt	48.2	76.9	64.3	34.0	71.8	3.2	44.2
Discussion. The performance gap between  $\mathrm{mUSE}_{\mathrm{small}}$  and  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$  demonstrates the effectiveness of the proposed CL-ReLKT framework. In particular, we can use CL-ReLKT to generalize a base sentence embedding model to handle languages that were not originally included in the training process.
Regarding the CL-ReLKT performance on Hindi (HI), one important observation is that Hindi is the only language in this study whose family is not represented in the original training data, which results in a lot of OOV tokens from unknown characters. We provide more explanation in Appendix A.6.
5.4 Run-time Efficiency on Query Encoding
Let us now consider the efficiency of the methods. Since this investigation focuses on the embedding methods, we consider only the sentence embedding computational time. We used a DGX-1 machine using one Intel Xeon E5-2698 and one NVIDIA Tesla V100 GPU to benchmark the models.
Results. As shown in Table 5, the experimental result shows that  $\mathrm{mUSE}_{\mathrm{small}}$  is the fastest. The result shows that  $\mathrm{mUSE}_{\mathrm{small}}$  took only  $\sim 7.9$  ms on average to encode one query at a time. Since our method is based on the  $\mathrm{mUSE}_{\mathrm{small}}$  architecture, we also obtain a similar run time. For the LM-based methods, we found that LaBSE is the quickest one. However, the method is still slower than  $\mathrm{mUSE}_{\mathrm{teacher}}$  and  $\mathrm{mUSE}_{\mathrm{cl-relkt}}$  by at least  $80\%$ . We can also find that MT-assisted, i.e., GNMT and MBart, are significantly slower than our methods. For the MT-assisted results, GNMT used  $258.3~\mathrm{ms}$  for one query while MBart used  $9,132~\mathrm{ms}$ .
Table 4: Precision at 1 (P@1) on the CL-ReQA task in unsupported languages.
Model	XORQA	XQuAD	MLQA
MT-assisted
DPR+GNMT	258.3±37.9	339.3±54.2	395.0±58.1
DPR+MBart	9,132±8,111	6,527±5,838	5,382±274
LM-based
mBERT-triplet	20.2±1.0	30.6±1.5	30.2±1.7
XLM-R-nli-stsb	20.5±1.1	22.1±2.7	23.4±4.6
XLM-R←SBERT	22.2±3.7	31.6±1.9	31.3±2.4
DPR	58.3±28.5	110.3±47.6	103.0±52.5
CORA	197.7±9.9	369.7±16.4	274.7±145.3
LaBSE	15.2±2.6	14.3±2.2	14.6±2.3
Multilingual Universal Sentence Encoding (mUSE)
mUSEsmall	8.4±1.0	7.9±1.1	7.6±1.2
mUSElarge	23.8±2.5	30.3±5.0	27.0±4.7
Our proposed methods
mUSEteacher	8.6±1.0	8.3±1.1	8.1±1.0
mUSEcl-relkt	8.5±1.5	8.3±1.5	8.5±1.4
Table 5: The average computational sentence encoding time and standard division in ms.
Discussion. Since  $\mathrm{mUSE}_{\mathrm{small}}$  is a much smaller model than BERT-base and XLM-R-base, it is advantageous to use our proposed methods when efficiency is a concern, e.g., edge deployment. While GNMT and MBart were effective in improving the performance of LM-based models (Appendix A.3), the additional machine translation cost renders the approach less desirable.
5.5 Ablation Studies on the Training Objective for LKT
This study compares our Cross-Linguual Retrieval Language Knowledge Transfer method with other
LKT techniques using the same baseline model. To directly assess the effect of the LKT method, we use the original  $\mathrm{mUSE}_{\mathrm{small}}$  instead of the  $\mathrm{mUSE}_{\mathrm{teacher}}$  as the starting model in this study. We compare four training objectives:
- We apply the training objective following Reimers and Gurevych (2020)'s work denoted  $\mathrm{mUSE}_{\mathrm{mse}}$  where the training objective contains the first CL-ReLKT's objective with an additional loss term that minimizes the difference between English and English embeddings from the previous iteration;
- As mentioned in Section 5.3 the CL-ReLKT loss has three objectives.  $\mathrm{mUSE}_{\mathrm{cl - relkt}}^{\mathrm{q}}$  uses only the first objective;
-  $\mathrm{mUSE}_{\mathrm{cl - relkt}}^{\mathrm{qd}}$  uses the first and the second objective; and
- Lastly,  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$  uses the full version of the CL-ReLKT loss function.
Results. The experiment results are given in Table 6. As expected, our training objective outperformed competitive training objectives. The performance of  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$  outperformed other training objectives from six out of seven cases. Especially Reimers and Gurevych (2020)'s training objective, our method outperformed with significant results on six out of seven cases except for HI  $\rightarrow$  EN in MLQA.
Model	XORQA	XQuAD	MLQA
FI	RO	EL	HI	VI	HI	VI
mUSEsmall	18.3	41.6	10.9	4.2	25.6	2.0	25.4
mUSEmse	31.0	65.5	17.6	8.0	35.3	1.4	26.0
mUSEqcl-relkt	33.7	67.2	16.8	5.5	34.5	2.2	37.0
mUSEqdcl-relkt	38.5	62.6	18.9	8.4	34.5	3.7	38.0
mUSEcl-relkt	39.4	73.9	58.0	27.0	71.4	3.7	38.7
Table 6: Comparison of different LKT training objectives. Precision at 1 (P@1) on the CL-ReQA task in unsupported languages.
In this paper, we propose a novel Cross-Lingual Retrieval Language Knowledge Transfer framework for CL-ReQA. Our framework is designed to improve the general performance and enable the baseline model to handle unsupported languages by exploiting the concepts of Dominant Language and Language Knowledge Transfer. Our method outperformed competitive methods in all cases of supported languages and five out of seven cases of unsupported languages. Furthermore, we demonstrated that grouping passages associated with the
same document together could benefit machine reading QA.
7 Responsible NLP Research Checklist
Did you discuss the limitations of your work? The limitation of our work is out-of-domain problems. We strongly advise against using our model with out-of-domain data.
Did you discuss any potential risks of your work? There is a risk of retrieving incorrect documents causing the machine reading comprehension part to produce wrong answers.
Did you discuss the license or terms for use and/or distribution of any artifacts? The XORQA dataset is under the MIT License, while XQuAD and MLQA are under CC-BY-SA 4.0.
Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? XORQA, XQuAD, and MLQA were created for retrieval QA assessments; we use them for this exact purpose.
Did you discuss the steps taken to check whether the data that was collected/used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? From our careful inspection, there is no offensive content or sensitive data included in the three datasets: XORQA, XQuAD, and MLQA.
Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We reported the mean average score for all experiments, which is a common practice for reporting the performance.
If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? All experiments were done by Tensorflow because mUSE is only available on Tensorflow.
Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019. ReQA: An evaluation for end-to-end answer retrieval models. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering. Association for Computational Linguistics.
Naveen Arivazhagan, Ankur Bapna, Orhan First, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021a. XOR QA: Cross-lingual open-retrieval question answering. In NAACL-HLT.
Akari Asai, Xinyan Yu, Jungo Kasai, and Hannaneh Hajishirzi. 2021b. One question answering model for many languages with cross-lingual dense passage retrieval. NeurIPS.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. CoRR.
Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language processing from bytes. In Proceedings of the 2016 Conference of the North American Chapter of the Associ-
ation for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2021. Multireqa: A cross-domain evaluation for retrieval question answering models. Proceedings of the Second Workshop on Domain Adaptation for NLP.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Mahmut Kaya and Hasan Sakir Bilge. 2019. Deep metric learning: A survey. Symmetry, 11(9):1066.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Patrick Lewis, Barlas Oguz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Association for Computational Linguistics.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Trans. Assoc. Comput. Linguistics.
Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020. Zero-shot cross-lingual transfer with meta learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference
on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Holger Schwenk and Matthijs Douze. 2017. Learning joint multilingual sentence representations with neural machine translation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 2017, Vancouver, Canada, August 3, 2017.
Nattapol Trijakwanich, Peerat Limkonchotiwat, Wannaphong Phatthiyaphaibun, Raheem Sarwar, Ekapol Chuangsuwanich, and Sarana Nutanong. 2021. Robust fragment-based framework for cross-lingual sentence retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Andrew Trotman, Antti Puurula, and Blake Burgess. 2014. Improvements to bm25 and language models examined. In Proceedings of the 2014 Australasian Document Computing Symposium, ADCS '14.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
Liang Wang, Wei Zhao, and Jingming Liu. 2021. Aligning cross-lingual sentence representations with dual momentum contrast. In EMNLP. Association for Computational Linguistics.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
Yan Zhang, Ruidan He, Zuozhu Liu, Lidong Bing, and Haizhou Li. 2021. Bootstrapped unsupervised sentence representation learning. In ACL. Association for Computational Linguistics.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
A.1 CL-ReQA: Passage- vs Document-bases
This study reports the effect of the passage-based and document-based input on retrieval performance. The experiment was conducted in supported and unsupported languages of three datasets: XORQA, XQuAD, and MLQA.
Results. As shown in Table 7, the document-based input substantially outperform the passage-based one in all languages and all datasets. The performance of  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$  on passage-based for supported and unsupported languages is lower than document-based significant on every unsupported language. These results conform with those of the downstream task presented in Table 2.
Model	XORQA	XQuAD	MLQA
RU	JA	FI	DE	ES	VI	DE	ES	VI
Passage-base
mUSEcl-relkt	56.4	45.9	45.6	51.7	51.3	23.5	61.9	62.4	32.3
Document-base
mUSEcl-relkt	58.2	49.5	48.2	83.2	84.0	72.3	64.8	62.8	44.2
Discussion. This experiment shows that the efficiency of document-based is more robust than passage-based significant. Moreover, when we applied both inputs to a downstream task, MR-QA, the experiment results showed that changing the input from passage-based to document-based is more robust.
A.2 CL-ReQA: Precision at 5,10
In this experiment, we study the effectiveness of our method on mUSE's supported languages. We report the precision score at 5 and 10 on the XORQA dataset.
Results. As shown in Table 8, our proposed model,  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$ , on  $\mathrm{P@5}$  and  $\mathrm{P@10}$  have the same narrative as  $\mathrm{P@1}$  (Table 3). That is, the cross-lingual retrieval language knowledge transfer model improved the performance from the teacher model,  $\mathrm{mUSE}_{\mathrm{teacher}}$ , and it outperformed every LM-based model.