SlowGuess/NAACL_2022_OCR huggingface dataset:
Discussion. Experimental results verify that precision at other  $\mathbf{k}$ 's values does not change any conclusion from our work. The results show that our model outperformed other models in precision at 1, 5, and 10 settings.
Table 7: Comparison of different retrieval inputs such as passage- and document-bases on the CL-ReQA task in supported and unsupported languages.
Model	XORQA
P@5	P@10
RU	KO	JA	RU	KO	JA
LM-based
mBERT-triplet	59.3	37.6	63.3	68.2	46.6	70.4
XLM-R-nli-stsb	48.7	48.1	46.6	54.7	55.5	55.7
XLM←SBERT	43.8	46.2	43.0	54.7	55.7	53.4
DPR	61.6	4.0	56.7	74.2	15.0	67.9
CORA	48.1	34.0	31.1	62.2	48.4	41.7
LaBSE	50.1	45.9	51.8	59.6	57.0	59.8
mUSE-based
mUSEsmall	69.6	64.1	59.6	77.7	71.6	69.2
mUSElarge	75.1	68.1	71.0	82.2	74.3	78.5
Our proposed methods
mUSEteacher	78.2	69.6	69.2	82.8	78.1	78.5
mUSEcl-relkt	79.7	70.7	72.0	85.4	78.3	78.8
Table 8: Comparison of different Precision at  $k$  (P@k) where  $k$  values are equal to 5 and 10 on the CL-ReQA task in supported languages on XORQA.
A.3 CL-ReQA: Monolingual vs Cross-lingual Retrievals
In this study, we compare the CL-ReQA approach against the MT-assisted monolingual retrieval one. For CL-ReQA methods, we chose DPR and  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$ . For the MT-assisted methods, we used two translators, GNMT and MBart (Liu et al., 2020), to translate all questions into English which is the documents' language. These two translators were then applied to assist DPR in the same manner as the XORQA investigation (Asai et al., 2021a).
Results. Table 9 shows that the two translators provide substantial improvements to DPR and mUSE-based. The GNMT-assisted tended to perform better than the MBart-assisted in almost all cases. As state in Section 3.2, our language knowledge transfer comprise of GNMT in the training data process. Thus, the performance of our method with/without MT-assisted is similar. Our method was the best performer in six out of nine cases. Moreover, in seven out of nine cases, the performance of our method is decreased when MBart is applied. Since the performance of MBart is lower than GNMT, the performance of  $\mathrm{mUSE}_{\mathrm{cl - relkt}} + \mathrm{MBart}$  is dropped significantly.
Discussion. Machine translators can provide a quick solution to improve the performance of cross-lingual retrieval. In this way, the problem of cross-lingual retrieval is converted into a monolingual one. This approach can be useful when the language pair has a reliable translator, but there is insufficient QA data to create a cross-lingual retrieval
Model	XORQA	XQuAD	MLQA
RU	JA	FI	DE	ES	VI	DE	ES	VI
Cross-lingual retriever (LM-based)
DPR	33.8	26.9	39.1	51.3	58.0	10.1	56.6	59.0	32.1
LM-based + MT-assisted
DPR+GNMT	45.0	36.3	32.0	58.8	61.8	56.3	61.3	59.4	56.0
DPR+MBart	39.8	26.2	36.4	59.2	61.5	52.5	58.8	57.4	51.9
Multilingual Universal Sentence Encoding (mUSE)
mUSEsmall	43.3	41.2	18.3	73.1	66.8	25.6	60.4	57.0	25.4
mUSE-based + MT-assisted
mUSEsmall+GNMT	52.8	45.0	43.0	81.1	82.1	42.5	60.4	62.0	60.3
mUSEsmall+MBart	51.6	37.6	42.3	79.1	78.6	68.0	59.4	58.8	53.6
Our proposed method
mUSEcl-relkt	58.2	49.5	48.2	83.2	84.0	72.3	64.8	62.8	44.2
Our proposed method + MT-assisted
mUSEcl-relkt+GNMT	64.8	46.2	51.7	82.1	83.6	62.2	64.5	62.8	61.8
mUSEcl-relkt+MBart	48.2	37.7	38.2	83.2	83.5	34.0	63.0	62.4	58.5
model. However, we consider the multilingual sentence embedding approach to be superior to the MT-assisted one due to the following reasons: (i) the computational cost benefits of skipping the MT process; (ii) the MT-assisted approach can be negatively affected by a poor MT performance; (iii) the reliance of MT models can be a limitation for some language pairs. See Appendix A.7 for a further analysis.
A.4 CL-ReLKT on Other Architectures
This study demonstrates our cross-lingual retrieval language knowledge transfer method on LM-based models. For diversity, we select BERT-based (mBERT-triplet) and RoBERTa-based (XLM-R-nli-stsb) because these models trained only on the dominant language, English, same as  $\mathrm{mUSE}_{\mathrm{teacher}}$ .
Results. As shown in Table 10, we applied the CL-ReLKT framework on other architectures, i.e., BERT and RoBERTa, on supported languages of XORQA. The experimental results show that CL-ReLKT on LM-based models significantly improves XLM-R-nli-stsb. Furthermore, when we use mBERT-triplet as the teacher instead of XLM-R-nli-stsb. The result shows a small improvement over the student instead of using XLM-R as the teacher model.
Discussion. Our CL-ReLKT can be applied to any pre-trained model not limited to only in mUSE's architecture. However, to give the best results, let the student initialize the weight from the teacher. In addition, the results from both models are not over mUSE's performance.
Table 9: Precision at 1 (P@1) on the CL-ReQA task in supported and unsupported languages.
Model	XORQA
RU	KO	JA
LM-based
mBERT-triplet (1)	41.0	19.0	46.4
XLM-R-nli-stsb (2)	28.7	26.6	28.0
LM-based + CL-ReLKT
T=(1)	XLM-R-nli-stsbcl-relkt	29.0	27.1	28.3
T=(2)	XLM-R-nli-stsbcl-relkt	33.2	30.8	31.9
Table 10: CL-ReLKT on different architecture between student and teacher models. Where T is the teacher model.
A.5 Ablation Studies
This study presents the effect of each design decision in the triplet loss and cross-lingual retrieval language knowledge transfer proposed. Here, we investigate the following components: (i) training strategies; (ii) training data settings; (iii) reference languages for cross-lingual retrieval language knowledge transfer (CL-ReLKT); and (iv) distance functions for CL-ReLKT. In each investigation, we use the best setting from the previous steps. All experimental results were obtained from XX→EN retrieval on the XORQA test set across four languages, where XX can be one of these languages Russian (RU), Korean (KO), Japanese (JA), and Finnish (FI).
A.5.1 Training strategies
As shown in Table 11, we compare training strategies with/without each of the following components: initialization, online updates, and other deep metric learning techniques. As expected, the result shows that online negative sampling and initializing with BM25 helps improve the performance of triplet loss. Furthermore, we also study the effect of replacing triplet loss with contrastive learning as presented in the current state-of-the-art work (Karpukhin et al., 2020). We found that contrastive loss consistently provides a performance improvement over the original  $\mathrm{mUSE}_{\mathrm{small}}$  model but still lags behind the triplet loss.
A.5.2 Training data settings
In terms of training data for the teacher training, there are two decisions we need to consider: (i) the answer representation unit: whether to use one passage or one document as the retrieval unit in the training process; (ii) the question language: whether to use the original questions (in multiple languages) or translate them all to English. In the case of English, all English questions were trans
lated using GNMT. As shown in Table 11, the passage and English combination provides the best performance.
A.5.3 Teacher's language for CL-ReLKT
In this study, we explore the choice of language to function as the reference (teacher) in the CL-ReLKT process. Intuitively, we want a language that is well-represented in the training corpora when constructing the original model. Consequently, we compare English, Spanish, and German. As expected, English, which is the dominant language in the training corpora of mUSE, provides the best performance. These results also conform with the discussion on the dominant language provided in Section 2.1.
A.5.4 Distance functions for CL-ReLKT
The distance function is critical to the CL-ReLKT performance. While there exists many distance functions we can apply to the LKT process, we consider two of the most widely used ones, cosine and squared L2. As we can see, squared L2 provides the best performance.
A.5.5 Discussion
From the results, we conclude that the default settings of our proposed methods are (i) triplet loss as the training objective (ii) English question + passages as the answer representation unit (iii) English as the teacher language, and (iv) square L2 distance as the distance function for CL-ReLKT.
A.6 Language Analysis
In this section, we discuss Hindi, which our method performs worse than other competitive methods, and examines other languages that we have not shown in the tables, such as Telugu and Bengali.
As shown in Table 4, the  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$  's performance is the best in every language except Hindi. This is due to the mUSE encoder's tokenizer (sentencepiece), which cannot handle Hindi well compared to other unsupported languages (i.e., FI, RO, EL, VI). For instance, we measured the tokenizer's out-of-vocabulary (OOV) rate of the mUSE's tokenizer on Hindi (XQuAD) and found that the OOV rate of Hindi is  $\sim 14.5\%$ . On the other hand, the OOV rate on Greek on the same dataset is only  $2.2\%$  which is  $\sim 12.3\%$  lower than Hindi. Since the language families of the languages used in mUSE are not Indo-Aryan (Hindi and Bengali) nor Dravid
Component	XORQA
RU	KO	JA	FI
mUSEsmall	43.3	35.5	41.2	18.3
Training strategies
Triplet loss	+ online	45.0	35.7	42.5	23.4
+ BM25	53.6	38.6	42.5	20.8
+ BM25, + online	54.2	44.5	47.7	25.0
Contrastive loss	+ BM25, + online	52.1	39.9	42.5	21.8
Training data settings
English questions + documents	53.0	40.7	44.3	21.8
English questions + passages	54.2	44.5	47.7	25.0
multilingual questions + documents	53.1	44.3	46.9	22.6
multilingual questions + passages	50.7	40.9	42.7	20.5
Teacher's language for CL-ReLKT
mUSEcl-relkt	English as teacher	58.2	47.7	49.5	48.2
Spanish as teacher	56.4	41.4	47.7	45.8
German as teacher	56.4	41.3	48.7	43.1
Distance functions for CL-ReLKT
mUSEcl-relkt	Squared L2 distance	58.2	47.7	49.5	48.2
Cosine distance	57.3	44.7	49.0	47.8
Table 11: Comparison between training strategies, training data settings, teacher's language for CL-ReLKT, and distance functions for CL-ReLKT measured with P@1 score on XX→EN, XORQA test set.
ian (Telugu)  $^{4}$ , the mUSE's sentencepiece cannot handle Indo-Aryan and Dravidian language families well. To make the mUSE encoder handle these language families better, we might need to retrain the mUSE sentencepiece tokenizer by using other tokenizers (i.e., sentencepiece in mBERT). Since mBERT's tokenizer is trained on more than 100 languages, mBERT and XLM-R perform better than the  $\mathrm{mUSE}_{\mathrm{cl - relkt}}$  on Indo-Aryan languages. Another solution would be to use the universal tokenizer (Gillick et al., 2016) which represents the input as bytes instead of characters.
A.7 Error Analysis: MT-assisted vs Ours
We provide some sample questions that our model and the MT-assisted model (DPR+GNMT) answer differently in Table 12. We notice that our method does better than MT-assisted when the questions are very specific, such as the question relating to Brothers Grimm. In general, we find that our model generally performs better when the question contains names. This is because machine translation can sometimes fail to translate names properly.
On the other hand, when the questions are not specific, it is a toss-up whether the prediction is correct for both models. Rows 2 and 3 show examples of such vague questions. Lastly, we found that our model performed particularly worse than MT-
4 according to Ethnologue
Lan	Question	Predict context	Correct?
DE-EN
(1)	Welchem März der Gebäu der Grimm entspricht die Geschichte Diebstahl der Butter des Partners von Aarne Thompson?
(Which fairy tale by the Brothers Grimm corresponds to the story of the theft of butter from Aarne Thompson's partner?)	mUSEcl-relkt: Cat and Mouse in Partnership" (German: Katze und Maus in Gessellschaft) is a Brothers Grimm fairy tale.
It is Aarne-Thompson type 15, Stealing the Partner's Butter.	✓
DPR+GMT: Tubman and her brothers, Ben and Henry, escaped from slavery on September 17, 1849.
Tubman had been hired out to Dr. Anthony Thompson, who owned a large plantation in an area called Poplar Neck in neighboring Caroline County [...]	×
VI-EN
(1)	Edward gét loai nhac não?
(Edward hated any kind of music?)	mUSEcl-relkt: [....] Edward is musical, able to play the piano like a virtuoso. He enjoys a wide range of music, including classical, jazz, progressive metal, alternative rock, and punk rock, but dislikes country. [...]	✓
DPR+GMT: [....] Born in Woodside, Dudley, Edwards signed for Manchester United as a teenager and went on to become the youngest player to play in the Football League First Division [...]	×
DE-EN
(2)	Was war das Durchschnittseinkommen pro Person in der Stadt?
(What was the median income per person in the city?)	mUSEcl-relkt: The median income for a household in the city was $33,295, and the median income for a family was $39,250.
Males had a median income of $31,875 versus $18,594 for females.
The per capita income for the city was $14,606	×
DPR+GNMT: The median income for a household in the city was $46,795, and the median income for a family was $60,424.
Males had a median income of $41,192 versus $29,454 for females.
The per capita income for the city was $23,562.	✓
VI-EN
(2)	Sú kiên não diēn r túnhūng nám 1793 {_den} 1802?
(What events took place between 1793 and 1802?)	mUSEcl-relkt: [....] Tabinshwehti's brother-in-law, Bayinnaung, succeeded to the throne in 1550 and reigned 30 years, launching a campaign of conquest invading several states, including Manipur (1560) and Ayuthaya (1564). [...]	×
DPR+GNMT: [....] These wars were the War of the Austrian Succession (1740–1748), the Seven Years' War (1756–1763), the American Revolution (1765–1783), the French Revolutionary Wars (1793–1802) and the Napoleonic Wars (1803–1815). [...]	✓
VI-EN
(3)	Iron Manultimatephát網頁 vào nám não?
(Iron Man was released in what year?)	mUSEcl-relkt: [....] Created by Stan Lee, Larry Lieber and Jack Kirby, Ant-Man's first appearance was in Tales to Astonish #35 (September 1962). [...]	×
DPR+GMT: [....] After the successful release of Iron Man (2008) in May, the company set a July 2011 release date for The Avengers. [...]	✓
Table 12: Examples from  ${\mathrm{{mUSE}}}_{\mathrm{{cl}} - \text{relkt }}$  and DPR+GNMT with the highest question-context similarity
assisted on contents related to numbers, as shown in the last two examples. Embedding numerical information is generally hard when the data is scarce. The model in our method has to map questions from multiple languages and numbers close together. This makes learning numerical concepts such as in example number four challenging. We believe this is a good avenue for further research.