SlowGuess/NAACL_2022_OCR huggingface dataset:
(I) The effect of the first of mentioned features could be analysed basing on Tab. 1 (upper part of the table, showing results on development set) and 2. Self-training for Bi-Encoder model improves results by  $3\%$  on average for the XOR-Full task and about  $5\%$  for the XOR-Retrieve task (BiEncoder $_1$  vs Bi-Encoder $_2$ ).
(II) The usage of a single encoder could be estimated as again  $3\%$  for the XOR-Full task and about  $5 - 6\%$  for the XOR-Retrieve task (Senti + ext. reader vs. Bi-Encoder2). In Tab. 4 we demonstrate key motivation of using shared encoder model. We observe that the single encoder approach superior to the Bi-Encoder in terms of memory efficiency and overall performance. With the same size (less than  $2\%$  difference) it achieves more than  $12\%$  relative improvement or  $\sim 6 - 7$  difference in absolute points in retrieval task. Note that Sentri and BiEncoder2 trained in the same setting and same QA datasets except used base model variants (base and large respectively). We do it for the sake of the matching number of parameters, matched memory consumption and similar training time of both
Base architecture	#params overall	XOR
Eq	Ep	R@2kt	R@5kt
Bi-Encoder2	XLM-Rbase	XLM-Rbase	540 M	45.4	53.9
Sentri	XLM-Rlarge (shared)	550 M	51.0	60.8
Table 4: Comparison of architectures of single and bi-encoder models. The single encoder approach significantly outperform dual-encoder one using almost the same memory amount.
Most-effective top-k	Macro Average
F1	EM	BLEU
Sentri + ext. reader + MT	5	16.1	10.1	16.5
Sentri + CORA Reader	15	30.7	-	-
Sentri + MFiD	100	41.3	34.9	30.7
Table 5: Comparison of different reader models.
(III) The replacement of standard extractive reader, i.e. span-tagging model, with a generative one, MFiD model in our case, turned out to add up to  $34\%$  of F1 measure (for Japanese) and  $25\%$  on average. We observe that with the number of contexts more than 5 performance of extractive reader degrades. Contrary that using more contexts for answer generation can significantly improve model quality. We further evaluate generative reader usage by adding the one described in (Asai et al., 2021b), it uses only 15 top-ranked contexts due to memory constraints. Unlike that, MFiD can use up to 100 top-ranked contexts thanks to the independent processing of passages in the reader's encoder. The results are presented in Tab. 5.
(IV & V) The importance of the last two features could be estimated basing on Tab. 6. While the former one adds up to 2 per cent, the latter is of crucial importance adding up to  $20\%$  depending on task and measure.
We could conclude that all the features are important for our approach to present state-of-the-art results in retrieval and end-to-end tasks.
5.1 Languages Used
It is also interesting to know if the system is actually using the data in other languages for answer generation. In other words, if our model is truly cross-lingual. The analysis of the top-100 paragraphs for all the questions in validation set of XOR TyDi, namely the breakdown on the languages used, is shown on Tab. 7. As one can see, there are several interesting features could be spotted in the table. The highest self-usage (i.e. when a question and a paragraph are in the same language) percentage is shown by Korean language (almost entirely,  $97.6\%$ ), with negligible usage of other languages, except English with  $1.4\%$ . On the contrary, Finnish language is shown the lowest self-usage of  $52.9\%$  with  $34.6\%$  usage of English. These two facts could be explained as Korean having unique writing system, thus it has almost nonexistent intersection with other languages in terms. In the contrary Finnish is Latin-based and is in close contact with Swedish language since the Middle Ages, while Swedish is close German language for English. But this speculation has its downside: the second most-used language for Finnish is Japanese  $(7.4\%)$ , which is both unrelated and uses other script. Interestingly, Japanese language is the
R@2kt	R@5kt
MKQA	XOR	MKQA	XOR
Sentri	53.3	51.0	60.3	60.8
Sentri w/o false negative filtering	52.1	49.3	60.1	60.3
Sentri w/o weight-sharing (Bi-Encoder approach)	45.3	45.4	52.9	53.9
Sentri w/o multi-language translations of training set	41.5	30.8	45.8	42.3
Table 6: Ablation experiments on MKQA and XOR development sets.
second language by usage for almost all the languages, including Korean but with only  $0.7\%$ . For Telugu the second most used language is Bengali, which is related to it. But surprisingly, the other way around it is almost unused. Another peculiar feature is that Russian language is significant in usage for almost all the languages with exception for Korean. We hypothesise that it is due to the proportion of Russian data in the training set, this language being the second by size in the dataset. It is important to mention that the largest present language is Arabic, but its influence is lower than Russian. The influence of Arabic, Russian, and Japanese need more in-depth future investigation.
Datasets The cross-lingual question answering datasets were scarce before recent years. Fortunately, these years left us with several publicly available datasets. Lewis et al. (2020) introduced MLQA dataset. It consists of parallel QA pairs in several languages. Liu et al. (2019) have presented XQA dataset, with training set in English and validation and test sets in the other languages. Cross-lingual Question Answering Dataset (XQuAD) benchmark presented in Artetxe et al. (2020). It consists of a subset of 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 (Rajpurkar et al., 2016) together with their translations into ten languages.
Systems Open-domain question answering task assumes answering factoid questions without a predefined domain (Kwiatkowski et al., 2019). Recent research was focused on creating non-English question answering datasets and applying crosslingual transfer learning techniques, from English to other languages. Until recently, the availability of appropriate train and test datasets has been a key factor in the development of the field: however, in recent years, many works have focused on the collection of loosely aligned data obtained
through automatic translation or by parsing similar multilingual sources. Artetxe et al. (2020) studied cross-lingual transferability of monolingual representations of a transformer-based masked language model.
In most previous approaches the authors use extractive models to generate the actual answer. This could be explained by the mental inertia from SQuAD-like datasets. By SQuAD-like we mean a dataset where labelled data includes an explicitly stated question, a passage, containing an answer, and a span markup for the answer. Such markup was presented for the question answering task called SQuAD in (Rajpurkar et al., 2016). But recently there were presented cross-lingual generation of answers from raw texts. Kumar et al. (2019); Chi et al. (2019) studied cross-lingual question generation. Shakeri et al. (2020) proposed a method to generate multilingual question and answer pairs by a generative model (namely, a fine-tuned multilingual T5 model), it is based on automatically translated samples from English to the target domain. Generative question answering was mostly considered in previous work for long answers datasets. However, FiD model (Izacard and Grave, 2020b) archives competitive results on SQuAD-like datasets, where an answer is supposed to be short text span. For open domain question answering, one of the first approaches named RAG used generative models was presented in (Lewis et al., 2021). A key idea of this RAG model is to process several (top k) passages from the retriever in the encoder simultaneously. The produced dense representations of the passages are used in the decoder for the answer generation, this process is called fusion. Processing the passages independently in the encoder allows a model to scale to many contexts, as it only runs self-attention over one context at a time. FiD model follows this paradigm further improving the results in question generation.
Ar	Bn	Fi	Ja	Ko	Ru	Te	En
Ar	80.6	0.2	0.1	7.1	0.2	1.2	0.0	10.5
Bn	0.3	89.8	0.1	4.8	0.2	0.8	0.1	3.7
Fi	0.8	1.4	52.9	7.4	0.3	2.1	0.1	34.6
Ja	0.8	3.7	0.2	77.5	1.1	1.7	0.2	14.5
Ko	0.0	0.0	0.0	0.7	97.6	0.1	0.0	1.4
Ru	0.9	0.8	0.3	8.7	0.3	74.5	0.0	14.2
Te	0.2	8.3	0.0	3.8	0.4	0.4	75.5	11.2
Table 7: Breakdown on the languages that the Sentri+MFiD model uses for answer generation.
Nowadays multi-lingual and cross-lingual problems are coming to the stage once the natural language models become more and more powerful. One of these problems is that where the systems answer the questions using various mutually disjoint language data, as it stated in XOR TyDi task. This task is based on a specific XOR TyDi dataset (Asai et al., 2021a), which ensured such information asymmetry in the different language data. We introduced the cross-lingual system to solve the XOR task. While the XOR TyDi is a challenging test that stimulates cross-linguality in NLP systems, we have outperformed the existing models in two subtasks: XOR-Retrieve and XOR-Full without using external APIs. The first task is a classical passage retrieval task, while the second one is an end-to-end question-answering task. Besides showing the state of the art results on these two subtasks, our system is demonstrated the ability the transfer to the unseen languages in retrieval task, including the languages which were not presented in the pre-trained language model we use as an encoder for the retriever part of our Sentri model. And last we found that the previous works ignored the existence of the morphology in XOR TyDi presented languages, thus missing many results in information retrieval. We propose to solve this issue by using stemming or lemmatization for such languages.
Our system has five differentiating features, which are self-training (using the output of the previously trained models), single encoder (allowing us to reduce the number of parameters about twice in retriever), usage of a generative model to get the question from retrieved passages, in-batch negative filtering, and usage of the machine translated data during the training process. All of these features are proved to make a share in the achieved significant quality improvement demonstrated by our
model. Although, our system has several flaws, e.g. passage selection strategy and stemming for the languages, we consider these flaws as our future work. But we hope that current study will foster research in cross-lingual question answering tasks.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021a. Xorqa: Cross-lingual open-retrieval question answering. Proceedings of NAACL-HLT'2021.
Akari Asai, Xinyan Yu, Jungo Kasai, and Hannaneh Hajishirzi. 2021b. One question answering model for many languages with cross-lingual dense passage retrieval. Proceedings of NeurIPS 2021.
Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, XianLing Mao, and Heyan Huang. 2019. Cross-lingual natural language generation via pre-training.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in ty po- logically di verse languages. Transactions of the Association for Computational Linguistics, 8:454-470.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandep Baines, Onur Celebi, Guillaume Wenzek,
Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multilingual machine translation. CoRR, abs/2010.11125.
Anthony Ferritto, Sara Rosenthal, Mihaela Bornea, Kazi Hasan, Rishav Chakravarti, Salim Roukos, Radu Florian, and Avirup Sil. 2020. A multilingual reading comprehension system for more than 100 languages. In Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations, pages 41-47.
Luyu Gao and Jamie Callan. 2021. Unsupervised corpus aware language model pre-training for dense passage retrieval. ArXiv, abs/2108.05540.
Matthew L. Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. CoRR, abs/1705.00652.
Gautier Izacard and Edouard Grave. 2020a. Distilling knowledge from reader to retriever for question answering.
Gautier Izacard and Edouard Grave. 2020b. Leveraging passage retrieval with generative models for open domain question answering.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. 2019. Cross-lingual training for automatic question generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4863–4872. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Patrick Lewis, Barlas Oguz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2020. *Mlqa: Evaluating cross-lingual extractive question answering*.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Roektaschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv, abs/2004.01401.
Jiahua Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2019. Xqa: A cross-lingual open-domain question answering dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2358–2368.
Shayne Longpre, Yi Lu, and Joachim Daiber. 2020. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. TACL.
Barlas Ouguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen tau Yih, Sonal Gupta, and Yashar Mehdad. 2021. Domain-matched pre-training tasks for dense retrieval. ArXiv, abs/2107.13602.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. ArXiv, abs/2010.08191.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. 2021. Xtreme: Towards more challenging and nuanced multilingual evaluation. arXiv preprint arXiv:2104.07412.
Mark Sanderson. 2010. Christopher d. manning, prabhakar raghavan, hinrich schütze, introduction to information retrieval, cambridge university press. 2008. ISBN-13 978-0-521-86571-5, xx+ 482 pages. Natural Language Engineering, 16(1):100-103.
Siamak Shakeri, Noah Constant, Mihir Sanjay Kale, and Linting Xue. 2020. Multilingual synthetic question and answer generation for cross-lingual reading comprehension. arXiv preprint arXiv:2010.12008v1.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
For our system, we have adapted models from the Huggingface Transformers library (Wolf et al., 2019). We trained the question and passage encoders using the in-batch negative sampling with a batch size of 16, one hard negative per question. We trained the system for 40 epochs with a learning rate of  $10^{-5}$  using Adam, linear scheduling with warm-up and dropout rate of 0.1. For training and validation, we used NQ, Trivia, XOR and TyDi QA datasets. The number of hard negative passages was 32 and 50 for the first and second stages respectively. All experiments were carried out on four NVIDIA V100 GPUs (with 32Gb RAM each).
B Data Distribution
The overall statistics on mined data from the mentioned datasets are available in Tab. 8. As one can see, the data acquired is following the rough pattern: most of the samples come from TriviaQA, the second source is NQ, while the rest comes from XOR TyDi QA. In the perspective of languages the pattern follows the XOR TyDi distribution, with Arabic being the largest non-English language, and Korean being the smallest one.
NQ	TriviaQA	XOR QA	Total
Ar	16420	38652	10496	65568
Bn	10165	25178	1973	37316
Fi	10787	23145	4688	38620
Ja	15357	26877	2869	45103
Ko	1327	2247	721	4295
Ru	18499	35081	3981	57561
Te	4964	12880	1481	19325
C Effects of Normalisation
Since we are using normalisation for the retrieval, we decided to look into the evaluation process. The metrics used for the evaluation, namely pentoken F1, Exact Match, and BLEU, are based on simple token comparison. Such comparison is inefficient for the languages with rich morphology, like Russian or Japanese. So we applied the same normalisation as in retrieval for the generated and gold answers. Table 9 shows the achieved results. As one can see normalisation helps to achieve less strict and thus more informative comparison for the morphology-rich languages. Since Japanese and Finnish are both synthetic agglutinative languages, we suppose that the results on them could also be improved with usage of the stemming/lemmatization, thus improving the crosslingual average further. Given that we think the usage of some kind of normalisation should be recommended for any cross-lingual QA task including morphology-rich languages.
Table 8: The number of mined samples from training part of Natural Questions, TriviaQA, and XOR TyDi QA datasets
w/o stemming	with stemming
F1	EM	BLEU	F1	EM	BLEU
Ar	18.9	12.2	19.7	21.2	14.6	18.0
Bn	11.2	6.1	12.9	11.7	6.6	13.1
Fi	21.1	14.8	22.9	21.1	14.8	22.9
Ja	3.9	0.9	4.0	3.9	0.9	4.0
Ko	10.6	6.7	7.5	11.8	8.1	8.3
Ru	8.1	4.1	11.0	14.6	7.4	18.1
Te	13.9	8.8	16.7	14.2	8.9	15.3
Avg	12.5	7.7	13.5	14.1	8.8	14.2
Table 9: Results for Sentri model with and without normalisation.
D Zero-Shot Cross-Linguual Transfer
In addition to the MKQA results on which were presented in main contents, we used M2M100 model to translate the MKQA English subset to all known to it languages, thus extending MKQA to 98 languages. We provide the results on the this extended dataset in Tab. 10.
E Unseen Languages
We thoroughly analysed the (partially) unseen languages and found out that our system performs reasonably well even for those languages, which are not present in the pre-trained XLM Roberta used as an encoder in our model. The results are presented
in Tab. 11. We also provide aggregated results for the unseen on training stage languages and seen in the training stage ones. As one could see, our model performs surprisingly well on the languages which were not presented on any training stage, although the training improves Recall by 12 and 13 per cent on average, while pre-training adds  $13.8\%$  of Recall. We think this is another evidence of the great generalizability of the pre-trained language models.
Query language	Metric	Query language	Metric
R@2kt	R@5kt	R@2kt	R@5kt
Af	47.9	56.4	Lv	44.1	52.6
Sw	35.8	44.2	Bs	46.0	54.5
Is	44.1	52.3	Ps	35.9	44.0
Tl	44.3	54.2	Hu	51.8	59.8
Id	49.3	56.5	Lt	42.8	51.1
Ff	9.9	15.9	Ln	9.2	14.6
Sd	33.0	41.1	Gl	45.5	53.4
Bg	47.5	55.6	Pa	25.1	31.2
Ast	36.5	44.4	Sk	49.0	56.8
Ro	48.7	56.4	Oc	35.8	44.5
Yo	18.4	25.3	Lb	35.8	39.1
Ig	23.9	32.2	Br	10.3	14.1
Et	43.8	52.6	Jv	39.6	48.1
Sr	45.4	53.8	Gd	21.8	29.6
Hr	47.2	55.4	Sq	43.6	51.9
Or	6.7	11.1	Ml	40.0	47.7
Tn	37.2	19.6	Yi	32.7	40.7
Bn	13.5	49.2	Ss	21.0	27.6
Kn	25.2	31.4	Ba	21.3	27.0
Fa	44.0	51.9	Az	40.0	47.8
Fy	41.1	49.2	Ca	45.0	53.2
Hi	45.1	53.6	Lg	20.0	26.3
Ilo	39.0	46.0	Cy	33.9	41.9
El	46.3	53.9	So	26.9	34.9
Mr	38.8	47.3	Ne	14.6	19.9
Mg	21.4	36.5	Ceb	28.5	36.6
Ha	28.6	40.0	Cs	48.5	56.9
Gu	31.1	44.3	Ht	19.1	25.6
Tn	41.4	19.6	Ns	14.0	19.6
Bn	30.9	49.2	mn	36.2	43.2
Kn	25.2	31.4	Xh	28.7	36.8
Fa	44.0	51.9	Mk	46.7	54.6
Fy	41.1	49.2	Be	41.5	49.9
Hi	45.1	53.6	Ga	9.6	14.3
ILo	46.0	46.0	Si	41.5	49.0
El	46.3	53.9	Su	37.0	45.1
Mr	38.8	47.3	Uz	27.4	34.5
Kk	38.8	28.2	Am	27.3	34.7
Mg	21.4	36.5	Wo	14.0	19.2
Ha	28.6	40.0	Ta	40.0	47.5
Ur	45.5	49.5	Ka	40.5	48.5
Pa	25.1	31.2	Hy	36.9	44.0
Table 10: Results on MKQA dataset translated to 98 languages.
Setting	Language	Script	R@2kt	R@5kt
Unseen at training and pre-training	Bashkir	Cyrillic	21.3	27.0
Armenian	Armenian	36.9	44.0
Haitian	Latin	19.1	25.6
Cebuano	Latin	28.5	36.6
Lao	Thai	39.0	46.0
Occitan	Latin	35.8	44.5
Luxembourgish	Latin	35.8	39.1
Yiddish	Hebrew	32.7	40.7
Fulah	Latin	9.9	15.9
Igbo	Latin	23.9	32.2
Ganda	Latin	20.0	26.3
Lingala	Latin	9.2	14.6
Swati	Latin	21.0	27.6
Tswana	Latin	13.5	19.6
Wolof	Latin	14.0	19.2
Yoruba	Latin	18.4	25.3
Zulu	Latin	30.9	38.6
Unseen at training and pre-training	Avg	—	24.1	30.8
Unseen at training	Avg	—	36.4	44.0
Seen	Avg	—	50.2	57.8
Table 11: Recall of Sentri model on machine-translated MKQA dataset.