SlowGuess/NAACL_2022_OCR huggingface dataset:
Nikita Sorokin
Dmitry Abulkhanov
Moscow Institute
Irina Piontkovskaya
Valentin Malykh
Huawei Noah's Ark lab
of Physics and Technology
sorokin.na@phystech.edu, abulkhanov.dmitry@huawei.com, piontkovskaya.irina@huawei.com, valentin.malykh@huawei.com
Cross-lingual question answering is a thriving field in the modern world, helping people to search information on the web more efficiently. One of the important scenarios is to give an answer even there is no answer in the language a person asks a question with. We present a novel approach based on single encoder for query and passage for retrieval from multi-lingual collection, together with cross-lingual generative reader. It achieves a new state of the art in both retrieval and end-to-end tasks on the XOR TyDi dataset outperforming the previous results up to  $10\%$  on several languages. We find that our approach can be generalized to more than 20 languages in zero-shot approach and outperform all previous models by  $12\%$ .
Question answering (QA) is an important tool for information search on the Internet. Since it is natural for a person to ask a question to get some information, the QA systems are designed to meet this requirement. QA provides a wide range of tasks, from engineering to cornerstone scientific tasks. Open-domain QA, in this direction, is an interesting example of a problem that connects both: multilingual knowledge sources form differing knowledge and supplement gaps in each specific language. The requirement for modern language models to be cross-lingual is gradually becoming more and more important and is being incorporated into popular benchmarks such as XGLUE (Ruder et al., 2021) or XTREME (Liang et al., 2020), where the systems are evaluated not only by their performance metrics on single language tasks but the ones on many languages.
Figure 1: Overview of Sentri and a case example of answering a Finnish question using non-Finnish sources. In short, at the first step system retrieves information from factoid knowledge sources (Wikipedia's) on diverse languages. In the second step it fuses the retrieved information regardless of the language of each part, even in the absence of texts on query language. Finally, it produces an answer on the query language that aggregates all (as it can) diverse pieces from other languages.
For this particular example, there is no answer in Finnish Wikipedia. One of the reasons is that there aren't many articles in Finnish Wikipedia because Finnish is a low-resource language in general. Moreover, the answer can be found in rich-resources languages such as English or Russian for example.
Human Language Technologies, pages 395 - 406
Generally, passage retriever is based on the so called dual-encoder, i.e. two independent modules of the same architecture for the question and context encoding. However, in previous works (Qu et al., 2021; Gao and Callan, 2021) authors found out that on the one hand dual-encoder is not noise-resistant and on the other hand large batch training can improve stability of resulting embedding space. For instance, in (Qu et al., 2021; Gao and Callan, 2021) authors used  $512 \times 8$  and  $512 \times 4$  respectively.
Ouguz et al. (2021) present DPR-PAQ model combining large semi-supervised corpus for pre-training with the better and larger LM RoBERTalarge, DPR-PAQ achieves the state of the art results on Natural Questions dataset. Nevertheless, large models and large batch training require abundance of GPU memory. The mentioned models are represent dual-encoder scheme, where two independent encoders are used - one for the questions and one for the passages. To reduce memory usage, we present a system including a single encoder used for both tasks, moreover, we show that the system using single encoder can learn an embedding space better suited for transfer learning and thus improve result in cross-lingual question answering in zero-shot scenario.
The system, we call Sentri, achieves a new state of the art on XOR TyDi QA cross-lingual dataset (Asai et al., 2021a) outperforming the previous approaches by  $10\%$  on retrieval task and  $7\%$  on end-to-end question answering task. In addition, on MKQA multi-lingual dataset (Longpre et al., 2020), which contains translations of Natural Questions to different languages, in zero-shot scenario our system outperforms a strong baseline by  $8\%$ .
The overall contribution of this paper is two-fold: (i) we present a system, including single encoder for questions and contexts, that achieves state-of-the-art results in the retrieval and end-to-end tasks of the XOR TyDi dataset, (ii) we provide an analysis of the system behaviour in zero-shot scenario on unseen languages proving the its transferability and lower resource consumption.
The rest of the paper is organized as follows. Section 6 presents an overview of the recent studies on multilingual QA models. Section 2 which details the dataset design choices, outlines the data preparation pipeline and data used for evaluation.
Figure 2: Iterative training framework that adopts the idea of self-training. At first, we retrieve top- $k$  passages from Wikipedia for each question from the initial QA training set with Sentri $_j$  model. Secondly, we select the positive  $(p^+)$  passages for each question and treat the rest as hard negative  $(p^{- - })$  examples. Finally we train the new Sentri $_{j + 1}$  model closing a circle.
Section 3 presents the engineering choices and describes the resulting model and its training process. We describe experimental setup and describe the achieved results in Section 4. We provide additional results analysis in Section 5, and Section 7 concludes the paper.
In this work, we use XOR TyDi and MKQA to evaluate our system onto and several datasets to (pre-)train it.
XOR TyDi (Asai et al., 2021a) is a multilingual open-retrieval QA dataset that enables crosslingual answer retrieval. The dataset, based on questions from TyDi QA (Clark et al., 2020), articulates three new tasks that involve finding documents in different languages using multilingual and English resources. It consists of questions written by information-seeking native speakers in 7 typologically diverse languages: Arabian, Bengali, Finnish, Japanese, Korean, Russian and Telugu. Answer annotations are retrieved from multilingual document collections. XOR-Retrieve is a cross-lingual retrieval task where a question is written in the target language (e.g., Japanese), and a system is required to retrieve an English document that answers the question. XOR-English Span is a cross-lingual retrieval task where a question is written in the target language (e.g., Japanese), and a system is required to output a short answer in English. XOR-Full is a cross-lingual retrieval task where a question is written in the target language (e.g., Japanese), and a system is required to output a short answer in the target language. In our work, we concentrate on XOR-Retrieve and XOR-Full
<sup>1</sup>There exists only one Wikipedia page on Finnish related to tumbleweed but there is no mention that tumbleweed is an invasive plant in North America.
tasks and use XOR TyDi to train and evaluate our system.
2.1 Training Dataset Pre-processing
Since there are low-resource languages in the XOR TyDi dataset, the task of preparing data is of priority importance. To generate data in different languages, we used NQ and Trivia QA datasets described above. Both NQ and Trivia QA are English language datasets. To use them for training purposes in our setup we had to translate them to the languages of XOR TyDi. The quality of machine translation still does not match the human translation quality in most of the languages and domains. However, questions and answers in NQ and Trivia QA datasets are short and easy to translate.
For each pair of question  $q$  and answer  $a$  from the datasets, we made translations from English to each language  $L_{i}$  using the M2M100 model described in (Fan et al., 2020), it is a state-of-the-art model for translation for many languages, including the ones we are interested in. Thus we translated question-answer pairs and we got an aligned dataset. The pairs are not enough, since the open-domain QA task is based on so-called support passages retrieved from some document collection. To overcome this issue we mined the positive and hard negative sample passages for each language  $L_{i}$ . We consider a paragraph from a document to be a positive sample if it is ranked high by a retriever model and includes the answer. We use complicated morphology-aware answer detection technique which we describe in Appendix. If a paragraph is highly ranked but contains no answer, we consider it as a hard negative sample. The resulting statistics and analysis of the training dataset we also present in Appendix.
Information Retrieval We took into account that most XOR TyDi languages have complex morphology and other linguistic features, which makes information retrieval less effective for the models using token comparison. Thus we decided to normalize the morphology for at least the languages, which has publicly available stemmers, namely, Arabic $^{1}$ , Bengali $^{2}$ , Korean $^{3}$ , and Russian $^{1}$ . The Telugu language has no publicly available stemmer,
but there is a lemmatizer, which we used.
For Korean we apply token splitting by the part-of-speech tag, i.e. modifier POS as a Josa and Eomi are treated as separate tokens. Unfortunately, we have not found accessible stemmers and/or lemmatizers for Japanese and Finnish languages. We use this normalisation to improve positive passage mining for self-training procedure. More details on normalisation could be found in Appendix.
Natural Questions (NQ) (Kwiatkowski et al., 2019) dataset is designed for end-to-end question answering. The questions are mined from real Google search queries and the answers are spans in Wikipedia articles identified by annotators. We use this dataset in two ways. One way is for training and another one is for zero-shot evaluation. The latter option is provided to us by MKQA dataset (Longpre et al., 2020). It is a translation of 10 thousand question-answer pairs from NQ to 26 different languages, thus giving us an aligned dataset of 260 thousand question-answer pairs total. The former option is described below.
We also use Trivia QA for pre-training of our model. Joshi et al. (2017) presented Trivia QA, a large-scale question-answering dataset that includes so-called evidence documents, allowing one to state a task of information retrieval. Trivia QA includes 95 thousand question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average ending with 650 thousand total triples.
The open domain question answering task heavily relies on retrieval from some (possibly more than one) document collections. In the case of the cross-lingual variant of this task, the usage of several (at least two - in English and in a target language) document collections is almost inevitable. We evaluate our model in two cross-lingual setups: using English Wikipedia ( $W_{eng}$ ) to search for a relevant passage containing the answer to the question or using collection of multilingual reference passages from Arabic, Russian, English, Finnish, Telugu, Bengali, Japanese, Korean Wikipedia ( $W_{multi}$ ). More formally, given a question  $q$  in language  $L_i$ , a system retrieves the documents from  $W_{eng}$  or  $W_{multi}$ , and formulates an answer  $a$ . Thus the system could be virtually split to retriever, which creates a list of
4https://bitbucket.org/sivareddyg/ telugu-part-of-speech-tags/src/master/
1https://pypi.org/project/nltk/ 2https://github.com/MIProtick/ Bangla-stemmer.git 3https://pypi.org/project/ mecab-python3/
relevant documents, and reader, which generates an answer using the most relevant documents. The sample of the system output is presented on Fig. 1.
3.1 Single Encoder Retriever
We follow common (Qu et al., 2021; Asai et al., 2021b; Ouguz et al., 2021) dual-encoder approach in data representation. The system consists of question encoder  $E_{q}(\cdot)$  and passage encoder  $E_{p}(\cdot)$  which maps text to  $d$ -dimensional real-valued vectors. Before run-time,  $E_{p}(\cdot)$  applied to all passages in knowledge source to create search index. To find out relevant passages to certain question system operates a similarity function:
$$
\operatorname {s i m} (q, p) = E _ {q} (q) ^ {\intercal} \cdot E _ {p} (p). \tag {1}
$$
i.e. similarity between the question and the passage defined by the dot product of their vectors.
In this work we investigate case when  $E_{q}(\cdot) = E_{p}(\cdot)$  and call this approach as Single Encoder. In addition to it, a model with  $E_{q}(\cdot) \neq E_{p}(\cdot)$  we call Bi-Encoder to avoid confusion.
The architecture that utilizes Single encoder approach for retrieval (Sentri) shares one encoder for  $E_{q}(\cdot)$  and  $E_{p}(\cdot)$  contrary to bi-encoder which based on two separate models.
Since our model is used in a multi-lingual setting, the choice of multilingual models is natural for base model. We use XLM-RoBERTa (Conneau et al., 2020) (large) in our experiments.
Figure 3: An example of overlap in positive passages  $(p^{+})$  for different instances of a question.
Training Sentri is trained to give positive passages higher scores than negative passages. More specifically, given a question  $q_{i}$  in a language from  $L$  together with its positive passage  $p_{i}^{+}$  and  $m$  negative passages  $\{p_{i,j}^{-}\}_{j=1}^{m}$  sampled from  $W_{multi}$ , we minimize the loss function:
$$
\begin{array}{l} \mathcal {L} \left(q _ {i}, p _ {i} ^ {+}, \left\{p _ {i, j} ^ {-} \right\} _ {j = 1} ^ {m}\right) \\ = - \log \frac {e ^ {\operatorname {s i m} \left(q _ {i} , p _ {i} ^ {+}\right)}}{\sum_ {j = 1} ^ {m} e ^ {\operatorname {s i m} \left(q _ {i} , p _ {i , j} ^ {-}\right)} + e ^ {\operatorname {s i m} \left(q _ {i} , p _ {i} ^ {+}\right)}}, \tag {2} \\ \end{array}
$$
where we aim to optimize the negative log-likelihood of the positive passage against a set of  $m$  negative passages.
For each question, we treat other passages in the training batch that do not answer this particular question as negative passages (in-batch negative trick, Henderson et al. 2017; Karpukhin et al. 2020). In particular, for batch size  $n$  each question can be further paired with  $m = n - 1 + n$  negatives (i.e., positive and hard negative passages of the rest questions) without sampling additional negatives. Furthermore, in the case of multilingual data, it helps enforce the cross-lingual ability of the model because of an increasing number of cross-language pairs.
3.2 In-batch False Negatives Filtering
Although the above strategy can increase the number of negatives, some of them may turn out to be false negatives. We analyze the batches generated for the training and found out that different questions in the same batch could have the same positive passages. Since these positive passages are used for in-batch negative training that produces false negative pairs. For English Wikipedia this overlap is significant but not so crucial like for lower resource Wikipedias. For instance, for Natural Questions passages in  $44\%$  training triplets (questions, answer, passage) are used more than once in the dataset. The sample of positive passage overlap for NQ is presented on Fig. 3. We use in-batch filtering allowing us to eliminate this overlap from generated batches and thus improve the overall system quality.
3.3 Self-Training
Several works (Qu et al., 2021; Izacard and Grave, 2020a) refer to iterative learning as a source of model quality improvement. We use this idea in the form described below, which we call self-training. Fig. 2 presents the framework which we use in this work. Sentri  $j$  model retrieves top-  $k$  passages from Wikipedia for each question from the initial QA training set. Then we select the positive  $(p^{+})$  passages for each question (we know the ones for the initial training set) and treat the rest as hard
R@2kt	Avg	R@5kt
Ar	Bn	Fi	Ja	Ko	Ru	Te	Ar	Bn	Fi	Ja	Ko	Ru	Te	Avg
Dev Set
DPR + BM25 + MT	43.4	53.9	55.1	40.2	50.5	30.8	20.2	42.0	52.4	62.8	61.8	48.1	58.6	37.8	32.4	50.6
CORA (Asai et al., 2021b)	32.0	42.8	39.5	24.9	33.3	31.2	30.7	33.5	42.7	52.0	49.0	32.8	43.5	39.2	41.6	43.0
Bi-Encoder1	47.0	38.8	40.7	49.7	30.2	42.1	34.1	40.4	54.0	41.5	49.5	56.0	40.2	52.6	43.7	48.2
Bi-Encoder2	47.8	39.1	48.9	51.2	40.2	41.2	49.4	45.4	55.1	43.3	59.5	59.4	51.2	52.0	56.9	53.9
Sentri	47.6	48.1	53.1	46.6	49.6	44.3	67.9	51.0	56.8	62.2	65.5	53.2	55.5	52.3	80.3	60.8
Test Set
DPR + BM25 + MT	48.3	54.4	56.7	41.8	39.4	39.6	18.7	42.7	52.5	63.2	65.9	52.1	46.5	47.3	22.7	50.0
GAAMA (Ferritto et al., 2020)	-	-	-	-	-	-	-	52.8	-	-	-	-	-	-	-	59.9
Sentri	53.8	66.7	55.4	42.9	46.8	55.1	48.7	52.8	63.0	72.4	63.5	53.1	56.9	61.8	56.4	61.0
Table 1: Performance on XOR-Retrieve task. The best result is given in bold, the second best is in underlined italic. We note that at extremely low-resource languages such as Finnish and Telugu Sentri shows consistent performance, similar to results on moderate-resource languages such as Russian.
negative  $(p^{- - })$  examples. Afterwards, we train new iteration Sentri  $j + 1$  model using the passages marked up previously. In contrast to (Asai et al., 2021b) we train iteratively only a retrieval part of the whole system.
At stage 0 when there is no trained model available, we use well-known BM25 model (Sanderson, 2010) as a retriever in our experiments. The important feature of this model is that it does not need any kind of training, thus it could be used to retrieve documents from collections in languages with little or no training data.
For Sentri system we report the results for the second stage of self-training below. For BiEncoder we report results for stages 1 and 2 adding the specifying index.
3.4 Answer Generation
We have experimented with both extractive and abstractive answer generation and found out that abstractive is more profitable. Here we describe the abstractive reader approach we use as primary one. We decided to use the FiD model (Izacard and Grave, 2020a) as a reader model in Sentri for end to end question answering task XOR-Full since it allows us to exclude the translator from a pipeline and to aggregate information in a cross-lingual setup. Since the original FiD model is monolingual, we present extension of this work, multilingual version which we call MFiD. To train MFiD, we use several QA datasets, listed in Sec. 2, namely XOR-Full, XOR TyDi, Natural Questions, and Trivia QA datasets, the same ones used for training the retriever part of the Sentri model. For each question from the QA datasets we retrieve top-50 passages from multi-language knowledge source using our retriever model. And then use it for training
MFiD as cross-lingual fusion reader. We also experimented with standard extractive reader. The details on extractive approach could be found in Sec. 5.
We have conducted a series of experiments with number of models, namely these are Sentri model combined with different reader parts and BiEncoder model with one or two stages of selftraining. Bi-Encoder model is using standard extractive reader (plus machine translation where applicable). The main difference between Sentri and Bi-Encoder, that the latter is based on classic dual-encoder architecture, while the former is using single encoder for questions and paragraphs.
4.1 Results on XOR TyDi
Tables 1 and 2 contain results for our system in retrieval and end-to-end setups, XOR-Retieve and XOR-Full respectively. These two tables contain the results of the models' evaluation on the development and test parts of the XOR TyDi dataset. It is important to mention that we use name Sentri for our model in both tasks, while in XOR-Retieve task the reader part is not used, since it is essentially a passage ranking evaluation. Also, you can find results for our models titled as  $\mathrm{Bi - Encoder}_1$  and  $\mathrm{Bi - Encoder}_2$  (for first and stages of self-training respectively). As one can see Sentri significantly outperforms these baseline models and existing state-of-the-art models. We provide more detailed analysis in section Ablation Study.
Tab. 1 displays recall scores for 2000 and 5000 first tokens (R@2kt and R@5kt respectively). That means that we expect to find an answer span in the first  $l$  tokens. This metric was proposed
Target Language Li, F1	Macro Average
Ar	Bn	Fi	Ja	Ko	Ru	Te	F1	EM	BLEU
Dev Set
DPR + BM25 + MT	9.2	15.8	14.4	4.8	7.9	5.2	0.5	8.3	4.6	7.5
CORA (Asai et al., 2021b)	42.9	26.9	41.4	36.8	30.4	33.8	30.9	34.7	25.8	23.3
Bi-Encoder1	15.0	8.7	11.5	6.2	7.5	8.5	9.1	9.5	5.7	10.5
Bi-Encoder2	18.9	11.2	21.1	3.9	10.6	8.1	13.9	12.5	7.7	13.5
Sentri + ext. reader + MT	20.8	14.5	21.3	10.7	16.1	12.1	17.3	16.1	10.1	16.5
Sentri + MFiD	52.5	31.2	45.5	44.9	43.1	41.2	30.7	41.3	34.9	30.7
Table 2: End-to-end performance on XOR-Full task. Our best model, Sentri + MFiD, at large margin outperforms existing systems.
R@2kt
Da	De	Es	Fr	He	Hu	It	Km	Ms	Nl	No
CORA (Asai et al., 2021b)	44.5	44.6	45.3	44.8	27.3	39.1	44.2	22.2	44.3	47.3	48.3
BM25 + MT*	44.1	43.3	44.9	42.5	36.9	39.3	40.1	31.3	42.5	46.5	43.3
Bi-Encoder2	50.0	47.8	48.7	47.4	37.7	43.4	41.8	37.8	49.5	47.3	49.1
Sentri	57.6	56.5	55.9	55.1	47.9	51.8	54.3	43.9	56.0	56.3	56.5
Pl	Pt	Sv	Th	Tr	Vi	Zh-cn	Zh-hk	Zh-tw	Avg
CORA (Asai et al., 2021b)	44.8	40.8	43.6	45.0	34.8	33.9	33.5	41.5	41.0	41.1
BM25 + MT*	46.5	45.7	49.7	46.5	42.5	43.5	37.5	37.5	36.1	42.0
Bi-Encoder2	47.0	47.7	50.0	46.5	45.6	47.3	42.6	41.5	41.0	45.3
Sentri	55.8	54.8	56.9	55.3	53.0	54.4	50.2	50.7	49.4	53.3
Table 3: Zero-shot cross-lingual retrieval results on MKQA dataset.
in (Asai et al., 2021a) as alternative to more common  $Recall@N$  in purpose to make more fair comparison across various models with different passage size used.
Our system outperforms the previous state-of-the-art system in both R@2kt and R@5kt metrics by a wide margin on four languages, namely Arabic, Japanese, Russian, and Telugu. More importantly, our system outperforms the previous system on average for all the languages. Interestingly, selftraining improves the results in all the languages, with the intriguing exception of the Russian language. This fact requires an additional investigation, we leave it as future work for now.
Table 2 displays F1, Exact Match (EM), and BLEU scores for the end-to-end setup where given a question in target language  $L_{i}$  and Wikipedia in both English and  $L_{i}$ , a system is required to generate an answer in the target language. F1 measure is computed per token for an answer span. Exact Match compares the golden answer span with the system output for exact equality. BLEU metric, defined as in (Papineni et al., 2002), computes
the number of overlapping  $n$ -gram between the golden answer and the system output. In this experiment, we see a somewhat different behaviour of the model. Our model outperform previous state of the art system for all languages, with exception for Telugu where CORA model (Asai et al., 2021b) shows insignificantly higher score.
4.2 Zero-Shot Cross-Linguual Transfer
We investigated the transferability across the languages for the trained system. We used the MKQA dataset in similar to the XOR-Retieve setup, i.e. we retrieved the passages from English Wikipedia, extracted the answer from the top-ranked passage, and translated it with a machine translation model. Here we again use M2M100 model for machine translation task. As a baseline for this task, we utilized BM25 with extractive reader and the machine translation model at the end of pipeline. We selected from MKQA such unseen languages that were not presented to the system during the training process. The achieved results presented in Tab. 3 show that even in such a zero-shot setting our sys
tem significantly outperforms both the strong baseline and previous approaches in all languages. Additional details on zero-shot transfer could be found in Appendix.
Sentri model has five important features which differentiate it from the previous work: self-training, a single encoder model for passage and question processing as a retriever, a generative model as a reader, in-batch negative filtering, usage of the machine translated data during the training process.