SlowGuess/NAACL_2021_OCR huggingface dataset:
Akari Asai\*, Jungo Kasai\*, Jonathan H. Clark\*, Kenton Lee\*, Eunsol Choi\*, Hannaneh Hajishirzi\*
\*University of Washington \*Google Research
The University of Texas at Austin Allen Institute for AI
{akari, jkasai, hannaneh}@cs.washington.edu
{jhclark, kentonl}@google.com, eunsol@cs.utexas.edu
Multilingual question answering tasks typically assume that answers exist in the same language as the question. Yet in practice, many languages face both information scarcity—where languages have few reference articles—and information asymmetry—where questions reference concepts from other cultures. This work extends open-retrieval question answering to a cross-lingual setting enabling questions from one language to be answered via answer content from another language. We construct a large-scale dataset built on 40K information-seeking questions across 7 diverse non-English languages that TYDI QA could not find same-language answers for. Based on this dataset, we introduce a task framework, called Cross-lingual Open-Retrieval Question Answering (XOR QA), that consists of three new tasks involving cross-lingual document retrieval from multilingual and English resources. We establish baselines with state-of-the-art machine translation systems and cross-lingual pretrained models. Experimental results suggest that XOR QA is a challenging task that will facilitate the development of novel techniques for multilingual question answering. Our data and code are available at https://nlp.cs.washington.edu/xorqa/.
Information-seeking questions—questions from people who are actually looking for an answer—have been increasingly studied in question answering (QA) research. Fulfilling these information needs has led the research community to look further for answers: beyond paragraphs and articles toward performing open retrieval<sup>1</sup> on large-scale document collections (Chen and Yih, 2020). Yet
Figure 1: Overview of XOR QA. Given a question in  $L_{i}$ , the model finds an answer in either English or  $L_{i}$  Wikipedia and returns an answer in English or  $L_{i}$ .  $L_{i}$  is one of the 7 typologically diverse languages.
the bulk of this work has been exclusively on English. In this paper, we bring together for the first time information-seeking questions, open-retrieval QA, and multilingual QA to create a multilingual open-retrieval QA dataset that enables cross-lingual answer retrieval.
While multilingual open QA systems would benefit the many speakers of non-English languages, there are several pitfalls in designing such a dataset. First, a multilingual QA dataset should include questions from non-English native speakers to represent real-world applications. Questions in most recent multilingual QA datasets (Lewis et al., 2020; Artetxe et al., 2020; Longpre et al., 2020) are translated from English, which leads to English-centric questions such as questions about American sports, cultures and politics. Second, it is important to support retrieving answers in languages other than the original language due to information scarcity of low-resource languages (Miniawatts Marketing Group, 2011). Moreover, questions strongly related to entities from other cultures are less likely to have answer content in the questioner's language
<sup>1</sup>We use open retrieval—instead of open domain—to refer to models that can access answer context from large document collections. We avoid using open domain due to its double meaning as “covering topics from many domains.”
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 547-564 June 6-11, 2021. ©2021 Association for Computational Linguistics
due to cultural bias (information asymmetry, Callahan and Herring, 2011). For example, Fig. 1 shows that the Japanese Wikipedia article of an American politician, Ron Paul, does not have information about his college degree perhaps because Japanese Wikipedia editors are less interested in specific educational backgrounds of American politicians.
In this paper, we introduce the task of cross-lingual open-retrieval question answering (XOR QA) which aims at answering multilingual questions from non-English native speakers given multilingual resources. To support research in this area, we construct a dataset (called XOR-TYDI QA) of 40k annotated questions and answers across 7 typologically diverse languages. Questions in our dataset are inherited from TYDI QA (Clark et al., 2020), which are written by native speakers and are originally unanswerable due to the information scarcity or asymmetry issues. XOR-TYDI QA is the first large-scale cross-lingual open-retrieval QA dataset that consists of information-seeking questions from native speakers and multilingual reference documents.
XOR-TYDI QA is constructed with an annotation pipeline that allows for cross-lingual retrieval from large-scale Wikipedia corpora (§2). Unanswerable questions in TYDI QA are first translated into English by professional translators. Then, annotators find answers to translated queries given English Wikipedia using our new model-in-the-loop annotation framework that reduces annotation errors. Finally, answers are verified and translated back to the target languages.
Building on the dataset, we introduce three new tasks in the order of increasing complexity (§3). In XOR-RETRIEVE, a system retrieves English Wikipedia paragraphs with sufficient information to answer the question posed in the target language. XOR-ENGLISHSPAN takes one step further and finds a minimal answer span from the retrieved English paragraphs. Finally, XOR-FULL expects a system to generate an answer end to end in the target language by consulting both English and the target language's Wikipedia. XOR-FULL is our ultimate goal, and the first two tasks enable researchers to diagnose where their models fail and develop under less coding efforts and resources.
We provide baselines that extend state-of-the-art open-retrieval QA systems (Asai et al., 2020; Karpukhin et al., 2020) to our multilingual retrieval setting. Our best baseline achieves an average of
18.7 F1 points on XOR-FULL. This result indicates that XOR-TYDI QA poses unique challenges to tackle toward building a real-world open-retrieval QA system for diverse languages. We expect that our dataset opens up new challenges to make progress in multilingual representation learning.
2 The XOR-TyDI QA Dataset
Our XOR-TyDI QA dataset comprises questions inherited from TyDI QA (Clark et al., 2020) and answers augmented with our annotation process across 7 typologically diverse languages. We focus on cross-lingual retrieval from English Wikipedia because in our preliminary investigation we were able to find answers to a majority of the questions from resource-rich English Wikipedia, and native speakers with much annotation experience were readily available via crowdsourcing in English.
2.1 XOR-TYDI QA Collection
Our annotation pipeline proceeds with four steps: 1) collection of questions from TYDI QA without a same-language answer which require cross-lingual reference to answer (§2.1.1); 2) question translation from a target language to the pivot language of English where the missing information may exist (§2.1.2); 3) answer retrieval in the pivot language given a set of candidate documents (§2.1.3); 4) answer verification and translation from the pivot language back to the original language (§2.1.4). Fig. 2 shows an overview of the pipeline.
2.1.1 Question Selection
Our questions are collected from unanswerable questions in TYDI QA. A question is unanswerable in TYDI QA if an annotator cannot select a passage answer (a paragraph in the article that contains an answer). We randomly sample 5,000 questions without any passage answer annotations (unanswerable questions) from the TYDI QA training data, and split them into training (4,500) and development (500) sets. We use the development data from TYDI QA as our test data, since the TYDI QA's original test data is not publicly available.2 We choose 7 languages with varying amounts of Wikipedia data out of the 10 non-English languages based on the cost and availability
$^{2}$ Furthermore, despite the benefits of hidden test sets, the resource-intensive nature of open-retrieval QA is not suitable to code-submission leaderboards. This further precluded the use of the original TyDI QA test sets.
Figure 2: Overview of the annotation process for XOR-TYDI QA.
of translators:3 Arabic, Bengali, Finnish, Japanese, Korean, Russian and Telugu.
2.1.2 Question Translation
We use a professional translation service, Gengo, to translate all collected questions into English. Since named entities are crucial for QA, we instruct translators to carefully translate them by searching for common English translations from English Wikipedia or other external sources. We perform manual quality assessment by native speakers on 50 translation samples, finding that more than  $95\%$  are correct. Note that while these translations are a part of the annotation procedure (due to the inherently cross-lingual nature of this task), they are not provided to models during evaluation.
2.1.3 Answer Retrieval in English
We use Amazon Mechanical Turk to retrieve answers to translated English questions given English Wikipedia articles. Annotators are instructed to select passage answers (gold paragraphs) and minimal answer spans as in Clark et al. (2020).
To annotate answers to information-seeking queries, previous work first identifies relevant Wikipedia articles using Google Search, and then annotators attempt to find answers there. Asai and Choi (2020) show that in information-seeking QA datasets many questions were annotated as "unanswerable" due to two systematic errors: retrieval error where the search engine failed to retrieve a relevant article and answer annotation error where the annotator overlooks answer content. Importantly, these two types of annotation errors present a tradeoff: if we retrieve many articles, retrieval errors will be reduced at the expense of answer
annotation errors because annotators have to find answer context among many candidate articles.
Collaborative model-in-the-loop. To find a middle ground in the tradeoff, we introduce a collaborative model-in-the-loop framework that uses Google Search and a state-of-the-art paragraph ranker. We first run Google Search to retrieve as many as top 10 Wikipedia articles, resulting in 387 paragraphs per question on average. We score them with Path Retriever (Asai et al., 2020) and present the five highest scoring paragraphs. Annotators are asked to skim these five paragraphs first; if they cannot find any answer content, they are asked to read the rest of the paragraphs, where the Wikipedia section headings guide their reading. To incentivize workers to find answers beyond the pre-selected ones, we carefully communicate with workers and send additional rewards to annotators who actively read the rest of the paragraphs and find answers for questions that other annotators may overlook. We found about  $70\%$  of the answers from the 5 paragraphs and  $30\%$  from the rest of the paragraphs in the top 10 articles. This means that while our paragraph ranking was effective, the annotators did not fully rely on it, thereby mitigating the influence of the passage ranking model on the dataset. See Appendix §B.1 for annotation interface details.
Quality control for QA annotation. We first recruit MTurkers with a high approval rate ( $\geq 96\%$ ) located in English-speaking countries, and all workers first annotate the same qualification batch. We assess the quality of those submissions and select high-quality annotators. Consequently, 40 out of more than 200 workers were qualified and 24 workers annotated most of our data. More details are in Appendix B.3.
3The cost of translations depends on the number of available translators, and the estimated translation cost for the other three non-English languages was considerably higher.
4 https://gengo.com/.
%	Ar	Bn	Fi	Ja	Ko	Ru	Te	All
TYDI QA	82	42	57	50	29	69	28	50
XOR-TYDI QA	92	82	83	77	68	83	44	72
Improvement	10	40	26	27	39	14	16	22
2.1.4 Answer Verification and Translation
We verify the annotated answers and translate those answers back to the target languages (cross-lingual data). Finally, we mix the annotated cross-lingual data with the same-language data from TYDI QA to reflect the actual question distributions from native speakers (in-language data).
Answer verification. We trained undergraduate students who are native English speakers to verify the annotated paragraphs and short answers. Only  $8\%$  of the answers were marked as incorrect through the verification phase and were later corrected by our pool of high-quality crowdworkers who yielded less than  $1\%$  annotation error.
Answer translation. We again use Gengo to translate answers from English back to the original languages. We give translators further instructions to normalize answers such that they are consistent with answers in TYDI QA. For example, some languages use their own unique set of numerals rather than Arabic numerals to represent numeric answers (e.g., Bengali numerals, Chinese numerals in Japanese text). The details of the answer translation process are described in Appendix §B.4. Note that because of the cost of answer translations, we conduct this answer translation process for evaluation sets only.
2.2 The XOR-TyDI QA Corpus
Dataset statistics. Table 1 shows the percentages of the questions annotated with short answers in the original TYDI QA and our XOR-TYDI QA, and Table 2 shows statistics of XOR-TYDI QA. As seen in Table 1, cross-lingual retrieval significantly increases the answer coverage in all languages by up to  $40\%$  (Bengali), and consequently we found answers for more than  $50\%$  of the origi
Table 1: Percentage of the questions with short answers (answerable questions) in the original TYDI QA dataset (dev) and XOR-TYDI QA. The third row (Improvement) represents the percentage of the questions that become answerable by searching the English Wikipedia articles.
Cross-lingual	In-language
Train	Dev	Test	Train	Dev	Test
Ar	2,574	350	137	15,828	358	1132
Bn	2,582	312	128	2,428	115	139
Fi	2,088	360	530	7,680	255	1,197
Ja	2,288	296	449	5,527	137	867
Ko	2,469	299	646	1,856	72	505
Ru	1,941	255	235	7,349	313	1,125
Te	1,308	238	374	5,451	113	712
Table 2: Dataset size of the XOR-TYDI QA corpus (answered data). Cross-lingual data comes from our reannotated questions that did not originally have same-language answers in TYDI QA. In-language data are taken directly from answerable questions in TYDI QA.
nal information-seeking questions in 6 out of the 7 languages. This result confirms the effectiveness of searching multilingual document collections to improve the answer coverage. Detailed statistics of the numbers of long answers, short answers, and unanswered questions are in Appendix  $\S \mathbf{B}.5$  .We also release the  $30\mathrm{k}$  manually translated questions for our training set, which could be used to train multilingual models or machine translation models.
Qualitative examples. Table 3 illustrates that finding relevant articles from multilingual document collections is important to answer questions asked by users with diverse linguistic and cultural backgrounds. The first question is unanswerable in Korean Wikipedia, but there is a clear description about who was the prime minister of France at the time in English Wikipedia. The second example shows English Wikipedia sometimes contains rich information about a target language-specific topic (e.g., economy in Krasnodar, a city in Russia). Those examples demonstrate the effectiveness of searching for answers in another language with more abundant knowledge sources. In the last question of Table 3, on the other hand, only the Wikipedia of the target language can provide the answer. XOR QA allows for both retrieval paths.
Comparison with other datasets. Table 4 compares XOR-TyDI QA and existing multilingual QA datasets. XOR-TyDI QA has three key properties that are distinct from these QA benchmarks. First, since all questions are inherited from TyDi QA, they are information-seeking questions written by
<sup>6</sup>We found in the Telugu data, certain types of questions are very frequent (e.g., what is the pin code of X mandal?). Those questions often ask some specific information of local administration districts, and are often unanswerable because (a) they are typically not described in English Wikipedia and (b) the overall coverage of Telugu Wikipedia is quite low.
After our initial release in November 2020, we modified the XOR-TYDI QA data, and released a new version as XOR-TYDI QA (v1.1). All results are based on v1.1.
L	Original Question: \( {Q}_{L}\left( {Q}_{en}\right) \)	Passage Answer: \( {P}_{en} \) or \( {P}_{L} \)	Minimal Answer in English: \( {A}_{en} \)	Final Answer: \( {A}_{L} \)
Ko	1993年連鎖水準はリフロー?(Who was the French Prime Minister in 1993?)	Mayor of Neuilly-sur-Seine from 1983 to 2002, he was Minister of the Budget under Prime Minister Édouard Balladur (1993-1995).	Édouard Balladur	あてたア理由を発状の
Ru	Kakay splaney zapillata B Kpacnoiape Ha cegoiHnHnHn? (What is the average wage in Krasnodar?)	Krasnodar has the lowest unemployment rate among the cities of the Southern Federal District at 0.3% of the total working-age population. In addition, Krasnodar holds the first place in terms of highest average salary—21,742 rubles per capita.	21,742 rubles	21,742 pu6ji
Ja	速水堅曹はとて製組件技衣を学んだ? (Where did Kenso Hayami learn the silk-reeling technique?)	藩宮前橋製組件所を前橋に開設。カスル・ ハデラ一から直接、器械製組件技衣を学む (he founded Hanei Maebashi Silk Mill and learned instrumental silk reeling techniques directly from Caspal Müller)	-	藩宮前橋製組件所 (Hanei Maebashi Silk Mill)
Table 3: Examples newly annotated for Korean (Ko) and Russian (Ru) questions. The bottom example is an answerable question from TYDI QA for which only Japanese Wikipedia includes the correct answer.
Dataset	Asked by native speakers	Open-retrieval	Cross-lingual
TYDI QA	✓	✗	✗
MLQA	✗	✗	✓
XQuAD	✗	✗	✗
MKQA	✗	WikiData	✗
MLQA-R	✗	21k sents	✓
XQuAD-R	✗	13k sents	✓
XOR-TYDI QA	✓	Wikipedia	✓
Table 4: Comparison with recent multilingual QA datasets. MKQA's answers are aligned to WikiData.
native speakers, and better reflect native speakers' interests and their own linguistic phenomena. This distinguishes XOR-TYDI QA from translation-based datasets such as MLQA (Lewis et al., 2020) and MKQA (Longpre et al., 2020). Second, our dataset requires cross-lingual retrieval unlike other multilingual datasets such as TYDI QA or XQuAD (Artetxe et al., 2020), which focus on same-language QA. Lastly, questions in XOR-TYDI QA require open retrieval from Wikipedia, whereas MLQA-R and XQuAD-R (Roy et al., 2020) limit the search space to matching each question with the predetermined  $21\mathrm{k} / 31\mathrm{k}$  sentences.
3 XOR QA Tasks and Baselines
We introduce three new tasks (Fig. 3): XOR-RETRIEVE, XOR-ENGLISHSPAN, and XOR-FULL with our newly collected XOR-TYDI QA dataset and construct strong baselines for each task. XOR-FULL defines our goal of building a multilingual open-retrieval QA system that uses both cross
lingual and in-language questions from XOR-TYDI QA. To diagnose where models fail and to allow researchers to use the data with less coding effort or computational resource, we also introduce the first two intermediate tasks that only use the crosslingual data (Table 2). We denote the target language by  $L_{i}$ . We also denote the English Wikipedia collection by  $W_{eng}$  and the Wikipedia collection in each target language  $L_{i}$  by  $W_{i}$ . We experiment with baselines using black-box APIs as a reference, but we encourage the community to use white-box systems so that all experimental details can be understood. Nonetheless, we release the intermediate results from those external APIs to make our results reproducible. All of the white-box system results can be reproduced using our codebase.
3.1 XOR-RETRIEVE: Cross-lingual Paragraph Retrieval
Task. Given a question in  $L_{i}$  and English Wikipedia  $W_{eng}$ , the task is to retrieve English paragraphs for the question. Finding evidence paragraphs from large-scale document collections like Wikipedia is a challenging task, especially when a query and documents are in different languages and systems cannot perform lexical matching.
Evaluation. Different open-retrieval QA models use different units for retrieval. To make fair comparisons across various models, we measure the recall by computing the fraction of the questions for which the minimal answer is contained in the top  $n$  tokens selected. We evaluate with  $n = 2k$ ,  $5k$ : R@2kt and R@5kt (kilo-tokens).
Figure 3: Overview of the tasks and baselines. Each dotted rectangle represents one of the three tasks and surrounds used pipeline modules.
Translate baselines. We first translate queries into English, and then paragraphs are retrieved in a monolingual way. For query translation, we train transformer machine translation (MT) models on publicly available corpora for easy replication. We also run Google's online machine translation service (GMT). This is not completely reproducible as these systems get constantly updated; nor do we know what model and training data they use. We encourage the community to use open MT systems where system details are available. For retrieval, we explore term-based retrieval (BM25, Robertson and Zaragoza 2009), term-based retrieval followed by neural paragraph ranking (Path Retriever, Asai et al. 2020), and end-to-end neural retrieval (DPR, Karpukhin et al. 2020).
Multilingual baselines. Alternatively, we can directly apply a multilingual pretrained model to retrieve paragraphs. We initialize and train a DPR encoder with multilingual BERT to enable multilingual document retrieval (Devlin et al., 2019).
3.2 XOR-ENGLISHSPAN: L-to-English Open-Retrieval QA
Task. Given a question in  $L_{i}$  and English Wikipedia  $W_{eng}$ , a system retrieves paragraphs from  $W_{eng}$  and extracts an answer. This task is equivalent to existing open-retrieval QA tasks (Chen et al., 2017), except that the query is not in English. This task involves challenging cross-lingual retrieval and question answering on the  $L_{i}$  query and English evidence paragraphs.
Evaluation. We use Exact Match (EM) and F1 over the annotated answer's token set following prior work (Rajpurkar et al., 2016).
Baselines. Our pipeline uses a machine reading
model to find a minimal span that answers the question given paragraphs selected from the previous XOR-RETRIEVE step. In particular, for the translate baselines, we use the same approach as state-of-the-art models (Asai et al., 2020; Karpukhin et al., 2020) that jointly predicts a span and a relevance score of each paragraph to the question. For the multilingual baseline where queries are not automatically translated during evaluation, we build a reader model with multilingual BERT.
3.3 XOR-FULL: Round Trip
Task. Given a question in target language  $L_{i}$  and Wikipedia in both English and  $L_{i}$  ( $W_{eng}$  and  $W_{i}$ ), a system is required to generate an answer in  $L_{i}$ . In this task, a system does not know a priori in which language we can find information that the user is seeking. Note that the XOR-FULL evaluation data includes both cross-lingual and in-language data, while XOR-RETRIEVE and XOR-ENGLISHSPAN only use cross-lingual data during evaluation.
Evaluation. Some answers in XOR-FULL are translated from English so the same spans may not exist in the target language's Wikipedia. For this reason, we use token-level BLEU scores (Papineni et al., 2002) over a ground-truth token set in addition to F1 and EM. The same tokenizer is applied to ground-truth and predicted answers to compute token-level F1 and BLEU.<sup>7</sup>
Baselines. Unlike the previous two tasks, evidence paragraphs can be found both in the target language and English, and a system has to output final answers based on the most plausible paragraphs. In this work, we introduce a simple multi
<sup>7</sup>We use the Moses tokenizer (Koehn et al., 2007) for all languages except we apply MeCab (Kudo, 2006) to Japanese.