SlowGuess/ACL_2020_OCR huggingface dataset:
Patrick Lewis*† Barlas Oğuz* Rudy Rinott* Sebastian Riedel*† Holger Schwenk*
*Facebook AI Research †University College London
{plewis,barlaso,ruty,sriedel,schwenk}@fb.com
Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making building QA systems that work well in other languages challenging. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area.<sup>1</sup> MLQA contains QA instances in 7 languages, English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. MLQA has over 12K instances in English and 5K in each other language, with each instance parallel between 4 languages on average. We evaluate state-of-the-art cross-lingual models and machine-translation-based baselines on MLQA. In all cases, transfer results are significantly behind training-language performance.
Question answering (QA) is a central and highly popular area in NLP, with an abundance of datasets available to tackle the problem from various angles, including extractive QA, cloze-completion, and open-domain QA (Richardson, 2013; Rajpurkar et al., 2016; Chen et al., 2017; Kwiatkowski et al., 2019). The field has made rapid advances in recent years, even exceeding human performance in some settings (Devlin et al., 2019; Alberti et al., 2019).
Despite such popularity, QA datasets in languages other than English remain scarce, even for relatively high-resource languages (Asai et al., 2018), as collecting such datasets at sufficient scale and quality is difficult and costly. There
are two reasons why this lack of data prevents internationalization of QA systems. First, we cannot measure progress on multilingual QA without relevant benchmark data. Second, we cannot easily train end-to-end QA models on the task, and arguably most recent successes in QA have been in fully supervised settings. Given recent progress in cross-lingual tasks such as document classification (Lewis et al., 2004; Klementiev et al., 2012; Schwenk and Li, 2018), semantic role labelling (Akbik et al., 2015) and NLI (Conneau et al., 2018), we argue that while multilingual QA training data might be useful but not strictly necessary, multilingual evaluation data is a must-have.
Recognising this need, several cross-lingual datasets have recently been assembled (Asai et al., 2018; Liu et al., 2019a). However, these generally cover only a small number of languages, combine data from different authors and annotation protocols, lack parallel instances, or explore less practically-useful QA domains or tasks (see Section 3). Highly parallel data is particularly attractive, as it enables fairer comparison across languages, requires fewer source language annotations, and allows for additional evaluation setups at no extra annotation cost. A purpose-built evaluation benchmark dataset covering a range of diverse languages, and following the popular extractive QA paradigm on a practically-useful domain would be a powerful testbed for cross-lingual QA models.
With this work, we present such a benchmark, MLQA, and hope that it serves as an accelerator for multilingual QA in the way datasets such as SQuAD (Rajpurkar et al., 2016) have done for its monolingual counterpart. MLQA is a multi-way parallel extractive QA evaluation benchmark in seven languages: English, Arabic, German, Vietnamese, Spanish, Simplified Chinese and Hindi. To construct MLQA, we first automatically identify sentences from Wikipedia articles which have the same or similar meaning in multiple languages. We
$^{1}$ MLQA is publicly available at https://github.com/facebookresearch/mlqa
7315
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
extract the paragraphs that contain such sentences, then crowd-source questions on the English paragraphs, making sure the answer is in the aligned sentence. This makes it possible to answer the question in all languages in the vast majority of cases.2 The generated questions are then translated to all target languages by professional translators, and answer spans are annotated in the aligned contexts for the target languages.
The resulting corpus has between 5,000 and 6,000 instances in each language, and more than 12,000 in English. Each instance has an aligned equivalent in multiple other languages (always including English), the majority being 4-way aligned. Combined, there are over 46,000 QA annotations.
We define two tasks to assess performance on MLQA. The first, cross-lingual transfer (XLT), requires models trained in one language (in our case English) to transfer to test data in a different language. The second, generalised cross-lingual transfer (G-XLT) requires models to answer questions where the question and context language is different, e.g. questions in Hindi and contexts in Arabic, a setting possible because MLQA is highly parallel.
We provide baselines using state-of-the-art crosslingual techniques. We develop machine translation baselines which map answer spans based on the attention matrices from a translation model, and use multilingual BERT (Devlin et al., 2019) and XLM (Lample and Conneau, 2019) as zero-shot approaches. We use English for our training language and adopt SQuAD as a training dataset. We find that zero-shot XLM transfers best, but all models lag well behind training-language performance.
In summary, we make the following contributions: i) We develop a novel annotation pipeline to construct large multilingual, highly-parallel extractive QA datasets ii) We release MLQA, a 7-language evaluation dataset for cross-lingual QA iii) We define two cross-lingual QA tasks, including a novel generalised cross-lingual QA task iv) We provide baselines using state-of-the-art techniques, and demonstrate significant room for improvement.
2 The MLQA corpus
First, we state our desired properties for a crosslingual QA evaluation dataset. We note that whilst some existing datasets exhibit these properties,
none exhibit them all in combination (see section 3). We then describe our annotation protocol, which seeks to fulfil these desiderata.
Parallel The dataset should consist of instances that are parallel across many languages. First, this makes comparison of QA performance as a function of transfer language fairer. Second, additional evaluation setups become possible, as questions in one language can be applied to documents in another. Finally, annotation cost is also reduced as more instances can be shared between languages.
Natural Documents Building a parallel QA dataset in many languages requires access to parallel documents in those languages. Manually translating documents at sufficient scale entails huge translator workloads, and could result in unnatural documents. Exploiting existing naturally-parallel documents is advantageous, providing high-quality documents without requiring manual translation.
Diverse Languages A primary goal of crosslingual research is to develop systems that work well in many languages. The dataset should enable quantitative performance comparison across languages with different linguistic resources, language families and scripts.
Extractive QA Cross-lingual understanding benchmarks are typically based on classification (Conneau et al., 2018). Extracting spans in different languages represents a different language understanding challenge. Whilst there are extractive QA datasets in a number of languages (see Section 3), most were created at different times by different authors with different annotation setups, making cross-language analysis challenging.
Textual Domain We require a naturally highly language-parallel textual domain. Also, it is desirable to select a textual domain that matches existing extractive QA training resources, in order to isolate the change in performance due to language transfer.
To satisfy these desiderata, we identified the method described below and illustrated in Figure 1. Wikipedia represents a convenient textual domain, as its size and multi-linguality enables collection of data in many diverse languages at scale. It has been used to build many existing QA training resources, allowing us to leverage these to train QA models, without needing to build our own training dataset. We choose English as our source language as it has the largest Wikipedia, and to easily source crowd
2The automatically aligned sentences occasionally differ in a named entity or information content, or some questions may not make sense without the surrounding context. In these rare cases, there may be no answer for some languages.
7316
Figure 1: MLQA annotation pipeline. Only one target language is shown for clarity. Left: We first identify  $N$ -way parallel sentences  $b_{en}$ ,  $b_{1} \ldots b_{N-1}$  in Wikipedia articles on the same topic, and extract the paragraphs that contain them,  $c_{en}$ ,  $c_{1} \ldots c_{N-1}$ . Middle: Workers formulate questions  $q_{en}$  from  $c_{en}$  for which answer  $a_{en}$  is a span within  $b_{en}$ . Right: English questions  $q_{en}$  are then translated by professional translators into all languages  $q_{i}$  and the answer  $a_{i}$  is annotated in the target language context  $c_{i}$  such that  $a_{i}$  is a span within  $b_{i}$ .
workers. We choose six other languages which represent a broad range of linguistic phenomena and have sufficiently large Wikipedia. Our annotation pipeline consists of three main steps:
Step 1) We automatically extract paragraphs which contain a parallel sentence from articles on the same topic in each language (left of Figure 1).
Step 2) We employ crowd-workers to annotate questions and answer spans on the English paragraphs (centre of Figure 1). Annotators must choose answer spans within the parallel source sentence. This allows annotation of questions in the source language with high probability of being answerable in the target languages, even if the rest of the context paragraphs are different.
Step 3) We employ professional translators to translate the questions and to annotate answer spans in the target language (right of Figure 1).
The following sections describe each step in the data collection pipeline in more detail.
2.1 Parallel Sentence Mining
Parallel Sentence mining allows us to leverage naturally-written documents and avoid translation, which would be expensive and result in potentially unnatural documents. In order for questions to be answerable in every target language, we use contexts containing an  $N$ -way parallel sentence. Our approach is similar to WikiMatrix (Schwenk et al., 2019) which extracts parallel sentences for many language pairs in Wikipedia, but we limit the search
de	es	ar	zh	vi	hi
5.4M	1.1M	83.7k	24.1K	9.2k	1340
Table 1: Incremental alignment with English to obtain 7-way aligned sentences.
for parallel sentences to documents on the same topic only, and aim for  $N$ -way parallel sentences.
To detect parallel sentences we use the LASER toolkit, which achieves state-of-the-art performance in mining parallel sentences (Artetxe and Schwenk, 2019). LASER uses multilingual sentence embeddings and a distance or margin criterion in the embeddings space to detect parallel sentences. The reader is referred to Artetxe and Schwenk (2018) and Artetxe and Schwenk (2019) for a detailed description. See Appendix A.6 for further details and statistics on the number of parallel sentences mined for all language pairs.
We first independently align all languages with English, then intersect these sets of parallel sentences, forming sets of N-way parallel sentences. As shown in Table 1, starting with 5.4M parallel English/German sentences, the number of N-way parallel sentences quickly decreases as more languages are added. We also found that 7-way parallel sentences lack linguistic diversity, and often appear in the first sentence or paragraph of articles.
As a compromise between language-parallelism
$^{3}$ https://github.com/facebookresearch/LASER
7317
and both the number and diversity of parallel sentences, we use sentences that are 4-way parallel. This yields 385,396 parallel sentences (see Appendix A.6) which were sub-sampled to ensure parallel sentences were evenly distributed in paragraphs. We ensure that each language combination is equally represented, so that each language has many QA instances in common with every other language. Except for any rejected instances later in the pipeline, each QA instance will be parallel between English and three target languages.
2.2 English QA Annotation
We use Amazon Mechanical Turk to annotate English QA instances, broadly following the methodology of Rajpurkar et al. (2016). We present workers with an English aligned sentence,  $b_{en}$  along with the paragraph that contains it  $c_{en}$ . Workers formulate a question  $q_{en}$  and highlight the shortest answer span  $a_{en}$  that answers it.  $a_{en}$  must be a subspan of  $b_{en}$  to ensure  $q_{en}$  will be answerable in the target languages. We include a "No Question Possible" button when no sensible question could be asked. Screenshots of the annotation interface can be found in Appendix A.1. The first 15 questions from each worker are manually checked, after which the worker is contacted with feedback, or their work is auto-approved.
Once the questions and answers have been annotated, we run another task to re-associate English answers. Here, workers are presented with  $q_{en}$  and  $c_{en}$ , and requested to generate an  $a_{en}'$  or to indicate that  $q_{en}$  is not answerable. Two additional answer span annotations are collected for each question. The additional answer annotations enable us to calculate an inter-annotator agreement (IAA) score. We calculate the mean token F1 score between the three answer annotations, giving an IAA score of  $82\%$ , comparable to the SQuAD v1.1 development set, where this IAA measure is  $84\%$ .
Rather than provide all three answer annotations as gold answers, we select a single representative reference answer. In  $88\%$  of cases, either two or three of the answers exactly matched, so the majority answer is selected. In the remaining cases, the answer with highest F1 overlap with the other two is chosen. This results both in an accurate answer span, and ensures the English results are comparable to those in the target languages, where only one answer is annotated per question.
We discard instances where annotators marked
the question as unanswerable as well as instances where over  $50\%$  of the question appeared as a subsequence of the aligned sentence, as these are too easy or of low quality. Finally, we reject questions where the IAA score was very low  $(< 0.3)$  removing a small number of low quality instances. To verify we were not discarding challenging but high quality examples in this step, a manual analysis of discarded questions was performed. Of these discarded questions,  $38\%$  were poorly specified,  $24\%$  did not make sense/had no answer,  $30\%$  had poor answers, and only  $8\%$  were high quality challenging questions.
2.3 Target Language QA Annotation
We use the One Hour Translation platform to source professional translators to translate the questions from English to the six target languages, and to find answers in the target contexts. We present each translator with the English question  $q_{en}$ , English answer  $a_{en}$ , and the context  $c_x$  (containing aligned sentence  $b_x$ ) in target language  $x$ . The translators are only shown the aligned sentence and the sentence on each side (where these exist). This increases the chance of the question being answerable, as in some cases the aligned sentences are not perfectly parallel, without requiring workers to read the entire context  $c_x$ . By providing the English answer we try to minimize cultural and personal differences in the amount of detail in the answer.
We sample  $2\%$  of the translated questions for additional review by language experts. Translators that did not meet the quality standards were removed from the translator pool, and their translations were reallocated. By comparing the distribution of answer lengths relative to the context to the English distribution, some cases were found where some annotators selected very long answers, especially for Chinese. We clarified the instructions with these specific annotators, and send such cases for re-annotation. We discard instances in target languages where annotators indicate there is no answer in that language. This means some instances are not 4-way parallel. "No Answer" annotations occurred for  $6.6\% - 21.9\%$  of instances (Vietnamese and German, respectively). We release the "No Answer" data separately as an additional resource, but do not consider it in our experiments or analysis.
2.4 The Resulting MLQA corpus
Contexts, questions and answer spans for all the languages are then brought together to create the
7318
fold	en	de	es	ar	zh	vi	hi
dev	1148	512	500	517	504	511	507
test	11590	4517	5253	5335	5137	5495	4918
Table 2: Number of instances per language in MLQA.
de	es	ar	zh	vi	hi
de	5029
es	1972	5753
ar	1856	2139	5852
zh	1811	2108	2100	5641
vi	1857	2207	2210	2127	6006
hi	1593	1910	2017	2124	2124	5425
final corpus. MLQA consists of 12,738 extractive QA instances in English and between 5,029 and 6,006 instances in the target languages. 9,019 instances are 4-way parallel, 2,930 are 3-way parallel and 789 2-way parallel. Representative examples are shown in Figure 2. MLQA is split into development and test splits, with statistics in Tables 2, 3 and 4. To investigate the distribution of topics in MLQA, a random sample of 500 articles were manually analysed. Articles cover a broad range of topics across different cultures, world regions and disciplines.  $23\%$  are about people,  $19\%$  on physical places,  $13\%$  on cultural topics,  $12\%$  on science/engineering,  $9\%$  on organisations,  $6\%$  on events and  $18\%$  on other topics. Further statistics are given in Appendix A.2.
Table 3: Number of parallel instances between target language pairs (all instances are parallel with English).
en	de	es	ar	zh	vi	hi
# Articles	5530	2806	2762	2627	2673	2682	2255
# Contexts	10894	4509	5215	5085	4989	5246	4524
# Instances	12738	5029	5753	5852	5641	6006	5425
Table 4: Number of Wikipedia articles with a context in MLQA.
Monolingual QA Data There is a great variety of English QA data, popularized by MCTest (Richardson, 2013), CNN/Daily Mail (Hermann et al., 2015) CBT (Hill et al., 2016), and WikiQA (Yang et al., 2015) amongst others. Large span-based datasets such as SQuAD (Rajpurkar et al., 2016, 2018), TriviaQA (Joshi et al., 2017), NewsQA (Trischler et al., 2017), and Natural Questions (Kwiatkowski et al., 2019) have seen extractive QA become a dominant paradigm. However,
large, high-quality datasets in other languages are relatively rare. There are several Chinese datasets, such as DURader (He et al., 2018), CMRC (Cui et al., 2019b) and DRCD (Shao et al., 2018). More recently, there have been efforts to build corpora in a wider array of languages, such as Korean (Lim et al., 2019) and Arabic (Mozannar et al., 2019).
Cross-lingual QA Modelling Cross-lingual QA as a discipline has been explored in QA for RDF data for a number of years, such as the QALD-3 and 5 tracks (Cimiano et al., 2013; Unger et al., 2015), with more recent work from Zimina et al. (2018). Lee et al. (2018) explore an approach to use English QA data from SQuAD to improve QA performance in Korean using an in-language seed dataset. Kumar et al. (2019) study question generation by leveraging English questions to generate better Hindi questions, and Lee and Lee (2019) and Cui et al. (2019a) develop modelling approaches to improve performance on Chinese QA tasks using English resources. Lee et al. (2019) and Hsu et al. (2019) explore modelling approaches for zero-shot transfer and Singh et al. (2019) explore how training with cross-lingual data regularizes QA models.
Cross-lingual QA Data Gupta et al. (2018) release a parallel QA dataset in English and Hindi, Hardalov et al. (2019) investigate QA transfer from English to Bulgarian, Liu et al. (2019b) release a cloze QA dataset in Chinese and English, and Jing et al. (2019) released BiPar, built using parallel paragraphs from novels in English and Chinese. These datasets have a similar spirit to MLQA, but are limited to two languages. Asai et al. (2018) investigate extractive QA on a manually-translated set of 327 SQuAD instances in Japanese and French, and develop a phrase-alignment modelling technique, showing improvements over backtranslation. Like us, they build multi-way parallel extractive QA data, but MLQA has many more instances, covers more languages and does not require manual document translation. Liu et al. (2019a) explore cross-lingual open-domain QA with a dataset built from Wikipedia "Did you know?" questions, covering nine languages. Unlike MLQA, it is distantly supervised, the dataset size varies by language, instances are not parallel, and answer distributions vary by language, making quantitative comparisons across languages challenging. Finally, in contemporaneous work, Artetxe et al. (2019) release XQuAD, a dataset of
7319
(a)
Figure 2: (a) MLQA example parallel for En-De-Ar-Vi. (b) MLQA example parallel for En-Es-Zh-Hi. Answers shown as highlighted spans in contexts. Contexts shortened for clarity with “[...].”
1190 SQuAD instances from 240 paragraphs manually translated into 10 languages. As shown in Table 4, MLQA covers 7 languages, but contains more data per language – over 5k QA pairs from 5k paragraphs per language. MLQA also uses real Wikipedia contexts rather than manual translation.
Aggregated Cross-lingual Benchmarks Recently, following the widespread adoption of projects such as GLUE (Wang et al., 2019), there have been efforts to compile a suite of high quality multilingual tasks as a unified benchmark system. Two such projects, XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020) incorporate MLQA as part of their aggregated benchmark.
4 Cross-lingual QA Experiments
We introduce two tasks to assess cross-lingual QA performance with MLQA. The first, cross-lingual transfer (XLT), requires training a model with  $(c_x, q_x, a_x)$  training data in language  $x$ , in our case English. Development data in language  $x$  is used for tuning. At test time, the model must extract answer  $a_y$  in language  $y$  given context  $c_y$  and question  $q_y$ . The second task, generalized cross-lingual transfer (G-XLT), is trained in the same way, but at test time the model must extract  $a_z$  from  $c_z$  in language  $z$  given  $q_y$  in language  $y$ . This evaluation setup is possible because MLQA is highly parallel,
allowing us to swap  $q_{z}$  for  $q_{y}$  for parallel instances without changing the question's meaning.
As MLQA only has development and test data, we adopt SQuAD v1.1 as training data. We use MLQA-en as development data, and focus on zero-shot evaluation, where no training or development data is available in target languages. Models were trained with the SQuAD-v1 training method from Devlin et al. (2019) and implemented in PyText (Aly et al., 2018). We establish a number of baselines to assess current cross-lingual QA capabilities:
Translate-Train We translate instances from the SQuAD training set into the target language using machine-translation. Before translating, we enclose answers in quotes, as in Lee et al. (2018). This makes it easy to extract answers from translated contexts, and encourages the translation model to map answers into single spans. We discard instances where this fails ( $\sim 5\%$ ). This corpus is then used to train a model in the target language.
Translate-Test The context and question in the target language is translated into English at test time. We use our best English model to produce an answer span in the translated paragraph. For all languages other than Hindi, we use attention
<sup>4</sup>We use Facebook's production translation models.