X OR-T Y D I QA Datasheet♣♣♦♦♥♣♠

Akari Asai , Jungo Kasai , Jonathan H. Clark , Kenton Lee , Eunsol Choi , Hannaneh Hajishirzi♣♦
University of Washington Google Research♥♠
The University of Texas at Austin Allen Institute for AI{akari, jkasai, hannaneh}@cs.washington.edu{jhclark, kentonl}@google.com, eunsol@cs.utexas.edu

1

Motivation for Datasheet Creation

ロン・ポールの学部時代の専攻は？[Japanese](What did Ron Paul major in during undergraduate?)

Why was the dataset created? We bring together for the first time information-seeking questions, open-retrieval QA, and multilingual QA tocreate a multilingual open-retrieval QA datasetthat enables cross-lingual answer retrieval. Thistask framework reflects well real-world scenarios
where a QA system uses multilingual document
collections and answers questions asked by users
with diverse linguistic and cultural backgrounds(Fig. 1). Despite the common assumption that we
can find answers in the target language, web resources in non-English languages are largely limited compared to English (information scarcity),or the contents are biased towards their own cultures (information asymmetry). To solve these issues, X OR-T Y D I QA (Asai et al., 2020) provides abenchmark for developing a multilingual QA system that finds answers in multiple languages.
Has the dataset been used already? All papers
reporting results on X OR-T Y D I QA are required
to submit their system outputs to https://nlp.
cs.washington.edu/xorqa/.
Who funded the dataset? X OR-T Y D I QA wasfunded by Google, ONR (N00014-18-1-2826),
DARPA (N66001-19-2-403), NSF (IIS1252835,IIS-1562364), the Allen Distinguished Investigator Award, the Sloan Fellowship, and the Nakajima Foundation Fellowship.

2Dataset Composition

What are the instances? At their core areinformation-seeking questions from native speakers of one of the 7 typologically diverse languages.
How many instances are there? Shown in Table 1 are the data sizes for the 7 languages. In total, about 40k questions are newly annotated withanswers from English Wikipedia (cross-lingual

Multilingual document collections(Wikipedias)
ロン・ポール (ja.wikipedia)

高校卒業後はゲティスバーグ大学へ進学。
(After high school, he went to Gettysburg College.)
Ron Paul(en.wikipedia)

Paul went to Gettysburg College, where he was amember of the Lambda Chi Alpha fraternity. He
graduated with a B.S. degree in Biology in 1957.
生物学 (Biology)

Figure 1: Overview of X OR QA. Given a ques-tion in a target language (e.g., Japanese, Korean,or Russian), the model finds an answer in either
English or the the target language Wikipedia andreturns an answer in the target language.

Ar
BnFi
Ja
KoRuTeTot.

Cross-lingual
Train Dev Test

In-language
Train DevTest2574 351 137
2582 312 128
2088 362 5302288 296 449
2469 299 647
1941 255 235
1308 238 375
15250 2113 2501

15828 357
2428 115
7680 253
5527 140
1856
747349 313
5451 11346119 1365

1133
138
1197
869507
1125
712
5681

Table 1: Dataset size of the X OR-T Y D I QA corpus.

Cross-lingual data come from our re-annotatedquestions that did not originally have samelanguage answers in T Y D I QA. In-language data
are taken directly from answerable questions inT Y D I QA. During evaluation, we exclude thequestions for which we cannot find any minimal
answer annotations (about 28% of the questions).

data). In-language data are annotated with answers from the target language Wikipedia articles

LKoRuJaOriginal Question

Passage Answer1993ᄂᆫ펴ᅳᄅᆼ사ᅳᄎᆼ로ᅵᄂᆫ느ᅮ기ᅮᄋᄀᆫᅡ요? (Who was theFrench Prime Minister in1993?)

Mayor of Neuilly-sur-Seine from 1983 to2002, he was Minister of the Budget underPrime Minister Édouard Balladur(1993–1995).
Krasnodar has the lowest unemployment rate
among the cities of the Southern Federal
District at 0.3% of the total working-age
population. In addition, Krasnodar holds thefirst place in terms of highest averagesalary—21,742 rubles per capita.
藩営前橋製糸所を前橋に開設。カスパ
ル・ミュラーから直接、器械製糸技術を学び (he founded Hanei Maebashi Silk Milland learned instrumental silk reelingtechniques directly from Caspal Müller)

Kaka sredn zarplatav Krasnodare nasegodnxni den~?
(What is the average wagein Krasnodar?)
速水堅曹はどこで製糸技術を学んだ？ (Where did
Kenso Hayami learn thesilk-reeling technique?)MinimalAnswer

FinalAnswer

ÉdouardBalladurᄋ데ᅮ아ᅳᄅᆯ라바뒤르

21,742rubles21,742–rubl

藩営前橋製糸所(HaneiMaebashi
Silk Mill)

Table 2: Examples newly annotated for Korean (Ko) and Russian (Ru) questions. The bottom example isan answerable question from T Y D I QA for which only Japanese Wikipedia includes the correct answer.
and taken from T Y D I QA.
What data does each instance consist of? Eachinstance contains a question written by a native
speaker who is actually interested in knowing ananswer. We annotate each question with paragraphs from Wikipedia articles that has answer
content. And these paragraphs are further annotated with minimal answer spans that answer thequestion. See Table 2 for examples.
Each data point contains a {query, answer span}
written in the same language, Li , and the Li isone of 7 typologically diverse languages. For the
newly annotated portion, we also have {English
query, English answer span, a English Wikipedia
paragraph where the answer span is found}, andfor the existing TyDiQA annotation, we have {aWikipedia paragraph in Li where the answer wasfound}.
Does the data rely on external resources? Allnecessary data are available in the release, including questions with in-langauge answers fromT Y D I QA.
Are there recommended data splits or evaluation measures? To facilitate a wide rangeof research with X OR-T Y D I QA, we defined
three tasks in the order of increasing complexity: XOR-R ETRIEVE, X OR -E NGLISH S PAN,
and X OR -F ULL. In XOR-R ETRIEVE, a system retrieves paragraphs from English Wikipediathat contain information to answer the question
posed in the target language. This is evaluatedwith recall-based measures. X OR -E NGLISH S PANtakes one step further and finds a minimal answer
span from the retrieved English paragraphs. Thisis evaluated with F1 and exact matching scores.
Finally, X OR -F ULL expects a system to generate
an answer end to end in the target language by
consulting both English and the target language’sWikipedia. We measure F1, exact matching, andBLEU scores in the target language for evaluation.
We use the same random data split for all threetasks, and the answer annotations for the test data
are all hidden from public.

3Data Collection Process

How was the data collected? Our annotation
pipeline consists of four steps: 1) collection ofrealistic questions that require cross-lingual references by annotating questions from T Y D I QA
without a same-language answer; 2) questiontranslation from a target language to the pivot
language of English where the missing information may exist; 3) answer span selection in thepivot language given a set of candidate documents;
4) answer verification and translation from thepivot language back to the original language. The
first step was done by collecting questions without answer annotations from an existing dataset,
T Y D I QA. We proceeded with the second andfourth steps by collaborating with our university

volunteers and a third party translation service,
1Gengo. We performed crowdsourcing via Ama2
zon Mechanical Turk for the third step of answer
annotation.
Who was involved in the collection process andwhat were their roles? As mentioned above,
our university volunteers, the Gengo translation
service, and crowdworkers from Amazon Mechanical Turk were involved in the collection process.
Over what time frame was the data collected?
The dataset was collected over a period of March2020 through September 2020, during which weconstantly monitored the progress and data quality.
Does the dataset contain all possible instances?
Ultimately, the goal of multilingual question answering research is to build a system that can answer any question from anyone in any language.
For this reason, our dataset is obviously a sample
from the infinite space of potential languages, people, and questions.
If the dataset is a sample, then what is the population? We sample 7 diverse languages from7 distinct language families: Arabic, Bengali,
Finnish, Japanese, Korean, Russian, and Telugu.
For each language, questions are written by nativespeakers who read a randomly chosen Wikipedia
article as a prompt. These 7 languages represent
a great degree of typological diversity that will be
key in deploying QA systems to many languages
in the world. For example, Japanese is an agglutinative language with SOV word order and four
distinct alphabets. Bengali is a morphologicallyrich language that has inflection, affixation, compounding, and reduplication. Moreover, Bengali
Wikipedia has fewer than 100k Wikipedia articles
as of fall 2020, serving as a benchmark for lowresource question answering.

4

Data Preprocessing

What preprocessing / cleaning was done?
Our questions are originally sampled fromT Y D I QA (Clark et al., 2020). In particular, werandomly sample 5000 questions without any passage answer annotations (unanswerable questions)from the T Y D I QA train data, and split it into1
2

https://gengo.com/
https://www.mturk.com/

training (4,500) and development (500) sets. We
downloaded Wikipedia dumps for the 7 languages
and English, which are archived by T Y D I QA au3thors. We extract plain text from the Wikipedia
4dump data by wikiextractor, and then we process
the data by following common practice in open domain QA work (Chen et al., 2017).
To clean the final data, we trained undergraduate students who are native English speakers toverify the annotated paragraphs and short answers.
Only 8% of the answers were marked as incorrect
through the verification phase and were later corrected by our pool of high-quality crowdworkers
who yielded less than 1% annotation error.
Was the raw data saved in addition to thecleaned data? As we re-annotate those noisy
annotations, we do not include the raw data in ourmain dataset to avoid confusion. Those raw data
will be available upon request. In the raw annotation data, we preserve the comments and feedback
provided by our annotators and students who verified the data, which often provide useful feedback
on why the data were flagged as noisy.
Does this dataset collection/preprocessing procedure achieve the initial motivation? X ORT Y D I QA is a dataset that allows researchers
to explore phenomena specific to cross-lingualopen-retrieval QA and to work towards building a real-world open-retrieval QA system fordiverse languages. This is the first attempt toconstruct a large-scale cross-lingual open-retrieval
QA dataset, and we limit the retrieval target toEnglish (i.e., a system is required to search English or the target language Wikipedia) to collect high-quality data at scale from crowdworkers. Although English Wikipedia is by far thelargest among all of the Wikipedias and we observe our questions often become answerable
given English Wikipedia information, searching
more Wikipedias to answer questions about certain cultures (e.g., Japanese Wikipedia is likely tomore information about a Japanese manga thanany other Wikipedias). We also sample 7 languages from T Y D I QA based on the cost andavailability of translators. We hope future work
will extend the retrieval target as well as the target
languages to a more diverse set of languages.
3

https://storage.googleapis.com/
tydiqa/tydiqa.pdf
4
https://github.com/attardi/
wikiextractor

5

Dataset Distribution

Howisthedataset
distributed? Itis available for downloads at https://nlp.cs.washington.edu/xorqa/.
When was it released? Oct. 2020.
What license (if any) is it distributed under?
X OR-T Y D I QA is distributed under the CC BY5
SA 4.0 license.
Who is supporting and maintaining thedataset? X OR-T Y D I QA will be maintained by
the first two authors of the paper: Akari Asai andJungo Kasai. All updates will be posted on thedataset website.

6

Legal and Ethical Considerations

Were workers told what the dataset would be
used for and did they consent? Crowdworkersconsented to have their responses used in this way
through the Amazon Mechanical Turk Participation Agreement.
If it relates to people, could this dataset expose
people to harm or legal action? Our dataset
can include incorrect information to the extent that
Wikipedia can have wrong information about people. Nonetheless, we performed extensive quality control and answer verification to minimize therisk of harming people.
If it relates to people, does it unfairly advantage or disadvantage a particular social group?
One fundamental problem with the recent question answering benchmarks is that most of theirquestions are written by native English speakers
and overly represent English-centric topics, suchas American politics, sports, and culture. As such,models trained and developed on those datasetsare likely to fail to serve people with diverse language and cultural backgrounds. X OR-T Y D I QA
partially remedies this long-standing problem byannotating questions from native speakers of diverse languages. Thus, we encourage researchersand developers to benchmark on X OR-T Y D I QA
to mitigate the potential bias and unfairness of QA
systems. We acknowledge, however, this dataset
still covers a very limited subset of languages inthis world.
5

https://creativecommons.org/licenses/by-sa/4.0/legalcode

References
Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi.
2020. XOR QA: Cross-lingual open-retrieval
question answering.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In ACL.
Jonathan H. Clark, Eunsol Choi, Michael Collins,Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA:
A benchmark for information-seeking question
answering in typologically diverse languages.
TACL.