SlowGuess/NAACL_2021_OCR huggingface dataset:
Limits of traditional annotation. In addition to fundamental problems of information scarcity and asymmetry in multilingual QA, questions can be labeled as unanswerable simply because of annotation errors. Annotation procedures for information-seeking QA data usually have each annotator read a single Wikipedia article retrieved by a search engine and label a correct answer span or label the question as not answered by the article (Kwiatkowski et al., 2019; Clark et al., 2020). In this procedure, the answer coverage is underestimated when the search engine fails to retrieve relevant articles (retrieval errors) or the annotator overlooks answer content from the selected articles (answer annotation errors, Asai and Choi, 2020). Importantly, these two types of annotation errors present a tradeoff: if we retrieve many articles, retrieval errors will be reduced at the expense of answer annotation errors because annotators have to find answer context among many candidate articles. An annotation procedure that misses too many answers will lead to an artificially small dataset.
B Additional Details of Dataset Creation
B.1 Annotation Interface
In this section, we describe the details of the annotation interface we used for answer annotation in English (§2.1.3). The annotation interface can be seen in Figs. 4 and 5. To maximize the answer coverage for open-retrieval questions, we first rank paragraphs from top articles retrieved by Google
Search. During this paragraph ranking process, we only consider top 5 paragraphs and exclude the articles ranked from top 6 to 10. Increasing the number of the initial articles introduces more noise and confuses our paragraph ranking model, while human annotators sometimes found that those low-ranked articles relevant and retrieved answers from them as discussed in §2.1.3. In the annotation interface, we first present those top 5 paragraphs first (the ones highlighted in light blue in Fig. 4). When annotators do not find answers in the pre-selected top 5 paragraphs, they will explore more paragraphs and articles by expanding originally collapsed articles as in Fig. 5.
Figure 4: Annotation interface (expanded). The blue highlighted paragraphs are ranked high by the BERT paragraph ranker, and the orange highlighted paragraph is the one clicked by the annotator.
Figure 5: Annotation interface (collapsed). Annotators can choose to read full articles or collapse articles.
B.2 Quality Control for Question Translation
We first ask Gengo translators to translate 20 sample questions following our detailed instruction before starting the task, and ask native speakers to assess the quality of translations. We filter out translators who do not provide translation results that meet our standard (e.g., wrong translations of entities, heavy reliance on public machine translation systems). We have found that some of the translators almost copy and paste outputs of existing APIs without fixing errors even when there are crucial errors. After this initial qualification process, we observe that the translation quality is sufficiently high.
B.3 Quality Control for QA annotation
To control the QA annotation quality, we recruit workers with a high approval rate  $(\geq 96\%)$  located in English-speaking countries and conducted a rigorous qualification procedure. In our qualification stage, we post small calibration batches and evaluate the workers' performance by expert judgements from authors and agreement with other annotators. To keep the high quality of annotations, we randomly sample qualified workers weekly and manually monitor their annotations by comparing them with gold annotations by authors. We remove qualifications when we detect too many incorrect annotations (e.g., label a paragraph about a different person as a gold paragraph) and remove the annotations done by those disqualified annotators, which are later reannotated by a qualified worker. Over 200 annotators participated in our calibration tasks. About 40 workers are qualified with 24 actively working on the final dataset. Each HIT contains 5 questions with a reward ranging from 1.5 to 2.5 USD. Qualified annotators generally spend 1-2 minutes to answer each question. We give special rewards to annotators who actively search additional paragraphs or articles; the amounts of the rewards are calculated based on the numbers of the HITs they have submitted, resulting in 5-10 USD for each payment.
B.4 Answer Translation Instructions
During answer translation, we asked annotators to follow the instructions listed below:
- Translators need to use metric units by default, instead of imperial units.
- If the original answers are expressed in an imperial unit, translators are encouraged to
convert them into a metric unit (e.g., Height  $5^{\prime}3^{\prime \prime}\rightarrow$  身長  $160~\mathrm{cm}$  ）
- When translating proper nouns, translators are asked to use an official translation if it is available in Wikipedia; otherwise they are encouraged to transliterate them.
We also specify some language-specific instructions to make the translated answers consistent with the ones in the original TYDI QA dataset.
- For Japanese and Korean, translators do not need to spell out the numbers (e.g., 1954 -> 千九百五十四) as people usually use Arabic numerals.
- For Bengali, we expect the numbers will be spelled out in Bengali numerals as Bengali speakers rarely use Arabic numerals.
- For Japanese and Korean, translators use appropriate measure words (e.g., 1867年，57歳) if those measure words are commonly added in those languages.
- For the languages where the date needs to be expressed in some rigid format, translators need to follow the format.
B.5 Full Data Statistics of Cross-lingual data
Seen in Table 9 are full data statistics of crosslingual data of XOR-TYDI QA. Among the questions with "Long" answer annotations are some questions without any short answers as in Natural Questions or TYDI QA. We do not include those "Long answer only" examples in our XOR-TYDI QA evaluations.
We describe the details in training our baselines to facilitate easy replication of our results.
C.1 Machine Translation Models
Table 10 lists hyperpameters for training our transformer machine translation models. We generally follow the hyperpameters for the base-sized transformer (Vaswani et al., 2017). The one exception is English  $\leftrightarrow$  Russian where we used pretrained transformer large models. For each language direction, all data are encoded into subwords by Moses tokenization (Koehn et al., 2007, for Arabic, Finnish, and Russian) and BPE (Sennrich et al., 2016) or SentencePiece (Kudo and Richardson, 2018, for
Bengali, Japanese, Korean, and Telugu). We train an autoregressive transformer (Vaswani et al., 2017) with the fairseq library (Ott et al., 2019).
C.2 Retrieval Models
Training DPR and Path Retriever. To train an English DPR and Path Retriever, we first initialize the parameters of the models with the ones trained on Natural Questions Open data, which is available on their repository. During finetuning on XOR-TyDI QA, we use the human translated questions with the annotated gold paragraph data.
Choice of negative and positive context. Selection of positive and negative examples is crucial to train competitive neural retriever models (Karpukhin et al., 2020). We follow the hyperparameters used in the original papers (Karpukhin et al., 2020; Asai et al., 2020). To construct effective negative and positive context, we follow the approaches introduced by the authors of those works.
To train DPR, we use the original gold paragraphs (long answers) annotated by MTurkers as positive passages. Following the experimental settings of DPR on Natural Questions, we first split gold paragraphs into 100-token units, and consider the units with the original short answer annotations as positive context. For negative context, we first randomly sample one negative paragraph per question from the top 5 paragraphs pre-selected by our paragraph reranking model in §2.1.3, split the negative paragraph into 100-token units, and then randomly pick one to use it as a negative context. We also reuse the in-batch negative paragraphs as discussed in Karpukhin et al. (2020).
Regarding the training of Path Retriever, we randomly sample top 50 paragraphs from the top 10 articles retrieved for annotations and use them as negative paragraphs. We also use the annotated long answers as positive paragraphs.
Implementation details of BM25 Retrievers. To implement BM25-based retrievers for the 7 languages, we use ElasticSearch's Python client (Python Elasticsearch Client).<sup>12</sup> We apply the default tokenizers and analyzers for Arabic, Bengali, Finnish and Russian. Japanese and Korean are not supported by the default ElasticSearch language
"https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md.
12. https://elasticsearch-py.readthedocs.io/en/master/.
Li	Train (1 way)	Dev (2 way)	Test (2 way)
Total	Long (%)	Short (%)	Total	Long (%)	Short (%)	total	Long (%)	Short (%)
Arabic	4,500	2,862 (63)	2,574 (57)	500	357 (71)	350 (70)	235	144 (61)	137 (58)
Bengali	4,500	2,822 (63)	2,582 (57)	500	330 (66)	312 (62)	185	131 (70)	128 (69)
Finnish	4,500	2,454 (55)	2,088 (46)	500	372 (74)	360 (72)	800	556 (69)	530 (66)
Japanese	4,500	2,557 (57)	2,288 (51)	500	320 (64)	296 (60)	779	477 (61)	449 (58)
Korean	4,500	2,674 (59)	2,469 (55)	500	314 (63)	299 (60)	1,177	684 (58)	646 (55)
Russian	4,500	2,178 (48)	1,941 (43)	500	270 (54)	255 (51)	470	252 (53)	235 (50)
Telugu	4,500	1,515 (33)	1,308 (29)	500	258 (52)	238 (47)	1,752	394 (22)	374 (21)
Table 9: Dataset statistics of the resulting XOR QA corpus (cross-lingual data only). "Long" denotes the questions with paragraph answer annotations, and "Short" denotes the questions with short answer annotations. During evaluation, we disregard the questions without short answer annotations.
Hyperparameter	Value
label smoothing	0.1
# max tokens	4096
dropout rate	0.3
encoder embedding dim	512
encoder ffn dim	2048
# encoder attn heads	8
decoder embedding dim	512
decoder ffn dim	2048
# decoder attn heads	8
max source positions	10000
max target positions	10000
Adam lrate	\( 5 \times 10^{-4} \)
Adam \( \beta_1 \)	0.9
Adam \( \beta_2 \)	0.98
lr-scheduler	inverse square
warm-up lr	\( 1 \times 10^{-7} \)
# warmup updates	4000
# max updates	300K
length penalty	1.0
analyzers, so we use Kuromoji $^{13}$  and Nori plugins $^{14}$  for Japanese and Korean respectively. Note that we do not implement a BM25-based retriever for Telugu, since it is not supported by the default language analyzer and we could not find an official plugin for Telugu.
C.3 Machine Reading Models
We use the official hyperparameters for machine reading components of DPR and Path Retriever. Table 11 shows the list of the hyperparameters used to train a multilingual machine reading model for the monolingual pipeline in XOR-FULL. We lowercased input paragraphs and questions.
Table 10: Hyperparameters for our transformer machine translation models.
Hyperparameter	Value
max sequence length	384
document stride	128
max query length	64
Adam lrate	5 × 10-4
Adam ε	1 × 10-8
max gradient norm	1.0
# train epochs	3.0
seed	42
Table 11: Hyperparameters for our machine reading model in the monolingual pipeline.
Choice of negative and positive examples. For the Path Retriever and BM25 baselines' reader, we sample three negative paragraphs per annotated question-gold paragraph pair and train a model that jointly predicts an answer span and relevance score of each paragraph to the question, following Asai et al. (2020). In DPR, the training examples are retrieved by the trained retriever, and we train the reader with 24 negative paragraphs by distant supervision (Karpukhin et al., 2020). We use human translated English questions to train English reader models, and use the original questions in  $L_{i}$  to train a multilingual reader model.
D Additional Results and Analysis
XOR-RETRIEVE. We present the R@2kt scores of the retrieval baselines in Table 12. As shown in Table 5, given human translations, DPR generally outperforms other two retrieval baselines. We also present R@2kt and R@5kt of our DPR models on our development set in Table 13, and we observe a similar performance trend to the test set: models with queries translated by GMT outperform other models in all of the XOR-TYDI QA languages. Comparing the two baselines that do not use external black-box APIs, we see that
13 https://www(elastic.co/guide/ en/elasticsearch/plugins/7.9/ analysis-kuromoji.html. 14 https://www(elastic.co/guide/en/ elasticsearch/plugins/7.9/analysis-nori. html.
Human	GMT	Our MT	Multi.
DPR	PATH	BM	DPR	PATH	DPR	PATH	DPR
Ar	65.8	65.0	41.6	61.7	59.1	48.3	45.0	41.2
Bn	72.8	78.1	57.7	72.0	58.2	54.4	60.9	43.9
Fi	66.5	68.0	43.7	60.6	60.3	56.7	56.6	50.3
Ja	62.0	59.0	38.8	52.1	50.0	41.8	36.7	29.1
Ko	65.0	60.0	43.8	57.9	50.3	39.4	33.8	34.5
Ru	57.5	59.9	35.2	51.2	54.1	39.6	34.7	35.3
Te	66.3	59.6	44.6	59.4	58.0	18.7	15.7	37.2
Av.	65.1	64.3	43.5	59.3	58.2	42.7	40.5	38.8
Table 12: R@2kt (§3.1) on the test data in the XOR-RETRIEVE setting. PATH and BM denote Path Retriever and BM25 respectively. The rightmost column is a multilingual approach that bypasses the query translation step (§3.1).
GMT	Our MT	Multi.
R@2kt	R@5kt	R@2kt	R@5kt	R@2kt	R@5kt
Ar	62.5	69.6	43.4	52.4	38.8	48.9
Bn	74.7	82.2	53.9	62.8	48.4	60.2
Fi	57.3	62.4	55.1	61.8	52.5	59.2
Ja	55.6	64.7	40.2	48.1	26.6	34.9
Ko	60.0	68.8	50.5	58.6	44.2	49.8
Ru	52.7	60.8	30.8	37.8	33.3	43.0
Te	72.3	79.0	20.2	32.4	39.9	55.5
Av.	62.2	69.6	42.0	50.6	40.5	50.2
the translation approach (Our MT) outperforms the multilingual one (Multi.) in Arabic, Bengali, Finnish Japanese, and Korean, while it performs poorly in Telugu. These results are consistent with the ones on the test data in Table. 5.
XOR-ENGLISHSPAN. Table 14 shows the F1 and EM scores of our DPR models on the development data in the XOR-ENGLISHSPAN setting. Similar to the results on XOR-RETRIEVE, GMT significantly outperforms our MT and our multilingual model. Probably due to the error propagation, the Telugu performance of our MT baseline is low, indicating the importance of developing a multilingual baseline that could perform well on languages with little parallel data for translation training.
XOR-FULL. We present F1, BLEU and EM scores for XOR-FULL in Tables 15, 16 and 17. We also present F1 scores and average F1, BLEU and EM scores on the development set in Table 18.
Table 13: R@5kt (§3.1) of DPR models (translate DPR and multilingual DPR) on the development data in the XOR-RETRIEVE setting.
GMT	Our MT	Multi.
F1	EM	F1	EM	F1	EM
Ar	35.4	27.7	20.9	14.9	17.2	12.3
Bn	42.1	35.3	25.2	20.5	21.8	17.3
Fi	31.8	23.1	31.9	23.3	27.6	20.8
Ja	27.2	20.9	19.6	15.5	15.5	12.8
Ko	32.5	22.7	25.3	18.1	18.5	14.4
Ru	34.7	28.2	16.1	11.4	21.3	17.3
Te	35.0	27.4	3.6	1.7	17.7	13.1
Av.	35.0	27.4	20.4	15.1	19.9	15.4
Table 14: F1 and EM scores of our DPR models (translate DPR and multilingual DPR) on the development data in the XOR-ENGLISHSPAN setting.
D.2 Additional Analysis
Single language Wikipedia ablations in XOR-FULL. In XOR-FULL, a system is expected to answer a question in the target language by consulting multilingual Wikipedia corpora, but which language answer content exists in is not known a priori (§3.3). To understand the benefit of retrieving evidence from a multilingual document pool, we run single language Wikipedia ablations. In this study, we conduct ablations in which systems only use either English Wikipedia ( $W_{eng}$ ) or the target language's Wikipedia ( $W_i$ ). We run the monolingual baselines for  $W_i$  only and the cross-lingual baselines for  $W_{eng}$  only. For the cross-lingual baseline, all predicted answers will be translated back to the target languages.
The bottom section of Table 15 shows the full results of single language Wikipedia ablations on XOR-FULL. In a majority of the languages, we observed performance drops from the full models that use both  $W_{eng}$  and  $W_{i}$  (e.g., 20.1 vs. 14.3 F1 in Korean). In Japanese and Finnish, our English Wikipedia only baselines outperform the full models. Currently, the answer aggregation process prioritizes answers predicted by monolingual models, but the monolingual models perform poorly in those two languages. Future work can address the challenges of improving evidence and answer aggregation from multilingual document collections.
Per-difficulty retrieval performance. We split our data by annotation difficulty i.e., whether or not a gold paragraph is selected by the BERT retriever used during annotation in our our collaborative annotation framework (§2.1.3). Table 19 presents retrieval performance broken down by difficulty. We observed a large performance gap between the easy and hard subsets (65.3 for easy vs. 59.9 for
Wiki Corpus	Translation	Retrieval	Target Language Li
Query	Answer	Li	Eng.	Ar	Bn	Fi	Ja	Ko	Ru	Te	Avg.
Wi,eng	GMT	GMT	GS	DPR	31.5	19.0	18.3	8.8	20.1	19.8	13.6	18.7
Our MT	Our MT	GS	DPR	29.6	6.6	15.5	7.6	16.4	18.7	1.7	13.7
Our MT	Our MT	BM25	DPR	12.1	22.0	9.3	5.4	9.7	7.4	0.8	9.5
-	GMT	GS	mDPR	30.5	5.2	16.9	8.2	17.6	19.8	6.0	15.7
Weng	GMT	GMT	-	DPR	23.9	18.5	22.9	24.1	17.5	16.8	13.2	19.5
Our MT	Our MT	-	DPR	7.6	5.9	16.2	9.0	5.3	5.5	0.8	7.2
-	GMT	-	mDPR	12.4	9.7	19.1	14.0	8.2	10.9	5.4	11.3
Wi	-	-	GS	-	29.0	0.9	9.5	6.2	14.3	18.5	0.9	11.3
-	-	BM25	-	12.0	22.0	9.3	5.3	9.7	7.4	-	-
Table 15: Performance on XOR-FULL task (F1 scores on the test data). "GS" denotes Google Search retrieval. The bottom section shows results from single Wikipedia baselines. ElasticSearch for BM25 does not support Telugu. "mDPR" denotes a DPR model where query and context encoders are initialized with multilingual BERT.
Wiki Corpus	Translation	Retrieval	Target Language Li
Query	Answer	Li	Eng.	Ar	Bn	Fi	Ja	Ko	Ru	Te	Avg.
Wi,eng	GMT	GMT	GS	DPR	22.1	10.9	13.3	3.0	20.1	11.4	9.1	12.1
Our MT	Our MT	GS	DPR	20.9	2.2	10.9	2.3	12.6	10.5	1.4	8.7
Our MT	Our MT	BM25	DPR	7.7	15.4	6.4	1.3	6.7	3.9	0.6	6.0
-	GMT	GS	mDPR	21.4	5.2	12.1	2.7	13.3	11.3	3.9	10.0
Weng	GMT	GMT	-	DPR	12.3	10.1	16.6	14.1	11.5	10.4	8.5	12.0
Our MT	Our MT	-	DPR	2.5	1.5	10.3	3.3	2.9	2.5	0.5	3.4
-	GMT	-	mDPR	6.7	4.5	13.5	8.1	8.2	6.5	3.1	6.8
Wi	-	-	GS	-	20.6	0.7	7.1	1.5	11.5	10.4	0.8	7.5
-	-	BM25	-	7.7	15.3	6.4	1.3	6.7	3.9	-	-
Table 16: Performance on XOR-FULL (EM scores on the test data). "GS" denotes Google Search retrieval. The bottom section shows results from single Wikipedia baselines. ElasticSearch for BM25 does not support Telugu. "mDPR" denotes a DPR model where query and context encoders are initialized with multilingual BERT.
Wiki Corpus	Translation	Retrieval	Target Language Li
Query	Answer	Li	Eng.	Ar	Bn	Fi	Ja	Ko	Ru	Te	Avg.
Wi,eng	GMT	GMT	GS	DPR	29.7	22.1	18.8	2.2	13.3	18.0	13.5	16.8
Our MT	Our MT	GS	DPR	27.8	7.4	10.9	2.0	12.6	17.0	1.1	8.9
Our MT	Our MT	BM25	DPR	12.8	22.9	6.4	1.2	7.0	7.3	0.3	12.0
-	GMT	GS	mDPR	27.8	7.0	13.9	1.8	11.3	17.0	5.3	13.9
Weng	GMT	GMT	-	DPR	24.5	21.4	20.6	6.1	10.0	14.2	13.3	15.7
Our MT	Our MT	-	DPR	8.6	6.7	16.7	2.8	3.9	5.0	0.3	6.3
-	GMT	-	mDPR	12.2	10.2	16.7	2.4	4.7	8.2	8.3	9.0
Wi	-	-	GS	-	27.3	0.7	10.4	1.6	10.4	16.6	0.8	9.7
-	-	BM25	-	12.8	22.9	10.6	1.2	7.0	7.3	-	-
Table 17: Performance on XOR-FULL (BLEU scores on the test data). "GS" denotes Google Search retrieval. The bottom section shows results from single Wikipedia baselines. ElasticSearch for BM25 does not support Telugu. "mDPR" denotes a DPR model where query and context encoders are initialized with multilingual BERT.
Translation
Query	Answer	Retrieval
Li	Eng.	Target Language Li	Macro Average
Ar	Bn	Fi	Ja	Ko	Ru	Te	F1	EM	BLEU
GMT	GMT	GS	DPR	18.0	29.1	13.8	5.7	15.2	14.9	15.6	16.0	9.9	14.9
Our MT	Our MT	GS	DPR	17.7	4.5	13.0	5.7	15.0	14.9	8.8	11.4	6.3	10.3
Our MT	Our MT	BM25	DPR	9.2	15.8	14.4	4.8	7.9	5.2	0.5	8.3	4.6	7.5
-	GMT	GS	mDPR	17.8	15.3	12.6	5.6	15.2	15.0	10.1	13.1	7.7	12.2
Table 18: Performance on XOR-FULL (dev data F1 scores and average F1, EM and BLEU scores). "GS" denotes Google Search retrieval, and "mDPR" denotes a DPR model where query and context encoders are initialized with multilingual BERT.
Query Translatorator	Easy	Hard
R@2kt	R@5kt	R@2kt	R@5kt
Human	65.3	72.5	59.9	68.9
GMT	61.1	67.7	54.3	63.4
Our MT	41.9	49.9	37.7	44.5
Multilingual	34.3	44.3	36.1	40.9
Table 19: Macro-averaged retrieval recall on the easy and hard subsets of the development set. All configurations use DPR for retrieval. The Multilingual model avoids query translation.
hard subsets in R@2kt with human translation and DPR), suggesting that the questions from the hard subset are clearly more challenging than the ones from the easy subset.
D.3 Qualitative Analysis on Translation Errors
One primary challenge in question translation is precisely translating key words (e.g., entities, year); our MT correctly translates a Japanese question,アーマンドアはい生た？(When was Almond Eye born; Almond Eye is a Japanese popular race horse)15 while Helsinki (Tiedemann and Thottingal, 2020) translates it to "When was al-. mond born?" This resulted in retrieval errors, and Wikipedia articles related to almonds were selected. Intrinsic metrics such as BLEU would not consider the importance of these translation mistakes.
15 https://en.wikipedia.org/wiki/Almond_Eye.