One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval
Authors: Akari Asai
Xinyan Yu
Jungo Kasai
Hannaneh Hajishirzi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Question answering
Multilingual NLP
Open-domain QA
Cross-lingual Information Retrieval

Abstract: We present Cross-lingual Open-Retrieval Answer Generation (CORA), the first unified many-to-many question answering (QA) model that can answer questions across many languages, even for ones without language-specific annotated data or knowledge sources.
We introduce a new dense passage retrieval algorithm that is trained to retrieve documents across languages for a question.
Combined with a  multilingual autoregressive generation model, CORA answers directly in the target language without any translation or in-language retrieval modules as used in prior work. We propose an iterative training method that automatically extends annotated data available only in high-resource languages to low-resource ones. Our results show that CORA substantially outperforms the previous state of the art on multilingual open QA benchmarks across 26 languages, 9 of which are unseen during training. Our analyses show the significance of cross-lingual retrieval and generation in many languages, particularly under low-resource settings. 

One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval
Akari Asai', Xinyan Yu Jungo Kasai,, Hannaneh Hajishirzi"
UUiieeriity of Washington, $ *Allen Institute for AI
(akari, xyu530, jkasai, hannaneh (cs www.wwwwwhwwwn..ommmmmm edu
Abstract
We present Cross-lingual Open-Retrieval Answer Generation (CORA), the first unified many-to-many question answering (QA) model that can answer questions across many languages, even for ones without language-specific annotated data or knowledge sources. We introduce new dense passage retrieval algorithm that is trained to retrieve documents across languages for a question. Combined with a multilingual autoregressive generation model, CORA answers directly in the target language without any translation or in-language retrieval modules used in prior work. We propose an iterative training method that automatically extends annotated data available only in high-resource languages to low-resource ones. Our results show that CORA substantially outperforms the previous state the art on multilingual open QA benchmarks across 26 languages, 9 of which are unseen during training. Our analyses show the significance of cross-lingual retrieval and generation in many languages, particularly under low-resource settings. Our code and trained model are publicly available at utlatps::/ happ:://githutee ccm www.///ai/aaaaaeeeeeeaaeaaaaaeae AkariAsai/
CORA
Introduction
Multilingual open question answering (QA) is the task of answering a question from a large collection of multilingual documents. Most recent progress in open QA made for English by building pipeline based on dense passage retriever trained on large-scale English QA datasets to find evidence passages English (Lee al.) 2019 Karpukhin et al. 2020). followed by reader that extracts an answer from retrieved passages. However, extending this approach to multilingual open QA poses new challenges. Answering multilingual questions requires retrieving evidence from knowledge sources of other languages than the original question since many languages have limited reference documents or the question sometimes inquires about concepts from other cultures (Asai al.) 202T] Lin et al. 2020). Nonetheless, large-scale cross-lingual open QA training data whose questions and evidence are different languages are not available in many those languages.
To address these challenges, previous work in multilingual open QA (Ture and Boschee, 2016 Asai
al., 2 translates questions into English, applies an English open QA system to answer in English. and then translates answers back to the target language. Those pipeline approaches suffer from eтror propagation of the machine translation component into the downstream QA. especially for low-resource languages. Moreover, they are not able to answer questions whose answers can be found in resources written in languages other than English or the target languages.
In this paper, we introduce unified many-to-many QA model that can answer questions in any target language by retrieving evidence from any language and generating answers in the target language. Our method (called CORA, Fig. [T] extends the retrieve-then: serrive-thenn-geeertte
generate approach of English open QA (Lewis al. 2020 Izaarr and Grave 2021b) with single cross-lingual retriever and generator that do not rely on dunguage-spectte retrivers or machine translation modules. The multilingual
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

mDPR: Multilingual Dense Passage Retriever
mGEN: Multilingual Answer Generator
Pimatts
a Frankie Sullivan
eye of the tiger (whu wrole thar strigg the tiguz)
g/x quíén escribió la canción
Question Encoder
Eye the Tiger
Bye of the Tiger
opot niuaz 1 11n000 yarrano trsmmasn 12 1937
Maximum Inner
Product Search
Cmuutii
Passage Encoder
Eye Tiger
Generator
<Bye the Tigers fue escrita Frankic Sullivan y Jim Peterik
Figure 1: Overview of CORA (mDPR and mGEN).
retrieval module (mDPR) produces dense embeddings of a question and all multilingual passages, thereby retrieving passages across languages. The generation module (mGEN) is trained to output an answer in the target language conditioned on the retrieved multilingual passages. To overcome the aforementioned data scarcity issue, we automatically mine training data using external language links and train mDPR and mGEN iteratively. particular, each iteration proceeds over two stages of updating model parameters with available training data and mining new training data cross-lingees pross-lingually by Wikipedia language links and predictions made by the models. This approach does not require any additional human annotations machine translation, and can be applied to many new languages with low resources.
Our experiments show that CORA advances the state of the art on two multilingual open QA datasets, XOR-TYDI QA (Asaí et al. 202T) and MKQA (Longpre et al. 2020), across 26 typologically diverse languages; CORA achieves gains of 23.4 and 4.7 Fl points in XOR- RYDTT QA and MKQA respectively, where MKQA data is not used for training. Moreover, CORA achieves F1 scores of roughly 30 over 8 languages documents, outperforming the state-of-the-art approach by 5.4 points. Our controlled experiments and human analyses illustrate the impact of many-to-many cross-lingual retrieval in improving multilingual open QA performance. We further observe that through cross-lingual retrieval, CORA can find answers to 20% of the multilingual questions that are valid but are originally annotated as unanswerable by humans due to the lack of evidence in the English knowledge sources.
MKQA that have no training data or even reference Wikipedia
2 Method
We define multilingual open QA the task of answering question q r. in target language J. given collection of multilingual reference passages mudtti where evidence passages can be retrieved from any language. These passages come from Wikipedia articles that are not necessarily parallel over languages. We introduce CORA, which runs a relrieve-then- seherrte procedure to achieve this goal (Fig.) We further introduce a novel training scheme of iterative training with data mining ($ 2.2
2.1 CORA Inference
CORA directly retrieves evidence passages from any language for questions asked in any target language, and then generates answers in the target language conditioned on those passages. More for- mally, the CORA inference consists of two steps of (i) retrieving passages mmmttii and (ii) generating an answer L based on the retrieved passages. Pp" pmmaiss can be in any language included in mullt muult mDPR(g L Cmuultty ), d mGEN(g", p
Multilingual Dense Passage Retriever (mDPR). mDPR extends Dense Passage Retriever (DPR; Karpukhin et al. 2020) to a multilingual setting. mDPR uses an iterative training approach to fine-tune pre-trained multilingual language model (e.g., mBERT; Devlin et al. 2019) to encode passages and questions separately. Once training is done, the representations for all passages from Corulli are computed offline and stored locally. Formally, passage encoding is obtained as follows: ey' mBERT, (p). where a passage p fixed-length sequence of tokens from multilingual documents. At inference, mDPR independently obtains d-dimensional (d 768) encoding of the

---

Training
Crose-Niognal Mining
Iteratively training
Training
Labeling
Inference
matti
EAAN outpuss:
Positive
mGEN,
Cmultti
mDPR,
mDPR
WikiDobs
Negative
a, PPotes Initad QA data
l %,. p
Ia
mDDR retrieved data
Wikilata-boued data
New training data
Figure 2: Overview of CORA iterative training and data mining.
question C² mmBERT, ). It retrieves k passages with the k highest relevance scores to the question, where the relevance score between a passage p and question q² is estimated by the inner product of their encoding vectors, (e,", ep))
Multilingual Answer Generator (mGEN), We multilingual sequence-to-sequencc model (e.g., mT5; Xue et al. 202T) to generate answers in the target language token-by-token given the retrieved multilingual passages pmmutes We choose generation approach because it can generate an answer in the target language I. from passages across different languages) Moreover, the generator can be adapted to unseen languages, some of which may have little or no translation training data. Specifically, the generator outputs the sequence probability for follows:
P(a'la", pmultti [ Iplallaas pranitt).
where of denotes the i-th token in the answer, and T is the length of the answer. We append language tag to the question to indicate the target language.
2.2 CORA Training
We introduce an iterative training approach that encourages cross-lingual retrieval and answer generation conditioned on multilingual passages (sketched in Fig. and Alg. []. Each iteration proceeds
stages: parameter updates 2.2. where mDPR and mGEN are trained on the current training data and cross-lingual data mining ($ (.2.2) where training data are automatically expanded by Wikipedia language links and model predictions.
Initial training data. The initial training data combination of multilingual QA datasets: XOR- TYDI QA and TYDI QA (Clark et al. (022)) and an English open QA dataset (Natural Questions, Kwiatkowski et al., 2019). Each training instance from these datasets comprises question, a positive passage, and an answer. Note that annotations in the existing QA datasets have critical limitations: positive passages
taken either from English (Asai al. 202 l) or the question's 's language (Clark) et al. 2020). Further, most of the non-English languages are not covered. Indeed, when we only train mDRR on this initial set, often learns to retrieve passages in the same languages or similar languages with irrelevant context or context without sufficient evidence to answer.
2.2.1 Parameter Updates
mDPR updates (line in Alg. Let (la' P".,,. be training instances. Each instance consists of a question 4." passage that answers the question (positive passage) Pi and n passages that do not answer the question (negative passages) Pi.j- For each question, we use positive passages for the other questions in the training batch as negative passages (in-batch negative, Gillick et al., 2019 Karpukhin et al. 2020). mDPR is updated by minimizing the negative log likelihood of positive passages:
exp((e,,, ep:)) exp((e, ep; Σn= exp((e, epi,,
Emdgr log
(2)
An alternative approach of answer extraction requires translation for all language pairs Asai al.
2021

---

Algorithm 1: Iterative training that automatically mines training data.
Data: Input QA pairs: (q" (g",a") )²)
initialize training data B² = (q" a +Poota), L = (Eag, L); while t < T do
O'DPR= + Train(fmbbRt B') Train MDPR
pmmulii - mDPR(q embedding( (mmulti)) ))/x Retrieve passages
OnGEN - Train (rriinmmmmn (q", aa",", ppmitti)// Train mGEN
For L == Eng. pmmdhi + = LangTink(q"" Cumull)) /* Mine data using Wikidata For P1 E pmmutt if MGGN((" pp)) == a" then posiliees.add(())) else negalives.add((p) taaaiiiie,ng / Add new training data
B
+= (9",
f t-
10 end
mGEN updates (lines 4-5 in Alg. D. After updating mDPR, we use mDPR to retrieve top k passages P pmmttt for each q". Given these pairs of the question and the retrieved passages (4 pmultti) as input, mGEN is trained to generate answer a autoregressively (Eq. (1) and minimize the cross- entropy loss. To train the model to generate in languages not covered by the original datasets, we translate a I to other languages using Wikipedia language links and create new synthetic answers See Appendix $ A.2 for more detail.
2.2.2 Cross-lingual Data Mining
After the parameter updates, we mine new training data using mDPR and Wikipedia language links and label the new data by mGEN predictions. This step skipped the final iteration.
Mining by trained mDPR and language links (line 4, 6 in Alg. D. Trained mDPR can discover positive passages in another language that is not covered by the initial training data. At each iteration, we use retrieved passages pmmultii for q (line 4 in Alg. as a source of new positive and negative passages. This enables expanding data between language pairs not in the original data.
To cover even more diverse languages, we use language links and find passages in other languages that potentially include sufficient evidence to answer. Wikipedia maintains article-level language links that connect articles on the same entity over languages. We use these links to expand training data from the English QA dataset of Natural Questions (line 6 in Alg. TT. Denote training instance by (q En En links. We use language links again to look up the English Wikipedia article that the gold passage Pgotd comes from. We then find articles in non-Englon non-English languages in the reference documents miuite that correspond to this article. Although the language link-based automatic translation cannot handle non-entity answers (e.g., short phrases), this helps us to scale to new languages without additional human annotation or machine translation. We add all passages from these articles to p muulti as positive passage candidates, which are then passed to mGEN to evaluate whether each of them leads to a not.
.Pgold). We first translate the English answer En to a target language L using language
.a
Automatic labeling by mGEN predictions (lines 7-8 in Alg. passage p; from mitti may not always provide sufficient information to answer the question even when it includes the answer string a To filter out those spurious passages (Lin et al. 2018 Min et al. 2019), we take instances generated from the two mining methods described above, and run mGEN on each passage predict an answer for the question. the answer matches the correct answer a', then the passage p; is labeled positive passage; otherwise we label the input passage negative passage. We assume that when mGEN fails to generate a correct answer given the passage, the passage may not provide sufficient evidence to answer, this helps filter out spurious passages that accidentally contain an answer string yet do not provide any clue to answer. We add these new positive and negative passages to the training data, and in the next iteration, mDPR is trained this expanded training set 2.2.1
2 "This automatic answer translation is only done after the third epoch of initial training to prevent the model from overfitting synthetic data.

---

3 Experiments
We evaluate CORA on two multilingual open QA datasets across 28 typologically diverse languages CORA achieves state-of-the-ar art performance across 26 languages, and greatly outperforms previous approaches that use anguagg-speccife components such as question or answer translation.
3.1 Datasets and Knowledge Sources
Multilingual open QA datasets differ in covered languages, annotation schemes, and target application scenarios. We evaluate FI and EM scores over the questions with answer annotations from two datasets, following the common evaluation practice in open QA (Lee et al. 2019).
XOR-TYDI QA. XOR-TYDI QA (Asai et al. 2021) is a multilingual open QA dataset consisting of typologically diverse languages, where questions are originally from TYDI QA (Clark al., 2020)
and posed by information-sedd seeking native speakers. The answers are annotated by extracting spans from Wikipedia in the same language as the question (in-language data) or by translating English spans extracted from English Wikipedia to the target language (cross-lingual data). XOR-TYDI QA offers both training and evaluation data.
MKQA. MKQA (Longpre et al. (020)) is an evaluation dataset created by translating 10k Natural Questions Wiiatkkwski et al. 2019) 25 target languages. The parallel data enables us compare the models' performance across typologically diverse languages, in contrast to XOR-TYDI QA. MKQA has evaluation data only; XOR-TYDI QA and MKQA have five languages common.
Collection of multilingual documents m We use the February 2019 Wikipedia dumps of 13 diverse languages from all XOR-TYDI QA languages and a subset of MKQA languages We choose 13 languages to cover languages with a large number of Wikipedia articles and a variety of both Latin and non-Latin scripts. We extract plain text from Wikipedia articles using wikiextractor and split each article into 100-token segments as in DPR (Karpukhin et al. 20200) We filter out disambiguation pages that distinguish pages that share the same article title well as pages with fewer than 20 tokens, resulting in 43.6M passages. See more details in Appendix $ B.2
Language categories. To better understand the model performance, we categorize the languages based on their availability during our training. We call the languages with human annotated gold paragraph and answer data seen languages. XOR-TYDI QA provides gold passages for 7 languages. For the languages in mr without human-annotated passages, mine new mDPR training data by our iterative approach ($ 2.2). We call these languages, which are seen during mDPR training, mDPR-seen. We also synthetically create mGEN training data explained in $ 2.2. by simply replacing answer entities with the corresponding ones in the target languages. The languages that are unseen by mDPR but are seen by mGEN are called mGEN-seen, and all other languages (i.e.. included neither in mDPR nor mGEN training; 9 of the MKQA languages) unseen languages.
3.2 Baselines and Experimental Setting
We compare CORA with the following strong baselines adopted from Asai et al. (2021).
Translate-test (MT DPR). As used in most previous work (s.g..Asai Asai al. 2021). this method
translates a question to English, extracts an answer in English using DPR, and then translates the answer back the target language. The translation models are obtained from MarianMT (Junczys Dowmunt et al. 2018) and trained on the OPUS-MT dataset (Tiedemann) 2012).
Monolingual baseline (BM25). This baseline retrieves passages solely from the target language and extracts the answer from the retrieved passages. Training neural network models such as DPR is infeasible with few thousands of training examples. Due to the lack of training data in most of
A full list of language families and script types are the appendix.
'Downloaded from http:////eeaoooooo/////////////t
year33A3222019322
htttp:://en. http://///w..oooooa///////////aa
5

---

Models
Target Language
Fl
Macro Average
Bn
Ja
Ko
Te
F1
EM
BLEU
CORA
59.8
40.4
42.2
44.5
27.1
45.9
44.7
43.5
33,5
31.1
SER
32.0
23.1
23.6
14.4
13.6
11.8
22.0
20.1
13.5
20.1
GMT+GS
31.5
19.0
18.3
8.8
20.1
19.8
13.6
18.7
12.1
16.8
MT+Mono
25.1
12.7
20.4
12.9
10.5
15.7
0.8
14.0
10.5
11.4
MT+DPR
7.6
5.9
16.2
9.0
5.3
5.5
0.8
7.2
3.3
6.3
BM25
31.1
21.9
21.4
12.4
12.1
17.7
Closed-book
14.9
10.0
11.4
22.2
9.4
18.1
10.4
13.8
9.6
7.4
Table I: Performance XOR-FULL (test data F1 scores and macro-averaged F1, EM and BLEU scores). "GMT+GS" denotes the previous state-of-the-are model, which combines Google Custom Search in the target language and Google Translate English DPR for cross-lingual retrieval (Asai eet al. 2021). Concurrent to our work, "SER" is state-of-the-arr model, Single Encoder Retriever, submitted anonymously on July 14 to the XOR-FULL leaderboard (ht /nlp. cs washington edu/xorqa/ We were not able to find a BM25 implementation that supports Telugu.
the target languages, we use BM25-based lexical retriever implementation by Pyserini (Lin et al. 2021]. We then feed the retrieved documents multilingual QA model to extract final answers.
MTnoono. This baseline combines results from the translate-test method and the monolingual method to retrieve passages in both English and the target language. Following Asai et al. (2021). we prioritize predictions from the monolingual pipeline they are over certain threshold tuned on XOR-TYDI QA development set; otherwise we output predictions from the translate-test method
Closed-book baseline. This model uses an mT55base question as input and generates answer in the target language without any retrieval at inference time (Roberts et al.) 2020). This baseline assesses the models' ability to memorize and retrieve knowledge from its parameters without retrieving reference documents.
model that takes a
CORA details. For experiments, we use a single retriever (mDPR) and a single generator (mGEN) that use the same passage embeddings. mDPR uses multilingual BERT base uncased] and the generator fine-tunes mT5-base. We found that using other pre-trained language models such as mBART (Liu et al. 2020) for mGEN or XLM-R (Conneau et al. 2020) for mDPR did not improve performance and sometimes even hurt performance. We first tine-tune mDPR using gold passages from Natural Questions, and then further fine-tune it using XOR-TYDI QA and TYD1 QA's 's gold passage data. We exclude the training questions in Natural Questions and TYDI QA that were used to create the MKQA or XOR-TYDI QA evaluation set. We run two iterations of CORA training ($ (.2) after the initial fine-tuning. All hyperparameters are Appendix $ B.5
Results and Analysis
4.1 Multilingual Open QA Results
XOR-TYDr QA. Table r reports the scores of CORA and the baselines in XOR-TYDI QA. CORA, which only uses single retriever and a single generator, outperforms the baselines and the previous state-of-the-art model on XOR-TYDI QA by large margin across all 7 languages. CORA achieves gains of 24.8 macro-averaged points over the previous state-of-the-art method (GMT+GS), which uses external black-box APIs, and 23.4 points over the concurrent anonymous work (SER).
MKQA. Tables 2 and report the Fl scores of CORA and the baselines on over 6.7k MKQA questions with short answer annotation under seen and unseen settings. CORA significantly
10
For the languages not supported by Pyserini, we always output translate-test's predictions.
WW did not use hizerr-izzed variants duc computational budget.
'The alternative of
(Cnneaa et al. 2020) did not improve our results.
10 Following previous work in open QA but different from the official script of MKQA (Longpre et al 2020). we disregard the questions labeled as "no answer". : As shown in our human analysis. it is difficult to prove an answer does not exist in the millions of multilingual documents if the annotation says

---

Setting
Seen (Included in DI QA)
mDPR-seen
Avg. over
En
Ar
Fi
Ja
Ko
CORA
21.8
40.6
12.8
26.8
19.7
12.0
19.8
32.0
30.9
15.8
8.5
MT+Mono
14.1
19.3
6.9
17.5
9.0
7.0
10.6
21.3
20.0
8.9
8.3
MT+DPR
17.1
43.3
16,0
21.7
9.6
5.7
17.6
28.4
19.7
8.9
6.9
BM25
19.4
5.9
9.9
9.1
6.9
8.1
14.7
10.9
4.9
Closed
4.5
8.0
4.6
3.6
6.5
4.1
2.1
Table 2: F1 scores on MKQA seen and mDPR-seen languages.
Setting
Unseen
Da
De
Fr
It
NI
P1
Pt
Hu
Vi
Ms
Km
No
Tr
lik
[w
CORA
30.4
30.2
30.8
29.0
32.1
25.6
28.4
18.4
20.9
27.8
5.8
29.2
22.2
5.2
6.7
5.4
MT-Mono
20.9
24.6
17.4
16.6
3.8
MT+DPR
26.2
25.9
21.9
25.1
28.3
24.6
24.7
15.7
15.1
12.6
18.3
18.2
3.8
3.8
BM25
9.5
12.5
13.6
12.8
134
7.4
9.4
8.8
2.8
3.3
Closed
4.7
5.3
4.0
44
533
3.8
2.3
2.4
"en": "Zhien" (Chinese, simplifedd) "bk": : "Zh.kk" (Chinese, Hong Kong). "tw":"Z Iw" (Chinese, Iraditiona).
Table 3: Fl scores on MKQA in mGEN-seen and unseen languages.
outperforms the baselines in all languages by large margin except for Arabic and English. Note that [ongpre et al.) (2020) report results simplified setting with gold reference articles from original Natural Questions dataset given in advance, and thus their results
not comparable. CORA
yields larger improvements over the translate-test baseline in the languages that are distant from English and with limited training data such Malay (Ms; 27.8 vs. 12.6) and Hebrew (He; 15.8 vs. 8.9). The performance drop of the translate-tes: model from English (43.3 F1) to other languages indicates the error propagation from the translation process. BM25 performs very poorly in some low-resource languages such as Thai because the lack of answer content in the target languages' Wikipedia. MToO underpeforms the MT+DPR baseline in MKQA since it challenging to rerank answers from two separate methods with uncaliberated confidence scores. In contrast, CORA retrieves passages across languages, achieving around 30 Fl majority of the 26 languages.
4.2
Analysis
Setting
XOR-TYDI QA
MKQA
Avg.
Ar
Ja
Te
Avg. F1
Fi
Ru
Es
Th
Vi
CORA
31.4
42.6
33.4
26.1
22.3
25.9
20.6
33.2
6.3
22.6
(i) mDPR, mGEN,
27.9
36.2
29.8
21.1
17.3
23.1
13.1
28.5
5.7
18.6
(ii) DPR (trained NQ)+mGEN
24.3
30.7
29.2
19.0
17.9
20.1
16.9
29.4
5.5
18.2
(iii) CORA, Cm
19.1
20.5
23.2
11.5
20.5
24.7
15.4
28.3
8.3
21.9
(iv) InDR--Ext.raddr-MM
11.2
11.8
10.8
5.6
12.2
16.1
10.9
25.2
1.2
12.7
Table 4: Ablation studies on XOR-TYDI QA development set and a subset of MKQA.
Impact of CORA components. We compare CORA with the following four variants to study the impact of different components. (i) mDPR, + mGEN, only trains CORA using the initial labeled, annotated data and measures the impact of the iterative training. (ii) DPR (trained NQ) + mGEN replaces mDPR with a multilingual BERT-based DPR trained English data from Natural
Ablations:
Questions (NQ), and encodes all passages in C montti This configuration assesses the impact of cingsllingual training data. (iii) CORA, mollti =(En} only retrieves from English during inference. This variant evaluates English reference documents suffice answer multilingual questions. (iv) mDPR+Ext.C DOPR-Exxtreeddrr-MF replaces mGEN with an extractive reader model (Karpukhin et al. (0200) followed by answer translation. This variant quantifies the effectiveness of using multilingual generation model over the approach that combines an extractive reader model with language-specifie translation models. Note that for MKQA experiments, we sample the same 350 questions (~5%) from the evaluation set for each language reduce the computational cost over varying configurations.

---

34
50
54.5
54.5
5
77
4
30
47
0.5
72 1
00
B1
34
0
71.3
86.0
85
62.1
7
5.0
1.7
5.7
1.6
3.6
0.8
0.1
8
6.8
2.7
3.5
42
4.9
Ja Es
0.5
40.8
1.7
0.2
0.2
0.3
0.3
0.0
0.6
1.6
0.3
1.4
0.8
1.1
SV
0.3
1.3
38.1
0.3
0.4
0.6
02
0.0
0.7
7.4
0.3
0.7
0.6
0.7
retrieval errors
28 48
ja
0.0
0.1
0.0
37
0.0
0.1
0.1
0.0
0.0
0.0
2.9
0.3
0.0
0.1
different lang
18 0
KD
0.1
0.1
0.1
0.1
18.
0.1
0.1
0.0
0.1
0.1
0.1
0.1
0.1
0.1
incorrect answer
22 36
ru
0.9
0.9
1.1
0.3
64.0
0.5
0.0
0.9
1.0
0.5
0.4
0.7
he
0.4
0.2
0.5
02
0.1
0.1
48.6
0.0
0.6
0.3
0.2
02
0.3
0.2
annotation error
22 12
th
0.0
0.0
0.0
00
022
0.
0.0
99
4
0.0
0.0
0.0
0.0
0.0
underspecified q 10
id
0.8
1.1
2.1
06
0.7
0.6
1.1
0.0
1,7
2.3
1.6
12
22.2
3.1
es
ru
he
pt
no
Table 6: Error categories (%) on 50 errors sampled from Japanese (Ja) and Spanish (Es) data.
Figure 3: Breakdown of the languages of retrieved reference pas- sages for sampled MKQA questions (%). The x and y axes indicate target (question) and retrieval reference languages respectively.
Results in Table 4 show performance drops in all variants. This supports the following claims: (i) the iterative learning and data mining process is useful, (ii) mDPR trained with cross-lingual data substantially outperforms DPR with multilingual BERT trained on monolingual data only, (iii) reference languages other than English are important in answering multilingual questions, and (iv) multilingual generation model substantially boosts the model performance.
Setting Lang Script
mDPR-Seen
Unseen
Es
Fi
Ja
Ru
Th
Pt
Ms
Tr
Zh-Cn
Zh-Hk
Km
Lain
Jpan
Cyrl
Thai
Lain
Hant
Khmr
2'810
53.7
52.8
32.9
42.3
14.9
50.0
49.4
42.0
12.6
16.6
15.7
mDPR
ruies 810
63.4
60.9
42.0
54.0
28.0
62.6
63.4
55.4
40.6
42.3
25.1
[
R R
52.3
46.0
24.6
36.0
12.6
45.7
48.8
32.0
9.1
14.0
13.4
810
63.1
53.1
32.9
49.1
29.4
56.8
58.0
44.0
36.3
39.4
23.4
Table 5: Retrieval recall performance on MKQA the percentage of the questions where least out of the top 10 passages includes an answer string in the target language (R " ee),), in any language (R mimiiii @1 0). The same subset of the MKQA evaluation data used in the ablations.
Retrieval performance and relationship to the final QA performance. We evaluate CORA's retrieval performance MKQA using two recall metrics that measure the percentage of questions with at least one passage among the top 10 that includes a string in an answer set in the target language (R* (R*@10) or in the union of answer sets from all languages that are available in MKQA (R" muiti MKQA provides answer translations across 26 languages.
@ (10).
Table reports retrieval results for mDPR and multilingual BERT-based DPR trained on NQ: DPR (NQ). This is equivalent (ii) from the ablations. We observe that mDPR performs well in Indo- European languages with Latin script, even when the language is unseen. Interestingly, there is a significant performance gap between R @10 0 and R multi 310 0 in languages with non-Latin script (e.g.. Japanese, Russian, Chinese); this suggests that our model often uses relevant passages from other languages with Latin script such as English or Spanish to answer questions in those languages with non-Latin script. Our mDPR outperforms DPR (NQ) by a large margin in unseen languages with limited resources, which are consistent with the findings in Table 3 Nevertheless, we still see low performance on Khmer and Thai even with the R maiti @10 metric. We also observe that passage and query embeddings for those languages are far from other languages, which can be further studied in future work. We provide two-dimensional visualization of the encoded passage representations in the appendix.
Breakdown of reference languages. Fig. breaks down retrieved reference languages for each target language. Our multilingual retrieval model often retrieves documents from the target language (if its reference documents available), English, its typologically similar languages. For example, mDPR often retrieves Spanish passages for Portuguese questions and Japanese passages for Chinese questions; while they are considered phylogenetically distant, Japanese and Chinese overlap in script.
To further evaluate this, we conduct controlled experiment: we remove Spanish, Swedish and Indonesian document embeddings and evaluate CORA on related languages: Danish, Portuguese and
8

---

Exampler: French Question:
Exampley: Norweigian Question
Examples: Japanese Question
gi'; Quand est-ce que The life of pablo sorti? (when was The life pablo releasedr) a" 14 fevtier 2016
g/T; hveem spiller black panther filmen black panther (who plays black pontherin in the monts black panthee)
2:1 Malla/ttt /2/ 2
(wlan diss the third musoon of rick and morty coms ou)
"t"; Chadwick Buserman
mbPR
a= Answer MKQA
The Life of Pablo ru.rrkipodia
The Life of Pablo" Gass samyuee 1d
Pantera Negra (pelicula) wikipeddii
duenpa 2016 (The Liks Pablo
protagonizada Chadwick Boseman como TChalla/ Black Panther stars Charbrick Rosemmnn as (Challs/ / Black Panther)
Dy 92 ja.ciiiipedia
released Eebeurry 2016)
The Life of Pablo- sw.ukkippeia
y w1w.://ooo5//111. A6.
The Life of Pablo var planerat att alippas fabruari 2016... slayptss slutligen
Black Panther (film) swaeikipedda
S (Busk Morty's
fehruarl 2016 (The Publlo... waw schedhild release on February 11, 2016 Suuansh
Huvudrollen soe Black Panther spelis av Chadwick Roseman (The main rule as Black Funther is played by Chodsick Roseman)
siring 1.00)
was finally released on Jehmuree is. pont))
mGAN
Pred: 14 Dempant 2016
X
Pred: Chadwick Boseman
Pred: 2
Figure 4: Cross- lingual retrieval and generation examples for three MKQA questions.
Malay. We observe performance drops of 1.0 in Danish, 0.6 in Portuguese, and 3.4 F1 points in Malay, This illustrates that while CORA allows for retrieval from any language in principle (many-to-many), cross-lingual retrieval from closer languages with more language resources is particularly helpful.
Error analysis and qualitative examples. Table 6 analyzes errors from CORA by manually inspecting 50 Japanese and Spanish wrong predictions from MKQA. We observe six major error categories: (a) retrieval errors, (b) generating correct answers in a different language (different lang), (c) incorrect answer generation (incorrect answer). (d) answer annotation errors (e.g., a correct alias isn't covered by gold answers, or Wikipedia information is inconsistent with English.), and (e) ambiguous or underspecified questions such as "who won X this year" (underspecified q). The table shows that both in Japanese and Spanish, the retrieval errors are dominant. In Japanese, CORA often generates correct answers in English, not in Japanese (different lang).
Fig. 4 shows some qualitative examples. The first example shows error in (b): mGEN is generating an answer in Russian, not in French though the answer itself is correct. This type of error happens especially when retrieved passages are in languages other than the target and English.
Human evaluation on cross-lingual retrieval results. To observe how cross- pooss-lnngual retrieval between distant languages is actually helping, we sample 25 Norwegian questions for which Spanish passages are included among the top 10 retrieved results. As seen in Fig. 3 CORA retrieves Spanish (es) passages for 6.8% of the Norwegian (no) questions. A Spanish speaker judges if the retrieved Spanish passages actually answer the given Norwegian questions 11| cases, the retrieved Spanish passages are relevant in answering the question. One such example is presented in Fig. (the second example).
We found that in 96% of the
Human analysis on unanswerable questions. CORA retrieves passages from larger multilingual document collection than the original human annotations. Thus, CORA may further improve the answer coverage over the original human annotations. MKQA includes questions that are marked as unanswerable by native English speakers given English knowledge sources. We sample 400 unanswerable Japanese questions whose top one retrieved passage from a non- English Wikipedia article. Among these, 329 unanswerable questions are underspecified (also discussed in Asai and Choi 202T). For 17 out of the 71 remaining questions, the answers predicted by CORA are correct. This finding indicates the significance of cross-lingual retrieval and potential room for improvement in annotating multilingual open QA datasets. The third example Fig. 4 shows one of these cases.
Related Work and Broader Impacts
English and non- non-English open QA. Despite the rapid progress in open QA (Chen et al. 2017 Karpukhin et al. (2020), most prior work has been exclusively on English (Lewis et al. 2020 Izacard and Grave 2021b). Several prior attempts to build multilingual open QA systems often rely on machine translation or -nguage--ppecficc retrieval models (Ture and Boschee 2016 Asai et al. 2021
During evaluation, we provide the original English questions from MKQA.

---

approaches. Izacard and Grave (202 a) introduce an iterative training framework that uses attention weights from a generator model as a proxy for text relevance scores. [Tran et al. (022)) introduce CRISS, a self-supervised pre-training approach consisting of parallei sentence mining module and a sequence-to-seque -e-euunece model, which are trained iteratively. Several recent work such as Xiong et al. (202T) improves DPR by mining and learning with hard examples. Our work is the first work that introduces a unified multilingual system for many-to-mann open QA, which is challenging task requiring massive-scale cross-lingual retrieval and has not been addressed in prior work. We introduce an iterative training and data mining approach guided by filtering from answer generation model to automatically extend annotated data available only in high-resouree languages to low-ress reworrcee.. This approach contributes to significant performance improvements in languages without annotated training data.
Lewis et al. ) and Guu et al. (2020) introduce retrieve-then-gene
Many-languages-one models. Several recent work introduces single multilingual models for many languages using pre-trained multilingual models such as mBERT or mT55 in many NLP tasks (e.g.. entity linking: |Botha et al. 2020 De Cao et al.) 202T; semantic role labeling: Mulcaire et al.) 2019b Lyu al. 2019; Fei et al., 2020, syntactic parsing: Mulcaire et al., 2019a Kondratyuk and Straka 2019). This work conducts the first large-scale study of a unified multilingual open QA model across many languages and achieves state-of-the-an performance in 26 typologically diverse languages.
Synthetic data creation for machine reading comprehension. Alberti et al. (2019) introduce a method of generating synthetic machine reading comprehension data by automatically generating questions and filtering them out by trained machine reading comprehension model. Several studies augment multilingual machine reading comprehension training data by generating new question- answer pairs from randomly sampled non- non-English Wikipedia paragraphs (Riabi et al. 2021 | et al.] (020). This work focuses on multilingual open QA, which involves not only machine reading comprehension but also cross-lingual retrieval. A similar augmentation method for machine reading comprehension can be applied to further improve the answer generation component in CORA.
Societal impacts. Our code and data are publicly available. CORA can perform open QA in unseen languages and can benefit society in building QA systems for low-resource languages, hence enabling research in that direction. Unlike previous models, CORA removes the necessity of external black-box APIs, and thus we can examine and address wrong answers due to model errors or misinformation present on Wikipedia. This would help us mitigate the potential negative impact from CORA or its subsequent models outputting wrong answer when it is used by people who seek information.
6 Conclusion
To address the information needs of many non-English speakers, a QA system has to conduct cross- lingual passage retrieval and answer generation. This work presents CORA, a unified multilingual many-to-many open QA model that retrieves multilingual passages in many different languages and generates answers in target languages. CORA does not require language-specific translation or retrieval components and can even answer questions in unseen, new languages. We conduct extensive experiments on two multilingual open QA datasets across 28 languages, 26 of which CORA advances the state of the art on, outperforming competitive models by up to 23 Fl points. Our extensive analysis and manual evaluation reveal that CORA effectively retrieves semantically relevant passages beyond language boundaries, and can even find answers to the questions that were previously considered unanswerable due to lack of sufficient evidence in annotation languages (c.g.. English). Nonetheless, our experimental results show that the retrieval component still struggles to find relevant passages for queries in some unseen languages. Our amalysis also showed that CORA sometimes fails to generate an answer in the target language. In future work, we aim to address these issues to further improve the performance and scale our framework to even more languages.
Acknowledgement
This research was supported by NSF IIS- 2044660, ONR N00014-18-1-2826 gifts from Google, the Allen Distinguished Investigator Award, the Sloan Fellowship, and the Nakajima Foundation Fellowship. We thank anonymous reviewers, area chairs, Eunsol Choi, Sewon Min, David Wadden, and the members of the UW NLP group for their insightful feedback this paper, and Gabriel Ilharco for his help on human analysis.
10

---

References
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019, Synthetic QA corpora generation with roundtrip consistency In ACL.
Akari Asai and Eunsol Choi. 2021. Challenges in information seeking QA: Unanswerable questions and paragraph retrieval ACL.
Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingenn open-retrieval question answering. In NAACL.
Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020. Entity Linking in 100 Languages
In EMNLP.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer
open-domain questions. In ACL.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkosski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages, TACL.
Alexis Conncau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.
Unsupervised liossllingual representation learning at scale In ACL.
Nicola De Cao. Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov,
Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2021. Multilingual autoregressive entity linking.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, BERT: Pre-training of deep bidirectional transformers for language understanding, In NAACL.
Hao Fei, Mcishan Zhang, and Donghong Ji. 2020. [Cross-lingual semantic role labeling with high quality translated training corpus In ACL.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and
Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval In CoNL.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wer Wei Chang. 2020. Realm
[ettiieal-auugeeetee language model pre-training In ICML.
Gautier Izacard and Edouard Grave. 2021a. Distilling knowledge from reader to retriever for question answering. In ICLR.
Gautier Izacard and Edouard Grave. 2021b. Leveraging passage retrieval with generative models for open domain question answering
In EACL.
Marcin
Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++ ACL (System Demonstrations).
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering In EMNLP.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Dan Kondratyuk and Milan Straka. 2019. 75 languages, model: Parsing Universal Dependencies universally In EMNLP.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstcin, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A benchmark for question answering research, TACL.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering In ACL.

---

Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal. Heinrich Küttler. Mike Lewis, Wen-tau Yih, Tim Rocktäschel. et al. 2020. generation for knowledge-intensivee NIP tasks In NeurIPS.
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserimi: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations/
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates, 2020. BERT and beyond
Pretrained transformers for text ranking
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering In ACL.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad. Mike Lewis. and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation TACL.
Shayne Longpre, Yi Lu, and Joachim Daiber. 2020. MKQA: A linguistically diverse benchmark for multilingual open domain question answering
Chunchuan Lyu, Shay B. Cohen, and Ivan Titov. 2019. Semantic role labeling with iterative structure refinement In EMNLP.
Sewon Min, Jordan Boyd-Graber Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, and Tom al Kwiatkowski. 2021. NeurlPS 2020 EfficientQA competition: Systems, analyses and lessons learned. In PMLR.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. discrete hard em approach for weakly supervised question answering. In EMNLP.
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019a. Low-resource parsing with crosslingual contextualized representations] In CoNLL
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019b. Polyglot contextual representations
improve crosslingual transfer In NAACL.
Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano.
2021. Synthetic data augmentation for zero-shot cross-lingual question answering In EMNLP.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In EMNLP.
Siamak Shakeri, Noah Constant, Mihir Sanjay Kale, and Linting Xue. 2020. Towards zero-shot multilingual synthetic question and answer generation for cross-lingual reading comprehension
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. comprehension questions easier? In EMNLP.
What makes reading
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS In LREC.
Jörg Tiedemann and Lars Nygaard. 2004. The OPUS corpus parallel and free In LREC.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT Building open translation services for the World In EAMT.
Chau Tran, Yuqing Tang. Xian Li, and Jiatao Gu. 2020. Cross-lingual retrieval for iterative self supervised training In NewrIPS.
Ferhan Ture and Elizabeth Boschee. 2016. Learning to translate for multilingual question answering In EMNLP.
12

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. (Transformers: State-of-the-art natural language processing In EMNLP (System Demonstrations).
Lee Xiong, Chenyan Xiong, Ye Li. Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval In ICLR.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mTS: A massively multilingual pre-trained texi-to-text transformer In NAACL.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section $ 4
(b) Did you describe the limitations of your work? [Yes] See Section $
$
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section $
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Our code and com/ (tt///aaraaaa/CRR)
trained model are publicly available at
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All hyperparameters will be included in appendix. Data splits are from the original datasets.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] That information is included in Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [Yes] See Section 3
(b) Did you mention the license of the assets? [Yes] Both two datasets are under MIT license and we discuss that information in Appendix.
(c) Did you include any assets either in the supplemental material [Yes] Our code and trained model are publicly available at at/htts:://go / httt::://githb.... com/
URL?
AkariAsal/CoRa,
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Section $
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] As the dataset used in this work is originally created for general purpose QA and we believe that those datasets contain few examples that risk our participants. We will describe that in details Appendix.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] We will include the instructions in Appendix.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes] As the dataset used in this work originally created for general purpose QA and we believe that those datasets contain few examples that risk our participants. We will describe that in details Appendix.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rvE7I0-iHxe", "reviewer": "Reviewer_XaUF", "summary": "The paper attempts to provide a solution to an interesting problem of cross language retrieval based question answering in a zero shot (without requirement of any language-specific annotated training data). Authors propose a new DPR algorithm that to be able to retrieve documents relevant to the question across languages (for multiple languages). The paper extends retrieve-the-generate based open domain QA system to a cross-lingual setting. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "How does the proposed approach differ in terms of techniques with something already proposed like RAG (https://arxiv.org/abs/2005.11401). It looks like the proposed approach is just a multilingual version of RAG thus the paper has limited novelty.  Does the system expects the  gold document annotations to be available for training the retriever module? Section 2.2., talks about positive passages, this requires huge time and effort for annotation, so how does authors claim in abstract that the proposed technique does not require any language specific annotated training data?  What happens in the scenario where a wikipedia page about a specific entity is not available in a particular language in which the question has been asked?  Section 2.2 needs to have some more intuition and clarity. What is intuition behind translating English answers to target language. Is there a mechanism to filter out spurious passages by the automatic passage labeling technique? How does the model learns cross lingual embeddings? do authors use a pre-trained model like m-bert? How does the proposed model results compare when compared with m-bert based methods? \n\nUpdate: I think the problem formulation is novel. So, after reading author responses I am increasing the score to 6. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the work has not been discussed by the authors. Authors did promise to make the code publicly available to general public to enable further research in the direction ( societal impact)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ri7x9kCJLeq", "reviewer": "Reviewer_qfFE", "summary": "This paper studies the task of multilingual Open-Domain Question Answering. \nThe task setting requires building a system that can answer natural language questions in any language on an open set of domains. Models are required to answer in the same language the question was asked in. \nThe authors tackle the task using a fairly conventional retrieve-and-read ODQA model, with the necessary modifications to make both the retriever and reader component language-agnostic, by exploiting multi-lingual pretrained models, training on available multilingual training data, and using and a clever data augmentation setup to increase multilingual training data. \n\nThe proposed solution “CORA”, operates over wikipedias in many different languages. \n\nThe retrieval component “MDPR” is a multi-lingual extension of DPR, using MBERT. This model can retrieve evidence documents across different languages, rather than having to match the language of the question. This means that a question expressed in a low-resource language can benefit from retrieving evidence from high resource languages. It also allows for a question written in one language on a cultural topic associated with a different language to retrieve from the most appropriate language’s wikipedia. \n\nThe reader “mGEN” is a generative reader, implemented using mT5, that takes as input the concatenation of the top-K retrieved passages from the retriever (which may be several different languages) and the question, and generates the answer in the question language, token by token. \n\nThe main difficulty of the proposed solution is how to train the MDPR and MGEN model components, given the limited multilingual raining data. The initial model is trained with NQ (english) and TyDi QA and Xor-TiDi QA (multilingual data).\nTo increase the amount of multilingual training data and language coverage,  the authors introduce a data augmentation strategy where the initial model is used to harvest potential additional data. This exploits the wikipedia “language links” resource. This iterative approach works by retrieving passages, looking up passages on the same topic in other languages using “language links”, and using the reader to see whether these can be used to generate the correct answer - if they can, they are added as additional training data for the next round. Another expansion mechanism exploits the fact that most answers are wikipedia entities, and can be translated using annotations available in the wikipedia “language links” resource.\n\nCORA is thoroughly evaluated on TyDI QA and an evaluation-only dataset called MKQA, and performs very strongly, improving the state-of-the-art by a wide margin in almost every setting..\nThorough ablations give insights and quantify the the effects of the system, and demonstrate zero-shot question answering abilities  \n", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an empirically very strong piece of work that performs sensible (if straightforward and somewhat incremental on the modelling side) extensions of work from open-domain QA and multilingual open-domain QA. However, the result is stronger than the sum of its parts, unlocking the ability to combine information across many languages’ wikipedias, and enjoy the resultant improvements in performance. This is a really good piece of work, and well executed. \n\nThe iterative training and data expansion techniques are interesting and important contributions, which leverage the wealth of annotations and metadata available in wikipedia to improve cross-lingual QA, without requiring explicit question translation or extra annotation efforts by humans. One limitation is that this won’t always work well if the language links aren’t high quality for some languages, or in other domains where language links aren’t available (either because wikipedia isn’t a good background corpus, or because answers are not wikipedia/wikidata entities).\n\nThe experiments and ablations are thorough and detailed and the analysis is deep. \nIn terms of some constructive criticism, the MGen model could probably be made more powerful by using the Fusion-in-Decoder architecture, which would allow the model to scale to reading more passages per question, which has shown to greatly improve accuracy on open-domain QA. \n\nI also wonder why the models’ performances  are so different and sometimes inconsistent between MKQA and XOR-full (comparing tables 1 and 2)  - for example, CORA scores 54.8 F1 for arabic on XOR-full, but loses a lot of performance arabic on MKQA (only 13.1 F1) . Conversely, MT+DPR performs poorly for arabic on XOR-full (7.6 F1) but  outperforms CORA on MKQA (16.0 F1).\n\nSummary:\n* Originality: This paper takes a standard ODQA approach, but converts it to a massively cross-lingual setting in a novel and very powerful way.\n( Quality: the experiments are well-thought out, the results are very strong, and the ablations and analysis are execllent\n* Clarity: The paper is very clear and well written\n* Significance: I expect this to be a high impact paper and serve as inspiration for many followup works and tasks.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a note on societal impacts which summarize the main impacts well. This kind of work will likely have a positive impact on societal equality, unlocking the strong ODQA performance we see in high resource languages for lower resource languages. There are risks that CORA may provide wrong answers that might be harmful or offensive, but this is the case for monolingual systems too. The authors rightly point out that not relying on external blackbox translators allows the model to be more interpretable and debuggable.\nThe use of wikipedia language links is useful, but does limit the textual domains the data expansion technique can be used for to similar domains to wikipedia.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "klEigqHs8gL", "reviewer": "Reviewer_hSDX", "summary": "This paper addresses the multilingual open QA task by combining a cross-lingual passage retrieval algorithm with a multilingual autoregressive generation model. The approach is thus capable of answering questions where the answer and the question are in different languages.  The authors extended existing annotated data using an iterative training approach that improves retrieval for languages with limited resources. The experimental results showed that the proposed method (CORA) outperformed the previous state of the art on multilingual open question answering benchmarks across 26 languages.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper targets at Multilingual Open Domain QA, which is a novel task with many application scenarios. The proposed method, CORA, showed relatively good performance on 26 languages on benchmark datasets. \nHowever, the overall idea is very similar to CRISS [1]. The CRISS approach consists of a cross-lingual retrieval module and a seq2seq module, and they are optimized alternately. Similarly, the CORA approach in this paper also consists of DPR (Dense Passage Retriever, mbert structure) and GEN(Answer Generator, mt5 structure), and they are also optimized alternately. Although CRISS [1] doesn’t target at multilingual open QA,  it has strong performance on cross-lingual sentence retrieval, which is related to multilingual open QA. The authors should compare their work with CRISS [1].\nAuthors claim that they introduce a new dense passage retrieval algorithm, but don’t mention any similar work about dense passage retrieval [2].  \nOverall, this paper combines DPR and GEN module and achieves good experimental results. However, the technical contribution of the paper is quite limited. \n[1] Cross-lingual Retrieval for Iterative Self-Supervised Training\n[2] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see potential negative social impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0aZCOTpjL-3", "reviewer": "Reviewer_nAYP", "summary": "This paper presents a multilingual dense passage retrieval model called mDPR to deal with the multilingual/cross-lingual IR and QA tasks. The mDPR part is firstly trained using existing labeled multilingual Q-P pairs. Then, possible passage pairs are selected from a given multilingual corpus based on the trained mDPR and used to train an answer generation model called mGEN. Next, given each question, the trained mDPR is further used to retrieve more possible passage candidates in other languages from Wikipedia corpus, where the retrieved passages that can lead to correct answers are considered as positive instances, the others as negative instances. With these expanded new question-passage pairs, the training of mDPR is performed again. The above procedure is conducted in an iterative manner. Evaluations are conducted on TyDi QA and MKQA in the open-domain QA setting. Several baselines are included and compared. The key contribution is to do DPR in the multilingual/cross-lingual setting.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a good paper that extends dense IR from monolingual to multilingual/cross-lingual. But the novelty is limited. The most important part, I think, is the data expansion part described in Section 2.2. From model perspective, there seems be have few innovation. The baselines compared in Table 1 and 2 are also weak. Why not compare with the submissions on the TyDi QA leaderboards? Why stronger multilingual pre-trained models such as XLM-R or mBART are not used and fine-tuned for comparison?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the problem of multilingual open domain question answering, introducing an iterative approach to mining answer passages.

There is not a clear consensus amongst the reviewers. It is clear that this is an interesting and under-explored problem, and that the proposed solution works well. Reviewer qfFE argues strongly for acceptance based on the novelty of the setting, the strong empirical comparisons, and the potential for inspiring future work.

The major criticism made of the paper is its lack of technical novelty, and reviewers agree that similar iterative retrieval ideas have been proposed in models such as RAG and CRISS. However, there is a non-trivial modeling contribution over these papers to extend them successfully to multilingual open domain QA. Reviewer nAYP also feels that the baselines are weak, but I agree with reviewer qfFE that the strong set of baselines is a significant contribution of the work.

Overall, I recommend acceptance.


Decision: Accept (Poster)