One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval

1 Introduction . Multilingual open question answering ( QA ) is the task of answering a question from a large collection of multilingual documents . Most recent progress in open QA is made for English by building a pipeline based on a dense passage retriever trained on large-scale English QA datasets to find evidence passages in English ( Lee et al. , 2019 ; Karpukhin et al. , 2020 ) , followed by a reader that extracts an answer from retrieved passages . However , extending this approach to multilingual open QA poses new challenges . Answering multilingual questions requires retrieving evidence from knowledge sources of other languages than the original question since many languages have limited reference documents or the question sometimes inquires about concepts from other cultures ( Asai et al. , 2021 ; Lin et al. , 2020 ) . Nonetheless , large-scale cross-lingual open QA training data whose questions and evidence are in different languages are not available in many of those languages . To address these challenges , previous work in multilingual open QA ( Ture and Boschee , 2016 ; Asai et al. , 2021 ) translates questions into English , applies an English open QA system to answer in English , and then translates answers back to the target language . Those pipeline approaches suffer from error propagation of the machine translation component into the downstream QA , especially for low-resource languages . Moreover , they are not able to answer questions whose answers can be found in resources written in languages other than English or the target languages . In this paper , we introduce a unified many-to-many QA model that can answer questions in any target language by retrieving evidence from any language and generating answers in the target language . Our method ( called CORA , Fig . 1 ) extends the retrieve-then-generate approach of English open QA ( Lewis et al. , 2020 ; Izacard and Grave , 2021b ) with a single cross-lingual retriever and a generator that do not rely on language-specific retrievers or machine translation modules . The multilingual 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . retrieval module ( mDPR ) produces dense embeddings of a question and all multilingual passages , thereby retrieving passages across languages . The generation module ( mGEN ) is trained to output an answer in the target language conditioned on the retrieved multilingual passages . To overcome the aforementioned data scarcity issue , we automatically mine training data using external language links and train mDPR and mGEN iteratively . In particular , each iteration proceeds over two stages of updating model parameters with available training data and mining new training data cross-lingually by Wikipedia language links and predictions made by the models . This approach does not require any additional human annotations or machine translation , and can be applied to many new languages with low resources . Our experiments show that CORA advances the state of the art on two multilingual open QA datasets , XOR-TYDI QA ( Asai et al. , 2021 ) and MKQA ( Longpre et al. , 2020 ) , across 26 typologically diverse languages ; CORA achieves gains of 23.4 and 4.7 F1 points in XOR-TYDI QA and MKQA respectively , where MKQA data is not used for training . Moreover , CORA achieves F1 scores of roughly 30 over 8 languages on MKQA that have no training data or even reference Wikipedia documents , outperforming the state-of-the-art approach by 5.4 F1 points . Our controlled experiments and human analyses illustrate the impact of many-to-many cross-lingual retrieval in improving multilingual open QA performance . We further observe that through cross-lingual retrieval , CORA can find answers to 20 % of the multilingual questions that are valid but are originally annotated as unanswerable by humans due to the lack of evidence in the English knowledge sources . 2 Method . We define multilingual open QA as the task of answering a question qL in a target language L given a collection of multilingual reference passages Cmulti , where evidence passages can be retrieved from any language . These passages come from Wikipedia articles that are not necessarily parallel over languages . We introduce CORA , which runs a retrieve-then-generate procedure to achieve this goal ( Fig . 1 ) . We further introduce a novel training scheme of iterative training with data mining ( § 2.2 ) . 2.1 CORA Inference . CORA directly retrieves evidence passages from any language for questions asked in any target language , and then generates answers in the target language conditioned on those passages . More formally , the CORA inference consists of two steps of ( i ) retrieving passages Pmulti and ( ii ) generating an answer aL based on the retrieved passages . Pmulti can be in any language included in Cmulti . Pmulti = mDPR ( qL , Cmulti ) , aL = mGEN ( qL , Pmulti ) . Multilingual Dense Passage Retriever ( mDPR ) . mDPR extends Dense Passage Retriever ( DPR ; Karpukhin et al. , 2020 ) to a multilingual setting . mDPR uses an iterative training approach to fine-tune a pre-trained multilingual language model ( e.g. , mBERT ; Devlin et al. , 2019 ) to encode passages and questions separately . Once training is done , the representations for all passages from C multi are computed offline and stored locally . Formally , a passage encoding is obtained as follows : epL = mBERTp ( p ) , where a passage p is a fixed-length sequence of tokens from multilingual documents . At inference , mDPR independently obtains a d-dimensional ( d = 768 ) encoding of the question eqL = mBERTq ( qL ) . It retrieves k passages with the k highest relevance scores to the question , where the relevance score between a passage p and a question qL is estimated by the inner product of their encoding vectors , ⟨eqL , ep⟩ . Multilingual Answer Generator ( mGEN ) . We use a multilingual sequence-to-sequence model ( e.g. , mT5 ; Xue et al. , 2021 ) to generate answers in the target language token-by-token given the retrieved multilingual passages Pmulti . We choose a generation approach because it can generate an answer in the target language L from passages across different languages.1 Moreover , the generator can be adapted to unseen languages , some of which may have little or no translation training data . Specifically , the generator outputs the sequence probability for aL as follows : P ( aL∣qL , Pmulti ) = T ∏ i p ( aLi ∣aL < i , qL , Pmulti ) , ( 1 ) where aLi denotes the i-th token in the answer , and T is the length of the answer . We append a language tag to the question to indicate the target language . 2.2 CORA Training . We introduce an iterative training approach that encourages cross-lingual retrieval and answer generation conditioned on multilingual passages ( sketched in Fig . 2 and Alg . 1 ) . Each iteration proceeds over two stages : parameter updates ( § 2.2.1 ) where mDPR and mGEN are trained on the current training data and cross-lingual data mining ( § 2.2.2 ) where training data are automatically expanded by Wikipedia language links and model predictions . Initial training data . The initial training data is a combination of multilingual QA datasets : XORTYDI QA and TYDI QA ( Clark et al. , 2020 ) , and an English open QA dataset ( Natural Questions , Kwiatkowski et al. , 2019 ) . Each training instance from these datasets comprises a question , a positive passage , and an answer . Note that annotations in the existing QA datasets have critical limitations : positive passages are taken either from English ( Asai et al. , 2021 ) or the question ’ s language ( Clark et al. , 2020 ) . Further , most of the non-English languages are not covered . Indeed , when we only train mDPR on this initial set , it often learns to retrieve passages in the same languages or similar languages with irrelevant context or context without sufficient evidence to answer . 2.2.1 Parameter Updates . mDPR updates ( line 3 in Alg . 1 ) . Let D = { ⟨qLi , p+i , p−i,1 , ⋯ , p−i , n⟩ } mi=1 be m training instances . Each instance consists of a question qLi , a passage that answers the question ( positive passage ) p + i , and n passages that do not answer the question ( negative passages ) p−i , j . For each question , we use positive passages for the other questions in the training batch as negative passages ( in-batch negative , Gillick et al. , 2019 ; Karpukhin et al. , 2020 ) . mDPR is updated by minimizing the negative log likelihood of positive passages : Lmdpr = − log exp ( ⟨eqLi , ep+i ⟩ ) exp ( ⟨eqLi , ep+i ⟩ ) +∑ n j=1 exp ( ⟨eqLi , ep−i , j ⟩ ) . ( 2 ) 1An alternative approach of answer extraction requires translation for all language pairs ( Asai et al. , 2021 ) . Algorithm 1 : Iterative training that automatically mines training data . Data : Input QA pairs : ( qL , aL ) 1 initialize training data B1 = ( qL , aL , pgold ) , L = { Eng , L } ; 2 while t < T do 3 Θ t mDPR ← Train ( θt−1mDPR , Bt ) / * Train mDPR * / 4 Pmulti ← mDPR ( qL , embedding ( Cmulti ) ) / * Retrieve passages * / 5 θ t mGEN ← Train ( θt−1mGEN , ( qL , aL , Pmulti ) ) / * Train mGEN * / 6 For L == Eng , Pmulti+ = LangLink ( qL , Cmulti ) ) / * Mine data using Wikidata * / 7 For pi ∈ P multi : if mGEN ( qL , pi ) == aL then positives.add ( pi ) else negatives.add ( pi ) 8 B t+1 += ( qL , aL , positives , negatives ) / * Add new training data * / 9 t← t + 1 10 end mGEN updates ( lines 4-5 in Alg . 1 ) . After updating mDPR , we use mDPR to retrieve top k passages Pmulti for each qL . Given these pairs of the question and the retrieved passages ( qL , Pmulti ) as input , mGEN is trained to generate answer aL autoregressively ( Eq . ( 1 ) ) and minimize the crossentropy loss . To train the model to generate in languages not covered by the original datasets , we translate aL to other languages using Wikipedia language links and create new synthetic answers.2 See Appendix § A.2 for more detail . 2.2.2 Cross-lingual Data Mining . After the parameter updates , we mine new training data using mDPR and Wikipedia language links and label the new data by mGEN predictions . This step is skipped in the final iteration . Mining by trained mDPR and language links ( line 4 , 6 in Alg . 1 ) . Trained mDPR can discover positive passages in another language that is not covered by the initial training data . At each iteration , we use retrieved passages Pmulti for qL ( line 4 in Alg . 1 ) as a source of new positive and negative passages . This enables expanding data between language pairs not in the original data . To cover even more diverse languages , we use language links and find passages in other languages that potentially include sufficient evidence to answer . Wikipedia maintains article-level language links that connect articles on the same entity over languages . We use these links to expand training data from the English QA dataset of Natural Questions ( line 6 in Alg . 1 ) . Denote a training instance by ( qEn , aEn , pgold ) . We first translate the English answer aEn to a target language aL using language links . We use language links again to look up the English Wikipedia article that the gold passage pgold comes from . We then find articles in non-English languages in the reference documents C multi that correspond to this article . Although the language link-based automatic translation can not handle non-entity answers ( e.g. , short phrases ) , this helps us to scale to new languages without additional human annotation or machine translation . We add all passages from these articles to Pmulti as positive passage candidates , which are then passed to mGEN to evaluate whether each of them leads to aL or not . Automatic labeling by mGEN predictions ( lines 7-8 in Alg . 1 ) . A passage pi from P multi may not always provide sufficient information to answer the question qL even when it includes the answer string aL . To filter out those spurious passages ( Lin et al. , 2018 ; Min et al. , 2019 ) , we take instances generated from the two mining methods described above , and run mGEN on each passage to predict an answer for the question . If the answer matches the correct answer aL , then the passage pi is labeled as a positive passage ; otherwise we label the input passage as a negative passage . We assume that when mGEN fails to generate a correct answer given the passage , the passage may not provide sufficient evidence to answer ; this helps us filter out spurious passages that accidentally contain an answer string yet do not provide any clue to answer . We add these new positive and negative passages to the training data , and in the next iteration , mDPR is trained on this expanded training set ( § 2.2.1 ) . 2This automatic answer translation is only done after the third epoch of initial training to prevent the model from overfitting to synthetic data .

Summary: This paper studies the task of multilingual Open-Domain Question Answering.  The task setting requires building a system that can answer natural language questions in any language on an open set of domains. Models are required to answer in the same language the question was asked in.  The authors tackle the task using a fairly conventional retrieve-and-read ODQA model, with the necessary modifications to make both the retriever and reader component language-agnostic, by exploiting multi-lingual pretrained models, training on available multilingual training data, and using and a clever data augmentation setup to increase multilingual training data.   The proposed solution “CORA”, operates over wikipedias in many different languages.   The retrieval component “MDPR” is a multi-lingual extension of DPR, using MBERT. This model can retrieve evidence documents across different languages, rather than having to match the language of the question. This means that a question expressed in a low-resource language can benefit from retrieving evidence from high resource languages. It also allows for a question written in one language on a cultural topic associated with a different language to retrieve from the most appropriate language’s wikipedia.   The reader “mGEN” is a generative reader, implemented using mT5, that takes as input the concatenation of the top-K retrieved passages from the retriever (which may be several different languages) and the question, and generates the answer in the question language, token by token.   The main difficulty of the proposed solution is how to train the MDPR and MGEN model components, given the limited multilingual raining data. The initial model is trained with NQ (english) and TyDi QA and Xor-TiDi QA (multilingual data). To increase the amount of multilingual training data and language coverage,  the authors introduce a data augmentation strategy where the initial model is used to harvest potential additional data. This exploits the wikipedia “language links” resource. This iterative approach works by retrieving passages, looking up passages on the same topic in other languages using “language links”, and using the reader to see whether these can be used to generate the correct answer - if they can, they are added as additional training data for the next round. Another expansion mechanism exploits the fact that most answers are wikipedia entities, and can be translated using annotations available in the wikipedia “language links” resource.  CORA is thoroughly evaluated on TyDI QA and an evaluation-only dataset called MKQA, and performs very strongly, improving the state-of-the-art by a wide margin in almost every setting.. Thorough ablations give insights and quantify the the effects of the system, and demonstrate zero-shot question answering abilities