Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper attempts to provide a solution to an interesting problem of cross language retrieval based question answering in a zero shot (without requirement of any language-specific annotated training data). Authors propose a new DPR algorithm that to be able to retrieve documents relevant to the question across languages (for multiple languages). The paper extends retrieve-the-generate based open domain QA system to a cross-lingual setting. 
main_review: How does the proposed approach differ in terms of techniques with something already proposed like RAG (https://arxiv.org/abs/2005.11401). It looks like the proposed approach is just a multilingual version of RAG thus the paper has limited novelty.  Does the system expects the  gold document annotations to be available for training the retriever module? Section 2.2., talks about positive passages, this requires huge time and effort for annotation, so how does authors claim in abstract that the proposed technique does not require any language specific annotated training data?  What happens in the scenario where a wikipedia page about a specific entity is not available in a particular language in which the question has been asked?  Section 2.2 needs to have some more intuition and clarity. What is intuition behind translating English answers to target language. Is there a mechanism to filter out spurious passages by the automatic passage labeling technique? How does the model learns cross lingual embeddings? do authors use a pre-trained model like m-bert? How does the proposed model results compare when compared with m-bert based methods? 

Update: I think the problem formulation is novel. So, after reading author responses I am increasing the score to 6. 
limitations_and_societal_impact: Limitations of the work has not been discussed by the authors. Authors did promise to make the code publicly available to general public to enable further research in the direction ( societal impact)
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses the multilingual open QA task by combining a cross-lingual passage retrieval algorithm with a multilingual autoregressive generation model. The approach is thus capable of answering questions where the answer and the question are in different languages.  The authors extended existing annotated data using an iterative training approach that improves retrieval for languages with limited resources. The experimental results showed that the proposed method (CORA) outperformed the previous state of the art on multilingual open question answering benchmarks across 26 languages.
main_review: This paper targets at Multilingual Open Domain QA, which is a novel task with many application scenarios. The proposed method, CORA, showed relatively good performance on 26 languages on benchmark datasets. 
However, the overall idea is very similar to CRISS [1]. The CRISS approach consists of a cross-lingual retrieval module and a seq2seq module, and they are optimized alternately. Similarly, the CORA approach in this paper also consists of DPR (Dense Passage Retriever, mbert structure) and GEN(Answer Generator, mt5 structure), and they are also optimized alternately. Although CRISS [1] doesn’t target at multilingual open QA,  it has strong performance on cross-lingual sentence retrieval, which is related to multilingual open QA. The authors should compare their work with CRISS [1].
Authors claim that they introduce a new dense passage retrieval algorithm, but don’t mention any similar work about dense passage retrieval [2].  
Overall, this paper combines DPR and GEN module and achieves good experimental results. However, the technical contribution of the paper is quite limited. 
[1] Cross-lingual Retrieval for Iterative Self-Supervised Training
[2] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

limitations_and_societal_impact: I don't see potential negative social impact of this work.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper studies the task of multilingual Open-Domain Question Answering. 
The task setting requires building a system that can answer natural language questions in any language on an open set of domains. Models are required to answer in the same language the question was asked in. 
The authors tackle the task using a fairly conventional retrieve-and-read ODQA model, with the necessary modifications to make both the retriever and reader component language-agnostic, by exploiting multi-lingual pretrained models, training on available multilingual training data, and using and a clever data augmentation setup to increase multilingual training data. 

The proposed solution “CORA”, operates over wikipedias in many different languages. 

The retrieval component “MDPR” is a multi-lingual extension of DPR, using MBERT. This model can retrieve evidence documents across different languages, rather than having to match the language of the question. This means that a question expressed in a low-resource language can benefit from retrieving evidence from high resource languages. It also allows for a question written in one language on a cultural topic associated with a different language to retrieve from the most appropriate language’s wikipedia. 

The reader “mGEN” is a generative reader, implemented using mT5, that takes as input the concatenation of the top-K retrieved passages from the retriever (which may be several different languages) and the question, and generates the answer in the question language, token by token. 

The main difficulty of the proposed solution is how to train the MDPR and MGEN model components, given the limited multilingual raining data. The initial model is trained with NQ (english) and TyDi QA and Xor-TiDi QA (multilingual data).
To increase the amount of multilingual training data and language coverage,  the authors introduce a data augmentation strategy where the initial model is used to harvest potential additional data. This exploits the wikipedia “language links” resource. This iterative approach works by retrieving passages, looking up passages on the same topic in other languages using “language links”, and using the reader to see whether these can be used to generate the correct answer - if they can, they are added as additional training data for the next round. Another expansion mechanism exploits the fact that most answers are wikipedia entities, and can be translated using annotations available in the wikipedia “language links” resource.

CORA is thoroughly evaluated on TyDI QA and an evaluation-only dataset called MKQA, and performs very strongly, improving the state-of-the-art by a wide margin in almost every setting..
Thorough ablations give insights and quantify the the effects of the system, and demonstrate zero-shot question answering abilities  

main_review: This is an empirically very strong piece of work that performs sensible (if straightforward and somewhat incremental on the modelling side) extensions of work from open-domain QA and multilingual open-domain QA. However, the result is stronger than the sum of its parts, unlocking the ability to combine information across many languages’ wikipedias, and enjoy the resultant improvements in performance. This is a really good piece of work, and well executed. 

The iterative training and data expansion techniques are interesting and important contributions, which leverage the wealth of annotations and metadata available in wikipedia to improve cross-lingual QA, without requiring explicit question translation or extra annotation efforts by humans. One limitation is that this won’t always work well if the language links aren’t high quality for some languages, or in other domains where language links aren’t available (either because wikipedia isn’t a good background corpus, or because answers are not wikipedia/wikidata entities).

The experiments and ablations are thorough and detailed and the analysis is deep. 
In terms of some constructive criticism, the MGen model could probably be made more powerful by using the Fusion-in-Decoder architecture, which would allow the model to scale to reading more passages per question, which has shown to greatly improve accuracy on open-domain QA. 

I also wonder why the models’ performances  are so different and sometimes inconsistent between MKQA and XOR-full (comparing tables 1 and 2)  - for example, CORA scores 54.8 F1 for arabic on XOR-full, but loses a lot of performance arabic on MKQA (only 13.1 F1) . Conversely, MT+DPR performs poorly for arabic on XOR-full (7.6 F1) but  outperforms CORA on MKQA (16.0 F1).

Summary:
* Originality: This paper takes a standard ODQA approach, but converts it to a massively cross-lingual setting in a novel and very powerful way.
( Quality: the experiments are well-thought out, the results are very strong, and the ablations and analysis are execllent
* Clarity: The paper is very clear and well written
* Significance: I expect this to be a high impact paper and serve as inspiration for many followup works and tasks.



limitations_and_societal_impact: The authors provide a note on societal impacts which summarize the main impacts well. This kind of work will likely have a positive impact on societal equality, unlocking the strong ODQA performance we see in high resource languages for lower resource languages. There are risks that CORA may provide wrong answers that might be harmful or offensive, but this is the case for monolingual systems too. The authors rightly point out that not relying on external blackbox translators allows the model to be more interpretable and debuggable.
The use of wikipedia language links is useful, but does limit the textual domains the data expansion technique can be used for to similar domains to wikipedia.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a multilingual dense passage retrieval model called mDPR to deal with the multilingual/cross-lingual IR and QA tasks. The mDPR part is firstly trained using existing labeled multilingual Q-P pairs. Then, possible passage pairs are selected from a given multilingual corpus based on the trained mDPR and used to train an answer generation model called mGEN. Next, given each question, the trained mDPR is further used to retrieve more possible passage candidates in other languages from Wikipedia corpus, where the retrieved passages that can lead to correct answers are considered as positive instances, the others as negative instances. With these expanded new question-passage pairs, the training of mDPR is performed again. The above procedure is conducted in an iterative manner. Evaluations are conducted on TyDi QA and MKQA in the open-domain QA setting. Several baselines are included and compared. The key contribution is to do DPR in the multilingual/cross-lingual setting.
main_review: This is a good paper that extends dense IR from monolingual to multilingual/cross-lingual. But the novelty is limited. The most important part, I think, is the data expansion part described in Section 2.2. From model perspective, there seems be have few innovation. The baselines compared in Table 1 and 2 are also weak. Why not compare with the submissions on the TyDi QA leaderboards? Why stronger multilingual pre-trained models such as XLM-R or mBART are not used and fine-tuned for comparison?
limitations_and_societal_impact: no potential negative societal impact.
review_title: null

Meta-review: The paper tackles the problem of multilingual open domain question answering, introducing an iterative approach to mining answer passages.

There is not a clear consensus amongst the reviewers. It is clear that this is an interesting and under-explored problem, and that the proposed solution works well. Reviewer qfFE argues strongly for acceptance based on the novelty of the setting, the strong empirical comparisons, and the potential for inspiring future work.

The major criticism made of the paper is its lack of technical novelty, and reviewers agree that similar iterative retrieval ideas have been proposed in models such as RAG and CRISS. However, there is a non-trivial modeling contribution over these papers to extend them successfully to multilingual open domain QA. Reviewer nAYP also feels that the baselines are weak, but I agree with reviewer qfFE that the strong set of baselines is a significant contribution of the work.

Overall, I recommend acceptance.


Decision: Accept (Poster)