Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper makes end-to-end training work for multi-document readers (e.g., FiD style readers). The key idea is an EM algorithm to update the reader and retriever given the top-k selection of previous iteration models. The proposed approach shows good performance on multiple open-domain QA benchmarks.
main_review: This paper makes end-to-end training work for multi-document readers (e.g., FiD style readers). The key idea is an EM algorithm to update the reader and retriever given the top-k selection of previous iteration models.

Although I think that the novelty of the EM algorithm is a bit limited (mathematically, the Hard-EM, REINFORCE, REALM and iteratively updating are closely related), the proposed approach demonstrates a clear advantage on multiple open-domain QA benchmarks. Also given that multi-document readers like FiD have become the mainstream in many QA tasks, it is necessary to have a joint-training algorithm like REALM be extended to these readers. Therefore I still would like to vote for acceptance.

It will be great if the following two questions can be addressed in the author rebuttal:

(1) To deal with the posterior, in Eq. 4 the authors make an approximation, which reduces the multi-document reader to single-document reader. On the other hand, could I understand the L_{alt-2} objective in Section 3.6 is equivalent to either Hard-EM [1] or REINFORCE [2]. If the answer is NO, detailed discussions about the differences from [1,2] should be added; if YES, this L_{alt-2} should be added as a baseline to Table 2 under "Our Implementation". In whichever case, I think the discussion of L_{alt-2} should be highlighted in Section 3.4, because it is related to a straightforward way of applying what has been developed by the community [1,2], while can form an apple-to-apple comparison to the proposed approximation.

[1] Min et al., 2019. A discrete hard em approach for weakly supervised question answering.

[2] Wang et al., 2018. R3: Reinforced ranker-reader for open-domain question answering.

(2) The reader of EMDR^2 is based on T5. If I am correct, the RAG and FiD works use BART instead. I am wondering how much benefit is caused by switching BART to T5. Therefore it would be helpful if EMDR^2 can be applied to the BART reader for comparison.

Typo:

Line 103: Eq. 3 (should be the equation in Line 101?)


limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes EMDR2, an end-to-end differentiable training method of multi-document reader and retriever for open-domain question answering. Based on expectation-maximization algorithm, EMDR2 iteratively estimates the set of relevant documents for a given question and uses the estimated set to update the parameters of retriever and reader. EMDR2 achieves new SOTA performance for models of comparable size on three benchmark OpenQA datasets.
Contributions:
-	An **EM-based** end-to-end training method of **multi-document** reader and retriever is proposed, which achieves SOTA performance on three benchmark OpenQA datasets.
-	The observation that EMDR2 implementations with different retriever initializations achieve similar final retrieval accuracies may be instructive.

main_review: Originality: 
-	This is new end-to-end training method for open-domain QA. Although the differences from previous work (such as REALM, RAG, and FiD) are clear, EMDR2 looks like a better combination of their techniques. The new contributions compared to Joint Top-k (Sachan et al., 2021) in Table 3 seem to be somewhat incremental and conventional.
-	Although a few end-to-end training methods for OpenQA are mentioned in introduction, there is no more comprehensive and in-depth literature review for such methods in related work.

Quality: 
-	The handling of technology and its terminology sounds correct and makes sense, but some intuitive explanations are lacked, e.g., the assumption on line 129 and the reason for using stop-gradient operator.
-	The work is complete and its empirical results seem to be impressive. But the necessity of considering multiple documents during end-to-end training and inference for single-hop questions from the evaluated three datasets is not intuitive explained and experimentally proved. Maybe a variant of EMDR2 whose reader only takes as input a question and one document (i.e., Z_reader = Z_{top-1}) should be considered. The result comparison between the variant and EMDR2 could prove such necessity, and the comparison between it and the Individual Top-k (Sachan et al., 2021) can show the effectiveness of updating the retriever using the posterior.

Clarity: 
-	The paper is easy to follow, and an expert reader may reproduce the proposed methods and results with some effort. However, the core motivation and main contributions are not highlighted, which brings ambiguity to the proposed method's novelty.
-	The conclusion on lines 47 to 48 is inconsistent with that on lines 260 to 261.
-	Should the metric P@50 be Recall@50? It is abnormal for P@50 having values greater than 80%, which means there are more than 40 relevant documents for a question from the three datasets. 

Significance: 
-	EMDR2 provides a better way for end-to-end training of multi-document retriever and reader than Joint Top-k (Sachan et al., 2021), and achieves SOTA performance on three benchmark OpenQA datasets. It may bring improvements in training methods for generative OpenQA and other retrieval-augmented NLG tasks.
-	The observation that EMDR2 implementations with different retriever initializations achieve similar final retrieval accuracies may be instructive.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on the open-domain question answering task. They use the generate framework which first retrieve relevant documents for the given question and then generate the answer based on the question and the retrieved documents. They model the retrieved documents in the latent space. Previous studies usually consider only a single document as the latent variable. In this paper, they consider to use a set of documents as the latent variables. They then apply an expectation-maximization algorithm to update the reader and the retriever. The reader is optimized to generate the answer based on the question and the retrieved set of documents. The retriever is updated using posterior estimation of the retrieved results given the question and the answer. They conduct experiments on three open domain question answering tasks. The experimental results show that their approach outperforms other baselines by a large margin (2%-3% absolute improvement). They conducted ablation studies to show the performance of the model with different number of retrieved documents. They also compared different retriever initialization  strategies and showed that masked salient spans pretraining is helpful.
main_review: The originality:
The authors propose to model a set of retrieved document instead of single document in the latent space. This is different from previous work such as REALM. Based on this intuition, the EM algorithm is straightforward to come up with and the objective is similar to the one in REALM.

Quality:
Their experimental results show that their model can significantly outperforms all the baselines. The also conducted a set of ablation study which show that their model performs consistently good with different number of retrieved documents. But they did not do the experiments to show that it is important to model a set of retrieved documents instead of a single one. For example, they could conduct preliminary experiments to show how many questions need to be answered based on multiple documents, and do the analysis to show that their model can performs much better especially on the cases that required multi-document modeling. Without showing this,  their motivation is not well supported by the experiments.

Clarity:
The paper is well-written and easy to understand. The approach is intuitive and straightforward. The authors will release the code for further research and reproducibility.

Significance:
The authors achieve state-of-the-art performance on three popular dataset for open domain question answering, which is significant.

limitations_and_societal_impact: The model may have typical biases caused by the large language models. Another potential limit is the model could be attacked to generate fake answers when the evidence documents are modified.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes end-to-end multi-document retriever-reader training for open-domain question answering. The proposed model is based on a pipeline approach where the retriever based on a dual encoder retrieves a set of documents from a large text corpus and a reader processes them and generates the answer. Different from previous approaches which independently train the retriever and the reader (Karpukhin et al, Izacard & Gave, and others), the paper jointly trains them using the EM algorithm. It is also different from previous work with joint training (Lee et al, Guu et al, and others) in that the generation of the answer is conditioned on multiple documents, not just a single one. In particular, this difference makes joint training more difficult, so the paper proposes to use the EM algorithm to iteratively update the parameters of the retriever and the reader. The paper also came up with clever initialization tricks and asynchronous refreshes of the MIPS index, which leads to successful training of the proposed model.
The proposed model is evaluated on widely-used open-domain QA benchmarks (NQ, TriviaQA, WebQ). The proposed model outperforms a set of other, recently-proposed models by a large margin, which is impressive given that the baselines are highly competitive.
main_review: Strengths
- The problem is well-motivated, with clear distinctions from prior work.
- The model is well-executed, with the key method like EM algorithm, and tricks including initialization and asynchronous updates. All the descriptions of the models are very clear.
- The model achieves impressive results on competitive open-domain QA benchmarks.

Weaknesses
- I think the paper should discuss more thorough comparisons to REALM (Guu et al) because there are a lot of shared aspects. For example:
    * The asynchronous updates are almost identical.
    * The high-level idea of REALM pretraining is very similar to the initialization in this paper (using masked LM as sort of pseudo data to pretrain the model). Do you think these two methods have exactly the same role, or do you think there are features that your initialization can capture but REALM pretraining cannot?
  * The difference in answer generation being conditioned on single vs. multi documents (hence leading to the need for the EM algorithm) is clearer. Additional discussions I want to see are mainly about the above two points.
- Framing retrieval documents as a latent variable is first done by Lee et al. 2019. It should be more highlighted.


Questions
- Asking based on ablation in Table 3
    * What is P@50? What “retrieval accuracy” means is unclear. Do you mean recall accuracy? (The model gets a credit if at least one of top-50 docs contain the answer text.) In that case, it might be better to say “R@50” or “Rec@50” then, and clarify in the caption?
    * It looks like DPR-style supervised training does not impact the performance very much based on this metric. Have you compared these two methods based on the EM score?
- Have you tried a baseline where you initialize the retriever and the reader using MSS pretraining, but then train them separately, in a pipeline manner? I think this is an important baseline.


Minor comments
- L19: “train a language model”: I am not sure if it is necessary to train a “language model”?
- L32: I would cite Karpukhin et al for the pipeline training as it is the first paper trying it, and Izacard & Grave mainly took the model from Karpukhin et al.
- Table 1: It might be good to add a column “requiring document supervision” which makes the proposed model more appealing - I think this column will be checked only for REALM and the proposed model.
- Section 3.2: Which T5 variant was used for the reader? I am guessing it is T5-base, but it isn’t clearly mentioned in the paper.
- Section 3.3: I think the terms “parametric” and “semi-parametric” are confusing, and are not used in any prior work. What about “closed-book” vs. “open-book” (following terms in Roberts et al) or “no retrieval” vs. “retrieval”?


Conclusion & Overall recommendation

The paper proposes a model for open-domain QA, where the answer is generated conditioned on multiple documents jointly, and the retriever and the reader are trained end-to-end. The proposed idea is new and substantially different from prior work; the model is well-executed; the experiments are comprehensive, showing impressive results on competitive open-domain QA benchmarks. I have minor concerns about more comparisons with prior work and additional baselines, which are relatively easy to address. Overall, the quality of the paper is substantially higher than the bar to be published.
limitations_and_societal_impact: The paper adequately discusses limitations and societal impact. It would be also good to mention that training is relatively resource-heavy (requiring 16 GPUs), potentially having environmental concerns.
review_title: null

Meta-review: This paper proposes a new algorithm for training of multi-document question answering model containing a reader and a retriever. The key idea is to introduce latent variables over the sets of retrieved documents, employ the EM algorithm to approximately estimate the variables, and update the parameters of the reader and the retriever. Experimental results show that the proposed approach outperforms the baselines for about 2-3% on three benchmark datasets.

Pros
* The problem studied is important.
* The paper is generally clearly written.
* The proposed approach appears to be reasonable and sound.
* Experimental results show the efficacy of the proposed approach. The improvements are significant.

Cons
* There were related models proposed in the past. The idea of the work is not very novel. The paper mainly demonstrates that a new way of combining the existing ideas does work better.
* There are some details that are not explained very clearly. The authors have promised to make them clearer in the revised version.
* The authors have also successfully addressed many of the issues pointed out by the reviewers in the rebuttal.

Overall this is a solid work. We think that the contribution to ML is mainly from the empirical results.


Decision: Accept (Poster)