End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering
Authors: Devendra Singh Sachan
Siva Reddy
William L. Hamilton
Chris Dyer
Dani Yogatama
Venue: NeurIPS 2021 Poster (2021)
Keywords: open-domain question answering
information retrieval
closed-book QA

Abstract: We present an end-to-end differentiable training method for retrieval-augmented open-domain question answering systems that combine information from multiple retrieved documents when generating answers. We model retrieval decisions as latent variables over sets of relevant documents. Since marginalizing over sets of retrieved documents is computationally hard, we approximate this using an expectation-maximization algorithm. We iteratively estimate the value of our latent variable (the set of relevant documents for a given question) and then use this estimate to update the retriever and reader parameters. We hypothesize that such end-to-end training allows training signals to flow to the reader and then to the retriever better than staged-wise training. This results in a retriever that is able to select more relevant documents for a question and a reader that is trained on more accurate documents to generate an answer. Experiments on three benchmark datasets demonstrate that our proposed method outperforms all existing approaches of comparable size by 2-3% absolute exact match points, achieving new state-of-the-art results. Our results also demonstrate the feasibility of learning to retrieve to improve answer generation without explicit supervision of retrieval decisions.

End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering
Devendra Singh Sachan 1.2 Siva Reddy 1.2, William Hamilton 1:2, Chris Dyer², Dani Yogatama
Mila Quebec Al Institute
2 School of Computer Science, McGill University DDeepMind
sachande@mi la quubbbec,, (siva, w1h) ) @cs. dics.mogill. 11.1c.... (cdyer, dyogatama Redeepminds nd. com
Abstract
We present an end-to-end differentiable training method for retrieval-augmentt open-domain question answering systems that combine information from multiple retrieved documents when generating answers. We model retrieval decisions as latent variables over sets of relevant documents. Since marginalizing over sets of retrieved documents is computationally hard, we approximate this using an expectation-maxiiizzin maximization algorithm. We iteratively estimate the value of our latent variable (the set of relevant documents for given question) and then use this estimate to update the retriever and reader parameters. We hypothesize that such end-to-end training allows training signals to flow to the reader and then to the retriever better than stage-wise training. This results in a retriever that is able to select more relevant documents for question and reader that is trained on more accurate documents to generate an answer. Experiments on three benchmark datasets demonstrate that our proposed method outperforms all existing approaches comparable size by 2-3 absolute exact match points, achieving new state-of-the- art results. Our results also demonstrate the feasibility of learning to retrieve improve answer generation without explicit supervision of retrieval decisions.
expectation
Introduction
Open-domain question answering (OpenQA) a question answering task where the goal is train language model to produce an answer for given question. In contrast to many question answering tasks, an OpenQA model only provided with the question as its input without accompanying documents that contain the answer. One of the most promising approaches to OpenQA is based on augmenting the language model with an external knowledge source such as Wikipedia (often referred to as the evidence documents). In this approach, the model consists of two core components (Chen et al., 2017): (i) an information retrieval system to identify useful pieces of text from the knowledge source (the retriever); and (ii) system to produce the answer given the retrieved documents and the question (the reader).
We can view such model as a latent variable model, where the latent variables represent retrieved documents that are used to produce answers given questions (Lee et al., 2019). End-to-end (joint) training of this model is challenging since we need to learn both to generate an answer given retrieved documents and what to retrieve. Previous work considers two potential solutions (see Table for high-level summary). First, they adopt stage-wise training, where the retriever is trained while freezing the reader and vice versa (Karpukhin et al., 2020, Izacard and Grave, 2021b,a), Another
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Reader and Retriever Training
Model
Multi-Dos Doc
Retriever Adaptation /
Disjoint End-to- End
Multi-Step Unsupervised Retriever
Reader
REALM (Guu et al., 2020)
/
DPR (Karpukhin el al., 2020)
RAG (Lewis et al., 2020b)
FiD (Izacard and Grave, 2021b)
FiD-KD (Izacard and Grave, 202
EMDR* (Our Approach)
Bird's s-eye view of the recent OpenQA approaches. Multi-Doc reader indicates whether the reader architecture uses multiple documents single document. Retriever adaptation shows whether the retriever gets feedback from the reader to update its parameters. Disjoint denotes that first the retriever is trained and then the reader trained. End-to-end denotes that the reader and retriever are trained jointly in one cycle. Multi-step indicates that the reader and retriever are trained iteratively in multiple eycles. Unsupervised retriever indicates whether the retriever is initialized using unsupervised approaches or using supervised data.
Table 1:
alternative is constraint the reader to condition on each retrieved document individually! (Guu et al., 2020) -oometmmess with extra supervision for the latent variables in the form of the relevant document for a question (Lewis et al., 2020b).
In this paper, we consider a from multiple documents when generating answers. Expectation-maste maximization (Dempster al., 1977) offers principled template for learning this class of latent variable models. We present EMDR²: End-to-end training feedback from the model itself as "pseudo labels" of the latent variables for optimizing the retriever and reader parameters. We use two estimates of the latent variables: (i) prior scores for updating the reader parameters and (ii) approximate posterior scores given all observed variables for the retriever parameters.
question answering model that combines information
Reader and Retriever ($2). EMDR² iteratively uses
We evaluate our proposed method by experimenting on three commonly used OpenQA datasets: Natural Questions, TriviaQA, and WebQuestions ($3). EMDR2 achieves new state-of-the-an results for models of comparable size on all datasets, outperforming recent approaches by 2-3 absolute exact match points. We also show that EMDR² is robust retriever initialization. achieves high accuracy with unsupervised initialization, suggesting that supervised training of the retriever may not be an essential component of the training process as suggested in prior work (Karpukhin al., 2020).
In summary, our contributions are as follows: (i) we present an end-to-end training method (EMDR²) for aifieval-augmeneeed question- -answernnn systems; (ii) we demonstrate that EMDR outperforms other existing approaches of comparable size without any kind of supervision on the latent variables; (iii) provide ablation studies for better understanding of the contributions of different components of our proposed method; and (iv) we release our code and checkpoints to facilitate future work and for reproducibility.
EMDR² is a framework that can be used to train retrieval- aetrmeel-augmmented text generation models for any task. We believe that our estimation technique in EMDR² is also useful for learning similar latent variable models in other domains.
2
Model
Our proposed model EMDR² consists of two components: (i) neural retriever and (ii) neural reader, which we train jointly in an end-to-end setting. Figure shows an illustration of our model and training procedure. We discuss each component and our training objective in detail below.
This makes marginalization over latent variables easier since we only need to consider one document a time rather than multiple documents at once.
Our code is available https: h/github...//////////////.tb.htt com www.//ewsiigghs.ccanncccocee

---

2.1 Neural Retriever: Dual Encoder
Let the collection of evidence documents be denoted by [di...., d Given question q. the goal of the retriever module is to select subset of documents 2 C D to answer the question. We model the retriever as dual-encoder network (Bromley al., 1994). where one encoder fy encodes the question and another fa encodes the evidence document (to vector). The retrieval score defined as the dot product between the two resulting vectors:
score d, 4) fg(q: 2.)) fa(d;; Ba).
(1)
where + f
g. Pa] denotes the retriever parameters. select top-K documents for the question q from 'D based on the retrieval scores. We denote the set of retrieved documents by == (-1..... 2x)
We use transformer encoders (Vaswani et al., 2017) as our fg and fa. Our transformer architecture is similar to BERT with 12 layers and 768 hidden size (Devlin et al., 2019). We use the final representation of the first token (i.e., the standard [CLS] token from BERT's tokenization) as our question (and similarly document) embedding. Initializing fg and fa with BERT weights has been shown to lead to a poor retrieval accuracy (Lee al., 2019, Sachan al., 2021). Therefore, we initialize the retriever with an unsupervised training procedure. We discuss our initialization technique detail in $3.2.
2.2 Neural Reader: Fusion-in-Decode
The reader takes as input a question q and a set of retrieved documents (to be read) 2 to generate an answer. Our reader is based on the Iusion-in-Decodee (FiD; Izacard and Grave, 2021b) model, which is built on top of T5 (Raffel et al., 2020). T5 is pretrained sequence-to-sequence transformer that consists of encoder Ge and decoder ga-
In FiD, each retrieved document 2k is first appended with its title (tz,) and the question:
[CLS] [SEP] l2, [SEP] # [SEP],
where [CLS] is used to indicate the start of document and [SEP] is used separator for the different parts of the document as well as the final token.
Each ææ is then independently given corresponding to of the retrieved documents are concatenated as:
input to the T5 encoder ge. The output representations
X 2 = (x)); ...).....s yo
(. (xx)] E [R(*K))H
where is the number of tokens in each æ 3 and H is the hidden size of the T5 encoder ge- this work, we use the T5-base configuration with N = 512 and H = 768.
X2 is then given attends to both previously generated tokens (i.e., causal attention) as well as the tokens encoded in X (i.e., cross attention). Since X contains information from multiple documents, the decoder has the ability to aggregate useful signals contained multiple documents and jointly reason over them. We define the probability of the answer as:
input to the T5 decoder 9d- When generating answer token, the decoder
p(a 2, I (a, (a, a q. 2;6), [=1
(2)
where ④ denotes the reader parameters (i.e., T5 encoder and decoder) and T is the number of answer tokens. We keep generating answer tokens until the decoder outputs a special EOS token or pre-specified maximum answer length reached.
2.3 End-to-End Training of Reader and Retriever
In contrast to previous work on generative question answering, we train both the reader and the retriever jointly in an end-to-end differentiable fashion.
Denote our latent variable which represents a set of retrieved documents by and let Z a possible value of Z. The marginal likelihood of an answer (marginalizing over all the possible values of Z)
We truncate and pad necessary such that every æ; has the same length N. See $3.2 for details.
3

---

Retriever Training
Reader Training
question (q)
p(a q A
Shared Parameters
p(a
q, 2)
Question Encoder
T5 Decoder
Decoder
Evidence Docs (D)
Inner Product
fa(z))
X
a
fe(g)
Stale Evidence Doc Encoder
Evidence
(a: ) (xt)
Concat
lop-k MIPS
Encoder
Distributed Evidence Doc Index over GPUs
Asynchronous Index Refresh
Top-k Docs (Z)
T5 Encoder
Preprocessor
22 2 26
Figure An illustration of the different components of EMDR?. Colored blocks indicate components which contain trainable parameters.
is: p(a @; = p(a q, 2)p)p(Z q: Φ). The goal of our training procedure is to find and θ that would maximize the above objective. Exactly optimizing Eq. 3 is intractable as it is combinatorial in nature." For one particular value 2, the log-likelihood is simpler compute:
log q, 2; O)p( 2 q: b) = log p(a q, 2; θ) + logp(2 2 q; Φ)
Expectation maaiaaiiiiiaaiiiiiiiiii (EM) algorithm (Dempster et al., 1977) offers solution to learning this latent variable model. In classical EM, we iteratively compute the posterior Z given all observed variables and use it to update θ and 4
We propose using two estimates of Z Zezrer and Zieuiever for updating the two components the model (reader parameters θ and retriever parameters 4):
p(a q, Zreader; Q) log pl Zrettieerr q: +)
reader
In the first term, we set the value of the latent variable = Zweder based on the prior scores. In the second term, we seek to maximize approximate posterior of Z Zeetriovr- We discuss them in more detail below.
Reader parameters For updating (the first term of Eq. 3), we use the top-K documents with the highest individual scores (as computed by Eq. 1 based the current value of ④) to construct Zreader- This is equivalent to relying on the prior p( q; ②) to estimate Zredeer (without using information from the answer a). We choose to use the prior to train reader parameters since the prior scores are also used at evaluation time to obtain the top-K documents. As a result, there is no mismatch between training and test computations when computing p(a q, 2; Q) (i.e., Z that is used at test time is obtained in exactly the same way as Zreader Zup-k).
Retriever parameters 4. For updating (the second term of Eq. 3), we propose to use the posterior estimate. In other words, train Φb Using the posterior allows our retriever to learn from richer training signals as opposed to relying only the prior.
additional information from when evaluating Zeeriever to
We need to be able to compute p( Zretrieerr g, a; θ +) to maximize the retriever parameters. However, computing this quantity is difficult since it is probability of a set.5 Consider set of K documents (e.g., Ziop-K) ). where 23 denotes a document in the set. We approximate the maximization of the probability of the set by assuming that its probability is maximized if the sum of the probability of
retrieved document ø when generating an answer. In this case, the latent variable represents a document assignment instead set retrieved documents.
'Contrast objective with REALM (Guu al., 2020), where the reader only conditions
"This true whether we choose the posterior probability the prior probability.

---

each document in the set is maximized. 6 With this approximation, we arrive at a simpler quantity: E p(= q, a; θ D). Note that using Bayes rule, we can rewrite:?
p(z 9, a O +) x p(a q, =x; O)p(z* q; Φ)
The reader now only conditions on one document when computing the probability of answer p(a q: 2g; @). This simpler reader uses the same parameters as the more sophisticated one O but it only uses one document zς instead of set of documents.
To compute Eq. 4, we first obtain K documents with the highest scores as computed by Eq. based on the current value of &. We compute the probability of document z Z K
p(z q, Zwp K; ①) exp(score =x)/T; ②))
(5)
Σj= exp(score(g 2,)/T;
where is temperature hyperparameter and the approximation assumes that documents beyond the top- K contributes very small scores so we do not need to sum over all evidence documents M in the denominator (which is in the order of tens of millions in our experiments). We then compute p(a q: 2g; θ) similarly to Eq. 2.
Overall training objective of EMDR?. Combining the above derivations, our end-to-end training objective that we seek to maximize for particular example becomes:
K
L = log q: Ki θ) + log SG (p(a q, =x: θ)) p(zx Ki +),
(6)
reader
retriever
where SG is the stop-gradient operator so that the reader parameters θ are not updated to also perform well given a single document #k. The nop-gradieett operator in the second term of EMDR² has several benefits. First, the FiD reader is trained from the first term of the EMDR² objective in which its likelihood is conditioned on all the retrieved documents, similar how the reader used test time. Second, it also makes training faster since the backward pass which is computationally more expensive than the forward pass is not needed, which in turn reduces the usage of GPU RAM as intermediate activations need not be saved.
Given training example, update O and & by taking gradients of Eq. with respect O and & in an end-to-end fashion. Intuitively, we train the reader to generate the correct answer given K highest scoring documents Z For the retriever, has a high score of generating an answer (since the sum over K is inside the log in the second term) while taking into account feedback from the reader. Algorithm summarizes our training algorithm.
train it to select K documents which collectively
Algorithm 1: End-to-end training of multi-document reader and retriever.
Input: Model parameters θ and 4 evidence documents D.
while not converged do
Compute Z k using the current retriever parameters +.
// E-step
Compute p(a q. =x) for each 2 using the current reader parameters O. // E-step
Update model parameters O and & to maximize the log-likelihood in Eq. 6. // M-step
end
3 Experiments
3.1 Datasets
We experiment with three commonly used open-domain question answering datasets:
'The intuition is that each element of the set contributes independently, which greatly simplifies the computa- tion to find the maximum of the set.
We choose not to normalize with p(a q: ④ 2)) since computing this quantity would require summing over all evidence documents M. While this makes the resulting objective that we optimize not correspond to proper probability distribution anymore, observe that training method still behaves well in practice.
5

---

Natural Questions (NQ; Kwiatkowski et al., 2019). NQ contains questions asked by users of the Google search engine. Similar Lee al. (2019), we use the short answer subset. TriviaQA (Joshi et al., 2017). TriviaQA a collection of trivia question-ansn -nnswrr pairs that were collected from multiple sources on the web.
WebQuestions (WehQ; Berant et al., 2013). WebQ questions were collected using Google Suggest API and the answers were annotated using Mechanical Turk. We use the version from Chen et al. (2017) where Freebase IDs in the answers are replaced by entity names,
Evidence documents D. We use the preprocessed English Wikipedia dump from December 2018 released by Karpukhin et al. (2020) our evidence documents. Each Wikipedia article is split into non-overlapping 100 words long segments. Each segment corresponds document in our case. There are a total of 21,01 21,015,324 documents in total.
We provide descriptive statistics and other preprocessing details in Appendix A.
3.2 Implementation Details
Hardware and library. We run all of our experiments on a machine with 96 CPUs, 1.3TB physical memory, and 16 A100 GPUs We use PyTorch (Paszke al., 2019) to implement our proposed model and relevant baselines.
Model configurations. For both the retriever and reader, we use the base configuration that consists of 12 layers, 768 dimensional hidden size, and 12 attention heads. In all experiments, we retrieve 50 documents, unless stated otherwise. We only use the base configuration our experiments due to GPU memory constraints. However, we believe that our results would generalize to larger configurations as well.
To support fast retrieval, we pre-compute evidence document embeddings and store them distributed fashion over all the GPUs. We refer to these document embeddings as the document index. For each question, we retrieve documents in an online (on-the-fly) manner by performing exact maximum inner product search (MIPS), implemented using asynchronous distributed matrix multiplication over the document index. These documents are converted to subwords using BERT's tokenization and are given as input to the T5 reader. If tokenized document is shorter than 512 tokens, it is padded using the tokens from the neighboring documents until the maximum token limit is reached. Such padding additionally helps to provide an extended context for answer generation.
Retrieval.
Initialization and training details. We initialize the parameters of the model with unsupervised pre-training before performing supervised training using the question-answer training examples. Unsupervised pre-training is essential it helps to warm-start the retriever so that outputs relevant documents for given question.
We first pre-train the retriever parameters with unsupervised Inverse Cloze Task training (Lee et al., 2019) for 100,000 steps. We then extract sentences containing named entities from the evidence documents. Next, we replace 15% of the named entity tokens with masked tokens, which are often referred to as masked salient spans (MSS; Guu et al., 2020). The masked sentence can be considered as the question and its salient spans (i.e, named entities) can be considered as the answer to train the model with Eq. 6. We train the model on these question-answer (masked sentence-named entities) pairs for 82,000 steps with batch size of 64 using Adam (Kingma and Ba, 2015). We refer this initialization method as unsupervised pre-training with masked salient spans. We provide further description in Appendix C.
After MSS training, we finetune the model on the setescc- question-anse answeer training examples with EMDR². We perform training for 10 epochs on NQ and TriviaQA with a batch size of 64, and for 20 epochs on WebQ with a batch size of 16. During training, we save a checkpoint every 500 steps and select the best checkpoint based on its performance on the development set.
During end-to-end training, since the parameters of the document encoder (fa) are also updated at every step, the pre-computed document embeddings become stale as training progresses. We use the most recent document encoder checkpoint to compute fresh document embeddings asynchronously with which the document index is updated after every 500 training steps to prevent staleness.

---

Model
top-K
NQ
TriviaQA
WebQ
# of params
dev
test
dev
test
dev
test
Closed-Book QA Models
TS-base (Roberts et al., 2020)
25.7
24.2
28.2
220M
TS-large (Roberts et al., 2020)
27.3
28.5
29.5
770M
T5-XXL (Roberts et al., 2020)
32.8
42.9
35.6
11B
GPT-3 (Brown et al., 2020)
29.9
41.5
175B
Open-Book QA Models
BM25 + BERT (Lee et al., 2019)
24.8
26.5
47.2
47.1
27.1
21.3
220M
ORQA (Lee et al. 2019)
31.3
33.3
45.1
45.0
36.8
30.1
330M
REALM (Guu al., 2020)
5
38.2
40.4
40.7
330M
DPR (Karpukhin ol., 2020)
25
41.5
56.8
34.6
330M
ReConsider (lyer et al., 2021)f
30
43.1
59.3
44.4
440M
RAG-Sequence (Lewwis ef al., 2020b)
50
44.0
44.5
55.8
56.8
44.9
45.2
626M
Individual Top-K K (Sachan al. 2021)
45.9
56.3
440M
Joint Top-K (Sachan al., 2021)
50
49.2
64.8
440M
FiD (Izacard Grave. 2021b)
100
48.2
65.0
440M
FiD-KD (Izacard and Grave, 2021a)
100
48.0
49.6
68.6
68.8
440M
Our Implementation (Base Configuration)
TT-baas
26.0
25.1
26.7
27.8
31.0
32.4
220M
FiD (DPR retriever, T5 reader)
37.3
38.4
50.8
50.4
40.2
38.3
440M
FiD (DPR retriever, T5 reader)
50
47.3
48.3
65.5
66.3
46,0
45,2
440M
FiD (MSS DPR retriever, T5 reader)
50
48.8
50.4
68.0
68.8
43.5
46.8
440M
FiD (MSS retriever, MSS reader)
50
38.5
40.1
60.0
59.8
39.1
40.2
440M
EMDR (MSS retriever, MSS reader)
50.4
52.5
71.1
71.4
49.9
48.7
440M
Table 2: Exact match scores on three evaluation datasets. Top-K denotes the number of retrieved documents that are used by the reader to produce an answer. To provide a fair comparison with our reimplementations, we show results from other papers with the base configuration, except for RAG-Sequence that uses BART-large (Lewis al., 2020a). f indicates that their results on WebQ use NQ training data to pretrain the model.
Inference.
We use greedy decoding for answer generation at inference time.
3.3 Baselines
We compare model to other approaches for OpenQA that can be categorized under the following two classes:
Closed-book QA models. Large-scale language models capture lot of world knowledge in their parameters derived from the corpus they have been trained on (Petroni et al., 2019). We compare with the work of Roberts al. (2020) who show that larger T5 moeels-when finetuned with question-answe pairs-can perform remarkably well. We also compare with the few-shot results of GPT-3 (Brown al., 2020). 8
Open-book QA models. Similar to this work, these models consist of retriever and reader components and adopt the retrieve then predict approach for answering questions given a collection of evidence documents. These models mainly differ in how the retriever is initialized (ORQA: Lee el al., 2019, DPR; Karpukhin el al., 2020), whether the reader processes single document (ORQA, DPR, RAG; Lewis el al., 2020b) or multiple documents (FiD; Izacard and Grave, 2021b), or whether the reader and retriever are trained jointly or in multistage process (REALM; Guu al., 2020, FiD-KD) Izacard and Grave, 2021a).

---

3.4 Results
We follow standard conventions and report exact match (EM) scores using the reference answers included in each dataset. Table 2 shows our main results. We divide the table into three main sections: flosed-book QA models, open-book QA models, and our implementation. The first two sections contain results from other papers, which we include for comparisons. The last section includes results from our proposed model, as well as our reimplementation of relevant baselines to control for our experimental setup.
Our reimplementation of T5-base provides strong baselines when the number of retrieved documents is set to 0 (no retrieval) and 1. From Table 2, we see that the setting of top-1 vastly improves performance over the setting with no retrieved documents, signifying the importance of retrieval for OpenQA tasks. When further increasing the top-k documents to 50, the performance of the FiD models substantially improves over the top-1 retrieval, verifying the observation from (Izacard and Grave, 202 about the importance of modeling the retrieved documents as a set.
Comparing EMDR² with our reimplementation of FiD illustrates the benefit of our end-to-end training approach. The underlying model is similar in both cases, but the training method is different. FiD adopts a two-stage approach to first train the retriever and then the reader. We have three variants of FiD: (i) the reader and retriever are initialized with MSS training, (ii) the retriever is initialized with DPR training. which is the setting used in the original paper (Izacard and Grave, 2021b), and (iii the retriever is initialized with MSS + DPR training from (Sachan el al., 2021), as it further improves DPR recall. EMDR² outperforms all the variants by large margins on all the datasets.
The current best approach for training multi-document reader and retriever is FiD-KD (Izacard and Grave, 202 aa). FiD-KD is complex training procedure that requires multiple training stages and performs knowledge distillation with inter-attention scores, We take the results from the original paper when comparing our model with FiD-KD. EMDR² outperforms the reported numbers of FiD-KD by more than 2.5 points on NQ and TriviaQA to obtain new state-of offte-of-thh-arr results on these benchmarks.
In addition to better performance, EMDR² also has three other advantages compared to FiD-KD: (i) EMDR² is more efficient since it only uses 50 evidence documents, whereas FiD-KD leverages 100 documents; (ii) FiD-KD is based on a distillation approach which requires multiple cycles of retriever and reader training, while EMDR² only requires one cycle of end-to-end training; and (iii) FiD-KD relies supervised initialization of the retriever achieve its best performance. EMDR² is more robust to the retriever initialization, as demonstrated by state-of-the-art results even with unsupervised initialization of the retriever.
For the WebQ dataset, the training set size is much smaller compared to the other datasets (Table 5). Previous approaches such RAG rely supervised transfer (i.e., they finetune model pre-trained on NQ) to obtain good results. In contrast, EMDR² improves over the results from this RAG model by 3.5 points without the supervised transfer step. This result demonstrates the applicability of our approach to the low-resource setting where we only have a limited number of training examples. We also perform qualitative analysis of the model outputs, which is included in Appendix E.
3.5 Ablations
Number of retrieved documents. We investigate the performance of EMDR² and FiD as we vary the number of retrieved documents K in Figure 2. We observe that when the number of retrieved documents is increased, both EMDR² 2 and FiD improve in performance. When K is small, the gap between EMDR² and FiD is larger. This indicates the efficacy of EMDR² in a more constrained setting where we can only retrieve small number of documents (e.g., due to memory limitations).
Retriever initialization. We explore the effect of different parameter initialization strategies when training with EMDR²: (i) unsupervised MSS pre-training, (ii) supervised retriever training (DPR), and (iii) MSS pre-training followed by supervised retriever training (MSS- DPR; Sachan al. (2021)). Table 3 shows our results. We can see that on NQ, MSS pre-training being unsupervised leads to a lower initial retriever recall than DPR. After EMDR² training, the recall improves by 20% (highlighted in yellow cells). Training with DPR initialization leads to the same final recall as obtained by MSS
We note that GPT-3 is not trained on the full training examples that we use, so the results are not directly comparable.
8

---

Natural Questions (dev)
TriviaQA (dev)
WehQuestions (dev)
Exact soore
50
Exact Match score
Exact Match score
60
60
EMDR
10
EMDR
FID
FID
10
201
53
10
20
top- documents
lop-k
number
top-k documents
Figure Performance NQ, TriviaQA, and WebQ we vary the number of retrieved documents.
NQ (dev)
TriviaQA (dev)
WebQ (dev)
Retriever Initialization
Reader
R@50
EM
@550
EM
R@
EM
Initialization
B.T.
A.T.
B.T.
A.T.
B.T.
A.T.
MSS pre- see-training
MSS pre-training
66.4
86.3
50.4
74.8
86.2
71.1
59.8
88.6
49.9
MSS pre- vre-training
T5
66.4
86.3
50.3
74.8
86.3
70.9
59.8
88.6
47.7
DPR training
TS
82.3
86.3
50.0
83.2
86.2
70.5
84.2
88.6
49.0
MSS DPR
MSS pre-training
84.5
86.3
50.5
85.3
86.3
71.2
85.0
88.6
49.9
Table 3: R@50 denotes the retrieval recall from the top-50 retrieved documents. B.T. and A.T. indicates R@50 score Before Training and After Training the model, respectively.
pre-training, suggesting that DPR initialization of the retriever may not be essential component to obtain good performance in OpenQA tasks. Similar trends are also observed on TriviaQA and WebQ. Similarly, MSS DPR initialization has better initial recall but leads to marginal or no improvements in answer extraction performance over MSS pre-training. Finally, we also observe that MSS pre-training also provides an improvement of 2 points in answer extraction WebQ when compared to the T5 reader (shown in orange cells), highlighting its importance in the low-resource OpenQA tasks.
3.6 Alternative End-to-End Training Objectives
We compare EMDR² objective (Eq. 6) to two alternative formulations for end-to-end training.
In the first alternative formulation, when training the re- triever parameters we simply factorize p(Z q; 4)
Method top-k
NQ
TriviaQA
WebQ
I p(=x qi 2) to arrive at the following objective:
FiD EMDR²
50
47.3
65.5
46.0
K
50.4
71.1
49.9
Can logp(a q, 2) +
Σ 1=1
log p(z((z 2)
Car Lanz
14.1
11.9
28.0
49.9
69.6
28.8
The second term in this objective is maximised by uni- form retrieval, in other words, by removing any discrim- ination between documents in the retriever. We include it show the impact of adversarial objective.
Table 4: EM scores on the development set for alternative training objectives.
In the second formulation, for each retrieved document, we approximate its posterior under the as- sumption that have uniform prior over the set of retrieved documents: p(2x q, a, 2up- Q) x
p(a q, zg; +) x - We use this train reader and retriever parameters follows:
Cak-2 p(a q. Z: θ) + KSG (p(* q. a, Zop-K: θ)) || p(** q, Z:
Intuitively, we try to match the probability of retrieving document 2 with the "contribution" of that document to the generated answer a, regardless whether the retriever relatively more or less likely to retreieve the document priori.
Table shows our results on the development set of NQ. We observe that training with the adversarial Catt11 objective diverges, leading to poor performance, expected. This shows that harming the retriever during training can significantly harm performance of the QA system. contrast, although it disregards the estimated prior, the Cak-2 objective still improves over the FiD baseline for NQ and

---

TriviaQA. However, it still lags behind EMDR². On WebQ, the Lak-2 objective diverges and leads to poor performance. We leave further analysis on the convergence of L objective as a part of future work.
4 Related Work
Our work is based on end-to-end training of neural readers and retrievers, which we discuss in $1, $2, and $3. Here we instead focus discussing previous work related to standalone neural retrievers, neural readers, and their application in other natural language processing tasks.
Neural retrievers. There are two broad classes of neural retrievers based on the number of embed- dings computed for document: dual encoders (Yih al., 2011, Lee al., 2019) and multivector encoders (Khattab and Zaharia, 2020, Luan et al., 2021). Dual encoders store one embedding for each evidence document. Multivector encoders require multiple embeddings, which can be computa- tionally expensive for large-scale retrieval. Due to the large size of the evidence document collection in OpenQA, our work uses the more efficient dual-encoder. Sachan et al. (2021) show that the performance of supervised dual encoders in OpenQA can be improved when pre-training with the Inverse Cloze Task for the high-resouree setting or masked salient spans for the low-resource setting.
Neural readers. Neural readers output answer given retrieved documents as its input. There are also two broad classes of neural readers: extractive and generative. Extractive readers (Clark and Gardner, 2018, 8, de Masson d' Autume et al., 2019, Wang et al., 2019, Guu et al., 2020, Karpukhin et al., 2020) extract a span from a retrieved document to produce an answer. Generative readers (Izacard and Grave, 2021b), on the other hand, generates answer conditioned on the retrieved documents.
Other application areas. In addition to question answering, retrieval- -aggenetd methods have been successfully applied to other natural language processing tasks. In left-to-right language modeling. retrieving similar words from an external memory has been shown to improve perplexity (Khandelwal al., 2020, Yogatama al., 2021). In machine translation, retrieving domain-speciff target language tokens has improved performance in domain adaptation (Khandelwal et al., 2021). Finally, in dialog modeling, retrieving mnowledgeiiformeed informed text has helped improve factual correctness the generated conversations (Fan et al., 2021).
We provide detailed comparison of EMDR² with some of the previous work in Appendix C and D.
5 Discussion
Summary of contributions. We presented EMDR?, an end-to-end training method for retrieval- augmented question answering systems. We showed how arrive at our training objective using the expectation-maxiiizttioo maximization algorithm. We demonstrated that EMDR² achieves state-of-the-art performance on three benchmark OpenQA datasets.
Technical limitations. EMDR² shares few limitations with other retrieval- -urieete- question answering models. particular, as evidence documents are stored in an uncompressed format, maintaining them and searching for relevant documents can be expensive (both in terms of compute and memory consumption).
our experiments, we only focused on open-domain question answering. It would be interesting to see how EMDR² performs for other text generation models as well. We also note that training is relatively resource-heavy (requiring 16 GPUs), potentially having environmental concerns.
Potential negative socictal impacts. While EMDR" has the potential to improve language models in the low-resource setting (as demonstrated by results on WebQ in $3.4), it could exhibit typical biases that are associated with large language models. For example, our model does not have an explicit mechanism to generate answers that are calibrated for fairness across all spectra. As auumentl- method, it also could be more prone to generating fake answers an attacker manages to have access and modify information in the collection of evidence documents.
10

---

Acknowledgements
The authors would like to thank the Deep Language team, Mila's students, and anonymous reviewers for providing us valuable feedback and useful suggestions about this work that helped us improve the paper.
Funding Statement
DSS was supported by the Canada CIFAR AI Chair held by Prof. William Hamilton.
References
Berant, J., Chou, A., Frostig, R., and Liang, P. (2013). Semantic parsing on Freebase from question- answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.
Bromley, J., Guyon, I,, LeCun, Y., Säckinger, E., and Shah, R. (1994). Signature verification using a "siamese" time delay neural network. In Advances in Neural Information Processing Systems.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems.
Chen, D., Fisch, A., Weston, J., and Bordes, A. (2017). Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers).
Clark, C. and Gardner, M. (2018). Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
de Masson d' d'Autume, C., Ruder, S., Kong, L.,, and Yogatama, D. (2019). Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems.
Dempster, A., Laird, N., and Rubin, D. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, Series B, 1(39), 1-38.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirec- tional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Fan, A., Gardent, C., Braud, C., and Bordes, A. (2021). Augmenting Transformers with KNN-Based Composite Memory for Dialog. Transactions of the Association for Computational Linguistics,
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). Retricval augmented language model pre-training. In Proceedings of the 37th International Conference Machine Learning.
Iyer, S., Min, S., Mehdad, Y., and Yih, W. (2021). Reconsider: Re-ranking using span-focused cross- gross-attention for open domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers).
Izacard, G. and Grave, E. (2021a). Distilling knowledge from reader retriever for question answering. In International Conference on Learning Representations.
Izacard, G. and Grave, E. (2021b). Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.
11

---

Joshi, M.. Choi, E.. Weld, D., and Zeitlemoyer, L. (2017). TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers).
Karpukhin, V., Oğuz, B., Min, S., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. (2020). Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations.
Khandelwal, U., Fan, A., Jurafsky, D... Zettlemoyer, L., and Lewis, M. (2021). Nearest neighbor machine translation. In International Conference on Learning Representations.
Khattab, O. and Zaharia, M. (2020). Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In The 2015 International Conference for Learning Representations.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, Lee, K., Toutanova, K. N., Jones, L.., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics.
Lee, K. Chang, M.-W., and Toutanova, K. (2019). Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020a). BART: Denoising sequence-to-seen sequeene pre- tre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t,, Rocktäschel, T., Riedel, S., and Kiela, D. (2020b). Retrieval-augmented generation for knowledge-intenet intensive nip tasks. In Advances in Neural Information Processing Systems.
Luan, Y., Eisenstein, J., Toutanova, K., and Collins, M. (2021). Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9.
Min, S., Chen, D., Hajishirzi, H., and Zettlemoyer, L. (2019). discrete hard EM approach for weakly supervised question answering. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.., Gimelshein, N., Antiga, I.,, Desmaison, A., Kopf, A., Yang. E.,, De Z.,, Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems.
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. (2019). Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
Qi, P., Zhang, Y., Zhang. Y., Bolton, J., and Manning, C. D. (2020). Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
12

---

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M.. Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2144) 1-67.
Roberts, A., Raffel, C., and Shazeer, N. (2020). How much knowledge can you pack into the parameters language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Robertson, S. and Zaragoza, H. (2009). The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval.
Sachan, D. S., Patwary, M., Shoeybi, M., Kant, N., Ping, W., Hamilton, W. L.. and Catanzaro, B. (2021). End-to-end training of neural retrievers for open-domain question answering. Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP).
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-Im: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint 1xX::1900..0053.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing
Systems.
Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., and Jiang. J. (2018). R3: Reinforced ranker-reader for open-domain question answering. In AAAI.
Wang, Z., Ng. P., Ma, X., Nallapati, R., and Xiang, B. (2019). Multi-passage BERT:
globally normalized BERT model for open-domain question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNLP)),
Yih, W.-t., Toutanova, K., Platt, J. C., and Meek, C. (2011). Learning discriminative projections for text similarity measures. In Proceedings of the Fiffeenth Conference on Computational Natural Language Learning.
Yogatama, D., de Masson d' Autume, C., and Kong, L. (2021). Adaptive Semiparametric Language Models. Transactions of the Association for Computational Linguistics, 9, 362-373.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Please see the model ($2) and result ($3) sections that solidify the claims made in the abstract and introduction sections.
(b) Did you describe the limitations of your work? [Yes] Please see limitations in $5.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see negative societal impact in $5,
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We include the code, data, and instructions in the supplemental material and $3.2.
(b) Did you specify all the training details (c.g., data splits, hyperparamettrs, how they were chosen)? [Yes] We specify these details in the appendix included in the supplcmentary material.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Our experiments are compute expensive and it is not feasible to perform multiple runs of the same experiment with different seeds. All our training runs the same seed value (1234). As alternative to running multiple seeds, we perform number of ablation studies ($3.5).
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] Please $3.2 under hardware and library.
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Please see $3.1 for the details.
(b) Did you mention the license of the assets? [Yes] Our work based open-source data and framework. When applicable, we describe the license information in the appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We include our code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zO2zKTPoL7Y", "reviewer": "Reviewer_gGYZ", "summary": "This paper focuses on the open-domain question answering task. They use the generate framework which first retrieve relevant documents for the given question and then generate the answer based on the question and the retrieved documents. They model the retrieved documents in the latent space. Previous studies usually consider only a single document as the latent variable. In this paper, they consider to use a set of documents as the latent variables. They then apply an expectation-maximization algorithm to update the reader and the retriever. The reader is optimized to generate the answer based on the question and the retrieved set of documents. The retriever is updated using posterior estimation of the retrieved results given the question and the answer. They conduct experiments on three open domain question answering tasks. The experimental results show that their approach outperforms other baselines by a large margin (2%-3% absolute improvement). They conducted ablation studies to show the performance of the model with different number of retrieved documents. They also compared different retriever initialization  strategies and showed that masked salient spans pretraining is helpful.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality:\nThe authors propose to model a set of retrieved document instead of single document in the latent space. This is different from previous work such as REALM. Based on this intuition, the EM algorithm is straightforward to come up with and the objective is similar to the one in REALM.\n\nQuality:\nTheir experimental results show that their model can significantly outperforms all the baselines. The also conducted a set of ablation study which show that their model performs consistently good with different number of retrieved documents. But they did not do the experiments to show that it is important to model a set of retrieved documents instead of a single one. For example, they could conduct preliminary experiments to show how many questions need to be answered based on multiple documents, and do the analysis to show that their model can performs much better especially on the cases that required multi-document modeling. Without showing this,  their motivation is not well supported by the experiments.\n\nClarity:\nThe paper is well-written and easy to understand. The approach is intuitive and straightforward. The authors will release the code for further research and reproducibility.\n\nSignificance:\nThe authors achieve state-of-the-art performance on three popular dataset for open domain question answering, which is significant.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The model may have typical biases caused by the large language models. Another potential limit is the model could be attacked to generate fake answers when the evidence documents are modified.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ky5h7BUsaMD", "reviewer": "Reviewer_aV2Y", "summary": "The paper proposes end-to-end multi-document retriever-reader training for open-domain question answering. The proposed model is based on a pipeline approach where the retriever based on a dual encoder retrieves a set of documents from a large text corpus and a reader processes them and generates the answer. Different from previous approaches which independently train the retriever and the reader (Karpukhin et al, Izacard & Gave, and others), the paper jointly trains them using the EM algorithm. It is also different from previous work with joint training (Lee et al, Guu et al, and others) in that the generation of the answer is conditioned on multiple documents, not just a single one. In particular, this difference makes joint training more difficult, so the paper proposes to use the EM algorithm to iteratively update the parameters of the retriever and the reader. The paper also came up with clever initialization tricks and asynchronous refreshes of the MIPS index, which leads to successful training of the proposed model.\nThe proposed model is evaluated on widely-used open-domain QA benchmarks (NQ, TriviaQA, WebQ). The proposed model outperforms a set of other, recently-proposed models by a large margin, which is impressive given that the baselines are highly competitive.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The problem is well-motivated, with clear distinctions from prior work.\n- The model is well-executed, with the key method like EM algorithm, and tricks including initialization and asynchronous updates. All the descriptions of the models are very clear.\n- The model achieves impressive results on competitive open-domain QA benchmarks.\n\nWeaknesses\n- I think the paper should discuss more thorough comparisons to REALM (Guu et al) because there are a lot of shared aspects. For example:\n    * The asynchronous updates are almost identical.\n    * The high-level idea of REALM pretraining is very similar to the initialization in this paper (using masked LM as sort of pseudo data to pretrain the model). Do you think these two methods have exactly the same role, or do you think there are features that your initialization can capture but REALM pretraining cannot?\n  * The difference in answer generation being conditioned on single vs. multi documents (hence leading to the need for the EM algorithm) is clearer. Additional discussions I want to see are mainly about the above two points.\n- Framing retrieval documents as a latent variable is first done by Lee et al. 2019. It should be more highlighted.\n\n\nQuestions\n- Asking based on ablation in Table 3\n    * What is P@50? What “retrieval accuracy” means is unclear. Do you mean recall accuracy? (The model gets a credit if at least one of top-50 docs contain the answer text.) In that case, it might be better to say “R@50” or “Rec@50” then, and clarify in the caption?\n    * It looks like DPR-style supervised training does not impact the performance very much based on this metric. Have you compared these two methods based on the EM score?\n- Have you tried a baseline where you initialize the retriever and the reader using MSS pretraining, but then train them separately, in a pipeline manner? I think this is an important baseline.\n\n\nMinor comments\n- L19: “train a language model”: I am not sure if it is necessary to train a “language model”?\n- L32: I would cite Karpukhin et al for the pipeline training as it is the first paper trying it, and Izacard & Grave mainly took the model from Karpukhin et al.\n- Table 1: It might be good to add a column “requiring document supervision” which makes the proposed model more appealing - I think this column will be checked only for REALM and the proposed model.\n- Section 3.2: Which T5 variant was used for the reader? I am guessing it is T5-base, but it isn’t clearly mentioned in the paper.\n- Section 3.3: I think the terms “parametric” and “semi-parametric” are confusing, and are not used in any prior work. What about “closed-book” vs. “open-book” (following terms in Roberts et al) or “no retrieval” vs. “retrieval”?\n\n\nConclusion & Overall recommendation\n\nThe paper proposes a model for open-domain QA, where the answer is generated conditioned on multiple documents jointly, and the retriever and the reader are trained end-to-end. The proposed idea is new and substantially different from prior work; the model is well-executed; the experiments are comprehensive, showing impressive results on competitive open-domain QA benchmarks. I have minor concerns about more comparisons with prior work and additional baselines, which are relatively easy to address. Overall, the quality of the paper is substantially higher than the bar to be published.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper adequately discusses limitations and societal impact. It would be also good to mention that training is relatively resource-heavy (requiring 16 GPUs), potentially having environmental concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "eaeVhSoBlSk", "reviewer": "Reviewer_6UDc", "summary": "This paper proposes EMDR2, an end-to-end differentiable training method of multi-document reader and retriever for open-domain question answering. Based on expectation-maximization algorithm, EMDR2 iteratively estimates the set of relevant documents for a given question and uses the estimated set to update the parameters of retriever and reader. EMDR2 achieves new SOTA performance for models of comparable size on three benchmark OpenQA datasets.\nContributions:\n-\tAn **EM-based** end-to-end training method of **multi-document** reader and retriever is proposed, which achieves SOTA performance on three benchmark OpenQA datasets.\n-\tThe observation that EMDR2 implementations with different retriever initializations achieve similar final retrieval accuracies may be instructive.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n-\tThis is new end-to-end training method for open-domain QA. Although the differences from previous work (such as REALM, RAG, and FiD) are clear, EMDR2 looks like a better combination of their techniques. The new contributions compared to Joint Top-k (Sachan et al., 2021) in Table 3 seem to be somewhat incremental and conventional.\n-\tAlthough a few end-to-end training methods for OpenQA are mentioned in introduction, there is no more comprehensive and in-depth literature review for such methods in related work.\n\nQuality: \n-\tThe handling of technology and its terminology sounds correct and makes sense, but some intuitive explanations are lacked, e.g., the assumption on line 129 and the reason for using stop-gradient operator.\n-\tThe work is complete and its empirical results seem to be impressive. But the necessity of considering multiple documents during end-to-end training and inference for single-hop questions from the evaluated three datasets is not intuitive explained and experimentally proved. Maybe a variant of EMDR2 whose reader only takes as input a question and one document (i.e., Z_reader = Z_{top-1}) should be considered. The result comparison between the variant and EMDR2 could prove such necessity, and the comparison between it and the Individual Top-k (Sachan et al., 2021) can show the effectiveness of updating the retriever using the posterior.\n\nClarity: \n-\tThe paper is easy to follow, and an expert reader may reproduce the proposed methods and results with some effort. However, the core motivation and main contributions are not highlighted, which brings ambiguity to the proposed method's novelty.\n-\tThe conclusion on lines 47 to 48 is inconsistent with that on lines 260 to 261.\n-\tShould the metric P@50 be Recall@50? It is abnormal for P@50 having values greater than 80%, which means there are more than 40 relevant documents for a question from the three datasets. \n\nSignificance: \n-\tEMDR2 provides a better way for end-to-end training of multi-document retriever and reader than Joint Top-k (Sachan et al., 2021), and achieves SOTA performance on three benchmark OpenQA datasets. It may bring improvements in training methods for generative OpenQA and other retrieval-augmented NLG tasks.\n-\tThe observation that EMDR2 implementations with different retriever initializations achieve similar final retrieval accuracies may be instructive.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "cKTqBWmYak", "reviewer": "Reviewer_p6SD", "summary": "This paper makes end-to-end training work for multi-document readers (e.g., FiD style readers). The key idea is an EM algorithm to update the reader and retriever given the top-k selection of previous iteration models. The proposed approach shows good performance on multiple open-domain QA benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes end-to-end training work for multi-document readers (e.g., FiD style readers). The key idea is an EM algorithm to update the reader and retriever given the top-k selection of previous iteration models.\n\nAlthough I think that the novelty of the EM algorithm is a bit limited (mathematically, the Hard-EM, REINFORCE, REALM and iteratively updating are closely related), the proposed approach demonstrates a clear advantage on multiple open-domain QA benchmarks. Also given that multi-document readers like FiD have become the mainstream in many QA tasks, it is necessary to have a joint-training algorithm like REALM be extended to these readers. Therefore I still would like to vote for acceptance.\n\nIt will be great if the following two questions can be addressed in the author rebuttal:\n\n(1) To deal with the posterior, in Eq. 4 the authors make an approximation, which reduces the multi-document reader to single-document reader. On the other hand, could I understand the L_{alt-2} objective in Section 3.6 is equivalent to either Hard-EM [1] or REINFORCE [2]. If the answer is NO, detailed discussions about the differences from [1,2] should be added; if YES, this L_{alt-2} should be added as a baseline to Table 2 under \"Our Implementation\". In whichever case, I think the discussion of L_{alt-2} should be highlighted in Section 3.4, because it is related to a straightforward way of applying what has been developed by the community [1,2], while can form an apple-to-apple comparison to the proposed approximation.\n\n[1] Min et al., 2019. A discrete hard em approach for weakly supervised question answering.\n\n[2] Wang et al., 2018. R3: Reinforced ranker-reader for open-domain question answering.\n\n(2) The reader of EMDR^2 is based on T5. If I am correct, the RAG and FiD works use BART instead. I am wondering how much benefit is caused by switching BART to T5. Therefore it would be helpful if EMDR^2 can be applied to the BART reader for comparison.\n\nTypo:\n\nLine 103: Eq. 3 (should be the equation in Line 101?)\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new algorithm for training of multi-document question answering model containing a reader and a retriever. The key idea is to introduce latent variables over the sets of retrieved documents, employ the EM algorithm to approximately estimate the variables, and update the parameters of the reader and the retriever. Experimental results show that the proposed approach outperforms the baselines for about 2-3% on three benchmark datasets.

Pros
* The problem studied is important.
* The paper is generally clearly written.
* The proposed approach appears to be reasonable and sound.
* Experimental results show the efficacy of the proposed approach. The improvements are significant.

Cons
* There were related models proposed in the past. The idea of the work is not very novel. The paper mainly demonstrates that a new way of combining the existing ideas does work better.
* There are some details that are not explained very clearly. The authors have promised to make them clearer in the revised version.
* The authors have also successfully addressed many of the issues pointed out by the reviewers in the rebuttal.

Overall this is a solid work. We think that the contribution to ML is mainly from the empirical results.


Decision: Accept (Poster)