Distilling Knowledge from Reader to Retriever for Question Answering
Authors: Gautier Izacard
Edouard Grave
Venue: ICLR 2021 Poster (2021)
Keywords: question answering
information retrieval

Abstract: The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based on retrieved documents, to obtain synthetic labels for the retriever. We evaluate our method on question answering, obtaining state-of-the-art results.

Published as a conference paper at ICLR 2021
DISTILLING KNOWLEDGE FROM READER TO RETRIEVER FOR QUESTION ANSWERING
Gautier Izacard 1,2,3 Edouard Grave! Faaebook AI Research, École normale supérieure, PSL University, 'Inria {gi zacardl eggaaee @ fb. com
ABSTRACT
The task of information retrieval is an important component of many natural lan- guage processing systems, such as open domain question answering. While tra- ditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods to obtain supervised data train the retriever model, cor- responding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based retrieved documents, to obtain synthetic labels for the retriever. We eval- uate our method on question answering, obtaining stae-of-the--n results.
INTRODUCTION
Information retrieval is an important component for many natural language processing tasks, such as question answering (Voorhees et al., 1999) fact checking (Thorne et al.] 2018). For example, many real world question answering systems start by retrieving set of support documents from a large source of knowledge such as Wikipedia. Then, finer-grained model processes these docu- ments to extract the answer. Traditionally, information retrieval systems were based on hand-crafted sparse representations of text documents, such as TF-IDF or BM25 (Jones) 1972] Robertson et al.] 1995). Recently, methods based on dense vectors and machine learning have shown promising re- sults Karpukhin al., 2020 Khattab al.) 2020). Deep neural networks based on pre-training, such as BERT (Dvlin et 2019), have been used to encode documents into fixed-size representa- tions. These representations are then queried using approximate nearest neighbors (Johnson et al. 2019). These techniques have lead to improved performance on various question answering tasks.
challenge of applying machine learning to information retrieval is to obtain training data for the retriever. To train such models, one needs pairs of queries and the corresponding list of documents that contains the information corresponding to the queries. Unfortunately. hand-labeling data to that end is time consuming, and many datasets and applications lack such annotations. An alternative approach is to resort to heuristics, or weakly supervised learning, for example by considering that all documents containing the answer are positive examples. However, these approaches suffer from the following limitations. First, frequent answers or entities might lead to false positive examples. As an example, consider the question "where was Ada Lovelace born?". The sentence "Ada Lovelace died in 1852 in London " would be considered as positive example, because it contains the answer "London" ". A second limitation that for some tasks, such as fact checking or long form question answering, such heuristics might not be applicable directly.
In this paper, we propose procedure to learn retriever systems without strong supervision in the form of pairs of queries and documents. Following previous work (Chen et al. 2017), our approach uses two models: the first one retrieves documents from large source of knowledge (the retriever), the second one processes the support documents to solve the task (the reader). Our method is inspired by knowledge distillation (Hinton et al., 2015). and uses the reader model to obtain synthetic labels to train the retriever model. More precisely, we reader, and use the attention activations over the input documents as synthetic labels to train the retriever. Said otherwise, assume that attention activations are good proxy for the relevance of
sequence-tor pequence-to-sequence model as the

---

Published as a conference paper at ICLR 2021
documents. We then train the retriever to reproduce the ranking of documents corresponding that metric.
We make the following contributions:
First, we show that attention scores from a sequence-to-sequeecc reader model are a good measure of document relevance (Sec. 3.2)
Second, inspired by knowledge distillation, we propose to iteratively train the retriever from these activations, and compare different loss functions (Sec. 3.4)
Finally, evaluate method three question-ansserrng benchmarks, obtaining state- of-the-art results (Sec.
Our code is available at: gi hta.ab.ccm/acccbbokkeeeacccc/c//
RELATED WORK
We briefly review information retrieval based on machine learning. We refer the reader to Manning et al. 2008) and Mitra et al. (018) for a more exhaustive introduction to the subject.
Vector space models. In traditional information retrieval systems, documents and queries rep- resented as sparse vectors, each dimension corresponding to a different term. Different schemes have been considered to weigh the different term, the most well known being based on inverse doc- ument frequency, or term specificity (Jones) 1972). This technique was later extended, leading to the BM25 weighting scheme which still widely used today (Robertson et al. 1995). limita- tion of sparse representations is that the terms of the query need to match the terms of the returned documents. To overcome this, Deerwester et al. (1990) proposed to use latent semantic analysis for indexing, leading to low-dimension dense representations of documents.
Neural information retrieval. Following the success of deep learning for other natural process- ing tasks, neural networks were applied to the task of information retrieval. [Huang al. (2013)
proposed a deep bag-of-wordh model, where queries and documents were embedded independently. technique known as bi-encoder: Documents the query, and the model was trained on clickthrough data from a search engine. This technique was later extended by using convolutional neural networks (Shen et al. 2014) and recurrent neural net- works (Palangi et al. 2016). A limitation of independently embedding documents and query is that it does not capture fine- onn-grained interactions between the query and documents. This lead Nogueira & Cho (2019) and Yang et al. (2019) to use BERT model to jointly embed documents and query, a technique known as cross-encoder.
then ranked by using the cosine similarity with
End-to-end retrieval. Most of the methods described in the previous paragraph were used to re- rank a small number documents, usually returned by a traditional IR systems. the context of ad-hoc document retrieval, Gillick al. (2018), showed that bi-encoder models could be com- petitive with traditional IR systems. For open domain question-answertee answering, Karpukhin et al. (2020) introduced dense passage retrieval (DPR), which uses dense embeddings and nearest neighbors search. More precisely, question and passage embeddings are obtained using a BERT-based bi- encoder model, which is trained a small dataset of question and passage pairs. Then, the full knowledge source (Wikipedia) is encoded using this model, and passages are queried by computing the k-nearest neighbors of the embedding of the question. Jointly embedding the query and docu- ments makes the application of cross- cross-encoder models intractable to large database. To address this limitation, [Humeau al. (2019) introduced the poly-encoder architecture, in which each documents is represented by multiple vectors instead of one. Similarly,KKaattbb Khattab et al. (2020) proposed scoring function where each term of the query and documents represented by single vector. To make the method tractable, their system retrieves documents with an approximate score, which are then re-ranked with the exact one. Finally, Luan et al. (2020) conducts a theoretical and empirical study of sparse, dense and cross-attention information retrieval systems.
Unsupervises learning. Closest to our work, there is growing body of work trying to learn infor- mation retrieval systems from unsupervised data. Lee al. (2019) introduced the inverse cloze task
2

---

Published as a conference paper at ICLR 2021
for pre-training retrievers, which can then be fine-tuned end-to-end on dulestion-answerrn tasks. This pre-training scheme was later evaluated for ad-hoc document retrieval by Chang et al. (2020).
Guu et al. (2020) proposed to augment language model pre-training with a retrivvr module, which is trained using the masked language modeling objective. Similarly, |ewis et al. (2020a) introduced sequence-to-sequence model that is pre-trained by generating target text, after retrieving set of related texts. Lewis et al. (2020b) further train the retriever obtained in Karpukhin et al. (2020) by
backpropagating to the retriever the error between the generated output and the gold answer.
Simultaneously to our work. Yang & Seo 2020) proposes to train retriever with knowledge dis- tillation. The main difference with our method is the nature of the synthetic labels that are used to train the retriever. Yang & Sco) (2020) uses the DPR reader, which includes a classifier that predicts which passage contains the answer, and can be seen as a cross-encoder reranker. This technique thus performs the distillation of a cross-encoder retriever to bi-encoder retriever. In contrast, our method uses the internal attention scores of the reader, which does not require additional supervision besides pairs of question and answer.
3 METHODOLOGY
Our system is composed of two modules, the retriever and the reader, following the standard pipeline for open-domain question answering. Given an input question these modules are used in a two-step process generate answer. First the retriever selects support passages in a large knowledge source. Then these passages are processed by the reader, along with the question, to generate an answer. For the reader module we use the Fusion-in iesionnin-Decoder model (Izacard Grave 2020). which achieves state-of-thh-ae performance when combined with BM25 or DPR (Karpukhin et al. 2020). It based on prruuecce-o-sequeer
architecture, and is initialized from pre-trained models such as T5 or BART (Raffel et al. 2019, Lewis et al.. 2019).
The focus of this work to train the retriever without strong supervision or weakly supervised learning based on heuristics. For this we propose to train the retriever by learning to approximate the attention score of the reader. The training scheme outlined here can be seen as a student-teacher pipeline, where the teacher, the reader module, produces targets which are used to train student network, the reader. By doing so, we hope to leverage the signal extracted from the question-answer pairs by the reader. Since the goal of the retriever is to retrieve the most relevant passages, by training the retriever to estimate the reader attention scores, we implicitly make the assumption that these scores
good proxy for the usefulness of a passage answer the question.
In this section we will first describe the signal which is used train the retriever, the design of the retriever, and how this module is trained.
architecture, before elaborating on the
3.1 CROSS- MECHANISM
First, let us briefly review the Tusion-in-Decoder model (FiD, Izacard & Grave 2020). The under- lying architecture is goquence-to-squemee model, composed of encoder and a decoder. The encoder independently processes ntp different text inputs (sk)i<k<n,. In the case of open-domain question answering based on Wikipedia, each input Sy the concalenation of the question q and a support passage, with special tokens guestion: title: and context: : added before the question, the title of the Wikipedia article and the text of each passage. The output representations of the encoder are then concatenated to form a global representation X of dimension (E; 1. (x) x d, where l is the length of the k-th segment and d the dimension of the embeddings and hidden rep- resentations of the model. Then, the decoder processes this representation as a regular autoregressive model, alternating self-attention, cross- attention and feed-forward modules.
Only the cross-attention module explicitly takes as input the global output representation X of the encoder. H E IR denotes the output the previous self-attention layer the decoder, the cross- attention operation consists in the following operations. First, queries Q keys K. and values V are computed by applying linear transformations:
Q= @ K WxX, V= W.X.
3

---

Published as a conference paper at ICLR 2021
Then similarity score between the query position i, Q2, and the key at position j. K,, is obtained by computing the dot- nro-product between these two elements, and normalized over the dimension:
exp(0s.)
o = Q K₁,
Oli E exp(Ormm)
A new representation is obtained as sum of the values, weighted by the attention probabilities, before going through a final linear transformation Wo:
O Wo Σ āigV
The operations described above are performed in parallel with different linear transformations in the case of multi-head attention. Finally a normalization layer is applied, and this pipeline is wrapped by skip connection. See Vaswani et al. (2017) for more details on the structure of Transformers.
3.2 CROSS-ATTENTIO SCORE AS A RELEVANCE MEASURE FOR PASSAGE RETRIEVAL
In some sense, the attention scores a:, involving the j-th key measures the importance this key, and corresponding value, to compute the next representation. We hypothesize that it is good proxy to estimate the relevance of a passage - the more the tokens in a text segment are attended to, the more relevant the text segment is to answer the question.
Given the reader model, an input question and a corresponding set of support passages D, (p*) we obtain relevance scores (G Siskán for each passage by aggregating attention scores. In particular, the score Ga.ny is obtained by averaging the pre-attention scores co, over all the tokens in the input s3 corresponding to the passage Pk. all the layers and all the heads of the decoder. Note that the FiD decoder jointly processes the passages, and thus the score Gg.p.. de- pends on the other support passages. We consider other pooling operators, such as max, to aggregate attention scores over layers, heads and tokens and empirically compare them in Sec. 5.2
Before we proceed, let us consider the following simple experiment, which first indication that reader attention scores are indeed a strong relevance signal. Given question and 100 passages retrieved with DPR, our goal is to select the 10 best passages. When using the top 10 passages from DPR instead of the top 100, the performance of our reader drops from 48.2 EM to 42.9 EM. On the other hand, we select the top 10 documents according to the attention scores, the performance only drops to 46.8 EM.
3.3 DENSE BI-ENCODER FOR PASSAGE RETRIEVAL
Ideally, we would like to rank passages according to the reader cross-attention scores. In practice however, since the passages and the question need to be processed simultaneously by the reader module it is impractical to query a large knowledge source this way. Thus, we use a retriever model composed of an embedder function E. that maps any text passage d-dimensional vector, such that the similarity score between a question & and passage is defined as So(g. p) = F(q)" E(p). This similarity metric enables us index all passages in the knowledge source as a preprocessing step. Then at runtime, passages with the highest similarity score with the input question are retrieved, by using an efficient similarity search library such as FAISS (Johnson et al. 2019).
For the embedder we use BERT and follow DPR by considering that the encodings E(g) and E(p) are obtained by extracting the representation of the initial [CLS] token. This leads to a represen- tation of dimension d = 768 in the case of a base model. Differently from DPR, we use the same encoding function E. for the questions and passages by sharing parameters.
3.4 DISTILLING THE CROSS-ATTENTION SCORE TO BI-ENCODER
In this section, we describe how to train the retriever model, based on the relevance scores obtained in Sec. [3.2] For the training objective of the retriever, we propose to minimize the KL-divergence between the output So p) and the score Gg.p after normalization:
(x, Q) = Σ
Glog GG.pr - log So So(g.))) p)),
VEQ,PED2
4

---

Published as a conference paper at ICLR 2021
where
exp(Ggp) g.p))
exp(So(g.p)) En exp(Sa(q,7))
G.pp
So(g. p)
Ey'ED, exp(Gg,r')
In Sec. 5.1 we present results obtained when using alternatives this training objective. We con- sider two other objectives which have been used in Dehghani et al. (2017). where BM25 is used as a teacher model to train neural ranker. A first option consists in training the retriever with regression approach by minimizing the mean squared error:
MSE (8, Q)
E
(So(g.p) G.
PER.pEDo
The second option we consider is to use a max max-margin loss that explicitly penalizes inversions the ranking estimated by the retriever:
Cranking (0, ②) =
Σ
max (0, sign(G (S) So(q,
TE PEQ.PI.PEED
In words, if P1 is more relevant to answer the question g than P2 i.e. Ggpr. > Gq.p.. the loss pushes the retriever score of p) to larger than the score of p2 by at least a margin
3.5 ITERATIVE TRAINING
In this section, explain how iterative training be used with the student-teacher scheme de- scribed in the previous section, similarly to Khattab et al.] (2020). This iterative procedure can be interpreted using the current retriever sample negative examples, in order train triever. When learning a retriever with discriminative training, negative samples play an important role, and various strategies have been considered in previous work. Karpukhin et al. (2000) com- pared random sampling with using the top-k passages from BM25 which do not contain the answer and with using the positive passages from other queries. Consider that for each question, have an initial set of support documents D We propose to use an iterative pipeline where each iteration can be described as the following step process:
1. Train the reader R using the set of support documents for each question Do
2. Compute aggregated attention scores (Gg.p))e with the reader It.
3. Train the retriever E using the scores DE
4. Retrieve top-passe pa-passsge with the new trained retriever E.
This multi-step procedure can be repeated multiple times. A critical point of the training procedure the initial of documents corresponding to each question. Sec. we compare retrievers obtained by starting from documents obtained using BM25 or cosine similarity from BERT model. In particular, we show that while the initial performance with BERT is low, the iterative procedure allows to greatly improve the performance of the model.
4 EXPERIMENTS
In this section we evaluate the student-teacher training procedure from the previous section. We show that we obtain competitive performance without strong supervision for support documents.
4.1 EXPERIMENTAL SETTING
Datasets. We perform experiments TriviaQA (Joshi al. 2017) and NaturalQues- tions (Kwiatkowski et al. 2019), two standard benchmarks for open-domain question answering. TriviaQA is made of questions from trivia and quiz league websites, and does not contain gold support documents. NaturalQuestions contains questions corresponding to web search queries, and
gold support documents from Wikipedia. Following the setting from Lee et al. (2019) Karpukhin
5

---

Published as a conference paper at ICLR 2021
BERT
BM25
Iter.
P@20
P@ 100
Dev EM
P@20
P@ 100
Dev EM
0
4.8
12.0
9.8
59.3
74.0
41.2
32.2
45.8
16.9
76.4
84.3
46.8
Natural QueutioQes
51.1
62.6
28.6
80.4
86.7
47.9
3
67.8
76.8
39.3
80.0
86.3
46.2
0
4.6
12.0
9.7
75.0
82.3
65.3
37.1
59.4
19.6
79.0
85.5
66.7
TriviaQA
2
60.8
73.4
43.3
82.1
86.5
67.5
72.0
83.2
52.0
81.6
86.6
67.7
4
76.4
84.6
62.3
Table I: Iterative training starting with documents retrieved with BERT and BM25. Iteration corresponds the performance of the reader trained the set of initial support documents. We report all metrics on the validation set.
al. 2220), use the original evaluation set as test set, and keep 10% of the training data for validation. We use the Wikipedia dump from Dec. 20, 2018 8 for support documents, splitting articles into non-overlapping passages 100 tokens, and applying the same preprocessing [Chen et al. 2017).
We also evaluate on NarrativeQuustions (Kočiský et al. 2018), using a publicly available prepro- cessed version This is reading comprehension dataset built on corpus of books and movie scripts. For each story, questions are generated by human annotators based on a summary of the given document. We consider the full story setting, where the task is to answer questions given the entire story and not the summary used to generate question-answer -nnswrr pairs. Here the knowledge source is not the same for all questions: given question the retrieval operation is performed all passages of the associated story. These passages are obtained by dividing the story in chunks of 100 words. These stories are long documents, with average of 60k words. While part of the documents could be processed entirely by the Fusion-in-Decodee module, it is interesting to limit the number of support passages to reduce the computational cost of the reading step.
While answers in TriviaQA and NaturalQuestions are short, NarrativeQA answers are about five words long on average, with medium length answers such "He dismantles and attaches it his mother's jeep" which answers the question "What does Mark do with his radio "tation".. Notably significant number of answers do not correspond to spans in the story. is thus not straightforward to train the retriever with heuristics using question-any answer pairs. In our case we use the same pipeline as for TriviaQA and NaturalQuestions, demonstrating the flexibility of our approach.
Evaluation. The model performance is assessed in two ways. First, following previous work such as DPR and ColbertQA, we report the top-k retrieval accuracy (P@k). which is the percentage of questions for which least one passage of the top-k retrieved passages contains the gold answer. is unclear how well this metric evaluates the retriever performance, since the answer can be contained passage without being related the question. This notably true for common words or entities.
We also report the final end-to-end performance of the question answering system composed of the retriever and reader modules. This the metric we are fundamentally interested in. For TriviaQA and NaturalQuestions predicted answers are evaluated with the standard exact match metric (EM), as introduced by RRjjuurarr et al. (2016). For NarrativeQA we report the metrics proposed in original paper: ROUGE-L, BLEU BLEU-4 and METEOR.
4.2 TECHNICAL DETAILS
Initialization. Similarly to DPR, we initialize the retriever with the BERT base model, pretrained with uncased text. The Fusion-in- inDecoeer reader is initialized with the T5 base model. A critical component of the iterative training procedure is the initialization of the support passages D asso-

---

Published as a conference paper at ICLR 2021
Model
NQ
TriviaQA
dev.
test
dev.
test
DPR (Karpukhin et al.
2020)
41.5
57.9
RAG (Lewis et al. (2220b)
44.5
56.1
ColBERI-QA (Khattab al.
2020
48.2
63.2
Fusion-in-Decoderr (15 base)
Izacard & Grave 2020)
48.2
65.0
Fusion-in-Decoder (T5 large)
Izaaard Grave 2020
51.4
67.6
Ours (starting from BERT, T5 base)
39.3
40.0
62.5
62.7
Ours (starting from BM25, T5 base)
47.9
48.9
67.7
67.7
Ours (starting from DPR, T5 base)
48.0
49.6
68.6
68.8
Ours (starting from DPR, T5 large)
51.9
53.7
71.9
72.1
Table 2: Comparison to state-of-the-art models on NaturalQuestions and TriviaQA.
ciated with each question q. For this we consider different options. The first one to use passages retrieved using BM25. We use the implementation from Apache Lucene with default parameters, and tokenize questions and passages with SpaCy We also use passages obtained with BERT as a retriever without fine-tuning, this leads to poor initial performance. Finally in Table 2 we show that initializing D. with passages obtained with DPR (Karpukhin et al. 2020) outperforms the two pre- 100 passages
vious initializations. We train all retrievers using 100 passages. For the reader, for NaturalQuestions and TriviaQA and 20 passages for NarrativeQA.
Iterative training. We apply the iterative training procedure on each dataset independently. Both the reader and the retriever are fine-tuned using the ADAM algorithm (Kingma & Ba 2014). with batch of size 64. The reader is trained for 10k gradient steps with a constant learning rate of 10-1, and the best model is selected based on the validation performance. The retriever trained with constant learning rate of 5 10 until the performance saturates. To monitor the performance of the retriever during training, we measure the similarity between the reader and the retriever rankings. At each new training iteration the reader is reinitialized from T5 base, while we pursue the training of the retriever. We found that restarting from T5 base is important for the first iterations when starting with BERT documents. We have not tried to reinitialize the retriever between each iteration. More details on the hyperparameters and the training procedure are reported in Appendix A.2
4.3 RESULTS
In I we report the performance of our approach for different number of self-training iterations. Generally, we observe that the accuracy of our system increases with the number of iterations, obtaining strong performance after few iterations. Interestingly, while the initial performance with documents retrieved with BERT is very poor, our method still reach competitive scores on TriviaQA, and to lesser extent, NaturalQuestions. However, a second observation is that the quality of the initial document sets plays an important role on the performance of the end system. Indeed, we observe that starting the procedure from BM25 documents, which are higher quality indicated by the performance of the system at iteration O, leads to stronger results than using BERT documents. An interesting research question would be to explore pre-training of the initial BERT model for retrieval, for example by using the inverse cloze task.
In Table 2 report the performance of approach, well existing state-of-th-aat systems on TriviaQA A and NaturalQuestions. In addition to initializing our method with documents retrieved with BM25 and BERT, we also train system by starting from DPR documents. First, observe that our method improve the performance over the state-of-the-art, even when starting from BM25 documents. This validates our assumption that it is possible obtain strong retrievers without the need of supervision for the documents. Second, when starting from DPR passages, our method leads +4.5 EM improvement on TriviaQA and +2.3 EM improvement on NaturalQuestions when the final evaluation is carried out with a large reader.
Jueeee www.ne..pache.c.. apache.org
spacy. ic

---

Published as a conference paper at ICLR 2021
Method
Iter.
Rouge-I.
Bleu-I I
Bleu-4
Meteor
dev.
test
dev.
test
dev.
test
dev.
test
Best from Kočiský al. (0088)
14.5
14.0
20.0
19.1
2.23
2.1
4.6
4.4
DPR + Fil
29.7
30.8
33.0
34.0
6.7
6.9
10.3
10.8
Ours starting from BM25 Ours starting from BM25
0
29.9
30.3
34.6
33.7
7.1
6.5
10.5
10.4
31.6
32.0
34.9
35.3
7.6
11.0
11.1
Table 3: Performance NarrativeQA.
In 3 report the performance of our method on the NarrativeQA dataset. We use the setting where the knowledge source corresponds to the whole document, and in particular, we do not use the summary. We compare our results the best ones reported in the original paper for this setting. Similar to results obtained on NaturalQuestions and TriviaQA, we observe that training the retriever by using the attention scores of the reader leads improvements, compared the BM25 baseline.
ABLATIONS
In this section, investigate design choices regarding two key elements of our approach: the training objective and the aggregation of cross-attention scores, For all experiments, we consider simplified experimental setting: single training iteration is performed on NaturalQuestions, starting from BM25 passages.
5.1 TRAINING OBJECTIVES
In Table 4 we report the performance of our model trained with the different training objectives described in Sec. 3.3 We observe that using the KL--divergence between the aggregated scores of the reader and the scores of the retriever outperforms the other objective functions.
Method
P@5
P@ 20
P@ 100
Dev EM
Mean Squared Error
46.5
61.2
73,9
40.6
Max-margin loss, =
60.3
73.6
82.7
45.4
Max-margin loss, 7 0.2
60.3
73.5
82.6
45.8
Max-margin loss, = 0.1
60.2
73.5
82.6
45.1
KL-divergence
64.7
76.4
84.3
46.8
Table 4: Comparison of training objectives on NaturalQuestions after one iteration. We report all the metrics on the validation set.
5.2 How TO AGGREGATE
SCORES?
In Section 4 the cross-attention scores a are aggregated in specific way, order obtain a single scalar used to train the retriever. Formally let denote by a.,jA,, the cross-attention scores between token i the output and token j of the input, for the k-th layer and h-th head. Then, the scores Gg.p for p D, used in Section 4 are computed as follows:
G mean O'o.j.k,l
q.P
j.k,h
where j describes the input tokens corresponding to p. Table we explore alternatives this choice by considering different aggregation schemes. In particular, we consider (1) taking the max over the input tokens corresponding to passage instead the average, (2) taking the average over the output tokens instead of taking the score of the first token, (3) taking the mean over the last six layers instead all the layers, (4) taking the max over the layers instead of the average, (5) taking the max over the heads instead of the average. We observe that the performance of approach relatively stable to the choice of aggregation, and that the best result is obtained by averaging, except the output tokens where it is best only consider the first token.
8

---

Published as a conference paper at ICLR 2021
Method
P@5
P@20
100
Dev EM
(0) j, &.,jkk,h
64.7
76.4
84.3
46.8
m maxj k.
61.2
72.5
81.0
46.0
(2) mean A
63.5
75.3
83.1
45.8
(3) mean75k<12.j.. A0,
64.1
75.7
83.8
46.4
(4) meallj, m&x; A.,
63.9
75.5
83.7
46.5
(5) j. max, &..j.ks.
64.2
76.1
83.9
46.8
Table 5: Comparison of attention aggregation schemes on NaturalQuestions after one iteration. The index i corresponds to output tokens, j corresponds to input tokens given passage, to heads and & to layers of the decoder. We report all metrics on the validation set.
6 CONCLUSION
In this paper, we introduce a method to train an information retrieval module for downstream tasks, without using pairs of queries and documents as annotations. Our approach inspired by knowl- edge distillation, where the retriever module corresponds to the student model and the reader module corresponds to the teacher model. In particular, we use the cross-attention scores, from a sequence- to to-sequence reader, to obtain synthetic targets for the retriever. We compare different ways to aggre- gate the scores, as well as different training objectives to learn retriever. We show that iteratively training the reader and the retriever leads to better performance, and obtain state-of-the-art perfor- mance on competitive question answering benchmarks. In the future, we would like to explore better pre-training strategies for the retriever module, well as better scoring functions for the retriever.
REFERENCES
Wei-Cheng Chang, Felix X Yu, Yin- Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-baat based large-scale retrieval. arXiv preprint arXiv:2002.03332. 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-
domain questions. In Proc. ACL, 2017.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, (6):391-407, 1990.
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 11, pp. 65-74, New York, NY, USA, 2017. Association for Computing Machinery. doi: 10.1145/307713633000333
URL https: /doi: wttp:o.orgl1.... 1145/ 307 136. 3080832
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019.
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. End-to-end retrieval in continuous space. arXiv preprint arXiv:/811.0000 1.08008, 2018.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.08909 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338, 2013.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Trans- former architectures and pre-tree are-training strategies for fast and accurate multi-sentence scoring. arXiv preprint 1905.0 2019.

---

Published as a conference paper at ICLR 2021
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019.
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. ACL, 2017.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906. 2020.
Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. TACL, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lec, Kristina N. Toutanova, Llion Jones, Ming-Wa Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: a benchmark for question answering research. TACL, 2019.
Kenton Lee, Ming-Wei Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proc. ACL, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeece pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461, 2019.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettle- moyer. Pre-training via paraphrasing. arXiv preprint (xXxi:0006.150200 2020a.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. eetrieval-augmentee gener- ation for knowledge-intnssive intensive nlp tasks. arXiv preprint arXiv:2005.11401, 2020b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. arXiv preprint AXV::2005000181 2020.
Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan. Introduction to information retrieval. Cambridge university press, 2008.
Bhaskar Mitra, Nick Craswell, al. An introduction neural information retrieval. Foundations and Trends in Information Retrieval, 13(1):1-12,, 2018.
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint 1XXiv/900.00000 2019.
Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 24(4):694-707, 2016.
10

---

Published as a conference paper at ICLR 2021
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. EMNLP, 2016.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock- Beaellk-eaa Mike Gatford, et al. Okapi at TREC-3. NIST Special Publication Sp, 1995.
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. Learning semantic rep- resentations using convolutional neural networks for web search. In Proceedings of the 23rd international conference on world wide web, pp. 373-374, 2014.
James Thore, Andreas Vlachos, Christos Christodouulppullo, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint 1803. 05355, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30, pp. 5998-6008. 2017.
Ellen Voorhees et al. The TREC-8 question answering track report. In TREC, 1999.
Sohee Yang and Minjoon Seo. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010.10990 10999, 2020.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with BERTserini. In Proc. NAACL (Demonstra- tions), 2019.
11

---

Published as a conference paper at ICLR 2021
Hyperparameter
Reader-base
Reader-large
Retriever
Number of parameters
220M
770M
110M
Number of heads
12
16
12
Number layers
24
48
12
Hidden size
768
1024
768
Batch size
64
64
64
Dropout
0.1
0.1
0.1
Learning rate schedule
constant
linear
constant
Peak learning rate
0.0001 1.
0.00005 1.
0.00005
Gradient clipping
1.
Table 6: Hyperparameters for retriever and reader training.
A EXPERIMENTAL DETAILS
A.1 SETTING
For NaturalQuestions and TriviaQA we follow the standard open-domain question answering setting used in Lee et al. 2019);; Karpukhin et al. (200)). In this setting the original development set used as test set, and 10% of the training set is used for development purpose. Moreover, for NaturalQuestions, all questions with answers longer than five tokens are discarded.
For TriviaQA we use the unique human--eneratte answer train the reader. In this dataset part of the answers are in uppercase. We normalize uppercase answers by converting the first letter in each word to uppercase and remaining characters lowercase using the title Python string method.
For NarrativeQA, questions and answers in uppercase are converted to lowercase.
A.2 TRAINING
For every datasets, both the reader and the retriever are ine-tuned with a dropout rate of 10%. All models at the exception of the large reader trained using the ADAM algorithm (Kingma & Ba 2014) with a constant learning rate of 10-4 for the base reader and 5 10 for the retriever. The base reader is trained for 10k gradient steps with batch size of 64. We train the large reader with the ADAMW algorithm (Loshchilov & Hutter 2019) with a peak learning rate of 5 10-5 and linear warmup for 600 gradient steps followed by linear decrease of the learning rate for 14.4k gradient steps.
We perform model selection on the validation performance. The retriever is trained until its perfor- mance saturates with a batch size of 64. To monitor the performance of the retriever during training, we measure the similarity between the ranking obtained with the reader score, and the ranking of the retriever. We use different metrics for this: the number of inversions between the two rankings, the proportion of passages the retriever top-k that are also in the reader top-k and the number of passages to obtain all top-k passage of the reader.
During training and at test time, each text input of the encoder is restricted to be most 250 token long. For NaturalQuestion and TriviaQA, we use wikipedia as knowledge source, thus for each passage there is an associated article title. Each input is composed of the concatenation of question, title and support passage with special tokens que st ion: title: and context: added before the question, the title and the text of each passage. In the case of NarrativeQA, the question and each passage concatenated to form the different inputs.
A.3 INFERENCE
At test time, for TriviaQA and NaturalQuestions we use greedy decoding, and Beam Search with beams for NarrativeQA.
12

---

Published as a conference paper at ICLR 2021
NaturalQuestions
TriviaQA P@ 100
Iter.
P@20
P@ 100
Dev EM
P@-0
Dev EM
0
77.1
84.3
46.4
78.2
84.7
65.0
80.3
86.7
47.8
81.4
86.4
67.1
2
82.4
87.9
48.2
83.5
87.4
68.1
Table 7: Iterative training starting with documents retrieved with DPR. Iteration corresponds to the performance of the reader trained on the of initial support documents. We report all metrics on the validation set. Contrary to results reported in Table the reader model was not re-initialized between each iteration.
13


Reviews:
[{"review_id": "oUZzDTUkOY-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Many Information Retrieval systems rely on two components: a retriever that identifies a small set of \"support\" documents from a large corpus, followed by a reader that re-scores these support documents more finely. For retrievers, metrics like BM25 were once common but they are increasingly replaced by machine learned components. However, most datasets do not provide direct supervision information for the retriever.\n\nAssuming a Transformer reader, this paper proposes to train the reader and retriever iteratively, using cross-attention scores from an increasingly better reader as a way to identify increasingly better sets of support documents. The novelty of this paper is the use of reader cross-attention scores as a proxy to train the retriever. The models are otherwise standard.\n\nThis paper is well motivated and clearly written. It seems to be improving the state-of-the-art on three datasets and over several recent baselines. The training procedure (including hyper-parameters) is detailed, facilitating reproducibility.\n\nQuestions and comments:\n- Since DPR plays a central role in this paper, it deserves a longer introduction, either in Section 2 or later. Further discussion on the use of already trained models as starting point would be interesting.\n- Similarly, it would be nice to summarize the negative example sampling strategy in Section 3.5.\n- On the NarrativeQA dataset, the authors mention \"it is not straightforward to train the retriever with heuristics using Q-A pairs\". Isn't BM25 (used for the initial set of documents) a heuristic? Can this be clarified?\n- I assume the gains against the baselines are statistically significant, but it would be nice to mention in writing.\n- An iterative training procedure is going to be slower than other approaches. Some discussion on this point would be interesting. Or referencing a discussion in a previous paper would also be OK.\n- Are there any other downsides to this approach?\n\nMinor comments:\n- In the third paragraph of the first section, it sounds like the authors introduce the retriever/reader architecture, which is standard (and otherwise clear from the rest of the paper).\n- Typo, page 2: well know -> well known\n- Please introduce d (embedding dimensionality) in page 3.\n- Please add a citation in the first row of Table 3.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ihKwhpYZkca", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper targets an important problem in open-domain QA - the training of the retriever for the purpose of determining a segment that may contain the answer. In the most traditional setting, the retriever is just a traditional IR system such as BM25. In some existing work, the retriever has been trained to locate the documents containing the answer (e.g. inverse cloze task, or DPR). This paper goes in the same direction. The difference is that it uses the attention weights as relevance signals to train the retriever, instead of the inclusion of the answer in the passage.\n\nOverall, the paper describes an interesting contribution to the problem. I vote for accepting it.\n\nStrengths:\nThe proposed idea is interesting and complements those used in the existing work. In the experiments, it is shown that the idea can indeed improves the existing ones: it takes the existing work DPR as the starting point, and shows that the iterative process incorporating the attention weights can improve the QA performance.\n\nThe paper provides solid experiments that support the claim.\n\nWeaknesses:\nWhile the basic idea sounds interesting, the paper does not provide a strong intuition behind it. One could imagine that strong attention could be paid to wrong passages, as the reader does not know the ground truth. In contrast, the idea used in the previous work - the inclusion of the answer in a passage is a positive signal - sounds more intuitive. It would be interesting to estimate how the attentions correlate to with the distribution of correct answer in the passages. This could provide some empirical indication that attention weights point to the right direction. The paper contains Table 1, which partly show this correlation. A stronger intuitive motivation could make the paper stronger.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PWS7IuRysr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a training technique for information retrieval models in the context of (open domain) question answering. Assuming the existence of some reader model, the idea is to use internal information of that model as a training signal for a retriever. Specifically, they use the attention activations over the input documents as synthetic labels for the retriever.\n \nThe paper is written well, and proposes an interesting idea. The technique is well motivated, I particularly like that they motivate the use of the cross-attention score through a simple experiment, where they compare the top 10 DPR passages with the top 10 passages by attention score.\nThe results over the SOTA seem moderate though, and the number of iterations seems to be an important (and potentially underexplored) variable there.\n \nI support the acceptance of the paper, because I believe the technique and the choice of model score (cross-attention score) are both interesting contributions.\n \nAs the authors say, training retrieval systems is tricky, since there is usually not sufficient labelled data available and it might depend heavily on the task. The iterative training that exploits internal state of the “downstream” model that they describe is an interesting idea that deserves attention from the community.\n \nIn Table, only up to 4 iterations in the table, which still show large improvements from one to the next. It would be interesting to know at what point no additional gains are seen.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9qCTC_TzOEh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary:\n* This paper proposes a technique to learn retriever models for question answering that does not require annotated pairs of query and documents. The proposed technique uses  attention scores of a reader model to obtain synthetic labels for the retriever.  Experimental results with NaturalQuestions, TriviaQA, and NarrativeQA show that the technique achieved state-of-the-art results.\n\nStrengths:\n* This paper is well organized and well written.\n* The proposed method is novel.\n* Experimental results are strong.\n\nComments:\n* We can see from Table 1 of this paper and Table 2 of the DPR paper that the P@20 and P@100 scores of the proposed retriever, which used iterative training starting with documents retrieved with BM25, outperformed those of the DPR model trained in a supervised manner.  This is a surprising result. I think this paper can benefit a lot with a more analysis with this point.  How much false positives and negatives are included in the reader's attention scores? Why are the readers' attention scores better than supervised  labels?\n* I also would like to see the results of the proposed system that uses supervised labels of relevant passages (e.g. a cross entropy loss) in addition to the KL divergence loss.  Moreover, I would like you to add the results of iterative training starting with documents retrieved with DPR to Table 1.\n* There is a concurrent study.  I think it would be very useful to discuss the study: \"Is Retriever Merely an Approximator of Reader?\" https://openreview.net/forum?id=dvXFpV6boX\n\nUpdate:\nThank you for the answers to my questions and additional experiments!", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper attempts to improve retrieval in open domain question answering systems, which is a very important problem. In this regards, the authors propose to utilize cross-attention scores from a seq2seq reader models as signal for training retrieval systems. This approach overcomes typical low amount of labelled data available for retriever model. The reviewers reached a consensus that the proposed approach are interesting and novel. The proposed approach establish new state-of-the-art performance on three QA datasets, although the improvements over previous methods are marginal. Overall, reviewers agree that the paper will be beneficial to the community and thus I recommend an acceptance to ICLR.

Decision: Accept (Poster)