Venue: NAACL 2021

# ReConsider: Improved Re-Ranking using Span-Focused

Cross-Attention for Open Domain Question Answering

Srinivasan Iyer\({}^{\dagger}\) Sewon Min\({}^{\ddagger}\) Yashar Mehdad\({}^{\dagger}\) Wen-tau Yih\({}^{\dagger}\)

\({}^{\dagger}\)Facebook AI \({}^{\ddagger}\)University of Washington

{sviyer,mehdad,scottyih}@fb.com sewon@cs.washington.edu

Work done while at Facebook AI.github.com/facebookresearch/reconsider1280

###### Abstract

State-of-the-art Machine Reading Comprehension (MRC) models for Open-domain Question Answering (QA) are typically trained for span selection using distantly supervised positive examples and heuristically retrieved negative examples. This training scheme possibly explains empirical observations that these models achieve a high recall amongst their top few predictions, but a low overall accuracy, motivating the need for answer re-ranking. We develop a successful re-ranking approach (ReConsider) for span-extraction tasks that improves upon the performance of MRC models, even beyond large-scale pre-training. Reconsider is trained on positive and negative examples extracted from high confidence MRC model predictions, and uses in-passage span annotations to perform span-focused re-ranking over a smaller candidate set. As a result, ReConsider learns to eliminate close false positives, achieving a new extractive state of the art on four QA tasks, with 45.5% Exact Match accuracy on Natural Questions with real user questions, and 61.7% on TriviaQA. We will release all related data, models, and code1.

Footnote 1: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1280–1287 June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Open-domain Question Answering [3] (QA) involves answering questions by extracting correct answer spans from a large corpus of passages, and is typically accomplished by a light-weight passage retrieval model followed by a heavier Machine Reading Comprehension (MRC) model [9]. The span selection components of MRC models are trained on distantly supervised positive examples (containing the answer string) together with heuristically chosen negative examples, typically from upstream retrieval models. This training scheme possibly explains empirical findings [22, 10] that while MRC models can confidently identify top-K answer candidates (high recall), they cannot effectively discriminate between top semantically similar false positive candidates (low accuracy). In this paper, we develop a general approach to make answer re-ranking successful for span-extraction tasks, even over large pretrained models, and improve the state of the art on four QA datasets.

Earlier work [22, 10] on open-domain QA have recognized the potential of answer re-ranking, which we continue to observe despite recent advances using large pre-trained models like BERT [10]. Figure 1 shows the top-3 predictions of a BERT-based SOTA model [11] on a question from Natural Questions (NQ) [12], "_Who was the head of the Soviet Union when it collapsed?_" While all predictions are very relevant and refer to Soviet Union heads, _Mikhail Gorbachev_ is correct and the rest are close false positives. Table 1 presents accuracies obtained by the same model on four QA datasets, if the answer exactly matches

Figure 1: Top-3 passage-spans predicted by a BERT-MRC model on a question from NQ (answer spans are underlined). ReConsider re-evaluates the passages with marked candidate answers, eliminates close false positives and ranks _Mikhail Gorbachev_ as correct.



any of the top-k predictions for \(k=1,5,10\) and \(25\). We observe that an additional \(10\%\) and \(~{}20\%\) of correct answers exist amongst the top-5 and top-25 candidates respectively, presenting an enormous opportunity for span reranking models.

Our re-ranking model is trained using positive and negative examples extracted from high confidence MRC model predictions, and thus, learns to eliminate hard false positives. This can be viewed as a coarse-to-fine approach of training span selectors, with the base MRC model trained on heuristically chosen negatives and the re-ranker trained on finer, more subtle negatives. This contrasts with multi-task training approaches Wang et al. (2018c), whose re-scoring gains are limited by training on the same data, especially when coupled with large pre-trained models. Our approach also scales to any number of ranked candidates, unlike previous concatenation based cross-passage re-ranking methods Wang et al. (2018b) that do not transfer well to current length-bounded large pre-trained models. Similar to MRC models, our re-ranking approach uses cross-attention between the question and a candidate passage Seo et al. (2016). However, we now demarcate a specific candidate answer span in each passage, to assist the model to perform span-focused reasoning, in contrast to MRC models, which must reason across all spans. Therefore, the re-ranker performs span ranking of carefully chosen candidates, rather than span selection like the MRC model. Similar focused cross-attention methods have recently proved to be effective for Entity Linking Wu et al. (2020) tasks, although they annotate the query rather than the passage.

We use our broadly applicable span-focused re-ranking approach on models from Karpukhin et al. (2020) and achieve a new extractive state of the art on four QA datasets, including 45.5% on the open-domain setting of NQ (real user queries, +1.6% on small models) and 61.1% on TriviaQA Joshi et al. (2017) (+2.5% on small models). To our knowledge, we are the first to successfully leverage re-ranking to improve over large pre-trained models on open-domain QA.

## 2 Background

Open-domain Question Answering (QA) aims to answer factoid questions from a large corpus of passages Voorhees et al. (1999) (such as Wikipedia) in contrast with single passage MRC tasks Rajpurkar et al. (2016). Prior works use pipelined approaches, that first retrieve candidate passages and subsequently use a neural MRC model to extract answer spans Chen et al. (2017), with further improvements using joint learning Wang et al. (2018a); Tan et al. (2018). Recent successes involve improving retrieval, thereby increasing the coverage of passages fed into the MRC model Guu et al. (2020); Karpukhin et al. (2020). In this paper, we significantly improve MRC model performance by making re-ranking successful using span-focused re-ranking of its highly confident predictions.

For Open-domain QA, it is crucial to train MRC models to distinguish passage-span pairs containing the answer (_positives_) from those that do not (_negatives_). Using negatives that appear as close false positives can produce more robust MRC models. However, prior work relies on upstream retrieval models to supply distantly supervised positives (contain answer string) and negatives Asai et al. (2020), that are in-turn trained using heuristically chosen positives and negatives. Our approach leverages positives and negatives from highly confident MRC predictions which are hard to classify, and thus, improve upon MRC model performance.

Jia and Liang Jia and Liang (2017) motivate recent work on answer verification for QA by showing that MRC models are easily confused by similar passages. Wang et al. (2018b) use a weighted combination of three re-rankers and rescore a concatenation of all passages with a particular answer using a sequential model, while, Wang et al. (2018c) develop a multi-task end-to-end answer scoring approach. Although the main idea is to consider multiple passage-span candidates collectively, such approaches either used concatenation, which is prohibitively expensive to couple with length-restricted models like BERT, or are trained on the same data without variations only to realize marginal gains. Hu et al.Hu et al. (2019) use answer verification to predict the unanswerability of a question

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Dataset & Top-1 & Top-5 & Top-10 & Top-25 \\ \hline NQ & 40.3 & 49.5 & 50.9 & 62.4 \\ TriviaQA & 57.2 & 64.6 & 65.7 & 73.1 \\ WebQ & 42.6 & 49.0 & 50.7 & 60.4 \\ TREC & 49.6 & 58.7 & 60.9 & 71.4 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Top-k EM accuracies using a state-of-the-art model Karpukhin et al. (2020) on four open-domain QA tasks (dev set). Improvements of up to 22% can potentially be achieved by re-ranking top-25 candidates.

 

passage pair for traditional MRC tasks. To our knowledge, our work is the first to (i) successfully demonstrate a re-ranking approach that significantly improves over large pre-trained models Devlin et al. (2019) in an open domain setting, and (ii) use annotated top model predictions as harder negatives to train more robust models for QA.

## 3 Model

We assume an extractive MRC model \({\mathcal{M}}\) coupled with a passage retrieval model, that given a question \(q\) and a passage corpus \({\mathcal{P}}\), produces a list of \(N\) passage and span pairs, \(\{(p_{j},s_{j})\}_{j=1}^{N},p_{j}\in{\mathcal{P}}\) and \(s_{j}\) is a span within \(p_{j}\), ranked by the likelihood of \(s_{j}\) answering \(q\). Note that \(\{p_{j}\}_{j=1}^{N}\) is not disjoint as a passage can have multiple answer spans. In this section, we develop a span-focused re-ranking model \({\mathcal{R}}\), that learns a distribution \(p\), over top-\(K~{}(p_{j},s_{j})\) pairs \(1\leq j\leq K\), given question \(q\). Essentially, model \({\mathcal{R}}\) first scores every \((q,p_{j},s_{j})\) triple using scoring function \(r\), and then normalizes over these scores to produce \(p\):

\[p(q,p_{j},s_{j})=\frac{e^{r(q,p_{j},s_{j})}}{\sum_{1\leq k\leq K}e^{r(q,p_{k} ,s_{k})}}. \tag{1}\]

Specifically, if \(E(q,p_{j},s_{j})\in{\mathbb{R}}^{H}\) is a dense representation of \((q,p_{j},s_{j})\), \(r\) is defined as:

\[r(q,p_{j},s_{j})={\mathbf{w}}^{T}E(q,p_{j},s_{j}), \tag{2}\]

where \({\mathbf{w}}\in{\mathbb{R}}^{H}\) is a learnable vector.

Span-focused tuple encodingWe compute \(E\) using the representation of the [CLS] token of a BERT model Devlin et al. (2019) applied to a span-focused encoding of \((q,p_{j},s_{j})\). This encoding is generated by first marking the tokens of \(s_{j}\) within passage \(p_{j}\) with special start and end symbols [A] and [/A], to form \(\hat{p}_{j}\), followed by concatenating the [CLS] and question tokens, with the annotated passage tokens \(\hat{p}_{j}\), using separator token [SEP]. We find span marking to be a crucial ingredient for answer re-ranking, without which, performance deteriorates (Section 5).

TrainingWe obtain top \(K\) predictions \((p_{j},s_{j})\) of model \({\mathcal{M}}\) for each question \(q_{i}\) in its training set, which we divide into positives, where \(s_{j}\) is exactly the groundtruth answer, and remaining negatives. We train \({\mathcal{R}}\) using mini-batch gradient descent, where in each iteration, for question \(q\), we include 1 randomly chosen positive and \(M-1\) randomly chosen negatives, and maximize the likelihood of the positive. Unlike the heuristically chosen negatives used to train \({\mathcal{M}}\), \({\mathcal{R}}\) is trained using negatives from high confidence predictions of \({\mathcal{M}}\), which are harder to classify. Thus, this can be viewed as an effective coarse-to-fine negative selection strategy for span extraction models (Section 5).

## 4 Baseline Model \({\mathcal{M}}\)

We use the state-of-the-art models of Karpukhin et al. (2020) which consists of 1) a dense passage retriever, and 2) a span extractive BERT reader, as our model \({\mathcal{M}}\). The retriever uses a passage encoder \(f_{p}\) and a question encoder \(f_{q}\) to represent all passages and questions as dense vectors in the same space. During inference, it retrieves top-100 passages similar to question \(q\) based on their inner product, and passes them on to the MRC reader.

The MRC reader is an extension of model \({\mathcal{R}}\) of Section 3, to perform span extraction. We briefly describe it but Karpukhin et al. (2020) has complete details. Its input is a question \(q\) together with positive and negative passages \(p_{j}\) from its retrieval model. \((q,p_{j})\) tuples are encoded as before (\(enc(q,p_{j})=q\) [SEP]\(p_{j}\)), but without spans being marked (as spans are unavailable). A distribution over passages \(p_{s}\) is computed as before using scoring function \(r\) and context encoder \(E\). In addition, a start-span probability, \(p_{st}(t_{i}|q,p_{j})\) and an end-span probability, \(p_{e}(t_{i}|q,p_{j})\) is computed for every token \(t_{i}\) in \(enc(q,p_{j})\). The model is trained to maximize the likelihood of \(p_{s}(p_{j})\times p_{st}(s|q,p_{j})\times p_{e}(t|q,p_{j})\) for each correct answer span \((s,t)\) in \(p_{j}\), and outputs the top-K scoring passage-span pairs during inference.

## 5 Experiments

DatasetsWe use four benchmark open-domain QA datasets following Lee et al. (2019):

**Natural Questions (NO)** contains real user questions asked on Google searches; we consider questions with short answers up to 5 tokens.

**TriviaQA** Joshi et al. (2017) consists of questions collected from trivia and quiz-league websites; we take questions in an unfiltered setting and discard the provided web snippets.

**WebQuestions (WebQ)**Berant et al. (2013) is a collection of questions extracted from the Google Suggest API, with answers being Freebase entities.

**CuratedTREC**Baudis and Sedivy (2015) contains curated questions from TREC QA track 

Implementation detailsFor all datasets, we use the retrieval model (without retraining) and setup from Karpukhin et al. (2020), retrieving 100-token passages from a Wikipedia corpus (from \(2018\)-\(12\)-\(20\)). We also use their MRC model with their best performing hyperparameters as model \({\cal M}\). For model \({\cal R}\), we experiment with both BERTbase and BERTlarge, use top-100 predictions from model \({\cal M}\) during training (top-5 for testing), and use \(M=30\). We use a batch size of \(16\) on NQ and TriviaQA and \(4\) otherwise. For WebQ and TREC, we start training from our trained NQ model.

ResultsTable 2 presents end-to-end test-set exact match accuracies for these datasets, compared with previous models. The BERTbase version of ReConsider outperforms the previous state-of-the-art DPR model of Karpukhin et al. (2020) (our model \({\cal M}\)) by \(1.6\%\) on NQ and \(\sim 2\%\) on TriviaQA and WebQ. For training on the smaller WebQ and TREC datasets, we initialize models using the corresponding NQ model. Table 2 demonstrates the effectiveness of a coarse-to-fine approach for selecting negative passages, with dense retrieval based negatives (DPR) outperforming BM25, and in turn, improved upon by our reranking approach. We obtain gains despite \({\cal R}\) being not only very similar in architecture to the MRC reader \({\cal M}\), but also trained on the same QA pairs, owing to (i) training using harder false-positive style negatives, and (ii) answer-span annotations that allow a re-allocation of modeling capacity from modeling all spans to reasoning about specific spans with respect to the question and the passage. Re-ranking performance suffers without these crucial methods. For example, replacing answer-span annotations with answer concatenation reduces accuracy by \(\sim\)1% on the dev set of NQ.

We train a large variant of ReConsiderusing BERTlarge for model \({\cal R}\), trained on predictions from a BERTlarge model \({\cal M}\). For a fair comparison, we re-evaluate DPR using BERTlarge.ReConSinderHor performs it by \(\sim\)1% on all datasets (+ \(\sim\)2% on TREC). This model is also comparable in size to RAG Lewis et al. (2020) (which uses BARTlarge) but outperforms it on all tasks (+1 on NQ, +5.5 on TriviaQA, +3 on TREC), demonstrating that retrieve-extract architectures can perform better than answer generation models.

We find \(K\)=5 (testing) to be best for all datasets, and increasing \(K\) has little effect on accuracy, despite training on top-100 predictions. Although in contrast with our expectations based on Table 1, this is anticipated since very low-ranked predictions are less likely to be remarked highly, but this also presents an opportunity for future work.

In Table 3, we present examples from the validation set of NQ, of cases where 1) DPR-BERTbase

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Model & NQ & TriviaQA & WebQ & TREC \\ \hline BM25+BERT Lee et al. (2019) & 26.5 & 47.1 & 17.7 & 21.3 \\ ORQA Lee et al. (2019) & 33.3 & 45.0 & 36.4 & 30.1 \\ HardEM Min et al. (2019a) & 28.1 & 50.9 & - & - \\ GraphRetriever Min et al. (2019b) & 34.5 & 56.0 & 36.4 & - \\ PathRetriever Asai et al. (2020) & 32.6 & - & - & - \\ REALM Guu et al. (2020) & 39.2 & - & 40.2 & 46.8 \\ REALMNews Guu et al. (2020) & 40.4 & - & 40.7 & 42.9 \\ \hline
**Models that use DPRmulti**Karpukhin et al. (2020) & & & & \\ \hline DPR-BERTbase Karpukhin et al. (2020) & 41.5 & 56.8 & 42.4 & 49.4 \\ ReConsiderbase (Ours) & **43.1** & **59.3** & **44.4** & **49.3** \\ \hline DPR-BERTlarge Karpukhin et al. (2020) & 44.6 & 60.9 & 44.8 & 53.5 \\ RAGlarge Lewis et al. (2020) & 44.5 & 56.1 & 45.5 & 52.2 \\ ReConsiderlarge (Ours) & **45.5** & **61.7** & **45.9** & **55.3** \\ \hline \hline \end{tabular}
\end{table}
Table 2: End-to-end QA test-set (Exact Match) accuracy. Models in the lower half use dense passage retrieval from Karpukhin et al. (2020). ReConsider outperforms previous methods under both base and large versions. Table layout and baselines are borrowed from Karpukhin et al. (2020) (published extractive SOTA). Dataset stats and dev set results are in Appendix, and baseline descriptions can be found in Karpukhin et al. (2020). \({}^{\dagger}\): numbers from our own experiments.

 

produces an incorrect top answer, which is corrected after re-ranking with ReConsider (top 2 examples), and 2) DPR-BERTbase's answer is correct but is ranked lower after re-ranking. Of the 15.4% validation examples that were amenable for correction by re-ranking the top-5 candidates from DPR-BERTbase, ReConsider was able to fix 6.1%. However, in this process, 4.3% of answers that were originally correct (top-ranked), lost their top-rank after ReConsider, and this presents an opportunity for further improving re-ranking.

## 6 Conclusion

We use a synergistic combination of two techniques viz. retraining with harder negatives, and, spann focused cross attention, to make re-ranking successful for span-extractive tasks over large pretrained models. This method achieves SOTA extractive results on four open domain QA datasets, also outperforming recent generative pre-training approaches.

## References

* Asai et al. [2020] Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over Wikipedia graph for question answering. In _International Conference on Learning Representations (ICLR)_.
* Baudis and Sedivy [2015] Petr Baudis and Jan Sedivy. 2015. Modeling of the question answering task in the yodaqa system. In _International Conference of the Cross-Language Evaluation Forum for European Languages_.
* Berant et al. [2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Chen et al. [2017] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer opendomain questions. In _Association for Computational Linguistics (ACL)_.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _North American Association for Computational Linguistics (NAACL)_.
* Guu et al. [2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. In _International Conference on Machine Learning (ICML)_.
* Hu et al. [2019] Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, and Dongsheng Li. 2019. Read+ verify: Machine reading comprehension with unanswerable questions. In _Conference on Artificial Intelligence (AAAI)_.
* Jia and Liang [2017] Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In _Association for Computational Linguistics (ACL)_.
* Karpukhin et al. [2018] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen,

\begin{table}
\begin{tabular}{l l} \hline \hline Model & Prediction \\ \hline Question & Where are zebra mussels found in the United States? \\ DPR-BERbase & ... on the genetic algorithm for rule - set production (garp), a group of researchers predicted that the **southeastern united states** is moderately to highly likely to be inhabited by zebra mussels ... \\ +ReConsider & ... of zebra mussel **in the great lakes** alone exceeds §500 million a year. \\ \hline Question & Where do you find neurons in the brain? \\ DPR-BERbase & ... there is strong evidence for generation of substantial numbers of new neurons in two brain areas, the **hippocampus** and olfactory bulb. A neuron is a specialized type of cell found in the bodies ... \\ +ReConsider & The brain is in the most complex organ in a vertebrate’s body. In a human, **the cerebral cortex** contains approximately 14–16 billion neurons. \\ \hline Question & Who said if I have seen further it is by standing on the shoulders of giants? \\ DPR-BERbase & Standing on the shoulders of giants ... this concept has been traced to the 12th century, attributed to Bernard of Chartres. Its most familiar expression in English is by **Isaac Newton** in 1675: “If I have seen further it is by standing on the shoulders of giants ... \\ +ReConsider & Standing on the shoulders of giants ... this concept has been traced to the 12th century, attributed to **Bernard of Chartres**. Its most familiar expression in English is by Isaac Newton in 1675: “If I have seen further it is by standing on the shoulders of giants ... \\ \hline \hline \end{tabular}
\end{table}
Table 3: Top passage with answer span (in bold) for example questions from the validation set of NQ, both with and without re-ranking using ReConsider. For the first two examples, ReConsider re-ranks to obtain the correct answer, while in the last example, re-ranking eliminates the already correct top answer.

 

and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. _Transactions of the Association of Computational Linguistics (TACL)_.
* Lee et al. [2019] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In _Association for Computational Linguistics (ACL)_.
* Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Pitkins, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In _Advances in Neural Information Processing Systems (NeurIPS)_.
* Min et al. [2019a] Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. A discrete hard EM approach for weakly supervised question answering. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Min et al. [2019b] Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Knowledge guided text retrieval and reading for open domain question answering. _arXiv preprint arXiv:1911.03868_.
* Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Dopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Seo et al. [2016] Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. In _International Conference on Learning Representations (ICLR)_.
* Tan et al. [2018] Chuanqi Tan, Furu Wei, Nan Yang, Bowen Du, Weifeng Lv, and Ming Zhou. 2018. S-net: From answer extraction to answer generation for machine reading comprehension. In _Conference on Artificial Intelligence (AAAI)_.
* Voorhees et al. [1999] Ellen M Voorhees et al. 1999. The TREC-8 question answering track report. In _TREC_.
* Wang et al. [2018a] Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018a. R^3: Reinforced ranker-reader for open-domain question answering. In _Conference on Artificial Intelligence (AAAI)_.
* Wang et al. [2018b] Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In _International Conference on Learning Representations (ICLR)_.
* Wang et al. [2018c] Yizhong Wang, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. 2018c. Multi-passage machine reading comprehension with cross-passage answer verification. In _Association for Computational Linguistics (ACL)_.
* Wu et al. [2020] Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In _Empirical Methods in Natural Language Processing (EMNLP)_.

 

Computing Infrastructure Used

All experiments were run on a machine with 2 chips of Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz with 20 cores (40 threads) each, equipped with \(8\) NVIDIA TESLA V100 GPUs, each with \(32\) GB of memory.

## Appendix B Average Run-time and #Parameters

We report average run-times for training and inference on NQ (TriviaQA is similar), as well as number of model parameters, in Table 4. WebQ and TREC are much smaller datasets and have lower runtimes.

## Appendix C Validation Performance

Table 8 presents validation set performance for the experiments that we ran for this paper.

## Appendix D Hyperparameters

For training ReConsider, we use top-\(100\) predictions of the baseline MRC model. This was chosen based on validation set accuracy, and other values that were experimented with were \(50\) and \(75\). For training ReConsider we use 1 positive and \(M-1\) negatives during each iteration. We tried values of \(M\) between \(5\) and \(40\) in increments of \(5\) and chose \(M=30\) based on validation set accuracy (see Table 6). Similarly, we re-rank \(K=5\) candidates during inference, and this value was chosen by experimenting with values \(2,3,4\) and values between \(5\) and \(20\), in increments of \(5\) (see Table 5).

## Appendix E Dataset Statistics

Table 7 presents the number of examples in the training, validation and testing splits of the four open-domain QA datasets that we use, based on the dataset prepared by Karpukhin et al. (2020).

\begin{table}
\begin{tabular}{l r} \hline \hline
**Train-M** & **Accuracy (\%)** \\ \hline
25 & 42.34 \\
30 & 42.48 \\
40 & 41.25 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyper-parameter tuning for the number of negative passages i.e. train–\(M\), on the validation set of NQ.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **Params** & **Train** & **Inf.** \\ \hline DPR-BERTlarge & 335M & 37 h & 2.8 h \\ ReConsidERbase & 109M & 13 h & 2 m \\ ReConsiderLarge & 335M & 28 h & 2 m \\ \hline \hline \end{tabular}
\end{table}
Table 4: Runtime for training and inference, and number of parameters of the models that we executed, on NQ. Runtimes for ReConsider do not include the time required to train and obtain predictions from DPR.

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Dataset** & **Train** & **Dev** & **Test** \\ \hline NQ & 67,098 & 8,757 & 3,610 \\ TriviaQA & 67,975 & 8,837 & 11,313 \\ WebQ & 2,898 & 361 & 2,032 \\ TREC & 1,240 & 133 & 694 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Training, validation and testing set sizes for the four open-domain QA tasks evaluated in our paper.

\begin{table}
\begin{tabular}{l r} \hline \hline
**K** & **Accuracy (\%)** \\ \hline
3 & 41.54 \\
5 & 42.50 \\
10 & 42.26 \\
15 & 42.03 \\
20 & 41.91 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Hyper-parameter tuning for \(K\) at inference time on the validation set of NQ.

 



\begin{table}
\begin{tabular}{l c r r r} \hline
**Model** & **NQ** & **TriviaQA** & **WebQ** & **TREC** \\ \hline ReConsiderbase (Ours) & 42.5 & 60 & 46.3 & 49.6 \\ \hline DPR-BERTlarge [17] & 42.2 & 60.1 & 43.8 & 54.9 \\ ReConsiderlarge (Ours) & 44.2 & 61.7 & 46 & 54.9 \\ \hline \end{tabular}
\end{table}
Table 8: Validation set performance for our experiments.