Venue: NAACL 2021

# Designing a Minimal Retrieve-and-Read System

for Open-Domain Question Answering

Sohee Yang1 and Minjoon Seo1,2

1KAIST AI \({}^{~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} 

quantization (Zafrir et al., 2019), and knowledge distillation (Hinton et al., 2014). In this paper, we utilize some of these generic approaches and combine them with problem-specific techniques to size down a conventional retrieve & read system. We first train a passage filter and use it to reduce the corpus size (Section 2.1). We further apply parameter sharing strategies and knowledge distillation to make a single-encoder lightweight model that can perform both retrieval and reading (Section 2.2). In addition, we adopt multiple engineering tricks to make the whole system even smaller (Section 2.3).

We verify the effectiveness of our methods on the dev set and test set of EfficientQA2Min et al. (2021). By applying our strategies to a recent extractive retrieve & read system, DPR Karpukhin et al. (2020), we reduce its size by 160x with little loss of accuracy, which is still higher than the performance of a purely parametric T5 Roberts et al. (2020) baseline with a comparable docker-level storage footprint. In Appendix A.5, we also report the performance on two more open-domain QA datasets, Natural Questions Kwiatkowski et al. (2019) and TriviaQA Joshi et al. (2017), to test the generalizability of our methods and suggest a future research direction.

Footnote 2: A recently-hosted open-domain QA challenge at NeurIPS 2020 in which 18 teams participated. [https://efficientqa.github.io](https://efficientqa.github.io)

## 2 Method

In this section, we discuss three techniques for reducing the storage footprint of a generic retrieve & read system, namely passage filtering (Section 2.1), unifying retriever and reader into a single model through parameter sharing (Section 2.2), and posttraining compression (Section 2.3). We assume that the initial system takes the conventional composition of a trainable (neural) retriever with a question encoder and a passage encoder that create dense vectors used for search, a neural extractive reader (possibly with passage ranking), and a text corpus and the corresponding index that serve as the knowledge source. Figure 1 shows how we start from one such retrieve & read system and apply each of the methods, in the order they are introduced in this section, to successively reduce its system footprint without sacrificing much accuracy.

### Passage Filtering

Index and corpus files can take up a significant portion of the storage footprint of a retrieve & read system if a large text corpus is utilized as the knowledge source. Therefore, to drastically reduce the system size, we train a binary classifier and use it to exclude passages that are relatively unlikely to be useful for question answering.

Let the set of indices of all passages in the corpus be \(J_{\text{total}}\). To create the training data, we split \(J_{\text{total}}\) into two disjoint sets, \(J^{+}_{\text{train}}\) and \(J^{-}_{\text{train}}\), such that the former contains the indices of the passages we would like to include in the minimal retrieve & read system.3 Denoting \(E(\cdot)\) as a trainable dense encoder which maps a passage to a \(d\)-dimensional embedding, such that \(v_{j}=E(p_{j})\in\mathbb{R}^{d}\), the score \(s_{j}=v_{j}^{\top}w\), where \(w\in\mathbb{R}^{d}\) as a learnable vector, represents how likely a passage \(p_{j}\) would hold the answer to an input question. The classifier is trained with binary cross entropy on the minibatches of half-positive and half-negative passages, \(J^{+^{\prime}}_{\text{train}}\) and \(J^{-^{\prime}}_{\text{train}}\) drawn from \(J^{+}_{\text{train}}\) and \(J^{-}_{\text{train}}\), respectively.

Footnote 3: All the details, including how we split the data, are in Appendix A.1.

During training, we sample several checkpoints and evaluate them using the hit ratio on a validation set: \(\operatorname*{hit_{val}}=|J^{[1:]J^{+}_{\text{val}}||}_{\alpha}\cap J^{+}_{ \text{val}}|/|J^{+}_{\text{val}}|\), where \(J^{+}_{\text{val}}\) is a set of indices of the ground truth passages that hold the answer for the questions in the validation set and \(J^{[1:]J^{+}_{\text{val}}||}_{\alpha}\) is the set of indices \(j\) of the passages whose inferred score \(s_{j}\) is in the top-\(|J^{+}_{\text{val}}|\) scores sorted in descending order, among all \(s_{j}\) such that \(j\in J^{+}_{\text{val}}\cup J^{-}_{\text{val}}\). \(J^{-}_{\text{val}}\) is a disjoint set randomly sampled from \(J^{-}_{\text{train}}\).

We select the checkpoint with the highest \(\operatorname*{hit_{val}}\) and calculate \(s_{j}\) for all \(p_{j}\), where \(j\in J_{\text{total}}\), using the selected checkpoint. Then, we retrieve \(J_{\text{subset}}=J^{[1:n]}_{\text{total}}\), the set of indices of the \(n\) top-scoring passages, to indicate the passages to include in our minimal retrieve & read system.

### Retriever-Reader with Single Encoder

In this subsection, we introduce how to obtain a unified retriever-reader with a single encoder (which results in a smaller system footprint) that can perform both retrieval and reading without much drop in accuracy. The unified retriever-reader is trained by successively applying (1) retriever encoder sharing, (2) distilling a reader into the retriever-reader network, and (3) iterative finetuning.

#### 2.2.1 Lightweight Encoder and Embedding Dimension Reduction

To make the system small, we utilize a lightweight pretrained encoder. Specifi 

cally, MobileBERT Sun et al. (2020) (4.3x smaller than BERT-base Devlin et al. (2019)) is employed as the encoder of our retriever-reader model.

We use the dense embedding vectors of the passages in the knowledge source as the index. Therefore, reducing the embedding dimension results in a linear decrease in the index size. We use only the first 128 dimensions (out of 512) to encode the questions and passages.

#### 2.2.2 Retriever Encoder Sharing

Let \(E_{\psi}(\cdot)\) and \(E_{\phi}(\cdot)\) be the the question encoder and passage encoder of a retriever, where each of the encoders produces a vector for question \(q\) and passage \(p\).

We share the parameters of the encoders, so that \(\psi=\phi=\theta\), and differentiate the question inputs from passages inputs using an additional input signal: different token type ids of 0 for questions and 1 for passages. The retrieval score for a pair of question \(q\) and passage \(p\) is calculated as \(\text{sim}_{\theta}(q,p)=E_{\theta}(q,0)\cdot E_{\theta}(p,1)\).

We minimize the negative log-likelihood of selecting the passage which holds the answer, namely the positive passage, while training on mini-batches that consist of questions that are each paired with one positive passage and several negative passages. This procedure creates a retriever with a single encoder of parameters \(\theta\) that can encode both questions and passages.4

Footnote 4: In a setting where the index is frozen (addition or editing of index items does not occur), the system does not need a passage encoder. However, we assume a self-contained system with the full ability to update the index, so the passage encoder is considered in the system composition.

#### 2.2.3 Unified Retriever-Reader Through Knowledge Distillation

The previous subsection describes how to make a retriever that holds only one encoder. Here, we further train the parameters of the retriever \(\theta\) so that it can also acquire the ability of a reader; we make a unified retriever-reader model that shares all the encoder parameters and eliminate the need for a separate reader. Specifically, using a fully trained reader of parameters \(\xi\) as the teacher, we adopt knowledge distillation to transfer its reading ability to the unified retriever-reader network. The training starts after initializing the parameters of the retriever-reader as \(\theta\), which is obtained from the retriever encoder sharing procedure described in the previous subsection.

Let \(J_{\text{read}}\subset J_{\text{subset}}\) be the set of indices of the passages whose retrieval score \(\text{sim}_{\omega}(q,p_{j})\), calculated for question \(q\) using a retriever with parameters \(\omega^{5}\), is among the top-\(k^{\prime}\) scores for all \(j\in J_{\text{subset}}\). \(J_{\text{read}}\) serves as the candidate pool of the indices of the training set passages.

During training, for question \(q\), a set of passages \(P_{q}=\{p_{i}|1\leq i\leq m\}\) where \(m\geq 2\) is sampled from \(\{p_{j}|j\in J_{\text{read}}\}\) to construct a part of the training batch, such that only \(p_{1}\) contains the answer to question \(q\) among \(p_{i}\in P_{q}\).

Then, we train the unified retriever-reader network with parameters \(\theta\) using a multitask loss \(\mathcal{L}_{\text{read}}+\mathcal{L}_{\text{ret}}\), such that the former is used to train the reader part of the network, and the latter is used to keep training the retriever part. The resulting retriever-reader model has the ability to perform both retrieval and reading.

\(\mathcal{L}_{\text{read}}\) is designed to distill the knowledge of a reader teacher into the reader part of the retriever-reader student; the KL divergence between the sharpened and softmaxed answer span scores of the teacher and the student, \(D_{KL}(\mathcal{P}_{\xi,q}^{\text{span}}||\mathcal{P}_{\theta,q}^{\text{span}})\). If the teacher reader additionally contains a passage ranker, distillation is also jointly done on the passage ranking scores (\(m\)-dim vector outputs).

Retrieval loss \(\mathcal{L}_{\text{ret}}\) is jointly optimized in a multitask-learning manner to prevent the retriever part of the unified network from forgetting the retrieval ability while training the reader part. The loss can either be the negative log-likelihood described in the previous subsection or another knowledge distillation objective function with a fully trained retriever teacher. If the reader teacher used for \(\mathcal{L}_{\text{read}}\) has a passage ranker, the passage ranking score of the teacher can serve as the distillation target Yang and Seo (2020).

#### 2.2.4 Iterative Finetuning of Unified Retrieval-Reader

We have observed that finentuing the unified retriever-reader for a few more epochs leads to better retrieval and reading performance. While the most simple method is to jointly train the model on the standard reader loss and retriever loss6, we additionally try iterative finentuing of each of the retriever and reader part as described in Algorithm 1. The motivation here is to apply a loose reconstruction constraint \(\mathcal{L}_{\text{recon}}\) to keep the retrieval score as it is before and after the model is optimized for reading, with an assumption that this would be helpful 

to alleviate the train-inference discrepancy in the input distribution of the reader, created because the unified retriever-reader is not trained using a pipelined manner (training the reader on top of the retrieval result of a fixed retriever).

Input\(\theta^{(t)}\) (parameters of the model at time \(t\)), knowledge distillation temperature \(\tau\), and training batch of question \(q\) and passages \(P_{q}=\{p_{i}|1\leq i\leq m\}\) drawn from \(J_{\text{read}}\) such that \(m\geqslant 2\), \(Y(q,p_{1})=1\), and \(Y(q,p_{i})=0,\forall 2\leqslant i\leq m\). (batch size of \(1\) is assumed here for a simple presentation) Output Updated parameters \(\theta^{(t+1)}\)

```
1:\(\ell^{(t)}\leftarrow[E_{\theta}^{(t)}(p_{1},1),\cdots,E_{\theta}^{(t)}(p_{m},1 )]^{\top}E_{\theta}^{(t)}(q,0)\)
2:\(\hat{\theta}^{(t)}\leftarrow\text{GradientUpdate}(\mathcal{L}_{\text{reso}}(q, p_{1},\cdots,p_{m});\theta^{(t)})\)
3:\(\hat{\ell}^{(t)}\leftarrow[E_{\theta}^{(t)}(p_{1},1),\cdots,E_{\theta}^{(t)}(p _{m},1)]^{\top}E_{\theta}^{(t)}(q,0)\)
4:\(\mathcal{L}_{\text{reco}}\gets D_{\mathrm{KL}}\left(\text{softmax}(\ell^{( t)}/\tau)||\text{softmax}(\hat{\ell}^{(t)}/\tau)\right)\)
5:\(\mathcal{L}_{\text{all}}\leftarrow\text{CrossEntropy}(\text{softmax}(\hat{ \ell}^{(t)}),Y)\)
6:\(\theta^{(t+1)}\leftarrow\text{GradientUpdate}(\mathcal{L}_{\text{reco}}+ \mathcal{L}_{\text{all}};\hat{\theta}^{(t)})\)
```

**Algorithm 1** A single iterative finetuning step on the unified retriever-reader with parameters \(\theta\) at time \(t\)

### Post-Training Compression Techniques

In addition to the training methods to decrease the corpus, index, and model size, several post-training engineering tricks are applied to compress the system footprint further: (1) INT8 quantization of index items, (2) saving model weights as FP16, (3) resource compression, and (4) utilizing token IDs as the corpus instead of raw texts.

INT8 Quantization of Index ItemsThe dense embeddings that serve as the items in the search index are of type FP32 in the default state. INT8 quantization can be applied to reduce the index size by four times with a little bit of drop in the accuracy. We make use of the quantization algorithm implemented in FAISS Johnson et al. (2019) IndexScalarQuantizer7. During inference, the embeddings are de-quantized, and the search is performed on the restored FP32 vectors.

Footnote 7: [https://github.com/facebookresearch/faiss/blob/v1.5.2/IndexScalarQuantizer.cpp](https://github.com/facebookresearch/faiss/blob/v1.5.2/IndexScalarQuantizer.cpp)

Saving Model Weights as FP16Half precision can be used to size down the model weights of originally FP32 tensors with almost no drop in accuracy. In PyTorch, this can be done by calling .half() on each FP32 tensor in the model checkpoint.

In TensorFlow, model graphs saved as the data type of FP16 may result in unacceptably slow inference according to the used hardware. We have found out that keeping the tensor types of the graph as FP32 but making the actual assigned values as FP16 enables a higher compression ratio when the model weights are compressed as described below.

Resource CompressionData compressors with a high compression ratio are effective at reducing the initial system footprint. Our observation is that bzip2 is better for binary files such as model weights or index of embedding vectors, whereas lzma is better for human-readable text files. System resources can also be compressed if necessary. We use - 9 option for both compressors.

Utilizing Token IDs as the CorpusA corpus file must be included in the system to get the actual text of the item retrieved by search (an embedding vector in our case). We have found out that using the file of the encoded token ids of the tokenized texts as the corpus, instead of the raw texts, is beneficial not only because it reduces the inference latency by preprocessing the texts, but also the compressed output size is often slightly smaller.

## 3 Experiments

Experimental SetupWe apply our storage reduction methods to a recent extractive retrieve & read system, DPR Karpukhin et al. (2020), which consists of three different BERT-base encoders: question encoder of the retriever, passage encoder of the retriever, and encoder of the reader with a ranker. All experiments are done on Naver Smart Machine Learning (NSML) Platform Sung et al. (2017); Kim et al. (2018). The training and evaluation details are in Appendix A.1, A.2, and A.3.

Experimental ResultsFigure 1 shows how each of the discussed strategies changes DPR's system size and Exact Match (EM) score on the EfficientQA dev set (see Table 3 and Table 4 in Appendix for details). Our starting point is a standalone open-domain QA system with DPR whose estimated size is 77.5 GB: 1.4 (system) + 0.8 (retriever) + 0.4 (reader) + 61 (index) + 13 (text) GB. The red plot shows from left to right one path to successively apply each strategy to reduce the system footprint to 484.69MB, which is 160 times smaller. Although the methods are described as sequential for easier presentation, the methods with filled markers and dotted lines are orthogonal to each other and thus can be applied in any other order. The methods with unfilled markers and solid lines are built on top of the previous method for each.

Sizing down the corpus from 21,015,325 to 1,224,000 (5.8%) passages (SS2.1) decreases the sys 

tem footprint by a large margin of about 70.5GB with only 2.72% of drop in EM. Using a smaller passage embedding dimension of 128D (SS2.2.1), changing the encoder to MobileBERT (SS2.2.1), and sharing the encoders of the retriever (SS2.2.2) save further 4.1GB of storage with little drop in accuracy of 1.28%. The process of unifying the retriever and reader into a single model (SS2.2.3) drops EM by 1.11, but the accuracy increases by 2.77% (to 34.44%) with iterative finetuning (SS2.2.4). In ablation studies on the three-step training procedure, omitting the knowledge distillation step drops EM by 1.5%, and omitting \({\cal L}_{\text{recon}}\) drops EM by 0.38%.

Applying post-training compression techniques further reduces the system footprint by a large margin while sacrificing little accuracy. EM changes to 34.39% with INT8 quantization, and the rest of the tricks do not affect the accuracy. Converting the PyTorch checkpoint to a binary for TensorFlow Serving to reduce system library dependency and applying bzip2 compression on some of the system resources creates the final system of 484.69MB with an accuracy of 34.33%. Figure 1 shows that this accuracy is higher than the performance of the parametric T5 Roberts et al. (2020) baseline with a comparable docker-level system footprint.8

Footnote 8: The accuracy of the T5 baselines are calculated using the SSM models finetuned on Natural Questions: [https://github.com/google-research/google-research/tree/master/t5_closed_book_qa#released-model-checkpoints](https://github.com/google-research/google-research/tree/master/t5_closed_book_qa#released-model-checkpoints).

In Table 1, we show the test set accuracy of our final system and other baselines. In summary, the performance of our system is higher than all of the parametric baselines, and the accuracy drop from DPR is only 2.45% on the EfficientQA dev set and about 4% on the test set while reducing the system footprint to about 0.6% of the original size.

Our final system achieves the first place in the human (manual) evaluation and the second place in the automatic evaluation on "Systems Under 500MB Track" of the EfficientQA competition. While the accuracy of our system is 32.06% on the EfficientQA test set in the automatic evaluation, which is 1.38% behind the top-performing system Lewis et al. (2021), its accuracy is 42.23% in the human evaluation which is 2.83% higher than the other system. Interestingly, when possibly correct answers are also counted as correct, the accuracy rises to 54.95% (7.58% higher than the other system). Please refer to Table 2 of Min et al. (2021) for more details.

In addition to the EfficientQA dataset, we also perform experiments on open-domain Natural Questions (NQ) Kwiatkowski et al. (2019) and TriviaQA Joshi et al. (2017) to test the generalizability of the proposed methods. The results and detailed analysis are presented in Appendix A.5.

## 4 Related Works

There has recently been a line of work that targets to create storage-efficient open-domain QA systems, especially following the EfficientQA competition. Here, we introduce several approaches concurrent to ours that interested readers may refer to Izacard et al. (2020) and Fajcik et al. (2021) explore the trade-off between storage budget and accuracy, and their retrieve & read systems take up only about 6GB with state-of-the-art performance. Lewis et al. (2021) propose a QA-pair retrieval system for open-domain QA, which enjoys the benefits of high flexibility and low latency. Their retriever answers 1100 questions per second with 41.2% accuracy on NQ, which rises to 47.7% when equipped with a reranker. The variants optimized for small system footprint are the winning systems of two storage-constrained tracks at EfficientQA. Min et al. (2021) review the EfficientQA competition with detailed analysis and summarize all of the top-performing systems.

## 5 Conclusion

We discuss several orthogonal approaches to reduce the system footprint of a retrieve-and-read-based open-domain QA system. The methods together reduce the size of a reference system (DPR) by 160 times with an accuracy drop of 2.45% and 4% on EfficientQA dev and test, respectively. We hope that the presented strategies and results can be helpful for designing future retrieve-and-read systems under a storage-constrained serving environment.

\begin{table}
\begin{tabular}{l r c l} \hline \hline
**Model** & **EM** & **System Size** & **Mechanism** \\ \hline T5-1.1-small+SSM & 18 & 486.61 MB & parametric \\ T5-1.1-XL+SSM & 28 & 5.65 GB & parametric \\ REALM & 35 & 27.19 GB & retrieve \& read \\ \hline DPR & 36 & 77.5 GB & retrieve \& read \\ + Our Methods & 32 & **484.69 MB** & retrieve \& read \\ \hline \hline \end{tabular}
\end{table}
Table 1: System size and Exact Match (EM) score of several standalone open-domain QA systems, reported on the test set of EfficientQA. The value of the baseline systems except for DPR are reported in the EfficientQA leaderboard9. The system size of DPR is estimated as described in Section 3.

 



## Acknowledgements

The authors would like to thank the members of NAVER Clova for proofreading this paper. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).

## References

* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. In NeurIPS, Cited by: SS4.3.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, Cited by: SS1, SS2.2.
* M. Fajcik, M. Docekal, K. Ondrej, and P. Smrz (2021)Pruning the index contents for memory efficient open-domain qa. arXiv preprint arXiv:2102.10697. External Links: Link Cited by: SS1, SS2.2.
* Kelvin Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang (2019)Realm: retrieval-augmented language model pre-training. In ICML, Cited by: SS1, SS2.2.
* S. Han, H. Mao, and W. J. Dally (2016)Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, Cited by: SS1, SS2.2.
* Geoffrey Hinton, O. Vinyals, and J. Dean (2014)Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop, Cited by: SS1, SS2.2.
* G. Izacard and E. Grave (2021)Leveraging passage retrieval with generative models for open domain question answering. EACL. Cited by: SS1, SS2.2.
* G. Izacard, F. Petroni, L. Hosseini, N. De Cao, S. Riedel, and E. Grave (2020)A memory efficient baseline for open domain question answering. arXiv preprint arXiv:2012.15156. Cited by: SS1, SS2.2.
* J. Johnson, M. Douze, and H. Jegou (2019)Billion-scale similarity search with gpus. In IEEE Transactions on Big Data, Cited by: SS1, SS2.2.
* M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017)Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. In ACL, Cited by: SS1, SS2.2.
* V. Karpukhin, B. Oguz, S. Min, L. Wu, S. Edunov, D. Chen, and W. Yih (2020)Dense passage retrieval for open-domain question answering. In EMNLP, Cited by: SS1, SS2.2.
* H. Kim, M. Kim, D. Seo, J. Kim, H. Park, S. Park, H. Jo, K. Kim, Y. King, Y. Yang, Y. Kim, N. Sung, and J. Ha (2018)Nsml: meet the mlas platform with a real-world case study. arXiv preprint arXiv:1712.05902. Cited by: SS1, SS2.2.
* T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelecy, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019)Natural questions: a benchmark for question answering research. In TACL, Cited by: SS1, SS2.2.
* K. Lee, M. Chang, and K. Toutanova (2019)Latent retrieval for weakly supervised open domain question answering. In ACL, Cited by: SS1, SS2.2.
* Patrick Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W. Yin, T. Rocktaschel, et al. (2020)Retrieval-augmented generation for knowledge-intensive nlp tasks. In EMNLP, Cited by: SS1, SS2.2.
* P. Lewis, Y. Wu, L. Liu, P. Minervini, H. Kuttler, A. Pitkus, P. Stenetorp, and S. Riedel (2021)Paq: 65 million probably-asked questions and what you can do with them. arXiv preprint arXiv:2102.07033. Cited by: SS1, SS2.2.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS1, SS2.2.
* Sewon Min, J. Boyd-Graber, C. Alberti, D. Chen, E. Choi, M. Collins, K. Gouu, H. Hajishirzi, K. Lee, J. Falomaki, C. Raffelt, A. Roberts, T. Kwiatkowski, P. Lewis, Y. Wu, H. Kuttler, L. Liu, P. Lin, P. Minervini, P. Stenetorp, S. Riedel, S. Yang, M. Seo, G. Izacard, F. Petroni, L. Hosseini, N. De Cao, E. Grave, I. Yamada, S. Shimadaoaka, M. Suzuki, S. Miyawaki, S. Sato, R. Takahashi, J. Suzuki, M. Fajcik, M. Docekal, Karel Onderaj, P. Smrz, H. Cheng, Y. Shen, X. Liu, P. Hee, W. Chen, J. Gao, B. Oguz, Xilun Chen, V. Karpukhin, S. Peshterliev, D. Okhonko, M. Schlichktrull, S. Gupta, Y. Mehdad, and W. Yih (2021)Neurips 2020 efficientqa competition: systems, analyses and lessons learned. arXiv preprint arXiv:2101.00133. Cited by: SS1, SS2.2.
* A. Roberts, C. Raffel, and N. Shazeer (2020)How much knowledge can you pack into the parameters of a language model?. In EMNLP, Cited by: SS1, SS2.2.
* Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou (2020)Mobilebert:

a compact task-agnostic bert for resource-limited devices. In _ACL_. Sung et al. (2017) Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, and Sunghun Kim. 2017. Nsml: A machine learning platform that enables you to focus on your models. In _NeurIPS Systems for ML Workshop_. Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Dramee, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv preprint arXiv:1910.03771_. Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwook-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _ICLR_. Yang and Seo (2020) Sohee Yang and Minjoon Seo. 2020. Is retriever merely an approximator of reader? _arXiv preprint arXiv:2010.10999_. Zafir et al. (2019) Ofir Zafir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In _NeurIPS EMC2 Workshop_.

## Appendix A Appendix

### Training Details of the Passage Filter

\(J_{\text{total}}\) is the set of all 21M passages that serve as the knowledge source of DPR. \(J^{+}_{\text{train}}\) consists of the top-200 passages retrieved for each of the questions in Natural Questions (Kwiatkowski et al., 2019) train, dev, test set, and EfficientQA dev set. To retrieved the passages, we use the retriever of Yang and Seo (2020) trained on Natural Questions. Smoothed frequency \([\log_{10}(\text{freq}_{p_{j}})+1]\) is considered to create a candidate pool of the positive passages, and oversampling from the pool is done to make \(J^{+^{\prime}}_{\text{train}}\), whereas \(J^{-^{\prime}}_{\text{train}}\) is randomly and uniformly sampled from \(J^{-^{\prime}}_{\text{train}}\). The objective function is defined as follows:

\[-\frac{\sum_{j\in J^{+^{\prime}}_{\text{train}}}\log\sigma(s_{j})+\sum_{j\in J ^{-^{\prime}}_{\text{train}}}\log\left[1-\sigma(s_{j})\right]}{|J^{+^{\prime} }_{\text{train}}|+|J^{-^{\prime}}_{\text{train}}|}.\]

We finetune a RoBERTa-base (Liu et al., 2019) classifier with a batch size of 18 (\(|J^{+^{\prime}}_{\text{train}}|=|J^{-^{\prime}}_{\text{train}}|=9\)), learning rate of 1e-5, dropout rate of 0.1, max norm gradient clipping of 2.0, and warmup steps of 1000, using one V100 GPU. We use the code of HuggingFace Transformers (Wolf et al., 2019), and no additional preprocessing is used on the data other than the tokenization for RoBERTa-base. We train the model for one epoch until all the positive passages oversampled according to the smoothed frequency are seen by the model at least once.

We evaluate the model on the validation set after every 2000 steps of gradient update. We compose \(J^{+}_{\text{val}}\) as a set of indices of the passages that hold the answer for one of the questions in the EfficientQA dev set and are retrieved by an existing retriever as the most relevant passage to the question. We use \(|J^{+}_{\text{val}}|=893\) positive passages and \(|J^{-}_{\text{val}}|=10000-893=9107\) randomly selected negative passages. \(n=1,224,000\) passages are selected for use in our minimal retrieve & read system to fit in the storage budget of 500MB.

### Training Details of the Retriever-Reader with Single Encoder

We have not searched for hyperparameters in almost all experiments on parameter sharing and mainly followed the training setup of DPR.10

Footnote 10: [https://github.com/facebookresearch/DPR](https://github.com/facebookresearch/DPR)

For the experiments on EfficientQA, the training set of Natural Questions (Kwiatkowski et al., 2019) is used to train the 

that report the best result on the EfficientQA dev set12 are selected. Our code is built on top of the official implementation of DPR, so the datasets are preprocessed as done in the code of DPR. Table 2 shows the statistics of the datasets including Natural Questions and TriviaQA used for the experiments in Appendix A.5. We train the models using four to eight P40 or V100 GPUs.

Footnote 12: [https://github.com/google-research-datasets/natural-questions/blob/master/nq_open/NQ-open.efficientqa.dev](https://github.com/google-research-datasets/natural-questions/blob/master/nq_open/NQ-open.efficientqa.dev). 1.1.json

Retriever Encoder SharingWe use MobileBERT Sun et al. (2020) as the pretrained encoder. The encoder output vector is what corresponds to the [CLS] token in the input, and only the first 128 out of 512 dimensions are utilized to calculate the retrieval score. Following the setup of Karpukhin et al. (2020), we use a learning rate of 2e-5, max norm gradient clipping of 2.0, warmup of 1237, sequence length of 256, in-batch negative training of 1 positive and 127 negatives, and training epochs of 40 to 70, applying early stopping if there is a lack of resource. The models are evaluated on the dev set after every epoch.

Unified Retriever-Reader Through Knowledge**Distillation\(k^{\prime}=200\) and \(m=24\) is used to create the training dataset. To train the unified retriever-reader, we use a learning rate of 1e-5, max norm gradient clipping of 2.0, no warmup steps, sequence length of 350, batch size of 16, knowledge distillation temperature \(\tau\) of 3, and training epochs of 16 to 30, applying early stopping when the score seems to be converged.

Since the reader teacher (DPR reader) has a ranker, \(\mathcal{L}_{\text{read}}\) is defined as the sum of the KL divergence between the span scores and the KL divergence between the ranking scores of the teacher and the student. \(\mathcal{L}_{\text{ret}}\) also takes the passage ranking score from the ranker as the distillation target.

The model is evaluated at every 2000 steps, and we select the checkpoint with the highest average EM on \(m^{\prime}\) retrieved passages, where \(m^{\prime}\in\{1,10,20,30,40,50\}\), along with an acceptable reranking accuracy (how many times the positive passage is ranked at the top among 50 candidates).

Iterative Finetuning of Unified Retriever-ReaderWe finetune the model for at most six epochs. The rest of the hyperparameters are the same as described in the previous paragraphs.

### Evaluation Details

The reported EM is the highest EM on \(m^{\prime}\) retrieved passages where \(m^{\prime}\in\{1,10,20,\cdots,100\}\). The original code of DPR searches the answer only in the passage scored the highest by the passage ramker, and thus the answer span with the highest span score in the single passage is selected as the final answer. All of the EM scores presented in this work are also calculated this way.

On the other hand, we have found out that the end-to-end QA accuracy can be slightly increased by using the weighted sum of the passage ranking score \(\mathcal{P}_{\text{rank}}\) and the answer span scores, \(\mathcal{P}_{\text{start}}\) and \(\mathcal{P}_{\text{end}}\) for the start and end positions, respectively, to compare the answer candidates at inference time. Therefore, we have used this scoring method for the model submitted to the EfficientQA leaderboard. Specifically, we use \((1-\lambda)(\log\mathcal{P}_{\text{start}}+\log\mathcal{P}_{\text{end}})+2 \lambda\log\mathcal{P}_{\text{rank}}\) as the score. The answer spans with the top five weighted sum scores in each retrieved passage are selected as the candidate answers, and the one with the highest score is chosen as the final answer. We select \(\lambda=0.8\) based on the performance on the dev set. This method increases the dev set accuracy after the iterative finetuning stage (SS2.2.3) from 34.44 to 34.61.

Due to the discrepancy between the validation accuracy during and after training (described in detail in Appendix A.5), we select up to five checkpoints based on the dev set accuracy observed during training and evaluate them to obtain the one with the actual highest dev set accuracy after the iterative finetuning is done.

### System Footprint

System footprint is measured by the command du  -h   / inside the standalone docker container right after its launching as stated in the EfficientQA competition guideline. The system footprint at runtime may be larger when the resources are initially compressed at the time of launching the container.

Table 3 shows from the top to bottom the detailed

\begin{table}
\begin{tabular}{l r r r} \hline \hline
* & 1,800 & 1,800 \\ Natural Questions & 79,168 (58,880) & 8,757 & 3,610 \\ TriviaQA & 78,785 (60,413) & 8,837 & 11,313 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of the number of questions in each dataset. The values in parenthesis denote the number of questions filtered by DPR preprocessing and used for actual training. There is no training data for EfficientQA, so we use the training set of Natural Questions to train the model for EfficientQA.
 

ablations on how the system size changes from 77.5 GB to 484.69 MB by applying each of the methods discussed in Section 2. The values in the table use MB as the unit. Decreased values are marked in red and increased values are marked in blue.

The docker image is initially assumed to be bitnami/pytorch:1.4.012, and it changes to python:3.6.11-slim-buster13 after adopting TensorFlow (TF) Serving that does not require heavy system libraries as PyTorch does. The most lightweight docker image with python uses Alpine, but TF Serving does not run on an Alpine docker container due to the lack of support of system library requirements.

Footnote 12: [https://hub.docker.com/r/bitnami/pytorch](https://hub.docker.com/r/bitnami/pytorch)

### Experiments: NQ and TriviaQA

Experimental SetupMost of the details to train the models on Natural Questions (NQ) and TriviaQA (Trivia) follow what is written in Appendix A.1 and Appendix A.2, and here we describe only the differences. To train the passage filter, we use log with base 2 instead of 10 for Trivia due to its higher validation set accuracy. The questions used to create the training data are from the train and dev set of the datasets which correspond to the targets of the filter models. To train the unified retriever-reader through knowledge distillation, a batch size of 8 with gradient accumulation steps of 2 is used to train the models using only four V100 GPUs. The maximum number of training epochs is set to 30, but training is stopped around the 16th epoch to shorten the training time even when the scores do not seem to be fully converged. For iterative finetuning, a batch size of 8 with gradient accumulation steps of 2 is again used with four V100 GPUs. The maximum number of training epochs is also set to 30, but the training is stopped before the 10th epoch.

Experimental ResultsFigure 2 shows the EM and docker-level system footprint when each of the discussed strategies is applied to DPR. In the case of the EfficientQA dataset, the step-wise evaluation result on the test set cannot be reported because the answer set is not publicly available. On the other hand, for NQ and Trivia, we present the step-wise accuracy on the test set along with that on the dev set to show how the strategies affect the actual performance on the test set. The evaluation results of all the cases are presented in Table 4.

Let us define the relative performance drop at step \(t\) as the percentage of \(\frac{\text{EM}_{t-1}-\text{EM}_{t}}{\text{EM}_{t-1}}\) where EM\({}_{t}\) is the EM score at the \(t\)-th phase. As shown in the figures and the table, applying the methods to different datasets does not show consistent trends. Because the EfficientQA dataset is constructed in the same way as NQ Min et al. (2021), the trends on these two datasets are similar except that changing the backbone from BERT to MobileBERT (SS2.2.1) results in a significant relative performance drop of 8.21% on the dev set of NQ while the value is only 0.18% on EfficientQA. On the other hand, the same change results in about 4% of relative performance _gain_ on Trivia. A different phenomenon also appears when the retriever encoders are shared (SS2.2.2) that the accuracy rises on EfficientQA and NQ while it drops on Trivia.

The percentage of the final accuracy to the ac

\begin{table}
\begin{tabular}{l r r r r r r} \hline \hline  & **Docker** & **Retriever** & **Reader** & **Index** & **Text File** & **Total** \\ \hline DPR & 1,270 & 836 & 418 & 61,919 & 13,065 & 77,508 \\ §2.1 Passage Filtering (21,015,325 \(\rightarrow\) 1,224,000 passages) & 1,270 & 836 & 418 & 3,681 & 756 & 6,961 \\ §2.2.1 Embedding Dimension Reduction (76SD \(\rightarrow\) 128D) & 1,270 & 836 & 418 & 614 & 756 & 3,894 \\ §2.2.1 Lightweight Encoder (BERT \(\rightarrow\) MobileBERT) & 1,270 & 188 & 94 & 614 & 756 & 2,922 \\ §2.2.2 Retrieer Encoder Sharing & 1,270 & 94 & 94 & 614 & 756 & 2,828 \\ §2.2.3 Unified Retriever-Reader Through Knowlege Distillation & 1,270 & 94 & 0 & 614 & 756 & 2,734 \\ §2.2.4 Iterative Finetuning of Unified Retriever-Reader & 1,270 & 94 & 0 & 614 & 756 & 2,734 \\ §2.3 INT8 Quantization of Index Items & 1,270 & 94 & 0 & 170 & 756 & 2,290 \\ §2.3 Saving Model Weights as FP16 & 1,270 & 47 & 0 & 170 & 756 & 2,243 \\ §2.3 Resource Compression & 1,270 & 42 & 0 & 145 & 187 & 1,644 \\ §2.3 Utilizing Token IDs as the Corpus & 1,270 & 42 & 0 & 145 & 177 & 1,634 \\ §3 TF Serving, Minimizing Library Dependencies, & & & & & & \\ Fusing Index into Model Graph & 312 & 177 & 0 & 0 & 177 & 666 \\ §2.3 System Resource Compression & 130 & 177 & 0 & 0 & 177 & **484** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Detailed ablations on how the system size changes from 77.5 GB to 484.69 MB by applying each of the discussed methods from the top to bottom. The values in the table use MB as the unit. Decreased values are marked in red and increased values are marked in blue.

 

curacy at the start also differs among the datasets: 93.3% and 89.0%14 on the EfficientQA dev and test set, 87.6%, 87.5%, and 85.4% on the NQ dev set, Trivia dev set, and Trivia test set, respectively, but 78.5% on the NQ test set. While the gap between the percentages on the dev and test set is small on Trivia, the value is considerably large on NQ. Also, the gap between the dev and test set accuracy divided by the latter is about 7% on EfficientQA and NQ, while it is only 2% on Trivia.

Footnote 14: 36.0 is used as an approximation for the accuracy of DPR on the EfficientQA test set, which is reported as 36 in [https://github.com/google-research-datasets/natural-questions/tree/master/tn_open](https://github.com/google-research-datasets/natural-questions/tree/master/tn_open).

Meanwhile, a common observation is that passage filtering (SS2.1), embedding dimension reduction (SS2.2.1), and unifying the retriever and the reader through knowledge distillation (SS2.2.3) consistently result in the drop of accuracy. The relative performance drop of each of the methods is 7.40%, 4.08%, and 3.39% on the EfficientQA dev set, 7.61%, 1.67%, and 3.38% on the NQ dev set, 12.07%, 2.14%, and 3.97% on the NQ test set, 7.37%, 4.60%, 7.12% on the Trivia dev set, and 8.60%, 4.25%, 8.03% on the Trivia test set.15

Footnote 15: Figure 1 of Izacard et al. (2020) also shows the trade-off between the index size and system accuracy. Note that the implementation details of their passage filtering and embedding dimension reduction are different from ours.

In the case of unifying the retriever and the reader into one model, one possible cause of the accuracy drop might have come from its currently suboptimal checkpoint selection method. From the moment the retriever and reader are unified into one model and jointly trained, the validation accuracy reported during training uses the outputs of the _initial_ retriever parameters while the actual evaluation must use outputs of the _updated_ retriever parameters at the time of validation. Due to this discrepancy, checkpoint selection based on the validation accuracy at training does not lead to the model with the actual highest dev set accuracy. The discrepancy may further necessitate measuring the true dev set accuracy at several different checkpoints (possibly with high validation accuracy during training) to choose the final model after iterative finetuning. To deal with this issue and fairly compare the best checkpoints, future research may be conducted to refresh the retrieval index during training as in the work of Guu et al. (2019); Xiong et al. (2021), so that the evaluation (and training) may not be done on the stale retrieval outputs.

\begin{table}
\begin{tabular}{l r r r r r} \hline \hline  & **EfficientQA Dev** & **NQ Dev** & **NQ Test** & **TriviaQA Dev** & **TriviaQA Test** \\ \hline DPR & 36.78 & 40.20 & 41.52 & 56.84 & 57.10 \\ §2.1 Passage Filtering & 34.06 & 37.14 & 36.51 & 52.65 & 52.19 \\ §2.2.1 128D Embedding & 32.67 & 36.52 & 35.73 & 50.23 & 49.97 \\ §2.2.1 MobileBERT & 32.61 & 33.52 & 31.66 & 52.20 & 52.02 \\ §2.2.2 Retriever Encoder Sharing & 32.78 & 34.03 & 31.99 & 52.11 & 51.66 \\ §2.2.3 Unified Retrieval-Reader Through KD & 31.67 & 32.88 & 30.72 & 48.40 & 47.51 \\ §2.2.4 Iterative Finentuing & 34.44 & 35.19 & 32.63 & 50.23 & 49.06 \\ §2.3 Post-Training Compression & 34.33 & 35.22 & 32.60 & 49.76 & 48.75 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Detailed ablations on how the Exact Match (EM) score on each dataset changes by applying each of the discussed methods from the top to bottom.

Figure 2: System footprint vs. Exact Match (EM) accuracy on Natural Questions and TriviaQA.