SlowGuess/NAACL_2021_OCR huggingface dataset:
We discuss several orthogonal approaches to reduce the system footprint of a retrieve-and-read-based open-domain QA system. The methods together reduce the size of a reference system (DPR) by 160 times with an accuracy drop of  $2.45\%$  and  $4\%$  on EfficientQA dev and test, respectively. We hope that the presented strategies and results can be helpful for designing future retrieve-and-read systems under a storage-constrained serving environment.
<sup>9</sup>https://ai.google.com/research/NaturalQuestions/efficientqa
The accuracy of the T5 baselines are calculated using the SSM models finetuned on Natural Questions: https://github.com/google-research/google-research/tree/master/t5 Closed book_qa#released-model-checkpoints.
The authors would like to thank the members of Naver Clova for proofreading this paper. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Martin Fajcik, Martin Docekal, Karel Ondrej, and Pavel Smrz. 2021. Pruning the index contents for memory efficient open-domain qa. arXiv preprint arXiv:2102.10697.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2019. Realm: Retrievalaugmented language model pre-training. In ICML.
Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. *EACL*.
Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. 2020. A memory efficient baseline for open domain question answering. arXiv preprint arXiv:2012.15156.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. In *IEEE Transactions on Big Data*.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, Nako Sung, and Jung-Woo Ha. 2018. Nsml: Meet the mlaas platform with a real-world case study. arXiv preprint arXiv:1712.05902.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. In TACL.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In ACL.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In EMNLP.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. Paq: 65 million probably-asked questions and what you can do with them. arXiv preprint arXiv:2102.07033.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Kuttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen tau Yih. 2021. Neurips 2020 efficientqa competition: Systems, analyses and lessons learned. arXiv preprint arXiv:2101.00133.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In EMNLP.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert:
a compact task-agnostic bert for resource-limited devices. In ACL.
Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, and Sunghun Kim. 2017. Nsml: A machine learning platform that enables you to focus on your models. In NeurIPS Systems for ML Workshop.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtopicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv preprint arXiv:1910.03771.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In ICLR.
Sohee Yang and Minjoon Seo. 2020. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010.10999.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In NeurlPS EMC2 Workshop.
A.1 Training Details of the Passage Filter
$J_{\mathrm{total}}$  is the set of all 21M passages that serve as the knowledge source of DPR.  $J_{\mathrm{train}}^{+}$  consists of the top-200 passages retrieved for each of the questions in Natural Questions (Kwiatkowski et al., 2019) train, dev, test set, and EfficientQA dev set. To retrieved the passages, we use the retriever of Yang and Seo (2020) trained on Natural Questions. Smoothed frequency  $\left[\log_{10}(\mathrm{freq}_{p_j}) + 1\right]$  is considered to create a candidate pool of the positive passages, and oversampling from the pool is done to make  $J_{\mathrm{train}}^{+'}$ , whereas  $J_{\mathrm{train}}^{-'}$  is randomly and uniformly sampled from  $J_{\mathrm{train}}^{-}$ . The objective function is defined as follows:
$$
- \frac {\sum_ {j \in J _ {\mathrm {t r a i n}} ^ {+ \prime}} \log \sigma (s _ {j}) + \sum_ {j \in J _ {\mathrm {t r a i n}} ^ {- \prime}} \log [ 1 - \sigma (s _ {j}) ]}{| J _ {\mathrm {t r a i n}} ^ {+ \prime} | + | J _ {\mathrm {t r a i n}} ^ {- \prime} |}.
$$
We finetune a RoBERTa-base (Liu et al., 2019) classifier with a batch size of 18  $(|J_{\mathrm{train}}^{+^{\prime}}| = |J_{\mathrm{train}}^{-^{\prime}}| = 9)$ , learning rate of 1e-5, dropout rate of 0.1, max norm gradient clipping of 2.0, and warmup steps of 1000, using one V100 GPU. We use the code of Hugging-Face Transformers (Wolf et al., 2019), and no additional preprocessing is used on the data other than the tokenization for RoBERTa-base. We train the model for one epoch until all the positive passages oversampled according to the smoothed frequency are seen by the model at least once.
We evaluate the model on the validation set after every 2000 steps of gradient update. We compose  $J_{\mathrm{val}}^{+}$  as a set of indices of the passages that hold the answer for one of the questions in the EfficientQA dev set and are retrieved by an existing retriever as the most relevant passage to the question. We use  $|J_{\mathrm{val}}^{+}| = 893$  positive passages and  $|J_{\mathrm{val}}^{-}| = 10000 - 893 = 9107$  randomly selected negative passages.  $n = 1,224,000$  passages are selected for use in our minimal retrieve & read system to fit in the storage budget of 500MB.
A.2 Training Details of the Retriever-Reader with Single Encoder
We have not searched for hyperparameters in almost all experiments on parameter sharing and mainly followed the training setup of DPR. $^{10}$
For the experiments on EfficientQA, the training set of Natural Questions (Kwiatkowski et al., 2019) is used to train the models. The checkpoints
$^{10}$ https://github.com/facebookresearch/DPR
Table 2: Statistics of the number of questions in each dataset. The values in parenthesis denote the number of questions filtered by DPR preprocessing and used for actual training. There is no training data for EfficientQA, so we use the training set of Natural Questions to train the model for EfficientQA.
Dataset	Train	Dev	Test
EfficientQA	-	1,800	1,800
Natural Questions	79,168 (58,880)	8,757	3,610
TriviaQA	78,785 (60,413)	8,837	11,313
that report the best result on the EfficientQA dev set $^{11}$  are selected. Our code is built on top of the official implementation of DPR, so the datasets are preprocessed as done in the code of DPR. Table 2 shows the statistics of the datasets including Natural Questions and TriviaQA used for the experiments in Appendix A.5. We train the models using four to eight P40 or V100 GPUs.
Retriever Encoder Sharing We use MobileBERT (Sun et al., 2020) as the pretrained encoder. The encoder output vector is what corresponds to the [CLS] token in the input, and only the first 128 out of 512 dimensions are utilized to calculate the retrieval score. Following the setup of Karpukhin et al. (2020), we use a learning rate of 2e-5, max norm gradient clipping of 2.0, warmup of 1237, sequence length of 256, in-batch negative training of 1 positive and 127 negatives, and training epochs of 40 to 70, applying early stopping if there is a lack of resource. The models are evaluated on the dev set after every epoch.
Unified Retriever-Reader Through Knowledge Distillation  $k' = 200$  and  $m = 24$  is used to create the training dataset. To train the unified retriever-reader, we use a learning rate of 1e-5, max norm gradient clipping of 2.0, no warmup steps, sequence length of 350, batch size of 16, knowledge distillation temperature  $\tau$  of 3, and training epochs of 16 to 30, applying early stopping when the score seems to be converged.
Since the reader teacher (DPR reader) has a ranker,  $\mathcal{L}_{\mathrm{read}}$  is defined as the sum of the KL divergence between the span scores and the KL divergence between the ranking scores of the teacher and the student.  $\mathcal{L}_{\mathrm{ret}}$  also takes the passage ranking score from the ranker as the distillation target.
The model is evaluated at every 2000 steps, and we select the checkpoint with the highest aver
age EM on  $m^{\prime}$  retrieved passages, where  $m^{\prime} \in \{1, 10, 20, 30, 40, 50\}$ , along with an acceptable reranking accuracy (how many times the positive passage is ranked at the top among 50 candidates).
Iterative Finetuning of Unified Retriever-Reader We finetune the model for at most six epochs. The rest of the hyperparameters are the same as described in the previous paragraphs.
The reported EM is the highest EM on  $m'$  retrieved passages where  $m' \in \{1, 10, 20, \dots, 100\}$ . The original code of DPR searches the answer only in the passage scored the highest by the passage ranker, and thus the answer span with the highest span score in the single passage is selected as the final answer. All of the EM scores presented in this work are also calculated this way.
On the other hand, we have found out that the end-to-end QA accuracy can be slightly increased by using the weighted sum of the passage ranking score  $\mathcal{P}_{\mathrm{rank}}$  and the answer span scores,  $\mathcal{P}_{\mathrm{start}}$  and  $\mathcal{P}_{\mathrm{end}}$  for the start and end positions, respectively, to compare the answer candidates at inference time. Therefore, we have used this scoring method for the model submitted to the EfficientQA leaderboard. Specifically, we use  $(1 - \lambda)(\log \mathcal{P}_{\mathrm{start}} + \log \mathcal{P}_{\mathrm{end}}) + 2\lambda \log \mathcal{P}_{\mathrm{rank}}$  as the score. The answer spans with the top five weighted sum scores in each retrieved passage are selected as the candidate answers, and the one with the highest score is chosen as the final answer. We select  $\lambda = 0.8$  based on the performance on the dev set. This method increases the dev set accuracy after the iterative finetuning stage (§2.2.3) from 34.44 to 34.61.
Due to the discrepancy between the validation accuracy during and after training (described in detail in Appendix A.5), we select up to five checkpoints based on the dev set accuracy observed during training and evaluate them to obtain the one with the actual highest dev set accuracy after the iterative finetuning is done.
A.4 System Footprint
System footprint is measured by the command du -h / inside the standalone docker container right after its launching as stated in the EfficientQA competition guideline. The system footprint at runtime may be larger when the resources are initially compressed at the time of launching the container.
Table 3 shows from the top to bottom the detailed
$^{11}$ https://github.com/google-research-datasets/natural-questions/blob/master/nq_open/NQ-open.efficientqa.dev. 1.1.jsonl
Table 3: Detailed ablations on how the system size changes from 77.5 GB to 484.69 MB by applying each of the discussed methods from the top to bottom. The values in the table use MB as the unit. Decreased values are marked in red and increased values are marked in blue.
Docker	Retriever	Reader	Index	Text File	Total
DPR	1,270	836	418	61,919	13,065	77,508
§2.1 Passage Filtering (21,015,325 → 1,224,000 passages)	1,270	836	418	3,681	756	6,961
§2.2.1 Embedding Dimension Reduction (768D → 128D)	1,270	836	418	614	756	3,894
§2.2.1 Lightweight Encoder (BERT → MobileBERT)	1,270	188	94	614	756	2,922
§2.2.2 Retriever Encoder Sharing	1,270	94	94	614	756	2,828
§2.2.3 Unified Retriever-Reader Through Knowledge Distillation	1,270	94	0	614	756	2,734
§2.2.4 Iterative Finetuning of Unified Retriever-Reader	1,270	94	0	614	756	2,734
§2.3 INT8 Quantization of Index Items	1,270	94	0	170	756	2,290
§2.3 Saving Model Weights as FP16	1,270	47	0	170	756	2,243
§2.3 Resource Compression	1,270	42	0	145	187	1,644
§2.3 Utilizing Token IDs as the Corpus	1,270	42	0	145	177	1,634
§3 TF Serving, Minimizing Library Dependencies, Fusing Index into Model Graph	312	177	0	0	177	666
§2.3 System Resource Compression	130	177	0	0	177	484
ablations on how the system size changes from 77.5 GB to 484.69 MB by applying each of the methods discussed in Section 2. The values in the table use MB as the unit. Decreased values are marked in red and increased values are marked in blue.
The docker image is initially assumed to be bitnami/pytorch:1.4.0 $^{12}$ , and it changes to python:3.6.11-slim-buster $^{13}$  after adopting TensorFlow (TF) Serving that does not require heavy system libraries as PyTorch does. The most lightweight docker image with python uses Alpine, but TF Serving does not run on an Alpine docker container due to the lack of support of system library requirements.
A.5 Experiments: NQ and TriviaQA
Experimental Setup Most of the details to train the models on Natural Questions (NQ) and TriviaQA (Trivia) follow what is written in Appendix A.1 and Appendix A.2, and here we describe only the differences. To train the passage filter, we use log with base 2 instead of 10 for Trivia due to its higher validation set accuracy. The questions used to create the training data are from the train and dev set of the datasets which correspond to the targets of the filter models. To train the unified retriever-reader through knowledge distillation, a batch size of 8 with gradient accumulation steps of 2 is used to train the models using only four V100 GPUs. The maximum number of training epochs is set to 30, but training is stopped around the 16th epoch to shorten the training time even when the scores do not seem to be fully converged.
For iterative finetuning, a batch size of 8 with gradient accumulation steps of 2 is again used with four V100 GPUs. The maximum number of training epochs is also set to 30, but the training is stopped before the 10th epoch.
Experimental Results Figure 2 shows the EM and docker-level system footprint when each of the discussed strategies is applied to DPR. In the case of the EfficientQA dataset, the step-wise evaluation result on the test set cannot be reported because the answer set is not publicly available. On the other hand, for NQ and Trivia, we present the stepwise accuracy on the test set along with that on the dev set to show how the strategies affect the actual performance on the test set. The evaluation results of all the cases are presented in Table 4.
Let us define the relative performance drop at step  $t$  as the percentage of  $\frac{\mathrm{EM}_{t-1} - \mathrm{EM}_t}{\mathrm{EM}_{t-1}}$  where  $\mathrm{EM}_t$  is the EM score at the  $t$ -th phase. As shown in the figures and the table, applying the methods to different datasets does not show consistent trends. Because the EfficientQA dataset is constructed in the same way as NQ (Min et al., 2021), the trends on these two datasets are similar except that changing the backbone from BERT to MobileBERT ( $\S 2.2.1$ ) results in a significant relative performance drop of  $8.21\%$  on the dev set of NQ while the value is only  $0.18\%$  on EfficientQA. On the other hand, the same change results in about  $4\%$  of relative performance gain on Trivia. A different phenomenon also appears when the retriever encoders are shared ( $\S 2.2.2$ ) that the accuracy rises on EfficientQA and NQ while it drops on Trivia.
The percentage of the final accuracy to the ac
<sup>12</sup>https://hub.docker.com/r/bitnami/pytorch
13 https://hub.docker.com/_/python
(a) Natural Questions
(b) TriviaQA
Figure 2: System footprint vs. Exact Match (EM) accuracy on Natural Questions and TriviaQA.
Table 4: Detailed ablations on how the Exact Match (EM) score on each dataset changes by applying each of the discussed methods from the top to bottom.
EfficientQA Dev	NQ Dev	NQ Test	TriviaQA Dev	TriviaQA Test
DPR	36.78	40.20	41.52	56.84	57.10
§2.1 Passage Filtering	34.06	37.14	36.51	52.65	52.19
§2.2.1 128D Embedding	32.67	36.52	35.73	50.23	49.97
§2.2.1 MobileBERT	32.61	33.52	31.66	52.20	52.02
§2.2.2 Retriever Encoder Sharing	32.78	34.03	31.99	52.11	51.66
§2.2.3 Unified Retriever-Reader Through KD	31.67	32.88	30.72	48.40	47.51
§2.2.4 Iterative Finetuning	34.44	35.19	32.63	50.23	49.06
§2.3 Post-Training Compression	34.33	35.22	32.60	49.76	48.75
curacy at the start also differs among the datasets:  $93.3\%$  and  $89.0\%^{14}$  on the EfficientQA dev and test set,  $87.6\%$ ,  $87.5\%$ , and  $85.4\%$  on the NQ dev set, Trivia dev set, and Trivia test set, respectively, but  $78.5\%$  on the NQ test set. While the gap between the percentages on the dev and test set is small on Trivia, the value is considerably large on NQ. Also, the gap between the dev and test set accuracy divided by the latter is about  $7\%$  on EfficientQA and NQ, while it is only  $2\%$  on Trivia.
Meanwhile, a common observation is that passage filtering ( $\S 2.1$ ), embedding dimension reduction ( $\S 2.2.1$ ), and unifying the retriever and the reader through knowledge distillation ( $\S 2.2.3$ ) consistently result in the drop of accuracy. The relative performance drop of each of the methods is  $7.40\%$ ,  $4.08\%$ , and  $3.39\%$  on the EfficientQA dev set,  $7.61\%$ ,  $1.67\%$ , and  $3.38\%$  on the NQ dev set,  $12.07\%$ ,  $2.14\%$ , and  $3.97\%$  on the NQ test set,  $7.37\%$ ,  $4.60\%$ ,  $7.12\%$  on the Trivia dev set, and  $8.60\%$ ,  $4.25\%$ ,  $8.03\%$  on the Trivia test set.[15]
In the case of unifying the retriever and the reader into one model, one possible cause of the accuracy drop might have come from its currently suboptimal checkpoint selection method. From the moment the retriever and reader are unified into one model and jointly trained, the validation accuracy reported during training uses the outputs of the initial retriever parameters while the actual evaluation must use outputs of the updated retriever parameters at the time of validation. Due to this discrepancy, checkpoint selection based on the validation accuracy at training does not lead to the model with the actual highest dev set accuracy. The discrepancy may further necessitate measuring the true dev set accuracy at several different checkpoints (possibly with high validation accuracy during training) to choose the final model after iterative finetuning. To deal with this issue and fairly compare the best checkpoints, future research may be conducted to refresh the retrieval index during training as in the work of Guu et al. (2019); Xiong et al. (2021), so that the evaluation (and training) may not be done on the stale retrieval outputs.
1436.0 is used as an approximation for the accuracy of DPR on the EfficientQA test set, which is reported as 36 in https://github.com/google-research-datasets/natural-questions/tree/master/nq_open.