SlowGuess/NAACL_2022_OCR huggingface dataset:
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them. Transactions of the Association for Computational Linguistics, 9(0):1098-1115.
Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021. In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 163-173, Online. Association for Computational Linguistics.
S. Lloyd. 1982. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129-137.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9:329-345.
Yu A. Malkov and D. A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824-836.
Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin'ichi Satoh. 2018. A survey of product quantization. ITE Transactions on Media Technology and Applications, 6(1):2-10.
Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Kuttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen-tau Yih. 2021. NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned. In Proceedings of the NeurIPS 2020 Competition and Demonstration Track, volume 133 of Proceedings of Machine Learning Research, pages 86-111. PMLR.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2012. Foundations of machine learning. Cambridge, MA: MIT Press. Journal Abbreviation: Adapt. Comput. Mach. Learn. Publication Title: Adaptive Computation and Machine Learning.
Barlas Oğuz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, and Yashar Mehdad. 2021. Domain-matched Pre-training Tasks for Dense Retrieval. arXiv:2107.13602 [cs]. ArXiv: 2107.13602.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Roktäschel, and Sebastian Riedel. 2021. KILT: a Benchmark for Knowledge Intensive Language Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2021. Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees? In International Conference on Learning Representations.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2010.08191 [cs]. ArXiv:2010.08191.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Stephen Robertson. 2008. On the history of evaluation in IR. Journal of Information Science, 34(4):439-456. _eprint: https://doi.org/10.1177/0165551507086989.
Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr., 3(4):333-389. Place: Hanover, MA, USA Publisher: Now Publishers Inc.
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, and Bryan Catanzaro. 2021. End-to-End Training of Neural Retrievers for Open-Domain Question Answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6648–6662, Online. Association for Computational Linguistics.
Rob Schapire. 2007. Theory and Applications of Boosting. In Neural Information Processing Systems, Tutorials, page 104.
Robert E. Schapire. 1990. The strength of weak learnability. Machine Learning, 5(2):197-227.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple Entity-Centric Questions Challenge Dense Retrievers. arXiv:2109.08535 [cs]. ArXiv: 2109.08535.
Sivic and Zisserman. 2003. Video Google: a text retrieval approach to object matching in videos. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 1470-1477 vol.2.
Ingo Steinwart and Andreas Christmann. 2008. Support vector machines. Springer Science & Business Media.
Nandan Thakur, Nils Reimers, Andreas Rückle, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. CoRR, abs/2104.08663. ArXiv: 2104.08663.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Ellen M. Voorhees and Dawn M. Tice. 2000. Building a Question Answering Test Collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00, pages 200-207, New York, NY, USA. ACM. Event-place: Athens, Greece.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Qiang Wu, C. Burges, K. Svore, and Jianfeng Gao. 2009. Adapting boosting for information retrieval measures. Information Retrieval.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. arXiv:2007.00808 [cs]. ArXiv: 2007.00808.
Jun Xu and Hang Li. 2007. AdaRank: a boosting algorithm for information retrieval. In SIGIR.
Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. 2021. Efficient Passage Retrieval with Hashing for Open-domain Question Answering. arXiv:2106.00882 [cs]. ArXiv:2106.00882.
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of Lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253-1256.
Sohee Yang and Minjoon Seo. 2020. Is Retriever Merely an Approximator of Reader? arXiv:2010.10999 [cs]. ArXiv:2010.10999.
Sohee Yang and Minjoon Seo. 2021. Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering. arXiv:2104.07242 [cs]. ArXiv: 2104.07242.
Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained Transformers for Text Ranking: BERT and Beyond. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM '21, pages 1154-1156, New York, NY, USA. Association for Computing Machinery. Event-place: Virtual Event, Israel.
Wen-tau Yih, Kristina Toutanova, John C. Platt, and Christopher Meek. 2011. Learning Discriminative Projections for Text Similarity Measures. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, CoNLL '11, pages 247-256, USA. Association for Computational Linguistics. Event-place: Portland, Oregon.
Tan Yu, Junsong Yuan, Chen Fang, and Hailin Jin. 2018. Product Quantization Network for Fast Image Retrieval. In Computer Vision - ECCV 2018, volume 11205, pages 191-206, Cham. Springer International Publishing. Series Title: Lecture Notes in Computer Science.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021a. Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance. arXiv:2108.00644 [cs]. ArXiv: 2108.00644.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021b. Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval. arXiv:2110.05789 [cs]. ArXiv: 2110.05789 version: 1.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021c. Optimizing Dense Retrieval Model Training with Hard Negatives. arXiv:2104.08051 [cs]. ArXiv:2104.08051.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Adversarial retriever-ranker for dense text retrieval. arXiv preprint arXiv:2110.03611.
Methods	Round	Total Dim.	MSMARCO MRR@10
DPR w/ iteratively-sampled negs. (Initial Hard Negs. BM25)	1	768	28.6
2	768	32.2
3	768	32.3
4	768	32.8
5	768	32.6
1	160	28.9
2	160	31.4
3	160	31.7
4	160	32.1
5	160	32.3
6	160	32.5
7	160	32.3
DrBoost (32-dim subvectors) (Initial Negs. Random)	1	32	22.2
2	64	31.5
3	96	33.8
4	128	34.3
5	160	34.4
Table 5: Ablations for the number of rounds for DPR with iterative negatives and DrBoost for MSMARCO
A.1 Number of Rounds
The performance of DPR and DrBoost on MS-MARCO for different numbers of rounds are shown in Table 5. We find that all models saturate at about 4 or 5 rounds. Note DrBoost does not need more iterations to train, even though it does not use BM25 negatives for the first round. On NQ, adding a  $6^{\text{th}}$  model slightly improves DrBoost's precision, at the expense of recall (see Table 1).
While iterative training is expensive, we find that subsequent rounds are much cheaper than the first round, with the first round taking  $\sim 20\mathrm{K}$  steps in our experiments to converge, with additional DrBoost rounds converging after about  $3\mathrm{K}$  steps.
Bagging Dense Retrieval We also trained a simple ensemble of six 32-dim DPR models for NQ, which we compare to our  $6 \times 32$ -dim component DrBoost. This experiment investigates whether the improvement over DPR is just a simple ensembling effect, or whether it is due to boosting effects and specialization of concerns. This DPR ensemble performs poorly, scoring 74.5 R@20 (not shown in tables),  $6.8\%$  below the equivalent DrBoost, confirming that the boosting formulation is important, not simply having several ensembled dense retrievalers.
We implement our models architectures based on HuggingFace's Transformers (Wolf et al., 2020)
and run our experiments on 16 V100 GPUs. For all training rounds, we used the same set of training hyperparameters — we set learning rate as 3e-5, dropout as 0.1, weight decay as 0.01, batch size as 2 (per GPU) and max training steps as 30k. The maximum question and passage lengths are set as 40 and 200 respectively and we accompany each question with 50 passages during training. Using our training infrastructure, the first round of the training takes about 8 hours and each additional training round takes about 1.5 hours until convergence. We always use the dev loss for model selection.
A.3 Detailed results
DrBoost	DPR	DPR, 160 dim.
(n=4096)	(n=16384)	(n=65536)	(n=4096)	(n=16384)	(n=65536)	(n=4096)	(n=16384)
Exhaustive	0.3438	0.3438	0.3438	0.328	0.328	0.328	0.3248	0.3248
1	0.1905	0.1884	0.2057	0.1277	0.1186	0.1088	0.1669	0.1637
2	0.2338	0.2359	0.2458	0.172	0.1599	0.1479	0.2129	0.2072
4	0.2694	0.2652	0.2719	0.2095	0.1996	0.1836	0.2465	0.2395
8	0.2919	0.2873	0.2955	0.2433	0.2326	0.2155	0.2722	0.2637
16	0.3106	0.3018	0.3094	0.2693	0.2532	0.2415	0.2906	0.2822
32	0.324	0.3161	0.3179	0.2855	0.2715	0.2629	0.3027	0.297
64	0.3314	0.3236	0.3266	0.2994	0.2864	0.2791	0.3127	0.3063
128	0.3382	0.332	0.3312	0.31	0.2982	0.2922	0.3179	0.3129
256	0.34	0.3375	0.3354	0.3161	0.3092	0.3011	0.3206	0.3182
512	0.3424	0.34	0.3395	0.3226	0.3141	0.3085	0.3232	0.3212
1024	0.3437	0.3416	0.3415	0.325	0.3197	0.3139	0.3243	0.3229
2048	0.3438	0.343	0.342	0.3279	0.3243	0.3188	0.3247	0.3242
4096	0.3435	0.3433	0.3268	0.3228	0.3249
8192	0.3438	0.3432	0.3278	0.3254	0.3248
16384	0.3435	0.3268
32768	0.3437	0.3278
Table 6: IVF indexing results on MSMARCO. Metric is MRR@10.  $n$  refers to number of clusters used for IVF training.
DrBoost, 160 dim	DrBoost-distilled, 160 dim	DrBoost, 192 dim	DrBoost-distilled, 192 dim	DPR	DPR, 192 dim.
Exhaustive	0.876	0.868	0.874	0.870	0.879	0.866
1	0.683	0.706	0.684	0.701	0.595	0.623
2	0.751	0.767	0.750	0.760	0.670	0.688
4	0.795	0.800	0.793	0.803	0.717	0.741
8	0.819	0.821	0.819	0.825	0.756	0.778
16	0.836	0.837	0.835	0.840	0.795	0.806
32	0.850	0.849	0.845	0.848	0.816	0.826
64	0.859	0.855	0.858	0.856	0.835	0.838
128	0.865	0.858	0.864	0.859	0.845	0.847
256	0.870	0.862	0.868	0.863	0.855	0.855
512	0.874	0.864	0.870	0.866	0.864	0.861
1024	0.874	0.865	0.871	0.866	0.871	0.864
2048	0.875	0.865	0.873	0.867	0.874	0.865
Table 7: IVF indexing results on NQ. Metric is Recall@100. The number of clusters used for IVF training was 65536.
DrBoost, 160 dim	DrBoost-distilled, 160 dim	DrBoost, 192 dim	DrBoost-distilled, 192 dim	DPR	DPR, 192 dim.
Exhaustive	0.809	0.809	0.813	0.809	0.827	0.808
1	0.624	0.650	0.625	0.647	0.518	0.557
2	0.686	0.703	0.684	0.703	0.597	0.625
4	0.732	0.744	0.730	0.746	0.647	0.679
8	0.758	0.764	0.755	0.764	0.690	0.717
16	0.771	0.779	0.775	0.780	0.732	0.743
32	0.784	0.793	0.786	0.791	0.760	0.766
64	0.794	0.797	0.799	0.799	0.779	0.780
128	0.799	0.800	0.805	0.801	0.791	0.789
256	0.804	0.803	0.810	0.804	0.803	0.797
512	0.807	0.805	0.812	0.807	0.813	0.803
1024	0.808	0.806	0.812	0.807	0.820	0.805
2048	0.808	0.806	0.813	0.808	0.823	0.807
Table 8: IVF indexing results on NQ. Metric is Recall@20. The number of clusters used for IVF training was 65536.
DrBoost, 160 dim	DrBoost-distilled, 160 dim	DrBoost, 192 dim	DrBoost-distilled, 192 dim	DPR	DPR, 192 dim.
Exhaustive	0.710	0.706	0.715	0.703	0.731	0.710
1	0.544	0.560	0.535	0.557	0.439	0.475
2	0.597	0.615	0.594	0.605	0.506	0.540
4	0.634	0.646	0.636	0.644	0.551	0.593
8	0.662	0.663	0.663	0.665	0.597	0.623
16	0.678	0.676	0.681	0.680	0.640	0.653
32	0.691	0.689	0.692	0.688	0.666	0.671
64	0.699	0.692	0.702	0.695	0.687	0.685
128	0.704	0.696	0.708	0.699	0.698	0.694
256	0.707	0.701	0.710	0.702	0.709	0.703
512	0.709	0.703	0.712	0.703	0.718	0.707
1024	0.710	0.704	0.713	0.703	0.726	0.708
2048	0.710	0.704	0.714	0.704	0.728	0.709
Table 9: IVF indexing results on NQ. Metric is Recall@5. The number of clusters used for IVF training was 65536.
Rounds	who got the first Nobel prize in physics?	when is the next deadpool movie being released?
1	0: Title: Nobel Prize in PhysicsThe Nobel Prize in Physics is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the ...1: Title: Nobel Prize in Physics...receive a diploma, a medal and a document confirming the prize amount. Nobel Prize in Physics ...2: Title: Nobel Prize controversies...research CERN, commented in a scientific meet in Kolkata titled "Frontiers of Science" that "it is unfortunate that pioneering ...	0: Title: Deadpool (film)...was written by Reese and Wernick and played in front of "Logan". "Deadpool 2" was released on May 18, 2018, with ...1: Title: Deadpool 2...chimichangas, traditionally Deadpool's favorite food, as well as "Deadpool"-inspired Harder drinks. The campaign also ...2: Title: Deadpool 2...the final two hours. By May 2018, Leitch was working on an official extended edition of the film with Fox wanting to "spin that ...
2	0: Title: George B. McClellanGeorge Brinton McClellan (December 3, 1826-October 29, 1885) was an American soldier, civil engineer, railroad executive ...1: Title: Johannes BrahmsJohannes Brahms ; 7 May 1833 – 3 April 1897) was a German composer and pianist of the Romantic period. Born in Hamburg ...2: Title: Bedebede ( ; 672/3 – 26 May 735), also known as Saint Bede, Venerable Bede, and Bede the Venerable ( ), was an English Benedictine ...	0: Title: Here and Now (2018 TV series)Here and Now is an American drama television series created by Alan Ball. The series consists of ten episodes and ...1: Title: Deadpool 2...is dedicated to her memory. The film's score is the first to receive a parental advisory warning for explicit content, and ...2: Title: I'm New HereI'm New Here is the 13th and final studio album by American vocalist and pianist Gil Scott-Heron. It was released on February ...
3	0: Title: Henri PoincareJules Henri Poincaré ( ; 29 April 1854 – 17 July 1912) was a French mathematician, theoretical physicist, engineer, and ...1: Title: Marie Curie...named in her honor. Marie Curie Marie Skłodowska Curie ( ; ; born Maria Salomea Skłodowska; 7 November 18674 July 1934 ...2: Title: Alberto Santos-DumontAlberto Santos-Dumont ( ; 20 July 187323 July 1932, usually referred to as simply Santos-Dumont) was a Brazilian inventor ...	0: Title: Deadpool 2...is dedicated to her memory. The film's score is the first to receive a parental advisory warning for explicit content, and ...1: Title: Deadpool (film)...was written by Reese and Wernick and played in front of "Logan". "Deadpool 2" was released on May 18, 2018, with ...2: Title: Kong: Skull Island...later moved to Warner Bros. in order to develop a shared cinematic universe featuring Godzilla and King Kong. ...
Table 10: Example retrieval results from each round of DrBoost. Only the beginning of each passage is shown.