SlowGuess/NAACL_2022_OCR huggingface dataset:
4.2 Approximate MIPS
Table 1 also shows how DPR and DrBoost behave under IVF MIPS search, which is shown graphically in Figure 1 as well. We find that DrBoost dramatically outperforms DPR in IVF search, indicating that much faster search is possible with DrBoost. High-dimensional embeddings suffer under IVF due to the curse of dimensionality, and thus compact embeddings are important. Using 8
Figure 1: Search accuracy vs the number of clusters visited in IVF search (proportional to latency). Accuracy drops as search speed increases, but the accuracy drop-off for DrBoost is much slower than for DPR.
search probes, DrBoost outperforms DPR by  $10.5\%$  on MSMARCO and  $6.3\%$  on NQ in R@100. The dimensionally-matched DPR is stronger, but still trails DrBoost by about  $4\%$  using 8 probes. The strongest exact search model is thus not necessarily the best in practical approximate MIPS settings. For example, if we can tolerate a  $10\%$  relative drop in accuracy from the best performing system's exact search, DrBoost requires 16 (4) probes for MSMARCO (NQ) to reach the required accuracy, whereas DPR will require 1024 (16), meaning DrBoost can be operated approximately 64x (4x) faster.
The distilled DrBoost is also shown for NQ in Table 1. The precision (low R@K values) is essentially unaffected, (exact search drops by  $0.1\%$  for R@20), but recall drops slightly  $(-0.7\% \mathrm{R}@\mathrm{100})$ . Interestingly, the distilled DrBoost performs even better under IVF search, improving over DrBoost by  $\sim 1\%$  at low numbers of probes. Crucially, whilst the distilled DrBoost is only slightly better than the 192-dim DPR under exact search, it is  $4-5\%$  stronger under IVF with 8 probes (alternatively, 8x faster for equivalent accuracy).
Aside from fast retrieval, small indices are also important for edge devices, or for scalability reasons. While DrBoost can already produce high quality compact embeddings, Product Quantization
Methods	Total Dimension	MSMARCO	Natural Questions
MRR@10	R@20	R@100
Exact Search	IVF 8	IVF 64	Exact Search	IVF 4	IVF 8	IVF 32	Exact Search	IVF 4	IVF 8	IVF 32
BM25 (Yang et al., 2017)	-	18.7	-	-	59.1	-	-	-	73.7	-	-	-
AR2 (Zhang et al., 2021)	768	39.5	-	-	86.0	-	-	-	90.1	-	-	-
DPR w/ iteratively-sampled negatives	768	32.8	21.6	27.9	82.7	64.7	69.0	76.0	87.9	71.7	75.6	81.6
160 / 192*	32.5	26.8	30.2	80.8	67.9	71.7	76.6	86.6	74.1	77.8	82.6
DrBoost (32-dim subvectors)	160 (5x32d)	34.4	29.6	32.7	80.9	73.2	75.8	78.4	87.6	79.5	81.9	85.0
192 (6x32d)	-	-	-	81.3	73.0	75.5	78.6	87.4	79.3	81.9	84.5
DrBoost-Distilled	160	-	-	-	80.8	74.4	76.4	79.3	86.8	80.0	82.1	85.0
Table 1: Summary of Results on MSMARCO development set and NaturalQuestions test set. "Exact" indicates Exact MIPS results, IVF indicates IVF MIPS search with 65K centroids, with the number of search probes (proportional to search speed) indicated. *Dimensional-matched DPR is 160 dims for MSMARCO and 192 for DPR.
Methods	Total
Dim.	Size
(GB)	NQ
R@20	R@100
DPR (Yamada et al., 2021)	768	64.6	78.4	85.4
+ PQ (8-dim subvecs)	2.0	72.2	81.2
BPR (Yamada et al., 2021)	768*	2.0	77.9	85.7
DrBoost	160	13.5	80.9	87.6
+ PQ (4-dim subvecs)	0.84	80.3	86.8
+ PQ (8-dim subvecs)	0.42	76.7	84.8
Table 2: Product Quantization Results. * Indicates Binary vector.
can reduce this even further. Table 2 shows that DrBoost's NQ index can be compressed from 13.5 GB to 840MB with less than  $1\%$  drop in performance. We compare to BPR (Yamada et al., 2021), a method specifically designed to learn small indices through binarization. DrBoost's PQ index is  $2.4\mathrm{x}$  smaller than the BPR index reported by Yamada et al. (2021), whilst being  $2.4\%$  more accurate  $(R@20)$ . A more aggressive quantization leads to a 420MB index —  $4.8\mathrm{x}$  smaller than BPR — whilst only being  $1.2\%$  less accurate.
5.1 Qualitative Analysis
Since each round's model is learned on the errors of the previous round, we expect each learner to "specialize" and learn complementary representations. To see if this is qualitatively true, we look at the retrieved passages from each round's retriever in isolation (Table 10 in §A.3). Indeed, we find that each 32-dim sub-vector tackles the query from different angles. For instance, for the query "who got the first nobel prize in physics?", the first subvector captures general topical similarity based on keywords, retrieving passages related to the "Nobel
Figure 2: Quantiles of the top-20 margin on the NQ training set, for each iteration of DrBoost.
Prize". The second focuses mostly on the first paragraphs of articles of prominent historical personalities, presumably because these are highly likely to contain answers in general; and the third one retrieves from the pages of famous scientists and inventors. The combined DrBoost model would favor passages in the intersection of these sets.
5.2 In-distribution generalization
Boosting algorithms are remarkably resistant to over-fitting, even when the classifier has sufficient capacity to achieve zero training error. In their landmark paper, Bartlett et al. (1998) show that this generalization property is a result of the following: the training margins increase with each iteration of boosting. We empirically show the same to be true for DrBoost. For a fixed query embedding, dense retrieval acts as a linear binary classifier, where the gold passage is positive and all other passages are negatives (Eq. (1)). We adopt the classical definition of margin for linear classifiers to dense retrieval by defining a top- $k$  margin:
$$
\operatorname {T o p - k} \operatorname {m a r g i n} _ {i} = \frac {h \left(q _ {i} , c ^ {+}\right) - \max  _ {c ^ {-}} ^ {\{k \}} h \left(q _ {i} , c ^ {-}\right)}{\left| \left| q _ {i} \right| \right| \mu_ {c}} \tag {2}
$$
Method	SciFact NDCG@10	FiQA NDCG@10	Quora NDCG@10	ArguAna NDCG@10
SotA Dense	64.3	30.8	85.2	42.9
DPR (160 dim)	50.9	22.8	84.3	42.5
DrBoost (160 dim)	49.7	22.4	78.8	39.9
Table 3: BEIR results. The SotA row is copied from Thakur et al. (2021), and the numbers represent the best model for each dataset.
Method	EntityQuestions
R@20	R@100
BM25 (Chen et al., 2021)	71.2	79.7
DPR (Chen et al., 2021)	49.7	63.4
DPR (192 dim)	47.1	60.6
DrBoost (160 dim)	51.2	63.4
Table 4: Entity Questions Results.
where  $\mu_{c}$  is the average norm of passage embeddings and the operator  $\max \{k\}$  returns the  $k$ -th maximum element in the set. For a fixed  $q_{i}$  and  $k = 1$ , this definition is identical to the classical margin definition. Figure 2 plots the  $50^{th}$ ,  $75^{th}$  and  $90^{th}$  percentiles of the top-20 margin for DrBoost on the NQ training set. We clearly see that margins indeed increase at each step, especially for cases that the model is confident in (high margin). We hypothesize this property to be the main reason for the strong in-distribution generalization of DrBoost that we observed, and potentially also for the surprisingly strong IVF results, since wide margins should intuitively make clustering easier as well.
5.3 Cross-domain generalization
It has been observed in previous work (Thakur et al., 2021) that dense retrievers still largely lag behind sparse retrievers in terms of generalization capabilities. We are interested to test whether our method could be beneficial for out-of-domain transfer as well. We show the results for zero-shot transfer on a subset of the BEIR benchmark in Table 3 and the EntityQuestions dataset in Table 4. While DrBoost improves slightly over the dimension-matched baseline on EntityQuestions, where the passage corpora stays the same, it produces worse results on the BEIR datasets. We conclude that boosting is not especially useful for cross-domain transfer, and should be combined with other methods if this is a concern.
5.4 Representation Probing
One of the hypothesis we formulate for the stronger performance of DrBoost over DPR is that the for
Figure 3: Topic classification accuracy when probing DrBoost and DPR representations with an SVM.
mer might better capture topical information of passages and questions. To test this, we collected topics for all Wikipedia articles in Natural Questions using the strategy of Johnson et al. (2021) and associate them with both passages and questions. We then probed both DPR and DrBoost representations with an SVM (Steinwart and Christmann, 2008) classifier considering a 5-fold cross-validation over 500 instances and 8 different seeds. Results (in Figure 3) confirms our hypothesis: the topic classifier accuracy is higher with DrBoost representations with respect to DPR ones of the same dimension (i.e., 192), for both questions and passages.
5.5 Related Work
Boosting for retrieval Boosting has been studied in machine learning for over three decades (Kearns and Valiant, 1989; Schapire, 1990). Models such as AdaBoost (Freund and Schapire, 1997) and GBMs (Friedman, 2001) became popular approaches to classification problems, with implementations such as XGBoost still popular today (Chen and Guestrin, 2016). Many boosting approaches have been proposed for retrieval and learning-to-rank (LTR) problems, typically employing decision trees, such as AdaRank (Xu and Li, 2007), RankBoost (Freund et al., 2003) and lambdaMART (Wu et al., 2009). Apart from speed and accuracy, boosting is attractive due to promising theoretical properties such as convergence and generalization. (Bartlett et al., 1998; Freund et al., 2003; Mohri et al., 2012). Boosted decision trees have recently been demonstrated to be competitive on LTR tasks (Qin et al., 2021), but, in recent years, boosting approaches have generally received less attention, as (pretrained) neural models began to dominate much of the literature. However, modern neural models and boosting techniques need not be exclusive, and a small amount of work exploring
boosting in the context of modern pre-trained neural models has been carried out (Huang et al., 2020; Qin et al., 2021). Our work follows this line of thinking, identifying dimensionally-constrained biencoders as good candidates as neural weak learners, adopting a simple boosting approach which allows for simple and efficient MIPS at test time.
Dense Retrieval Sparse, term-based Retrievers such as BM25 (Robertson and Zaragoza, 2009) have dominated retrieval until recently. Dense, MIPS-based Retrieval using bi-encoder architectures leveraging contrastive training with gold pairs (Yih et al., 2011) has recently shown to be effective in several settings (Lee et al., 2019; Karpukhin et al., 2020; Reimers and Gurevych, 2019; Hofstätter et al., 2021b). See Yates et al. (2021) for a survey. The success of Dense Retrieval has led to many recent papers proposing schemes to improve dense retriever training by innovating on how negatives are sampled (Xiong et al., 2020; Qu et al., 2021; Zhan et al., 2021c; Lin et al., 2021, inter alia.), and/or proposing pretraining objectives (Oğuz et al., 2021; Guu et al., 2020; Chang et al., 2020; Sachan et al., 2021; Gao and Callan, 2021). Our work also innovates on how dense retrievers are trained, but is arguably orthogonal to most of these training innovations, since these could still be employed when training each component weak learner.
Distillation We leverage a simple distillation technique to make DrBoost more efficient at test time. Distillation for dense retrievers is an active area, and more complex schemes exist which could improve results further (Izacard and Grave, 2021; Qu et al., 2021; Yang and Seo, 2020; Lin et al., 2021; Hofstätter et al., 2021a; Barkan et al., 2020; Gao et al., 2020).
Multi-vector Retrievers Several approaches represent passages with multiple vectors. Humeau et al. (2020) represent queries with multiple vectors, but retrieval is comparatively slow as relevance cannot be calculated with a single MIPS call. ME-BERT (Luan et al., 2021) index a fixed number of vectors for each passage and ColBERT (Khattab and Zaharia, 2020) index a vector for every word. Both can perform retrieval with a single MIPS call (although ColBERT requires reranking) but produce very large indices, which, in turn, slows down search. DrBoost can also be seen as a multi-vector approach, with each weak learner produc
ing a vector. However, each vector is small, and we index concatenated vectors, rather than indexing each vector independently, leading to small indices and fast search. This said, adapting DrBoost-style training to these settings would be feasible. SPAR (Chen et al., 2021) is a two-vector method: one from a standard dense retriever, and the other from a more lexically-oriented model. SPAR uses a similar test-time MIPS retrieval strategy to ours, and SPAR's lexical embeddings could be trivially added to DrBoost as an additional subvector.
Efficient retrievers There have been several recent efforts to build more efficient retrieval and question answering systems (Min et al., 2021). Izacard et al. (2020) and Yang and Seo (2021) experiment with post-hoc compression and lower-dimensional embeddings, Lewis et al. (2021) index and retrieve question-answer pairs and Yamada et al. (2021) propose BPR, which approximates MIPS using binary vectors. There is also a line of work learning embeddings specifically suited for approximate search (Yu et al., 2018; Zhan et al., 2021a,b) Generative retrievers (De Cao et al., 2021) can also be very efficient. DrBoost also employs lower-dimensional embeddings and off-the-shelf post-hoc compression for its smallest index.
In this work we have explored boosting in the context of dense retrieval, inspired by the similarity of iteratively-sampling negatives to boosting. We find that our simple boosting approach, DrBoost, performs largely on par with a 768-dimensional DPR baseline, but produces more compact vectors, and is more amenable to approximate search. We note that DrBoost requires maintaining more neural models at test time, which may put a greater demand on GPU resources. However, the models can be run in parallel if latency is a concern, and if needed, these models can be distilled into a single model with little drop in accuracy. We hope that future work will build on boosting approaches for dense retrieval, including adding adaptive weights, and investigating alternative losses and sampling techniques. We also suggest that emphasis in dense retrieval should be placed on more holistic evaluation than just exact retrieval accuracy, demonstrating that models with quite similar exact retrieval can perform very differently under practically-important approximate search settings.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A Human Generated MAchine Reading CComprehension Dataset. arXiv:1611.09268 [cs]. ArXiv: 1611.09268.
Oren Barkan, Noam Razin, Itzik Malkiel, Ori Katz, Avi Caciularu, and Noam Koenigstein. 2020. Scalable Attentive Sentence Pair Modeling via Distilled Sentence Embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3235-3242. Number: 04.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E. Schapire. 1998. Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651 - 1686. Publisher: Institute of Mathematical Statistics.
Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training Tasks for Embedding-based Large-scale Retrieval. arXiv:2002.03932 [cs, stat]. ArXiv:2002.03932.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pages 785-794, New York, NY, USA. ACM. Event-place: San Francisco, California, USA.
Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2021. Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One? arXiv:2110.06918 [cs]. ArXiv: 2110.06918.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive Entity Retrieval. In International Conference on Learning Representations.
Yoav Freund, Raj Iyer, Robert E. Schapire, and Yoram Singer. 2003. An Efficient Boosting Algorithm for Combining Preferences. J. Mach. Learn. Res., 4(null):933-969. Publisher: JMLR.org.
Yoav Freund and Robert E. Schapire. 1997. A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. Journal of Computer and System Sciences, 55(1):119-139.
Jerome H. Friedman. 2001. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5):1189-1232. Publisher: Institute of Mathematical Statistics.
Luyu Gao and Jamie Callan. 2021. Condenser: a Pretraining Architecture for Dense Retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981-993, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020. Understanding BERT Rankers Under Distillation. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval, IC-TIR '20, pages 149-152, New York, NY, USA. Association for Computing Machinery.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasapat, and Ming-Wei Chang. 2020. Retrieval Augmented Language Model Pre-Training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2021a. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 [cs]. ArXiv:2010.02666.
Sebastian Hofstätter, Sheng-Chieh Lin, Zheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021b. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, pages 113-122, New York, NY, USA. Association for Computing Machinery. Event-place: Virtual Event, Canada.
Tongwen Huang, Qingyun She, and Junlin Zhang. 2020. BoostingBERT: Integrating Multi-Class Boosting into BERT for NLP Tasks. arXiv:2009.05959 [cs]. ArXiv: 2009.05959.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. In International Conference on Learning Representations.
Gautier Izacard and Edouard Grave. 2021. Distilling Knowledge from Reader to Retriever for Question Answering. In International Conference on Learning Representations.
Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. 2020. A Memory Efficient Baseline for Open Domain Question Answering. arXiv:2012.15156 [cs]. ArXiv: 2012.15156.
Isaac Johnson, Martin Gerlach, and Diego Sáez-Trumper. 2021. Language-agnostic topic classification for wikipedia. In Companion Proceedings of the Web Conference 2021, pages 594-601.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, pages 1-1.
H Jégou, M Douze, and C Schmid. 2011. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117-128.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
M. Kearns and L. G. Valiant. 1989. Cryptographic Limitations on Learning Boolean Formulae and Finite Automata. In Proceedings of the Twenty-First Annual ACM Symposium on Theory of Computing, STOC '89, pages 433-444, New York, NY, USA. Association for Computing Machinery. Event-place: Seattle, Washington, USA.
Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 39-48, Virtual Event China. ACM.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics, 7:452-466.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Roektäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.