SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 1 shows the performance of Spider (measured by top-  $k$  retrieval accuracy) compared to other unsupervised baselines on three datasets, without additional fine-tuning. Results for remaining datasets are given in Table 5 and Table 6. Supervised baselines (i.e. DPR) are given for reference. Results demonstrate the effectiveness of Spider w.r.t. other dense pretrained models, across all datasets. For example, the average margin between Spider and ICT is more than 15 points. Moreover, Spider outperforms DPR-Single on three of the datasets (TREC, SQuAD and EntityQs). When DPR is better than our model, the gap narrows for higher values of  $k$ . In addition, it is evident that Spider is able to outperform BM25 in some datasets (NQ, WQ and TREC), while the opposite is true for others (TriviaQA, SQuAD and EntityQuestions). However, our hybrid retriever is able to combine the merits of each of them into a stronger model, significantly improving over both across all datasets. For example, on TriviaQA, Spider and BM25 achieve  $75.8\%$  and  $76.4\%$  top-20 retrieval accuracy, respectively. The hybrid model significantly improves over both
This is done to avoid running on all examples in each step, which might lead to overfitting. However, we did not test this hypothesis.
Model	# Examples	NQ	TriviaQA	WQ	TREC	SQuAD	EntityQs
DPR-NQ	58,880	-	69.0	68.8	85.9	48.9	49.7
DPR-TriviaQA	60,413	67.5	-	71.4	87.9	55.8	62.7
DPR-WQ	2,474	59.4	66.7	-	82.0	52.3	58.3
DPR-TREC	1,125	57.9	64.0	61.7	-	49.4	46.9
DPR-SQuAD	70,096	47.0	60.0	56.0	77.2	-	30.9
DPR-Multi	122,892	-	-	-	-	52.0	56.7
BM25	None	62.9	76.4	62.4	81.1	71.2	71.4
ICT	None	50.6	57.5	43.4	-	45.1	-
Spider	None	68.3	75.8	65.9	82.6	61.0	66.3
Spider-NQ	58,880	-	77.2	74.2	89.9	57.7	61.9
Spider-TriviaQA	60,413	75.5	-	73.7	91.2	68.1	72.9
Table 2: Top-20 retrieval accuracy in a cross-dataset "zero-shot" setting, where models are evaluated against datasets not seen during their training. DPR- $x$  and Spider- $x$  are models trained on the full dataset  $x$ , initialized from BERT and Spider, respectively. DPR-Multi was trained on NQ, TriviaQA, WQ and TREC. # Examples is the number of labeled examples used to train the model. Top-100 retrieval accuracy results are given in Table 7.
models and obtains  $80.0\%$ , better than DPR-Single and DPR-Multi (79.7% and 78.9%, respectively).
Moreover, we observe that Spider consistently surpasses Contriever, with substantial gains for lower values of  $k$ .
5.2 Cross-Dataset Generalization
An important merit of Spider is the fact that a single model can obtain good results across many datasets, i.e. in a "zero-shot" setting. Table 2 demonstrates the results of supervised models in these scenarios, where DPR models are tested on datasets they were not trained on. Spider outperforms four of the six DPR models (DPR-WQ, DPR-TREC, DPR-SQuAD and DPR-Multi) across all datasets. In addition, it significantly outperforms DPR-NQ, which is a widely-used retriever, $^{8}$  on three datasets out of five. Finally, DPR-TriviaQA outperforms Spider on three datasets.
When fine-tuning Spider on NQ and TriviaQA (see Sections 4.3;5.3), the resulting models show strong generalization to other datasets. For example, Spider-NQ outperforms DPR-NQ (initialized from BERT) by 4-12 points. Similar trends are observed for the models trained on TriviaQA. Specifically, Spider-TriviaQA is able to outperform BM25 on EntityQuestions, that is known to challenge dense retrievers (Sciavolino et al., 2021).
5.3 Supervised Setting
Table 3 shows the performance when fine-tuning pretrained models on 128 examples, 1024 examples and full datasets from NQ and TriviaQA. Spider establishes notable gains compared to all other
dense baselines on both datasets and for all training data sizes. When only 128 examples are available, Spider significantly outperforms all other models, with absolute gaps of  $3 - 11\%$  on both datasets. On TriviaQA, Spider fine-tuned on 128 examples is able to outperform all other baselines when they are trained on 1024 examples. Similar trends are observed for the 1024-example setting (absolute gaps of  $1.7 - 6.2\%$ ).
Even though Spider was mainly designed for unsupervised settings, it outperforms other pretrained models in the full dataset as well. On both datasets, Spider obtains the best results, improving over DPR models (initialized from BERT) by  $1.9 - 6.5\%$ .
We perform an ablation study on the query transformation applied on the query passage  $q$ . We then test the contribution of the negative passage  $p^{-}$  to the performance of our model. Last, we scale up both the batch size and the number of pretraining steps.
Choice of Query Transformation During pretraining, we apply a query transformation on the query  $q$ . We sample a random window containing the recurring span  $S$  and either remove or keep  $S$ . We now test the effect of these choices on our model. We consider two more options for the context taken from  $q$ : (1) the whole passage, for which we replace  $S$  with a [MASK] token (as the context is very long, it makes sense to provide the retriever with a signal on what span is sought in the answer), and (2) a prefix of random length preceding  $S$ , for which we always remove  $S$  from the context (as it is in any case, by definition, in the end of  $q'$ ). The
The model was downloaded from Hugging Face model hub 200,000 times during December 2021.
Model	NQ	TriviaQA
Top-1	Top-5	Top-20	Top-100	Top-1	Top-5	Top-20	Top-100
BM25	22.1	43.8	62.9	78.3	46.3	66.3	76.4	83.2
128 examples
BERT	12.7	27.3	43.5	60.6	16.7	33.4	49.4	65.4
ICT	22.8	45.5	64.1	78.3	32.7	54.5	68.9	79.5
Condenser	17.6	36.8	52.7	68.6	26.1	45.9	60.2	73.7
CoCondenser	23.2	47.9	65.2	79.2	36.3	60.1	72.8	81.6
Spider	31.7	57.7	74.3	84.6	47.5	68.5	78.5	85.1
1024 examples
BERT	26.6	49.6	65.3	78.1	32.6	52.7	66.1	77.9
ICT	30.4	55.8	72.4	83.4	38.8	60.0	72.8	82.3
Condenser	30.8	55.1	71.7	82.2	40.7	61.1	72.4	81.2
CoCondenser	32.7	60.1	75.6	84.8	43.3	65.4	76.2	83.6
Spider	37.0	63.0	77.9	86.5	49.5	69.5	79.3	85.5
Full Dataset
BERT	46.3	68.3	80.1	86.1	53.7	71.2	79.7	85.1
ICT	46.4	69.6	80.9	87.6	55.1	72.3	80.4	85.8
Condenser	47.0	70.1	81.4	87.0	57.4	73.4	81.1	86.1
CoCondenser	47.8	70.1	80.9	87.5	58.7	75.0	82.2	86.5
Spider	49.4	72.2	82.4	88.0	60.2	76.1	83.1	87.2
Table 3: Top-  $k$  retrieval accuracy of different pretrained models on the test sets of Natural Questions and TriviaQA, after fine-tuning on various sizes of training data: 128 examples, 1024 examples and the full datasets. All models are fine-tuned using the data produced by Karpukhin et al. (2020), i.e., BM25-based negative examples.
top two rows in Table 4 correspond to these ablations. Indeed, both are inferior to taking a random window surrounding  $S$  (one before the last row).
In addition, we test whether alternating between keeping and removing  $S$  is indeed better than applying only one of them consistently. The third, fourth and fifth rows of Table 4 verify that our motivation was indeed correct: Alternating between the two is superior to each of them on its own.
Effect of Negative Passages During pretraining, each query  $q_{i}^{\prime}$  has one positive passage  $p_{i}^{+}$  and one negative passage  $p_{i}^{-}$ . We pretrain a model without negative passages at all, i.e. the target is to select the positive  $p_{i}^{+}$ , given the positive passages of all other examples  $\{p_{j}^{+}\}_{j = 1}^{m}$ . This model corresponds to the row with # negatives  $= 0$  (i.e. the sixth row in Table 4). As expected, the top-  $k$  retrieval accuracy of the model drops significantly (2-6% for different  $k$  values) with respect to the same model with # negatives  $= 1$  as a result of this choice, which is consistent with Karpukhin et al. (2020).
Scaling up Batch Size and Training Steps We scale up the batch size and observe improvements of  $0.6\mathrm{-}1.2\%$ . We train our model for longer (200K steps instead of 100K), which leads to additional  $1.1\mathrm{-}1.8\%$  improvements (last two rows in Table 4).
Pretraining for dense retrieval has recently gained considerable attention, following the success of self-supervised models in many tasks (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020). While most works focus on fine-tuning such retrievers on large datasets after pretraining (Lee et al., 2019; Chang et al., 2020; Guu et al., 2020; Sachan et al., 2021a; Gao and Callan, 2021a), we attempt to bridge the gap between unsupervised dense models and strong sparse (e.g. BM25; Robertson and Zaragoza 2009) or supervised dense baselines (e.g. DPR; Karpukhin et al. 2020). A concurrent work by Oğuz et al. (2021) presented DPR-PAQ, which shows strong results on NQ after pretraining. However, their approach utilizes PAQ (Lewis et al., 2021b), a dataset which was generated using models trained on NQ, and is therefore not unsupervised.
Leveraging recurring spans for self-supervised pretraining has previously been considered for numerous tasks, e.g. coreference resolution and coreferential reasoning (Kocijan et al., 2019; Varkel and Globerson, 2020; Ye et al., 2020) and question answering (Ram et al., 2021; Bian et al., 2021; Castel et al., 2021). Glass et al. (2020) utilize recurring spans across documents to create pseudo-examples
Query Transformation	Batch Size	NQ (Dev Set)
Context	Recurring Span	# Negs	Size	Steps	Top-1	Top-5	Top-20	Top-100
Whole passage	Replace with a [MASK]	1	512	100,000	17.5	36.8	52.7	67.3
Prefix	Remove	1	512	100,000	18.9	39.7	58.0	72.4
Random window	Remove	1	512	100,000	18.6	39.2	56.8	71.7
Random window	Keep	1	512	100,000	20.3	42.0	61.1	75.9
Random window	Remove / Keep	1	512	100,000	21.5	44.5	62.3	76.2
Random window	Remove / Keep	0	512	100,000	16.3	38.8	58.1	74.1
Random window	Remove / Keep	1	1,024	100,000	22.1	45.4	63.5	77.0
Random window	Remove / Keep	1	1,024	200,000	23.4	46.5	65.3	78.2
Table 4: Ablation study on the development set of Natural Questions. The top rows of the table describe ablations on the query transformation: We first determine the context to take from the query passage, and then decide what operation will be applied on the recurring span. The bottom rows of the table study the contribution of the negative passage  $p^{-}$  (# Negs = 0 stands for no negative examples), as well as scaling up the batch size (i.e. the number of queries at each batch) and the total number of training steps. The last row corresponds to our model Spider.
for QA.
While we focus in this work on dual-encoder architectures, other architectures for dense retrieval have been introduced recently. Luan et al. (2021) showed that replacing a single representation with multiple vectors per document enjoys favorable theoretical and empirical properties. Khattab and Zaharia (2020) introduced late-interaction models, where contextualized representations of query and document tokens are first computed, and a cheap interaction step that models their fine-grained relevance is then applied. Phrase-based retrieval (Seo et al., 2018, 2019) eliminates the need for a reader during inference, as it directly retrieves the answer span given a query. Lee et al. (2021a) demonstrated strong end-to-end ODQA results with this approach, and Lee et al. (2021b) showed that it is also effective for passage retrieval. Our pretraining scheme can be seamlessly used for those architectures as well.
In this work, we explore learning dense retrievers from unlabeled data. Our results demonstrate that existing models struggle in this setup. We introduce a new pretraining scheme for dual-encoders that dramatically improves performance, reaching good results without any labeled examples. Our results suggest that careful design of a pretraining task is important for learning unsupervised models that are effective retrievers for ODQA.
We thank Yuval Kirstain and anonymous reviewers for valuable feedback and discussions, and Deven
dra Singh Sachan for his help with running the ICT and MSS baselines. This project was funded by the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080), the Blavatnik Fund, the Alon Scholarship, the Yandex Initiative for Machine Learning and Intel Corporation.
Petr Baudis and Jan Šedivý. 2015. Modeling of the question answering task in the YodaQA system. In Proceedings of the 6th International Conference on Experimental IR Meets Multilinguality, Multimodality, and Interaction - Volume 9283, CLEF'15, page 222-228, Berlin, Heidelberg. Springer-Verlag.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
Amrita Bhattacharjee, Mansoresh Karami, and Huan Liu. 2022. Text transformations in contrastive self-supervised learning: A review.
Ning Bian, Xianpei Han, Bo Chen, Hongyu Lin, Ben He, and Le Sun. 2021. Bridging the gap between language model and reading comprehension: Unsupervised MRC via self-supervision.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen,
Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems.
Or Castel, Ori Ram, Avia Efrat, and Omer Levy. 2021. How optimal is greedy decoding for extractive question answering?
Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In 8th International Conference on Learning Representations, ICLR 2020.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Danqi Chen and Wen-tau Yih. 2020. Open-domain question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 34-37, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yixing Fan, Xiaohui Xie, Yinqiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li, Ruqing Zhang, and Jiafeng Guo. 2021. Pre-training methods in information retrieval.
Luyu Gao and Jamie Callan. 2021a. Condenser: a pretraining architecture for dense retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981-993, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Luyu Gao and Jamie Callan. 2021b. Unsupervised corpus aware language model pre-training for dense passage retrieval.
Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, and Avi Sil. 2020. Span selection pretraining for question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2773-2782, Online. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Papat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39-48, New York, NY, USA. Association for Computing Machinery.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
Vid Kocijan, Oana-Maria Camburu, Ana-Maria Cretu, Yordan Yordanov, Phil Blunsom, and Thomas Lukasiewicz. 2019. WikiCREM: A large unsupervised corpus for coreference resolution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4303-4312, Hong Kong, China. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.
Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021a. Learning dense representations of phrases at scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6634-6647, Online. Association for Computational Linguistics.
Jinhyuk Lee, Alexander Wettig, and Danqi Chen. 2021b. Phrase retrieval learns passage retrieval, too. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3661-3672, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021a. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000-1008, Online. Association for Computational Linguistics.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Kuttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021b. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9(0):1098-1115.
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Zheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: An easy-to-use python toolkit to support replicable IR research with sparse and dense representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021. Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguistics, 9:329-345.
Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021. A replication study of dense passage retriever.
Barlas Oğuz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus. Xilun Chen, Sebastian Riedel, Wen tau Yih, Sonal Gupta, and Yashar Mehdad. 2021. Domain-matched pre-training tasks for dense retrieval.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835-5847, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3066-3079, Online. Association for Computational Linguistics.
Revanth Gangi Reddy, Vikas Yadav, Md Arafat Sultan, Martin Franz, Vittorio Castelli, Heng Ji, and Avirup Sil. 2021. Towards robust neural retrieval models with synthetic pre-training.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020 How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426. Online. Association for Computational Linguistics.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333-389.
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, and Bryan Catanzaro. 2021a. End-to-end training of neural retrievers for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).