SlowGuess/NAACL_2022_OCR huggingface dataset:
pages 6648-6662, Online. Association for Computational Linguistics.
Devendra Singh Sachan, Siva Reddy, William L. Hamilton, Chris Dyer, and Dani Yogatama. 2021b. End-to-end training of multi-document reader and retriever for open-domain question answering. In Advances in Neural Information Processing Systems.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6138-6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Minjoon Seo, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2018. Phrase-indexed question answering: A new challenge for scalable document comprehension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 559-564, Brussels, Belgium. Association for Computational Linguistics.
Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4430-4441, Florence, Italy. Association for Computational Linguistics.
Yuval Varkel and Amir Globerson. 2020. Pre-training mention representations in coreference models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8534-8540, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008.
Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 question answering track. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC'00), Athens, Greece. European Language Resources Association (ELRA).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing:
System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.
Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, page 1253-1256, New York, NY, USA. Association for Computing Machinery.
Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible ranking baselines using lucene. J. Data and Information Quality, 10(4).
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7170-7186, Online. Association for Computational Linguistics.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations.
A Baselines: Further Details
BM25 (Robertson and Zaragoza, 2009) A sparse bag-of-words model that extends TF-IDF (i.e. reward rare terms that appear in both  $q$  and  $p$ ) by accounting for document length and term frequency saturation.
BERT (Devlin et al., 2019) was pretrained on two self-supervised tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). We evaluate BERT only in the supervised setting, namely as a backbone for fine-tuning, similar to DPR.
ICT (Lee et al., 2019) A dual-encoder model which was pretrained on the Inverse Cloze Task. Given a batch of passages, ICT masks a sentence from each passage, and trains to predict what is the source passage for each sentence. ICT encourages lexical matching by keeping the sentence in the original passage with low probability. Note that unlike our approach, ICT is trained to produce representations to corrupted passages. In addition, we
Model	CuratedTREC	SQuAD	EntityQuestions
Top-5	Top-20	Top-100	Top-5	Top-20	Top-100	Top-5	Top-20	Top-100
Supervised Models
DPR-Single	66.6	81.7	89.9	40.8	58.4	74.9	38.1	49.7	63.2
DPR-Multi	80.0	89.2	93.9	35.6	52.0	67.8	44.7	56.7	70.0
DPR-Single + BM25	75.8	87.0	93.8	66.9	77.9	86.3	61.1	71.7	81.3
DPR-Mutli + BM25	84.7	90.3	95.4	58.5	72.1	83.0	63.2	73.3	82.6
Unsupervised Models
BM25	64.6	81.1	90.3	57.5	71.2	82.0	61.0	71.4	80.0
ICT*	-	-	-	26.5	45.1	65.2	-	-	-
Condenser	9.9	20.2	34.4	6.1	13.2	25.3	1.0	2.7	7.6
CoCondenser	11.7	22.5	39.3	8.5	16.5	28.8	0.5	1.4	8.7
MSS*	-	-	-	33.9	51.3	68.4	-	-	-
Spider	65.9	82.6	92.8	43.6	61.0	76.0	54.5	66.3	77.4
Spider + BM25	74.5	86.5	93.9	60.9	74.6	84.5	65.4	75.0	82.6
Table 5: Results for an evaluation setup as in Table 1 for the remaining datasets. Top-  $k$  retrieval accuracy (i.e.,the percentage of questions for which the answer is present in the top-  $k$  passages) for supervised and unsupervised approaches. DPR-Single is trained on the corresponding dataset only. We mark in bold the best unsupervised method for each dataset. *Results shared with us by the authors of Sachan et al. (2021a,b).
encourage lexical matching of individual terms in the query, rather than the entire query as ICT.
Sachan et al. (2021a) trained their own ICT model, which shows stronger performance than Lee et al. (2019). The authors shared new results with us, in which TREC and EntityQs are missing. Since their model is not public, for fine-tuning we use the model trained by Lee et al. (2019).
Condenser & CoCondenser (Gao and Callan, 2021a,b) Condenser is an architecture that aims to produce dense sequence-level (i.e. sentences and passages) representations via a variant of the MLM pretraining task. Specifically, to predict a masked token  $x_{t}$ , they condition the prediction on two representations: (1) a representation of  $x_{t}$  from an earlier layer in the encoder, and (2) a dense sequence-level representation of the [CLS] token at the last layer of the network. CoCondenser adds a "corpus-aware" loss alongside MLM to create better embeddings by sampling two sub-spans from each sequence and train in a contrastive fashion.
MSS (Sachan et al., 2021a,b) An unsupervised model in which a dense retriever and a reader are trained jointly end-to-end. First, salient spans (e.g. entities) are identified using a NER model. Then, some of them are masked. The training objective is to predict these missing spans while using retrieved documents as evidence. Due to the latent nature of the retrieval process in this model, its training is substantially more expensive than contrastive learning. In addition, it requires frequent updates
of the encoded evidence corpus.
Contriever (Izacard et al., 2021) A contemporary work. Contriever is an unsupervised dense model trained in a contrastive fashion, using random cropping to generate two views of a given input.
DPR (Karpukhin et al., 2020) A supervised model for ODQA based on dual-encoders and trained in a contrastive fashion (see Section 2). All DPR models considered in the paper are initialized with a BERT-base encoder, and trained on full datasets: DPR-Single models are trained on a single dataset, and are also referred to as DPR- $x$ , where  $x$  is the name of the dataset. DPR-Multi was trained onNQ, TriviaQA, WQ and TREC. For DPR-NQ and DPR-Multi, we use the checkpoints released by the authors. We re-train the other DPR-Single models (which were not made public) using the same hyper-parameters as Karpukhin et al. (2020). We do not train a DPR model on EntityQs. The models we trained are consistent with the results of Karpukhin et al. (2020), except for DPR-SQuAD, where we did not manage to reproduce the original results.
Table 5 and Table 6 show the top-  $k$  accuracy for the unsupervised setting (complements Table 1) for additional datasets. Table 7 shows the top-100 accuracy for the cross-dataset setting (complements Table 2).
Model	NQ (No Overlap)	TriviaQA (No Overlap)	WQ (No Overlap)
Top-5	Top-20	Top-100	Top-5	Top-20	Top-100	Top-5	Top-20	Top-100
Supervised Models
DPR-Single	54.5	68.7	76.8	48.0	56.4	62.6	47.2	60.0	71.1
DPR-Multi	54.2	68.8	77.1	46.9	55.8	62.4	48.0	60.5	70.4
DPR-Single + BM25	60.9	74.0	81.1	53.4	60.2	64.9	50.5	62.4	72.8
DPR-Mutli + BM25	61.7	73.7	80.9	54.6	60.9	65.2	53.3	63.8	73.0
Unsupervised Models
BM25	38.8	55.5	70.1	47.3	56.0	62.4	35.4	53.6	66.4
ICT*	27.6	44.1	58.8	26.2	38.5	51.5	19.4	33.2	52.5
Condenser	6.5	13.3	26.6	1.8	4.2	9.1	9.0	19.3	30.8
CoCondenser	23.8	36.8	52.4	4.6	8.0	13.8	21.3	38.3	54.2
MSS*	33.2	49.7	66.1	36.2	47.9	58.0	19.7	36.9	54.1
Spider	44.3	60.7	73.7	45.2	55.6	62.7	38.1	54.8	69.7
Spider + BM25	49.3	65.2	77.6	51.9	59.4	64.6	42.9	58.4	72.0
Table 6: Top-  $k$  retrieval accuracy (i.e., the percentage of questions for which the answer is present in the top-  $k$  passages) on the "no-answer-overlap" portion of the test sets of three datasets (Lewis et al., 2021a) for supervised and unsupervised approaches. DPR-Single is trained on the corresponding dataset only. We mark in bold the best unsupervised method for each dataset. *Results shared with us by the authors of Sachan et al. (2021a,b).
Model	# Examples	NQ	TriviaQA	WQ	TREC	SQuAD	EntityQs
DPR-NQ	58,880	-	78.7	78.3	92.1	65.2	63.2
DPR-TriviaQA	60,413	79.7	-	81.2	93.7	71.1	74.6
DPR-WQ	2,474	72.6	77.9	-	90.8	67.6	70.2
DPR-TREC	1,125	71.0	76.0	74.6	-	65.3	61.1
DPR-SQuAD	70,096	65.1	75.6	72.9	89.5	-	49.3
DPR-Multi	122,892	-	-	-	-	67.8	70.0
BM25	None	78.3	83.2	75.5	90.3	82.0	80.0
ICT	None	66.8	73.6	65.7	-	65.2	-
Spider	None	81.2	83.5	79.7	92.8	76.0	77.4
Spider-NQ	58,880	-	83.7	82.5	94.1	72.8	74.1
Spider-TriviaQA	60,413	85.0	-	83.3	95.4	80.6	81.4
C Limitations & Risks
We point to several limitations and potential risks of Spider. First, there is still a gap in performance between supervised and unsupervised models, as can be observed in Table 1. Second, self-supervised pretraining is heavier in terms of compute than standard supervised training like DPR. Third, Spider was trained on data solely from Wikipedia, which might hurt its performance when applied to other domains. Last, our model may introduce biases as other pretrained language models, e.g. against under-represented groups.
Table 8 shows the number of examples in each of the datasets used in our evaluation suite.
Table 7: Results for an evaluation setup as in Table 2, measured by top-100 retrieval accuracy in a cross-dataset "zero-shot" setting, where models are evaluated against datasets not seen during their training. DPR- $x$  is a model trained on the full dataset  $x$ , and DPR-Multi was trained on NQ, TriviaQA, WQ and TREC. # Examples is the number of labeled examples used to train the model.
Dataset	Train	Test
Natural Questions	58,880	3,610
TriviaQA	60,413	11,313
WebQuestions	2,474	2,032
CuratedTREC	1,125	694
SQuAD	70,096	10,570
EntityQs	-	22,075
No (Answer) Overlap Datasets
Natural Questions	-	1,313
TriviaQA	-	3,201
WebQuestions	-	856
Table 8: Dataset statistics: number of training and test examples in each dataset.