SlowGuess/NAACL_2022_OCR huggingface dataset:
Ori Ram Gal Shachaf Omer Levy Jonathan Berant Amir Globerson
Blavatnik School of Computer Science, Tel Aviv University ori.ram@cs.tau.ac.il
Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our "recurring span retrieval" approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseudo queries and relevant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreover, a hybrid retriever over Spider and BM25 improves over both, and is often competitive with DPR models, which are trained on tens of thousands of examples. Last, notable gains are observed when using Spider as an initialization for supervised training. $^{1}$
State-of-the-art models for retrieval in open domain question answering are based on learning dense text representations (Lee et al., 2019; Karpukhin et al., 2020; Qu et al., 2021). However, such models rely on large datasets of question-passage pairs for training. These datasets are expensive and sometimes even impractical to collect (e.g., for new languages or domains), and models trained on them often fail
Figure 1: Top- $k$  retrieval accuracy of various unsupervised methods (solid lines) on the test set of Natural Questions (NQ). DPR (dotted) is supervised (trained on NQ) and given for reference.
to generalize to new question distributions (Sciavolino et al., 2021; Reddy et al., 2021).
The above difficulty motivates the development of retrieval models that do not rely on large annotated training sets, but are instead trained only on unlabeled text. Indeed, self-supervision for retrieval has gained considerable attention recently (Lee et al., 2019; Guu et al., 2020; Sachan et al., 2021a; Fan et al., 2021). However, when applied in a "zero-shot" manner, such models are still outperformed by sparse retrievers like BM25 (Robertson and Zaragoza, 2009) and by supervised models (see Sachan et al. 2021a). Moreover, models like REALM (Guu et al., 2020) and MSS (Sachan et al., 2021a,b) that train a retriever and a reader jointly (i.e. in an end-to-end fashion), treating retrieval as a latent variable, outperform contrastive models like ICT (Lee et al., 2019), but are much more computationally-intensive.
In this work we introduce Spider (Span-based unsupervised dense retriever), a dense model pretrained in a contrastive fashion from self
1Our code and models are publicly available: https://github.com/oriram/spider, and: https://huggingface.co/tau/spider
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2687 - 2700 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 2: An example of our pretraining approach: Given a document  $\mathcal{D}$  (e.g. the article "Aaron" in Wikipedia), we take two passages that contain a recurring span  $S$ . One of them is transformed into a short query (left)  $q^{\prime}$  using a random window surrounding  $S$ , in which  $S$  is either kept (top) or removed (bottom). The second passage is then considered the target for retrieval  $p^+$ , while a random passage from  $\mathcal{D}$  that does not contain  $S$  is considered the negative  $p^-$  (right). Each batch is comprised of multiple such examples, and the pretraining task is to select the passage  $p_i^+$  for each query  $q_i^\prime$  (solid line) from the passages of all examples (in-batch negatives; dashed lines).
supervision only (Bhattacharjee et al., 2022), which achieves retrieval accuracy that significantly improves over unsupervised methods (both contrastive and end-to-end), and is much cheaper to train compared to end-to-end models.
Spider is based on a novel self-supervised scheme: recurring span retrieval. We leverage recurring spans in different passages of the same document (e.g. "Yoko Ono" in Figure 2) to create pseudo examples for self-supervised contrastive learning, where one of the passages containing the span is transformed into a short query that (distantly) resembles a natural question, and the other is the target for retrieval. Additionally, we randomly choose whether to keep or remove the recurring span from the query to explicitly model cases where there is substantial overlap between a question and its target passage, as well as cases where such overlap is small.
We evaluate Spider on several ODQA benchmarks. Spider narrows the gap between unsupervised dense retrievers and DPR on all benchmarks (Figure 1, Table 1), outperforming all contrastive and end-to-end unsupervised models in top-5 & top-20 accuracy consistently across datasets. Furthermore, we demonstrate that Spider and BM25 are complementary, and that applying their simple combination (Ma et al., 2021) improves retrieval accuracy over both, sometimes outperforming a supervised DPR model.
We further demonstrate the utility of Spider as an off-the-shelf retriever via cross-dataset evaluation (i.e., when supervised models are tested against datasets which they were not trained on), a setting that often challenges dense retrievers (Sciavolino et al., 2021; Reddy et al., 2021). In this setting, Spider is competitive with supervised dense retrievers trained on an abundance of training examples.
Last, Spider significantly outperforms other pretrained models when used as an initialization towards DPR training, and also shows strong cross-dataset generalization. For example, Spider finetuned on TriviaQA is, to the best of our knowledge, the first dense model to outperform BM25 on the challenging EntityQuestions dataset (Sciavolino et al., 2021).
Taken together, our results demonstrate the potential of pretraining for reducing the reliance of ODQA models on training data.
In open-domain question answering (ODQA), the goal is to find the answer to a given question over a large corpus, e.g. Wikipedia (Voorhees and Tice, 2000; Chen et al., 2017; Chen and Yih, 2020). This task has gained considerable attention following recent advancement in machine reading comprehension, where models reached human parity in extracting an answer from a paragraph given a question (Devlin et al., 2019; Raffel et al., 2020).
Due to the high cost of applying such reading comprehension models, or readers, over the entire corpus, state-of-the-art systems for ODQA first apply an efficient retriever – either sparse (Robertson and Zaragoza, 2009; Chen et al., 2017) or dense (Lee et al., 2019; Karpukhin et al., 2020) – in order to reduce the search space of the reader.
Recently, dense retrieval models have shown promising results on ODQA, even outperforming strong sparse methods that operate on the lexical-level, e.g. BM25. Specifically, the dominant approach employs a dual-encoder architecture, where documents and questions are mapped to a shared continuous space such that proximity in that space represents the relevance between pairs of documents and questions. Formally, let  $\mathcal{C} = \{p_1,\dots,p_m\}$  be a corpus of passages. Each passage  $p\in \mathcal{C}$  is fed to a passage encoder  $E_{P}$  such that  $E_P(p)\in \mathbb{R}^d$ . Similarly, the question encoder  $E_{Q}$  is defined such that the representation of a question  $q$  is given by  $E_{Q}(q)\in \mathbb{R}^{d}$ . Then, the relevance of a passage  $p$  for  $q$  is given by:
$$
s (q, p) = E _ {Q} (q) ^ {\top} E _ {P} (p).
$$
Given a question  $q$ , the retriever finds the top- $k$  candidates with respect to  $s(q, \cdot)$ , i.e.  $\operatorname{top} - k_{p \in \mathcal{C}} s(q, p)$ . In order to perform this operation efficiently at test time, a maximum-inner product search (MIPS) index (Johnson et al., 2021) is built over the encoded passages  $\{E_P(p_1), \dots, E_P(p_m)\}$ .
While considerable work has been devoted to create pretraining schemes for dense retrieval (Lee et al. 2019; Guu et al. 2020; inter alia), it generally assumed access to large training datasets after pretraining. In contrast, we seek to improve dense retrieval in the challenging unsupervised setting.
Our contribution towards this goal is twofold. First, we construct a self-supervised pretraining method based on recurring spans across passages in a document to emulate the training process of dual-encoders for dense retrieval. Our pretraining is simpler and cheaper in terms of compute than end-to-end models like REALM (Guu et al., 2020) and MSS (Sachan et al., 2021a). Second, we demonstrate that a simple combination of BM25 with our models leads to a strong hybrid retriever that rivals the performance of models trained with tens of thousands of examples.
3 Our Model: Spider
We now describe our approach for pretraining dense retrievers, which is based on a new self-supervised task (Section 3.1). Our pretraining is based on the notion of recurring spans (Ram et al., 2021) within a document: given two paragraphs with the same recurring span, we construct a query from one of the paragraphs, while the other is taken as the target for retrieval (Figure 2). Other paragraphs in the document that do not contain the recurring span are used as negative examples. We train a model from this self-supervision in a contrastive fashion.
Since sparse lexical methods are known to complement dense retrieval (Luan et al., 2021; Ma et al., 2021), we also incorporate a simple hybrid retriever (combining BM25 and Spider) in our experiments (Section 3.2).
3.1 Pretraining: Recurring Span Retrieval
Given a document  $\mathcal{D} \subset \mathcal{C}$  with multiple passages (e.g. an article in Wikipedia), we define cross-passage recurring spans in  $\mathcal{D}$  as arbitrary n-grams that appear more than once and in more than one passage in  $\mathcal{D}$ . Let  $S$  be a cross-passage recurring span in  $\mathcal{D}$ , and  $\mathcal{D}_S \subset \mathcal{D}$  be the set of passages in the document that contain  $S$ , so  $|\mathcal{D}_S| > 1$  by definition. First, we randomly choose a query passage  $q \in \mathcal{D}_S$ . In order to resemble a natural language question, we apply a heuristic query transformation  $T$ , which takes a short random window from  $q$  surrounding  $S$  to get  $q' = T(q)$  (described in detail below).
Similar to DPR, each query has one corresponding positive passage  $p^+$  and one corresponding negative passage  $p^-$ . For  $p^+$ , we sample another random passage from  $\mathcal{D}$  that contains  $S$  (i.e.  $p^+ \in \mathcal{D}_S \setminus \{q\}$ ). For  $p^-$ , we choose a passage from  $\mathcal{D}$  that does not contain  $S$  (i.e.  $p^- \in \mathcal{D} \setminus \mathcal{D}_S$ ). The article title is prepped to both passages (bot not to the query).
Figure 2 illustrates this process. We focus on the first example (in orange), which is comprised of three passages from the Wikipedia article "Aaron". The span "the priesthood for himself and his male descendants" appears in two passages in the article. One of the passages was transformed into a query (denoted by  $q_1'$ ), while the other  $(p_1^+)$  is taken as a positive passage. Another random passage from the article  $(p_1^-)$  is considered its negative.
As the example demonstrates, existence of recurring spans in two different passages often implies
semantic similarity between their contexts.
Query Transformation As discussed above, after we randomly choose a query passage  $q$  (with a recurring span  $S$ ), we apply a query transformation on  $q$ . The main goal is to make the queries more "similar" to open-domain questions (e.g. in terms of lengths).
First, we define the context to keep from  $q$ . Since passages are much longer than typical natural questions, we take a random window containing  $S$ . The window length  $\ell$  is chosen uniformly between 5 and 30 to resemble questions of different lengths. The actual window is then chosen at random from all possible windows of length  $\ell$  that contain  $S$ .
Second, we randomly choose whether to keep  $S$  in  $q'$  or remove it. This choice reflects two complementary skills for retrieval – the former requires lexical matching (as  $S$  appears in both  $q'$  and  $p^+$ ), while the latter intuitively encourages semantic contextual representations.
The queries in Figure 2 (left) demonstrate this process. In the top query, the recurring span "the priesthood for himself and his male descendants" was kept as is. In the bottom query, the span "Yoko Ono" was removed.
Span Filtering To focus on meaningful spans with semantically similar contexts, we apply several filters on recurring spans. First, we adopt the filters from Ram et al. (2021): (1) spans only include whole words, (2) only maximal spans are considered, (3) spans that contain only stop words are filtered out, (4) spans contain up to 10 tokens. In addition, we add another filter: (5) spans should contain at least 2 tokens. Note that in contrast to methods based on salient spans (Glass et al., 2020; Guu et al., 2020; Roberts et al., 2020; Sachan et al., 2021a,b), our filters do not require a trained model.
Training At each time step of pretraining, we take a batch of  $m$  examples  $\{(q_i', p_i^+, p_i^-)\}_{i=1}^m$ , and optimize the cross-entropy loss with respect to the positive passage  $p_i^+$  for each query  $q_i'$  in a contrastive fashion (i.e., with in-batch negatives), similar to Karpukhin et al. (2020):
$$
- \log \frac {\exp \left(s (q _ {i} ^ {\prime} , p _ {i} ^ {+})\right)}{\sum_ {j = 1} ^ {m} \left(\exp \left(s (q _ {i} ^ {\prime} , p _ {j} ^ {+})\right) + \exp \left(s (q _ {i} ^ {\prime} , p _ {j} ^ {-})\right)\right)}
$$
3.2 Hybrid Dense-Sparse Retrieval
It is well established that the strong lexical matching skills of sparse models such as BM25 (Robertson and Zaragoza, 2009) are complementary to dense representation models. Ma et al. (2021) demonstrated strong improvements by using hybrid dense-sparse retrieval, based on BM25 and DPR. Specifically, they define the joint score of a hybrid retriever via a linear combination of the scores given by the two models, i.e.  $s_{\mathrm{hybrid}}(q,p) = s(q,p) + \alpha \cdot \mathrm{BM25}(q,p)$ . They tune  $\alpha$  on a validation set of each of the datasets. A similar approach was considered by Luan et al. (2021). Since tuning hyperparameters is unrealistic in our settings, we simply set  $\alpha = 1.0$  for all hybrid models. Thus, we define:
$$
s _ {\mathrm {h y b r i d}} (q, p) = s (q, p) + \mathrm {B M} 2 5 (q, p)
$$
We adopt the normalization technique from Ma et al. (2021). We begin by fetching the top- $k'$  (where  $k' > k$ ) passages from each of the models. If a passage  $p$  is found in the top- $k'$  of a dense retriever but not of BM25, then  $\mathrm{BM25}(q,p)$  is set to the minimum value from the top- $k'$  results of BM25 (and vice versa).
To evaluate how different retrievers work on different settings and given different amounts of supervision, we simulate various scenarios by using existing datasets, with an emphasis on the unsupervised setting.
We evaluate our method on six datasets commonly used in prior work, all over Wikipedia: Natural Questions (NQ; Kwiatkowski et al. 2019), TriviaQA (Joshi et al., 2017), WebQuestions (WQ; Berant et al. 2013), CuratedTREC (TREC; Baudiš and Šedivý 2015), SQuAD (Rajpurkar et al., 2016) and EntityQuestions (EntityQs; Sciavolino et al. 2021). The datasets vary significantly in the distribution of questions and the size of training data.
Lewis et al. (2021a) showed that there exists a significant overlap between train and test questions in ODQA datasets, which poses an issue in our case: supervised models can memorize training questions while unsupervised methods cannot. Thus, we also report the results on the "no answer overlap" portion of the test sets created by Lewis et al. (2021a) for NQ, TriviaQA and WQ.
2In our case, passages contain 100 words, while Joshi et al. (2017) report an average length of 14 words for questions.
We consider a variety of baselines, including supervised and self-supervised dense models, as well as sparse methods. All dense models share the architecture of BERT-base (namely a transformer encoder; Vaswani et al. 2017), including the number of parameters (110M) and uncased vocabulary. In addition, all pretrained dense models use weight sharing between query and passage encoders (only during pretraining).  $E_{Q}(q)$  and  $E_{P}(p)$  are defined as the representation of the [CLS] token. Similar to Gao and Callan (2021a), we do not consider the models trained in Chang et al. (2020), as they rely on Wikipedia links, and were not made public.
We now list our baselines (see App. A for further details). As a sparse baseline model, we follow prior work and take BM25 (Robertson and Zaragoza, 2009). We consider several unsupervised dense retrieval models: ICT (Lee et al., 2019; Sachan et al., 2021a), Condenser and CoCondenser (Gao and Callan, 2021a,b). We also compare our approach with an unsupervised model trained in an end-to-end fashion (i.e. jointly with a reader): Masked Salient Spans (MSS; Sachan et al. 2021a,b). In addition, we add the results of the unsupervised Contriever model (Izacard et al., 2021), a contemporary work. Last, we add results of DPR (Karpukhin et al., 2020), a supervised model, for reference.
4.3 Evaluation Settings
We evaluate our method and baselines in a broad range of scenarios. We report top- $k$  retrieval accuracy, i.e. the percentage of questions for which the answer span is found in the top- $k$  passages.
Unsupervised Setting Models are trained only on unlabeled data, and evaluated on all datasets without using any labeled examples (i.e. in a zero-shot mode). As a reference point, we also compare to DPR, which is supervised.
Cross-Dataset Generalization To test the robustness of different models across datasets, we compare Spider to DPR models tested on datasets they were not trained on. The motivation behind these experiments is to determine the quality of all models as "off-the-shelf" retrievers, namely on data from unseen distributions of questions.
Supervised Setting We compare Spider to other pretrained models for retrieval when fine-tuned on different amounts of training examples, similar to Karpukhin et al. (2020). Specifically, we consider the settings where 128 examples, 1024 examples and full datasets are available. We restrict these experiments to NQ and TriviaQA due to the high cost of running them for all datasets and baselines.
We base our implementation on the official code of DPR (Karpukhin et al., 2020), which is built on Hugging Face Transformers (Wolf et al., 2020).
Passage Corpus We adopt the same corpus and preprocessing as Karpukhin et al. (2020), namely the English Wikipedia dump from Dec. 20, 2018 (following Lee et al. 2019) with blocks of 100 words as retrieval units. Preprocessing (Chen et al., 2017) removes semi-structured data (e.g., lists, infoboxes, tables, and disambiguation pages), resulting in roughly 21 million passages. This corpus is used for both pretraining and all downstream experiments.
Pretraining We train Spider for 200,000 steps, using batches of size 1024. similar to ICT and Condenser, the model is initialized from the uncased BERT-base model, and weight sharing between the passage and query encoders is applied. Each pseudo-query has one corresponding positive example and one negative example.4 Overall, the model is expected to predict the positive passage out of a total of 2048 passages.5 The learning rate is warmed up along the first  $1\%$  of the training steps to a maximum value of  $2 \cdot 10^{-5}$ , after which linear decay is applied. We use Adam (Kingma and Ba, 2015) with its default hyperparameters as our optimizer, and apply a dropout rate of 0.1 to all layers. We utilize eight 80GB A100 GPUs for pretraining, which takes roughly two days. In our ablation study (see Section 5.4), we lower the learning rate to  $10^{-5}$  and the batch size to 512 in order to fit in eight Quadro RTX 8000 GPUs.6 Each ablation takes two days.
Fine-Tuning For fine-tuning, we use the hyperparameters from Karpukhin et al. (2020), and do
<sup>4</sup>We perform an ablation on this choice in Section 5.4.
In-batch negatives are taken across all GPUs, as suggested in Qu et al. (2021).
One ablation does involve a batch size of 1,024, and was trained using A100 GPUs as well.
3For unsupervised models, this is essentially equivalent to the unsupervised setting.
Model	NQ	TriviaQA	WQ
Top-5	Top-20	Top-100	Top-5	Top-20	Top-100	Top-5	Top-20	Top-100
Supervised Models
DPR-Single	68.3	80.1	86.1	71.2	79.7	85.1	62.8	74.3	82.2
DPR-Multi	67.1	79.5	86.1	69.8	78.9	84.8	64.0	75.1	83.0
DPR-Single + BM25	72.2	82.9	88.3	75.4	82.4	86.5	64.4	75.1	83.1
DPR-Mutli + BM25	71.9	82.6	88.2	76.1	82.6	86.5	67.3	77.2	84.5
Unsupervised Models
BM25	43.8	62.9	78.3	66.3	76.4	83.2	41.8	62.4	75.5
ICT*	32.3	50.6	66.8	40.2	57.5	73.6	25.2	43.4	65.7
Condenser	13.0	25.5	43.4	4.5	9.6	18.5	20.3	35.8	51.9
CoCondenser	28.9	46.8	63.5	7.5	13.8	24.3	30.2	50.7	68.7
MSS*	41.7	59.8	74.9	53.3	68.2	79.4	29.0	49.2	68.4
Contriever**	47.2	67.2	81.3	59.5	74.2	83.2	-	-	-
Spider	49.6	68.3	81.2	63.6	75.8	83.5	46.8	65.9	79.7
Spider + BM25	55.1	72.1	84.1	71.7	80.0	85.5	51.0	69.1	81.1
Table 1: Top-  $k$  retrieval accuracy (i.e., the percentage of questions for which the answer is present in the top-  $k$  passages) on the test sets of three datasets for supervised and unsupervised approaches. DPR-Single is trained on the corresponding dataset only. We mark in bold the best unsupervised method for each dataset. Further results are given in Tables 5&6. *Results reported in Sachan et al. (2021a,b); **Results reported in Izacard et al. (2021).
not perform any hyperparameter tuning. Specifically, we train using Adam (Kingma and Ba, 2015) with bias-corrected moment estimates (Zhang et al., 2021), and a learning rate of  $10^{-5}$  with warmup and linear decay. We use batch size of 128 for 40 epochs with two exceptions. First, when fine-tuning DPR-WQ and DPR-TREC, we run for 100 epochs for consistency with the original paper. Second, when fine-tuning on 128 examples only, we lower the batch size to 32 and run for 80 epochs. We use BM25 negatives produced by Karpukhin et al. (2020), and do not create hard negatives by the model itself (Xiong et al., 2021).
Retrieval When performing dense retrieval, we apply exact search using FAISS (Johnson et al., 2021). This is done due to the high memory demand of creating an HNSW index for each experiment (Karpukhin et al., 2020). For sparse retrieval (i.e. BM25), we utilize the Pyserini library (Lin et al., 2021), built on top of Anserini (Yang et al., 2017, 2018). For hybrid retrieval, we set  $k' = 1000$  similar to Ma et al. (2021).
Our experiments show that Spider significantly improves performance in the challenging unsupervised retrieval setting, even outperforming strong supervised models in many cases. Thus, it enables
the use of such retrievers when no examples are available. When used for supervised DPR training, we observe significant improvements over the baselines as well. We perform ablation studies that demonstrate the importance of our pretraining design choices.