SlowGuess/NAACL_2022_OCR huggingface dataset:
Kexin Wang $^{1}$ , Nandan Thakur $^{2*}$ , Nils Reimers $^{3*}$ , Iryna Gurevych $^{1}$
<sup>1</sup> Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt
2 University of Waterloo, 3 Hugging Face
Dense retrieval approaches can overcome the lexical gap and lead to significantly improved search results. However, they require large amounts of training data which is not available for most domains. As shown in previous work (Thakur et al., 2021b), the performance of dense retrievers severely degrades under a domain shift. This limits the usage of dense retrieval approaches to only a few domains with large training datasets.
In this paper, we propose the novel unsupervised domain adaptation method Generative Pseudo Labeling (GPL), which combines a query generator with pseudo labeling from a cross-encoder. On six representative domainspecialized datasets, we find the proposed GPL can outperform an out-of-the-box state-of-the-art dense retrieval approach by up to 9.3 points nDCG@10.GPL requires less (unlabeled) data from the target domain and is more robust in its training than previous methods.
We further investigate the role of six recent pre-training methods in the scenario of domain adaptation for retrieval tasks, where only three could yield improved results. The best approach, TSDAE (Wang et al., 2021) can be combined with GPL, yielding another average improvement of 1.4 points nDCG@10 across the six tasks. The code and the models are available<sup>1</sup>.
Information Retrieval (IR) is a central component of many natural language applications. Traditionally, lexical methods (Robertson et al., 1994) have been used to search through text content. However, these methods suffer from the lexical gap (Berger et al., 2000) and are not able to recognize synonyms and distinguish between ambiguous words.
Recently, information retrieval methods based on dense vector spaces have become popular to address these challenges. These dense retrieval methods map queries and passages<sup>2</sup> to a shared, dense vector space and retrieve relevant hits by nearest-neighbor search. Significant improvement over traditional approaches has been shown for various tasks (Karpukhin et al., 2020; Xiong et al., 2021). This method is also adapted increasingly by industry to enhance the search functionalities of various applications (Choi et al., 2020; Huang et al., 2020).
However, as shown in Thakur et al. (2021b), dense retrieval methods require a large amount of training data to work well. Most importantly, dense retrieval methods are extremely sensitive to domain shifts: Models trained on MS MARCO perform rather poorly for questions for COVID-19 scientific literature (Wang et al., 2020; Voorhees et al., 2021). The MS MARCO dataset was created before COVID-19, hence, it does not include any COVID-19 related topics and models did not learn how to represent this topic well in a vector space.
In this work, we present Generative Pseudo Labeling (GPL), an unsupervised domain adaptation technique for dense retrieval models (see Figure 1). For a collection of paragraphs from the desired domain, we use an existing pre-trained T5 encoder-decoder to generate synthetic queries. These input passages are viewed as the positive passages for the generated queries. For each generated query, we retrieve the most similar paragraphs as the negative passages using an existing dense retrieval model. We term this step negative mining and term these negative passages hard negatives. Finally, we use an existing cross-encoder to score each (query, passage)-pair and train a dense retrieval
2We use passage to refer to text of any length.
<sup>3</sup>For reference, the popular MS MARCO dataset (Nguyen et al., 2016) has about 500k training instances; the Natural Questions dataset (Kwiatkowski et al., 2019) has more than 100k training instances.
*Contributions made while being employed at UKP Lab.
<https://github.com/UKPLab/gpl>
Human Language Technologies, pages 2345 - 2360
model on these generated, pseudo-labeled queries using MarginMSE-Loss (Hofstätter et al., 2020).
We use publicly available models for query generation, negative mining, and the cross-encoder, which have been trained on the MS MARCO dataset (Nguyen et al., 2016), a large-scale dataset from Bing search logs combined with relevant passages from diverse web sources. We evaluate GPL on six representative domain-specific datasets from the BeIR benchmark (Thakur et al., 2021b). GPL improves the performance by up to 9.3 points nDCG@10 compared to state-of-the-art model trained solely on MS MARCO. Compared to the previous state-of-the-art domain-adaptation method QGen (Ma et al., 2021; Thakur et al., 2021b), GPL improves the performance by up to 4.5 nDCG@10 points. Training with GPL is easy, fast, and data efficient.
We further analyze the role of six recent pretraining methods in the scenario of domain adaptation for retrieval tasks. The best approach is TSDAE (Wang et al., 2021), that outperforms the second best approach, Masked Language Modeling (Devlin et al., 2019) on average by 2.5 points nDCG@10. TSDAE can be combined with GPL, yielding another average improvement of 1.4 point nDCG@10.
Pre-Training based Domain Adaptation. The most common domain adaptation technique for transformer models is domain-adaptive pretraining (Gururangan et al., 2020), which continues pre-training on in-domain data before fine-tuning with labeled data. However, for retrieval it is often difficult to get in-domain labeled data and models are applied in a zero-shot setting on a given corpus. Besides Masked Language Modeling (MLM) (Devlin et al., 2019), different pre-trained strategies specifically for dense retrieval methods have been proposed. Inverse Cloze Task (ICT) (Lee et al., 2019) generates query-passage pair by randomly selecting one sentence from the passage as the query and the other part as the paired passage. ConDensor (CD) (Gao and Callan, 2021) applies MLM on top of the CLS token embedding from the final layer and the other context embeddings from a previous layer to force the model to learn meaningful CLS representation. SimCSE (Gao et al., 2021a; Liu et al., 2021) passes the same input twice through the network with different dropout masks and min
imizes the distance of the resulting embeddings, while Contrastive Tension (CT) (Carlsson et al., 2021) passes the input through two different models. TSDAE (Wang et al., 2021) uses a denoising auto-encoder architecture for representation learning: Words from the input text are removed and passed through an encoder to generate a fixed-sized embedding. A decoder must reconstruct the original text without noise. As we show in Appendix E, just using these unsupervised techniques is not sufficient and the resulting models perform poorly.
So far, ICT and CD have only been studied on in-domain performance, i.e. a large in-domain labeled dataset is available which is used for subsequent supervised fine-tuning. SimCSE, CT, and TSDAE have been only studied for unsupervised sentence embedding learning. As our results show in Appendix E, they do not work at all for purely unsupervised dense retrieval.
If these pre-training approaches can be used for unsupervised domain adaptation for dense retrieval was so far unclear. In this work, we transfer the setup from Wang et al. (2021) to dense retrieval and first pre-train on the target corpus, followed by supervised training on labeled data from MS MARCO (Nguyen et al., 2016). Performance is then measured on the target corpus.
Query Generation. Query generation has been used to improve retrieval performances. Doc2query (Nogueira et al., 2019a,b) expands passages with predicted queries, generated by a trained encoder-decoder model, and uses traditional BM25 lexical search. This performed well in the zero-shot retrieval benchmark BeIR (Thakur et al., 2021b). Ma et al. (2021) proposes QGen, that uses a query generator trained on general domain data to synthesize domain-targeted queries for the target corpus, on which a dense retriever is trained from scratch. As a concurrent work, Liang et al. (2020) also proposes the similar method. Following this idea, Thakur et al. (2021b) views QGen as a posttraining method to adapt powerful MS MARCO retrievers to the target domains.
Despite the success of QGen, previous methods only consider the cross-entropy loss with in-batch negatives, which provides coarse-grained relevance and thus limits the performance. In this work, we show that extending this approach by using pseudolabels from a cross-encoder together with hard neg
As shown in Wang et al. (2021), training in the reverse order, i.e. first on MS MARCO and then on the target corpus usually performs poorly.
Figure 1: Generative Pseudo Labeling (GPL) for training domain-adapted dense retriever. First, synthetic queries are generated for each passage from the target corpus. Then, the generated queries are used for mining negative passages. Finally, the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted dense retriever. The output at each step is marked with dashed boxes.
atives can boost the performance by several points nDCG@10.
Other Methods. Recently, Xin et al. (2021) proposes MoDIR to use Domain Adversarial Training (DAT) (Ganin et al., 2016) for unsupervised domain adaptation of dense retrievers. MoDIR trains models by generating domain invariant representations to attack a domain classifier. However, as argued in Karouzos et al. (2021), DAT trains models by minimizing the distance between representations from different domains and such learning objective can result in bad embedding space and unstable performance. For sentiment classification, Karouzos et al. (2021) proposes UDALM based on multiple stages of training. UDALM first applies MLM training on the target domain; and it then applies multi-task learning on the target domain with MLM and on the source domain with a supervised objective. However, as shown in section 5, we find this method cannot yield improvement for retrieval tasks.
Pseudo Labeling and Cross-Encoders: Bi-Encoders map queries and passage independently to a shared vector space from which the query-passage similarity is computed. In contrast, cross-encoders (Humeau et al., 2020) work on the concatenation of the query and passage and predict a relevance score using cross-attention between query and passage. This can be used in a re-ranking setup (Nogueira and Cho, 2019), where the relevancy is predicted for all query-passage-pairs for a small candidate set. Previous work has shown that cross-encoders achieve much higher performances (Thakur et al., 2021a; Hofstätter et al., 2020; Ren et al., 2021) and are less prone to domain shifts (Thakur et al., 2021b). But cross-encoders come with an extremely high computational overhead, making them less suited for a production set
ting. Transferring knowledge from cross-encoder to bi-encoders have been shown previous for sentence embeddings (Thakur et al., 2021a) and for dense retrieval: Hofstätter et al. (2020) predict cross-encoder scores for (query, positive)-pairs and (query, negative)-pairs and learns a bi-encoder to predict the margin between the two scores. This has been shown highly effective for in-domain dense retrieval.
This section describes our proposed Generative Pseudo Labeling (GPL) method for the unsupervised domain adaptation of dense retrievers. Figure 1 illustrates the idea of GPL.
For a given target corpus, we generate for each passage three queries (cf. Table 3) using an T5-encoder-decoder model (Raffel et al., 2020). For each of the generated queries, we use an existing retrieval system to retrieve 50 negative passages. Dense retrieval with a pre-existing model was slightly more effective than BM25 lexical retrieval (cf. Appendix A). For each (query, positive, negative)-tuple we compute the margin  $\delta = \mathrm{CE}(Q,P^{+},\cdot) - \mathrm{CE}(Q,P^{-})$  with CE the score as predicted by a cross-encoder,  $Q$  the query and  $P^{+} / P^{-}$  the positive / negative passage.
We use the synthetic dataset  $D_{\mathrm{GPL}} = \{(Q_i, P_i, P_i^-, \delta_i)\}_i$  with the MarginMSE loss (Hofstätter et al., 2020) for training a domain-adapted dense retriever that maps queries and passages into the shared vector space.
Our method requires from the target domain just an unlabeled collection of passages. Further, we use pre-existing T5- and cross-encoder models that have been trained on the MS MARCO passages dataset.
Query Generation: To enable supervised train
ing on the target corpus, synthetic queries can be generated for the target passages using a query generator trained on a different, existing dataset like MS MARCO. Previous work QGen (Ma et al., 2021) used the simple MultipleNegativesRanking (MNRL) loss (Henderson et al., 2017; van den Oord et al., 2018) with in-batch negatives<sup>5</sup> to train the model:
$$
\begin{array}{l} L _ {\mathrm {M N R L}} (\theta) = \\ - \frac {1}{M} \sum_ {i = 0} ^ {M - 1} \log \frac {\exp \left(\tau \cdot \sigma (f _ {\theta} (Q _ {i}) , f _ {\theta} (P _ {i}))\right)}{\sum_ {j = 0} ^ {M - 1} \exp \left(\tau \cdot \sigma (f _ {\theta} (Q _ {i}) , f _ {\theta} (P _ {j}))\right)} \\ \end{array}
$$
where  $P_{i}$  is a relevant passage for  $Q_{i}$ ;  $\sigma$  is a certain similarity function for vectors;  $\tau$  controls the sharpness of the softmax normalization;  $M$  is the batch size.
MarginMSE loss: MultipleNegativesRanking loss considers only the coarse relationship between queries and passages, i.e. the matching passage is considered as relevant while all other passages are considered irrelevant. However, the query encoder is not without flaws and might generate queries that are not answerable by the passage. Further, other passages might actually be relevant as well for a given query, which is especially the case if training is done with hard negatives as we do it for GPL.
In contrast, MarginMSE loss (Hofstätter et al., 2020) uses a powerful cross-encoder to soft-label (query, passage) pairs. It then teaches the dense retriever to mimic the score margin between the positive and negative query-passage pairs. Formally,
$$
L _ {\text {M a r g i n M S E}} (\theta) = - \frac {1}{M} \sum_ {i = 0} ^ {M - 1} | \hat {\delta} _ {i} - \delta_ {i} | ^ {2} \tag {1}
$$
where  $\hat{\delta}_i$  is the corresponding score margin of the student dense retriever, i.e.  $\hat{\delta}_i = f_\theta(Q_i)^T f_\theta(P_i) - f_\theta(Q_i)^T f_\theta(P_i^-)$ . Here the dot-product is usually used due to the infinite range of the cross-encoder scores.
This loss is a critical component of GPL, as it solves two major issues from the previous QGen method: A badly generated query for a given passage will get a low score from the cross-encoder,
hence, we do not expect the dense retriever to put the query and passage close in the vector space. A false negative will lead to a high score from the cross-encoder, hence, we do not force the dense retriever to assign a large distance between the corresponding embeddings. In section 6.3, we show that GPL is a lot more robust to badly generated queries than the previous QGen method.
In this section, we describe the experimental setup, the datasets used and the baselines for comparison.
We use the MS MARCO passage ranking dataset (Nguyen et al., 2016) as the data from the source domain. It has 8.8M passages and 532.8K query-passage pairs labeled as relevant in the training set. We select six representative datasets from the BeIR benchmark as the data from the target domain (cf. subsection 4.2). As Table 1 shows, a state-of-the-art dense retrieval model, achieving an MRR@10 of 33.2 points on the MS MARCO passage ranking dataset, performs poorly on the six selected domain-specific retrieval datasets when compared to simple BM25 lexical search.
We use the DistilBERT (Sanh et al., 2019) for all the experiments. We use the concatenation of the title and the body text as the input passage for all the models. We use a maximum sequence length of 350 with mean pooling and dot-product similarity by default. For QGen, we use the default setting in Thakur et al. (2021b): 1-epoch training and batch size 75. For GPL, we train the models with 140k training steps and batch size 32. To generate queries for both QGen and GPL, we use the DocT5Query (Nogueira et al., 2019a) generator trained on MS MARCO and generate queries using nucleus sampling with temperature 1.0,  $k = 25$  and  $p = 0.95$ . To retrieve hard negatives for both GPL and the zero-shot setting of MS MARCO training, we use two dense retrievers with cosine-similarity trained on MS MARCO: msmarco-distilbert-base-v3 and msmarco-MiniLM-L-6-v3 from Sentence-Transformers<sup>8</sup>. The zero-shot performance of these two dense retrievers is available in Appendix B. We retrieve 50 negatives
In-batch negatives mean that within the same batch, the passages in example  $i$  are viewed as the negatives for example  $j$  ( $i \neq j$ ).
For example, some generated queries are statements instead of questions.
7We use the script from BeIR at https://github. com/UKPLab/beir.
<sup>8</sup>https://github.com/UKPLab/sentence-transformers
using each retriever and uniformly sample one negative passage and one positive passage for each training query to form one training example. For pseudo labeling, we use the ms-marco-MiniLM-L-6-v2 $^{9}$  cross-encoder. For all the pre-training methods (e.g. TSDAE and MLM), we train the models for 100K training steps and with batch size 8.
As shown in Section 6, small corpora require more generated queries and for large corpora, a small down-sampled subset (e.g.  $50\mathrm{K}$ ) is enough for good performance. Based on these findings, we adjust the number of generated queries per passage  $q_{\mathrm{avg}}$  and the corpus size  $|C|$  to make the total number of generated queries equal to a fixed number,  $250\mathrm{K}$ , i.e.  $q_{\mathrm{avg}} \times |C| = 250\mathrm{K}$ . In detail, we first set  $q_{\mathrm{avg.}} > = 3$  and uniformly down-sample the corpus if  $3 \times |C| > 250\mathrm{K}$ ; then we calculate  $q_{\mathrm{avg.}} = \lceil 250\mathrm{K} / |C| \rceil$ . For example, the  $q_{\mathrm{avg.}}$  values for FiQA (original size  $= 57.6\mathrm{K}$ ) and Robust04 (original size  $= 528.2\mathrm{K}$ ) are 5 and 3, resp. and the Robust04 corpus is down-sampled to  $83.3\mathrm{K}$ . QGen and GPL share the generated queries for fair comparison.
As our methods focus on domain adaptation to specialized domains, we selected six domain-specific text retrieval tasks from the BeIR benchmark (Thakur et al., 2021b): FiQA (financial domain) (Maia et al., 2018), SciFact (scientific papers) (Wadden et al., 2020), BioASQ (biomedical Q&A) (Tsatsaronis et al., 2015), TREC-COVID (scientific papers on COVID-19) (Roberts et al., 2020), CQADupStack (12 StackExchange subforums) (Hoogeveen et al., 2015) and Robust04 (news articles) (Voorhees, 2005). These selected datasets each contain a corpus with a rather specific language and can thus act as a suitable test bed for domain adaptation.
The detailed information for all the target datasets is available at Appendix C. We make modification on BioASQ and TREC-COVID. For efficient training and evaluation on BioASQ, we randomly remove irrelevant passages to make the final corpus size to 1M. In TREC-COVID, the original corpus has many documents with a missing abstract. The retrieval systems that were used to create the annotation pool for TREC-COVID often ignored such documents. This leads to a strong
annotation bias on text length for these documents, since this dataset contains only titles and abstracts. Hence, we removed all documents with a missing abstract from the corpus. The evaluation results on the original BioASQ and TREC-COVID are available at Appendix D. Evaluation is done using nDCG@10.
Zero-Shot Models: We apply supervised training on MS MARCO or PAQ (Lewis et al., 2021) and evaluate the trained retrievers on the target datasets. (a) MS MARCO represents a distilbert-base dense retrieval model trained with MarginMSE on the MS MARCO dataset with batch-size 75 for 70k steps. (b) PAQ (Oguz et al., 2021) represents MNRL training on the PAQ dataset. (c)  $\mathbf{PAQ} + \mathbf{MS}$  MARCO represents MNRL training on PAQ followed by MarginMSE training on MS MARCO. (d) TSDAEms MARCO represents TSDAE (Wang et al., 2021) pre-training on MS MARCO followed by MarginMSE training on MS MARCO. (e) BM25 system based on lexical matching from Elasicsearch $^{10}$ .
Previous Domain Adaptation Methods: We include two previous unsupervised domain adaptation methods, UDALM (Karouzos et al., 2021) and MoDIR (Xin et al., 2021). For UDALM, we apply MLM training on the target corpus and then apply the multi-task training of MarginMSE training on MS MARCO and MLM training on the target corpus. For MoDIR, it starts from the ANCE checkpoint and apply domain adversarial training on MS MARCO and the target dataset. As of writing, the training code of MoDIR is not public, but domain adapted models for 5 out of 6 datasets have been released by the authors.
Pre-Training based Domain Adaptation: We follow the setup proposed in Wang et al. (2021) on domain-adapted pre-training: We pre-train the dense retrievers with different methods on the target corpus and then continue to train the models on MS MARCO with MarginMSE loss. The pre-training methods consist of: (a) CD (Gao and Callan, 2021) extracts the hidden representations from an intermediate layer and applies MLM on the CLS token representation and these extracted hidden representations $^{11}$ . (b) SimCSE (Gao et al., 2021b; Liu et al., 2021) simply encode the same
10https://www/elastic.co
$^{11}\mathrm{CD}$  can only be applied with CLS pooling.
$^{9}$ https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2
Dataset
Method	FiQA	SciFact	BioASQ	TRECC.	CQADup.	Robust04	Avg.
Zero-Shot Models
MS MARCO	26.7	57.1	52.9	66.1	29.6	39.0	45.2
PAQ	15.2	53.3	44.0	23.8	24.5	31.9	32.1
PAQ + MS MARCO	26.7	57.6	53.8	63.4	30.6	37.2	44.9
TSDAEMS MARCO	26.7	55.5	51.4	65.6	30.5	36.6	44.4
BM25	23.9	66.1	70.7	60.1	31.5	38.7	48.5
Previous Domain Adaptation Methods
UDALM	23.3	33.6	33.1	57.1	24.6	26.3	33.0
MoDIR	29.6	50.2	47.9	66.0	29.7	-	-
Pre-Training based Domain Adaptation: Target → MS MARCO
CT	28.3	55.6	49.9	63.8	30.5	35.9	44.0
CD	27.0	62.7	47.7	65.4	30.6	34.5	44.7
SimCSE	26.7	55.0	53.2	68.3	29.0	37.9	45.0
ICT	27.0	58.3	55.3	69.7	31.3	37.4	46.5
MLM	30.2	60.0	51.3	69.5	30.4	38.8	46.7
TSDAE	29.3	62.8	55.5	76.1	31.8	39.4	49.2
Generation-based Domain Adaptation (Previous State-of-the-Art)
QGen	28.7	63.8	56.5	72.4	33.0	38.1	48.8
QGen (w/ Hard Negatives)	26.0	59.6	57.7	65.0	33.2	36.5	46.3
TSDAE + QGen (Ours)	31.4	66.7	58.1	72.6	35.3	37.4	50.3
Proposed Method: Generative Pseudo Labeling
GPL	32.8	66.4	61.0	72.6	34.5	41.4	51.5
TSDAE + GPL	34.4	68.9	61.6	74.6	35.1	43.0	52.9
Re-Ranking with Cross-Encoders (Upper Bound, Inefficient at Inference)
BM25 + CE	33.1	67.6	72.8	71.2	36.8	46.7	54.7
MS MARCO + CE	33.0	66.9	57.4	65.1	36.9	44.7	50.7
TSDAE + GPL + CE	36.4	68.3	68.0	71.4	38.1	48.3	55.1
Table 1: Evaluation using nDCG@10. The best results of the single-stage dense retrievers are bold. TRECC. and CQADup. are short for TREC-COVID and CQADupStack. Our proposed GPL significantly outperforms other domain adaptation methods. For the first time, we investigate the TSDAE pre-training in domain adaptation for dense retrieval and find it can significantly improve both QGen and GPL. The results on the full 18 BeIR datasets can be found in Appendix D.
text twice with different dropout masks in combination with MNRL loss. (c) CT (Carlsson et al., 2021) is similar to SimCSE but it uses two independent encoders to encode a pair of text. (d) MLM (Devlin et al., 2019) uses the default setting in original paper, where  $15\%$  tokens in a text are sampled to be masked and are needed to be predicted. (e) ICT (Lee et al., 2019) uniformly samples one sentence from a passage as the pseudo query to that passage and uses MNRL loss on the synthetic data. We follow the setting in Lee et al. (2019) and masked out the selected sentence  $90\%$  of the time. (f) TSDAE (Wang et al., 2021) uses a denoising autoencoder to pre-train the dense retrievers with  $60\%$  random tokens deleted in the input texts.