A Thorough Examination on Zero-shot Dense Retrieval
Authors: Ruiyang Ren
Yingqi Qu
Jing Liu
Xin Zhao
Qifei Wu
Yuchen DIng
Hua Wu
Haifeng Wang
Ji-Rong Wen
Venue: EMNLP 2023 Findings (2023)
Keywords: Zero-shot dense retrieval; Information retrieval

Abstract: Recent years have witnessed the significant advance in dense retrieval (DR) based on powerful pre-trained language models (PLM). DR models have achieved excellent performance in several benchmark datasets, while they are shown to be not as competitive as traditional sparse retrieval models (e.g., BM25) in a zero-shot retrieval setting. However, in the related literature, there still lacks a detailed and comprehensive study on zero-shot retrieval. In this paper, we present the first thorough examination of the zero-shot capability of DR models. We aim to identify the key factors and analyze how they affect zero-shot retrieval performance. In particular, we discuss the effect of several key factors related to source training set, analyze the potential bias from the target dataset, and review and compare existing zero-shot DR models. Our findings provide important evidence to better understand and develop zero-shot DR models.

A Thorough Examination on Zero-shot Dense Retrieval
Ruiyang Ren
Yingqi Qu² Jing Liu Wayne Xin Zhao 13t Qifei Wu?
Yuchen Ding? Hua Wu² Haifeng Wang? Ji-Rong Wen
'Gaoling School of Artificial Intelligence, Renmin University of China Baidu Inc.
BBeijing Key Laboratory of Big Data Management and Analysis Methods (reyon.ren, jrwen) @ruc.edu.cn, batmanfly @ gmail.com (quyingqi, liujing46, wuqifei, dingyuchen, wu wu_hwat wanghaifeng @ @baidu.com
Abstract
to investigate the retrieval capabilities of DR mod- els under the zero-shot setting, where DR models trained on an existing domain (called source do- main) is directly applied to another domain (called target domain) with no available training data on the target domain. Considering the large discrep- ancy between different domains. the zero-shot ca- pability directly affects the widespread deploy- ment of DR models in real-world applications.
Recent years have witnessed the significant ad- vance in dense retrieval (DR) based on pow- erful pre-trained language models (PLM). DR models have achieved excellent performance in several benchmark datasets, while they shown to be not as competitive as traditional sparse retrieval models (e.g. BM25) in a zero- shot retrieval setting. However, in the related literature, there still lacks a detailed and com- prehensive study on zero-shot retrieval. In this paper, present the first thorough examina- tion of the zero-shot capability of DR models. We aim identify the key factors and ana- lyze how they affect zero-shot retrieval perfor- mance. In particular, we discuss the effect of several key factors related source training set, analyze the potential hias from the target dataset, and review and compare existing zero- shot DR models. Our findings provide impor- tant evidence to better understand and develop zero-shot DR models.
Recently, number of studies have been con- ducted to analyze the zero-shot capability of DR models (Thakur et al., 2021b; Chen et al., 2022), and report that DR models have poor zero-shot re- trieval performance when compared with lexical models such as BM25. However, existing works primarily focus on zero-shot performance across wide range of tasks, and lacks comprehensive and in-depth analysis the influence of different settings affected by various factors. Thus, the de- rived findings may not hold under some specific settings, which cannot ensure an accurate under- standing of the zero-shot retrieval capacity. For example, our empirical analysis has revealed that BM25 is not the absolute leader compared with DR models under the zero-shot setting, and differ- ent source training sets result in different zero-shot performances (Section 2.3). As result, thor- ough examination of the zero-shot capacity of DR models is necessary.
1 Introduction
With the massive success of pre-trained language models (PLM) (Devlin et al., 2019; Brown et al., 2020), dense retrieval (DR) has been empowered to become an essential technique in first-stage re- trieval. Instead of using sparse term-based rep- resentations, DR learns low-dimensional query and document embeddings for semantic match- ing, which has been shown to be competitive with or even better than sparse retrievers (e.g., BM25) (Karpukhin et al., 2020).
Considering these issues, this paper aims to pro- vide an in-depth empirical analysis of zero-shot DR. Specifically, we would like to conduct a more detailed analysis by analyzing the effect of dif- ferent factors on retrieval performance. Since the DR models are trained on the source domain data (called source training set), we focus the analy- sis by examining three major aspects of the source training set, including query set, document set, and data scale. Besides, we also consider other possi- ble factors such as query type distribution, vocab- ulary overlap between source and target sets, and
Unlike sparse retrievers, DR models typically require training on sufficient labeled data to achieve good performance in a specific domain. However, there are always new domains (or sce- narios) that need to be dealt with for an informa- tion retrieval system, where little in-domain la- beled data is accessable. Therefore, it is crucial
"The work was done during the internship at Baidu. Corresponding authors.

---

the bias of the target dataset. Our analysis is par- ticularly interested in answering the rescarch ques- tions of what influence factors affect the zero-shot capabilities of DR models and how does each of the influence factors affect such capability?
Datasets
#Truin Q
4Eval.
AD
Q Len.
Natural Questions (NQ)
58.812
3,610
21,015,324
9.2/ 1,0/000.0
MIMARCO (MM)
502,939
6,980
8,841,823
6.0/
MSMARCOv2 (MMv2)
277,144
8,184
138,364,196
57.5
HorporQA (HQA)
72.928
5,901
21,015,324
20.0/ 100.0
TriviaQA (TQA)
61,688
7,785
21,015,324
13.7 100.0
SearchQA (SQA)
117,384
NQRROA
104,071
6,775
21,015,324
11.1 100.0
In addition, we also systematically review the recent advances in zero-shot DR. We summarize and categorize the key techniques that can improve the zero-shot performance (with an associated dis- cussion of the potential influence factors).
HIQA-2018
5,500
6488
67,658
10.8 132.3
SciFact
809
300
5,183
21../
SciDaas
1,000
25.637
9.4/ 176.2
BiaASQ
3,742
500
14,914,602
8.1 202.6
Quora
10,000
522.951
9
ArguAna
1,406
8,674
193.0/ 166.8
By our empirical analysis, we find that: (i) Training data from source domain has important effect on the zero-shot capability of the trained DR model, either the query set or document set in it (Section 3.1 and Section 3.2). (ii) Vocabu- lary overlap and query type distribution are poten- tial factors that affect the zero-shot performance (Section 3.1.2 and Section 3.1.3). (iii) Increasing the query scale of source data can improve both the in-domain and out-of-domain performance of the DR model (Section 3.3.1). (iv) Increasing the document scale of the source domain may result in performance decrease (Section 3.3.2). (v) The lexical bias (overlap coefficient of queries and an- notated documents) of some datasets is probably one of the reasons that the performance of sparse retriever scems to be more robust than DR model on existing benchmarks (Section 3.4). Our study provides an important work to understand and im- prove the zero-shot capacity of the DR model.
Table 1: Dataset statistics, where "Q" denotes query and "D" denotes document. The upper part lists source datasets and the lower part lists target datasets in our zero-shot experiments.
We mainly consider the setting of zero-shot DR (Thakur et al., 2021b), where the labeled data from target domain is only available for testing. Besides, we assume that the labeled data from a different domain (called source domain) can be used to train the DR models. Such setting is slightly different from general zero-shot learning, where no labeled data is available at all. This paper aims to present a deep and comprehensive analysis on the zero-shot capacity of DR models.
2.2 Experimental Setup
Datasets In order to comprehensively study the zero-shot capability of the DR model, we col- lect 12 public datasets that cover multiple domains with different data characteristics. The statistics of the datasets is shown in Table 1, including source datasets and target datasets. In our ex- perimental design, datasets low conduct controlled experiments. The target datasets are the representative datasets in BEIR (Thakur al., 202 lb) benchmark with mul- tiple domains. The detailed description can be found in Appendix A.
2 Background and Setup
2.1 Zero-shot Dense Retrieval
We study the task of finding relevant docu- ments with respect to query from large doc- ument set!. In particular, retrieval (DR) (Zhao al., 2022) that learn low-dimensional semantic embeddings for both queries and documents and then measure the em- bedding similarities as the relevance scores. To implement DR, a PLM- based dual-encoder has been widely adopted (Karpukhin et al., 2020; Xiong et al., 2020: Zhou et al., 2023) by learn- ing two separate encoders for relevance matching. Compared with sparse retrievers (e.g.. BM25), DR models highly rely large-scale high-quality training data to achieve good performance (Qu et al., 2021; Ren et al., 2021a; Zhou et al., 2022).
selected that al-
focus dense
Training Data Construction To examine the ef- fect of query set and document set, which are two factors in source dataset, we incorporate a combi- nation form of <Query set, Document set> to de- note the resources of query set and document set, where they can come from different domains or datasets. For example, "<NQ, MM>" in Table 3 denotes that queries in NQ are used to construct training data with documents in MSMARCO.
Note that in this paper, "document" is generalized con- copt that can be in a variety of text forms, such as passage, sentence, and article.
There are two types of annotation including long answer and short answer, where long answer

---

Targel (-)
NQ
MSMARCO
Results on NQ and MSMARCO It also can be observed in Table 2 that the two DR models are better than BM25 when evaluated on NQ and
Models (with source dataset) (1)
M@10
RØ50
M@10
K@50
BM25
31.3
69.7
18.7
59.2
ANCE (MSMARCO)
39,9
78.9
33,0
79,1
RockerQAA (MSMARCD)
50.8
83.2
38.8
86.5
MSMARCO. Furthermore, the performance gap evaluating on MSMARCO development set be- tween RocketQAv2 (NQ) and RocketQAv2 (MS- MARCO) (38.8 vs 22.4) is greater than the perfor- mance gap NQ test set (61.1 vs 50.8), indicat- ing that the DR model trained on MSMARCO is stronger than that trained on NQ.
RocketQA>22 (NQ)
61.1
86.9
22.4
67.2
Table 2: Evaluation results of models trained on MS- MARCO and NQ.
denotes the whole relative document and short an- swer denotes the short span that directly answers the query. When examining the effects of query set and document set in the source dataset, we change the query set or document set respectively. In such cases, we leverage the annotated short answers to relabel the relevant documents in combined source dataset. Taking "<NQ, MM> in Table 3 as an example, we use queries in NQ and select can- didates containing short answers as positives from MSMARCO document set, negatives are sampled from candidates without short answers.
Results on Target Datasets We perform the zero-shot evaluation of RocketQAv2 on six tar- get datasets, including SciDocs, ScilFact, FiQA, BioASQ. Quora, and ArguAna. As shown in Ta- ble 3 (part A), RocketQAv2 (MSMARCO) outper- forms RocketQAv2 (NQ) on most target datasets, showing that the DR model trained on MS- MARCO has a stronger zero-shot capability. Moreover, can be observed that BM25 signif- icantly outperforms DR models on SciFact and BioASQ, and is also competitive on other datasets, which is strong zero-shot retrieval baseline.
Evaluation Metries We use MRR @ 10 and Re- 50 as evaluation metries. MRR @ 10 calcu- lates the averaged reciprocal of the rank of the first positive document for set of queries. Recall calculates a truncated recall value at position 50 of a retrieved list. For space constraints, we denote MRR @ 10 and Recall 50 by M@ 10 and R R@50, respectively.
Through the initial experiments, we can see that models trained on the two source datasets have dif- ferent zero-shot performance when tested on di- verse target datasets. It is difficult to directly draw concise conclusions about the zero-shot capacities of DR models, given the significant performance variations in different settings. Considering this issue, in what follows, we systematically investi- gate what factors relevant and how they affect the zero-shot performance in multiple aspects.
2.3 Initial Exploration
To gain basic understanding of zero-shot re- trieval capacities, we conduct analysis on two well-known public datasets, MSMARCO and NQ, considering both in-domain (training and testing on the same dataset) and out-of-domain settings (training and testing on two different datasets).
Experimental Analysis and Findings
In this section, we conduct a detailed analysis about multiple factors of the source training set, including source query set, source document set, and data scale. Besides, we also analyze the effect of the bias from the target dataset.
Backhone Selection We select RocketQAv2 (Ren al.. 2021b) as the major DR model for study, which is one of the ttate-of-the-art DR models. We train models on MSMARCO and NQ, and use RocketQAv2 (MSMARCO) and RocketQAv2 (NQ) denote RocketQAv2 model trained on MS- MARCO and NQ, respectively. For comparison, we adopt the ANCE (Xiong et al., 2020) and BM25 two baselines and report their corre- sponding performance. From Table 2, we find that RockerQAv2 outperforms ANCE evaluating on the two datasets (trained on MSMARCO). Thus, in the following experiments, we select Rock- etQAv2 as the backbone model.
3.1 The Effect of Source Query Set
We first analyze the effect of different source query sets by fixing the document set. The ex- amined query sets include NQ. MSMARCO, and datasets in MRQA (Fisch et al., 2019).
3.1.1 Experimental Results
First, we fix MSMARCO the document and use NQ queries and MSMARCO queries as two query sets to construct training data. After train- ing the backbone models with the two constructed

---

Target (-))
SeiDocs M@10
SciFact
FIQA-2018
BiAAS M@10
Quura M@10
ArguAna M@10
Averuge
Source ())
5
M0
R@50
M
R@50
R@:50
R
M6110
R@50
A. Initial Experiment
BM25
20.7
57.8
62.2
86.7
29.7
63.77
46.0
73.0
73.8
95.7
20.8
86.9
42.2
77.3
Wikipedins
24.3
61.2
40.5
76.8
97.8
9.7
65.9
33.2
6885
<MM. MM-
24.1
63.7
48.0
77.0
35.1
71.3
30.9
55.0
71.4
97.6
21.6
85.8
38,5
75.1
B. Effect Source Query Set
<NQ. MMs
21.0
56.0
33.7
66.7
17.9
55.
20.0
44.8
72.0
96.7
11.3
73.7
29.2
65.5
<MM. MM+
23.7
61.3
47.1
76.7
33.5
68.7
51.8
70.0
20.1
83.3
37.4
68.6
<TQA, Wikippdiins
23.1
61.0
43.9
17.3
18.4
54.6
23.9
47.6
62.5
92.9
15.4
79.2
51.2
68.8
<SQA, Wikipedias
22.3
59.9
44.1
77.7
19.9
54.8
20.1
41.4
66.2
94.7
15.8
78,9
31,4
67.9
<HQA, Wikipedinss
184
54.6
34.6
69.7
11.4
37.4
18.3
40.6
46.5
83.8
10.5
69.1
23.3
59.2
<NQRROA- Wikipedia>>
24.0
62.7
29.8
72.7
24.1
60.5
23.9
44.4
71.7
96.8
14.5
78.2
31,2
69.2
C. Effect of Souree Documeet Set
<NQ. Wikipediites
23.5
60.5
41.2
24.5
23.5
49.4
76.1
97.6
9.7
06.2
33.1
67.7
<NO. MMs
20.0
56.0
33.7
66.7
17.9
55.1
20.0
44.8
72.0
96.7
11.3
73.7
29.2
65.5
Wikipodia+MM>
73.8
96.8
58.1
30.7
D1. Effect of Query Scale
<NQ 10%, Wikipedia>
7.6
40.9
32.1
9.1
56.8
<NQ S0SS, Wikipedia>
22.1
59.6
38.7
70.0
18.9
54.8
18.0
42.4
61.4
92.4
12.0
71,9
28.5
65.2
<NQ Wikipedias
23.0
59.7
44.4
68.8
94.0
65.2
29.2
65.1
<MM 10%s MMs
23.0
60.2
43.1
73.7
30.7
67.8
27.8
50.8
79.4
58.5
19.0
83.1
57.2
72.4
<MM 50%. MMs
24.4
62.8
47.0
74.7
32.9
68.8
28.9
53.6
77.7
98.6
19.8
84.9
38.5
73.9
<MM 1001, MMs
23.4
62.9
47.6
77.7
34.0
69.4
31.8
55.6
80.1
99.0
20.7
85.4
39.6
75.0
D2. Dfffect
Scale
<MM+2 1%, MM+2>
25.6
64.4
49.2
79.3
31.9
31.9
55.4
80.6
98.9
15.2
76.3
39,1
73.9
<MM2 108, MMv2>
24.7
63.6
48.2
78.0
32.1
66.7
30.2
54.6
77.9
98.7
14.4
75.1
37.9
72.8
<MM+2 100%, MMv2>
24.2
63.2
49.5
79.3
32.8
69.3
30.1
52.8
77.1
98.6
14.9
75.8
38.1
73.2
Table 3: Zero-shot evaluation results on six target datasets with different settings. Document Ser> to denote the query and document sets used for training.
We use the form <Query Set,
training datasets (denoted by MNQ and MMARCO), we evaluate them on six target datasets. More- over, collect four QA datasets in MRQA task source datasets from similar domains, including TriviaQA, SearchQA, HotpotQA, and NQ. Since these datasets are constructed based on Wikipedia, we adopt Wikipedia as the document set to con- struct training data for queries. By fixing the doc- ument set, we can analyze the effect of query sets on the retrieval performance.
get query sets. Following the previous work (Gu- rurangan et al., 2020), we consider the top 10K most frequent words (excluding stopwords) in each query set. For each pair of source-target query sets, we calculate the percentage of vocabu- lary overlap by the weighted Jaccard (Ioffe, 2010).
Figure (red lines) present the relationship be- tween query vocabulary overlap and zero-shot per- formance on six target datasets, where we sort the source datasets according to their corresponding results from Table 3 (x-axis). The query vocabu- lary overlap between source and target datasets are shown in y-axis.
Table (part B) shows the zero-shot retrieval results on target datasets. It can be observed that the overall zero-shot capability on target datasets of MNO is worse than MMARCO- which is con- sistent with initial experiments (Table 3, part A). These results further verify that the source query set actually has important effects the zero-shot capability of the DR model.
Overall, we can observe that there a certain positive correlation between query vocabulary overlap and zero-shot performance, with only a small number of inconsistent cases. It is because larger vocabulary overlap indicates a stronger domain similarity. Similar results (blue lines in Figure 1) are also found for vocabulary overlap of documents across domains.
Given the above overall analysis, we next zoom into two more detailed factors for better under- standing the effect of the query
3.1.2 Query Vocabulary Overlap
3.1.3 Query Type Distribution
Since domain similarity is critical in cross-domain task performance (Van Asch and Daelemans, 2010), we consider studying a simple yet im- portant indicator to reflect the domain similarity, Le,, vocabulary overlap between source and tar-
Besides vocabulary overlap, we continue study the effect of another important factor, i.e., query type distribution, which summarizes the overall distribution of query contents. In particular, we analyze the distribution of query types for both

---

XSMABCO
18.7%
1,8%
1,8%
15.0%
4.1%
1.2%
1.%
13%
26/%
L664
Di
Dec
NQ
11,5%
31.3%
48.%%
4.3%
3.4%
4.5%
10%
6.5%
100%
BapprQ
15.5%
RUY6
10%
23%
48%
90%
82%
L410
Vecahdary
THiRQA
18.4%
Zera-bbut Peffermance
Zam-shet Parfermance 90A
>Aggahoe
0.5%
6.1%
00%
BOX (164)
5QA
TQx
MM
No (20))
No
MOA (442)
YOA
NM (473))
Quera
37.6%
3.2%
25.7%
18%
5.%%
4.7%
140%
TV0
(223)
(250)
(287)
(2:20
(4200
Bio4QQ
12.2%
4.9%
0.5%
4.5%
0.5%
4.%
11.4%
30.%
14.%
1554
(a) SciDocs
(b) SciFact
FiQ4
15.%%
0.9%
0.5%
15.5%
2.2%
6.5%
000
304%
50.4
14B
Sefhaor
8.2%
6.1%
0.1%
0.1%
8.%%
0.%
000%
6.7%
98.%
Qhen
QU
Doe
Auut
Nocthaler Dverbe
lie
SdFax
8.5%
393
What
When
Whw
Hox
Whore
Way
Which
V/N Daclasttic Eatrapy
40.6
41.6
41.3
2012
27.2
21.3
37.
Figure 2: Query type distribution on 11 datasets for cach dataset with information entropy in last column, where top five datasets are source datasets and bottom six datasets are target datasets.
38
Zerassoet Petfermaree
Zemosher Performonn
1OA
SQx
NO
NM
HOA
S0A
NO
TDA
MM
(c) FIQA
(d) BioASQ
sets share more similar query type distri- butions, the corresponding zero-shot perfor- mance also likely to improve. For exam- ple, SearchQA, containing a large proportion of declarative queries, performs better on datasets also with a large proportion of declarative queries, such as ArguAna and SciFact.
Quen
Uvery
SCA
35.
41.
43.5
43.7
Vocoludary
45.1
27.1
16.1
Zara-bot Porfermunce
Zere-thet Perforaancs
BMA
10A
MIA
XDM (NL0)
NO
BIJA
NO
TUA
NDA
NM
(c) Quora
(1) Arguana
3.2 The Effect of Source Document Set
Figure I: Relationship between vocabulary overlap and zero-shot performance on six target datasets. The x- axis corresponds to the sorted zero-shot performance the target dataset of models trained on different source datasets. y-axis denotes the query/document overlap of source datasets and the target dataset.
Following the analysis of query sets, we next study the effect of source document sets.
To conduct the analysis, we fix NQ queries as the source query set and use Wikipedia and MS- MARCO as source document sets. Furthermore, we merge the Wikipedia and MSMARCO docu- ment sets a more comprehensive document set to study the effect of additional documents from other domain. For each query, we can use the short answer annotated in NQ dataset to select documents from three document sets to construct training data. After training with different source datasets (same query set but different document sets: Wikipedia, MSMARCO, and their combi- nation), we evaluate the zero-shot performance of these three models on six target datasets.
source and target query sets, focusing on "WH" queries, "Y/N" queries, and declarative queries. The query type is determined according to pre- defined rules, which are detailed in Appendix A.4.
Figure 2 presents the distribution of each query set from 11 source and target datasets. For better understanding such a distribution, we also calcu- late the information entropy (Shannon, 1948) of query type distribution for each dataset, where a larger entropy value indicates a more balanced dis- tribution of query types. First, we find that models trained with more comprehensive query types are capable of performing better in the zero- shot retrieval setting, e.g., MSMARCO dataset. particular, MSMARCO contains the most di- verse and comprehensive queries among all source datasets, which leads to a more strong zero-shot capability of DR models as the source training set. Furthermore, when source and target query
Table 3 (part C') provides the evaluation results on six target datasets. A surprising finding is that the model trained on the merged document set does not outperform the model trained on Wikipedia
target datasets. A possible rea- son is that short answers of NQ queries annotated based on Wikipedia do not match well for other document sets, resulting in performance degrada- tion. On the whole, the effect of the underlying document set is not that significant as query set.

---

Turget (-))
Query
In-domain
Improvement
Source (1)
Scale
M@10
R@50
40%
Ratio
<NQ 105, Wikipedia>
5K
40.2
76.1
<NQ 50%, Wikipodia>
25K
53.4
84.1
<NQ 1005,, Wikipedias
50K
85.0
<MM 10%, MMs
50K
90.8
79.5
250K
83.0
<MM 100%, MM-
500K
34,9
83.5
Table 4: In-domain experiments on NQ and MS-
MARCO with different ratios of queries from the orig- inal query set.
20%
40%
Overlap Coefficient
3.3 The Effect of Data Scale
SeiDoes ArguAna FIQA SciFact BioASQ Quera (0.05) (0.15)
(0:34) (0.48) (0.64) (0.68)
Data scale is also an important factor to consider that affects the zero-shot retrieval performance.
Figure 3: Relationship between the improvement ratio of DR models over BM25 and the the overlap coeffi- cient different target datasets.
3.3.1 Query Scale
Query scale refers to the number of queries in the source training set. Since each training query corresponds to one (or multiple) labeled docu- ment(s) for retrieval tasks, query scale represents the amount of annotated data. We conduct ex- periments on NQ and MSMARCO by varying the query scale. For each dataset, we randomly sam- ple 10%, 50%, and 100% queries from the training queries and construct three training sets. We use these training sets to train three models and con- duct in-domain and out-of-domain evaluations.
constructed from different document sets. We then train three different models with 1%, 10%, and full document sets. More details and the in-domain evaluation results can be found in Appendix A.7.
We evaluate the models trained with different document scales on six target datasets and report the results in Table 3 (part D2). Surprisingly, the model trained on 1% document set outperforms those trained full document set and 10% doc- ument set, which seems to contradict the intuition. We suspect that since the main discrepancies be- tween training sets are negatives, the DR model is trained with more diverse source domain nega- tives using a larger document set, which absorbs more domain-speciff characteristics. Therefore, the DR model with larger document set for training is likely to suffer from the over-fitting issue in cross-domain setting.
Table 4 presents the results of in-domain eval- uation and Table (part D1) presents the out-of- domain results six target datasets. First, with the increase of the query scale in the train- ing set, the in-domain capability and out-of- domain zero-shot capability of models gradu- ally improve. Furthermore, considering the two settings with the same query scale, i.e., "NQ 100%" and "MSMARCO 10%" with 50K queries, find the model trained on "MSMARCO 10%" still achieves better zero-shot performance than that trained on "NQ 100%". Combining the results Section 2.3, we find the stronger zero-shot ca- pability of model trained on MSMARCO dataset does not simply come from the larger query scale.
3.4 The Bias from Target Datasets
Recently, BEIR (Thakur et al., 2021b) points out that sparse models are often used to create annota- tions for dataset construction, resulting in the lex- ical bias on the dataset, which also occurs in other tasks (Wang al., 2022: Sun al., 2023). Ac- cording to this, we consider quantitatively investi- gating how such bias affects the zero-shot perfor- mance of the sparse and dense retrieval models, by computing the overlap coefficient (Vijaymeena and Kavitha, 2016) over queries and labeled doc- uments in target test sets.
3.3.2 Document Scale
We further analyze the effect of document scale using MSMARCOv2 (passage) dataset (Craswell et al., 2021), which is the largest publicly available text collection up date. We randomly sample subsets of 1% (1.4 million) and 10% (14 mil- lion) from MSMARCOV2 document set. In each setting, we fix the training queries and correspond- ing annotations, while the negatives are specially
Specially, for each query from the set of target dataset, we compute the overlap coefficient for six target datasets, by dividing the number overlap terms of the query and its annotated docu-

---

ment by the total number of query terms. We sort the overlap coefficients according the value as- cendingly. In particular, we mainly consider com- paring sparse and dense retrieval models. The re- sults on the six target datasets are shown in Fig- ure 3. It can be observed that BM25 overall per- forms better on the target dataset with a larger overlap coefficient and the performance of DR models is better than BM25 with low overlap coefficient. This finding indicates that the bias of the target dataset may cause to fail to evaluate the zero-shot capabilities of different models ac- curately. Thus, how to construct fair datasets is also an important direction that needs to be further explored. An exception in Figure 3 is the Quora dataset. It because that this dataset is created for duplicate question identification, where the query and the document annotation are two related ques- tions with a larger surface similarity.
Method
QG
KD
CP
MSS
ISR
Factors
QGen
VO+OT=QS
AugSBERT
v
QS
SPAR
v
QS
GPL
VO+QT+QS
Contriever
QS
GTR
LaPraDRR
Q8
Table 5: Representative zero-shot DR methods with different techniques. We mark the corresponding in- fluence factors of each methods, where VO, QT, and QS denote the improving factors of vocabulary over- lap, query type and query scale, respectively.
hensive terms that may appear in other domains, and has a more balanced query type distribution.
Model Analysis
So far, we mainly focus on the discussions of data side. In this section, we further review and analyze existing zero-shot DR models.
3.5 Summarizing the Main Findings
4.1 Reviewing Existing Solutions
In this part, we summarize the main findings based on the above experimental analysis.
According to the specific implementations, we summarize the existing zero-shot DR models with the adopted techniques in Table 5. Next, we dis- cuss the major techniques and the representative methods. Note that this section is not a compre- hensive list of existing studies, but rather an analy- sis of the different techniques they use and factors from the previous analysis they improve on.
First, source training set has important ef- fect on the zero-shot retrieval capacities (Sec- tion 3.1 and 3.2). Various factors related to source training set will potentially affect the zero-shot performance, including vocabulary overlap (Sec- tion 3.1.2) and query type distribution (Sec- tion 3.1.3). Our empirical study shows that it is useful to improve the zero-shot performance by increasing the vocabulary overlap and query type distribution similarity between source and target domains and setting more balanced query type dis- tributions for the source domain.
Query Generation (QG) QG methods construct synthetic training data by using documents from the target domain to generate (pseudo) queries, which aims to augment the training data by gener- ating queries that fit the target domain. QGen (Ma et al., 2021) trains an Auto-encoder in the source domain to generate synthetic questions by a tar- get domain document. Similarly, GPL (Wang et al., 2021) generates synthetic queries with pre- trained T5 model. QG can enhance the vocabulary overlap between the source and target domains, meanwhile increasing the query scale and fitting the query type distribution of the target domain.
Second, query scale of source training data sig- nificantly affects the zero-shot capability. Increas- ing the query scale (the amount of annotation data) brings improvement on both in-domain and out- of-domain evaluation (Section 3.3.1). However, seems to bring negative effect when increasing the document scale, which shows performance de- crease in our experiments (Section 3.3.2).
Third, when the test set has a large overlap coefficient between queries and annotated docu- ments, the evaluation might be biased towards ex- act matching methods such as BM25 (Section 3.4). Finally, we find that MSMARCO is more capable source dataset for training zero-shot DR models, compared with NQ. The major reason is that it contains more queries, covers more compre-
Knowledge Distillation (KD) KD is a commonly used strategy in DR, which utilizes a powerful model (e.g., cross-encoder) as the teacher model to improve the capabilities of DR models (Hof- stätter et al., 2021). has been found that such a technique can improve out-of-domain perfor- mance. GPL (Wang et al., 2021) and AugS- BERT (Thakur et al., 2021a) use cross-encoder

---

annotate unlabeled synthetic query-doc pairs train the bi-encoder. Different from the above methods, SPAR (Chen et al., 2021) proposes to distill knowledge from BM25 to DR model. KD based approaches alleviate the data scarcity issue and play similar role in increasing the query scale (i.e., more supervision signals).
Method
PLM
BioASQ
SciFact
FRQA
BM25
46.5
66.5
23.6
QGen
DistiBERT
39.8
64.4
30.8
GTR
Them
27.1
60.0
34.9
GPL
DistiBERTT
42.5
65.2
33.1
LaPraDoR
DistlBERT
51.1
68.7
34.5
La
DisniBERT
30.8
59.9
31.4
Table 6: NDCG@ 10 results of different zero-shot methods on three target datasets.
Contrastive Pre-training (CP) With unsuper- vised contrastive learning showing great power in the field of NLP,, researchers start apply this ap- proach to zero-shot DR. Contriever (Izacard al., 2021) and LaPraDoR (Xu et al., 2022) are two typ- ical methods that build large-scale training pairs similar to SimCSE (Gao et al., 2021) and utilize contrastive loss in training. Such approaches use unsupervised corpus to construct training data, al- lowing the model to capture the matching relation- ship between two text contents in the pre-training phase, essentially enlarging the training data scale.
4.2 Comparison of Zero-shot Methods
We compare different zero-shot methods on three most commonly used target datasets, including BioASQ, SciFact, and FiQA. Table 6 reports the performance of these zero-shot methods. We show the results of QGen reported in BEIR paper since the results in QGen paper is incomplete, and we report the Taase results of GTR for faimess. We replicate the results of GPL on BioASQ with the full document set, since the results in GPL paper is performed by randomly removing irrelevant docu- ments. For LaPraDoR, we use the results of "FT" model fine-tuned MSMARCO dataset. We also report the results without considering BM25 scores (w/o LEDR) for LaPraDoR to investigate the effect of incorporating the sparse retriever.
Model Size Scaling (MSS) For PLM based ap- proaches, becomes broad consensus that scal- ing the model size can lead to substantial perfor- mance improvement (Brown et al., 2020; Fedus et al.. 2021). Recently MSS has shown the ef- fectiveness in zero-shot DR. GTR (Ni et al., 2021) is generalizable T5-based DR model that uses extremely large-scale training data and model pa- rameters, which obtain performance gains. More- over, large language models have attracted a lot of attention recently due to their excellent text gen- eration capabilities (Zhao et al., 2023; Ren et al., 2023), they can also be applied to improve the zero-shot capabilities of dense retrieval.
First, we can see that LaPraDoR overall outper- forms other DR models, when remove the BM25 enhanced strategy, there is huge drop in the zero- shot performance on BioASQ and SciFact. This corresponds our previous analysis (Section 3.4), introducing BM25 in DR model can greatly im- prove the zero-shot performance on datasets with high overlap proportion (BioASQ and SciFact), while does not bring much performance im- provement on FiQA that has low overlap pro- portion. Without considering ISR, GPL achieves competitive performance, since it mainly fits for the vocabulary overlap and query type between the source and target sets, well as increases the scale of pseudo labeled data (for training with knowledge distillation). Therefore, considering more positive influence factors does bring the DR model zero-shot performance improvement.
Integrating Sparse Retrieval (ISR) It has been shown that both sparse and dense retrieval mod- els have specific merits when dealing with differ- ent datasets. Thus, it is an important practical ap- proach that integrates the two methods for enhane- ing the zero-shot retrieval capacities. SPAR (Chen et al., 2021) trains a student DR model that distills BM25 to the DR model, then concats the embed- dings from the BM25- based DR model and nor- mal DR model. Moreover, LaPraDoR (Xu et al., 2022) enhances the DR model with BM25 by mul- tiplying the BM25 score with the similarity score of DR model. We consider ISR a special ap- proach since does not change the setup of source training set for DR model, but mainly combines the sparse and dense models in an ensemble way.
Conclusion and Future Work
In this paper, we thoroughly examine the zero-shot capability of DR models. We conduct empirical analysis by extensively studying the effect of vari- ous factors on the retrieval performance. In partic- ular, we find that the factors of vocabulary overlap, query type distribution, and data scale are likely

---

have varying degrees of influence on the zero- shot performance of the dense retriever. Besides, the performance between BM25 and DR mod- els varies significantly on different target datasets, where the dataset bias (e.g,, a dataset is created based on exact match) is likely to make such com- parison unfair. Overall, we find that the zero-shot performance of dense retrieval models still has room improve and deserves further study. As future work, we will consider designing more ro- bust and general zero-shot retrieval methods that can adapt to different settings or domains.
representation learning using citation-informed transformers. arXiv preprint arXiv:2004007880
Document-leve
Nick Craswell, Bhaskar Mitra, Emine Yil- maz, Daniel Campos, and Jimmy Lin. 2021. Trec 2021 deep learning track guidelines. cttps://microoofet wtt..gg..t.......aaaeaeeeeeeeeee CREC-Deep--earndee html.
Jacob Devlin. Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Languuge Technologies, NAACI-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Pa- pers), pages 4171 4186.
Limitations
In this work, we conduct a comprehensive em- pirical analysis of the zero-shot capability of the dense retrieval model, which requires large GPU resources, and cannot conduct the same large- scale experiments on all other existing dense re- trieval approaches due the huge experimental cost and space limitation. However, since the structure different dense retrieval models are relatively consistent, we believe that the experi- mental results in this paper are generalizable and can reflect the characteristies of the zero-shot ca- pabilitics of dense retrieval models.
Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqu: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv: 1704.05777
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arxhv:2101.03961.
Adam Fisch, Alon Talmor, Robin Jia. Minjoon Sco, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Work- shop on Machine Reading for Question Answering. MRQA @ EMNLP 2019, Hong Kong, China, Novem- ber 2019, pages 1-13.
A Aeknowledgements
This work was partially supported by National Natural Science Foundation China under Grant No. 62222215, Beijing Natural Science Founda- tion under Grant No. L.233008 and 4222027.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo. lz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models domains and tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Tao Chen, Mingyang Zhang, Jing Lu, Michael Bender- sky, and Mare Najork. 2022. Out-of-domaile seman- tics to the rescue! shot hybrid retrieval models. arXiv preprint arXiv:2201. 10582.
Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang. Jimmy Lin, and Allan Hanbury. 2021. Effi- ciently teaching an effective dense retriever with bal- anced topic aware sampling. 44th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pages 112-122.
Xilun Chen, Kushal Lakhotia, Barlas Oguz, An- chit Gupta, Patrick Lewis. Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2021. Salient phrase aware dense retrieval: Can dense retriever imitate sparse one? arXiv preprint 10.09:2110.06911.
Proceedings of the
Sergey Toffe. 2010. Improved consistent sampling, weighted minhash and II sketching. In 2010 IEEE International Conference on Data Mining, pages 246-255. IEEE
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specter:

---

Shankar Iyer. Nikhil Dandekar, and KornA@l Csemaii First quora dataset release: Ques-
Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings.
tion pairs. https: /ttp://guraaaaaa quora.com.
F Kirst-Quera-Dattee
Pair Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Hern annnzz 'Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Weig Chang, and Yinfei Yang. 2021. Large dual encoders are generalizable retrievers. ArXiv, abs/2 12.00999.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave 2021. Towards unsupervised dense information retrieval with contrastive learn- ing. arXiv preprint 1xXi1:211..09118.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An opti- mized training approach to dense passage retrieval for open-domaie question answering. In Proceed- ings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835-5847.
Mandar Joshi, Funsol Choi, Daniel S. Weld, and Luke Zettlemyer. 2017. Triviaqa: large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Vol- ume 1: Long Papers. pages 1601-1611.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage re- trieval for open- open-domain question answering. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781.
Nils Reimers and Iryna Gurevych. 2020. The curse of dense low-dimensional information retrieval for large index sizes. arXiv preprint 2Xxi:2012.142200
Ruiyang Ren, Shangwen Lv. Yingqi Qu Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021a. PAIR: Leverag- ing passage-centric similarity relation for improving dense passage retrieval. In Findings of the Associ- ation for Computational Linguistics: ACL-IICNLP 2021, pages 2173-2183.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ruiyang Ren, Yinggi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021b. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empiri- cal Methods Natural Language Processing, pages 2825-201 2235..
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Jacob De- vlin, Kenton Lee. Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-We Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natural questions: a benchmark for question an- swering research. Trans. Assoc. Comput. Linguis- tics, 7:452-466.
Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023. Investigating the fac- tual knowledge boundary of large language mod- els with retrieval augmentation. arXiv preprint arXiv:2307.11019
Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot neural passage synthetic question gen-
retrieval via domain-targennn eration. In Proceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main Volume, pages 1075- 1088.
Claude Elwood Shannon. 1948. ory of communication. The Bell system technical journal, 27(3):379-423.
A mathematical the-
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www' 18 open challenge: financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, pages 1941-1944
Teng Sun, Juntong Ni, Wenjie Wang. Liqiang Jing, Yinwei Wei, and Liqiang Nie. 2023. General de- biasing for multmmodal sentiment analysis. arXiv preprint arXiv:2307.105//
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021a. Augmented shert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 296-310
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset.
Proceedings of the Workshop on Cognitive Computation: Inte- grating neural and symbolic approaches 2016 co- located with the 30th Annual Conference on Neu- ral Information Processing Systems (NIPS 2016),
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021b. Beir:

---

heterogenous benchmark for zero-shot evalua- tion of information retrieval models. arXiv preprint 2Xiv:2104.086633
for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2369-2380.
George Tsatsaroniss Georgios Balikas, Prodromos Malakasiotis, Toannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenhom, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. 2015, An overview of the bioasq large- scale biomedical semantic indexing and question swering competition. BMC bioinformatics, 16(1):1- 28.
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji- Rong Wen. 2022. Dense text retrieval based on pre- trained language models: A survey. arXiv preprint arXiv:2211.1487.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolci Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 20..:3333.1823..
Vincent Van Asch and Walter Daelemans. 2010. Us- ing domain similarity for performance estimation. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 31-36.
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Ran- gan Majumder, Ji-Rong Wen, Nan Duan, et al. 2022. Simans: Simple ambiguous negatives sam- pling for dense text retrieval. arXiv preprint arXiv:2210.11773.
Vijaymeena and Kavitha. 2016. A survey on similarity measures in text mining, Machine Learn- ing and Applications: An International Journal, 3(2):19-22
Kun Zhou, Xiao Liu, Yeyun Gong, Wayne Xin Zhao, Daxin Jiang, Nan Duan, and Ji-Rong Wen. 2023. Master: Multi-task pre-trained bottlenecked masked autoencoders are better dense retrievers. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 630-647. Springer.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument with- out prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume Long Pupers), pages 241-251.
David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannanch Hajishirzi. 2020. Fact fiction: Veri- fying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7534-7550.
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2021. Gpl: Generative pseudo label- ing for unsupervised domain adaptation of dense trieval. arXiv preprint 1xXiv:211..07577
Wenjie Wang. Fuli Feng, Liqiang Nie, and Tat-Seng Chua. 2022. User-controllable recommendation against filter bubbles. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 1251-1261.
Lee Xiong, Chenyan Xiong, Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Aroold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning dense text retrieval. CoRR, abs/2007.00808.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2022. Laprador: Unsupervised pretrained dense retriever for zero-shot text retrieval. arXiv preprint arX1v:2003.06669.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset

---

A Appendix
Target (-)
TrivinQA
SearchQS
HotpotQA M@10 Res0
NO
Source (1)
Me
R
Me10 ReSI
M 10
R
A.1 Datasets
BMZ5
61.7
89.4
44.3
75.6
33.2
62.1
23.8
57.5
<TQA, Wikis
73.4
93.9
50.7
80.4
27.7
54.5
39.0
70.7
As mentioned Section 2.2, we divide the col- lected datasets into two categories: source datasets and target datasets, corresponding to in-domain training and out-of-domain evaluation, respec- tively. Note that source datasets can also be used for evaluation.
<SQA, Wikis
92.1
57.0
24.8
52.1
06.6
<BAA Wikls
64.2
91.3
47.1
78.1
34,9
64.4
36.6
67.7
<N0mg
Table 7: Results of models trained four MRQA datascts, where "Wiki" denotes "Wikipedia".
swering task contains question-answer pairs from financial data.
A.1.1 Source Datasets
For source datasets, collect six famous public datasets study various factors that may affect the transfer capability.
SciFact (Wadden et al.. 2020) is the task of scien- tific claim verification that selects scientific paper abstracts from the research literature to verify sci- tific claims.
Natural Questions (Kwiatkowski et al., 2019) is a dataset for open-domain QA originally, which contains queries in Google search, documents (Iong answers), and answer spans from the top- ranked Wikipedia pages.
BioASQ (Tsatsaronis et al., 2015) organizes chal- lenges on biomedical semantic indexing and ques- tion answering tasks. Semantic indexing task con- tains queries and corresponding documents.
MSMARCO (Nguyen et al., 2016) Passage Rank- ing dataset contains large number of queries with annotated passages in Web. Its queries are sam- pled from Bing search logs.
SciDoes (Cohan et al., 2020) is evaluation benchmark for the scientific domain, consisting of seven document-level tasks, including citation prediction, document classification, and recom- mendation.
MSMARCOv2 (Craswell et al., 2021) an en- hanced version of MSMARCO. It contains over 140 million passages and 11 million documents after being processed.
Quora (Lyer et al.) Duplicate Questions is a dataset with question pairs identify whether two questions are duplicates that convey the same se- mantic information.
HotpotQA (Yang et al., 2018) question answering dataset with multi-hop questions and Wikipedia-based question-answer pairs. Multi- hop reasoning is needed to answer the questions.
ArguAna (Wachsmuth et al., 2018) is a task of coumterargument retrieval that finds the best counterargument given an argument, consisting of argument-couue comttrrruett pairs from debatc website.
TriviaQA (Joshi et al., 2017) is reading com- prehension dataset consists of trivia questions with correct answers and evidences from the Web.
A.2 Experimental Setup
SearchQA (Dunn et al., 2017) is machine read- ing comprehension dataset containing question- answer pairs with snippets.
We adopted the joint training system of dense retrieval and re-ranking in RocketQAv2?, which leverages the dynamic listwise distillation for jointly training the dual-encoder-based dense re- triever and the cross-encoder-bbasee re-ranker.
A.1.2 Target Datasets
For target datasets, select and reuse six datasets used in BEIR (Thakur et al., 2021b) to cover as diverse scenarios as possible, including Bio- Medical, Finance, Misc., Quora, and Scientific domains, involving Information Retrieval, Ques- tion Answering. Aaplicate-Question Retrieval, Citation-Predictionn and Fact Checking tasks.
A minor modification made for the structure dif- ferent from RocketQAv2 is that we add point- wise constrain side by side with listwise constrain. since listwise score is not convenient for evaluat- ing the relevance score of single query-content pair. Through experiments, we find that this mod- ification does not affect the performance of joint
FiQA-2018 (Maia et al., 2018) a challenge in fi- nancial domain, where opinion-basse based question an-
Phtpps://o www..cco//eeeoe..ccooooooce.tcta www/ eeadd Rocket

---

Turget (-+)
NQ
MSMARCO
document sets on NQ and MSMARCO. Table 8 provides the evaluation results on NQ and MS- MARCO. We obtain similar findings as in Sec- tion 3.2 that the model trained on the merged doc- ument set does not outperform the model trained on Wikipedia on target datasets.
Source (1)
M@10
R
M@10
R@50
<NQ. Wikipedia>
60.7
86.6
21.5
64.3
<NQ. MMs
57.8
86.0
20.3
63.0
<NQ. Wikipedia+MMs
59.5
864
19.7
61.3
Table 8: Results of DR models trained on different source document sets.
A.6 Effect on Data Length
training system.
In addition to vocabulary overlap and query type, data length is a potential influencing factor on the data distribution, and we found that most of the source and target datasets do not differ much in the data length. Only special tasks such as Quora doc- ument set is shorter and ArguAna query is longer. We observe that the experimental results on these two datasets often differ from those on the other datasets (e.g., Figure ((e) and Figure 1(f),, and the specific impact of length itself on zero-shot perfor- mance is ambiguous.
For fair comparisons, we unify the experimen- tal settings as possible in all comparative experi- ments. We use the Adam optimizer (Kingma and Ba, 2015) with learning rate of 1e-5. We run the models up to 5 epochs with batch size of 192. All of our models are run on four NVIDIA Tesla V100 GPUs (with 32G RAM). To construct train- ing data, we use pre-trained DR model retrieve the top-k documents as candidates for each query in the training set. We sample undenoised in- stances through random sampling following Rock- etQAv2 and set the length of the instance list to 8 with ratio of the positive to the negative of 1:7.
A.7 Details in Document Scale Experiments
Training Details During the training process, we find that training on extreme large document set presents certain challenges. When we per- form random negative sampling on entire MS- MARCOv2 document set, we analyze the train- ing data and observe that the probability of mixing false negatives in it is much higher than that of the small document sets like 10% MSMARCOv2 doc- ument subset. We suspect that this is because the large document set may have more relevant con- tents for each topic, resulting in the serious prob- lem of unlabeled positives.
A.3 More Results of MRQA Trained Models
For futher study, we apply models trained on four source datasets in MRQA to these four datasets with different combinations. Table 7 represents the results evaluating on MRQA datasets. The zero-shot performances of models trained on four MRQA datasets are almost all better than BM25 when evaluated these four datasets separately, which shows that when there is not much discrep- ancy between the distribution of source and tar- get domain, the zero-shot capability of dense re- trieval is pretty good. We also find that the model trained on HotpotQA obtains the worst overall per- formance.
To overcome the obstacle, we propose a data cleaning strategy for the training data from the en- tire document set. Concretely, we check the re- call results on the 10% subset and find an average rank that roughly divides the relevant and irrele- vant passages. Then use the instances on the di- viding ranking position to locate rough dividing ranking position on large document set. We ig- nore the recalled documents with rank above the average dividing rank on such document set dur- ing random sampling.
A.4 Rules for Classifying Query Types
For queries starting with "WH" words, taking "what" as an example, queries with the first word of "whatt" or "what's" are classified as what-type queries. Queries starting with the first word "is", "was"", "are", "were", "do",, "does", "did", "have", "has", "had", "should", "can", "would", "could", "am", "shall". are classified as Y/N queries. The rest of the queries belong to declarative queries.
In-domain Evaluation Table 9 presents the zero- shot retrieval results with different document scales in the source training data. It seems that the model trained on a larger document set shows better retrieval capacity for in-domain evaluation. Moreover, we find that the performance decrease of DR model is not as significant as that of BM25
A.5 More Results for Source Document Set Effect
We also evaluate models that are trained using Wikipedia, MSMARCO and their combination as

---

Target (-))
MM+2
100%
MMv2
10%
MMv2
Source (1)
10
R050
M111
RØ50
M0:10 R6:00
BMZS
6.4
24.7
16.7
44.8
31.3
<MMv2, MMv2 100%
20.6
63.7
44.0
86.2
61.7
93.7
<MMv2, 1080
20.4
8600
62.0
<MMv2, MMv2 188
16.4
59.1
40.0
84.8
62.3
94.5
Table 9: Experiments with varying document scales on MSMARCOv2.
when increasing the document scale for evalua- tion, which is different from the findings of previ- ous work (Reimers and Gurevych, 2020). A pos- sible reason is that they utilize synthetic data to construct large document set, which have a large gap with the actual scene, and cannot be well ad- dressed by DR model.
In-depth Analysis Since MSMARCOv2 contains significantly more number of documents than MSMARCO, we would like to examine whether it leads a large expansion in vocabulary. We calculated the vocabulary overlap between MS- MARCO and MSMARCOV2 document sets at dif- ferent scales with the method in Section 3.2. We observe 99.2% overlap between "MSMARCOv2 10%" and full MSMARCOv2 document set and 93.8% overlap between "MSMARCOv2 1%" and full MSMARCOv2 document set. In addition, MSMARCO and MSMARCOv2 document sets have a overlap proportion of 79.1%. These find- ings show that MSMARCOv2 has similar vocab- ulary size with MSMARCO, which don't incorpo- rate substantial new topics. This may the rea- son for the poor zero-shot capability of the model trained on the large document set, since model is more likely to overfit the document set.


Reviews:
[{"review_id": "MRUe9wlZeg", "reviewer": "Reviewer_SQPY", "summary": "", "questions": "1. Do you have any solution to address the current lexical bias issue of evaluation?\n2. It is not very clear to me that how you construct NQ queries vs MARCO passages? If you use answer span to judge if a passage is relevant or not, there may be more than one relevant or no passages for each query. In the case of more than one relevant, you use all the passages as positive (this means you may include irrelevant passages)? In the case of no passage, you just ignore the queries?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback!\n\n\n#Reject reason 1: \n\nThis is a good suggestion. \n\nFirst of all, the main objective of this work is to conduct a comprehensive analysis of the factors that affect zero-shot dense retrievers and contribute reliable conclusions. There is presently no exhaustive exploration to disclose and summarize the zero-shot capabilities of dense retrievers. We believe our analysis is valuable and beneficial to the community.\n\nActually, based on the insights of the paper, we also examined some training strategies different from existing methods. For example, we attempted to generate queries using a T5-based model in the target domain to obtain a large set of queries. Then, based on the distribution of query types in the source domain, we constructed the same distribution in the training data of the target domain. The results show that controlling the query distribution led to better performance. We are considering incorporating them in the future version.\n\n#Reject reason 2: \n\nAlthough there are some zero-shot dense retrievers released, most of them are designed to improve on a specific aspect. The improvements of many existing methods can correspond to the influencing factors summarized in our paper. We believe that the value of this paper lies in the systematic analysis of the influencing factors, which is a more high-level summary. Moreover, as mentioned before, we are also considering increasing the discussion of our new training strategies.\n\n#Question 1: \n\nThe presence of lexical bias stems from how the dataset was annotated and is unrelated to the training method, which is a fairness issue. Thus, it is imperative to construct impartial datasets for fair comparisons in future research.\n\n#Question 2: \n\nFor each query, we only use one positive passage for training. When there are multiple positive passages, we use the passage with the highest score from a ranking model following RocketQAv2. We discard the query when there is no positive passage, but this rarely happens.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors conduct thorough experiments on the impact of query or document for dense retrieval training\n2. The conclusion paves way to training strategies, such as increasing query diversity (type and vocabulary overalp) rather than using larger documents. ", "reasons_to_reject": "1. It is pity that the authors could think about new training strategies given all the analysis rather than just reviewing existing methods.\n2. Although the experiments are thorough and give some good insights, in my opinion, the paper would be more valuable one year ago before all the other pre-training dense retrieval model published. With all the recent work, the insights provide by the paper may not seem to be very excited to me. Or the authors may provide a better training strategies given all the insights as mentioned.\n", "questions_to_authors": "1. Do you have any solution to address the current lexical bias issue of evaluation?\n2. It is not very clear to me that how you construct NQ queries vs MARCO passages? If you use answer span to judge if a passage is relevant or not, there may be more than one relevant or no passages for each query. In the case of more than one relevant, you use all the passages as positive (this means you may include irrelevant passages)? In the case of no passage, you just ignore the queries?", "paper_topic_and_main_contributions": "The paper conducts thoroughly investigation on the factors which might impact the zero-shot capability of dense retrieval. The paper conducts experiments on using different queries and document sets and also investigate the lexical bias of different datasets on different bias. I think the paper provides some good insights for future research. For example, query augmentation maybe the key to improve dense retrieval training and also mention the current lexical bias issue of evaluation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tDOkzELrMd", "reviewer": "Reviewer_8buS", "summary": "", "questions": "Please refer to Reasons To Reject.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback!\n\n#Reject reason 1: \n\nIt is a good question. When two datasets have an extremely high lexical overlap rate, they may be in the same domain. However, such a situation does not contradict the conclusion in our paper. Our conclusion is based on the comparison of the two overlap rates and is unrelated to their absolute values. Whether the vocabulary overlap rates are high or low, a relatively higher vocabulary overlap rate always leads to better zero-shot performance.\n\n#Reject reason 2: \n\nIt is also a good question. The first thing to note is that such an issue does not impact the conclusion. It is possible that a small number of queries may not be able to find positive passages among the candidate passages. However, this type of query makes up a tiny portion, less than 1%. We discard the query when there is no positive passage, which does not hurt the performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author's experiments and analysis have addressed the previously existing gap in detailed and comprehensive research on dense retrieval in a zero-shot setting.\n2. The author systematically analyzed various factors and performed corresponding experiments to illustrate the impact of different aspects of these factors on model performance. This analysis begins at a foundational level and progressively delves deeper.\n3. The author's conducted experiments are robust and offer substantial evidence to substantiate the author's perspective.\n4. The paper is well written and structured.\n", "reasons_to_reject": "1. The author highlights that the higher the vocabulary overlap between the source training set and the target test set, the better the model's performance in a zero-shot setting. However, with an exceedingly high overlap rate, do both datasets still meet the criteria for in-domain and out-of-domain?\n2. The author implemented certain allocations to control variables within the training set; however, these allocations appear to have certain issues. For instance, as mentioned in section 2.2, they utilized queries from NQ, while positive samples were extracted from the MSMARCO document set. Could this potentially result in certain queries lacking accurate answers?\n", "questions_to_authors": "Please refer to Reasons To Reject.", "paper_topic_and_main_contributions": "This paper mainly focuses on the study of dense retrieval in a zero-shot setting. Specifically, it identifies key factors influencing model performance by controlling a single variable and provides a detailed analysis of how these factors impact the performance. First, the authors analyze the impact of four variables on the DR model in a zero-shot setting, such as the source query set, source document set, data scale, and bias in the target dataset. Then, they conduct a detailed analysis of the specific factors within these variables. Additionally, the authors point out that bias in the target dataset may impact the evaluation. Finally, the authors also analyze which factors the existing technology improves performance by influencing.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kB1RwfWAnA", "reviewer": "Reviewer_19Ji", "summary": "", "questions": "1. Is there some way to use the conclusions drawn from this paper to improve performance in zero-shot setting?", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback!\n\n#Reject reason 1: \n\nTo the best of our knowledge, AR2, ColBERTv2, and RocketQAv2 are studies released around the same timeframe, with the longest time gap between their release dates being less than two months. \n\nFurthermore, the inconsistency among dense retrievers generally lies in the training process, while dual-encoder architectures are typically consistent. It will not affect the trend under different factor settings with such an architecture, and we have also examined it on ANCE (a well-known dense retriever). \n\nAnother advantage of using RocketQAv2 as the backbone model is that a well-trained re-ranker can also be obtained during training to easily facilitate the analysis of the re-ranker meanwhile.\n\n#Reject reason 2: \n\nWe understand that not all conclusions are exciting for everyone, but also have their values. Take your example as an illustration. We find that the model trained on MSMARCO has a stronger zero-shot capability. Therefore, people who require applying existing models in new domains can give priority considering models trained on MSMARCO, where our conclusion serves as a valuable application guideline, although it is not an important finding in our analysis.\n\nMoreover, for the community, there is presently no exhaustive exploration to disclose and summarize the law of zero-shot capabilities of dense retrieval models. We believe our analysis holds a unique and valuable contribution that is beneficial to the community.\n\n#Question 1:\n\nIt is a good question. Yes, besides the improvement of the existing methods mentioned in the paper can correspond to different factors, we also examined some training strategies based on our findings. For example, we attempted to generate queries using a T5-based model in the target domain to obtain a large set of queries. Then, based on the distribution of query types in the source domain, we constructed the same distribution in the training data of the target domain. The results show that controlling the query distribution led to better performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental evaluation is adequate and draws some useful conclusions.", "reasons_to_reject": "1. RocketQAv2 looks a bit dated. It would be better to adopt a newer model such as AR2 and ColBERTv2. It is not clear if these conclusions are applicable to other PLMs.\n2. Some of the conclusions weren't very exciting. For example, the model trained on MSMARCO dataset achieves better performance than NQ.", "questions_to_authors": "1. Is there some way to use the conclusions drawn from this paper to improve performance in zero-shot setting?", "paper_topic_and_main_contributions": "Since traditional sparse retrieval models (e.g., BM25) in a zero-shot retrieval setting perform better than dense retrieval (DR) model, this paper first presents a thorough examination of the zero-shot capability of DR models. This paper can help us to understand the zero-shot capacity of the DR model.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. ADVERSARIAL RETRIEVER-RANKER FOR DENSE TEXT RETRIEVAL, ICLR 2022\n2. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction, NAACL 2022", "Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 204, Furthermore, The ... ."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates dense retrieval in a zero-shot setting, with a focus on identifying factors that impact model performance. It examines four key variables: source query set, source document set, data scale, and bias in the target dataset, and provides an in-depth analysis of how these variables influence model performance. The paper also highlights the potential impact of bias on evaluation. Overall, it offers a comprehensive study of dense retrieval in a zero-shot context.

There are several strengths to this paper. It fills a gap in the research by providing a detailed and systematic analysis of dense retrieval in a zero-shot setting. The experiments conducted are robust and provide substantial evidence to support the findings. The paper is well-structured and well-written, making it accessible to the reader.

However, there are a few areas of concern. Reviewers suggests that the paper could have proposed new training strategies based on the insights gained from the analysis, rather than solely reviewing existing methods. Additionally, given the recent surge in pre-training-based dense retrieval models, the insights provided by the paper may not be as groundbreaking as they would have been a year ago. To address this, the authors might consider offering more innovative training strategies based on their findings.
Furthermore, there are questions raised regarding the allocation of queries from NQ and positive samples from the MSMARCO document set. This allocation could potentially result in queries without accurate answers, which should be addressed or clarified in the paper.
Decision: Accept-Findings