Towards Consistency Filtering-Free Unsupervised
Learning for Dense Retrieval⋆
Haoxiang Shi1,∗ , Sumio Fujita2 and Tetsuya Sakai1
12

Waseda University, 3-4-1 Ookubo, Shinjuku-ku, Tokyo, 169-8555, Japan
Yahoo Japan Corporation, Kioi Tower 1-3 Kioicho, Chiyoda-ku, Tokyo, 102-8282, Japan

Abstract
Domain transfer is a prevalent challenge in modern neural Information Retrieval (IR). To overcome
this problem, previous research has utilized domain-specific manual annotations and synthetic data
produced by consistency filtering to finetune a general ranker and produce a domain-specific ranker.
However, training such consistency filters are computationally expensive, which significantly reduces
the model efficiency. In addition, consistency filtering often struggles to identify retrieval intentions andrecognize query and corpus distributions in a target domain. In this study, we evaluate a more efficientsolution: replacing the consistency filter with either direct pseudo-labeling, pseudo-relevance feedback,or unsupervised keyword generation methods for achieving consistent filtering-free unsupervised dense
retrieval. Our extensive experimental evaluations demonstrate that, on average, TextRank-based pseudo
relevance feedback outperforms other methods. Furthermore, we analyzed the training and inference
efficiency of the proposed paradigm. The results indicate that filtering-free unsupervised learning cancontinuously improve training and inference efficiency while maintaining retrieval performance. In
some cases, it can even improve performance based on particular datasets.

Keywords
Neural Information Retrieval, Ad-hoc Ranking, Cross-domain Adaption, Consistency Filtering-free,
Unsupervised Learning

1. Introduction
Ad-hoc text ranking, which involves creating ordered lists of candidate documents from a large
static corpus in response to user queries, is a fundamental task in Information Retrieval (IR). On
the other hand, the Language models (LMs) have become the cornerstone of language tasks,
because of their extensive wide pre-trained knowledge [1, 2, 3, 4]. The application of an LMs
to ad hoc ranking has yielded significant performance improvements [5], which have driven a
paradigm shift [6] in IR. In this paradigm, the previous assumption was that domain transfer
could be generalized from one domain to all other domains. However, cross-domain retrieval
typically does not satisfy this assumption, limiting the performance.
To enhance domain adaptability, a two-stage approach has been widely adopted [7]. In
the first stage, a general ranker is trained on open-domain question–answer (QA) data pairs.
ReNeuIR’23: Workshop on Reaching Efficiency in Neural Information Retrieval, July 23-27, 2023, Taipei
We release codes at https://github.com/Haoxiang-WasedaU/Towards-Consistency-Filtering-Free-UnsupervisedLearning-for-Dense-Retrieval∗
Corresponding author.
Envelope-Open hollis.shi@toki.waseda.jp (H. Shi); sufujita@yahoo-corp.jp (S. Fujita); tetsuyasakai@acm.org (T. Sakai)⋆© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR

Workshop
Proceedings

http://ceur-ws.org
ISSN 1613-0073

CEUR Workshop Proceedings (CEUR-WS.org)

The solutions of the second stage include: (1) finetune the general ranker using manuallyannotated query–document pairs of the target domain and (2) generating synthetic data (i.e.,query–document pairs with noise) [8] in the target domain using a consistency filter for finetuning. However, both methods are inefficient during the second stage. Human labeling requires
domain expertise and is time-consuming and expensive. Additionally, the consistency filter
usually relies on an external question–answer model trained by QA pairs. The additional
training of the consistency filter is computationally expensive (for example, the T5 base modelhas 222.9M parameters)[9].
Furthermore, although the requirement of the labeled pairs is eliminated, the retrieval intents,
query and corpus distributions are not always well recognized, given a target domain. Dai et al.
[10] then showed that using a generator trained by a few of labeled query-document pairs in
the target domain can address these problems.
In this study, we aim at preserving the advantages of the two methods while circumventing
their disadvantages. This involves abandoning both the external dependency and the annotation
requirements, in addition to increasing the efficiency and meeting the real scenario in IR industry.
We aim to analyze whether a series of annotation-free and consistency filtering-free methods
are efficient and effective for improving the domain transfer ability of the general ranker.
We use three mainstream extractive methods on domain-specific documents as a replacement
for the filter in the second stage: (1) direct pseudo-labeling on the general ranker, (2) pseudorelevance feedback [11] (i.e., term frequency-inverse document frequency (TF-IDF) [12, 13, 14],
TextRank [15] and RAKE [16]) and (3) keywords generation using an unsupervised LM (i.e.,
KeyBERT [17]). We then feed the extractive synthetic data as the positive samples into an
unsupervised contrastive learning structure to finetune the general ranker [18]. The negative
samples are from BM25 retrieval or random selection.
We evaluate the performance of the extractive methods in the consistency filtering-freeframework for dense retrieval on two specific-domain IR datasets: the WWW-4 web searchdataset [19, 20] and the Robust04 News datasets [21]. The performance is measured using
four official evaluation metrics from the WWW-4 task [22]. The results suggest that utilizing
TextRank-based pseudo relevance feedback achieved better performance compared to Direct
Ranking and Vanilla BM25 across all metrics on average, despite the lack of statistical significance.
Through an analysis of the training and inference efficiency of the proposed paradigm, wefound that filtering-free unsupervised learning can continuously improve training and inference
efficiency while maintaining retrieval performance. In fact, in some cases, it can even improve
performance based on particular datasets.
We addressed the following research questions in this study:• Can cross-domain adaptation be realized using a consistency filtering-free unsupervised
approach?
• How does the sampling configuration affect the performance of domain-specific ranking?
• How efficient is our method in terms of training and inference compared with PROMPTAGATOR [10] and T5-based document expansion [23] by query prediction?

Target Domain Corpus

Open-domain Data
Answer Question
!!
"!
!"
""
……
!!
"!

Stage I

Unranked Q.

Q.
External
Pre-trained QA
ModelRankingLM

Unranked D.

LabeledPairs(", $)
Sy

Syn. Q

Encoder
DecoderQ.
D.-s.
G. Ranker Ranker
G. Ranker Ranking
G. Ranker
D.-s.
Stage II Ranker

Generator

QA Modelbased
Generator

LM
n.

Q

Figure 1: Mainstream domain adaptation paradigms in Dense Retrieval.

2. Related Work2.1. Dense Retrieval
In contrast to discrete bag-of-word matching, dense retriever embeds queries and documents
into a high-dimensional vector space and performs text matching in the embedding vector
space, overcoming the vocabulary mismatch problem of sparse retrieval [24].
Early dense retrievers utilize pre-trained LMs to encode queried and documents into biencoders to learn vector representations, via approximate nearest neighbor search [25]. Although this training strategy is more efficient, the performance is limited because only calculating the inner product between the latent representation of the query and document as aloss [5, 25]. To address this limitation, Khattab and Zaharia [26] proposed an interaction framework that increases the number of interactions between queries and documents by enabling
token-level interactions. Recent progress in the COIL framework by Gao et al. [27], combines
the exact matching and product of the latent presentations as the loss, thereby achieving both
efficiency and effectiveness.

2.2. Domain Transfer via Consistency Filtering
Domain adaptation remains a challenging topic in dense retrieval as current dense retrievalrankers are trained on open domain QA datasets, which provide limited insights into theirout-of-distribution generalization capabilities. To enhance domain adaptability, using in-domain
data to fine-tune a general ranker is the most effective method. However, owning to lack ofdomain-specific labeled data, various data augmentation methods were proposed, including
query augmentation [28] and document augmentation [29].
Among various query augmentation methods, the recently proposed technique of consistency
filtering has been shown to improve the quality of generated queries by guaranteeing roundtrip consistency; that is, the query must be answered by the passage it originated from [30].
In previous studies, to train the consistency filter, external QA models were preferred. For

Open-domain Data

Target Domain Corpus

Consistency Filtering Free
Unsupervised Alternatives

Unranked Q.

Direct Pseudo-labelingAnswer Question
!!
"!
!"
""
……
!!
"!

Pseudo-relevance Feedback
Unranked D.

LM

Unsupervised Keywords
Generation

Syn. Q

RankingStage I

General Ranker

Domain-specific Ranker
Stage II

Figure 2: The consistency filtering-free unsupervised dense retrieval paradigm.

instance, Lewis et al. [31] introduced probably asked questions to enhance the performanceof the closed-book QA (CBQA) model. To remove the QA models, Dai et al. [10] trained an
encoder-decoder as a consistency filter using a small group of in-domain query and document
pairs.

3. Experiment
Our task is to perform ad-hoc ranking on 𝐷 = {𝑞𝑖 , 𝑑𝑗 |𝑖 = 1, 2, ⋯ , 𝑛; 𝑗 = 1, 2, ⋯ , 𝑚}, where {𝑞𝑖 } and{𝑑𝑗 } represent the queries and documents, respectively. To accomplish this, we first fine-tune apre-trained language model (LM) using open domain question-answering (QA) data to obtain ageneral ranker R. We then further fine-tune the general ranker using a corpus specific to thetarget domain to obtain a domain-specific ranker R̃ based on R.

3.1. Method
In this method, we used the contextualized inverted list (COIL) framework [27] for both Stage I
and Stage II, because COIL is the most efficient in an exact match. As shown in Fig. 2, while
keeping Stage I unchanged, we modify the previous supervised cross-domain fine-tuning into
an unsupervised Stage II, and we adopt the widely-accepted contrastive learning loss in neuralIR, which can be calculated as follows:ℒ = − log

exp [𝑠(𝑞, 𝑑 + )]𝑚

exp [𝑠(𝑞, 𝑑 + )] + ∑𝑗 exp [𝑠(𝑞, 𝑑𝑗− )],(1)where 𝑠(⋅, ⋅) denotes the similarity which is the sum of the [ C L S ] product and the matching score.
The [ C L S ] product is the product of the [ C L S ] representations of a query and a document output
from the LM. The matching score is obtained by using each query token to look up its own
inverted list and computing the vector similarity with document vectors stored in the inverted

list, i.e., the contextualized exact lexical match. Previous methods distinguished a positive pairs
and a negative pairs by using human notations or consistency filtering. By contrast, we analyze
the performance of discriminating a positive pair (𝑞, 𝑑 + ) in the following ways.
Direct Pseudo-labeling. We introduce a corpus 𝐷 ∗ = {𝑞𝑘 , 𝑑𝑙 } that does not contain ranking
information but is relevant to 𝐷. We feed each 𝑞𝑘 ∈ 𝐷 ∗ into R to retrieve the top-𝑈 𝑑𝑙 as
positive pairs, which can be regarded as a pseudo-labeling of the retrieved documents by thecorresponding query. This method can be understood as a domain-relevant data augmentationand is a query-triggered process. In an information retrieval (IR) system, it is easy to collecta wider-domain 𝐷 ∗ that includes the domain of 𝐷, a narrow-domain 𝐷 ∗ that belongs to thedomain of 𝐷, or 𝐷 ∗ and 𝐷 can belong to the same wider domain.
Pseudo-relevance Feedback. Given a 𝑑𝑙 ∈ 𝐷 ∗ , we use the traditionally statistical methods
to generate the top 𝑉 keywords {𝑤𝑣 } of 𝑑𝑙 as the synthetic queries. Thus, ({𝑤𝑣 }, 𝑑𝑙 ) is the positive
pair. This approach is a document-triggered process and removes the requirements of 𝑞𝑘 ∈ 𝐷 ∗ .
As for the selected methods to be analyzed, because TF-IDF is a basic method, we only expand
the specific process of TextRank and RAKE.
• TextRank: The sentences in a document are parsed, and raw noun chunks and namedentities are identified. The sentences are then iterated through to construct a lemma
graph. Finally, PageRank, a form of eigenvector centrality, is used to calculate ranks foreach of the nodes, and the top-ranked phrases from the lemma graph are collected.
• RAKE: Keyword extraction begins with a document; the text is parsed into a set ofcandidate keywords. After every candidate keyword is identified and the lemma graph ofword co-occurrences is completed, a score is calculated for each candidate by summing
the scores for candidate member words. After all candidate keywords are scored, the top
𝑉 scoring candidates are selected as keywords for the document.
Unsupervised Keywords Generation. This is also a document-triggered method. First,
embeddings of a document 𝑑𝑙 ∈ 𝐷 ∗ are extracted using BERT to obtain a document-level
representation. Then, word embeddings are extracted for 𝑁-gram phrases. Finally, we use
cosine similarity to find the words/phrases that are the most similar to the document 𝑑𝑙 . The
most similar words {𝑤𝑣 } are then identified as the best description of the entire document.
Then, a negative sample (𝑞, 𝑑 − ) is generated in the following ways.
• Random: Given a pseudo-label or a synthetic 𝑞𝑘 , we randomly select 𝑚 documents in𝑑𝑙 ∈ 𝐷 ∗ .
• BM25: For the direct pseudo-labeling, given a query 𝑞𝑘 ∈ 𝐷 ∗ , we use the BM25 method toretrieve the inverted rank of the documents 𝑑𝑙 ∈ 𝐷 ∗ , and select the top 𝑚 as the negative
samples. Whereas, for the pseudo-relevance feedback and unsupervised keywords generation, a 𝑑𝑙 ∈ 𝐷 ∗ is replaced by a synthetic query {𝑤𝑣 }, and the remaining process is kept
the same.
The aforementioned methods use neither domain-specific labeled data nor additionally trained
collaborative filters. To verify the generalization ability, we evaluate these methods on different
couples for 𝐷 and 𝐷 ∗ .

3.2. Dataset and Baseline
We selected the following widely adopted large-scale ad hoc retrieval benchmarks from MSMARCO [32], which contained three million English documents with an average length ofapproximately 900 tokens, as the open-domain QA pairs in Stage I. To evaluate Stage II, We use
two couples of 𝐷 and 𝐷 ∗ , where the first couple is (𝐷: WWW-4, 𝐷 ∗ : WWW-2), and the second
couple is (𝐷: Robust04, 𝐷 ∗ : TREC Washington Post Corpus). WWW-2 and WWW-4 are boththe English web corpus [20]. WWW2 is constructed from ClueWeb12-B13, whereas WWW-4 isconstructed from Chuweb21 generated according to the April 2021 block of Common Crawl
dataset5. The TREC Washington Post Corpus [33, 34] contains news articles and blog posts
from January 2012 to December 2020. The Robust04 is generated from TREC disks 4 and 5,
containing news data of five media including Financial Times Limited et al. For WWW-2 andWWW-4 can be seen as the 𝐷 ∗ and 𝐷 are belong to the same wider domain, whereas for the
TREC Washington Post Corpus and Robust04 can be seen 𝐷 ∗ is in a narrower domain than 𝐷.
Our baselines are as follows.
• Direct Ranking: Only the COIL-based Stage I and no domain adaptation is conducted, andthe general ranker retrieves the document ranking of a query 𝑞𝑖 ∈ 𝐷.
• Vanilla BM25: Without the two-stage neural IR paradigm, the baseline retrieves the
document ranking of a query 𝑞𝑖 ∈ 𝐷.

3.3. Setup
Main Experiment. In Stage II, each positive sampling method and each negative sampling
methods are used in combination with each other (10 types in total). To enable a comparison
with direct labeling, we followed the COIL’s setting and fed one positive pair and 𝑚 = 7 negative
pairs, which were randomly selected and retrieved using the BM25 method to retrieve irrelevant
documents, into the general ranker. In direct pseudo-labeling, we selected the top 𝑈 = 5
and 𝑈 = 20 results from the WWW-2 and Washington Post datasets, respectively, to serve as
augmentation data for WWW-4 and Robust04. The token and [ C L S ] dimensions are 𝑛 = 768. A
document and a query are truncated to 128 tokens and 16 tokens, respectively. For the training,
the AdamW optimizer [35] with a learning rate of 2𝑒 − 6, 0.1 warm-up ratio, and linear learning
rate decay.
Oracle Experiments. We further analyze the effect of different 𝑈 on the performance. As
for pseudo-relevance feedback and unsupervised keywords generation, we further analyze theimpact of different 𝑉 on the performance of TextRank.

4. Results
4.1. Can cross-domain adaption be realized via consistency filtering-freeunsupervised approach?
The results of the main experiment experiments are presented in Tables 1 and 2. Each result is the average value of five runs. The following statements are based on average. The

n: Random
n: BM25

p: Pseudo-labeling
p: TF-IDF
p: TextRank
p: RAKEp: KeyBERT
p: Pseudo-labeling
p: TF-IDF
p: TextRank
p: RAKEp: KeyBERT
Direct Ranking
Vanilla BM25

nDCG@0010
55.60
58.86
60.79
56.08
57.27
58.71
57.05
56.36
58.22
58.13
57.50
51.70

WWW-4
Q@0010 nERR@0010
49.53
73.85
53.76
78.67
56.65
79.81
51.57
73.08
51.42
75.39
54.62
75.92
51.97
76.19
50.12
75.17
52.66
76.72
53.06
72.81
53.97
72.09
48.06
67.11

iRBU@0010
91.52
93.81
94.00
92.78
93.66
91.09
93.38
93.22
94.34
93.72
92.13
89.20

n: BM25

n: Random

Table 1
Experimental results on WWW-4, based on the COIL, where <p> and <n> refer to positive and negative
sampling, respectively.

p: Pseudo-labeling
p: TF-IDF
p: TextRank
p: RAKEp: KeyBERT
p: Pseudo-labeling
p: TF-IDF
p: TextRank
p: RAKEp: KeyBERT
Direct Ranking
Vanilla BM25

nDCG@0010
42.18
41.81
43.07
42.01
42.91
42.62
42.29
42.63
42.91
42.63
41.98
31.78

Robust04
Q@0010 nERR@0010
30.73
56.55
30.02
56.93
31.39
58.08
30.89
55.98
31.46
57.97
31.10
57.39
30.88
57.65
31.03
57.38
31.17
57.21
31.03
57.38
30.24
56.40
21.85
45.80

iRBU@0010
67.02
67.34
68.69
67.05
68.18
67.25
67.18
68.07
68.18
68.07
66.70
54.78

Table 2
Experimental results on Robust04, based on the COIL.

psuedo-labeling with BM25 negative sampling outperforms Direct Ranking in most of the metrics on both 𝐷. By contrast, psuedo-labeling with Random has a slight optimization effect on
Robust04, however, basically shows negative optimization on WWW-4. TF-IDF with Random
and BM25 show a random performance and negative optimizations for both the datasets, respectively. RAKE with both the negative sampling methods slightly outperforms Direct Ranking
on Robust04 (except for nERR@0010), however, the opposite trend was observed for WWW-4,

D.R.
D.R.
P.-l.&RND
P.-l.&BM25
V. BM25
TextRank&RND
TextRank&BM25
TF-IDF&RMD
TF-IDF&BM25
keyBERT&RND
keyBERT&BM25
RAKE&RND
RAKE&BM25

0.83790.434600.18080.6921
0.34840.59210.56050.46210.75880.4372P.-l.&
RND
0.83790.543300.23280.85030.44340.73310.69630.57530.91450.5488P.-l.&
BM25
0.4346
0.543300.56920.64210.90020.76580.80450.96520.61550.98

V. BM25
0
0
0
0
0
0
0
0
0
0
0

TextRank&
RND
0.18080.23280.569200.27370.63580.36440.393
0.54190.27720.5407

TextRank&
BM25
0.69210.85030.642100.27370.52830.86530.82330.67850.94110.6508RAKE&
RND
0.34840.44340.900200.63580.52830.65370.69290.8647
0.512
0.512

RAKE&
BM25
0.59210.73310.765800.36440.86530.65370.958
0.80270.81870.7791

KeyBERT&
RND
0.56050.69630.804500.393
0.82330.69290.958
0.84150.78040.8191

KeyBERT&BM25
0.46210.57530.965200.54190.67850.86470.80270.84150.649
0.9841

TF-IDF&
RND
0.75880.91450.615500.2772
0.94110.512
0.81870.7804
0.649

TF-IDF&BM25
0.4372
0.54880.98
00.54070.65080.512
0.77910.8191
0.98410.62340.6234

Table 3
Randomized Tukey HSD test p-values on WWW-4, where XXX&BM25 refers to using XXX and BM25
for positive and negative sampling, respectively; XXX&RND refers to using XXX and random for positive
and negative sampling, respectively; D.R. is direct ranking, V. BM25 is vanilla BM25, and P.-l. is
pseudo-labeling.

indicating a lack of generalization ability. TextRank with Random outperforms other methods,
and shows a stable performance for both the datasets, whereas the performance of TextRankwith BM25 deteriorates to be close to Direct Ranking. KeyBERT with BM25 outperforms thatwith Random for WWW-4 , and the opposite results were obtained for Robust04.
Statistical Significance Test. We use randomized Tukey HSD test p-values [36]. The differences between the runs (excepting BM25) are not statistically significant. The Tukey HSD test
𝑝-values are then shown in Table 3 and Table 4 for WWW-4 and Robust04, respectively. For
only a system pair containing vanilla BM25, 𝑝 < 0.05, whereas for other system pairs, all the𝑝 > 0.05, suggesting the improvements of our paradigms are not statistically significant.
These results indicate that it is feasible to remove the consistency filtering and domain-specificlabeling and use unsupervised contrastive fine-tuning to achieve the domain transfer ability of ageneral ranker. Its performance depends on the combination of positive and negative sampling
methods, suggesting that other unsupervised sampling methods are also worth exploring. In IR
industry scenarios where there are many domains and many documents in each domain, thisapproach involves a trade-off between performance and efficiency compared to introducing
complex and computationally expensive models. Possible reasons for the statistically insignificant improvements may be: (1) the query and corpus distributions are not significantly different
between MSMARCO and our selected two groups of 𝐷 and 𝐷 ∗ ; (2) the selected 𝐷 and 𝐷 ∗ arenot sufficiently strongly related in their corresponding domain; (3) the noise generated by the
positive and negative sampling methods is relatively large, and there is still room for further
optimization.

4.2. How sampling configuration impacts performance of domain-specificranking
We answer this question from three perspectives.
Impact on different of combinations of the sampling methods. It can be observed that
TextRank and pseudo-labeling exhibit opposite optimization effects under different negative
sampling methods.This could be attributed to the fact that for pseudo-labeling, the query 𝑞𝑖 ∈ 𝐷does not have any noise, and hence the non-correlation of the negative samples is higher under

D.R.
D.R.
V. BM25
P.-l.&RND
P.-l.&BM25
TextRank&RND
TextRank&BM25
RAKE&RND
RAKE&BM25
KeyBERT&RND
KeyBERT&BM25
TF-IDF
TF-IDF&BM25

V. BM25000.99590.76510.73570.86430.9981
0.78230.78280.86430.88330.9691

0
0
0
0
0
0
0
0
0
0

P.-l.&
RND
0.995900.89760.73370.86130.99780.78
0.78040.86130.88820.9652P.-l.&
BM25
0.765100.89760.653
0.91230.85430.76540.93450.87650.67540.8712

TextRank&
RND
0.735700.73370.653
0.8684
0.9511
0.9511
0.95240.86840.62890.7666

TextRank&
BM25
0.864300.86130.91230.8684RAKE&
RND
0.998100.99780.85430.73670.86390.91630.91670.91630.87320.75150.8956

0.7832
0.78320.86390.88640.9676RAKE&
BM25
0.782300.78
0.7654
0.95110.91670.7829KeyBERT&
RND
0.782800.78040.93450.95240.91630.78320.99

0.99
0.91670.673
0.8136

KeyBERT&BM25
0.864300.86130.87650.86840.75150.86390.91670.91630.91630.674
0.81380.75150.8956

TF-IDF&
RND
0.883300.88820.67540.62890.8956
0.88640.673
0.674
0.7515

TF-IDF&BM25
0.969100.96520.87120.7666
0.96760.81360.81380.89560.89560.8956

Table 4
Randomized Tukey HSD test p-values on Robust04.
100

nDCG@0010
iRBU@0010

Score

80

Q@0010nERR@0010

60
40
20
0

50
Top U

100

0

50
Top U

100

Figure 3: The retrieval performance vs. different 𝑈 in pesudo-labeling using augment datasets WWW-2
and Washington Post on (a) WWW-4 and (b) Robust04.

Score

100

V=3

U=10

50

0
nDC

Q@0 nERinDC iRBU Q@0 nER
R @0
G@0 @00 010 R@0
G@R0BU@00
0
010
010
010
010
10
10 10

Figure 4: The retrieval performance vs. different 𝑉 in TextRank on (a) WWW-4 and (b) Robust04.

the BM25 retrieval method compared to random sampling. This is because the latter method
may introduce false negative samples. In contrast, for TextRank, the extracted keywords serveas the synthetic query, which has noise and is more susceptible to false negative samples thanrandom sampling. On the other hand, TextRank avoids the influence of open domain QA pairs
on retrieval intents, query and document distribution in the target domain. We demonstrate that
TextRank has better positive sampling ability than TF-IDF, RAKE, and KeyBERT. This leads us toconclude that minimizing the positive sampling noise and improving the non-correlation of thenegative samples is a substantial factor for achieving successful domain adaptation performance.
Impact on different 𝑈 in pseudo-labeling. As shown in Fig. 3 (a) and (b), we find that inboth 𝐷, the domain-adaptive performance deteriorates when 𝑈 is not the most suitable 𝑈𝑠 . This𝑈 cannot be too large, since too many documents of the same query increases the noise of thepositive samples. Additionally, the performance on WWW-4 deteriorates more substantiallythan on Robust04 when 𝑈 > 𝑈𝑠 , suggesting that WWW-4 is more sensitive to this noise than
Robust04. Therefore, it is important to determine the optimal value of 𝑈 for pseudo-labeling toachieve good results.
Impact on different 𝑉 in TextRank.: We take TextRank, which has the best performance,
as a representative to analyze the impact of the keyword list length on Stage II. As shownin Fig. 4 (a) and (b), the domain-adaptation performance deteriorates as 𝑉 is increased, onboth 𝐷. This is because increasing the length of the may increase the noise of the synthetic
query, thereby increasing the positive sampling noise. The results of adjusting both 𝑈 and𝑉 simultaneously suggest the importance role of minimizing the positive sampling noise for
ensuring good domain-adaptive performance.

4.3. How our methods are efficient in terms of training and inferenceThis study compares the training and inference efficiency of our consistency filtering-freeunsupervised paradigm with PROMPTAGATOR [10] and T5-based document expansion by
query prediction [23]. Specifically, for model training, Compared to PROMPTAGATOR [10],the proposed paradigm avoids the additional use of the T5 encoder (i.e., to be trained as theconsistency filter), thereby achieving a reduction of 50% in total parameter count. Compare with
to full T5 model-based document expansion by query prediction, the present paradigm results
in a reduction of 67.8% in parameter count. Moreover, training a data generator with labeleddata easily causes under-fitting, which suggests that the consistency filtering-free paradigm
can solve such the problems.
With regard to model inference for query production (or keyword extraction in the proposed paradigm), TF-IDF-, TextRank- and RAKE-based consistency filtering-free unsupervised
paradigms eliminate the need for an additional pre-trained LM, thereby resulting in faster inference speed compared with the both PROMPTAGATOR and full T5-based document expressions
using the query predictions. Whereas, the pseudo-labeling- and KeyBERT-based paradigms
rely on BERT model, thus the inference speed tends to be the same with the PROMPTAGATOR
inference. However, the inference speed is till faster than the full T5 model-based inference. We
also conduct an inference test on an Nvidia GTX3090 GPU card, and observe a 2.1× accelerationin average when using TextRank-based paradigm, compared with the full T5 model-based
paradigm.

4.4. Failure analysis
Based on the results of a static significance test, our proposed methods did not show significant
improvements. In light of these findings, we conducted an in-depth analysis of several bad cases
to investigate potential areas for improvement. Specifically, we evaluated the performance of1.0

WWW4-Direct Labeling
WWW4-TextRank

0.8
0.6
0.4
0.2
0
0

5

10

15

20

25

30

35

40

45

50

Figure 5: The nDCG score by topics on the WWW4 dataset.

Good retrieval topics
Bad retrieval topics

QID
0208
0210
0232
0202
0211
0236

Content
cultural appropriation cases
hypothermia treatment
chrome download windows
New Orleans restaurantssocial anxiety
tennis score rules

Table 5The top 3 queries demonstrate superior and inferior nDCG@0010 scores in TextRank-based feedback
compared to Direct Labeling retrieval performance.

our TextRank-based pseudo relevance feedback method and direct ranking by topics on the
WWW4 dataset, using the nDCG@0010 scores of TextRank-based feedback. Based on theseresults, we identified the top 3 queries that demonstrated superior and inferior nDCG@ 0010
scores in TextRank-based feedback compared to Direct Labeling retrieval performance. Table 5
presents our findings, which suggest that the TextRank-based method can outperform directlabeling for queries that are likely to retrieve clear answers.

5. Conclusion
In this paper, we aim to eliminate the need for consistency filtering and manual labeling inthe second stage of the previous two-stage methods, to achieve more efficient and low-costdomain adaptation in neural IR. We analyze three positive sampling methods combined with
two negative sampling methods through unsupervised contrastive fine-tuning of the generalranker on the COIL framework.
The results of our study suggest that positive sampling using TextRank with Random negative
sampling outperforms the other methods on average, although these differences in performance
are not statistically significant. These findings suggest that the consistency filtering-free unsupervised paradigm may have some ability for domain adaptation that relies on actual datasets.
Additionally, we find that during unsupervised fine-tuning of the domain-specific ranker inthe second stage, it is a important factor to minimize the positive sampling noise and enhancethe non-correlation of the negative samples in order to improve domain-adaptive ability. We
additionally analyze the training and inference efficiency of the proposed paradigm. In training,
the proposed paradigm saves 50% and 67.8% parameters, compared with PROMPTAGATOR andT5 model-based document expansion by query prediction, respectively. In inference, average
2.1× acceleration has been tested compared to the full T5 model-based paradigm.

6. Acknowledgment
We sincerely thank Dr. Cen Wang, Mr. Jiaan Wang and Mr. Qijiong Liu for their feedback on
our paper. Additionally, we would like to extend our thanks to the Sakai Lab from Waseda
University for providing computational resources.

References[1] J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, J. Zhou, Is chatgpt a good nlg
evaluator? a preliminary study, arXiv preprint arXiv:2303.04048 (2023).
[2] J. Wang, F. Meng, D. Zheng, Y. Liang, Z. Li, J. Qu, J. Zhou,
A Survey on
Cross-Lingual Summarization,
Transactions of the Association for Computational Linguistics 10 (2022) 1304–1323. URL: https://doi.org/10.1162/tacl_a_00520.
doi:1 0 . 1 1 6 2 / t a c l _ a _ 0 0 5 2 0 .
arXiv:https://direct.mit.edu/tacl/articlepdf/doi/10.1162/tacl_a_00520/2062157/tacl_a_00520.pdf.
[3] J. Wang, F. Meng, Z. Lu, D. Zheng, Z. Li, J. Qu, J. Zhou, ClidSum: A benchmark dataset forcross-lingual dialogue summarization, in: Proceedings of the 2022 Conference on Empirical
Methods in Natural Language Processing, Association for Computational Linguistics, Abu
Dhabi, United Arab Emirates, 2022, pp. 7716–7729. URL: https://aclanthology.org/2022.
emnlp-main.526.
[4] J. Wang, F. Meng, D. Zheng, Y. Liang, Z. Li, J. Qu, J. Zhou, Towards unifying multi-lingualand cross-lingual summarization, arXiv preprint arXiv:2305.09220 (2023).
[5] R. Nogueira, K. Cho, Passage re-ranking with bert, arXiv preprint arXiv:1901.04085 (2019).
[6] J. Lin, R. Nogueira, A. Yates, Pretrained transformers for text ranking: BERT and Beyond,
Synthesis Lectures on Human Language Technologies 14 (2021) 1–325.
[7] M. Guo, Y. Yang, D. Cer, Q. Shen, N. Constant, Multireqa: A cross-domain evaluation for
retrieval question answering models, arXiv preprint arXiv:2005.02507 (2020).
[8] C. Alberti, D. Andor, E. Pitler, J. Devlin, M. Collins, Synthetic qa corpora generation withroundtrip consistency, arXiv preprint arXiv:1906.05416 (2019).
[9] H. Shi, R. Zhang, J. Wang, C. Wang, Y. Zheng, T. Sakai, Layerconnect: Hypernetworkassisted inter-layer connector to enhance parameter efficiency, in: Proceedings of the 29th
International Conference on Computational Linguistics, 2022, pp. 3120–3126.
[10] Z. Dai, V. Y. Zhao, J. Ma, Y. Luan, J. Ni, J. Lu, A. Bakalov, K. Guu, K. B. Hall, M.-W. Chang,
Promptagator: Few-shot dense retrieval from 8 examples, arXiv preprint arXiv:2209.11755(2022).
[11] S. Robertson, H. Zaragoza, et al., The probabilistic relevance framework: BM25 and beyond,Foundations and Trends® in Information Retrieval 3 (2009) 333–389.

[12] K. Spärck Jones, IDF term weighting and IR research lessons, Journal of documentation 60(2004) 521–523.
[13] G. Salton, Automatic text processing: The transformation, analysis, and retrieval of,
Reading: Addison-Wesley 169 (1989).
[14] K. Lang, Newsweeder: Learning to filter netnews, in: Machine learning proceedings 1995,
Elsevier, 1995, pp. 331–339.
[15] R. Mihalcea, P. Tarau, Textrank: Bringing order into text, in: Proceedings of the 2004
conference on empirical methods in natural language processing, 2004, pp. 404–411.
[16] S. Rose, D. Engel, N. Cramer, W. Cowley, Automatic keyword extraction from individual
documents, Text mining: applications and theory (2010) 1–20.
[17] P. Sharma, Y. Li, Self-supervised contextual keyword and keyphrase retrieval with selflabelling (2019).
[18] H. Shi, C. Wang, Self-supervised document clustering based on bert with data augment,
arXiv preprint arXiv:2011.08523 (2020).
[19] T. Sakai, S. Tao, Z. Zeng, Y. Zheng, J. Mao, Z. Chu, Y. Liu, M. Maistro, Z. Dou, N. Ferro,
et al., Overview of the NTCIR-15 we want web with centre (WWW-3) task, Proceedingsof NTCIR-15. to appear (2020).
[20] J. Mao, T. Sakai, C. Luo, P. Xiao, Y. Liu, Z. Dou, Overview of the NTCIR-14 we want web
task, Proceedings of NTCIR-14 (2019) 455–467.
[21] E. M. Voorhees, et al., Overview of the trec 2003 robust retrieval track., in: Proceedings of
NTCIR-4, 2003, pp. 69–77.
[22] T. Yamamoto, Z. Dou, Overview of ntcir-16, in: Proceedings of the 16th NTCIR Conference
on Evaluation of Information Access Technologies, 2022, pp. 3–7.
[23] R. Nogueira, W. Yang, J. Lin, K. Cho, Document expansion by query prediction, arXiv
preprint arXiv:1904.08375 (2019).
[24] H. Li, S. Wang, S. Zhuang, A. Mourad, X. Ma, J. Lin, G. Zuccon, To interpolate or not tointerpolate: PRF, dense and sparse retrievers, in: Proceedings of the 45th International
ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp.
2495–2500.
[25] V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W.-t. Yih, Dense
passage retrieval for open-domain question answering, arXiv preprint arXiv:2004.04906(2020).
[26] O. Khattab, M. Zaharia, Colbert: Efficient and effective passage search via contextualizedlate interaction over bert, in: Proceedings of the 43rd International ACM SIGIR conference
on research and development in Information Retrieval, 2020, pp. 39–48.
[27] L. Gao, Z. Dai, J. Callan, COIL: Revisit exact lexical match in information retrieval
with contextualized inverted list, in: Proceedings of the 2021 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Association for Computational Linguistics, Online, 2021, pp. 3030–3042. URL:
https://aclanthology.org/2021.naacl-main.241. doi:1 0 . 1 8 6 5 3 / v 1 / 2 0 2 1 . n a a c l - m a i n . 2 4 1 .
[28] J. Ma, I. Korotkov, Y. Yang, K. Hall, R. McDonald, Zero-shot neural passage retrieval via
domain-targeted synthetic question generation, arXiv preprint arXiv:2004.14503 (2020).
[29] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu,
Exploring the limits of transfer learning with a unified text-to-text transformer, The

Journal of Machine Learning Research 21 (2020) 5485–5551.
[30] C. Alberti, D. Andor, E. Pitler, J. Devlin, M. Collins, Synthetic QA corpora generation withroundtrip consistency, in: Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, Association for Computational Linguistics, Florence, Italy,
2019, pp. 6168–6173. URL: https://aclanthology.org/P19-1620. doi:1 0 . 1 8 6 5 3 / v 1 / P 1 9 - 1 6 2 0 .
[31] P. Lewis, Y. Wu, L. Liu, P. Minervini, H. Küttler, A. Piktus, P. Stenetorp, S. Riedel, Paq: 65
million probably-asked questions and what you can do with them, Transactions of the
Association for Computational Linguistics 9 (2021) 1098–1115.
[32] N. Craswell, B. Mitra, E. Yilmaz, D. Campos, J. Lin, Ms marco: Benchmarking ranking
models in the large-data regime, in: Proceedings of the 44th International ACM SIGIR
Conference on Research and Development in Information Retrieval, 2021, pp. 1566–1576.
[33] A. Bondarenko, M. Hagen, M. Völske, B. Stein, A. Panchenko, C. Biemann, Webis at TREC
2018: Common core track., in: TREC, 2018.
[34] I. Soboroff, S. Huang, D. Harman, TREC 2018 news track overview., in: TREC, volume 409,
2018, p. 410.
[35] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint
arXiv:1711.05101 (2017).
[36] T. Sakai, Laboratory experiments in information retrieval, The information retrieval series
40 (2018).