Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning
Authors: Che Wei Chen
Ching Wen Yang
Chun-Yi Lin
Hung-Yu Kao
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Retrieval
Domain adaptation
Denoise-finetuning
Unsupervised

Abstract: Dense retrieval models have exhibited remarkable effectiveness, but they rely on abundant labeled data and face challenges when applied to different domains. Previous domain adaptation methods have employed generative models to generate pseudo queries, creating pseudo datasets to enhance the performance of dense retrieval models. However, these approaches typically use unadapted rerank models, leading to potentially imprecise labels. In this paper, we demonstrate the significance of adapting the rerank model to the target domain prior to utilizing it for label generation. This adaptation process enables us to obtain more accurate labels, thereby improving the overall performance of the dense retrieval model. Additionally, by combining the adapted retrieval model with the adapted rerank model, we achieve significantly better domain adaptation results across three retrieval datasets. We release our code for future research.

Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Senoise-Finetunine
Che- Wei Chen', Ching- Ching-Wen Wen Yang", Chun-Y Lin', Hung- Yu Kao"
Institute of Medical Informatics'
Department of Computer Science and Information Engineering
National Cheng Kung University, Tainan, Taiwan (Q56104076, P76114511, NE6101 @5g ncku. edu. tw hykaoêmail nckku edu. tw
Abstract
Top-N relevant passuges
Ranked
pissages
Dense retricval models have exhibited remark- able effectiveness, but they rely on abundant labeled data and face challenges when applied different domains. Previous domain adapta- tion methods have employed generative models generate pseudo queries, creating pseudo datasets to enhance the performance of dense retrieval models. However, these approaches typically use unadapted rerank models, leading potentially imprecise labels. In this paper, we demonstrate the significance of adapting the rerank model the target domain prior to utiliz- ing it for label generation. This adaptation pro- cess enables us to obtain more accurate labels, thereby improving the overall performance of the dense retrieval model. Additionally, by combining the adapted retrieval model with adapted rerank model, achieve significantly better domain adaptation results across three re- trieval datasets. We release our code for future research."
Tep
Query
Rexioval modd
Pp?
Ceepa
Figure I: Two-stage information retrieval involves using a retrieval model to select small subset passages, which then reranked using rerank model.
(Karpukhin et al., 2020; Xiong et al., 2021) the mainstream approach in information retrieval (IR).
This approach effectively addresses the issue of missing words encountered lexical methods like BM25 (Robertson et al., 1995) or TF-IDF (Chris- tian et al., 2016) and has powerful retrieval ability. However, the introduction of the BEIR Bench- mark (Thakur et al., 2021) has highlighted the limi- tations of dense models. Surprisingly, traditional lexical-based models like BM25 have exhibited superior performance compared to dense retrieval models in out-of-domain scenarios. Additionally, training dense retrieval models from scratch ne- cessitates a significant amount of domain-specilfe training data, posing challenges in collecting large- scale data for each specific domain. As a result, there is a growing demand for domain adaptation methods that can enhance model performance with- out relying on labeled data.
1 Introduction
The goal of information retrieval (IR) is enable users input query and retrieve relevant passages documents from the retrieval system. standard IR system (Matveeva et al., 2006; Liu et al., 2009; Wang et al., 2011; Yang et al., 2019) typically com- prises two main stages (refer to Fig. l):
1. First-stage retrieval model: This model is de- signed to retrieve small subset of relevant passages based on the given query.
Previous unsupervised domain adaptation ap- proaches, such as Qgen (Ma et al., 2021) and GPL (Wang et al., 2021b), have employed generative models to generate pseudo queries and augment the training data. Qgen and GPL utilized fine- tuned rerank model (Nogueira and Cho, 2019) on the MSMARCO dataset (Nguyen al., 2017) as- sign relevance scores to the generated queries and passages, creating a pseudo dataset for training the
2. Rerank model: Responsible for reordering the retrieved passages, the rerank model aims to enhance the overall user experience.
Recent advancements in contextualized word em- beddings (Liu et al., 2019; Devlin et al., 2019; Lewis et al., 2020) have established dense retrieval
httas://ggthuu.ccm/rrii///////// http:////mmmmmeeccc5cee///

---

dense retrieval model. However, they did adapt the rerank model the target domain, resulting in potential label imprecision and failure to improve the two-stage retrieval score.
is widely adopted approach that combines the strengths of retrieval models and rerank models for effective information retrieval. has emerged as the preferred pipeline for competitive IR compe- tition tasks (Lassance, 2023; Huang et al., 2023; Zhang et al., 2023).
In this paper, we present novel technique for unsupervised domain adaptation (Fig.2). Our ap- proach focuses on simultaneously adapting the dense retrieval model and rerank model within two-stage retrieval system. The process begins by generating pseudo queries for target domain pas- sages, treating them as relative pairs. We employ sparse and dense negative miners to identify chal- lenging negatives and create pseudo dataset by combining queries with relevant and irrelevant pas- sages. The rerank model is then fine-tuned using this dataset.
Rerank models, such as cross-encoder (Nogueira and Cho, 2019) and mono-T5 (Nogueira al., 2020), demonstrate exceptional performance com- pared to retrieval models. However, their demand- ing computational requirements (Khattab and Za- haria, 2020) restrict their practical application to reranking a limited subset of passages.
In this study, we adapt the BERT base retrieval model (Devlin al., 2019) using the bi-encoder architecture (Reimers and Gurevych, 2019), while the rerank model implemented using the cross- encoder architecture (Nogueira and Cho, 2019). The bi-encoder serves as the initial retrieval model, where we input a query Q and passage P to obtain their respective mean-pooled output vectors E()) and EP). The similarity score be- tween the query and passage is computed by taking the dot product of these embeddings (Eq.1). To optimize efficiency, we pre-compute and store the passage embeddings, allowing for efficient retrieval by encoding the query input and calculating simi- larity scores.
Since this dataset is not labeled by human, it is likely to contain noisy labels. Therefore, we pro- pose a denoise-finetuning approach that incorpo- rates random batch warm-up and co-regularization learning. Our experimental results validate the ef- fectiveness of this approach in alleviating the im- pact of noisy labels.
In our domain adaptation approach, we utilize models that have been fine-tuned on the exten- sive MSMARCO (Nguyen al., 2017) IR dataset, which serves as a strong foundation for adapting to various domains. Remarkably, our approach outper- forms previous domain adaptation methods both one-stage and two-stage retrieval tasks, showcasing superior performance.
Sim
((Q))EEP)
(I)
In summary, our pipeline offers the following contributions:
On the other hand, the cross-encoder model treats reranking as a binary classification task. involves tokenizing each query and passage, denoted as (")) ........ ((()) and p()),... p respectively. The input sequence then is formed as follows:
Full adaptation of both the rerank model and dense retrieval model without requiring target domain labeling data.
Successful training of the rerank model on the generated dataset, effectively mitigating the in- fluence of noisy labels through random batch warm-up and co-regularization techniques.
[[CLS], "
("), (SEP),p()
(m))
(2)
This sequence is subsequently fed into the BERT model, where the [CLS] embedding is utilized as the input for single neural network. Cross- encoder exhibits strong performance but has higher latency than the bi-encoder (Hofstätter and Han- bury, 2019). Hence, it is more suitable choice to pair with the sirst-stuge retrieval model.
Transfer of domain domain-adapted adapted knowledge from the rerank model the dense retrieval model using knowledge distillation (Hofstätter et al., 2020), leading to significant performance improvements in the final two-stage retrieval.
2.2 Query Generation
2 Related Works
In previous research, docT5query (Nogueira al., 2019) 9) fine-tuned the T5 (Raffel al., 2020) model as a query generator using labeled pairs from the MSMARCO (Nguyen et al., 2017)
2.1 Two-stage Retrieval
Two-stage retrieval (Matveeva et al., 2006; Liu et al., 2009; Wang et al., 2011; Yang et al., 2019)

---

Source domain
Target domain
Denoise finetuning
Rerank model
Rerank model
Pseudo dataset
Distillation
Retrieve irrelevant passages
P
Negative miner
Q
p
Dense retrieval model
p
Q
p
Generate query
Passages
Query generator
p
Target domain corpus
Q
pt
Our adaptation pipeline transfers the rerank model and dense retrieval model from the source domain to the target domain. The process begins with the generation of ascudo-training dataset, which is then used to fine-tune the rerank model. And then it employs the denoise fine-tuning technique to mitigate the influence of noisy labels. Finally, the knowledge acquired from the rerank model is distilled into the dense retrieval model, completing the adaptation process.
Figure 2:
dataset. Their method involved generating ques- tions based on passages and concatenating them with the passages themselves, leading to improved retrieval performance for lexical-based methods.
model to the dense retrieval model.
3.1 Generation Pseudo Training Dataset
To generate pseudo queries for the passages in a given corpus, employ pre-trained T5 model trained the MSMARCO dataset. To promote diversity, we generate three pseudo queries for each passage, resulting in total of approximately 300K pscudo queries. For datasets with more than 100K passages, we randomly select 100K passages and generate three pseudo queries for each selected passage. The training data for the target domain comprises around 300K (pseudo query, passage) pairs, denoted Dpscudo {(Q,, P.)};-
In the context of domain adaptation, previous methods QGen (Ma et al., 2021) and GPL. (Wang et al., 2021b) utilized the generated questions as training data to enhance the dense retrieval model. These methods varied in their labeling approaches. with QGen using binary labels and GPL, utilizing a cross-cncoder (Nogucira and Cho, 2019) to assign soft labels to
pairs.
Building upon the work of GPL, we further re- fined the performance of the cross-cncoder model in the target domain. By leveraging the adapted cross-encoder, achieved more precise label- ing, resulting in improved performance of the dense retrieval model. Additionally, we employed a two-stage retrieval strategy by combining the adapted cross-encoder with the adapted retrieval model. This innovative approach led to significant 10% enhancement in NDCG@ 10 across all three datasets, surpassing the performance of previous methodologies.
3.2 Cross-Encoder (Rerank Model) Adaptation
The cross-encoder (Nogueira and Cho, 2019) treats the reranking of passages as a binary classification task. To fine-tune it for the target domain, the train- ing data should consist of (query, passage, label) triplets, where the label is either cate relevance or irrelevance. After obtaining the generated training data Dpsendo = {(2,, P.) we apply a filter to exclude queries which is shorter than 5 words. This filtering is necessary because we noticed that shorter queries often inquire about the meaning of word in the format "What is xxx", which does not align with the distribution of the queries in our testing datasets. For a clearer exam- ple, please refer to Appendix A.
0 indi-
3 Methodology
Our method, illustrated in Fig.2, achieves do- main adaptation for both the retrieval model and rerank model components through three steps: (1) constructing a pseudo training set, (2) denoise- finetuning the rerank model for domain adaptation, and (3) distilling the knowledge from the rerank
We leverage a cross-encoder trained on the MS-

---

Random batch warm-up
Co-regularization learning
Laabel
model
Mini batch
MY WAR
Mini batch
Rerank model
Ground truth
Label
Mini butch
Mini batch
M wic
Remank model
Rerank model
Ag8 prediction
Lagg
α %
Training steps
Figure 3: Denoise finetuning: In the random batch warm-up phase, train two separate cross- cross-encoder models using different data distributions. In the subsequent co-regularization learning stage, these models are jointly optimized with task-specific losses and regulated through an agreement loss. This regularization technique ensures consistent predictions and helps alleviate the influence of noisy labels, effectively preventing overfitting.
MARCO (Nguyen al., 2017) dataset assign scores to each (query, passage) pair in Dpscudo From these pairs, we extract the top 100k (Qi, P.) with the highest cross-encoder prediction scores positive training samples, forming the positive dataset D pos = {Qi, P., 1)0,00k
generator, (2) false positive passages resulting from imprecise reranking of the cross-encoder, and (3) false negative documents drawn from the top- 1000 negative pools.
To address the issue of potentially noisy labels, we propose learning approach called denoise- finetuning (Fig.3). In our experiments, this ap- proach has been proven to effectively mitigate the impact of incorrect labels.
To obtain challenging irrelevant passages, we utilize two retrieval approaches: (1) BM25 and (2) a bi-encoder trained on the MSMARCO dataset. For each query, we retrieve the sop-1000 similar passages using both models. From the retrieval pools of these two models, randomly select one passage from each pool, resulting in two negative samples. This process allows to create the nega- 200 tive training dataset Dneg = [(Q,, P., 0))?
Taking inspiration from common practices in data science, such N-fold validation and model ensemble, our objective for the models to learn from diverse data distributions in the initial stages and subsequently mutually correct each other dur- ing the later training phase.
By combining the positive samples from Ppos and the negative samples from Dneg. construct the final training dataset for the cross-encoder, de- noted as Dee Dneg pos- The cross-encoder is trained to predict the probability of relevance between the query and passage pairs, guided by the provided labels the training data.
We initialize two cross-encoder models {CE,)2 and employ linear warm-up as the learning rate scheduler. We train our models for T steps, this scheduler gradually increases the learning rate until it reaches its maximum value at T a%, where a is hyperparameter within [0, 100), and subsequently decreases it. During the first a% of training steps, we randomly discard batches with probability of P to allow the two models to learn from different data distributions.
This selection process of positive and negative samples ensures that the cross-encoder learns to discriminate between relevant and irrelevant query- passage pairs, enhancing its capability to capture the semantic relationship between queries and pas- sages.
During the later stage of training, we utilize an adapted version of the co-regularization method proposed by Zhou and Chen (2021). This method is inspired by previous studies (Arpit et al., 2017; Toneva et al., 2018) that highlight the delayed learn- ing curves associated with noisy labels. This ap- proach enables both models learn from the batch data as well their combined predictions. In this phase, the models benefit not vided labels but also from the aggregated predic-
3.3 Denoise Finetuning
However, it is probable that the dataset Der {(Q,, v.)) Wi E {0,1) contains noisy labels, which can negatively impact the model's perfor- mance (refer to Figure.4). These labels can come from (1) low-quality queries generated by the query
only from the pro-

---

tions. The influence of co-regularization is con- trolled by the hyperparameters 7. The total loss (Eq. 3) consists of two components: Lladel, which represents the loss based on the provided labels, and Lagg, which reflects the loss derived from the joint predictions.
Algorithm 1: Denoise Finetuning
Input: Dataset Dee - {(Qi, hi):
hyperparameters 7,7,00
Output: Adapted cross-encoders
Initialize cross-encoders {CF for steps + to do Select a batch B from Dor if step S a% x T then
[ Liabel + T Lagg
When new batch B is received, the first step is calculate the task-specific loss Liabet- In Eq.4, Uk represents the batch prediction of the cross-encoder Cy2,, and y represents the labels from the batch. We use the binary cross-entropy loss (BCE) for loss calculation.
for k: <- 1 to 2 do
With probability 1 1-16 P: Ub: (B) Liabel BCE y)
Update CE with Liabel
else
Compute predictions:
2
(ūx <- CF (B))
Ltabol
BCE
(4)
2
Compute label loss: Liabel by Eq. 4.
A=1
Compute mean prediction agg by Eq.5.
The agreement loss Lagg measures the discrep- ancy between the model's prediction and the ag- gregate prediction agg- To compute this, first obtain the aggregate prediction agg by averaging the batch predictions from the two cross-encoder models. Subsequently, we calculate the loss Lagg using the cross-cntropy loss.
Compute agreement loss Lagg by Eq.
Total loss: L Llabet + Lagg- Update model parameters with C.
1 agg = Σ (B)) 1=1
utilizing the Margin Mean Squared Error (Margin- MSE) loss and the margin value, they successfully trained the student model (bi-encoder) to discrimi- nate between relevant and irrelevant passages.
(5)
2
Logg
BCE ogg)
The Margin-MSE loss is defined in Eq. 8, where Q denotes the query, p+ denotes the relevant pas- sage, and p- denotes the irrelevant passage. The term MSE corresponds to the Mean Squared Er- ror loss function (Stein, 1992), while C'E and BE represent the output relevant scores of the cross- encoder and bi-encoder, respectively, given a query and passage.
Our experiments demonstrate that by incorpo- rating this co-regularization technique, the models can leverage their shared predictions to enhance their learning and improve overall performance.
3.4 Bi-Encoder (Retrieval Model) Adaptation
Knowledge distillation is widely used method to transfer knowledge from a high-compute model to a smaller and faster model while preserving per- formance to some extent. Applying this technique to transfer knowledge from a rerank model to a retrieval model is also meaningful.
- M= M=CE(Q Pt) - CF(Q, (7)
L MSD((M,BEQ.P')
(8)
Previous experiments conducted by Hofstätter et al. (2020) have demonstrated the effectiveness of ensembling the teacher's scores in improving the accuracy of margin estimation. In our study, we calculate the margin by averaging the scores from both the unadapted cross-encoder and the adapted cross-encoder (referred to as Mix). This approach yields more precise labels for the pseudo queries and passages. For a detailed example, please refer
In previous study, Hofstätter et al. (2020) pro- posed cross-architecture training approach to transfer knowledge from a cross-encoder (Nogueira and Cho, 2019) to various dense retrieval models. They used the cross-encoder as the teacher model to label the margin M between pairs of (query, rel- evant passage) and (query, irrelevant passage). By

---

Domain
Dataset
#Corpus
#Test query
consists of 300k samples, with 100k relevant labels and 200k irrelevant labels.
Finance
FiQA-2018
$7638
648
Science
SciFact
5183
300
DenoiseFFinetuning: We allocate the initial T x training steps for the random batch warm- warm-up stage, with a set to 0.1. The remaining steps are dedicated to the co-regularization stage. The cross- encoder is trained with batch size of 32 for 2 epochs. The maximum sequence length is set to 300 for all datasets.
Bio-Medical
TREC-COVID
171,332
50
Table The statistics of the three experimental datasets
to Appendix F.
4
Experiment Setup
Bi-Encoder (Retrieval Model) Adaptation: We use DistilBERT (Sanh et al., 2019) as the pre- trained model for the bi-encoder, following the con- figuration of GPL (Wang al., 2021b). For nega- tive retrieval, we employ two dense retrieval mod- els, msmarco-distilber? and Aamaaco-MiniLMM* obtained from Sentence-Transfomme Transformers. Each re- triever retrieves 50 negative passages. From these retrieved passages, we randomly select one neg- ative passage and one positive passage for each training query, forming single training example. The bi-encoder is trained for 140k steps with batch size of 32, The maximum sequence length is 350 for all datasets.
4.1 Datasets
Our experiments are performed on three datasets obtained from the BEIR benchmark (Thakur et al., 2021). Details regarding the test sizes and corpus sizes for each dataset are provided in Table 1 and be found in Appendix B.
4.2 Baselines
In our evaluation, we compare the performance of our adapted cross-encoder and adapted bi-encoder models with previous domain adaptation methods and zero-shot models. Further details about the baselines can be found in Appendix C.
4.3 Hyperparameters
Experiment Results and Analyses
We adapt existing models trained the MS- MARCO (Nguyen et al., 2017) dataset from the source domain to multiple target domains using our domain adaptation method.
5.1 Overall Performance
To generate queries for pas- sages in the target domain, we utilize the T5 query generator with temperature setting of 1, which has been shown produce high-quality queries in previous studies (Ma al., 2021; Wang al., 2021b).
Query Generation:
The main results are summarized in Table 2. A comparison of the recall rates for first-stage re- trieval is described in Appendix D. We utilize the adapted bi-encoder (BE(w/Ad)) for initial retrieval, retrieving 100 relevant passages query, and the adapted cross-encoder (CE(w/Ad)) for reranking. This combination yields substantial improvements in NDCG@ @ 10 compared previous domain adap- tation methods and zero-shot models. Notably, our adapted bi-encoder outperforms previous query generation- based domain adaptation methods, GPL and QGen, in the first-stage retrieval. We achieve a remarkable 10% performance increase. Note that we do not experiment with BE(Mix) + CE(Mix) setup, since an online system in practice typically comprises one BE and one CE ensure shorter trieval time. Introducing Mix the cross-encoder would double the inference time for reranking, mak- ing it impractical for real-world implementation. Therefore, we opted not to explore this particular setup in our experiments.
Encoder (Rerank Model) Adaptation: We adapt the Mini-LM cross-encoder (Wang et al., 2020b) to the target domains. For retrieving hard negatives, we employ BM25 from Elasticsearch and bi-encoder from From the t0p-1000 relevant passages retrieved by each retriever, we select one passage as a hard nega- tive. The input passage is created by concatenating the title and body text. The hyperparameter y in Eq.3 is set to 1. The pseudo training dataset Dr..
'nttps://huggiinggaas co/BeI//
qurry=eenmmsmmrrro-----rrrgev
1B-marro--MinLLtL-11--
httpppss//uugggiiaeeeeeee//eette eesentence-transfone
samarco-distilbert-.o

---

Model
FiQA
SciFact
TRECC
Avg.
Zero-shot sparse model
BM255
23.6
66.5
65.6
51.9
doc (Nogueira et al., 2019)
29.1
67.5
71.3
55.9
Zero-shot dense model
ANCE (Xiong et al., 2021)
29.5
50.7
63.4
48.5
ColBERTV2 (Santhanam et al., 2022)
31.7
67.1
67.7
55.5
Reranking
BM25+CE
35.0
68.8
75.4
59.7
Pre-training base domain adaptation
MLM (Gururangan et al., 2020)
30.2
60.0
69.5
53.2
TSDAE (Wang al., 2021a)
29.3
62.8
76.1
56.0
Query generation base domain adaptation
QGen (Ma et al., 2021)
28.7
63.8
72.4
54.9
GPL (Wang et al., 2021b)
32.8
66.4
72.6
57.2
GPL CE(wo/Ad)
36.8
68.4
76.5
60.5
Ours
BE(w/Ad)
33.9
69.9
74.6
59.4
BM25+CE(W/Ad)
36.7
71.1
78.6
62.1
BE(W/A))+CE(WAA))
38.4
71.1
79.6
63.0
Table 2: Our approach outperforms previous methods in terms of NDCG@ 10 for domain adaptation. For reranking by cross-encoder (CE), we utilize top 100 results from either the bi-encoder (BE) BM25. For BE(w/Ad), it denotes the bi-encoder (BE) distilled from the Mix cross-ence cross-encoder (See Table.3).
5.2 Impact of Denoise-finetuning
Teacher
FiQA
SciFact
TRECC
Avg.
During the evaluation of the denoise- -inboiiiiiintuuingg ap- proach, we conducted tests at intervals of 1000 training steps. In the denoise setting, reported the model with the higher score. The results de- picted in Figure illustrate the effectiveness of denoise-finetuning in stabilizing model training and mitigating the influence of noisy labels, particularly in situations involving diverse data distributions. We compare the two methods using three different random seeds. resulting in six scores for each test point.
CE(wo/Ad)
33.1
66.7
72.9
57.5
CE(w/Ad) Mix
32.3
69.1
69.7
57.0
33.9
69.9
74.6
59.4
Table 3: NDCG@ 10 results obtained using different cross-encoder models as bi-encoder teachers. "Mix" utilizes the average margin score of "CE(wo/Ad)" and "CE(w/Ad)" for training the bi-encoder model.
assess the models every 10k steps and report the best evaluation result.
5.3 Impact of Teacher Models in Distillation
Interestingly, we observe that while the adapted cross-encoder performs better in reranking, it's dis- tillation result do not surpass the unadapted cross- encoder over all datasets, such as TREC-COVID. After studying the distribution of the margin and some data cases, we found that the unadapted cross- encoder tends to assign a smaller absolute margin score, which means that exhibits a more conser-
our distillation experiments, we evaluate three teacher models: (1) adapted cross-encoder, (2) un- adapted cross-encoder and (3) Mix model. The Mix model is created by averaging the margin values of models (1) and (2), as described in Section 3.4. We
https:: http//hhugggggaace wtwpo//gga......eeeeeeeeeeeeeeee cw/sentenceeecceeeeee ereentence-trasffromeee msmarco-Minille Min-coo--iiLM------V

---

AIAA
SciFact
TREC COVID
1.14
M
4000
0.200
1000
Danckis
100
Normnl
Figure 4: We perform model evaluation on the testing set every 1000 steps during the training of the cross-encoder on the pseudo training dataset. total, we train the model for 18,750 steps across all datasets, with an allocation of 1870 steps for warm-up and the remaining steps for essential co-regularization. The impact of denoise fine-tuning becomes evident from the second test point, which occurs at approximately 2000 steps.
Reverse%
Conclusion
Dataset
Method
5
10
In conclusion, we have presented novel approach for unsupervised domain adaptation in information retrieval. Our method addresses the limitations of dense retrieval models and aims to enhance per- formance in out-of-domain scenarios. By simulta- neously adapting the rerank model and dense re- trieval model, we achieve significant improvements in two-stage retrieval performance.
Normal
30.3
24.7
23.2
FIQA
Denoise
33.1
26.4
24.9
Normal
70.1
62.9
63.6
SciFact
Denoise
70.1
67.5
68.4
Table 4: The performance of the cross-encoder model assessed at 18,000 training steps, comparing two training conditions: normal fine-tuning and denoise- finetuning.
Experiment results demonstrate that our domain adaptation approach outperforms previous meth- ods in both one-stage and two-stage retrieval tasks.
The fine-tuned rerank model effectively captures domain-specific information, which is then trans- ferred the dense retrieval model through knowl- edge distillation.
vative distinction between relevant and irrelevant passages. We speculate that the adapted model widens the gap between relevant and irrelevant passages, while the unadapted model maintains a certain level of regularization. Consequently, the adapted and unadapted cross-encoders together complement each other and surpass single models. Therefore, we combine both models as Mix and uti- lize it as the optimal teacher model for knowledge distillation. For detailed example, please refer to Appendix F.
Overall, our work contributes to the field of unsu- pervised domain adaptation in information retrieval and provides valuable framework for improving retrieval performance across diverse domains with- out the need for domain-specitn labeled data.
Limitations
5.4 Extreme Environment
Our proposed method exhibits superior perfor- mance compared to other baselines on FiQA (Fi- nance), SciFact (Science), and TREC-COVID (Bio-Medical) datasets by significant margin. However, is important acknowledge that trieval datasets from different domains may possess highly diverse distributions of passages and queries. Therefore, further experiments across additional domains are required to comprehensively evaluate the general performance of this method.
Table 4 presents the results of our experiments on a simulated dataset with significant label noise, demonstrating the effectiveness of denoise finetun- ing. We inverted {1, 5, 103% of the labels in the pseudo dataset D.. and report the final test scores. Additionally, we conducted experiments with 20% the labels inverted, testing with different values the hyperparameter a. Further detailed analysis can be found in Appendix E.

---

Acknowledgement
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39- 48.
This work was supported by the National Science and Technology Council, Taiwan, under Grant We would like to thank the reviewers for their insightful feedback.
NSTC 1 112-2233-F-006-000
Carlos Lassance. 2023, Extending english ir methods to multi-lingual ir. arXiv preprini arXiv:2302.1 14723.
References
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seque to-suueece pre-truc pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kan- wal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017. A closer look at mem- orization in deep networks. In International confer- ence on machine learning, pages 233-242. PMLR.
Hans Christian, Mikhael Pramodana Agus, and Derwin Suhartono. 2016. Single document automatic text summarization using term frequency-inverse docu- ment frequency (tf-idf). ComIech: Computer, Mathe- matics and Engineering Applications, 7(4):285-294,
Tie-Yan Liu et al. 2009. Learning to rank for informa- tion retrieval. Foundations and Trends@ in Informa- tion Retrieval, 3(3):225-331
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.11999
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Pupers), pages 4171 1-66, Minneapolis, Minnesota. Association for Computational Linguistics.
Ji Ma. Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot neural passage retrieval via domain-targeted synthetic question generation. In Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1075-1088, Online. Association for Computational Linguisties.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy. Doug Downey, and Noah A. Smith. 2020. Don't it stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1342-8360, Online. Association for Computational Linguistics.
Macedo Maia, Siegfried Handschuh, André Freitas. Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www' 18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941-1942.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking models with cross- architecture knowledge distillation.
Irina Matveeva, Chris Burges, Timo Burkard, Andy Lau- cius, and Leon Wong. 2006. High accuracy retrieval with multiple nested ranker. In Proceedings of the 29th annual international ACM SIGIR conference Research and development in information retrieval, pages 437 444.
Sebastian Hofstätter and Allan Hanbury. 2019. Let's measure run time! extending the ir replicability in- frastructure to include performance aspects. arXiv preprint 107.4440....614.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Deng- 2017. MS MARCO: A human generated MAchine reading COmprehension dataset.
Zhiqi Huang, Puxuan Yu, and James Allan. 2023. Cross-lingual knowledge transfer via distillation for multilingual information retrieval. arXiv preprint arXiv:2302.1 13400.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Pas- sage re-ranking with bert. arXiv preprint arXiv:
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Rodrigo Nogucira. Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with pre- trained sequence-to-seqqeece model. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 708-718, Online. Association for Computational Linguistics.

---

Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019, From doe2query to doetttttquery. Online preprint, 6.
Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021a. Tsdae: Using transformer-bast sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pages 67 1-688.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Leurning Rexearch, 21(11:5485-55 -455551...
Kexin Wang, Nandan Thakur, Nits Reimers, and Iryna Gurevych. 202 Gpl: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. arXiv preprint arXiv:2112.07577
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert, Sentence embeddings using siamese bert-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Lidan Wang, Jimmy Lin, and Donald Metzler. 2011. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Informa- tion Retrieval, pages 105-114.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulicn Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
Lucy Lu Wang, Kyle Lo. Yoganand Chandrasekhar. Russell Reas, Jiangjiang Yang. Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, et al. 2020a. Cord- 19: The covid-19 19 open research dataset. ArXiv.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arX1v:1910.011 1910.01108.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-577 -5888.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3715-3733 15-3734.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok- Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In International Conference on Learning Representations.
Charles Stein. 1992. Estimation with quadratic loss. Breakthroughs in statistics: Foundations and basic theory, pages 460.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong. Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with BeRTserini. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations), pages 72-77, Minneapolis, Minnesota. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Qi Zhang, Zijian Yang. Yilun Huang, Ze Chen. Zijian Cai, Kangxu Wang, Jiewen Zheng, Jiarong He, and Jin Gao. 2023. Enhancing model performance in multilingual information retrieval with comprehen- sive data engineering techniques. arXiv preprint arXiv:230..07010.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J Gordon. 2018. An empirical study of exam- ple forgetting during deep neural network learning. arXiv preprint arXiv: 181205515..
Ellen Voorhees, Tasmeer Alam, Steven Bedrick. Dina Demner-Fushman William R Hersh, Kyle Lo. Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information re- trieval test collection. In ACM SIGIR Forum, vol- ume 54, pages 1-12. ACM New York, NY, USA.
Wenxuan Zhou and Muhao Chen. 2021. Learning from noisy labels for cenity-centric information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang. Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. As- sociation for Computational Linguistics.

---

Dataset
Example
Diverse Domains: Our chosen datasets span diverse domains, including finance, scientific ab- stracts, and COVID-19 research, representing areas of significant relevance.
Gen
Testset
TRECC
what is smus
how does the coronavirus respond to changes in the weather
C Baseline
FIQA
what is step2?
what software do you use to track your net worth
We compare our approach with previous zero-shot and domain adaptation approaches. For GPL, MLM, Qgen and TSDAE, utilize the data pro- vided from Wang et al. (2021b). For doeTsquery, we use the data provided by the Thakur al. (2021), For BM25, we employ the Anserini toolkit with a pre-built index.
SciFact
what mdsc
ATF4 is general endoplasmic
reticulum stress marker.
Table 5: Example of pseudo queries and testing set queries.
A Example of short generated queries
Zero-shot sparse model: The BM25 (Robert- son et al., 1995) and docTSquery (Nogueira et al., 2019) models represent the sparse approach. The docTSquery model utilizes T5 model (Raffel et al., 2020) generate queries and append them to the passages, thereby improving the retrieval performance.
We have noticed that shorter generated queries fire- quently inquire about the definition meaning of word in the format of "What is xxx," " which de- viates from the distribution found in our testing dataset. See Table for the comparison of a gen- erated query and a query from the corresponding testing dataset.
Zero-shot dense model: ANCE (Xiong et al., 2021) is a bi-encoder that utilizes an Approximate Nearest Neighbor (ANN) index of the corpus to generate challenging negative examples. During model fine-tuning, the index is updated in parallel to select these instances. On the other hand, Col- BERTV2 (Santhanam et al., 2022) computes con- textualized embeddings at the token level, which of- fers better robustness compared to whole-sentence- based approaches.
B Dataset
We evaluate the performance of our method on three datasets from the BEIR benchmark. To en- sure fair evaluation, we utilize the datasets and evaluation code 5 provided by Thakur et al., 2021.
The FiQA dataset (Maia al., 2018) focuses on the financial domain and utilizes StackExchange posts from 2009 to 2017 as its corpus.
Pre-training: MLM (Gururangan et al., 2020) and TSDAE (Wang et al., 2021a) enable unsuper- vised training of the model in the target domain, facilitating subsequent fine-tuning for downstream tasks.
The TREC-COVID dataset(Voorheee et al.. 2021) is based on the CORD-19 (Wang et al., 2020a) challenge and includes recent publications and historical research on COVID-19 19 and related coronaviruses.
Question generation domain adaptation: QGen (Ma et al., 2021) and GPL (Wang et al., 2021b) leverage generated questions as training data to enhance the dense retrieval model. These methods differ in their labeling approaches, with QGen using binary labels, and GPL utilizing a cross-encoder (Nogueira and Cho, 2019) assign soft labels to query-passage pairs.
The SciFaat dataset (Wadden et al., 2020) com- prises evidence- connnnceecotiiingngggg abstracts that corre- spond to expert-written scientific claims. Each claim is labeled, and the dataset also includes ra- tionales that provide supporting evidence for the labels.
We selected these three datasets for the following reasons:
D Recall Performance
Accessibility: All three datasets are publicly
available and open for use.
In the context of two-stage retrieval, the recall rate of the first retrieval model holds paramount signifi- cance. This is because achieving maximum recall of relevant articles is essential to enable the subse- quent rerank model for effective sorting, In Table
Objective Evaluation: We employed the BEIR toolkit, utilizing provided corpus, queries, qrels, and validation methods, ensuring a fair and unbi- ased evaluation.
peittp://giiie thub. ccm//cattorrnn///meerinnn /tt.

---

Model/Dataset
FiQA
SciFact
TRECC
SciFact
BM25
53.9
92.5
47.1"
GPL
63.7
89.5
53.8"
NDCG@10 (Rerank BM25 100)
QGen
61.8
89.3
45.6
BE(w/ad) 63.1
91.6
54.7"
Table 6: Comparison between @ecall@ 100 of our ap- proach with BM25 and previous domain adaptation method. The **' denotes capped @ecall@k.
Dataset y
2
5w5
[w]
FIQA
25.8
29.0
27.7
30.1
35.0
SciFact
65.1
62.8
66.4
67.4
68.9
71.4
Steps
30000
Table Reranking BM25 top results with ranking model in the latest checkpoint evaluation on test set.
Figure 5: Experimental results with reversed 20% labels on the SciFact dataset.
6, conduct comparison between the adapted bi-encoder and other domain adaptation methods in terms of Recall@) 100.
FiQA
(Rerank 8M25 100)
Due the fact that the TREC-COVII dataset contains over 100 instances with relevance annota- tions for each query, utilizing the standard version of recall would result in very low values, as the number of relevant passages always exceeds the pa- rameter 'k'. Hence, adopt the capped recall @k 1Eq.10) as replacement measure, the same as BEIR Benchmark (Thakur al., 2021).
122
30
28
Normal
@ecall@k
1
N
TP:
(9)
5000
Steps
10000
15000
N
GT,
Figure 6: Experimental results with reversed 20% labels on the FiQA dataset.
N
1
TPF
Capped Recall@k
(10)
N
min (k, GT,)
of the denoise-finetie finetuning method. We conducted tests every 1000 training steps to track the model's fine-tuning progress. The results presented in Fig- ure and Figure 6 demonstrate the impact of the hyperparameter r (defined in Eq. 3). Table shows the last checkpoint rerank performance. These find- ings highlight the importance of carefully tuning this parameter to achieve optimal performance and minimize the influence of label noisc.
where:
N: Total number of queries.
TP; Number of true positives for retrieval
instance i.
GT, Number of ground truth positive items for retrieval instance i.
k: Cutoff value for recall calculation.
F Margin from different teacher models
E Extremely Noisy Setting
In Table and Figure 7, different models exhibit variations in margin predictions for positive and negative samples across different datasets. Hence, we can leverage these differences to complement each other's 's shortcomings.
our experiments, intentionally create noisy labels by inverting 20% the labels in the pseudo dataset Doc- This allowed us to simulate an ex- tremely noisy dataset and evaluate the performance

---

CE(wo/Ad): -0.41 CE(w/Ad): -11.36 Mix: -5.98
CF(wo/Ad): 4.63 CE(w/Ad): -9.15 Mix: -2.26
Q: how to calculate the rent you can get from own- ing your house
Q: why is honesty important in business
P+ ... Yes,, you should certainly compare the monthly rent to what your mortgage payments would be, as you have done. Yes, you should con- sider how long you might live there. If you do move out, how difficult will it be to sell the house, given the marker conditions in your area? If you try to rent it, how difficult is it find tenant, and what rent could you expect to receive? Speaking of moving out and renting the place: Who will man- age the property and do maintenance?.... ...
P+ ...I refuse to be dishonest in business and am lucky work for senior management that shares that view. We win some, we lose some. We are as transparent as possible, keep our commitments, and seek deals that drive value for all parties. Some- times am deeply frustrated by the lack of integrity of others and the benefits they seem to reap. But my reputation is excellent, my clients know they can take me at my word...
In order to arrive at a decision you need the numbers: suggest spreadsheet. List the monthly and annual costs (see other responses). Then de- termine what the market rate for rental. Once you have the numbers it will be clear from numbers standpoint. One has consider the hassle of own- ing property from distance, which factored into the spreadsheet
Even there's nobody outside who needs to see it right away (investors, etc.), an honestly-written business plan is a valuable exercise. If done well (and preferably with external guidance) it forces you to think all the way through your idea and make your bases covered.
Table the example on the left, CE(w/Ad) demonstrates better performance, whereas in the example on the right, CI((w/Ad) outperforms. Various models showcase diverse margin predictions for positive and negative samples across distinct datasets. Thus, we can exploit these variances to compensate for each model's limitations.
10
Mergin
-10-
CEtwiAs)
-20
TREC-COWID
Solfect
FIO
Dataset
Figure 7: The margin values of using adapted cross- encoder, unadapted cross-encoder, and mixed margin.


Reviews:
[{"review_id": "PZsrofBZgG", "reviewer": "Reviewer_zViP", "summary": "", "questions": "- How does the overall training cost compared to the baseline methods? The proposed method utilizes several separate steps to adapt the two components in a retrieval system. The training procedure might be more complicated and requires lots of tuning efforts?\n\nUpdate: The authors have provided the training cost in GPU hours. The method does require longer training time, but as the authors claimed, it is a reasonable investment to improve the overall performance.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review of our paper. \n\n\n> How does the overall training cost compared to the baseline methods? The proposed method utilizes several separate steps to adapt the two components in a retrieval system. The training procedure might be more complicated and requires lots of tuning efforts?\n\nYou've raised a important question regarding the training costs and complexity compared to baseline methods. \n\nOur primary comparison target is the GPL[RF1]  method. In comparison to the baseline method GPL, the steps are as follows:\n1. Generating queries for passages (~5hr)\n2. In the distillation phase, labeling using the cross-encoder (~4hr for labeling)\n3. Training the bi-encoder (~10hr)\n\nTotal: 19hr\n\n[RF1] Wang, Kexin, et al. \"GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\nIn our approach, building upon GPL, we further finetune the cross-encoder. This introduces the following steps:\n1. Apply domain adaptation on cross-encoder.\n2. Labeling with both the original cross-encoder CE(w/Ad) and the adapted CE(wo/Ad) cross-encoder, which adds extra labeling time but doesn't impact bi-encoder training.\n\nHence, the training process involves these stages:\n1. Generating queries for passages (~5hr)\n2. Preparing training data for cross-encoder and conducting denoise fine-tuning (~2hr)\n3. Labeling using both the cross-encoder and the adapted cross-encoder (~4hr + 4hr)\n4. Training the bi-encoder (~10hr)\n\nTotal: 25hr\n\nAll our experiments were performed on a single RTX 3090 GPU. Utilizing multiple GPUs would accelerate computation speed. It's important to note that inference times remain consistent. Even during denoise fine-tuning, we exclusively employ the model that demonstrates superior performance on test scores for inference and distillation purposes.\n\nWe acknowledge the increased training time as a consideration. However, the additional steps we introduce, such as cross-encoder training and using adapted cross-encoders for labeling, contribute to the effectiveness of our approach in achieving domain adaptation. We believe that the enhanced performance justifies the additional training investment.\n\n> line 195: a space is missing before \"(\"\nline 322: \"label\" should be subscript\n\nThank you for pointing out these issues! We will add the missing space and correct the subscript for \"label\" as suggested.\n\nOnce again, we appreciate your constructive feedback and suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and easy to understand.\n- The proposed method adapts both the rerank model and the retrieval model without labeled target data. It also adopts denoise-finetuning to mitigate the impact of noisy labels. This technique is overall novel.\n- The proposed method shows good performance and the ablation studies demonstrate the effectiveness.", "reasons_to_reject": "The method has three steps which can be complicated and can require more tuning efforts.", "questions_to_authors": "- How does the overall training cost compared to the baseline methods? The proposed method utilizes several separate steps to adapt the two components in a retrieval system. The training procedure might be more complicated and requires lots of tuning efforts?\n\nUpdate: The authors have provided the training cost in GPU hours. The method does require longer training time, but as the authors claimed, it is a reasonable investment to improve the overall performance.", "paper_topic_and_main_contributions": "This paper proposes a new method for unsupervised domain adaptation for both the dense retrieval model and rerank model in a two-stage retrieval system. It first constructs a pseudo training set and then denoise-finetuning the rerank model. It finally distills the knowledge from the rerank model to the dense retrieval model. The method achieves good performance improvement in several benchmarks. The ablation studies also demostrate the effectiveness of differetn components.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 195: a space is missing before \"(\"\n- line 322: \"label\" should be subscript"}, "extra_text": {}}, {"review_id": "DdkNVkT6Tz", "reviewer": "Reviewer_X5Zx", "summary": "", "questions": "1. How does the proposed denoise fine-tuning method compare to R-Drop?\n\n2. Is the validation set available? If not, is the performance reported for the last checkpoint?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions and comments, which have been addressed below.\n\n> The relationship between the proposed co-regularization and denoising is not yet clear. It seems to only constrain the losses of the two sub-models, reducing overfitting and making training more stable.\n\nWe introduce multiple models to collaboratively predict a shared label, allowing all models to learn together. We believe that this shared label becomes closer to the true underlying label because:\n+ Noisy labels exhibit delayed learning curves [RF1], indicating that models might not have learned these errors at this stage.\n+ The concept of predicting shared labels draws inspiration from prior research [RF2], and using multiple models for co-regularization has been explored in related studies to mitigate the impact of mislabeling.\n\n[RF1] Arpit, Devansh, et al. \"A closer look at memorization in deep networks.\" International conference on machine learning. PMLR, 2017.\n\n[RF2] Zhou, Wenxuan, and Muhao Chen. \"Learning from Noisy Labels for Entity-Centric Information Extraction.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n\nWhat distinguishes our approach from previous co-regularization  methods, is the potential for **integrating** cross-encoders from different source domains. \n+ For instance, in scenarios where we're building a retrieval system for a new disease, denoise fine-tuning with related diseases' retrieval systems could yield mutual benefits. The aggregated predictions (Eq.7) from multiple integrated models might enhance performance.\n\n> The experimental section lacks discussion on some necessary details, such as: how to choose the two key parameters alpha and P? which rerank model should be chosen for knowledge distillation? would it be better to fine-tune with more rerank models?\n\n\nThe choice of the two parameters alpha and P is challenging due to their sensitivity to various factors like training data size and batch size. However, we do believe it's meaningful to explore the coefficient $\\gamma$ in Eq.5, as it directly influences denoise fine-tuning.\n\nWe do a experimented with a variety of  $\\gamma$ to investigate the impact of denoise fine-tuning. In a simulated extreme noisy environment, where 20% of labels were flipped, we observed that denoise fine-tuning consistently led to improvements in the final checkpoint's NDCG@10 scores, in comparison to models trained without denoise-finetuning.\n\n| Dataset / $\\gamma$ | 0 | 1 |  2 |  5 | 10 | 20 |\n| -------- | -------- | -------- | - | -------- | -------- | -------- |\n| FiQA     | 25.8     | 29.0     |27.7 | 30.1| 33.4| 35.0 |\n| SciFact     | 65.1     | 62.8     | 66.4| 67.4|68.9 | 71.4|\n\n\nTo address your concern about the rerank model for knowledge distillation, we chose the one that performed the best on the test set. Using more rerank models for fine-tuning has been shown to not work well in previous research [RF1], and it also results in increased training costs.\n\n> No repeated experiments were conducted to eliminate the influence of randomness.\n\nWe acknowledge this concern and have taken steps to address it. We conducted experiments with three different random seeds in our laboratory setting, performing evaluations every 1000 training steps. In the final version of the paper, we will provide a comprehensive representation of the results by including the evaluation scores from all three seeds at the same step intervals. This will allow readers to observe the outcomes of denoise fine-tuning and normal fine-tuning for a total of six scores (three for denoise fine-tuning and three for normal fine-tuning).\n\n\n> How does the proposed denoise fine-tuning method compare to R-Drop?\n\nThis is a great question! We are also curious about this. To compare out work with R-Drop[RF1]:\n+ We followed the implementation detailed in the R-Drop paper but changed the KL divergence to binary cross-entropy loss due to cross-encoder's binary output.\n+ We conducting evaluations on the test set every 1000 training steps during finetuning, using BM25 top 100 passages to rerank.\n+ Our reported results highlight the best NDCG@10 scores achieved during the training process.\n+ In accordance with Section 4.4 of the R-Drop paper, we set the dropout rate to 0.3 and explored the α parameter within the search range {3, 5, 7}, as specified in Section 4.5.\n\n[RF3] Wu, Lijun, et al. \"R-Drop: Regularized Dropout for Neural Networks.\" Advances in Neural Information Processing Systems 34 (2021): 10890-10905.\n\n**Result**\n\nAccording to the table below, R-Drop leads to a slight improvement in scores, but the difference is not significant and remains closely comparable to scores from normal training (Normal).\n\n| Dataset  | Zero-shot | Normal | R-Drop | Denoise |\n|----------|-----------|--------|--------|---------|\n| FiQA     | 35.0      | 36.4   | 36.3   | **36.7**    |\n| Covid    | 75.4      | 77.5   | 78.1   | **78.6**    |\n| SciFact  | 68.8      | 70.6   | 70.8   | **71.1**    |\n| Avg.     | 59.7      | 61.5   | 61.7   | **62.1**    |\n\n\n> Is the validation set available? If not, is the performance reported for the last checkpoint?\n\nWe reported our results based on the checkpoint with the highest evaluation scores during the training process.\n+ For TREC-COVID and SciFact datasets, as there are no official dev sets, we conducted testing on the test set every 1000 training steps, re-ranking the BM25 top 100 results using NDCG@10 as the evaluation metric.\n+ For the FiQA dataset, since an official dev set is available, we directly used the dev set's queries. Similarly, during training, we evaluated the model every 1000 training steps by re-ranking the BM25 top 100 results using NDCG@10.\n\n**To Reviewer:**\nOnce again, we appreciate your constructive feedback and suggestions, which helps us to enhance the comprehensiveness of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to understand, and the entire method appears to be reliable. \n\n2. The experiments show a significant improvement compared to previous methods, e,g., QGen, GPL.", "reasons_to_reject": "1. The relationship between the proposed co-regularization and denoising is not yet clear. It seems to only constrain the losses of the two sub-models, reducing overfitting and making training more stable.\n\n2. The experimental section lacks discussion on some necessary details, such as: how to choose the two key parameters alpha and P? which rerank model should be chosen for knowledge distillation? would it be better to fine-tune with more rerank models?\n\n3. No repeated experiments were conducted to eliminate the influence of randomness.", "questions_to_authors": "1. How does the proposed denoise fine-tuning method compare to R-Drop?\n\n2. Is the validation set available? If not, is the performance reported for the last checkpoint?", "paper_topic_and_main_contributions": "This paper investigates domain adaptation of the two-stage retrieval system. The authors first construct a pseudo training set using an initial generation model and two negative sample retrieval strategies. Then, they propose a denoise fine-tuning method to train the rerank models, where a key co-regularization technique is introduced to constrain the loss of different sub-rerank models. Finally, they use the rerank model as a teacher and update the retrieval model using knowledge distillation. The authors carefully experimentally verify the effectiveness of their method in domain adaptation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2.3 is related to knowledge distillation, and it would be better to merge it into Section 3.4."}, "extra_text": {}}, {"review_id": "18YqRBozKi", "reviewer": "Reviewer_kWur", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, which we greatly appreciate. \n\n> The description in the related work (Section 2) is somewhat lengthy, which covers more than one page.\n\nWe understand your concern regarding the length of the related work section (Section 2). During the initial drafting phase, we aimed to make our work accessible to scholars outside the Information Retrieval (IR) domain. To achieve this, we aimed for a comprehensive description to provide context to non-IR researchers. We will carefully consider your input and decide whether to trim the content while ensuring the essential context is retained.\n\n> The reason why choose the three datasets is not clearly clarified.\n\n\nRegarding the choice of the three datasets, we have several reasons for our selection:\n+ **Accessibility:** All three datasets are publicly available and open for use.\n+ **Objective Evaluation:** We employed the BEIR toolkit, utilizing its provided corpus, queries, qrels, and validation methods, ensuring a fair and unbiased evaluation.\n+ **Diverse Domains:** Our chosen datasets span diverse domains, including finance, scientific abstracts, and COVID-19 research, representing areas of significant relevance.\n\nWe will incorporate these explanations into the paper to clarify our rationale for dataset selection. \n\nOnce again, we sincerely thank you for your detailed review and hope that our responses encourage you to reconsider the evaluation of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The novel approach to unsupervised domain adaptation with denoise-finetuning, which significantly improves the performance of dense retrieval models. \n2. A comprehensive evaluation of the proposed method on three different datasets from different domains, demonstrating its superior performance compared to other baselines.", "reasons_to_reject": "1. The description in the related work (Section 2) is somewhat lengthy, which covers more than one page.\n2. The reason why choose the three datasets is not clearly clarified.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a novel approach for unsupervised domain adaptation in information retrieval. The main contribution of this paper is the proposal of a method that addresses the limitations of dense retrieval models and aims to enhance performance in out-of-domain scenarios. By simultaneously adapting the rerank model and dense retrieval model, the authors achieve significant improvements in two-stage retrieval performance. The paper also provides a comprehensive evaluation of the proposed method on three different datasets from different domains, demonstrating its superior performance compared to other baselines. Overall, this paper contributes to the field of unsupervised domain adaptation in information retrieval and provides a valuable framework for improving retrieval performance across diverse domains without the need for domain-specific labeled data.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TCmfa8qiYJ", "reviewer": "Reviewer_XVNc", "summary": "", "questions": "Question A: \nNoisy Label Handling: The paper does not delve into how they handle cases with high label noise (>10%). Although the denoise fine-tuning method is a good step, a more explicit technique, such as a noise-robust loss function, might be beneficial to tackle extremely noisy labels.\nQuestion B:\nI found the authors pre-trained their model on the MS-MARCO dataset for adaptation to multiple target domains. However, using a single source for domain adaptation might limit the model's ability to generalize to other unseen domains. ", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions and comments, which have been addressed below.\n\n> The compared baselines are not strong. Most of them are before 2022\n\n\nOur primary focus in this study is domain adaptation, specifically comparing against the state-of-the-art methods in the domain adaptation methods, such as QGen[RF1] and GPL[RF2]. While there are indeed more complex rerank and retrieval models available, these often involve intricate training techniques, extra datasets, and model architectures.\n\nOur intention with this work was to demonstrate the effectiveness of our approach using even the simplest models, namely cross-encoder and bi-encoder, for domain adaptation. By doing so, we aim to showcase that our method remains effective even with basic models.\n\n[RF1] Yang, Ji Ma Ivan Korotkov Yinfei, and Keith Hall Ryan McDonald. \"Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation.\"\n[RF2] Wang, Kexin, et al. \"GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\n> The proposed method involves multiple steps: generating queries, adapting cross-encoder and bi-encoder, and denoise fine-tuning. All modules are with different hyperparameters. This complexity can lead it difficult to reproduce and can also increase computational costs.\n\n\n**Computational costs**\n\nYou've correctly identified that our method involves multiple steps and numerous hyperparameters. We acknowledge that finding suitable hyperparameters can be challenging due to the complexity. While it's true that our approach involves more steps than GPL, we believe these additional costs are justified for the following reasons:\n1. We cannot guarantee that an unadapted cross-encoder will accurately annotate the target domain.\n2. The integration of cross-encoders from different domains during denoise fine-tuning adds diverse perspectives, which could be more beneficial than traditional co-regularization methods. For instance, in scenarios where we're building a retrieval system for a new disease, denoise fine-tuning with related diseases' retrieval systems could yield mutual benefits. The aggregated predictions (Eq.7) from multiple integrated models might enhance performance.\n3. The majority of the added cost is incurred during the training process, with minimal computational cost increase during actual inference.\n\n**About reproduce**\nWe will release our code along with the generated queries and pseudo training dataset.\n\n\n> Question A: Noisy Label Handling: The paper does not delve into how they handle cases with high label noise (>10%). Although the denoise fine-tuning method is a good step, a more explicit technique, such as a noise-robust loss function, might be beneficial to tackle extremely noisy labels\n\nIn response to your question regarding handling high label noise, we have conducted additional experiments to investigate the performance of denoise fine-tuning under extreme noise conditions. We created a pseudo dataset with 20% flipped labels to simulate extremely noisy labels. We adjusted the impact of denoise fine-tuning using $\\gamma$ in Eq.5 to observe its effects on the final checkpoint's performance on the test set's BM25 top 100 results' NDCG@10 scores.\n\nThe results indicate that denoise fine-tuning remains effective even in extremely noisy conditions. This demonstrates the robustness of our approach in handling label noise.\n\n\n| Dataset / $\\gamma$ | 0 | 1 |  2 |  5 | 10 | 20 |\n| -------- | -------- | -------- | - | -------- | -------- | -------- |\n| FiQA     | 25.8     | 29.0     |27.7 | 30.1| 33.4| **35.0** |\n| SciFact     | 65.1     | 62.8     | 66.4| 67.4|68.9 | **71.4**|\n\n\n> Question B: I found the authors pre-trained their model on the MS-MARCO dataset for adaptation to multiple target domains. However, using a single source for domain adaptation might limit the model’s ability to generalize to other unseen domains.\n\nOur model's source domain for domain adaptation is MSMARCO[RF3] , which is currently one of the largest information retrieval datasets. This choice allows us to leverage various supervised models for domain adaptation. Other works in the field have also begun from MSMARCO as a starting point for domain adaptation, justifying our approach.\n\nWhat distinguishes our approach from previous domain-adaptation methods, such as GPL, is the potential for integrating cross-encoders from different source domains during denoise fine-tuning and utilizing them for distillation in the bi-encoder.\n\n[RF3] Nguyen, Tri, et al. \"Ms marco: A human-generated machine reading comprehension dataset.\" (2016).\n\n> Evaluation Metrics: The paper mainly focuses on NDCG@10 for evaluation, which may not fully reflect the model’s performance, more evaluation metrics should be introduced.\n\nWe will include additional metrics recall@100 in our evaluation process. In a two-stage retrieval system, the recall rate of the first stage is  crucial, as successful retrieval in the first stage is a prerequisite for subsequent ranking by rerank models. \n\nWe have compared the first stage retrieval model's recall@100, and our approach exhibits higher recall@100 in SciFact and TREC-COVID datasets compared to QGen and GPL, which signifies the effectiveness of our approach in the initial retrieval stage.\n\n\n| Method / Dataset | FiQA | SciFact | TRECC|\n| -------- | -------- | -------- | -|\n|   QGen   | 61.8     | 89.3     | 45.6|\n|   GPL   | **63.7**     | 89.5     | 53.8|\n| BE(w/Ad)     | 63.1     | **91.6**    | **54.7** |\n\n\n\n> Experimentation with Transformer-based Models: The authors could investigate other cutting-edge transformer models for the query generation task.\n\nIn the future, we are interested in exploring the use of ChatGPT as a query generator. However, due to budget constraints and the significant time investment required for generation, we temporarily adopted the same query generator as our previous work. This suggestion is intriguing, and I agree that using different query generator models could provide distinct insights into the target domain.\n\n> Teacher Model Selection: While the paper successfully demonstrated the usage of a Mix model as a teacher for distillation, it relies on empirical observations. More theoretical exploration into why the Mix model performs better could make the method more robust and interpretable.\n\nWhile the large number of label relations, consisting of 4,480,000 (140,000*32) (Query, positive passage, negative passage) triples used for cross-encoder knowledge distillation to the bi-encoder, lacks precise annotated data for statistical analysis, we can hypothesize several factors contributing to the success of the Mix model:\n+ The cross-encoder adapted through domain adaptation demonstrates enhanced ranking capability, highlighting its proficiency in distinguishing relevant and irrelevant passages concerning the query.\n+ Our case study reveals that both models excel in different sample scenarios.\n\nWe will enhance the interpretability of our method by incorporating case studies specific to each of the three datasets in future versions of the paper. This will provide a more comprehensive explanation of the rationale behind the Mix model's effectiveness.\n\n\n**To Reviewer:**\n\nOnce again, we appreciate your constructive feedback and suggestions, which helps us to enhance the comprehensiveness of our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents an approach to unsupervised domain adaptation in information retrieval.\n2. The experimental results present that the proposed method significantly outperforms existing methods on various datasets (FiQA, SciFact, and TREC-COVID), indicating the effectiveness of the proposed method. ", "reasons_to_reject": "1. The compared baselines are not strong. Most of them are before 2022.\n2. The proposed method involves multiple steps: generating queries, adapting cross-encoder and bi-encoder, and denoise fine-tuning. All modules are with different hyperparameters. This complexity can lead it difficult to reproduce and can also increase computational costs.", "questions_to_authors": "Question A: \nNoisy Label Handling: The paper does not delve into how they handle cases with high label noise (>10%). Although the denoise fine-tuning method is a good step, a more explicit technique, such as a noise-robust loss function, might be beneficial to tackle extremely noisy labels.\nQuestion B:\nI found the authors pre-trained their model on the MS-MARCO dataset for adaptation to multiple target domains. However, using a single source for domain adaptation might limit the model's ability to generalize to other unseen domains. ", "paper_topic_and_main_contributions": "The authors proposed a rerank method to improve unsupervised domain adaptation in dense retrieval tasks.  \n1. The authors leverage target domain data into rerank and retrieval modeling.\n2. A denoise algorithm is introduced to increase the dense retrieval performance. \n3. The authors enhance the dense retrieval model with knowledge distillation from rerank model. \n\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Suggestions for authors:\n1. Evaluation Metrics: The paper mainly focuses on NDCG@10 for evaluation, which may not fully reflect the model's performance, more evaluation metrics should be introduced. \n2. Experimentation with Transformer-based Models: The authors could investigate other cutting-edge transformer models for the query generation task. \n3. Teacher Model Selection: While the paper successfully demonstrated the usage of a Mix model as a teacher for distillation, it relies on empirical observations. More theoretical exploration into why the Mix model performs better could make the method more robust and interpretable."}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to unsupervised domain adaptation for both the dense retrieval model and rerank model in a two-stage retrieval system. Through meticulous experimentation, the authors validate the efficacy of their proposed method in domain adaptation. The reviewers' opinions on this work are not unanimously positive. While the author has addressed certain concerns raised by the reviewers, there are still some lingering apprehensions. zViP expressed concerns about the training time consumption of the method. The authors responded by providing a detailed comparison with previous methods and indicating the time consumption involved. Although the method does require a significant amount of time, zViP acknowledges that these costs are a reasonable investment. X5Zx raised concerns regarding method details, experimental details, and experimental repeatability. The authors provided a thorough response addressing these concerns. kWur recognizes the novelty of the method and provides suggestions for further revision of the paper. XVNc expressed concerns about the weaknesses in the comparison methods, the complexity of calculations leading to practical feasibility issues, and difficulties in reproducibility. The authors provided corresponding responses, but these concerns still remain.
Decision: Accept-Findings