UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Authors: Jon Saad-Falcon
Omar Khattab
Keshav Santhanam
Radu Florian
Martin Franz
Salim Roukos
Avirup Sil
Md Arafat Sultan
Christopher Potts
Venue: EMNLP 2023 Main (2023)
Keywords: Natural Language Processing
Information Retrieval
Domain Adaptation
TL;DR: We present UDAPDR, a self-supervised approach employs large language models to adapt multi-vector retrieval models to new domains.

Abstract: Many information retrieval tasks require large labeled datasets for fine-tuning. However, such datasets are often unavailable, and their utility for real-world applications can diminish quickly due to domain shifts. To address this challenge, we develop and motivate a method for using large language models (LLMs) to generate large numbers of synthetic queries cheaply. The method begins by generating a small number of synthetic queries using an expensive LLM. After that, a much less expensive one is used to create large numbers of synthetic queries, which are used to fine-tune a family of reranker models. These rerankers are then distilled into a single efficient retriever for use in the target domain. We show that this technique boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods.

UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Jon Saad-Falcon' Omar Khattab'
Keshav Santhanam'
Radu Florian Martin Franz²
Salim Roukos² Avirup Sil Md Arafat Sultan2
Christopher Potts
Stanford University
IBM Research AI
Abstract
Stage
Generate queries GPT prompts target domaln passages
Many information retrieval tasks require large labeled datasets for fine-tuming aneettunigg However, such datasets are often unavailable, and their util- ity for real-world applications can diminish quickly due to domain shifts. To address this challenge, develop and motivate method for using large language models (LLMs) to generate large numbers synthetic queries cheaply. The method begins by generating small number of synthetic queries using an ex- pensive LLM. After that, a much less expensive one is used to create large numbers of synthetic queries, which are used to fine-tune a family of reranker models. These rerankers then dis- tilled into single efficient retriever for use in the target domain. We show this technique boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods.
Stage
Stage
Create corpus adapted using synthelic
Distill rerankers
Stage
Stage
Ropoat
Generate new queries using adapted prompts LLM
Evaluate ColBERTv2 target domaii
su needed
Stage
Train reranker with synthetic queries
Figure Overview of UDAPDR. An expensive LLM like GPT-3 is used to create initial set of synthetic queries. These are incorporated into a set of prompts for less expensive LLM that can generate large numbers of synthetic queries cheaply. 'he queries stemming from each prompt are used to train separate rerankers, and these are distilled into single ColBERTv2 retriever for use in the target domain.
1 Introduction
The advent of neural information retrieval (IR) has led to notable performance improvements on docu- ment and passage retrieval tasks (Nogueira and Cho, 2019; Khattab and Zaharia, 2020; Formal et al., 2021) well as downstream knowledge- intensive NLP tasks such as open-domain question- answering and fact verification (Guu et al., 2020; Lewis al., 2020; Khattab et al., 2021: Izacard al., 2022). Neural retrievers for these tasks often benefit from fine-tuning on large labeled datasets such as SQuAD (Rajpurkar et al., 2018), Natural Questions (NQ) (Kwiatkowski et al., 2019), and KILT (Petroni et al., 2021). However, IR models can experience significant drops in accuracy due to distribution shifts from the training to the target do- main (Thakur al., 2021; Santhanam al., 2022b). For example, dense retrieval models trained on MS MARCO (Nguyen et al., 2016) might not gen- cralize well to queries about COVID-19 scientific publications (Voorhees et al., 2021; Wang et al.,
2020), considering for instance that MS MARCO predates COVID-19 and thus lacks related topics.
Recent work has sought to adapt IR models to new domains by using large language models (LLMs) to create synthetic target-domain datasets for fine-tuning retrievers (Bonifacio et al., 2022; Meng et al., 2022; Dua et al., 2022). For example, using synthetic queries, Thakur et al. (2021) and Dai et al. (2022) fine-tune the retriever itself and train a cross-encoder to serve as passage reranker for improving retrieval accuracy. This significantly improves retriever performance in novel domains, but comes at a high computational cost stemming from extensive use of LLMss. This has limited the applicability these methods for researchers and practitioners, particularly in high-demand, user-

---

facing settings. In this paper, we develop Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers (UDAPDR), an efficient strategy for us- ing LLMs to facilitate unsupervised domain adap- tation of neural retriever models. We show that UDAPDR leads to large gains in zero-shot settings on a diverse range of domains.
1000s of synthetic queries prove effective and is compatible with various LLMs de- signed for handling instruction-based tasks like creating synthetic queries (e.g., GPT-3, T3, Flan-T5).
We generate synthetic queries using multi- ple prompting strategies that leverage GPT- 3 and Flan-T5 XXL. This bolsters the effec- tiveness of our unsupervised domain adapta- tion approach. The broader set of synthetic queries allows us to fine-tune multiple passage rerankers and distill them more effectively.
The approach outlined in Figure We begin with a collection of passages from a target domain (no in-domain queries or labels are required) as well as prompting strategy incorporating these passages with the goal of query generation. A pow- erful (and perhaps expensive) language model like GPT-3 is used create modest number of syn- thetic queries. These queries form the basis for corpus-adapted prompts that provide examples of passages with good and bad queries, with the goal of generating good queries for new target domain passages. These prompts are fed to a smaller (and presumably less expensive) LM that can generate a very large number of queries for fine-tuning neu- ral rerankers. We train separate reranker the queries from each of these corpus-adapted prompts, and these rerankers are distilled into a single stu- dent ColBERTv2 retriever (Khattab and Zaharia, 2020; Santhanam et al., 2022b,a), which is evalu- ated on the target domain.
2 Related Work
2.1 Data Augmentation for Neural IR
LLMs have been used to generate synthetic datasets (He et al.. 2022: Yang et al., 2020; Anaby-Tavor et al., 2020; Kumnr et al., 2020), which have been shown to support effective domain adaptation in Transformer-based architectures (Vaswani et al., 2017) across various tasks. LLMs have also been used to improve IR accuracy in new domains via the creation of synthetic datasets for retriever fine- tuning (Bonifacio et al.. 2022: Meng et al., 2022). Domain shift is the most pressing challenge for domain transfer. Dua et al. (2022) categorize dif- ferent types of domain shifts, such as changes in query or document distributions, and provide inter- vention strategies for addressing each type of shift using synthetic data and indexing strategies.
By distilling from multiple passage rerankers instead of single one, improve the utility of ColBERTv2, preserving more retrieval accuracy gains while redueing latency at inference. Our core contributions are as follows:
Query generation can help retrieval models trained on general domain tasks adapt to more tar- geted domains through the use of generated query- passage pairs (Ma et al., 2020; Nogueira et al., 2019). Wang et al. (2022) also use generative models to pseudo-label synthetic queries, using the generated data to adapt dense retrieval mod- els to domain-specific datasets like BEIR (Thakur et al., 2021). Thakur et al. (2021) and Dai et al. (2022) generate millions of synthetic examples for fine-tuning dense retrieval models, allowing for zero-shot and few-shot domain adaptation.
We propose UDAPDR, novel unsupervised domain adaptation method for neural IR that strategically leverages expensive LLMs like GPT-3 (Brown et al., 2020) and less expensive ones like Flan-T5 XXL (Chung et al., 2022), as well as multiple passage rerankers. Our approach improves retrieval accuracy in zero- shot settings for LoTTE (Santhanam et al., 2022b), SQuAD, and NQ.
We preserve the accuracy gains of these rerankers while maintaining the competitive latency of Co1BERTv2. This leads to substan- tial reductions in query latency.
Synthetic queries can also be used to train pas- sage rerankers that assist neural retrievers. Cross- encoders trained with synthetic queries boost re- trieval accuracy substantially while proving more robust to domain shifts (Thakur al., 2020, 2021; Humeau al., 2019). Dai et al. (2022) explore training the few-shot reranker Promptagator++, leveraging an unsupervised comain-adaptation ap- proach that utilizes millions of synthetically gener-
Unlike number of previous domain adapta- tion approaches that utilize millions of syn- thetic queries, our technique only requires
pronounced: Yoo-Dap-ter

---

ated queries train passage reranker alongside a dense retrieval model. Additionally, Wang al. (2022) found using zero-shot cross-encoders for reranking could further improve quality.
triever's base model (Devlin et al., 2019), and we show that it combines effectively with multi-vector retrievers (Khattab and Zaharia, 2020; Santhanam et al., 2022b).
However, due to the high computational cost of rerankers at inference, both Dai et al. (2022) and Wang et al. (2022) found unlikely these ap- proaches would be deployed in user-facing settings for information retrieval. Our work seeks to bolster the utility of passage rerankers in information re- trieval systems. Overall, Dai et al. (2022) and Wang et al. (2022) demonstrated the efficacy of unsuper- vised domain adaptation approaches utilizing syn- thesized queries for fine-tuning dense retrievers or passage rerankers, By using distillation strategies, we can avoid their high computational cost while preserving the latent knowledge gained through unsupervised domain adaptation approaches.
3 Methodology
Figure 1 outlines each stage of the UDAPDR strat- egy. For the target domain T, our approach quires access to in-domain passages (i.e., within the domain of T). However, it does not require any in-domain queries or labels. The overall goal is to leverage our store of in-domain passages and LLM prompting to inexpensively generate large numbers of synthetic queries for passages. These synthetic queries are used to train domain-specifer reranking models that serve as teachers for single retriever. The specific stages of this process
follows:
We begin with set of prompts that em- bed examples of passages paired with good and bad queries and that seek have the model generate novel good query for a new passage. We sam- ple X in-domain passages from the target domain T, and we use the prompts to generate 5X syn- thetic queries. In our experiments, we test values of X such as 5, 10, 50, and 100. (In Appendix A, we explore different strategies for selecting the X in-domain passages from the target domain.)
Stage 1:
2.2 Pretraining Objectives for IR
Pretraining objectives can help neural IR systems adapt to new domains without annotations. Masked Language Modeling (MLM) (Devlin et al., 2019) and Inverse Cloze Task (ICT) (Lee et al., 2019) offer unsupervised approaches for helping retrieval models adapt to new domains. Beyond MLM and ICT, Chang al. (2020) proposed two unsuper- vised pretraining tasks, Body First Selection (BFS) and Wiki Link Prediction (WLP), which use sam- pled in-domain sentences and passages to warm-up a neural retriever to new domains. Additionally, Gysel et al. (2018) 88 developed the Neural Vector Space Model (NVSM), an unsupervised pretraining task for news article retrieval that utilizes learned low-dimensional representations of documents and words. Izacard al. (2021) also explore a con- trastive learning objective for unsupervised training dense retrievers, improving retrieval accuracy in new domains across different languages.
In this stage, we use GPT-3 (Brown et al., 2020), specifically the text-davinci-eet model. The guiding idea is to use a very effective LLM for the first stage, to seed the process with very high quality queries. We employ the five prompting strategies in Figure 2. Two of our prompts are from Bonifacio et al. 2022, where they proved successful for generating synthetic queries in few-shot set- ting. The remaining three use zero-shot strategy and were inspired by prompts in Asai et al. 2022.
The queries generated in Stage form the basis for prompts in which passages from the target domain are paired with good and bad syn- thetic queries. The prompt seeks to lead the model to generate a good query for a new passage. Our prompt template for this stage is given in Figure 3. We create Y cdrpus-adapted prompts in this fash- ion, which vary according to the demonstrations they include. In our experiments, eral values for Y. specifically, 1, 5, and 10. This programmatic creation of few-shot demonstrations for language models is inspired by the Demonstrate stage of the DSP framework (Khattab et al., 2022).
Stage 2:
These pretraining objectives can be paired with additional domain adaptation strategies. Wang et al. (2022) coupled ICT with synthetic query data to achieve domain adaptation in dense retrieval mod- els without the need for annotations. Dai et al. (2022) also paired the contrastive learning objec- tive in Izacard al. (2021) with their unsupervised Promptagator strategy. While our zero-shot domain adaptation approach can pair with other techniques, it does not require any further pretrainingfor bol- stered retrieval performance; our approach only needs the language model pretraining of the re-
test out sev-

---

Prompt #1
Prompt #2
Example It
Frample 1:
Document: We don't know lot about the ef ectts of caffeine during pregnancy you and your besy, So it's best to limit the smount you get each day. if you are pregnant, Imme calleine to 200 mill millgrams each day. This about the amount in 1%/ B-ounce cups coffee or one 12-ounce cup of coffee.
Document: We don't know lot about the effects of caf during pregrentyy on you and your beby So it's best to limit the amount you each cny, if you are pregrant, kmit caffeine to 200 millg miligramss each day, This about she amount in 1/s B-ounce cupe ooffee or one 12-aunce cup of coffee. Relevant Querry: Is little carte ne ok during pregnancy?
Good Queestion: How much caffe ne ok for pregnant woman to have?
Bad Question: a Itte caffeine ok during pregnancy??
Example 2:
Document Passfforr herbartane, rane possion fruit natuv Austra lo. Fruits are green skinnee, white fisshed. with an uninown edible rating Some sources list the fruit as cdlbie, swoet and tazry, while others list the fruits as beine bitter and ineciblc. Relovant Querry: What fruit is rative Australia?
Example 2:
Decument: Passiffora herberriana, A rare passion fruit notive to Australia, Fruits are green- skinned, white fleshed, with an unknoww cditio rating, Same sources list the fruit as adibio, sweet and tasty, while others lst the fruits as being bitter and indiible.
Good Quasilio: (What Passif ora hat be time and now daas taste like?
passion Truit)
Example 3:
Document: The Canadian Armid Faross. I Tha large scale Danadian peacelaeping missionn started in Egyot November 1858 There eccrocimately 65,000 Reguli Foroe and 25,000 reservist members the Canadien miltery. 3-tn Canada, August designa Natio Pescwheepers Rulovant Query: How ge the Cansdian libey?
Ead Question: What fruit is native to Australie?
Example 3:
Document: The sedien Armed Forces. 1 The first large-scale Canadin pessosoesping
stanted Egypt Novembur 1886. There are approsimately 000
force and 25,000 pD0 members the Canadian milliary sarliary. 3 Canadaa, fugust cesigrated us National Procoireepens Day Good Queestion: Informat
Example d:
Document:: <Rousage eevt heree- Relevannt Query'
Canacler Armedl Forces
history
Bad Question: How
Canadian milltry?
Example d:
Document: <Fassage
Good Question:
Prompt #3
Prompt #4
Prompt #5
Query
Question
by
by given Passagg: <Passugg hunes
following Decument.
following Passage.
Passage:
Pastage Query:
Passorge: xPousage
Query"
Figure 2: The five prompts used in Stage 1 (Section 3). The sew-shot prompts #1 and #2 were inspired by Bonifacio al. (2022) while the zero-shot prompts #3, #4, and #5 were inspired by Asai (2022). In our experiments, we prompt GPT-3 in this stage to generate an initial set of queries.
did in Stage 1. In our experiments, we test 1K. 10K, 100K, and 1M as values for Z. We primarily focus on Z = 10K and 100K in Scction 4.3.
Example 1:
Document: <sampled in-domain passage IT>> Good Question: <good synthetic question
Bad Question: <bad synthetic question #1>
We use multiple corpus-adapted prompts to mit- igate edge cases in which we create a low-quality prompt based on the chosen synthetic queries and in-domain passages from the target domain T. (See Stage below for a description of how low-quality prompts can optionally be detected and removed.)
Example 2:
Document: <sampled in-domain passage #2>
Good Question: egood synthetic question N2>
Bad Question: <bad synthetic question #2>
Example 3:
Document: <sampled in-domain passage #3>
Good Question: <good synthetic question #3>
Bad Question: <bad synthetic question #3>
As a quality filter for selecting synthetic queries, we test whether synthetic query can return its gold passage within the top 20 retrieved results using zero-shot ColBERTv2 retriever. We only use synthetic queries that pass this filter, which has been shown to improve domain adaptation in neural IR (Dai et al., 2022: Jeronymo et al., 2023).
Example 4:
Document: <Passage text here>
Good Question:
Figure 3:
The prompt template used in Stage 2. (Sec-
tion 3). In our experiments, we create Y variants of this prompt, and each one is used with Flan-T5 XXL to generate Z queries for each Y.
Stage 4:
With each set of synthetic queries gen- erated using the Y corpus-adapted prompts in Stage 3, we train an individual passage reranker from scratch for the target domain T. These will be used teachers for single ColBERTv2 model in Stage 5. Our multi-reranker strategy draws in- spiration from Hofstätter et al. (2021), who found a teacher ensemble effective for knowledge distil- lation into retrievers. At this stage, we can simply
Stage 3: Each of the corpus-adapted prompts from Stage 2 is used to generate unique set of Z queries with Flan-T5 XXL (Chung al., 2022). The gold passage for each synthetic query is the passage it was generated from. We have the option of letting be large because using Flan-T5 XXL is considerably less expensive than using GPT-3 as

---

use all Y of these rerankers for the distillation pro- cess. As an alternative, we can select the N best rerankers based on their accuracy on the validation set of the target domain. For our main experiments, we use all of these rerankers. This is the most general case, in which we do not assume that a validation set exists for the target domain. (In Ap- pendix A, we evaluate settings where subset of them is used.)
datasets (He et al., 2021; Brown et al., 2020). Sim- ilarly, the raw StackExchange answers and ques- tions (i.e., from which LoTTE-Forum is derived) may overlap in part with the training data of GPT-3. The overlap between pretraining and evaluation datasets may impact the efficacy of domain adap- tation on NQ and SQuAD, leading to increased accuracy not caused by our approach.
4.3 Multi-Reranker Domain Adaptation
The domain-specifie passage rerankers from Stage 4 serve as multi-teachers creating a single ColBERTv2 retriever in a multi-teacher dis- tillation process. For distillation, we use annotated triples that are created by using the trained domain- specific reranker to label additional synthetic ques- tions. Overall, our distillation process allows us to improve the retrieval accuracy of ColBERTv2 without needing to use the rerankers at inference.
Stage 5:
Table 1 provides our main results for UDAPDR accuracy. For these experiments, we total bud- get of 100K synthetic queries and distribute these equally across the chosen number of rerankers to be used as teachers in the distillation process. When exploring UDAPDR system designs, we report dev results, and we report core test results for LoTTE and BEIR in Section 4.7.
We compare UDAPDR to two baselines. In the leftmost column of Table I, we have Zero-shot ColBERTv2 retriever (no distillation). This model has been shown to be extremely effective in our benchmark domains, and it very low latency, so serves as an ambitious haseline. In the rightmost column, we have a Zero-shot ColBERTv2 retriever paired with a single non-distilled passage reranker, trained on 100K synthetic queries. We expect this model to be extremely competitive in terms of ac- curacy, but also very high latency.
Stage 6: We test BERTv2 retriever on the evaluation set for the tar- get domain T. This corresponds to deployment of the retriever within the target domain.
domainn-dappee Col-
4 Experiments
4.1 Models
We leverage the Mamoontrate-SSarch--Preeee (DSP) (Khattab et al., 2022) codehase for running our ex- periments. The DSP architecture allows us to build modular system with both retrieval models and LLMs. For our passage rerankers, we selected DeBERTaV3-Large (He al., 2021) as the cross- encoder after performing comparison experiments with RoBERTa-Large (Liu et al., 2019), BERT (De- vlin et al., 2019), and Mv2 (Wang et al., 2021). For our IR system, we use the ColBERTv2 retriever since it remains competitive for both ac- curacy and query latency across various domains and platforms (Santhanam et al., 2022c).
All versions of UDAPDR are far superior Zero-shot ColBERTv2 across all the domains we evaluated. In addition, two settings of our model are competitive with or superior to Zero-shot Col- BERTv2 plus a Reranker: distilling into Col- BERTv2 the scores from 5 rerankers, each trained on 20k synthetic queries, as well as 10 rerankers, each trained on 10k synthetic queries.
4,4 Query Latency
The accuracy results in Table show that UDAPDR is highly effective. In addition to this, Table 2 reports a set of latency evaluations using the LoTTe Lifestyle section. Our latency costs refer to the complete retrieval process for a single query, from query encoding to the last stage of passage retrieval. Zero-shot ColBERTv2 is known to have low re- trieval latency (Santhanam et al., 2022a). However, its accuracy (repeated from Table 1), which is state-of-the-art level (Santhanam et al., 2022c), trails by large margins the methods we propose in this work. UDAPDR (line 2) has similar latency but also achieves the best accuracy results. The
4.2 Datasets
For our experiments, we use LoTTE (Santhanam et al., 2022b), BEIR (Thakur et al., 2021), NQ (Kwiatkowski et al., 2019). and SQuAD (Rajpurkar et al., 2016). This allows us to cover both long-tail IR (LoTTE, BEIR) and meneral-knowledeee ques- tion answering (NQ, SQuAD).
We note that NQ and SQuAD were both part of Flan-T5's pretraining datasets (Chung et al., 2022). Wikipedia passages used in NQ and SQuAD were also part of DeBERTaV3 and GPT-3's pretraining

---

ColBERTv2 Distillation with UDAPDR
Zero-shot
Zero-shot ColBERTv2
Y = reranker,
Y rerankers,
Y= 10 rerankers,
ColBERTv2
= 100k queries
Z 20k queries
= 10k queries
Reranker
LoTTE Lifestyle
64.5
73.0
74.8
74,4
73.5
LoTTE Techology
44.5
50.2
51.3
51.1
50.6
LoTTE Writing
80.0
84.3
85.7
86,2
85.5
LoTTE Recreation
70.8
76.9
80,4
79.8
79.1
LoTTE Science
61.5
65.6
67.9
68.0
67.2
LoTTE Pooled
63.7
70.0
72.1
72.2
71.1
NaturalQuestions
68.9
72.4
73.7
74.0
73.9
SQuAD
65.0
71.8
73.8
73.6
72.6
Domain Adaptation Strategies with Different Synthetic Query Counts. LoTTE results are for the Forum configuration. All results are for dev sets. The reranker used is DeBERTa-v3 --EEra-----Large. with Z synthetic queries hefore distilling each reranker
Table Success 3 for
The ColBERTv2 distillation strategies train Y rerankers with the same C model. No selection process for rerankers needed access to annotated in-domain dev sets (cf. Table 7 in our Appendices). The non-distilled reranker in the final column is trained on 100K synthetic queries created using Flan-TS XXL model and the prompting strategy outlined Section
Passages
Query Latency
Retriever and Reranker
Success@5
Reranked
Zero-shot ColBERTv2
N/A
35 ms
64.5
ColBERTv2 Distillation:
rerankers, 20k queries
N/A
35 ms
74.8
Zero-shot ColBERTv2+ Reranker
20
412
73.3
Zero-shot ColBERTv2 Reranker
100
2060 ms
73.5
Zero-shot ColBERTv2 + Reranker
1000
20600 ms
73.5
Table 2: Average Single Query Latency for Retrieval + Reranker Systems. Latencies and Success@5 are for LoTTE Lifestyle. The ColBERTv2 distillation strategies train Y rerankers each with synthetic queries before distilling each reranker with the same ColBERTv2 model. These experiments were performed on a single NVIDIA V100 GPU with Py Torch, version 1.13 (Paszke et al., 2019). Query latencies rounded to three significant digits.
Zero-shot ColBERTv2 + Reranker models come close, but only with significantly higher latency.
for UDAPDR. We tested using DeBERTaV3-Baal instead of DeBERTaV3-Large for passage reranker, decreasing the number of model parame- ters by over 70%. We found that DeBERTaV3 was still effective, though results in a 4.1 point drop in Success@5 compared to DcBERTaV3: Large for LOTTE Pooled (Table 3). (In our initial explorations, we also tested using BERT-Base or RoBERTa-Large as the cross-encoder but found them less effective than DeBERTaV3, leading to 6-8 point drops in Success@5.)
4.5 Impact of Pretrained Components
UDAPDR involves three pretrained components: GPT-3 to generate queries, Flan-T5 XXL generate our second, larger set of synthetic queries, and DeBERTaV3- Large for the passage rerankers. What is the impact these specific components on system behavior? To begin to address this question, we explored range of variants. These results summarized in Table 4. Our primary setting for UDAPDR per- forms the best, but noteworthy that very com- petitive performance can be obtained with no use GPT-3 at all. Additionally, we tried using Flan- T5 XL instead of Flan-T5 XXL for the second stage of synthetic query generation, since it is more than 90% smaller than Flan-T5 XXL in terms of model parameters. This still leads to better performance than Zero-shot Co1BERTv2,
initial set of synthetic
4.6 Different Prompting Strategies
We tested whether a simpler few-shot prompting strategy might be better than our corpus-adapted prompting approach for domain adaptation. In Ta- ble 4, we compare the InPars (Bonifacio et al., 2022) few-shot prompt our corpus-adapted prompt approach for synthetic query generation and passage reranker distillation. We evaluate these using query generation with both Flan XXL and GPT-3. We find that our multi-reranker reranker, corpus-
We also explored using a smaller cross-encoder

---

Query Generators
Passage Reranker
Success @
GPT-3 Flan-T5 XXL
DeBERTav3 Large
71.1
GPT-3 Flan-T5 XL
DeBERTav3 Large
66.7
Flan-T5 XXL
DeBERTay3 Large
68.0
Flan-T5 XI.
DeRERTav3 Large
65.9
GPT-3 Flan-T5 XXL
DeBERTay3 Base
67.0
GPT-3 Flan-T5 XI.
DcRERTav3 Basc
64.1
Coto-sho ColBERTV2
Zero-shot
N/A
63.7
Table 3: Model Configurations for Synthetic Query Generation and Passage Reranker. We describe the first and second query generators for UDAPDR in Section 3. The Success@5 5 scores are for the LoTTE Pooled dev task. A single non-distilled reranker is trained on 100K synthetic queries for each configuration. We do not explore configuration with exclusively GPT-3 generated queries due to GPT-3 API costs.
Prompt Strategy
Query Generators
Reranker Count
Success@5
InPars Prompt
GPT-3 Flan-T55 XXL
65.8
InPars Prompt
67.6
InPars Prompt
Flan-T5 XXL.
67.1
Corpus-Adapted Prompts
GPT-3 Flan-TS XXL
67.4
Corpus- Adapted Prompts
GPT-3 + Flan-T5 XXL
71.1
Zero-shot ColBERTV2
N/A
N/A
63.7
Table 4: Model Configurations for Prompting Strategies. We specify the prompting strategy, query generators, and reranker counts for each configuration. The Success @ scores are for the LoTTE Pooled dev task. 100,000 synthetic queries total are used for each approach except for the top row, which uses 5,000 synthetic queries due the costs of the GPT-3 API. The synthetic queries are split evenly amongst the total rerankers used. The rerankers are distilled with a ColBERTv2 retriever for configuration.
adapted prompting strategy is more successful, leading 3.5 point increase in Success@5 af- ter ColBERTv2 distillation while using the same number of synthetic queries for training.
4.8 Additional Results
Table and Table explore only a limited range of potential uses for UDAPDR. In Appendix A, we consider a wider range of uses. First, we ask whether is productive to filter the set of rerankers based on in-domain dev set performance. We mostly find that this does not lead to gains over simply using all of them, and it introduces the re- quirement that have labeled dev set. Second, evaluate whether substantially increasing the value of Z from 100K leads improvements. We find that it does not, and indeed that substantally larger values of Z can hurt performance.
4.7 LoTTE and BEIR Test Results
Table and Table 6, we include the test set results for LoTTE and BEIR, respectively. For LoTTE, UDAPDR increases C zero-shot Success @5 for both Forum queries and Search queries, leading to 7.1 point and a 3.9 point av- erage improvement, respectively. For BEIR, calculated ColBERTv2 accuracy using nDCG@ 10. We found that UDAPDR increases zero-shot ac- curacy by 5.2 points on average. Promptagator++ Few-shot offers similar improvements to zero-shot accuracy, achieving a 4.2 point increase compared to a zero-shot ColBERTv2 baseline. However, Promptagator++ Few-shot also uses reranker dur- ing retrieval, Icading to additional computational costs inference time. By comparison, UDAPDR is a zero-shot method (i.e., that does not assume access to gold labels from the target domain) and only uses the ColBERTv2 retriever and thus has a lower query latency at inference time.
Discussion & Future Work
Our experiments with UDAPDR suggest several directions for future work:
While we show that our domain adaptation strategy is effective for the multi-vector Col- BERTv2 model, whether also effective for other retrieval models an open question for future research.
For our base model in ColBERTv2, use BERT-Base. However, ColBERTv2 now al- lows for other base models, such as DeBER- TaV3, ELECTRA (Clark et al., 2020), and

---

LoTTE Datasets
Forum
Scarch
Life.
Tech.
Writing Rec.
Science Pooled
Life.
Tech.
Writing Rec.
Science Pooled
BM25
60.6
39.4
64.0
55.4
37.1
47.2
63.8
41.8
60.3
56.5
32.7
48.3
SPLADEV2
74.0
50.8
73.0
67.1
43.7
60.1
82.3
62.4
77.1
69.0
55.4
68.9
Rvc
73.7
47.3
71.5
65.7
38,0
57.7
82.1
63.4
78.0
72.1
55.3
69.8
Zero-shot V
76.2
54.0
75.8
69.8
45.6
62.3
82.4
65.9
80.4
73.2
57.5
71.5
UDAPDR
84.9
59.9
83.2
78.6
48.8
70.8
86.8
67.7
84.3
77.9
61.0
76.6
Table 5: Buc for LoTTE Test Set Results. The ColBERTv2 distillation strategies train Y rerankers each with 2 synthetic queries before distilling each reranker with the same ColBERTv2 model. For UDAPDR, we use 5 rerankers and 20K distinct synthetic queries for training each them. BM25, SPLADEv2, and RocketQAv2, take results directly from Santhanam al. (2022h).
BEIR Datasets
ArguAna Touché Covid NFcopus HotpoiQA
DBPedia Climste-FEVER FEVER SciFact SCIDOCS FIQA
BM25
31.5
36,7
65.6
32.5
60.3
31.3
21.3
75.3
66.5
15.8
23.6
DPR (MS MARCO)
41.4
56.1
20.8
37.1
28.1
17.6
58.9
27.5
ANCE
41.5
65.4
23.7
45.6
28.1
19.8
66.9
50.7
12.2
29.5
ColBERT
67.7
59.3
TAS
42.7
48.1
31.9
58.4
38.4
2288
70.0
64.3
14.9
30,0
RacketQAv2
45.1
24.7
67.5
29.3
53.3
35.6
18.0
67.6
56.8
13.1
30.2
SPLADEv2
27.2
71.0
33.4
68.4
23.5
78.6
33.6
BM25 Reranking w/ Coheresarge
46.7
27.6
80.1
34.7
58.0
37.2
25.9
67.4
72.1
19.4
41.1
Reranking OpenAl
81.3
23.7
Zero-shot ColBERTv2
46.1
26.3
84.7
33.8
70.3
44.6
27.1
78.0
66.0
15.4
45.8
Gcoi
49.3
18.2
61.9
319
33.4
32.8
17.3
66.9
64.4"
14.3
"5088
GPL. TSDAE
74.9
22.2
34.4
UDAPDR
57.5
32.4
88.0
34.1
75.3
47.4
33.7
83.2
72.2
17.8
53.5
Promptagator Sew-shol
63
38.1
76.2
32
73.6
43.4
24.1
86.6
73.1
20.1
49.4
Table nDCG@10 for BEIR Test Set Results. For each dataset, the bighest-accuracy zero-shot result is marked in bold while the highest overall is underlined. For UDAPDR, use 5 rerankers and 20K distinct synthetic queries for training each of them. The Promptagator and GPL results are taken directly from their respective papers. For Promptagator, we include both the best retriever-only configuration (Promptagator Few-shot) and the best retriever reranker configuration (Promptagator++ Few-shot). We include the GPL+TSDAF, pretraining strategy. which is found to improve retrieval accuracy (Wang et al., 2022). We copy the results for BM25, GenQ, ANCE, TAS-B, and ColBERT from Thakur et al. (2021), for MoDIR and DPR-M from Xin et al. (2022), for SPLADEv2 from Formal et al. (2021), and for BM25 Reranking of Cohere and OpenAlada2 from Kamalloo et al. (2023).
RoBERTa (Liu al., 2019). We would be interested to see the efficacy of our domain adaptation strategy with these larger encoders.
Conclusion
We present UDAPDR, a novel strategy for adapting retrieval models to new domains. UDAPDR uses synthetic queries created using generative models, such GPT-3 and Flan- T5 XXL, to train multiple passage rerankers on queries for target domain pas- sages. These passage rerankers are then distilled into ColBERTv2 to boost retrieval accuracy while keeping query latency competitive as compared to other retrieval systems. We validate our approach acToss the LoTTE, BEIR, NQ, and SQuAD datasets. Additionally, we explore various model configura- tions that alter the generative models, prompting strategies, retriever, and passage rerankers used in our approach. We find that UDAPDR can boost zero-shot retrieval accuracy on new domains with- out the use labeled training examples. We also discuss several directions for future work.
We explored distillation strategies for combin- ing passage rerankers with ColBERTv2. How- ever, testing distillation strategies for shrink- ing the reranker itself could be viable direc- tion for future work.
We draw upon several recent publications, in- cluding Bonifacio al. (2022) and Asai et al. (2022), create the prompts used for GPT-3 and Flan-T5 XXL in our domain adaptation strategy. Creating a more systematic approach for generating the initial prompts would be an important item for future work.

---

7 Limitations
Rodrigo Nogueira. 2022. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202 2e..22222.....2. 04.44.
While our domain adaptation technique does not require questions or labels from the target domain, it does require a significant number of passages in the target domain. These passages are required for use in synthetic query generation with the help of LLMs like GPT-3 and Flan-T5, future work should evaluate how effective these methods are on extremely small passage collections.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Wei-Cheng Chang. Felix X. Yu, Yin-Wen Chang, Yim- ing Yang, and Sanjiv Kumar, 2020. Pre-training Tasks for humbedding-baser Large-scale Retrieval. In International Conference on Learning Representa- tions.
The synthetic queries created in our approach may also inherit biases from the LLMs and their training data. Moreover, the exact training data of the LLMs is not precisely known. Our understand- ing is that subsets of SQuAD and NQ, in particular, have been used in the pretraining of Flan-T5 T5 mod- els as we note in the main text. More generally, other subtle forms of data contamination are possi- ble as with all research based on LLMs that have been trained on billions of tokens from the Web, We have mitigated this concern by evaluating on a very large range of datasets and relying most heav- ily on open models (i.e., Flan-T5, DeBERTa, and ColBERT). Notably, our approach achieves consis- tently large gains across the vast majority of the many evaluation datasets tested (e.g., the individ- ual sets within BEIR), reinforcing our trust in the validity and transferability of our findings.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddharthha Brahma, et al. 2022. Scaling instruction-finetunee language models. arXiv preprint arXiv:2210.11411
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D Manning. 2020. ELECTRA: Pre- training text encoders as discriminators rather than generators. arXiv preprint urXiv:2003. 10555.
Zhuyun Dai, Vincent Y Zhao, Ji Ma. Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wes Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint arXiv:2209. 11755.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Additionally, the LLMs used in our technique benefit substantially from GPU-based hardware with abundant and rapid storage. These technolo- gies are not available to all NLP practitioners and researchers due to their costs. Lastly, all of our selected information retrieval tasks in English, a high-resource language. Future work can ex- pand on the applicability of our domain adaptation techniques by using non-Englise passages in low- resource settings, helping us better understand the approach's strengths and limitations.
Dheeru Dua, Emma Strubell, Sameer Singh, and Pat Verga. 2022. To adapt or to annotate: Chal- lenges and interventions for domain adaptation in open-domain question answering. arXiv preprint arXiv:2212.10388
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2021. Splade v2: Sparse lexical and expansion model for informa- tion retrieval. arXiv preprint arXiv:2109. 10086.
References
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In Pro- ceedings of the AAA/ Conference on Artificial Intelli- gence, volume 34, pages 7383-7390.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training, In International Con- ference on Machine Learning, pages PMLR
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Ha- jishirzi, and Wen-tau Yih. 2022. Task-aware retrieval with instructions. arXiv preprint arXiv:221 0et::22/1109966 1.09260.
Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2018. Neural Vector Spaces for Unsuper- vised Information Retrieval. ACM Trans. Inf. Syst., 36(4).
Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and

---

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3: Improving deberta using electra-style pre-training with cradient-diseetangledd embedding sharing. arXiv preprint arXiv:2111.09544
Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39-48. ACM.
Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haf- fari, and Mohammad Norouzi. 2022. Generate, An- notate, and Learn: NLP with Synthetic Text. Trans- actions of the Association for Computational Linguis- tics, 10:826-842.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2021. Improving efficient neural ranking models with cross- architecture knowledge distillation.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data Augmentation using Pre-trained Trans- former Models, In Proceedings of the 2nd Workshop Life-long Learning for Spoken Language Systems, pages 18-26, Suzhou, China. Association for Com- putational Linguistics.
Jeremy Howard and Sehastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volame I: Long Papers), pages 328-339, Melbourne, Australia, Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Samuel Humeau, Kurt Shuster, Marie Lachaux. and Jason Weston. 2019. Poly-encoders: Trans- former architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv prepriat arXiv: (90::::905009999
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense infor- mation retrieval with contrastive learning.
Patrick Lewis, Ethan Perez. Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmente generation for wowlledg-inttesiie NLP tasks. Advances in Neural Information Processing Systems, 33:9459- 9474.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299
Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadace, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.0188.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907. 1692.
Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-Hermelo, Mehdi Rezagholizadeh, and Jimmy Lin. 2023, Evaluating embedding apis for information retrieval.
Ji Ma. Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan MeDonald. 2020. Zero-shot neural passage retrieval via targen-targeted synthetic question generation. arXiv preprint arXiv:2004. 14503.
Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval. In Thirty-Fifth Fifth Conference on Neural Information Processing Systems.
Rui Meng, Ye Liu, Semih Yavuz, Divyansh Agarwal, Lifu Tu, Ning Yu, Jianguo Zhang, Meghana Bhat, and Yingbo Zhou. 2022. Unsupervised dense retrieval deserves better positive pairs: Scalable augmentation with query extraction and generation. arXiv preprint 222.0.221..08841.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-SSarch Predict: Composing retrieval and language mod- els for knowledge-intensivy nlp. arXiv preprint arXiv:2212. 14024.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPs

---

Rodrigo Nogueira and Kyunghyun Cho. 2019. Pas- sage re-ranking with BERT. arXiv preprint arXiv:1901.044 040055
Augmentation Method for Improving Bi- Bi-Encoders for Pairwise Sentence Scoring Tasks. arXiv preprint 200.:22000006240.
Rodrigo Nogueira, Jimmy Lin, and A1 Epistemic. 2019. From doc2query to docttttqquey. Online preprint, 6.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty.fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, peeh-eerformanc deep learning library. Advances in neural information processing systems, 32.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin, 2017. Attention is all you need, In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 30, pages ciates, Inc.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin. Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2523-2544, Online. Association for Computational Linguistics.
Curran Asso-
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Sohoroff, and Lucy Lu Wang. 2021. TREC-COVID: constructing pandemic information retrieval test collection. In ACM SIGIR Forum, vol- ume 54, pages 1-12. ACM New York, NY, USA.
Pranav Rajpurkar, Robin Jía, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers). pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022, GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 2345-2360, Seattle, United States. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Lucy Lu Wang, Kyle Lo. Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang. Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang. Nancy Xin Ru Wang. Christopher Wilhelm, Boya Xie, Douglas M. Ray- mond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020, CORD-19: The COVID- open research dataset. In Proceedings of the 1st Work- shop NLP for COVID-19 at ACL 2020, Online. Association for Computational Linguistics.
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022a. PLAID: an efficient en- gine for late interaction retrieval. In Proceedings of the st ACM International Conference on Informa- tion& & Knowledge Management, pages 1747-1756.
Keshav Santhanam, Omar Khattab, Jon Fald-Fa Christopher Potts, and Matei Zaharia. 2022b. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3715-3734, Seat- tle. United States. Association for Computational Linguistics.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong. and Furu Wei. 2021. Mv2:LM Multi-head self- attention relation distillation for compressing pre- trained transformers. In Findings of the Association for Computational Linguistics: ACI-IJCNILP 2021, pages 2140-2151, Online. Association for Computa- tional Linguistics.
Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avirup Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, and Christo- pher Potts. 2022c. Moving beyond downstream task accuracy for information retrieval benchmarking. arXiv preprimi arXiv:2212.01330..
Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. 2022. Zero- shot dense retrieval with momentum adversarial do- main invariant representations. In Findings of the As- sociation for Computational Linguisties: ACL 2022, pages 4008 40000 Dublin, Ireland. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020. Augmented SBERT: Data

---

Yiben Yang. Chaitanya Malaviya, Jared Fernandez. Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative data augmentation for common- sense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1008-1025, Online. Association for Computational Linguistics.

---

Appendix
A Reranker Configurations
We aim understand the impact of different model configurations on the efficacy of our domain adaptation technique. We expand on experiments from Section 4.3 and explore alternate configurations of key factors the UDAPDR approach. Specifically, we want to answer the following questions through their corresponding experiments:
1. (a) Question: Do corpus-adapted prompts improve domain adaptation and retriever distillation?
(b) Experiment: Compare zero/few-sho prompts to corpus-adapted prompts for synthetic question generation (Table 4).
2. (a) Question: How does the number of rerankers affect downstream retrieval accuracy?
(b) Experiment: Compare single-reranker to multi-reranker approach (with different reranker
selection strategies) across various target domains (Table 7).
3. (a) Question: How does the synthetic query count affect downstream retrieval accuracy?
(b) Experiment: Explore different query count configurations ranging from several thousand hundreds of thousands of synthetic queries (Table 1).
4. (a) Question: How do triple counts during distillation affect domain adaptation for ColBERTv22
(b) Experiment: Explore different triple counts ranging from several thousand to millions of triples (Table 8).
In Table 7 and Table 8 (and Table I in the main text). we outline the results of different experimental configurations in which we alter synthetic query generation and passage reranker training. Based on results for the LoTTE pooled dataset, we find that training multiple rerankers and selecting the best performing rerankers can improve our unsupervised domain adaptation approach. We generate multiple corpus-adaptedd prompts and rerankers prevent edge cases in which sampled in-domain passages and queries have low quality. Furthermore, distilling the passage rerankers with ColBERTv2 allows us preserve their accuracy gains while avoiding their high computational costs. However, training many rerankers and selecting the best five rerankers can be computationally intensive and ultimately unnecessary to achieve domain adaptation. The simpler approach of training several rerankers and using them for distillation, without any quality filtering, yields comparable results with only a 0.6 point drop in Success @5 average while computing 10x less synthetic queries (Table 7). Additionally, by using rerankers to generate more triples for distillation with ColBERTv2, we were able boost performance even further as shown in Table 8.
B Fine-tuning Rerankers and Retriever
For all passage reranker models that we fine-tune, optimize for cross-entropy loss using Adam (Kingma and Ba. 2014) and apply 0.1 dropout the Transformer outputs. We feed the final hidden state of the [CLS] token into a single linear classification layer. We fine-tune for epoch configurations. Additionally, we using a 5e-6 learning rate combined with a linear warmup and linear decay for training (Howard and Ruder, 2018). We use a batch size of 32 across all experimental configurations. For our Co1BERTv2 retriever, we use le-5 learning rate and a batch size of 32 during distillation. The ColBERTv2 maximum document length is set to 300 tokens. We use a BERT-Bass model (Devlin et al., 2019) as encoder.
experimental
Instead of fine-tuning the rerankers, we also tried fine-tuning ColBERTv2 directly with the synthetic datasets. We found that fine-tuning the retriever directly with the synthetic queries offered only limited benefits, only improving zero-shot retrieval by 1-3 points of accuracy at best and decreasing zero-shot accuracy at worst (for the LoTTE Forum dev set). Distilling the rerankers offered more substantive gains and better adaptation to the target domains more generally.

---

ColBERTv2 Distillation with UDAPDR
Zero-shot
Zero-shot ColBERTv2 Rerankers
1 of 50
5 of 50 Rerankers
10 of 50
5 of 5
ColBERTv2
Rerankers Rerankers
Reranker
LoTTE Lifestyle
64.5
67.5
73.2
73.6
72.8
73.5
LoTTE Techology
44.5
46.3
50.3
50.7
49.9
50.6
LoTTE Writing
80.0
81.5
83.4
84.2
83.0
85.5
LoTTE Recreation
70.8
73.7
77.8
78.3
76.7
79.1
LoTTE Science
61.5
63.0
66.3
66.8
65.5
67.2
LoTTE Pooled
63.7
66.3
70.3
70.7
69.6
71.1
NaturalQuestions 68.9
70.1
73.0
73.5
72.7
73.9
SQuAD
65.0
67.2
71.0
71.3
70.4
72.6
Domain Adaptation Strategies with Different Reranker Counts. LoTTE dataset results correspond to the Forum configuration. All results correspond to dev sets of each task. The reranker used in the experiments is 1asERRa--v--arge The ColBERTv2 distillation strategies train number rerankers before selecting the best Y based their performance the dev of the target domain. Through the selection process, we aim to find an upper bound for retrieval accuracy, even though access to an annotated dev set is not realistic for all domains. our distillation strategies, each reranker was trained on 2,000 synthetic queries. For non-disto dintiitileed reranker used in the final column, we trained it 100,000 synthetic queries. The synthetic queries were created using Flan-T5 XXL model and the prompting strategy outlined in Section 3.
Table 7: Success@5 for
ColBERTv2 Distillation with UDAPDR
Zero-shot Co1BERTv2
Zero-shot ColBERTv2
1000
10000 triples
100000 triples
triples
+ Reranker
LoTTE Lifestyle
64.5
67.4
70.1
72.4
73.5
LoTTE Techology
44.5
46.0
47.8
50.2
50.6
LOTTE Writing
80.0
82.5
83.4
85.0
85.5
LOTTE Recreation
70.8
72.3
76.5
77.7
79.1
LoTTE Science
61.5
62.0
64.2
66.5
67.2
LoTTE Pooled
63.7
66.0
68.4
70.4
71.1
NaturalQuestions 68.9
70.5
72.7
73.4
73.9
SQuAD
65.0
67.2
70.6
71.5
72.6
Table 8: @uccess@5 for Multi-Reranker Domain Adaptation Strategies with Various Distillation Triples Counts. LoTTE dataset results correspond the Forum configuration. All results correspond to dev sets of each task. The reranker used in the experiments is DeBBERTa-v3-Large The ColBERTv2 distillation strategies use a single reranker trained on 10,000 synthetic queries; this reranker then generates the specified number of labeled triples. For our non-disto non-distilled reranker used in the final column, trained it on 100,000 synthetic queries. The synthetic queries were created using Flan-TS XXL model and the prompting strategy outlined in Section

---

We ran additional experiments testing UDAPDR's efficacy on LOTTE Search dev, using one reranker trained with unique set of 2000 synthetic queries. We found that the approach boosted accuracy by 1.6 points, increasing accuracy from 71.5 to 73.1. However, since the synthetic queries could be generated so cheaply, we decided to scale to tens of thousands of synthetic queries for further experiments.


Reviews:
[{"review_id": "pjHkWQp2xS", "reviewer": "Reviewer_H5VJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback on our submission! We are delighted that you found that our key idea can help “unleash the power of LLMs” and that our results “show [its] effectiveness”.\nWe’d like to clarify that only the domain-adapted copy of our neural retriever, ColBERTv2, is used at inference; there is no role for rerankers in the final UDAPDR system. Furthermore, our submission provides several quantitative tables as well as a number of figures to illustrate the strengths of UDAPDR compared to other domain adaptation approaches. Here we address your concerns:\n\n> “this paper focuses solely on the target domains and does not utilize any data from the source domains. Consequently, it may not be appropriate to classify it as UDA..” \n\nWe share your definition of UDA and would like to clarify where we draw upon source domains for our adaptation technique. For UDAPDR, we leverage knowledge gained from the source domains for the initial retriever, the distilled reranker, and the language models used for synthetic data generation. Our retriever, ColBERTv2, uses MS MARCO training data in addition to the pretraining datasets used for BERT. Our reranker, DeBERTa-v3-Large, additionally has its own host of pretraining datasets, such as the Wikipedia passages. Furthermore, GPT-3 and FLAN-T5 also include massive data collections in their pretraining ensemble. For our domain adaptation approach, we combine the latent knowledge gained from pretraining on the various general domain datasets with in-domain passages to generate synthetic questions and fine-tune our rerankers before distillation with the retriever. Hence, we believe it is fitting to consider UDAPDR a method for domain adaptation (or UDA).\n\n\n> “The approach taken by LLMs in addressing the issue of domain divergence remains somewhat perplexing” \n\nTo address domain divergence in our use of LLMs, we prompt the FLAN-T5 and GPT-3 with in-domain passages for generating synthetic questions without the need for in-domain questions or labels. While the synthetic questions do not have the same quality as human-written in-domain questions, they provide sufficient material for adapting our retriever to the specialized domain through the use of reranker distillation. This is a central finding of our work that opens the door to domain transfer, even where human labels are too costly or difficult to obtain.\n\n\n> “The experimental results indicate that the majority of performance improvements stem from the utilization of the reranker”\n\nWe must clarify that our final system does not have a reranker but rather is a single efficient ColBERTv2 retrieval model that is used in the target domain (see Figure 1, Stages 5 and 6). Before inference, our trained rerankers are distilled with the retriever, avoiding their costly latency for users while preserving gains to accuracy. As a result, only our neural retriever is used at inference, which has proven competitive in terms of both accuracy and latency. ([Santhanam et al. 2022](https://aclanthology.org/2023.findings-acl.738/) offers a thorough set of recent benchmark numbers.)\n\n\n> “The paper exclusively provides qualitative results, and it would be more compelling to include additional quantitative examples or visualizations to enhance the persuasiveness of the claims made.”\n\nTable 1 through Table 6 are our _quantitative_ results and they amount to a very strong case for UDAPDR compared to other domain adaptive techniques, such as Promptagator and GPL, as well as effective lightweight retrieval techniques, such as BM25 and SPLADEv2. In Table 6, for example, we include a head-on comparison between current state-of-the-art approaches for the BEIR information retrieval benchmark and show where UDAPDR shows significant gains to other neural retrieval approaches. In terms of _qualitative_ results: we feel that the experimental findings yield a very rich picture of system performance and the impact of different design choices across a wide range of task types from community forums to argument retrieval. \nIn our revised paper, we would be happy to add a graphical comparison to better visualize our contributions. We will also provide qualitative examples of domains where UDAPDR is most effective (e.g. QA datasets like HotpotQA and FiQA) and domains where UDAPDR barely improves on zero-shot ColBERTv2 (medical/scientific datasets like NFCorpus and SCIDOCS).\n\nFinally, we want to mention that UDAPDR will be released as part of an open-source package for doing retrieval and OpenQA. This release will allow the reproduction of the results in the paper, and its primary goal is to make UDAPDR useful for users in general.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Overall, the idea of synthesizing augmented queries with two-staged LLMs is promising.  Efficiency issues are raised by the authors and relieved by a simple strategy like locate-and-refine that unleashes the power of LLMs.\n\n(2) The results on the four datasets including IR and QA show the effectiveness of the proposed method.", "reasons_to_reject": "(1) Based on my understanding, Unsupervised Domain Adaptation (UDA) typically involves transferring knowledge gained from source domains, which have abundant annotated training data, to target domains that have only unlabeled data. However, this paper focuses solely on the target domains and does not utilize any data from the source domains. Consequently, it may not be appropriate to classify it as UDA.\n\n(2) The approach taken by LLMs in addressing the issue of domain divergence remains somewhat perplexing, as these models are typically trained to generate general knowledge without specific fine-tuning for particular domains. The authors have not extensively elucidated this aspect or presented empirical evidence to substantiate their claims.\n\n(3) The experimental results indicate that the majority of performance improvements stem from the utilization of the reranker, a widely adopted component in information retrieval (IR) systems. This somewhat diminishes the significance of leveraging LLMs for addressing domain adaptation challenges.\n\n(4) The paper exclusively provides qualitative results, and it would be more compelling to include additional quantitative examples or visualizations to enhance the persuasiveness of the claims made.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of domain adaptation in neural information retrieval (IR) models. The authors propose a method called Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers (UDAPDR) to facilitate efficient adaptation of neural retriever models to new domains.\n\nThe main contributions of this paper are as follows:\n\n1. UDAPDR: The authors introduce UDAPDR as an unsupervised domain adaptation method for neural IR. It leverages both expensive language models (LLMs) like GPT-3 and less expensive ones like Flan-T5 XXL, along with multiple passage rerankers. This approach aims to improve retrieval accuracy in zero-shot settings for different domains while maintaining competitive latency.\n\n2. Efficient Query Generation: Unlike previous domain adaptation approaches that require millions of synthetic queries, UDAPDR proves effective with only thousands of synthetic queries. The method is compatible with various LLMs designed for instruction-based tasks like query generation. By using multiple prompting strategies, the authors enhance the effectiveness of unsupervised domain adaptation by allowing fine-tuning of multiple rerankers and more effective distillation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3AxFTWC1Lt", "reviewer": "Reviewer_1Fo5", "summary": "", "questions": "1. How does the choice of prompts in Stage 1 affect the overall performance? What if only a subset of them are used? \n2. The prompt template in Stage 2 includes both good and bad questions. However, only Prompt 1 in Stage 1 generates bad questions. Could you clarify how bad questions are generated for other prompts?\n3. What if the generated queries from stage 3 are used for training the retriever directly instead of training a reranker and then distilling? \n4. What's the total cost of running GPT-3 at the first stage?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback on our submission! We are glad that appreciate about our work that it “manages to improve the out-of-distribution generation of neural retrievers without adding latencies”, that its “improvement is consistent across multiple datasets”, and that it’s “more computationally efficient compared to previous adaptation methods using synthetic queries”. We wanted to address your concerns about the paper’s contributions:\n\n> “I don't see the claim in line 107-108, \"our technique only requires 1000s of synthetic queries to prove effective\" is well supported by the experiment.”\n\nThank you for noting this. The paper did previously operate mainly at the level of 10K examples. To justify this claim, we ran a new experiment testing UDAPDR’s efficacy on LoTTE Search dev, using one reranker trained with a unique set of 2000 synthetic queries. We found that the approach boosted accuracy by 1.6 points, increasing accuracy from 71.5 to 73.1. We will include this result and expand it to more settings for the final paper, as it provides important new insights into the method.\n\n\n> “Although Flan-T5 XXL is much smaller than GPT-3, it is a large model and its computational cost is not negligible.”\n\nFLAN-T5 XXL requires significant resources for running inference at full-precision, though that depends on the prompt length. In Table 3, we explore the success of using FLAN-T5 XL (90% smaller than XXL) instead for synthetic generation. Furthermore, we plan to evaluate reduced precision configurations for FLAN-T5 XXL in our revised paper.\n\nAdditionally, we wanted to address each of your questions:\n\n> “How does the choice of prompts in Stage 1 affect the overall performance? What if only a subset of them are used?”\n\nWe did not perform extensive prompt engineering experiments for stage #1 since we found the prompts selected to be effective enough for our approach. From initial exploration, we found that similar prompt configurations were not more effective than the tested prompts. Testing extensive prompt configurations might also lead to an excessive number of experimental configurations that was too large for our available resources.\n\n\n> “The prompt template in Stage 2 includes both good and bad questions. However, only Prompt 1 in Stage 1 generates bad questions. Could you clarify how bad questions are generated for other prompts?”\n\nTo generate all of our bad questions for the other prompts, we solely used prompt #1. We will make sure to clarify this distinction in the revised paper.\n\n\n> “What if the generated queries from stage 3 are used for training the retriever directly instead of training a reranker and then distilling?”\n\nWe found that fine-tuning the retriever directly with the synthetic queries offered only limited benefits, only improving zero-shot retrieval by 1-3 points of accuracy at best and decreasing zero-shot accuracy at worst (for the LoTTE dev set). Distilling the rerankers offered more substantive gains and better adaptation to the target domains more generally. We will include the retriever fine-tuning experiments in our revised paper.\n\n\n> “What's the total cost of running GPT-3 at the first stage?”\n\nDepending on the context lengths of the target domain, the cost of running GPT-3 for the first stage was ~$0.25-1 (for each run per domain) from November 2022 to March 2023. It is approximately the same cost in August 2023.\n\nAlso, thank you for pointing out the grammatical errors and presentational improvements! We will address each of them in our revised paper.\n\nFinally, we want to mention that UDAPDR will be released as part of an open-source package for doing retrieval and OpenQA. This release will allow the reproduction of the results in the paper, and its primary goal is to make UDAPDR useful for users in general.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method manages to improve the out-of-distribution generation of neural retrievers without adding latencies at the test time. The adaptation improvement is consistent across multiple datasets (BEIR & BEIR). This method is more computationally efficient compared to previous adaptation methods using synthetic queries.", "reasons_to_reject": "1. I don't see the claim in line 107-108, \"our technique only requires 1000s of synthetic queries to prove effective\" is well supported by the experiment.\n2. Although Flan-T5 XXL is much smaller than GPT-3, it is a large model and its computational cost is not negligible. \n3. Please see the Questions For The Authors section.", "questions_to_authors": "1. How does the choice of prompts in Stage 1 affect the overall performance? What if only a subset of them are used? \n2. The prompt template in Stage 2 includes both good and bad questions. However, only Prompt 1 in Stage 1 generates bad questions. Could you clarify how bad questions are generated for other prompts?\n3. What if the generated queries from stage 3 are used for training the retriever directly instead of training a reranker and then distilling? \n4. What's the total cost of running GPT-3 at the first stage?", "paper_topic_and_main_contributions": "This paper presents UDAPDR, a novel domain adaptation method for retrieval models. This method leverages an effective LLM to generate queries of the target domain, which are then used to prompt a less expensive LLM to generate more queries to train a reranker. Last, this reranker is distilled into a final retriever. The authors tested their method on LoTTE and BEIR.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. I found the dotted arrow in Figure 1 a bit misleading as it usually indicates an iterative process, which is not the case in this paper. \n2. An average column in Table 6 could be helpful."}, "extra_text": {}}, {"review_id": "6coTzOL0GW", "reviewer": "Reviewer_8wrV", "summary": "", "questions": "1) In section 4.5, when not using GPT3 at all, does it mean Flan-T5 is used in stage 1?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback on our submission! We are happy to see you recognize our in-depth evaluations and ablation studies for UDAPDR and are excited to see that you found it “easy but effective” as is our goal for this work. We hope the experiments illustrate the utility of synthetic query generation and the efficacy of our reranker distillation technique. Here, we wanted to address your concerns about the paper’s contributions:\n\n> “The major concern is that the proposed approach is not novel. The prompts used in stage 1 and 2 are seen from prior works.”\n\nOnly the initial, manual prompts used in Stage #1 are from prior work while the prompts used in Stage #2 are unique and corpus-adapted for each domain. The heart of our approach is a new technique for creating integrated retrievers that are robust across domains. For this, we leverage proven prompting strategies to avoid engaging in unnecessary prompt hacking. This is simply an experimental choice for UDAPDR, not the essence of the method. Overall, we use the queries to train rerankers before distilling them with the ColBERTv2 retriever, bolstering domain adaptation while preserving a competitive retrieval latency at inference.\n\n\n> “Also, compared with few-shot method proposed in promptagator, this paper uses GPT3 to generate corpus-adapted prompt instead of manually select a few examples in promptagator. It is not clear if the proposed approach is more effective and less cost than promptagator.”\n\nThe evidence in the paper strongly supports the claim that UDAPDR is more efficient and effective than Promptagator, GPL, and the other domain adaptation techniques we evaluate. The most successful configuration of Promptagator is Promptagator++, which combines a fine-tuned retriever and reranker. The approach requires several human-annotated examples and utilization of LLMs like FLAN-T5 for synthetic data generation, achieving a 6.2 point increase in nDCG@10 on average for BEIR (46.6 → 52.8). However, their reranker introduces a high cost at inference, which they do not quantify in the paper but recognize as a relevant challenge for future research. Without their reranker, they only achieve 1.2 point increase in nDCG@10 on average for BEIR.\n\nIn comparison, UDAPDR does not use any rerankers at inference, instead distilling them with the ColBERTv2 retriever to preserve competitive latency for users while achieving a 5.2 point increase in nDCG@10 on average for BEIR. Additionally, UDAPDR does not require any human-annotated examples; it also utilizes FLAN-T5 and offers the option to use GPT-3 for higher-quality synthetic queries. Overall, the accuracy gains and greater inference efficiency of UDAPDR give our approach an edge over Promptagator\n\n\n> “ For example, not all tasks in BEIR are question answering, but the prompts used in stage 1 are focused on generating question given document. It seems that it would be difficult to adapt to a different task, e.g. counter argument retrieval in ArguAna.”\n\nUDAPDR still proves effective for BEIR tasks not centered around question-answering by helping adapt the ColBERTv2 retriever towards the in-domain documents. Table 6 reports a full evaluation, which includes an outstanding score on ArguAna – the best zero-shot score. Our two-stage synthetic query generation approach combined with query filtering allows us to generate relevant query/document pairs for fine-tuning rerankers in the new domain. As a result, after distilling our rerankers with the retriever, our retrievers are much better equipped to handle the specialized domains in BEIR. Nonetheless, we fully agree that it is possible to obtain an even more general version of UDAPDR by improving the seed set of prompts. We will urge the community to consider such direct extensions for their own applications.\n\n\n> “In addition, it would be interesting to know the performance of using the generated synthetic set to train a first-stage retrieval model as well.”\n\nWe found that fine-tuning the retriever directly with the synthetic queries offered only limited benefits, only improving zero-shot retrieval by 1-3 points of accuracy at best and decreasing zero-shot accuracy at worst (for the LoTTE Forum dev set). Distilling the rerankers offered more substantive gains and better adaptation to the target domains more generally. We will include the retriever fine-tuning experiments in our revised paper.\n\nAdditionally, we wanted to address each of your questions:\n\nYes, in section 4.5 when we don’t use GPT-3 at all, we use FLAN-T5 in stage #1 instead.\n\nAlso, thank you for pointing out the grammatical errors and presentational improvements! We will address each of them in our revised paper.\n\nFinally, we want to mention that UDAPDR will be released as part of a popular open-source package for doing retrieval and OpenQA. This release will allow the reproduction of the results in the paper, and its primary goal is to make UDAPDR useful for users in general.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a domain adaptation approach that doesn't require any labeled data and a small amount of synthetic data. The proposed method is easy but effective. Extensive evaluation and ablation experiments show the effectiveness of UDAPDR.", "reasons_to_reject": "The major concern is that the proposed approach is not novel. The prompts used in stage 1 and 2 are seen from prior works. \n\nAlso, compared with few-shot method proposed in promptagator, this paper uses GPT3 to generate corpus-adapted prompt instead of manually select a few examples in promptagator. It is not clear if the proposed approach is more effective and less cost than promptagator. For example, not all tasks in BEIR are question answering, but the prompts used in stage 1 are focused on generating question given document. It seems that it would be difficult to adapt to a different task, e.g. counter argument retrieval in ArguAna. \n\nIn addition, it would be interesting to know the performance of using the generated synthetic set to train a first-stage retrieval model as well. ", "questions_to_authors": "1) In section 4.5, when not using GPT3 at all, does it mean Flan-T5 is used in stage 1?", "paper_topic_and_main_contributions": "The paper proposed an unsupervised domain adaptation approach using LLM for synthetic data generation and multiple reranker distillation, called UDAPDR. The proposed method includes 3 stages for synthetic data generation, including generating synthetic queries on a few target domain passages, converting the generated queries into corpus-adapted prompts, and generating large scale synthetic data using the prompts. Then multiple rerankers are trained using the generated set and are then distilled to a single ColBERTv2 retriever. The distilled ColBERTv2 model is efficient and maintain the performance of rerankers. The evaluation on several datasets show the effectiveness of UDAPDR. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. line 019 2k -> 20k?"}, "extra_text": {}}, {"review_id": "PnnO6jsra3", "reviewer": "Reviewer_LCfk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback on our submission! We are glad that you appreciate our ablation study and the practicality of the UDAPDR fine-tuning pipeline, particularly when it comes to reducing the high latency costs of LLMs. We wanted to address your concerns about the use of GPT-3 and the total computational pipeline for UDAPDR:\n\n> “it seems very interesting that by using GPT-3 alone, the performance is already better than zero-shot approaches. it raises the question on the future of this approach since hardware devices will eventually catch up, and using GPT-3 won't be a very time-consuming task anymore.”\n\nUDAPDR offers a general two-stage method for domain adaptation. If GPT-3 becomes a little cheaper, UDAPDR can be configured to make wider use of it, especially in smaller domains. More fundamentally, though, we postulate that the tradeoff between scale (and hence cost) and quality will continue to exist with newer and larger models. (Think GPT-4, GPT-5, etc. so to speak.)\n\nBecause of this, our two-stage approach will continue to have value for scaling down the quality of the strongest LLMs for synthetic query generation so it’s feasible over large corpora.\nIn addition, even if large language models like GPT-3 and FLAN-T5 XXL could be hosted locally and used for reranking, it would be unlikely that their latency costs would be conducive for user-facing settings of information retrieval. Therefore, distillation of rerankers would still be required at some stage to preserve competitive latency at inference.\n\n\n> “i am slightly curious about the total computational cost of training the entire pipeline. There are many recent work on parameter-efficient finetuning and also distillation that potentially also allow you to directly obtain a small model for fast inference at the end. Any of recent finetuning and/or distillation work on top of GPT-3 or FlanT5XXL would be a very strong competitor. Could the authors comment on this?”\n\nOverall, the computational pipeline for UDAPDR ranges from 3 to 6 hours, with additional training time further improving the accuracy and distillation gains of our approach.  For comparison: fine-tuning GPT-3.5 using existing frameworks takes several days when including data validation, uploading, training, and deployment. Parameter-efficient fine-tuning (PEFT) and reduced precision configurations for FLAN-T5 XXL could lead to reduced quality for synthetic query generation in UDAPDR, substantially decreasing the efficacy of our technique, but we could consider them for scenarios where 3-6 hours is too long. \n\n\nFinally, we want to mention that UDAPDR will be released as part of an open-source package for doing retrieval and OpenQA. This release will allow the reproduction of the results in the paper, and its primary goal is to make UDAPDR useful for users in general.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the pipeline takes the inference latency of very large language models into consideration, and only uses it for the very first stage.\n\n2. the pipeline then  reduces the inference latency of multiple rerankers by distilling them into a single retrieval model.\n\n3. ablation study is thorough, and addresses common questions regarding the use of large language models.", "reasons_to_reject": "1. it seems very interesting that by using GPT-3 alone, the performance is already better than zero-shot approaches. it raises the question on the future of this approach since hardware devices will eventually catch up, and using GPT-3 won't be a very time-consuming task anymore.\n\n2. i am slightly curious about the total computational cost of training the entire pipeline. There are many recent work on parameter-efficient finetuning and also distillation that potentially also allow you to directly obtain a small model for fast inference at the end. Any of recent finetuning and/or distillation work on top of GPT-3 or FlanT5XXL would be a very strong competitor. Could the authors comment on this?", "questions_to_authors": "", "paper_topic_and_main_contributions": "the submission proposed a practical pipeline that 1) leverages a very large language model for generating a small batch of samples, and then 2) uses a small language model to enrich the training samples, and  3) train multiple rerankers and distill them into a retrieval model. Although the pipeline seems rather tedious, the performance has improved significantly in comparison to baseline approaches. Ablation study also shows the necessity of individual components.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce UDAPDR, a method for adapting neural retriever models to new domains efficiently. UDAPDR leverages both expensive language models (LLMs) like GPT-3 and less expensive ones like Flan-T5 XXL, along with multiple passage rerankers, to improve retrieval accuracy in zero-shot settings while maintaining competitive latency. The contributions of the paper include a novel domain adaptation approach, efficient query generation, and empirical evaluation on datasets like LoTTE and BEIR.


Pros:
The strengths of this paper lie in its innovative approach to domain adaptation, particularly the focus on the target domain without using source domain data.   Reviewers acknowledge that UDAPDR improves out-of-distribution generation of neural retrievers without adding latencies at test time.
 This adaptation improvement is consistent across multiple datasets, demonstrating the method's reliability. The approach is also commended for its computational efficiency compared to previous adaptation methods that use synthetic queries. The paper's contributions, including the pipeline's consideration of inference latency and the comprehensive evaluation and ablation experiments, are highlighted.


Cons:
The computational cost of Flan-T5 XXL is also considered non-negligible. Additionally, questions are raised regarding the novelty of the proposed approach, as certain aspects, such as prompts, have been seen in prior works. The paper is compared to a prior method called "promptagator," and it's suggested that the paper should better demonstrate the effectiveness and cost-efficiency of UDAPDR compared to this method. There is also a suggestion to explore the performance of training a first-stage retrieval model using the generated synthetic set. Lastly, the long-term viability of relying on large language models like GPT-3 for the method is questioned, given the potential for hardware advancements and improved efficiency through other techniques like parameter-efficient finetuning and distillation.
A concern raised by the reviewers  is the lack of strong support for the claim that "only 1000s of synthetic queries". 

In summary, the paper presents a promising approach to unsupervised domain adaptation in neural information retrieval models, but concerns have been raised regarding the novelty of the approach, the support for certain claims, and its long-term sustainability with current hardware trends. Addressing these concerns and providing a clearer comparison with existing methods could strengthen the paper's contribution to the field.
Decision: Accept-Main