Promptagator: Few-shot Dense Retrieval From 8 Examples
Authors: Zhuyun Dai
Vincent Y Zhao
Ji Ma
Yi Luan
Jianmo Ni
Jing Lu
Anton Bakalov
Kelvin Guu
Keith Hall
Ming-Wei Chang
Venue: ICLR 2023 poster (2023)
Keywords: large language model
few-shot prompting
information retrieval

Abstract: Much recent research on information retrieval has focused on how to transfer from one task (typically with abundant supervised data) to various other retrieval tasks where supervision is limited, with the implicit assumption that it is possible to generalize from one task to all the rest. However, this overlooks the fact that there are many diverse and unique retrieval problems, each targeting different search intents, queries, and search domains. In this paper, we suggest to work on Few-shot Dense Retrieval, a setting where each task comes with a short description and a few examples. To address this, we introduce Prompt-based Query Generation forRetrieval (Promptagator): for each task, we feed the few-shot examples to a large language model (LLM) and prompt it to behave as a task-specific query generator. Using this, we can synthetically generate a large number of relevant queries for any document, yielding abundant data for training task-specific retrievers --- with no reliance on traditional resources such as Natural Questions (Kwiatkowskiet al., 2019) or MS MARCO (Nguyen et al., 2016). Surprisingly, Promptagator with only 8 annotated examples enables efficient dual encoder retrievers to outperform computationally more expensive models trained on MS MARCO such as ColBERT v2 (Santhanam et al., 2022) by more than 1.2 points nDCG@10 on average on 11 retrieval sets. Further training standard-size re-rankers using the same generated data yields another 5.0 points nDCG@10 improvement. Our studies show that synthetic query generation can be far more effective than previously observed, especially when a small amount of task-specific knowledge is given.

Published as a conference paper at ICLR 2023
PROMPTAGATOR FEW-SHOT DENSE RETRIEVAL FROM 8 EXAMPLES
Zhuyun Dai't, Vincent Y. Zhao" Ma" Luan"" Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall and Ming-Wo Wei Chang"
Google Research
(zhuyundai, vzhao, maji, luanyi, mingweichang Boennaannggggggggggo @google. com "equal contributions 'corresponding authors
ABSTRACT
Much recent research on information retrieval has focused on how to transfer from one task (typically with abundant supervised data) to various other retrieval tasks where supervision is limited, with the implicit assumption that it is possible to generalize from one task all the rest. However, this overlooks the fact that there are many diverse and unique retrieval problems, each targeting different search intents, queries, and search domains. this paper, we suggest to work on Few-shot Dense Retrieval, setting where each task comes with short description and few examples. To address this, we introduce Prompt-basek Query Generation for Re- trieval (PROMPTAGATOR ); for each task, we feed the few-shot examples to a large language model (LLM) and prompt it to behave as task-specific query generator. Using this, we can synthetically generate a large number of relevant queries for any document, yielding abundant data for training task-specitf retrievers with no reliance on traditional resources such as Natural Questions (Kwiatkowski et al.) 2019) MS MARCO (Nguyen al., 2016). Surprisingly, PROMPTAGATOR using only 8 annotated examples enables efficient dual encoder retrievers to outperform computationally more expensive models trained on MS MARCO such ColBERT v2 (Santhanam et al. 2022) by more than 1.2 points nDCG@ 10 on average on 11 retrieval sets. Further training standard-sin size rerankers using the same generated data yields another 5.0 points nDCG@ @ 10 improvement. Our studies show that synthetic query generation can far more effective than previously observed, especially when small amount of task-specific knowledge is given.
INTRODUCTION
Significant progress has been made on neural retrieval models such dual encoders, which can search over a large collection of documents containing millions to billions of passages (Yih et al.
2011 Lee Lee al., 2019 Karpukhin et al. 2020). However, Thakur al. (02T) recently proposed the BEIR heterogeneous retrieval benchmark, and showed that it is still difficult for neural retrievers to perform well on wide variety of retrieval tasks that lack dedicated training data. To address this problem, many previous approaches focus on transferring knowledge from high-resource question answering (QA) datasets such as MS MARCO (Nguyen al., 2016), and propose architectures that possess good inductive biases, such as models that allow fine- one-grained token-level interaction (e.g.. ColBERT (Khattab & Zaharial 2020 Santhanam al., 2022) and SPLADE (Formal et al.) 202 which often come with higher inference cost. Data augmentation via synthetic query generation has previously been explored (Ma et al., 2021: [Shakeri et al., 2020,, but these question generators are learned from high-resource QA datasets, and often cannot generalize well to new retrieval tasks.
We argue that it is hard to anticipate models based on one or two QA datasets to perform well across all retrieval tasks. First, different retrieval tasks have very different search intents: in other words, different definitions of "relevance". For example, consider Figure [)): both Dbpedia- Entity (Hasibi et al.) 2017) and FEVER (Thome al. 2018) are tasks to retrieve documents from Wikipedia. However, Dbpedia- Entity is a task to retrieve entities that are mentioned in the query, while FEVER is task to find evidence that either supports refutes given statement. Which

---

Published as a conference paper at ICLR 2023
MS MARCO (Web Starch)
1) Unsupervieed
Supertaed (M4 MARDO)
FRVER (Cllinn Clain Veriication)
No Retrieval Supervision
Retrieval Tazs
Prunpaaeetine
Promplageis"
Dinerss wtad
2) Supervised (MS MARCO or NQ)
Pronplagettive
52
Dbgedia Entity Search
(100K-500K) examples
Many
Retrieval
Argentine
Aryntna CComneer Argument Renterall
3) Few shot
45.5
Retneval
Tank
loyal adioe
Rotrivval
Vidi 1t. Ghatmm Retieval BooMelion
Tank
Rezioval Porformance on 11 BER Datasets
Figure I: Few-shot retrieval with PROMPTAGATOR. Left (a): Retrieval tasks from BEIR differ in query distribution, retrieval corpus, and search intents. Middle (b); Most prior work uses supervised setting (2) which trains model on a large QA retrieval datasets and transfer to other retrieval tasks. Right (c): Few-shot PROMPTAGATOR performance. Average nDCG@ 10 on 11 datasets from BEIR from our PROMPTAGATOR models and previously MS MARCO-supervised models (SPLADE v2).
document is relevant to the query can be very different from one task to another task even if they share the same domain. Moreover, different tasks have distinct distributions of queries even when their search intents are similar. For example, queries in HotpotQA (Yang et al. 2018) are long compositional questions, while queries FiQA (Maia et al. 2018) are short financial questions.
Motivated by these observations, advocate to work on the setting of Few-shot Retrieval for diverse retrieval tasks (2)). where each task comes with a short description and a few annotated examples to clearly illustrate the search intent. To address this challenge, we propose Prompt-based Query Generation for Retrieval (PROMPTAGATOR) ( for each new retrieval task, we feed the few-shot examples to large language model (LLM) such as FLAN (Wei et al. 2002a)) and prompt it to perform doc-to-query generation. Importantly, the few-shot examples ensure that we capture the specific search intent of that task. Using this query generator, we can synthetically generate large number of relevant queries for any document, yielding abundant data for training any retriever, including highly efficient dual encoder models.
We find that our few-shot LLM query generator can produce good queries without any fine-tuning (.3.I) In fact, as shown in Figure [b), our synthetically generated data strong enough to completely forego using annotated query- documeett pairs from traditional high-resource datasets such as Natural Questions (Kwiatkowski et al.) 2019) or MS MARCO (Nguyen et al., 2016).
While PROMPTAGATOR is not the first application of LLMs for retrieval, prior work did not explore task-specific few-shot adaptation, and often came with high inference cost. Neelakantan et al. (2022) proposes to use GPT-3 (Brown et al. 2020) in dual encoders. However, their embedding dimension is 12k, which makes the search index footprint and inference cost prohibitively high for many applications. (Sachan et al. (2022) and Bonifacio et al. (2022) prompt LLMs for question generation, but did not explore the idea of using task-specific few-shot prompts for rapid task adaptation ? They also focus primarily on models that rerank top retrievals from existing retriever, rather than directly adapting the underlying retriever which must efficiently search over millions or billions of documents.
To summarize, the contributions of the paper are as follows:
We highlight previously overlooked differences across retrieval tasks (e.g.. search intent and query distribution), and propose a Few-Shot Retrieval evaluation for the BEIR dataset.
We propose PROMPTAGATOR, a simple recipe for few-shot retrieval by prompting an LLM to generate synthetic task-specific training data. For the first time, we can train fully neural retrievers and rerankers solely based few supervised examples.
Our results show that, surprisingly, PROMPTAGATOR with two-to-eight examples produces significantly better retrievers than recent models trained on MS MARCO or NQ that have over 500k human annotated examples (Figure I()) and utilize more expensive architectures: PROMPTAGATOR outperforms ColBERT v2 and SPLADE v2 on 1I retrieval tasks we tested, while reranking boosts results by another points standard retrieval evaluation metric.
FLAN is a I.I that not trained on any document retrieval or document-to-query generation tasks. InPars (Bonifacio et al. (222)) used the same few-shot prompt constructed from MS MARCO to generate reranker data for multiple tasks, so no task-specific prompt is used.
2

---

Published as a conference paper at ICLR 2023
2 FEW-SHOT RETRIEVAL TASK
In this section, we first introduce the definition a few-shot retrieval task and discuss the differences among tasks. We then propose Few-Shot Retrieval setting for the BEIR benchmark.
2.1 RETRIEVAL TASK
Given a large corpus, a retrieval model is responsible for finding the documents that are most relevant provided query q according a pre-defined notion of relevance. Formally, retrieval task is:
T = (D, 2.T).
where = (d1, de,....dd,, d is a large corpus of documents for retrieval, Q is a query distribution, and I is the underlying search intent for the task. Depending on the task, DD can be any document collection, such as the web or Wikipedia. Q also varies across tasks, e.g., short keyword search queries, questions, arguments, etc. If I(y, d)) 1, it means the search intent of q has been satisfied by the document d. For example, in question answering task [o d) if d answers For the same (g. d) pair, relevance may be either 1 or 0 depending on the search intent. For example, some argument retrieval tasks only seek to retrieve supporting arguments, while others aim to retrieve counterarguments.
In this work, we assume a target retrieval corpus Dy is given, but the amount of annotated query- document pairs for the new task is limited. Most prior research efforts focused on adapting retrievers to new corpus Dy, but didn't 't fully account for divergence in queries QT or intents I7. Next, we explore how a search intent can be expressed with a short description and very few examples.
2.2 FEW-SHOT BEIR SETTING
Intuitively, a person can understand retrieval task by reading short prompt and going over few examples. In this work, we ask if a few (8 or fewer) examples are sufficient to learn a task-specific retriever. To facilitate our study and future research on few-shot retrieval, define new few-shot retrieval evaluation built upon the BEIR heterogeneous retrieval benchmark (Thakur et al. 2021).
BEIR has 18 information retrieval datasets across 9 domains, including Bio-Medical, Finance, News, Twitter, Wikipedia, StackExchange, Quora, Scientific, and Misc These datasets also cover a diverse range of search intents: QA retrieval wwwesttion-t-documeeee duplicate question discovery (question- to-question), fact checking
etc. Following Santhaanm et al. (2022) and Formal al.]|(220))) (2021), we narrow our focus to the publicly-availan datasets in BEIR. The original BEIR evaluation used a zero-shot setup, where no queries or relevant query-document pairs from the evaluation datasets can be used for training.
We relax BEIR to the few-shot setting by randomly taking few (2 to 8) in-domain relevant query- document examples as task-specific supervision - in realistic applications, this number of examples is almost always possible to obtain. The examples available. For BEIR tasks which only have a test set, we use samples from the test data. To be models, we always mark these test-set examples
sampled from the development set when it
fair when evaluating documents from our retrieved results when computing metrics, even they are correctly retrieved (the worst possible outcome). The prompts and few-shot examples will be released to the public.
'failed':
remove the
PROMPTAGATOR
The key idea of PROMPTAGATOR is to transform a few examples into many more examples by prompting an L.MM to generate more data, instead of using them to train a retriever directly.
PROMPTAGATOR consists of three components: prompt-based query generation, consistency filtering. and retriever training. During prompt-based query generation, a task-specific prompt will be combined with large language model to produce relevant queries for all documents in Dr. Then, filtering step cleans the generated data based on round-trip consistency. Surprisingly, we found that a retriever trained only our synthetic data be used to filter the synthetic data. Finally, retriever (in this paper, dual encoders) and cross attention reranker are trained based on the filtered data. Figure 5 in Appendix shows the overall procedure.
3

---

Published as a conference paper at ICLR 2023
3.1 PROMPT-BASER QUERY GENERATION
In this first step, we feed our task-specific few-shot examples into a large language model (LLM) and prompt it to perform document-to-queryy generation. More precisely, let {(gs d be the k few-shot examples, where each example is a query (q: ~ Qr) and a document relevant to that query (d. E Dr)) according to the target task T (g. di) 1).
Following FLAN (Wei et al. 2022a), we instruction the LLM with the following string prefix:
Cdoc(d,) Cquery(91) .... Cduc (dk) Equery (g:) ) cdoc(d),
where is a separator token, (do and Equery (g) are task-specific document and query descriptions respectively, and d is new document presented at inference time. Using the ArguAna task as an example, we set edoe(t) ( = "Argument: ; {d}" and Equery = "Counter Argument: {()}" to inform
the LLM to generate counterarguments Fl The LLM is expected to generate Equery (4). We consider
it generation failure if the query description does not precede the actual query; otherwise, we accept g and form a synthetic relevant example (q. d).
Running the prompt on all documents from Dr, we obtain large set of synthetic (q. d) examples, amplifying the information from a few examples into a large synthetic dataset whose query distribution is similar to the true task distribution 2r and whose query-document pairs convey the true search intent Tr. This form of few-shot data extrapolation is similar to Lee et al. (2021).
We FLAN (Wii et al., 2022a) our LLM, and refer to our query generator as PFLAN (qld). FLAN is trained on a collection of tasks described via instructions and was shown to have good zero/few- performance on unseen tasks. We use the 137B parameter checkpoint. During prompt engineering, we use at most 8 examples, and reduce the number if they exceed the input length limit of FLAN. We also manually truncate individual queries and documents in the examples if they are too long. We randomly sample up to I million documents from each corpus and generate 8 questions per document using sampling basee decoding with temperature 0.7.
3.2 ROUND-TRIP FILTERING GENERATED DATA
We employ round-trip filtering Alberti et al. 2019 Lewis et al. 2021) to improve the quality of our synthetic data. The main intuition is that for any synthetic query generated from passage d, a good ğ should also retrieve its original passage d. In other words, the original d should have high probability under some retriever, p( (the reverse direction of query generation). If not, then we filter out q.
Round-trip trip filtering has been very effective for synthetic question generation QA tasks. However, these techniques typically rely on a question- guestion-answering model for the reverse direction filter. Since not all retrieval tasks resemble question-answering. this will not suffice in our setting.
Instead, we train an initial retriever from the unfiltered synthetic data, and then use it to filter the synthetic data. This works surprisingly well over the different search intents observed in BEIR. More precisely, given a synthetic query-documer pair (ğ. d), we use the initial retriever to predict the most relevant passages for q. We keep only when d occurs among the Top-K passages returned by the retriever. We show this filter substantially reduces the number of synthetic queries and significantly improves retrieval performance. In Appendix [F], we provide more insight into why this can work by viewing our synthetic queries as latent variables.
3.3 FEW-SHOT PROMPTAGATOR RETRIEVER
Our synthetically generated data allows training task-specific neutral retrievers for tasks where in- domain fine-tuning is challenging due to data scarcity. In this work, we use standard dual encoder retrieval architecture and we propose a simple pretrain-fine finnetnne recipe.
We pretrain retriever on C4 with the independent cropping task from Contriever (Izacard et al. 2022ah, where we treat two random crops from the same document as an artificial positive (query, document) pair and train with cross-entropy loss over in-batch random negatives. Next, we fine-tune the dual encoder on (q, d) pairs from our prompt-bases query generation, again with in-batch random negatives. After training for set number of epochs, we apply round-trip filtering on our synthetic
"The full set of descriptions used in our prompts can he found in Table 5 in the Appendix.
4

---

Published as a conference paper at ICLR 2023
Training Rocipe
Architecture
QGen
Supervision
Rerrieval
Cross-rttn Distillatinn
Retriever
Tokes-level Retrieval
Reranking Doc.
Serving Model Size
Model
Contriever
110M
GIR-XXL
Splade v2
MARCO (500K)
self
110M
ColBERT v2
MS MARCO (500K)
self
110M
GenQ
MS MARCO (500K)
self
110M
TS (MS MARCO
GPI.
MARCO (500K)
T5 MARCO)
MonoTS
(500K)
InPars
Few from MARCO)
BM25
1000
3B
GPT3
UPR
NA
Contrieerr
1000
110M+3B
FLAN TO
PROMPTAGNTOR
Few
110M
PROXPTAGATOR++
PROMPINGATOR
200
110M+125M
Table I: Comparison of different retrieval frameworks. Our serving models are just 110M-size dual encoder PROMPTAGATOR and a 125M-size reranker PROMPTAGATOR++, as good quality generated data allows simple models/pipeline to achieve strong performance InPars's 's few-shot examples are from MS MARCO and is task-independent. See text for more details for UPR's QGen mode 3
data as described in $32 using this initial dual encoder, and then continue to fine-tune the dual encoder on the filtered data.
We also propose PROMPTAGATOR++ reranker trained on the same synthetic data generated from our prompt-based QGen, which refines the retrieved candidates using slower but more accurate cross-attention model. We train the reranker using cross-entropy loss with 31 sampled negatives from top 200 passages retrieved by the PROMPTAGATOR retriever, which approximates the inference time distribution (reranking top 200 from the retriever).
Zero-shot PROMPTAGATOR Our prompt-based query generation can run in zero-shot manner, where we universally apply the following prompt irrespective of the target task: (d) Read the passage and generate query.'s Here {d) denotes the document text. Training retrievers and rerankers on this data leads to zero-shot PROMPTAGATOR and zero-shot PROMPTAGATOR+* This recipe serves baseline to show the benefits of adapting the few-shot prompt to the target task.
3.4 COMPARISON WITH PRIOR METHODS
Table compares the setting of PROMPTAGATOR some recently proposed approaches. Several dimensions of our recipe are simpler: our dual encoder does not employ hard negative mining, distillation from a cross-attention teacher or token-level refrieval. Also, our 125M parameter reranker is smaller than most other rerankers. We aim to show that even simpler and smaller architectures can achieve excellent results if trained with synthetic data that has been few-shot adapted ($4.3).
Compared to InPars (Bonifacio et al. 2022) and UPR (Sachan et al.) 2022). our approach employs task-specific few-shot adaption, while In Pars and UPR's s prompts are task independent and thus bear the same limitation of previous query generation approach (Ma al.] [021 Wang et al., 2022).
Another key difference is that prior works focused on reranking, while we enable few-shot learning for both reranking and retrieval.
4 EXPERIMENTS
We evaluate PROMPTAGATOR on the BEIR benchmark. We then dive deeper into the results through ablation studies and qualitative analysis.
4.1
IMPLEMENTATION
The original FLAN training set overlapped with 2 datasets in BEIR: NQ and Quora Most existing systems use MS MARCO for fully supervised learning and therefore do not report few zero-
'Serving model size is shown here. FLAN ( 1377) is used for query generation, not for serving. *UPR uses TO query generation directly for reranking rather than synthetic data augmentation.
'Regarding NQ, FLAN is only trained on the question-to--nswerr task and never observes the question-passage supervision needed for retrieval training. We study the effects of NQ and Quora in FLAN in $4.3
5

---

Published as a conference paper at ICLR 2023
arg
touché
covid
nfc
hotpot
dbp
climate
Tever
scifact
scidocs
fiqa
AVG.
Retriever
Unsupervised
BM25
31.5
60.3
21.3
75.3
66.5
Contriever
37.9
19.3
27.4
31.7
48.1
29.2
15.5
68.2
64.9
14.9
24.5
34.7
Supervised by MS MARCO
GTR-XXL
$4.0
26.7
74.0
66.2
SPLADE v2
47.9
27.2
71.0
33,4
68.4
43.5
23.5
78.6
69.3
15.8
33.6
46.6
ColBERT v2
46.3
26.3
73.8
33.8
66.7
44.6
17.6
78.5
69.3
15.4
35.6
46.2
GenQ
53.4
17.5
GPL.
55.7
25.5
70.0
34,5
58.2
38.4
23.5
75.9
67.4
16.9
34.4
45.5
PROMPTAGATOR (110M)
Zero-shot
53.8
26.6
72.7
33.4
60.4
36.4
21.4
76,2
62.3
16.3
40.4
45.5
Few-shot
59.4
34.5
75.6
33.4
61.4
16.8 (24.0")
77.0
65.0
18.4
47.8
Retriever Rerunker
Unsupervised
(PB)
50.3
21.3
60.4
33.3
72.2
33.8
9.5
57.3
69.6
17.3
45.0
42.7
(3B)
Supervised by MS MARCO
mo (220M)
13.2
27.7
8
35.7
69.5
41.9
24.5
80.2
73.6
16.5
41.4
45.6
m5 (3B)
19.7
PROMPTAGATOR++
(110M
125M)
Zero-shot
52.1
27.8
76.0
36.0
71.2
41.3
22.6
83.8
73.2
19.1
45.9
49.9
Few-shot
63.0
38.1
76.2
37.0
23.6
43.4
20.3
(24.1")
86.6
73.1
20.1
49.4
52.8
Table 2: Main Results. nDCG@ on BEIR. Retriever Comparisons (Upper Half): Among the various kind of retrievers, both zero-shot and few-shot PROMPTAGATOR produce strong results. Retriever+Reranke
Comparisons (Lower Half): the scenario where speed is not a concern, reranker is often used. We train PROMPTAGATOR++ the generated data and get significant improvement. See text for more details for Climate-FEVER and Webis- Sobbis-Tuuch22221
shot results on MS MARCO. Therefore we exclude MS MARCO, NQ and Quora from our main evaluations. We report nDCG@ 10, the standard retrieval evaluation metric on BEIR.
For PROMPTAGATOR'S s query generation, we sample questions from FLAN with a temperature of 0.7. For round-trip filtering, we find that setting filtering threshold K to gives the best results on MS MARCO and thus use for all BEIR datasets, We implement PROMPTAGATOR'S dual encoders following GTR (Ni et al. 2021).To To ensure efficiency, we use the T5-base encoder architecture consisting of 10M parameters. For PROMPTAGATOR++ reranking models, we initialize from T5- base version 1.1 encoder checkpoint which has 125M parameters. More details of the reranker implementation can be found in Appendix C At inference time, we rerank the top 200 candidates retrieved from the PROMPTAGATOR dual encoder.
We mostly follow the hyperparameters used in Ni et al. (2021). By default, we use batch size 6k; however, some of the corpora in BEIR contain only few thousand documents, making multiple relevant documents appear in the same batch, which interacts negatively with our in-batch softmax loss. To address this issue, we split all datasets into three groups based on corpus size: small (<50k), medium (50k-500k) and large (>500k). For dual encoder training, we use 128 batch size for small datasets and 6k for others. We finetune for 5k steps for large datasets and Ik for others. For ranking models, we use batch size 64 for all datasets and finetune large datasets for 20k steps, 5k for others.
4.2 MAIN RESULTS
Table shows the experimental results. We first notice that zero-shot PROMPTAGATOR already serves strong baseline. comparing favorably other retrieval baselines trained on O(100)) examples from MS MARCO. Nonetheless, few-shot PROMPTAGATOR markedly improves upon zero- shot PROMPTAGATOR, increasing average nDCG@ 10 by over 2 points, which highlights the impact of few-shot learning. Few-shot PROMPTAGATOR despite having simple training procedure and model architecture, outperforms strong baselines such GenQ (Takur al. 2021 and GPL (Wang et al. 2022) which also use query generation to augment training data, as well as ColBERT v2 (Santhanam
CCimate-FEVER's 's relevant docymett pairs in BEIR are not well- 4.3 so we tried FEVER prompt Climate-FEVI FEVER. The results are reported in (). but they are not used for computing the average Vobis-loouh has updated version; we used the original version.

---

Published as a conference paper at ICLR 2023
7/3
35
50
N=50,000
42.8
nDCG@@0
4.0
3.0
nDCG@10
30
nDCG@11
2.0
25
25
2.0
fever climate
dop
hotpot touchi
covid
solfecs
ntc
sclacct
Supervised
8-shot Promptagator
50,000 100,000
GenQ NQ QGen, PGT Zero-shot
Impact of Round Trip Filtering
Human-annotate Examples
Figure 2: Left (a). Delta in nDCG@ 10 between few-shot PROMPTAGATON with and without filtering.
Middle (b): Comparing the effect of the generated data versus the number of supervised data MS MARCO. PROMPTAGATOR with 8 examples can catch up with 50k labeled examples, when simple dual encoders used. Right (c): Ablation on query generation model. GenQ is prior system from Thakur et al. (2021), while NQ-QGen is our in-house NQ-trained T5 query generation model. Other than the generated data, NQ-QGen and PROMPTAGATOR uses the same hyper parameters.
et al. 2022 and SPLADE v2 (Formal et al. and distillation recipes.
2021 which rely on token level interaction architectures
Our reranker PROMPTAGATOR++ boosts performance by another 5 points on nDCG@ 10. It sig-
nificantly outperforms UPR (Sachan et al. 2022) whose reranker uses TO (Sanh et al., 2022) an instruction tuned LLM similar to FLAN. It also outperforms monoT5-33 (Nogueira et al. (020)).
which achieved previous state-of-the-arr reranking performance on BEIR in a recent study (Rosa) et al. (022). Note that most of these reranker approaches use large 3B parameter model for better generalization, while + uses a standard 125M reranker.
Comparing few-shot PROMPTAGATOR to baselines, the biggest improvement is Toucs-0ou (touché), followed by ArguAna (arg). WobissToouch22220ss goal is to retrieve documents for a controversial topic, e.g., "should felons who have completed their sentence be allowed to vote?". ArguAna's 's goal is to find the counter-argun counter-arguments arguments that oppose the input argument, and the input arguments often several-sentence long. Both tasks extremely different from traditional QA retrieval data that other models use, which are dominated by factoid questions. On the other hand, few-shot PROMPTAGATOR successfully adapt this task with few examples.
4.3 ANALYSIS
Impact of round-trip filtering. In Figure 2(a), we show quality differences between few-shot PROMPTAGATOR with and without filtering. Filtering improves performance on 8 out of 11 datasets and leads to 2.5 points improvement strategy. Nonetheless, filtering hurts model quality on NFCorpus and SciFact. These are the smallest datasets terms of generated queries and may indicate overfitting of our retrievers.
average, demonstrating the effectiveness of our filtering
We find that the majority of filtered examples are either queries that are too generic that match many documents, or queries that contain additional terms that are irrelevant to the document. Examples are in Fig. i in the Appendix. There are also cases where high quality data was incorrectly removed. We suspect that designing dynamic filtering thresholds would help, and leave to future exploration.
Can generated queries replace human annotated queries? In Figure bb), we evaluate 8-shot PROMPTAGATOR on MS MARCO, comparing against dual encoders trained on MS MARCO's supervised data. Note that did not add other components,
make the comparison simple. We chose MS MARCO as there are enough labeled data for this task and neither FLAN nor our models trained on MS MARCO examples. The results show that eight examples plus LLM can replace a significant portion of supervised examples.
How does PROMPTAGATOR compare to other query generation approaches? Figure 2(c) com- pares zero-shot PROMPTAGATOR to two other query generation approaches: GenQ is prior system from Thakur et al. 2021) using MS MARCO trained T5 query generation model, and NQ-QGen
7

---

Published as a conference paper at ICLR 2023
arg
touché
covid
nfc
hotpot
dbp
climate
fever
scifact
scidocs
fiqa
AVG.
FLAN original
59.4
34.5
75.6
33.4
61.4
38.0
(24.0t)
77.0
65.0
46.2
48.5
FLAN wi NQ and Quera
58.8
33.3
70.2
33.7
61.7
34.4
(23.5f)
76.2
63.8
18.3
43.0
47.0
Table 3: Impact of different FLAN versions.
This study uses Fever prompt for Climate Fever ($4.3)
OTHER the 15.2
OTHERS
OTHERS
10.8
while
the
23.2
what the
30.4
31
25.0
0.7
9.1
there 8 25 11 0.8
there prog eodive 1.7
when
8.3
new
25.0
6.5
the
25.0
(a) Gold queries
(b) Few-shot
(c) NQ-QGen
(d) Prompts (4 examples)
Top first word distribution on queries generated from different models in the ArguAna dataset. Left (a)()(()): Compare gold queries (a) and generated queries (b)(c). Queries generated by few-shot models has closer distribution to the gold queries, while the NQ-QGen queries are mostly questions. Right (d): The few shot FLAN can generate diverse queries even though there are only examples in the prompt. Statistics of more datasets are available in the Appendix (Figure 4
Figure 3:
our in-house T5 QGen model finetuned on NQ. The figure shows the advantages of zero-shot PROMPTAGATOR, outperforming both baselines by large margins. Importantly, NQ-QGen uses the same filtering, dual-encoder training, batch sizes and training steps as PROMPTAGATOR, providing a faircomparison of query generators. This indicates that the main contributing factor to PROMPTAGATOR better queries from prompting an LLM, not the specific training recipe or hyperparameters.
Does few-shot always improve almost always outperforms zero-shot PROMPTAGATOR except for Clmmtt-FEVER The original Climate-FEVER dataset
zero-shot? As shown in Table few-shot PROMPTAGATOR
of three tags to annotate query-doeument pair, namely "supports", "refutes", or "not enough info". However, BEIR treats all these three annotations as relevant, which problematic. Using query-document pairs annotated "not enough info" in our prompt could be detrimental to generation quality. Therefore, we tried switching to FEVER's few-shot prompt, as the two datasets share same corpus and similar search intents. With the better annotated examples, few-shot PROMPTAGATOR indeed surpass zero-shot. This result provides some evidence that low quality few-shot examples negatively affect PROMPTAGATOR.
Impact of FLAN Versions FLAN was trained on a collection of datasets which have some overlap with BEIR; specifically, it includes Natural Questions (NQ) and Quora. was not trained on query- document pairs from NQ Quora; however, in order to determine whether the inclusion of this data biased the results on the final retrieval evaluation, we designed an additional ablation experiment. Following the original FLAN recipe (Wei et al.) (022a),, we trained an additional LLM excluding both the NQ and Quora datasets. Table 4 shows the results. While the accuracy drops slightly, the overall performance still outperform prior retrievers.
Qualitative Analysis In order to understand the advantages of few-shot PROMPTAGATOR, we analyze the distribution of queries generated by different query generation methods for ArguAna in Figure 3 To easily see the differences, for each distribution we plot the histogram of each query's ' first word. Note that the distribution of few-shot PROMPTAGATOR (Fig. 3b) is much closer the real distribution (Fig. Ba) while the NQ-QGen (Fig. 3c) mostly generated questions even when the queries in this task are generally arguments, not questions. More examples are showcased in Table 6 in the Appendix.
Use few-shot examples directly To study the effect of using the few-shot examples directly, conduct a study of fine-tuning task-specific models using the GTR dual encoder (1 10M) (Ni et al. 202I) with the few-shot examples in appendix A As expected, it does not provide a large amount of impact for GTR base, where the average nDCG decreases from 40.4 to 38.7.
5 RELATED WORK
Neural Retrieval and Reranking Models The majority of neural retrievers today employ a dual encoder architecture that encodes queries and documents independently into dense vectors and
8

---

Published as a conference paper at ICLR 2023
retrieves documents using maximum inner product search (MIPS). Recent research has primarily focused on the following aspects: developing better pre-training tasks (Lee et al.] 2019 Chang et al.)
2020; Izacard et al. 2022a Gao & Callan 2021; Oguz et al. 2022). improving contrastive negatives
(Qu et al. 2021 Xiong et al.
2021: Lu et al. 2021), and improving generalization across different
domains Thakur al. 2021 Ren al. 2022).
Although dual encoders enable fast retrieval, their expressivity is limited due to the fact that their score is just dot-product between a query vector and document vector. A common solution is to use a cross-attention model to rerank retrieved candidates (Nogueira & Cho 2019 Nogueira et al. 2020p, as cross- attention rerankers can explicitly model the interaction between query and document tokens. Distilling coss-attenttion models into dual encoders has been effective in closing the gap
between the two (Hofstätter et al. 2020 Ren et al. 2021 Reddi et al. 2021 Zhang et al. 2022).
Neural Retrieval with Fine-Grained Interactions An alternative for bridging dense retrievers and cross-attention models to allow some amount of fine-grained query-document interactions in the retriever. Huumauu al. (2020) and Luan al. (202I) represent and retrieve queries and documents with multiple vectors instead of a single vector. ColBERT (Khattab & Zaharia 2020), COIL (Gao) et al. a22 and SPLADE (Formal et al.) 2021) take this further by using token-level interactions between queries and documents. Because these models are not just modeling a dot product, MIPS algorithms cannot be used directly. Hence, these models usually have much higher inference/serving cost compared to dual encoders.
Prompt-based Query Generation The idea of using prompted LLMs for query generation has previously been proposed for improving retrieval reranking. UPR (Sachan et al. 2022) proposed
use prompted LLMs to rerank passages directly. InPars (Bonifacio al. (022) is probably the most closely related work to ours. They proposed to use few-shot prompting with GPT-3 to generate synthetic data for training a T5-based reranker. Though InPars was tested on multiple retrieval datasets, they used task-independent prompt constructed from MS MARCO and did not explore task-specific few-shot learning. They also focused exclusively on reranking, whereas we also address full-scale retrieval.
Few-shot Learning pre-trained LLMs have significantly advanced few-shot learning, thanks to prompting strategies such as in-context learning and instruction-pronen prompting (Brown et al., [2020] Weil ct al. 2022b). Some approaches fine-tune LLMs specifically for few-shot learning (Schick & Schüitze 2021a b Gao et al. [2021b] Logan IV et al. 2022 Izacard et al. 2022b) while others do not (Brown
ct al. 2020; Bonifacio et al. 2022). We use LLMs as few-shot data generators this form of data augmentation via few-shot example cxtrapolation is similar to Lee et al. (0021)
6 CONCLUSION AND DISCUSSIONS
In this paper, we present PROMPTAGATOR, a novel approach to few-shot retrieval. We showed that it is possible to create task-specific retrievers and rerankers with only a few annotated examples. The few-shot examples, amplified by prompt-base based LLM query generation, simplifies the complexity of training neural retrievers for new tasks and leads to promising performance gains. It hopefully inspires future research towards generalizable retrieval systems that can seamlessly and efficiently adapt to many tasks.
While we demonstrate that LLM-based query generation can be very effective, many questions remain. One of the key issue requiring further investigation is on the generated data efficiency. We have not yet explored exactly how many query-document pairs are needed for each task, or how to use these generated examples more efficiently. Another issue is the sensitivity of the final retriever's performance with respect to the prompt. Finally, we would like to draw a connection from PROMPTAGATON to distillation, as the final dual encoders indirectly "learn" from the LLM. Analyzing the headroom and understanding how we can better transfer knowledge from LLMs to retrievers would be a critical topic for the future.
As mentioned in |6 PROMPTAGATOR can be viewed distilling LLM to standard-sin sized dual encoders via prompt- query generation. While the distillation process is computationally expensive, significantly reduces cost for inference.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Kenton Lee, Tom Kwiatkowski, and Daniel Gillick for technical discussion and providing feedback on our manuscript. We thank Alex Salcianu for developing a bulk inference pipeline for large language models.
REFERENCES
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6168 6173, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 16553/v1/11-11620 URL
org/P 100//1991622 1620
Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. Inpars: Unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, pp. 2387-2392, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 450387323. doi: 0355:344777953333166
htt.httt////dd otap:i.org/0... 1eee..11533474555 B1E1499555331111 3531863
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL /tttps:://prooceeingss /tt.o///ooooo.///////////////ooo cc/ htc/paper/202/ 1tt1:1eee1110111111010
neur 967 Bb w2w.::1144444422
Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang. and Sanjiv Kumar. Pre-training tasks for embedding- -aaddd lcrge retrieval. In International Conference on Learning Representa- tions, 2020. httttt / http:::///eeeeeeeeeeeeee
net forum? id=rrgg-m4FrDr
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. JOURNAL OF THE ROYAL STATISTICAL SOCIETY, SERIES B, 39(1):1-338 1977.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2: Sparse lexical and expansion model for information retrieval. CoRR, abs/2109. 2021. URL
httt::://arxiv.rrggee
2109. 10086
Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 981-993, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1..1100...000 20.11866//11// 1t553//11222..mmlpppoo .emnlp-main.7.. httpstps: ac //waaaaaaaanoooooo.oooo.oo org/ 2021. www.p-main....m..mm.mmm..mmmmamm
Luyu Gao, Zhuyun Dai, and Jamie Callan. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3030-3042, Online, June 2021a. Association for Computational Linguisties. doi: 10. 2021.naacl- www.aaaaaa.nn---11 URL htt.lttppss//claatttoooooooooo/p hwtp://2221..aacmmaaaaa 241
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- ties and the I Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021b. Association for Computational Linguistics. doi: 10. N0.18653/v111 1/202/ wel-longg 295. URL /ollo///ao//////atttooooooooooo ort.:oolllo....//222202// acc.//2ooo.....on 295
10

---

Published as a conference paper at ICLR 2023
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. Dbpedia-entity v2: A test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference Retrieval, SIGIR 17, pp. 1265- 1268, New York, NY, USA, 2017. ISBN 9781 450350228. URL https://doi.. http://0....r1111111110000100110 308075511
Research and Development in Information
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with cross-architecture knowledge distillation. ArXiv, abs/2010.02666, 2020. URL /
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. BRLLtttpp://opeeeviiii wwtp://nneeeottttteeeeeeeemm. SkkggnnNN-
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022a. URL hattps://openreviess net/
[oormmidd JKN1pXi 7b0j
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models, 2022b. URL htt.tttttttttti.....ttt org. abs ab2//208..222// 03299
Vladimir Karpukhin. Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 10200emnlp-main main.550. URL //ac anthology wwt.oog//2000 emn womml-mmmaa p- in. 550
Omar Khattab and Matei Zaharia. COIBERT:
Efficient and effective passage search via contextualized late interaction over BERT. In Jimmy Huang. Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock. Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 39.48.9466998 ACM, 2020. doi: 10.1145/33922113440000 URL //doi. 10. 1ew..
https
3401075
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, March 2019, doi: 10. 10.1162/tacl_a0000 ogy. otg/y10rg/11111 9219--0066
URL /akttss:/aclaateee waaphhhooogy....
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 6086-6096, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1w.11165/11//9///1 1612. URL httpat/s//aa...tttooooooo///////
Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural data augmentation via example extrapolation. CoRR, abs/2102.0133 2021. URL httpp:://axxi..org//////
31C2.01335
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. PAQ: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguisties, 9:1098 1115, 2021, doi: 10.1162/tacl a 00415. URL https: http::////aaaaatttooooooorr////s tacl-1
65
Robert Logan IV, Ivana Balazevie, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Findings of the Association for Computational Linguistics: ACI. 2022, pp. 2824-2835, Dublin,

---

Published as a conference paper at ICLR 2023
Ireland, May 2022. Association for Computational Linguistics. doi: 10.18655//112022...mmm 222. URL htt.hhtsss//ccaaatooooooo// wwt.:orggg/000 www.iiggnnninn..222222
Jing Lu, Gustavo Hernandez. Abrego, Ji Ma, Jianmo Ni, and Yinfei Yang. Mtagi-stage training with improved negative contrast for neural passage retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6091-6103, Online and Punta Cana, Do- minican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/ ottp://w....00222222mmmmmmmmm0.. 492
2021 1e2lleenl---me main.492. URL
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, Dense, and Attentional Representations for Text Retrieval. Transactions of the Association for Computational Linguis- tics, 9:329-344 345, 04 2021. doi: 10.1162/tac_ a a0366. URL https:: / tacl 00369
Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. Zero-shot neural passage retrieval via domain-targeted synthetic question generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1075-1088, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.eacl-main.22 URL htth:st:: http:///aaaaaattttaaaeaaam wwwpttthoooooooooo22eeee
eacl-main. 92
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www 18 open challenge; financial opinion mining and question answering. https:
In Companion proceedings of the the web conference 2018, pp. 1941-1942, 2018. URL
//doi. 100
31 92301
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David P. Schnurr, Felipe Petroski Such, Kenny Sai-Kin Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training. ArXiv, abs/2201.10005. 2022. URL httttttss///axiiii
abs 2201 10005
Tri Nguyen, Mir Rosenberg. Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In @IPS 2016. UtLphttp:
cttp//s.uwwwweeee httptttppp//uurwss.... org
2016 wwppra.ppp..... F
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming- Wei Chang, and Yinfci Yang. Large dual encoders are generalizable retrievers. CoRR, abs/2112.07882 2021. URL /ttp:://aaxivvvee org. abs 2112. 07899
Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv, 2019, URL https
04085
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 708-718. Association for Computational Linguistics, 2020, doi: 10. /tt/:///////22000..onnnnnneemm// 18653/v1/2022 Fiindinngs-emnll.
URL /tt.ttts://oi.orr////
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mehdad. Domain-matched pre-training tasks for dense retrieval. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1524-1534, Seattle, United States, July 2022. Association for Computational URL https:://clatthhol
Linguistics. doi: 10. 10.11665/
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.
June 2021. doi: 10. 10.18653/v 1/2021 maacl-main.44 m66. URL https 7/aclanthir ogy. naagl -main. 466
12

---

Published as a conference paper at ICLR 2023
Sashank J. Reddi, Rama Kumar Pasumarthi, Aditya Krishna Menon, Ankit Singh Rawat, Felix X. Yu, Seungycon Kim, Andreas Veit, and Sanjiv Kumar. Rankdistil: Knowledge distillation for ranking. In AISTATS, pp. 2368-2376, 2021. URL http://procealings mlr .press/ html
v1
Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. PAIR: Leveraging passage-centris similarity relation for improving dense passage retrieval. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2173-2183, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 172201..inding--acl.11 URL https: a www.clannnoollloooooooooooooeeee wrg.logg..rgg2221111 findings acl
191
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qifei Wu, Yuchen Ding, Hua Wu, Haifeng Wang, and Ji-Rong Wen. A thorough examination on zero-shot dense retrieval, 2022. URL 2atbbs/2204..22
Guilherme Moraes Rosa, Luiz Henrique Bonifacio, Vitor Jeronymo, Hugo Abonizio, Marzieh Fadaee, Roberto de Alencar Lotufo, and Rodrigo Nogueira. No parameter left behind: How distillation and model size affect zero-shot retrieval. CoRR. abs/2206.02877, 2022. doi: 10.48550/arXiv22266. 02873. htttps:::://
org/ /00. 48550/ arvii 2206. 02873
Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. arXiv, 2022. URL http:://wwwww..../aaaa/aaaaaa//a http:/aab/222...00 07496
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. The Tenth International Conference on Learning Representutions, ICLR 2022, Virtual Event, April 25-29, 2022. Open 2022. URL ntttps://openrevien net/ www.orommid-vv/o9000
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 3715-3734, July 2022. URL www.ooggg222222 matpg//222.maac.... naac -aammaann. mm..n.22
https://acl
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269,, Online, April 2021a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 wtt.:02..eealllmnn eacl-main. URL http::://acaantthohoo
org. hwwp:/222...cmmma..........
Timo Schick and Hinrich Schütze. It's s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-2352, Online, June 2021 Association for Computational Linguistics. doi: 10.18653/v1/2027 185. URL https:tt////// htt.t::://acaanttoolo
1taa:/22.....mmmmallaaaaaaaaaa -naacl-main.
org 2021 seaac-mmain.3
Timo Schick and Hinrich Schütze. Few-shot text generation with natural language instructions.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pp. 390-402, Online and Punta Cana, Dominican Republic, November 2021c. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2 1/202/ comnl-mmin...
URL https
1/aclanthology
wwt.oogg/2222 hwwprhhooog..org/2211..... emnlp ceto-main.. 32
Siamak Shakeri, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. End-to-end synthetic data generation for domain adaptation of
13

---

Published as a conference paper at ICLR 2023
question answering systems. In Proceedings of the 2020 Conference Empirical Methods in Nat- ural Language Processing (EMNLP), pp. 5445-5460. Association for Computational Linguisties, 2020.
oz 2020. emnip-main. 439
Nandan Thakur, Nils Reimers, Andreas Rückllé Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021. URL https: http:://pppereeeeeee / net wwt/ www.:::dwwwwwwwwwdecccc jeJ.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang. Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin HoffmannJohnn Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language models for dialog applications. CoRR, http/2201.022333 2022. WRRLhttps: arriiv..vv....////aa//aa////avvv crq/
abs /bs//bb////// 2201
08239
James Thorne, Andreas Vlachos, Christos Christodoullppouls,, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N1B 1074. URL
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Seattle, United States, July 2022. Association for Computational Technologies, pp. 20266//////22...m./// /2w.//.222....2.2...m.. main. 168. URL ht https: http:::://aaaaaaoolooo/o/oo// org Linguistics. doi: 10. 18.11813/vvl// 2022. www.naaallla.cm mwin. 168
Jason Wei, Maarten Paul Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Mingbo Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022a. URL /RRl/attps////peeeeiee
net/ffoumm htt./ofrummm1mgg27GG..0.m. d=
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans- actions on Machine Learning Research, 2022b. URL htth:tts: /pwp/evveeewe nee forum? id-yzkSU5zdwD
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021. URL httpst
//openrevies iew
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-122606 2280, Brussels, Belgium, October-Novemble 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/DD 12259. URL hww.attpaa//cciannnoooo........ otg.:///tttoooooooooooo---- 1259
Wen-tau Yih, Kristina Toutanova, John C. Platt, and Christopher Meek. Learning discriminative pro- jections for text similarity measures. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pp. 247-256, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL URR.htttss//acclaaae
ogy ogw..oggg........... 0329
14

---

Published as a conference paper at ICLR 2023
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. Adversarial retriever-ranker for dense text retrieval, In International Conference on Learning Representations, 2022. URL :
15

---

Published as a conference paper at ICLR 2023
arg
touché
covid
Info
hotpot
dbp
climate
fever
sciffact
scidocs
fiqu
AVG.
GTR-base
51.1
20.5
53.9
(24.1%)
66.0
60.0
34.9
40.4
GTR buse with & exammlees
51,9
24.1
36.2
30.5
23.4
35.4
(25.4%)
68.2
60.1
14.8
35.3
38.7
Table 4: Directly using eight examples might not improve the results. To showcase this, we fine-tuned GTR-based model with the same few-shot examples used for tuning.
COMPUTE USAGE AND ENVIRONMENTAL IMPACT
We used the 137B FLAN, which is based on LaMDA (hoppilan et al. 2022 LaMDA was pre- trained on large corpus consisting of 1.56T words, costing 451 MWh energy and 25.2 iCO2e carbon footprint. In PROMPTAGATOR, generated 29.23M queries prompts 58.46M queries, for total
of 610M words.
A FINE-TUNING GTR WITH FEW-SHOT EXAMPLES
We study the effect of directly adding few-shot (no more than 8 examples) on top of 110M dual encoder, GTR-base Ni et al. (202T). To maximum the utilities of the 8 examples, we associated each positive example with 32 random negative examples from the target corpus construct batch, and ran 50 steps of fine-tuning for each task. As expected, the 8 examples did not provide large amount of impact for GTR base, where the average ndeg go from 40.4 38.7 after fine-tuning using few-shot examples.
B ANALYSIS ON PROMPTS
Table shows the list of prompt templates on different BEIR datasets. In order to further analysis the difference between zero-shot and few-shot prompts, compare the few-shot and zero-shot generated queries given the same paragraph, randomly sampled from three datasets in Table 6 We observe that in general, the few-shot generated queries closer to the original queries, while zero-shot queries are mostly questions. For example, in the ArguAna dataset, the few-shot queries are in general longer and more claim-like. contrary, the zero-shot queries are most short question-like queries. Interestingly, for the HotpotQA dataset, even though both few-shot and zero-shot queries are generating questions- queries, few-shot queries sometimes generate multi-hop questions, while zero-shot mostly generates single-hop questions. We further conduct first word distribution across different generation models for all datasets Figure
Dataset
Prompt
ArguAna
0 Argument: passage X 1 Counter argument:
query X
FiQA
0 passage x 1 query X
HotpotQA
Evidence: passage
Vexed question: query
DBPedia-Entity
entity: passage
query: query
NFCorpus
0 Article: passage x Query: query
Touché-2020
0 passage X 1 Depate: query X
TREC-Covid
0 passage x 1 Question: query X
SciFact
0 passage X 1 Finding: query X
SCIDOCS
0 passage 1 The passage about query
FEVER
passage 1 it that query X
Table 5: Prompt template for each dataset.
16

---

Published as a conference paper at ICLR 2023
Paragraph
Few-shot
Zero-shot
Analysis
Banning skin whiten- ing such basis also requires the acceptance of the racial over- tones. Some form of tan is popular in many societies of people of Eu- ropean ethnic origins- that is not a racial matter, is more based on economic social perceptions (that of holidays to warmer climates). Ascribing racial element to erything to do with skin tone best lazy analysis. Irrespective of issues of race and perceptions of ethnic origins, and its intersec- tion with beauty standards, some people will be given advantages in life due to their appearance. Ban- ning a certain form of cosmetic, even if can have some racial and ethnic undertones, won't change that.
ArguAna:
Query 1: Banning skin whiteming
Query I: What did they try ban?
In ArguAna dataset, the few-shot ex- amples are more statement-like and are longer than zero-shot.
such ba- sis also requires the accep- tance of the racial overtones. However, the racial over- tones simply a side ef- fect the product being used incorrectly in order to attain a lighter skin tone than they naturally born with. The product itself is not racist, but rather peo- ple who use it. For exam- ple, person of Asian de- scent may use skin whiten- ing cream attain lighter skin tone so that they may fit into a more white-dominant society. The cream itself is not racist, but rather the fact the person attain a lighter skin tone so that they may fit into a more white-dominant society.
Query 2: Is skin whitening racist?
Query 3: Banning skin whiten- ing on such basis also quires the acceptance the racial overtones. Some form of tan is popular in many so- cicties of people of European ethnic origins that is not racial matter, is more based on economic social perceptions (that of holidays to warmer cli- mates).
Query 4: What are some the racial overtones skin- whitening?
Query 5: Are tans sign of af- fluence?
used to
Touché-2020: Psychologically, dict-wise and academically. Housewives can be both educated and uneducated but the fact that they spend more time with their children means that they put in greater effort and since they are not preoccupied with their job(read: cellphone); they pay the close attention needed to keep their children out of trouble. Think of housewives as inspectors your home...
Query I: Housewives are better parents
Query 1: diet-wise and aca- demically.
The few-shot gener- ates argument-like queries that are more controversial, while zero-shot generate random statements that sometimes even have grammar problems.
Query Should house- wives be considered better parents?
Query 2: People want be relaxed all the time(At home AND at work) that is why all the research on work-life hal- ance.
Query 3: has shown that home- makers are more relaxed, less stressed, healthier, happier the list goes
The Yung See San Fong House, or Young's 's Home in the Heart of the Hills; is a house in Los Gatos in the U.S. state of California. It is combination of Asian decorative motifs and pagoda roof's together with West- em massing and layout, It was completed in 1917 for Ruth Com- fort Mitchell Young, writer, and her husband, Sanborn Young, gentleman farmer, conservationist and later California State Senator.
HotpotQA:
Query Which author was the husband of Ruth Com- fort Mitchell Young?
Query 1: Yung See San Fong House was completed in 1917 for Ruth Comfort Mitchell Young, writer, and her hus- band, Sanborn Young.
Few-shot examples sometimes will create multihop questions indicated in blue, which rarely hap- pens in zero-shot examples.
Query 2: Which is known as the Yount-Lee House Query 3: Who the archi- tect of the Yung See San Fong House?
Query How old the Yung See San Fong House?
Query 3: Yung See San Fong House was completed in 1917 for Ruth Comfort Mitchell Young, a writer, and her hus- band, Sanborn Young.
Query 4: The Yung See San Fong House, Young's Home in the Heart of the Hills; is a house in which U.S. state?
Table 6: Few-shot and zero-shot generated queries randomly sampled from ArguAna, FiQA and HotpotQA dataset.
17

---

Published as a conference paper at ICLR 2023
C DETAILED IMPLEMENTATION
Figure shows the overall process of PROMPTAGATOR++4 the details of which are in Section
Our cross attention reranker is a listwise model based on T5. Specifically, it takes a list of documents given a query as the input. We represent each query-document pair as "Query: {q) Document: (d)" " and feed it into the encoder of a T5 model. We then apply a projection layer on the output encodings of the first token and use the output the ranking score. We optimize the model using softmax cross entropy Ioss over over ranking list consisting of a positive (q, ddt) pair and 31 sampled negative (g, d' pairs. Unlike monoT5 (Nogueira et al. 2020), which is pointwise reranker that uses an mncoder-decodee model and is trained to generate relevance label, our model is a listwise reranker and is directly optimizes for ranking performance.
D QUERY GENERATION STATISTICS
In Table we analyze the length of the generated ques- tions by different query generation systems. Note that NQ-QGen always generates short queries due to the query generation models being fine-tuned on the NQ dataset, and all of the generated questions have similar length those questions of NQ. Interestingly, zero-shot PROMPTAGATOR already obtains more variance in terms of length compared to NQ-QGen. Finally, few-shot PROMPTAGATOR offers significantly more variance in terms of the length of generated queries.
Few-sbet
Zero-sbot
QGen
ArguAna
98.2
26.0
9.7
Touché- 2020
7.8
13.4
TREC-Covid
10.8
10.2
NFCospus
8.3
11.5
10.3
HotpoiQA
DRPedia Entity
8.2
13.8
8.8
Fever
10.7
Climate-Fever
12.9
10.7
8.8
SciFact
12.6
12.4
SCIDOCS
15.7
10.7
FIQA-2018
12.5
10.1
9.5
AVG.
17.8
13.5
E ROUND-TRIP FILTERING EXAMPLES
Table 7: Average query length.
Figure 6 shows examples of queries from few-shot PROMPTAGATOR that were removed by round-trip fil- tering.
ROUND-TRIP FILTERING AS LATENT VARIABLE MODELING
This section aims to share some insight into why our round-trip filtering method is effective, by viewing queries as latent variables that we estimate. First, consider a hypothetical graphical model which each query g a latent variable and the documents retrieved for that query are observed variables following some distribution, p(d]g, 0*), where 0* represents the parameters of hypothetical "optimal" retriever that always selects the "best" documents for any query. For synthetic data generation, make it goal to sample queries from the posterior, p(gld, 6*). which according to Bayes rule is:
p(dlg, 0 (p((0(") 0") Eg ('d 8" (0''
where p(g (q) is prior over queries. We will assume it is an "uninformative prior"that is uniform over g and later write it as just p(g). we knew 0" (the parameters of an optimal retriever), we could just directly compute the above expression. But in practice we do not, so we will estimate 0 using expectation maximization (EM) (Dempster al. 1977). which will turn out to mirror our round-trip filtering algorithm. EM and other latent variable learning methods have long been used to impute missing data with great success.
In EM, we estimate by approximately maximizing the marginal likelihood of the observed variables:
arg max
II p(d)
II
Σ
p(d]g, 0)p(g)
dE
dE
4
18

---

Published as a conference paper at ICLR 2023
The first step in EM is to make an initial estimate of p(qld) for every document d. We FLAN query generator as our initial estimate: (d)). We then proceed to the M-step EM, which computes:
Σ p
arg I18X
dEDr
This is equivalent to training an initial retriever p(dlg, 0) on documents from Dr that have been paired with FLAN--enerated queries (what do). Finally, we proceed to the E-step of EM, which estimates p(q|d, 0) for every document d:
p(gld, 0) p(dlq, ()p(()
From this, we that the highest probability queries under p(g (d. 0) are the ones with the highest probability of retrieving d under our initial retriever (since p(q) is an uninformative prior that is uniform over all q).
To sample from p(q/d, 0), we could employ importance sampling. In importance sampling, we first sample from any proposal distribution that we choose, Pprup (g)d). We would then weight that sample by the ratio p(gld, ( Pproo(ddd)) We choose PFLAN as our proposal distribution. Then, instead of actually importance-weighiinn each sample, our filtering procedure just discards samples with low value of p(d q, 6), which is proportional to the numerator in the importance weight. This reveals both the differences and connections between our method and EM.
Although we could repeat this EM-like procedure until convergence, we found that single round of filtering yielded sufficient quality gains. also worth noting that the training of our final retriever could be viewed another M-step.
19

---

Published as a conference paper at ICLR 2023
SciFact
the 31
14.9
what
31.9
eliminating
50.0
10
Where
13.1
Increased
thare
when
pyriogsaain 0.8
634
Ohere
this 0.6
74.4
Others
which
10.6
transacting
50.0
Gold queries
Few-shot examples
NQ-QGen
Prompts (4 examples)
HotpotQA
15.4
who where
31.6
who
33.3
which
what
13.4
22.4
which 11.5
16.3
25.2
in 16.7
who0
818
when 2.3
how 4.1
in 4.8
when
how
the
2.5
2.7
11.8
springtime
which 16.7
16.7
Others
38.9
Others 7.0
Others
8.1
ameesha 16.7
Gold queries
Few-shot examples
NQ-QGen
Prompts (6 examples)
FIQA
15.4
what 17.4
31.4
what
31.4
what 16.7
what
12.4
how
who
21.5
16.7
why
IS
11.7
when
14.7
16.7
can 4.3
8
why 9.3
where
how
10.5
why
16.7
should
2.5
can
why
3.7
how
16.7
Others
52.5
Others
22.7
Others
16.7
Gold queries
Few-shot examples
NQ-QGen
Prompts (6 examples)
Fever
811
the -4.8
10.4
what
who
24.5
33.7
what 66.7
2hore
In 3.4
where
15.4
John 0.9
what 2.5
how
3.6
Others
83.9
Others
61.2
Others
7.9
music
33.3
Gold queries
Few-shot examples
NQ-QGen
Prompts (3 examples)
Climate-Fever
the
10.3
25.3
33.3
2.3
when
20.5
while 1.6
where
15.1
in
33.3
giobal
the
Others
73.6
Others
2.9
60.9
Others
8.5
nevertheless
33.3
Gold queries
Few-shot examples
NQ-QGen
Prompts examples)
-REC-Covid
what
88.0
what
792
what
30.6
what
66.7
8.0
which 1.5
which 4.0
Why 1.1
when
12.1
Others
2.0
6.0
Others 3.5
are 0.5
Omeer
who
4.7
15.6
how
33.3
Gold queries
Few-shot examples
NQ-QGen
Prompts (3 examples)
Webis- Touché2020
should
67.3
should
24.6
what
24.5
(in
60 8.1
3fe
thc
where
when
744
9.6
should
100
Others
310 /20
Wn1ch
Others
how
4.6
10.9
Prompts examples)
Gold queries
Few-shot examples
NQ-QGen
20

---

Published as a conference paper at ICLR 2023
SCIDOCS
a 18.9
what
sentiment
20.0
an 2.8
the 16.0
20.0
the 2.7
automatic 1.0
hew 10.5
learingg 1.3
0.9
who 7.0
20
20.0
deep 1.0
on 1.0
what 0.9
how 0.8
where 6.9
when
6.8
statistics
20
Others
82.8
Others
74.8
Others
17.8
parasitic
20.0
Gold queries
Few-shot examples
NQ-QGen
Prompts (3 examples)
NFCORPUS
how 2.7
15 2.3
what 15.8
whot
3833
proventing
33.3
the 2.1
the 3.7
1.2
dietary 2.4
whon
10.8
33.3
preventing 0.8
what
00es
can 1-6
who
5.2
platelet
33.3
Others
89.9
Gold queries
Few-shot examples
NQ-QGen
Prompts examples)
EePedia-Entiiy
how 15.4
what who
36.8
who what
26.6
who marlin magazines
what 12.4
25.0
23.6
why 6.5
when
14.4
when
22.0
15 6.4
how 3.8
where
17.7
can 4.3
the 2.9
how 4.5
events Others
the 1.2
edward SOuth
15.4
Others
4.5
DB Gold queries
Few-shot examples
NQ-QGen
Prompts (8 examples)
Figure 4: Top first word distribution on queries generated from different models in all other BEIR datasets.
Zwro-tol
(document) Read passage and generate query
Tiler
Phetraming with adxpandert
MDD
m
Training
Large LM
Smaal
Target domain documents
Synthetic data (query, document)
Negelive mining
n D
Argument: [dsc1) Counter argumer-aggmmet:: (query1) Argument (loc2) Counter-arg.ment (query2) X Augument (doaurent)
Positive
Negathow synthettcc deits Opinny,
Promet
Figure 5: PROMPTAGATOR++ Training pipeline.
21

---

Published as a conference paper at ICLR 2023
Passage: As the COVID-19 pandemic sweeps the globe, evolving containment measures have created an unprecedented need for rapid and effective science communication that is able to engage the public in behavioural change on mass scale. Public health bodies, governments, and media outlets have turned to comics this time of need and found natural and capable medium for responding to the challenge...
Query: What is the authors' purpose for writing this article?
Remarks:
The query lacks context. This query is also generic and can used to all articles
Passage: Big Bad Love a 2001 film directed by Arliss Howard, who co-wrote the script with his brother, James Howard, based collection of short stories of the name by Larry Brown. 'he story recounts an episode in the life of an alcoholic Vietnam veteran and struggling writer named Leon Barlow, who is played by Arliss Howard, and his wife, played by Howard 's wife Debra Winger. The soundtrack includes music by Tom Verlaine the Kronos Quartet, and R. L. Burnside. Query: music artist is from the United States
Remarks: The query lacks context.
Passage: Aminopeptidase N (APN) is the major cell surface receptor for group 1 coronaviruses. In this study, we have isolated and characterized feline APN cDNA and shown that the transfection of human embryonic kidney cells with this cDNA renders them susceptible to infection with the feline coronavirus feline infectious peritonitis virus, the human coronavirus (HCV) 229E and the porcine coronavirus porcine transmissible gastroenteritis virus.
Query: What is the function of Aminopeptidase N?
Remarks:
The query is too general and can match many documents that mentions APN.
Passage: Selena Danielle Coppa (born February 25, 1983) was a military intelligence Sergeant the United States Army. She is primarily notable for her organizing and activism against the US Occupation of Iraq while serving as an active duty military member, including serving on the Executive Board of Iraq Veterans Against the War. In 2009 it was announced that she was heading a committee responsible for gaining and training more active duty anti-war soldiers.
Query: who was the first president of the senate? Remarks: Hallucination. The passage didn't 't mention president of the senate.
Passage: Porcine deltacoronavirus (PDCoV) is an emerging swine coronavirus that causes severe diarrhea, resulting high mortality neonatal piglets. Despite widespread outbreaks in many countries, no effective PDCoV vaccines are currently available. Here, we generated, for the first time, full-length infectious cDNA clone of PDCoV. We further manipulated the infectious clone by replacing the NS6 gene with green fluorescent protein (GFP) to generate rPDCoV-NS6-GFP
Query: What virus is the first one that we' ve been able to create a vaccine for?
Remarks: Hallucination. The passage didn't 't mention that Porcine deltacoronavirus is the first virus create a vaccine.
Passage: Nacoleia rectistrialis is a moth in the Crambidae family. It was described by Hampson in 1912. It found the Key Islands.
Query: Who composed the music for the movie the ring? Remarks: Hallucination. The passage is about moth and is unrelated "music for the movie the ring".
Figure 6: Examples of queries from few-shot PROMPTAGATOR'S prompt-based query generation that were removed by round- trip filtering.
22


Reviews:
[{"review_id": "T9ikO7dQooV", "reviewer": "Reviewer_Qvy4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an approach to using a large language model (LLM) to synthesize fine tuning data for retrieval. Specifically, a question is generated for a given document, where the document is \"answers\" the question. \"Answers\" can have a task specific meaning, hence the quotation marks. A small number of task specific examples are used as prompts in the generation of this question. This synthetic data is then used to fine tune a retrieval model (Promptagator, scores query and candidate document, independent of other docs), or a retrieval model plus cross-attention reranker model (Promptagator++, ie a model that models some relation between query and candidate documents).\nThe paper presents results across 10 retrieval tasks (datasets), showing that this approach results on average in accuracy gains, according to the retrieval @10 metric. \n\nThe paper is well written. It must be said that it combines and builds upon existing ideas and very related work, but it is likely of value and interest to the retrieval community. ", "summary_of_the_review": "A solid, iterative contribution to the document retrieval literature. ", "strength_and_weaknesses": "Strengths I saw: \n* Most details are clearly described on the proposed approach and the conducted experiments.\nThe results are on public datasets, and public \"foundation model\" checkpoints. \n\n* The argument that there are different types of retrieval problems, where for one task (queryA, docA) is a valid pair, and for another it isn't, is well motivated. \n\n* The results on using the initially trained retriever model, to then score the synthetic data it was being trained on, remove anything not meeting adequate confidence that (query, doc) is a valid pair, and then continue training is a nice result. \n\n* The strengths of the zero-shot data gen results are also promising. \n\nRegarding weaknesses, I don't see any red flags that question the results or merit of the work. I do have some questions though.\n* The first is around Promptagator using FLAN as it's LLM , compared to (Thakur et al. 2021) and this papers own use of T5 as the question generation model. The results in table 3 hint that if prior works had used FLAN rather than T5, GPT etc for their Q generation, then maybe their results would be at Promptagators, or better. Also, why is the \"FLAN original\" result in table 3 not one of the rows in table 2?\n* Is the Intent(doc, query) premise (raised in the intro about how there are different types of retrieval tasks) not weakened by the rows in Table 2 where pre-training on MSMARCO results in good gains?\n\n* Minor, but some of the discussion where the fact the model is smaller than others (125M params) is slightly misleading I thought, since a 139B param model was used for data synthesis. Point completely agreed on inference time though. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NvKxfGe6AGq", "reviewer": "Reviewer_HWjD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposed a few-shot retrieval evaluation setting, which addresses the difference in the search intent and query distribution for different retrieval tasks. They also proposed a simple recipe for few-shot retrieval by prompting an LLM to generate synthetic task-specific training data and then train a dual encoder on the generated dataset. The experiment results show that its few-shot performance with two-to-eight examples is even better than a fully supervised model that is trained on MS MARCO or NQ.", "summary_of_the_review": "This paper proposed (1) a new task setting for BEIR retrieval benchmark (2) a few-shot method for learning a retrieval model by prompting the LLMs to generate data. The results are solid. I would like to recommend this paper be accepted.", "strength_and_weaknesses": "Strength:\n- The authors proposed a new few-shot setting for the BEIR benchmark, which is more reasonable because of the wide discrepancy between all the tasks in BEIR.\n- With only 8 examples, PROMPTAGATOR is able to outperform supervised models that are trained on labeled paired datasets.\n- Baselines and references of prior work are sufficient for comparison.\n- Round trip filtering is designed and evaluated as a good method to clean the dataset for better performance.\n\nWeaknesses:\n- For all the baseline models you have for BEIR, they are not adapted with the 8 examples. So it's basically not a fair comparison. I know 8 examples may not help a lot for these baseline models, but I think it's better to add the results by adapting the models using 8 examples.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8HbJ44Tb3NQ", "reviewer": "Reviewer_FSYW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on low-resource text retrieval setting that is different from the more widely considered settings so far. In particular, to address the lack of annotated data for retrieval tasks, existing works usually aim to train a retriever using available labeled data (e.g. MS MARCO) with good generalization to a range of target tasks (e.g. BEIR) in a zero-shot fashion. This work, instead, tries to build a task-specific retriever for each target task using a handful of target-task annotations (few-shot).\n\nTo achieve this end, they prompt a very large LM (the 137B FLAN) with the few-shot annotations, as well as the entire document corpus, from the target retrieval task to generate millions of pseudo-queries. These pseudo query-document pairs are then filtered using a retriever to improve their quality. Eventually, the filtered synthetic query-document pairs can serve as training data to train i) a bi-encoder neural retriever and (optionally) ii) a cross-encoder reranker. This proposed model is evaluated on a number of retrieval tasks from BEIR which shows competitive results.", "summary_of_the_review": "This paper identifies a valid issue in low-resource retrieval, which is the difficulty of capturing the diverse \"search intents\" and \"query distributions\" in different target tasks, but the proposed task-specific few-shot setting is, in my opinion, less than ideal compared to the existing transfer learning paradigm. It proposes a straightforward method to generate synthetic data with LLMs and achieves competitive results on BEIR, although the comparison with baselines is not exactly apple-to-apple. Finally, the proposed approach is very expensive in practice, and may not be well justified given its performance improvement over other retrievers that are simply trained on MS MARCO and adapted to the target dataset in a zero-shot fashion.", "strength_and_weaknesses": "Strengths:\n- The argument that retrieval tasks differ in \"search intents\" and \"query distributions\" is sound and worth considering when designing a retriever in the low-resource setting.\n- The proposed method is straightforward and clearly presented. \n- As a general data augmentation approach, (theoretically) the proposed method can also be used to train other retrievers.\n\nWeaknesses:\n- In order to use the proposed approach in practice, for each target retrieval task, one needs to query a 100B LLM to generate millions of samples, which is impractical for almost everyone but a few who have access to an enormous amount of computation power. \n- In this work, a separate retriever is trained for each target retrieval task. Compared to the traditional \"transfer learning\" setting where a single retriever is trained to generalize to many new target tasks (in either zero-shot or few-shot fashion), the proposed paradigm is arguably more cumbersome and expensive. For instance, to address the concerns of different \"search intents\" and \"query distributions\", it is potentially better to train a general multi-task retriever that can be easily adapted to new target tasks with different intents using prompts (instructions) or other techniques.\n- As the proposed setting is different from virtually all previous works, the comparison is not exactly apple-to-apple. For instance, this work utilizes the target corpus during training (to generate queries), which is not the case for many baseline methods. Also this method is training a separate retriever for each target task, effectively increasing the model capacity, which is also different from the baselines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "55XZcGCOWW", "reviewer": "Reviewer_ZHw5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of training a few-shot dense retriever, i.e., a dual encoder model that can operate well in on a new distribution with only a small number of question-paragraph pairs. To do so, they train a large LM to generate questions given unlabeled paragraphs as input. This LM is trained in a instruction-tuned way, and then prompted at test time with the few examples of the desired task. After generating the synthetic question-paragraph examples, they train a seperate dual encoder model on the data.", "summary_of_the_review": "I think the paper is compelling enough to warrant an accept given its clear writing, new methods, and sensible evaluation.", "strength_and_weaknesses": "Strengths:\n* Novel and semi-practical setting of few-shot retrieval in novel domains.\n* Simple and also very timely solution (instruction-tuned LLMs)\n* Sensible evaluation, common datasets and pretty sensible baselines.\n* Paper is clear and well-written\n\nWeaknesses:\n* Would have loved to see more multitask baselines. In particular, part of the paper's motivation is that rather than multi-tasking and hoping to generalize to the target task, one would like to quickly adapt to the target distribution. In that sense, it'd be interesting to see some methods such as training a multi-task biencoder and running it zero-shot on the target domain, or finetuning that multi-task biencoder on the few-shot examples. (I am thinking of the multi-task DPR model from Maillard et al. 2021).\n* Not actually instruction-tuning a QA-specific model. Instead, the method is basically just querying off-the-shelf systems.\n\nRandom Idea:\n* It would be an interesting idea to see how well you could use LLMs for retrieval without needing to train a seperate retriever (although it would be incredibly slow). In particular, take a prompted LLM that takes as input a series of question-paragraph pairs, and then score test paragraphs using p(paragraph|question). The reason I think this is interesting is because in your current method, you are essentially distilling a large LM into a dual encoder by generating synthetic data. I am wondering if doing the distillation causes the dual encoder to actually be a worse or better retriever than the original slow LM.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster