Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the problem of training a few-shot dense retriever, i.e., a dual encoder model that can operate well in on a new distribution with only a small number of question-paragraph pairs. To do so, they train a large LM to generate questions given unlabeled paragraphs as input. This LM is trained in a instruction-tuned way, and then prompted at test time with the few examples of the desired task. After generating the synthetic question-paragraph examples, they train a seperate dual encoder model on the data.
strength_and_weaknesses: Strengths:
* Novel and semi-practical setting of few-shot retrieval in novel domains.
* Simple and also very timely solution (instruction-tuned LLMs)
* Sensible evaluation, common datasets and pretty sensible baselines.
* Paper is clear and well-written

Weaknesses:
* Would have loved to see more multitask baselines. In particular, part of the paper's motivation is that rather than multi-tasking and hoping to generalize to the target task, one would like to quickly adapt to the target distribution. In that sense, it'd be interesting to see some methods such as training a multi-task biencoder and running it zero-shot on the target domain, or finetuning that multi-task biencoder on the few-shot examples. (I am thinking of the multi-task DPR model from Maillard et al. 2021).
* Not actually instruction-tuning a QA-specific model. Instead, the method is basically just querying off-the-shelf systems.

Random Idea:
* It would be an interesting idea to see how well you could use LLMs for retrieval without needing to train a seperate retriever (although it would be incredibly slow). In particular, take a prompted LLM that takes as input a series of question-paragraph pairs, and then score test paragraphs using p(paragraph|question). The reason I think this is interesting is because in your current method, you are essentially distilling a large LM into a dual encoder by generating synthetic data. I am wondering if doing the distillation causes the dual encoder to actually be a worse or better retriever than the original slow LM.  
clarity,_quality,_novelty_and_reproducibility: New setting, simple and novel method, clear writing. Would like to see more baselines.
summary_of_the_review: I think the paper is compelling enough to warrant an accept given its clear writing, new methods, and sensible evaluation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The author proposed a few-shot retrieval evaluation setting, which addresses the difference in the search intent and query distribution for different retrieval tasks. They also proposed a simple recipe for few-shot retrieval by prompting an LLM to generate synthetic task-specific training data and then train a dual encoder on the generated dataset. The experiment results show that its few-shot performance with two-to-eight examples is even better than a fully supervised model that is trained on MS MARCO or NQ.
strength_and_weaknesses: Strength:
- The authors proposed a new few-shot setting for the BEIR benchmark, which is more reasonable because of the wide discrepancy between all the tasks in BEIR.
- With only 8 examples, PROMPTAGATOR is able to outperform supervised models that are trained on labeled paired datasets.
- Baselines and references of prior work are sufficient for comparison.
- Round trip filtering is designed and evaluated as a good method to clean the dataset for better performance.

Weaknesses:
- For all the baseline models you have for BEIR, they are not adapted with the 8 examples. So it's basically not a fair comparison. I know 8 examples may not help a lot for these baseline models, but I think it's better to add the results by adapting the models using 8 examples.
clarity,_quality,_novelty_and_reproducibility: Clarity&Quality: The paper is well-written and easy to follow
Novelty: Good novelty -- new task setting was developed.
Reproducibility: The authors didn't provide the code so we cannot reproduce it.
summary_of_the_review: This paper proposed (1) a new task setting for BEIR retrieval benchmark (2) a few-shot method for learning a retrieval model by prompting the LLMs to generate data. The results are solid. I would like to recommend this paper be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an approach to using a large language model (LLM) to synthesize fine tuning data for retrieval. Specifically, a question is generated for a given document, where the document is "answers" the question. "Answers" can have a task specific meaning, hence the quotation marks. A small number of task specific examples are used as prompts in the generation of this question. This synthetic data is then used to fine tune a retrieval model (Promptagator, scores query and candidate document, independent of other docs), or a retrieval model plus cross-attention reranker model (Promptagator++, ie a model that models some relation between query and candidate documents).
The paper presents results across 10 retrieval tasks (datasets), showing that this approach results on average in accuracy gains, according to the retrieval @10 metric. 

The paper is well written. It must be said that it combines and builds upon existing ideas and very related work, but it is likely of value and interest to the retrieval community. 
strength_and_weaknesses: Strengths I saw: 
* Most details are clearly described on the proposed approach and the conducted experiments.
The results are on public datasets, and public "foundation model" checkpoints. 

* The argument that there are different types of retrieval problems, where for one task (queryA, docA) is a valid pair, and for another it isn't, is well motivated. 

* The results on using the initially trained retriever model, to then score the synthetic data it was being trained on, remove anything not meeting adequate confidence that (query, doc) is a valid pair, and then continue training is a nice result. 

* The strengths of the zero-shot data gen results are also promising. 

Regarding weaknesses, I don't see any red flags that question the results or merit of the work. I do have some questions though.
* The first is around Promptagator using FLAN as it's LLM , compared to (Thakur et al. 2021) and this papers own use of T5 as the question generation model. The results in table 3 hint that if prior works had used FLAN rather than T5, GPT etc for their Q generation, then maybe their results would be at Promptagators, or better. Also, why is the "FLAN original" result in table 3 not one of the rows in table 2?
* Is the Intent(doc, query) premise (raised in the intro about how there are different types of retrieval tasks) not weakened by the rows in Table 2 where pre-training on MSMARCO results in good gains?

* Minor, but some of the discussion where the fact the model is smaller than others (125M params) is slightly misleading I thought, since a 139B param model was used for data synthesis. Point completely agreed on inference time though. 
clarity,_quality,_novelty_and_reproducibility: Some in-the-details questions
- in section 4.1, reranking the top 200 candidates. Is that standard? Do other works use 200 as well? For each task what % of the time was the correct document in the top 200?
- was the pretraining the retriever on the C4 data the same as Contriever? (Apologies, I haven't gone over that work). If so, why was this done, rather than starting with the Contriever model and fine-tuning that?
summary_of_the_review: A solid, iterative contribution to the document retrieval literature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on low-resource text retrieval setting that is different from the more widely considered settings so far. In particular, to address the lack of annotated data for retrieval tasks, existing works usually aim to train a retriever using available labeled data (e.g. MS MARCO) with good generalization to a range of target tasks (e.g. BEIR) in a zero-shot fashion. This work, instead, tries to build a task-specific retriever for each target task using a handful of target-task annotations (few-shot).

To achieve this end, they prompt a very large LM (the 137B FLAN) with the few-shot annotations, as well as the entire document corpus, from the target retrieval task to generate millions of pseudo-queries. These pseudo query-document pairs are then filtered using a retriever to improve their quality. Eventually, the filtered synthetic query-document pairs can serve as training data to train i) a bi-encoder neural retriever and (optionally) ii) a cross-encoder reranker. This proposed model is evaluated on a number of retrieval tasks from BEIR which shows competitive results.
strength_and_weaknesses: Strengths:
- The argument that retrieval tasks differ in "search intents" and "query distributions" is sound and worth considering when designing a retriever in the low-resource setting.
- The proposed method is straightforward and clearly presented. 
- As a general data augmentation approach, (theoretically) the proposed method can also be used to train other retrievers.

Weaknesses:
- In order to use the proposed approach in practice, for each target retrieval task, one needs to query a 100B LLM to generate millions of samples, which is impractical for almost everyone but a few who have access to an enormous amount of computation power. 
- In this work, a separate retriever is trained for each target retrieval task. Compared to the traditional "transfer learning" setting where a single retriever is trained to generalize to many new target tasks (in either zero-shot or few-shot fashion), the proposed paradigm is arguably more cumbersome and expensive. For instance, to address the concerns of different "search intents" and "query distributions", it is potentially better to train a general multi-task retriever that can be easily adapted to new target tasks with different intents using prompts (instructions) or other techniques.
- As the proposed setting is different from virtually all previous works, the comparison is not exactly apple-to-apple. For instance, this work utilizes the target corpus during training (to generate queries), which is not the case for many baseline methods. Also this method is training a separate retriever for each target task, effectively increasing the model capacity, which is also different from the baselines. 
clarity,_quality,_novelty_and_reproducibility: This work is clearly presented and well-written. The idea of using LLM to generate synthetic data for model training is not new, even for the retrieval task (UPR, InPars), but the authors claim that their approach differs in that i) they do task-specific adaptation, and ii) they focus on first-stage retrieval instead of (or in addition to) reranking.

The method should be relatively straightforward to reproduce theoretically, if one has the required computation power. In practice, very few would be able to query a 100B LLM to generate millions of samples for their retrieval task.
summary_of_the_review: This paper identifies a valid issue in low-resource retrieval, which is the difficulty of capturing the diverse "search intents" and "query distributions" in different target tasks, but the proposed task-specific few-shot setting is, in my opinion, less than ideal compared to the existing transfer learning paradigm. It proposes a straightforward method to generate synthetic data with LLMs and achieves competitive results on BEIR, although the comparison with baselines is not exactly apple-to-apple. Finally, the proposed approach is very expensive in practice, and may not be well justified given its performance improvement over other retrievers that are simply trained on MS MARCO and adapted to the target dataset in a zero-shot fashion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to use large language models to generate queries for a small set of documents in a few shot retrieval setting. The setting itself is part of the novelty. The idea is simple and practical. Since this is a new setting, the focus should be on the evaluation of the baseline and the improvement over the baseline. There are some discussion about whether the baseline is strong enough and whether it is fair for the baseline. Most reviewers (including me) are convinced.

There is a concern about whether using FLAN takes up too much compute. It is indeed a concern given that the paper has claims about the low compute it needs. The wording can be fixed. The compute problem will be fixed as GPUs get more powerful, so overall I would OK with this issue.
justification_for_why_not_higher_score: The idea is simplistic. It would have got a higher score if the paper has more technical depth. Many reviewers mention the multitask setting. The authors also agree that this is an interesting direction, but decides to limit the scope of the paper.
justification_for_why_not_lower_score: The idea is simple and effective.
Decision: Accept: poster