Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed the idea of learning a dense retriever with a list of learned prefixes. The model attends to prefixes to compute query-dependent prefixes, which will be used to compute final query vectors for retrieval. In order to make the prefixes non-uniform, the paper additionally introduced a contrastive prompt regularization loss.
strength_and_weaknesses: The idea of computing query-dependent prefix to compute query vectors for retrieval is an appealing idea. The results shows that having learned prefixes does improve models' performance in many downstream tasks. The retrieval performance is also improved in zero-shot setting. Overall, I think this is an interesting paper. Thanks for the excellent work.

My main concern is the additional computation cost coming from the added prefixes. As discussed in the paper, the length of the prefix is 100. For many retrieval tasks, queries are short (sometimes 20~30 tokens). The added prefix tokens may increase the computation cost by ten times or more. Have you tried measuring the efficiency of your model and how it compares to the baselines?

If the improvement comes from having more parameters, why is it different from increasing the size of the model? Or, can you append 100 special tokens to queries and let models learn their embeddings.

A few more questions:
1. Does the number of task affect the size of the prefixes? Do you need more prefixes or longer prefixes if there are, say hundreds of tasks?
2. Are model's parameters in SPLADE jointly finetuned with parameters of the prefixes?

There are many paper come out recently in few-shot and zero-shot retrieval. You may want to add some of their numbers to your paper, e.g. Promptagator [1]. The numbers may not be directly comparable, but it's good to have.

[1] Promptagator: Few-shot Dense Retrieval From 8 Examples

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to read. Nice work.

I don't see any challenges in reproducing the results, but it's always nice to open source your codes and checkpoints.
summary_of_the_review: The proposed methods is well motivated and strongly justified by the experiments. The only drawback is that it's not very clear where does the improvement come from (see my question above). But overall, it is a nice paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors investigate the out-of-domain generalization problem in neural retrieval. They propose a hyper-prompt training mechanism and contrastive prompt regularization to this problem. The hyper-prompt training predicts a distribution over the predefined prompt parameters given the query, and get the task-adaptive prompt embeddings as prefix for the each layer of query encoder to augment the query representation. Contrastive prompt regularization solves the mode collapse problem of attention score distributions. Experimental results show that the proposed approach achieves slight improvement in the multi-task training setting but leads to significant improvement in zero shot setting and on out of domain tasks.
strength_and_weaknesses: ### Strength
- The propose approach is novel on designing a hyper-prompt training  approach for neural retrieval out-of-domain generalization.
- The authors conduct comprehensive experiments and analysis to justify the effectiveness of every component of the proposed approach (hyper-prompt, contrastive prompt regularization). Results are shown on both multi-task and zero shot settings.

### Weakness
- This work claims to target the out-of-domain generalization for neural retriever but experiments only justify the out-of-dataset generalization. Most retrieval tasks are from the Wikipedia domain.   

clarity,_quality,_novelty_and_reproducibility: - Clarity: This work is generally well-written and easy to follow.
- Quality: This work is technically sound.
- Novelty: This work is somewhat novel.
- Reproducibility: No source code is shared, it might be challenging to reproduce.

summary_of_the_review: In summary, I feel that this is a good work in general, though there are some minor concerns on the definition of “out-of-domain generalization”.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hyper-prompted training mechanism for large-scale text retrieval across tasks of different domains. 
strength_and_weaknesses: Strength:
1. The problem of cross-domain text retrieval that this paper studies is important.
2. The paper is well written with clear motivation, method description, and experiment setup. 
3. The empirical results are promising and comprehensive, covering two popular benchmarks KILT and BEIR, two representative retrieval methods DPR and SPLADE, and two settings supervised and zero-shot.

Weakness:
The main weakness of this paper is the novelty of the proposed HYPER mechanism. It lacks the comparison with other transfer learning mechanisms such as the ones mentioned in [1] and [2]. Although those papers do not conduct experiments on retrieval, it's easy to extend them to the retrieval setting. It seems to me that HYPER is also very general and can be applied to other downstream settings like GLUE. Thus it's necessary to have a discussion with other transfer learning mechanisms. 

[1] He, Junxian, et al. "Towards a unified view of parameter-efficient transfer learning." arXiv preprint arXiv:2110.04366 (2021).
[2] Asai, Akari, et al. "Attentional Mixtures of Soft Prompt Tuning for Parameter-efficient Multi-task Knowledge Sharing." arXiv preprint arXiv:2205.11961 (2022).



clarity,_quality,_novelty_and_reproducibility: As mentioned above, the clarity and quality of this paper are good. The reproducibility is fine with clear experiment setups on page 5 and impacts of hyper-parameters on page 8. 
However, the novelty part is a concern. Although the proposed method is more effective than prefix-tuning, it needs to answer a key question: is it the first method to construct **query-specific** prompt for different tasks? If not, how's the proposed method compared to previous methods?
summary_of_the_review: Overall I found this paper clearly written with strong motivation and has extensive and promising experiment results, but the novelty contribution is unclear. Thus I think this paper as marginally above the acceptance threshold and highly recommend the authors to add the discussion and necessary baselines into this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes HYPER, a mechanism for transfer learning among retrieval tasks in different domains and datasets. For each query, it first dynamically generates a hyper-prompt to synthesize the query by attending to a given number of shared prompts, then encodes the synthesized prompt into prefixes, and employs the prefix-tuning method to train the retriever model in a multi-task setting. Contrastive learning is employed to cluster synthesized prompts based on the datasets they come from. Experiment results show the efficacy of HYPER, especially on 0-shot in-domain and cross-domain retrieval settings.

strength_and_weaknesses: Pros:
- The paper presents an effective approach to address a practical problem of domain adaptation in retrieval, following on the recent popular direction of prompt tuning and prefix tuning.
- The approach is novel. Compared with previous related work (e.g. Hyper-Prompt), HYPER addresses retrieval task across different domains/datasets, so it dynamically generates/conditions distinct prompts on the query-level instead of task-level.
- The approaches, model architecture, formulas, and experiments are sound and solid, very clearly explained, and easy to follow. The paper is well-organized and written. 
- Experiments are conducted on popular public datasets of KILT and BEIR, and are compared against the recent SOTA systems in the retrieval field. The results and conclusions are convincing.
- The discussions and analysis are detailed and insightful. The analysis around prompt attention similarity and task similarity, together with visualization, is interesting and informative.

Cons:
- The approach is employed on query encoder. It would be interesting to explore and discuss about the effect of transfer learning on the document encoder as well. Also, I am a little curious about employing HYPER in retrieval models where the query encoder and document encoder share a large part of parameters, for example, ColBERT.
- A small suggestion: It would be interesting to add some query examples (maybe in Appendix) with similar and dissimilar prompt attention scores.


Question:
Page 3, when describing how to obtain the query representation Q, should it be generalized to any retrieval model's query encoder? Or, maybe mention that this description is specific to a particular model, and the other models follow in a similar way?

Minor editing errors:
Page 7, 'we calculate the mean...' -> should it be 'we calculate the similarities between the mean...' ?
Page 9, 'considerably scare' -> 'considerably scarce'

clarity,_quality,_novelty_and_reproducibility: Clarity: very clear, with few minor changes to improve on.

Quality: solid and sound with extensive experiment results to back up the claim.

Novelty: is novel in dynamically conditioning prompts on the query-level instead of task-level.

Reproducibility: The authors state that "Our implemented codes will be made public." 

summary_of_the_review: The paper addresses a well motivated problem effectively with recent advances in prefix and prompt tuning, and with innovations on conditioning prompts on the query-level instead of task-level. The claims are backed up by extensive experiments and analysis on recent public datasets compared with SOTA systems.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission introduces an architecture for improving dense retrievers in low resource settings. The approach learns a query-specific mixture of soft prompts which are used to improve a dense retrieval model. During training, an additional contrastive loss is used to prevent all tasks learning the same prompt. 

The method tackles an important problem, and is well motivated and clearly explained. The approach shows convincing results when evaluated on the BEIR and KILT benchmarks, and outperforms relevant ablations such as prefix tuning.

Adding 100 extra soft-prompt tokens to the query reduces efficiency and may make the method less practical. The requirement of an additional contrastive loss somewhat complicates training.

Overall this is a good submission that should be accepted.
justification_for_why_not_higher_score: Adding 100 extra soft-prompt tokens to the query reduces efficiency and may make the method less practical. The requirement of an additional contrastive loss somewhat complicates training. 
justification_for_why_not_lower_score: The method is well motivated and clearly explained. The approach shows convincing results when evaluated on the BEIR and KILT benchmarks, and outperforms relevant ablations such as prefix tuning. Overall, this is a good submission that should be accepted.
Decision: Accept: poster