HypeR: Multitask Hyper-Prompted Training Enables Large-Scale Retrieval Generalization
Authors: ZeFeng Cai
Chongyang Tao
Tao Shen
Can Xu
Xiubo Geng
Xin Alex Lin
Liang He
Daxin Jiang
Venue: ICLR 2023 poster (2023)
Keywords: Uniformed Large-Scale Retrieval
Multi-Task hyper-prompted training
Retrieval Generalization

Abstract: Recently, large-scale text retrieval has made impressive progress, facilitating both information retrieval and downstream knowledge-intensive tasks (e.g., open-domain QA and dialogue). With a moderate amount of data, a neural text retriever can outperform traditional methods such as BM25 by a large step. However, while being applied to out-of-domain data, the performance of a neural retriever degrades considerably. Therefore, how to enable a retriever to perform more robustly across different domains or tasks  and even show strong zero-shot transfer ability is critical for building scalable IR systems. To this end, we propose HypeR, a hyper-prompted training mechanism to enable uniform retrieval across tasks of different domains. Specifically, our approach jointly trains the query encoder with a shared prompt-based parameter pool and a prompt synthesizer that dynamically composes hyper-prompt for encoding each query from different tasks or domains. Besides, to avoid the mode collapse of prompt attention distribution for different queries, we design a contrastive prompt regularization that promotes the mode of prompt attention to be aligned and uniform. Through multi-task hyper-prompted training, our retriever can master the ability to dynamically represent different types of queries and transfer knowledge across different domains and tasks. Extensive experiments show our model attains better retrieval performance across different tasks and better zero-shot transfer ability compared with various previous methods.

Published as a conference paper at ICLR 2023
HYPER: MULTITASK HYPER-PROMPTED TRAINING EN- ABLES LARGE-SCALE RETRIEVAL GENERALIZATION
Zefeng Cai Chongyang Tao", Tao Shen", Can Xu², Xiubo Geng", Lengg"XX, Xin Lin', Liang He! Daxin Jiang 2t
Department of Computer Science, East China Normal University?
Microsoft Corporation, Beijing, China?
cklen& foxmail 1. com, {lhe, xlinjocss @cs. ecnu. cn, (chotao, shentao, caxu, xi ging, dj e www
ABSTRACT
Recently, large-scale text retrieval has made impressive progress, facilitating both information retrieval and downstream knowledge-intensin intensiee tasks (e.g., open- domain QA and dialogue). With a moderate amount of data, a neural text retriever can outperform traditional methods such as BM25 by large step. However, while being applied to out-of-domain data". the performance grades considerably. Therefore, how enable retriever to perform more robustly across different domains or tasks and even show strong zero-shot transfer ability is critical for building scalable IR systems. To this end, we propose HYPER, hyper-prompter training mechanism to enable uniform retrieval across tasks different domains. Specifically, our approach jointly trains the query encoder with a shared prompt-based parameter pool and prompt synthesizer that dynamically composes syper-prompt for encoding each query from different tasks domains. Besides, to avoid the mode collapse of prompt attention distribution for different queries, we design contrastive prompt regularization that promotes the mode of prompt attention to be aligned and uniform. Through multi-task hyper-prompted training, our retriever can master the ability to dynamically represent different types of queries and transfer knowledge across different domains and tasks. Extensive experiments show our model attains better retrieval performance across different tasks and better zero-shot transfer ability compared with various previous methods.
neural retriever de-
INTRODUCTION
Large-scale retrieval aims to retrieve relevant documents from millions to billions of documents according a given query, which the so-called first stage retrieval (Cai et al.] 2021). b. It can benefit for resolving various knowledge-intensie tasks significantly (Guu et al.) |020| Lewis et al. 2020), since the retrieved relevant documents contain explicit knowledge of world (Petroni al. 2021). Traditional cem--atchiin methods including tf-idf and BM25 (Yang et al.) 2017) can effectively achieve retrieval by building an inverted index and perform fairly well regardless of domains, however, recent popular neural retrievers outperform them by a large step with a moderate amount of task-specific data (Karpukhin et al., 2020: Formal et al. 202 Khattab Zaharia 2020).
For neural retrieval, a common way is to use pre-trained language models (e.g., BERT) (Devlin et al., 2019) to encode queries and documents into vectors respectively, which is known as Bi-Encoder. Although neural retrievers can be optimized effectively by utilizing the samples of specific tasks, in real-world applications, the formats of queries are different and the expected prioritics of query vectors varying considerably from task to task. For example, in Naturals Questions dataset (Kwiatkowski et af. 2019). a query such as "what was the first capital city of Australia" is a simple question sentence, however, in Wizard of Wikipedia dataset (Dinan et al. 2018), query such "noopp Dogg is so
"Work done during the internship at Microsoft. 1 "Corresponding author. 'Noting that here out-of-domain data refers to different tasks with different types of queries or the same task with data from different domains.

---

Published as a conference paper at ICLR 2023
awesome, he's a great rapper and does a lot for his community well..." contain multiple declarative sentences with implicit retrieval target. Besides the difference in query formats, different tasks also require generating query vectors with different richness intents, in HotpotQA dataset (Yang et al.) 2018) an input query "which game was published first, Near and Far or King Tokyo?" ?" expects an input query that can retriever documents relevant to the two mentioned items which are fair different from the queries in Natural Question that require retrieving specific facts to only one item. Those differences between tasks cause significant performance degradation when a model is applied to different tasks. Moreover, there is also data sparse problem for recently popular tasks (Almeida & Matos 2020). which expects a better generalization of a neural retriever (Thakur et al.) 2021).
To resolve the above challenges, we aim to build a universal model that is capable to process queries uniformly regardless of the differences between different tasks including varying formats of input queries and the unique features of query vectors for specific tasks. Meanwhile, we expect our model can obtain stronger generalization abilities which can be reflected by promising zero-shot and few-shot performance in large-scale retrieval. Specifically, the first problem is how to enable universal query process. For a neural retriever, the ability to resolve a specific task means a set of parameters trained on this task. Although one can train different models for each tasks (Karpukhin et al. 2020) or simply use a shared encoder with tullti- setting (Maillard et al. 2021). the first method leads to heavy parameter cost while the second method results in potentially indifferent generalization abilities.
To this end, we propose HYPER, multi-task HyPer-prompted training mechanism that can be combined with any transformer-batt based neural Retrieves. HYPER consists of two key components. The first component is Query-conditional Prompt Synthesizer (QPS) that leverages the attention module to synthesize suitable parameters of query encoder for different queries, which enables our query encoder to master the ability to dynamically represent different types of queries and transfer learned parameters across different tasks and domains by multi-task training. Nevertheless, we find merely applying QPS results in mode collapse problem of attention scores distributions, which causes our query encoder fails to learn different abilities to process queries for different tasks. To deal with this problem, we propose the Contrastive Prompt Regularization (CPR) to encourage the parameter synthesizing of the same tasks to become similar for better training effectiveness while promoting our query encoder to distinguish queries of different tasks and thus avoid mode collapse problems. Through the above multi-tast -aak to dynamically represent different types of queries and transfer knowledge across different domains and tasks. Therefore, HYPER can enable large-scale retrieval generalization in the zero-shot and few-shot scenarios.
training, HYPER can master the ability
To conclude, our contributions three-fold follows, i) we present HYPER, multitask hyper- prompted training mechanism that enables a neural retriever to dynamically process different types of queries with different hyper-prompts and transfer learned knowledge across different domains and tasks. ii) to impede the uniform retrieval in model construction and optimization, we propose cory-conditional Prompt Synthesizer (QPS) along with Contrastive Prompt Regularization (CPR) to synthesize suitable prompts for different queries. iii) Experiments in zero-shot in-domain and cross-domain retrieval tasks reflect the superior generalization provided by HYPER and the strong multi-tasking performance indicates the achieving of uniform retrieval.
2
METHOD
Task Formation For the large-scale text retrieval, we aim to seek document d. containing relevant knowledge from large collection of documents D to answer the query 9. Although input queries vary from task to task, we propose employing only one retriever to process them uniformly. Specifically, for datasets C = [Ti .,...,, T and out-of-domain data = (T++1, T J J.here f,k: is the number of tasks with training samples and without training samples respectively, the goal is to learn a neural retriever model P(d+lg. D; 6) (0 denotes the parameters of the model) with C and perform well on these in in-oomain tasks, while transferring the learned knowledge to process a new query from out-of-domain datasets C. Thus, given any queries in C U C, one can find the proper knowledge documents d4 following P(d+lg, D:0).
Model Overview Building upon pre-trained neural retriever, HYPER aims to dynamically synthe- size suitable prefixes to enable the retriever to process different queries uniformly and an illustration
2

---

Published as a conference paper at ICLR 2023
Query Representation
Sample T
Preficc
Transformer Layer
Prefix Encoder
Prefisila
Traniformer Layer
Align
Qurry-codizional prompt (p))
Prefix-UN
Transformer Layer
Sample 2 Task
Query Encoder
Contrastive Proqupt Regulriizaion
Attention Module
Query
Input query
Uniform
Instance Encoder
Sampling
Value
Key
BBC
Sample 3
Task
Shared
Pooling
My Ug Ug
Prompts
Figure 1: An illustration of HYPER architecture in multi multi-tasking training.
of it is provided in Figure [ HYPER first leverage the instance encoder 0, to generate the Query representation of a query, while hyper-prompts P are used as Key and Value for attention module. Therefore, we can dynamically generate different prompts for different types of queries. Besides, our proposed contrastive prompt regularization used to avoid the mode collapse problem which is crucial for learning different parameters for different types of queries.
2.1 PROMPT SYNTHESIZING
There are three main components in our query encoding process including an instance encoder O,, shared basic prompts P {px p; Ruxd ; E {1, [N] where N is the number of shared prompts, m is the length of each basic prompt, and the prefix encoder 8g. To enable our model to process different tasks uniformly, we store learned knowledge process different queries into shared prompts and synthesize prompts for different queries dynamically through the attention module. Moreover, we introduce contrastive prompt regularization to prevent mode collapse of attention scores, which is crucial for HYPER to effectively learn diverse knowledge to synthesize different prompts for queries of different tasks. the following, we will first describe how HYPER generate the corresponding prefix for different queries. Then, we explain the mode collapse problem in our attention module and how CPR resolves it.
Query-conditional Prompt Synthesizing We intend to generate dynamic hyper-prompts with the global semantic of a query which enables neural retriever to adapt different types of queries across different domains. To generate query representation for our prompt attention module, we first map input query q into corresponding word embeddings representation X [wi, W2, w/] R'xa where is the length of query, d is the dimension of word embedding. Then we employ max pooling along with the sequence axis of X and obtain X MaxPooling(XX Finally, we utilize a non-linear transformation to generate the incipient query representation as follows:
H1 GELU X WT Q LayerNorm(Hr).
(1)
Here, W(12) E RRdx xd are the transformation matrices and d, is the dimension of the hidden variable. GELUis Gaussian Error Linear Unit (Hendrycks & Gimpel (016)) and LayerNorm is the Layer-wise Normalization (Ba et al. 2016). Similar to query encoding, we employ max pooling operation along with the prompt length axis to transform each prompt into p. MaxPoolingg (,.), (1..... ... Thus, we can use Qp. to imply the fitness of different parameters for an input query. We employ softmax to normalize these scores and form a prompt atlention distribulion A E RR which is further used for synthesizing the final query-conditional prompt P1. The process is formally described as:
exp(QpT// a; ΣN exp(OpT/7)
N
PI Σ a.P.
(2)
where a; is the normalized attention score of the i-th prompt, Pf E R is generated query- conditional prompt, and 7 is the pre-defined temperature. To further improve the effectiveness of our proposed method, transform the generated prompt into prefix (Liu et al. 202 that owns the better representational ability. Although one can simply employ up projection to generate prefix for each layer of retriever, we find this approach considerably increases the number of total
3

---

Published as a conference paper at ICLR 2023
parameters which may cause the degradation of the generalization. Therefore, employ a parameter- efficient transformation (Stickland & Murray, 2019; Houlsby et al.] 2019) as the prefix encoder 8p (Wap: W ) J to generate the prefix which can be described as follow:
h W WapP1, PL (anh(h)
where Wap E RRd and Wap E RLdddd are projection matrices, dp is down projection dimension of Pr. is the number of layers of query encoder fy. Tanh is the hyperbolic tangent function, h is the intermediary low-dimension representation and PL E R&d is the dynamically generated parameters that can be split into prefixes for each layer of the neural retriever.
Contrastive Prompt Regularization Although the above mechanisms can generate query- conditional prefixes and share parameters across different tasks, find it results in the so-called mode collapse problem of attention score distributions. Specifically, the attention score distributions of different queries very similar which may cause our module degenerating Moreover, we expect queries belonging to the same task generate similar attention scores distribution while queries belonging to different tasks own dissimilar attention scores distribution. To simulta- neously learn representations of different domains or tasks, we propose the Contrastive Prompt Regularization (CPR) that employs soft constraint to cluster the attention scores implicitly and thus avoid the mode collapse problem. CPR can be formally described as follows.
prefix-tuning.
and clustering for different types of queries across
(CPR(C)= =
E Σ
Dj A(()))
E
Dj (((())))
BEC 9EBB REB,
QMEB Aly,
aligmment
uniformity
Here, B is mini-batch of training samples randomly selected from C, A(g.) P(=) 0z2 (1, :N} is the attention score distribution of the input query qu, Iq. is an indicator function that represents the dataset of task that the query 9. belonging D, is divergence that measures the similarity of two distributions. Inspired by contrastive learning (Wang & Isola| 2020). the first term that contains D (A(%), A(g;)) can be viewed as alignment regularization that encourages similar queries generated by similar prefixes, and the second term that contains Dy (A(q:), A(gx)) cab be viewed as uniformity regularization that prevents mode collapse of distributions of attention score. In our implementation, we use
divergence (Manning & Schütze 2002) since it owns certain upper and lower bounds which avoids the numeric overflow in optimizing.
2.2 UNIFORM RETRIEVAL WITH QUERY-CONDITION PROMPT
Lexicon-weighted Retriever HYPER compatible with any deep neural text retriever based on Transformer (Vaswani et al., 2017) architecture, however, we find the lexicon- method is more promising to attain better zero-shot retrieval" Therefore, we adopt a lexicon- weight language model SPLADE (Formal et al. 2021bb LM(:, : 0g) as backbone network. Combining with the generated dynamic prefix, we can represent a text of query as follows:
retrieval
02 Max.Pooling(log (1 + ReLU X; 02)))) E E
where ReLU is the Rectified Linear Unit. Following the common practice, we adopt the contrastive loss (Karpukhin et al., 2020 Khattab & Zaharia 2020: Formal et al., 202 that utilize a limited number of positive documents d4 and d_ for each queries for training. Specifically, we employ BM255 to retrieve top- 100 relevant documents as d_ as negative samples except those also contain answers to query. To encode positive documents and negative documents into corresponding vector representation va+ and Uz-, we employ a document encoder 0, to encode them but skip the prefix generation va Max MaxPoolingloog (1 ReL dd))))) E Rd Thus, likelihood distribution can be formatted as follows,
exp(vTTug va+))
d:: = Ededd exp (u. v
(6)
We also report evaluation results using the dense model (e.g., DPR) the backbone network in Table 'We adopt the default setting of Anserini for BM25 where k1 = 0.9, 0.4.
*Using dynamic representations of different documents requires to rebuild index in real-time which causes heavy calculation cost.
4

---

Published as a conference paper at ICLR 2023
Following the common practice of training lexical retriever (Formal al.) 2021bb. we add additional floating point operations per second (FLOPS) regularization terms (Paria et al. (020) to reduce the computation cost of the retrieval process and improve the retrieval effectiveness. Then, the loss function of our methods can be defined as
Cg(C) = Σ P(d+ d0 0a) ()) + x,FLOPS(e.) + AdFLOPS(wa),
QEC
where FLOPS(.) is regularization term proposed by (Formal et al. 2021b) and we use hyper- parameters Ag. Ad to adjust the sparsity of representation vi and va respectively.
Model Training In training, we adopt multi-tasking training paradigm with a mini- mini-batch mixup which means that we randomly sample from all tasks to compose training batch. We train the entire model on KILT for in-domain testing for superior performance. For cross-domain zero-shot retrieval, we freeze the parameters of the backbone network and only tune the parameters of our proposed components. The objective function we used can be described as:
min Cg(C) + )cCCPR(C)
(8)
P.0
(0,
where C T - T is the mixed data of different tasks, A, is a fixed pre-defined weight to control the regularization of CPR.
EXPERIMENTS
Benchmark Datasets We use publicly available retrieval datasets for evaluation, including KILT (Petroni et al., (02I) and BEIR (Thakur et al. (02I)) KILT is a benchmark for knowledge- intensive tasks that require retrieving additional knowledge from the wiki, we select all 7 datasets containing corresponding training samples to train our model in a multi-task setting. Specifically, we use variety of tasks including fact-checking (FEVER), entity linking (AY2), slot filling (zsRE), question answering (NQ, TQA, HoPo), and dialogue (WoW). KILT provides the provenances of all tasks in one wiki corpus, which enables us to train models with share passage encoder (Maillard et al. 202T). For BEIR, it is widely known zero-shot information retrieval benchmark and we employ it to evaluate the transfer learning ability provided by different methods. Also, we remove the datasets that are contained in KILT which results in 10 tasks including TREC-COVID (TC), NFCorpus (NFC), ArguAna (Argu), Touche-2020 (Touche), FiQA-2018 8 (FiQA), DBPedia (DBP), SCIDOCS (SeiD), Climate-FEVER (CFever) for fair comparison.
When evaluation on KILT, we adopt R- P-Precision as the retrieval metric which the primary metric used their paper (Petroni et al.) 2021). R- Precissio can be calculated as a where R is the number of Wikipedia pages msiide each provenance set and is the number of relevant pages among the top-R retrieved pages. For experiments on BEIR, Normalised Cumulative Discount Gain (nDCG) (Gysel& & de Rijke 2018) is reported to represent performances of different methods. Specifically, we utilize the official TREC evaluation tool (Gysel & de Rijke 2018) and compute
Evaluation Metrics
nDCG@ @ 10 for all datasets.
Experiment Setups We train our model on KILT in multi-task learning paradigm. Since our method can be combined with any transformer-bass neural retriever, we adopt SPLADEv2 (Formal
et al. 2021b) and DRP (Karpukhin et al., 2020) provided by the original paper as our backbone mode The learning rate of the backbone network and the modules of HYPER is set 2 x 10-5 by following Formal et al. (b02 and 1 x 10-3 selected from {10-1 10-2 10-3), respectively. The temperature t is set to e, A, is set to 0.3, Xa is set to 0. d, is set to 400 and d, is set to 100. The number of train epochs is set up to 10 epochs. both max document length and max query length are set to 512 to fit the task with a very long query, and the batch size is set to 256. For each query, we provide 1 positive sample and 19 negative samples for training. We set the sequence length of each basic prompt to 100 selected from (10, 50, 100). The X₂ and the number of shared basic prompts N
'Both models are pre-trained MS-MARCO Nguuen et al. 2016 and can provide superior initial performance KILT.
5

---

Published as a conference paper at ICLR 2023
Table Page-level R-precision on KILT. All models in the multi-tasking part are trained on 7 tasks of KILT, while the models in the zero-shot part are trained with the leave-one-out setting that leaves out the dataset used for testing in training. Model names that end with "zero" mean they are tested directly without training and the "-prefix" follows the model name means the corresponding model is
trained through prefix-tuning. indicates results from (Maillard et al. 2021).
Model
FEVER AY2
zsRE
NQ
HoPo
TQA
WoW
AVG
Multi- Tasking
DPR (Karpukhin et al. 2020
75.35
28.45
81.4
58.53
41.95
60.39
43.52
55,66
DPR-prehx
75.71
30.32
81.24
59.42
42.12
61.67
46.16
56.65
DPR-HYPER
76.35
31.80
81.79
60.83
42.67
62.91
46.35
57.53
SPLADE (Formal et al. 2021a
81.23
34.29
84,59
58.17
50.29
60.30
47.38
59.46
SPLADE-prehx
81.56
35.83
83.98
59.25
50.63
61.34
50.13
60.38
SPLADE-HYPER
82.17
38.52
84.23
60.41
51.24
62.76
49.84
61.31
Zero-Shot
BM25
50.13
3.47
66.43
25.83
43.95
29.44
27.
35.25
DPR-zero (Krnukhin 2020
$4.66
2.03
34.69
53.22
21.95
45.01
27.07
34.08
DPR (Karpukinn et ar. 2020
54.75
1.62
36.65
51.62
13.14
48.25
24.24
32.90
DPR-prehx
56.49
2.65
36.31
51.53
18.34
50.27
27.06
34.66
DPR-HYPER
59.13
3.56
37.03
53.31
19.13
52.24
29.06
36.21
SPLADE-Zero (Furmal et al.
2m
72.05
2.90
76.77
49.66
47.65
48.12
43.35
48.64
SPLADE (Formal et al. 2021a
73.42
2.32
80.75
47.78
29.96
51.63
38.73
46.37
SPLADE Trenx
75.12
3.21
81.24
48.36
40.25
54.42
43.55
49.45
SPLADE HYPER
78.39
82.16
50.02
41.38
56.28
46.81
51.44
are tuned and we finally select 0.1 and 20 respectively experiments are conducted on eight A100 GPUs.
We fix the random seed always to 42 and all 6
3.1 MAIN EVALUATION
Supervised and Zero-shot Performance on KILT We conduct both supervised and zero-shot experiments on KILT and the results are shown in Table [ Since our hyper-promen prompted training mechanism is applied to the lexicon-weighted retrieval method (e.g.. SPLADE), we name it as SPLADE-HYPER, Besides, we also test the effectiveness of our HYPER on dense retrieval methods (e.g., DPR) which results in DPR-HVE HYPER. First, we can easily find that SPLADE can provide superior performance than dense representation methods (e.g.. DPR) in terms of both the supervised and zero- shot settings on KILT. Notably, the performance gap is extremely significant in the zero-shot setting where dense retrieval methods (a.k.a., DPR) can only achieve comparable results with traditional BM22 while eexicon-weighted retrieval methods sigmificant outperform BM25 and dense retrieval methods. Second, compared with fine-tuning model entirely or prefix-tuning (Liu al., 2021b), our SPLADE-HYPER can obtain better performance on most tasks of the supervised setting, which demonstrates the superiority of our HYPER in sharing and transferring knowledge across different retrieval tasks or domains. Meanwhile, we can notice that HYPER can also improve performance even all unseen tasks, which reflects our method can effectively transfer learned knowledge from previous tasks and adapt to different types of queries across different domains. The experimental results of the few-shot setting (shown in Appedix A) can also further prove the effectiveness of the proposed HYPER.
Zero-shot Performance on BEIR
We also directly test the performance of our SPLADE-HYPER without tuning BEIR, which is widely known zero-shot IR benchmark. Following the most recent works (Xu et al. 2022 Wang et al. 2022), we compare our methods with varieties of methods,
including DoeT5 (Nouueira al. 2019a). ColBERT (Khattab & Zaharia 2020), DPR (Karpukhin
et al. 2020). ANCE (Xiong et al. 2020). GenQ (Thakur et al. 2021). TAS B (Reimers & Gurevych
2019 MoDIR (Xin et al.) 2021) and LapraDOR (Xu et al. 2022). Experiment results are shown in
Table 2 as we can see, our HYPER occupies the best performance on 4 of 9 tasks of BEIR, and we
also attain the best average performance. Besides, our method consistently outperforms the backbone
'The effects of different hyperparameters are investigated in section 3.2
"Our Code is available at : httpt//ttttuut//tg/////////t com/ o' 1 en/HypeR
*The observation is consistent with several previous studies MMillard al. 2021 Thakur al.
2021

---

Published as a conference paper at ICLR 2023
Table Zero-shot generalization evaluated on datasets of BEIR. "indicates results from Thakur
et al. 2021). indicates results from Xu et al. 2022).
Model
TC
NFC
FIQA
Argu
Touche
DBP
SeiD
CFever
SciF
AVG
BM25.
32.5
23.6
31.5
36.7
31.3
21.3
66.5
38,8
BM25+CE*
75.7
35.0
34.7
31.1
27.1
40.9
16.6
25.3
68.8
39.5
DocT5"
Nogueira al. 2019a
71.3
32.8
29.1
34,9
34.7
33.1
16.2
20.1
67.5
37.3
ColBERT [Khattab & Zaharia
2020
67.7
30.5
31.7
23.2
20.2
39.2
14.5
18.4
67.1
34.7
DPR
Karpukhin al.
2020
33.2
18.9
11.2
17.5
13.1
26.3
7.7
14.8
31.8
19,4
ANCE K1ong et al.
2020
65.4
23.7
29.5
41.5
24.0
28.1
12.2
19.8
50.7
32.8
GenQ!
Thakur et al.
2021
61.9
31.9
30.8
49.3
18.2
32.8
14.3
17.5
64.4
35.7
TAS-B
[Reimers & Gurevych 2019
48.1
31.9
30.0
42.9
16.2
38.4
14.9
22.8
64.3
34,4
MoDIR
Xin et al. 2021
2021
67.6
24.4
29.6
41.8
31,5
28.4
12.4
20.6
50.2
34.1
LapraDORT (Xu ct 2022
72.8
34.6
31.7
50.7
32.2
36.1
18.5
22.8
69.7
41.0
SPLADE
57.4
42.3
23.0
26.6
21.3
38.1
13.9
23.6
64.6
34,5
SPLADE-Prelix SPLADE-HYPER
73.0
32,9
34.7
49.3
25.2
41.8
15.3
22.3
70.5
40.6
79.1
33.6
35.1
50.1
27.4
43.6
15.6
23.7
70.4
42.1
Table 3: Ablation study on KILT.
Model
FEVER
AY2
zsRE
NQ
HoPo
TQA
WoW
AVG
HYPED
82.17
38.52
84.23
60.41
51.24
62.76
49.84
61.31
SPLADE-HYPER HYPR wio conditionl
80.61
36.65
83.24
59.07
50.60
61.37
49.48
60.15
SPLADE-HYPER w/o alignment D,
80.82
37.47
83.97
59.18
50.19
61.54
50.57
60.53
OPLADE-HYPER wi uniformity D
81.39
36.34
83.67
59.42
50.53
61.52
60.55
SPLADE-prefis
81.56
35.83
83.98
59.25
50.63
61.34
50.13
60.38
network SPLADE, which indicates HYPER can enable model to transfer learned knowledge across different domains better and thus improve the generalization ability of models. Moreover, our OPLADE-HYPER is better than SPLADE-prefixx which shows the superiority of proposed query- conditional prompt synthesizing and the strong ability of the dynamic parameterization to adapt different types of queries across the different domains.
3.2 ANALYSES
Ablation Study To verify the effectiveness of our proposed mechanisms, we propose several variants of our model for comparison. Specifically, to evaluate the effectiveness of QPS, we replace the generated attention score distribution with a fixed uniform distribution which results in a variant of our model without the attention module. Also, to verify the effectiveness of CPR, remove the alignment regularization and uniformity regularization to constitute the other two variants. Experiment results shown in Table 3 cause a significant decrease in performance. Comparison between HYPER and HYPER w/o query conditional indicates the attention module in QPS successfully generate suitable prompts for different queries and thus improve the performance. Moreover, we can observe removing either regularization terms of CPR results in degraded performances. Therefore, we can conclude that both alignment regularization and uniformity are crucial for enabling the effective query-conditional prompt generation to process different queries.
separately
see, removing any modules in mechanisms
Prompt Attention Similarity Task Similarity We con-
duct additional experiments to investigate whether the similari-
www
0.95
ties of prompt attention distributions can reflect the similarities between different tasks. Specifically, we calculate the similar- ities between the mean values of attention score distributions A belonging to the same task, and the results are shown in Figure 2 Obviously, there are two different groups of simi- lar attention score distribution, AY2 and WoW in the top-left corner and others the bottom-right, which are bounded by green lines. After reviewing the data, we find the two groups of datasets can distinguished by the lengths input queries. In particular, the lengths of queries of AY2 and WoW usually composed of multiple sentences while queries of other datasets only contain one or few sentences. This implies our mechanism can recognize the different lengths of different queries, which is prerequisite for dynamically adopting
HoPo
0.90
TEVER
0.85
NO
ZRE
MoWV
TOA
HoPP
SEVER
NO
oRC
Figure 2:
Similarities of attention scores distributions of different tasks.
7

---

Published as a conference paper at ICLR 2023
FEVER
AY2
HoPa
TQA
WoW
CPR
(b) w/ CPR
Figure 3: Visualization of attention weight embeddings for different tasks.
different methods to process different queries, such as extracting important information from long queries or predicting implicit relations in short queries. Moreover, we take a closer look and observe the sub- groups of attention score distributions annotated by blue lines. Comparing the queries of these two groups, we find the group composed of TQA, HoPo, and FEVER requires more inference skills than the group composed of zsRE and NQ. This implies our query-conditional module can even distinguish more subtle differences queries, which qualitatively reflects the effectiveness QPS.
Visualization of Prompt Attention Distribution To further confirm that CPR works expectation that prompts the distributions of attention scores to be uniform and aligned, we employ dimension reduction such as t-SNE to visualize them. To this end, randomly draw the 2000 samples from the test split of each dataset and then employ t-SNE to transform normalized prompt attention (a. (a. A) into 2D vectors. Experiment results shown in Figure 3] The comparison between w/ CPR and w/o CPR indicates that (I) uniformity regularization isolates attention score distribution dissimilar queries belonging different tasks, which helps avoid mode collapse. (2) alignment regularization prompts the attention score distributions of similar queries belonging to the same tasks become closer each other, which may improve the effectiveness of training prefix merged by specific pattern attention scores and thus benefits the model performance.
Performance vs Efficiency We further investigate the effi- ciency of our method since real-world applications not only require better retrieval results but also low retrieval latency. To measure the latency. we adopt the Query Per Second (QPS) as
page-lever R-precision KILT
70
SPLADE-HypeR
60
SPLADE
DDRRHHppRR DPR
metric and the higher QPS means more queries can be processed in time. We evaluate both dense retrievers and sparse retrievers with or without HYPER on KILT and experiment results are shown in Figure 4 As we can see, compared with backbone models, HYPER can obtain better performance and higher QPS which demonstrates the better efficiency of our method. Mean- while, although BM25 obtains better efficacy, neural methods outperform on retrieval metric significantly.
50
40
BM25
10
100
1000
QP5 on CPU Machine
Figure 4: Performance versus QPS (latency).
Dataset
Dataset
Dataset
Dataset KILT
Nvg Scores
42.
BEIR
51
42
BEIR
40-
50
40
38
20
40
10
10
10
10
10
N
A-
Figure 5: Effect of the number of shared basic prompts (N) and weights of CPR (Az). The two figures on the left vary N, while the two on the right vary Xe-
Impact of Parameters To better understand the effect of our method, we change the number of basic prompts (N) and the weight of CPR (Ae) and test these variants on both KILT and BEIR. Specifically, vary the number of shared prompts in {2, 5, 10, 20, 30, 50) with the weight of CPR fixed to O. Meanwhile, we vary the weights of CPR in (0.001, 0.005, 0.01, 0.02, 0.05, 0.1. 0.5) with number of shared prompt is fixed to 20. The average scores of different variants are shown in Figure [ As the number of shared prompts varies across different values, we can observe the performance increase and then decrease. This phenomenon implies more shared prompts can enable the model to learn more patterns exits in data while too many shared prompts suffered from the sparsity of representation space and thus results in insufficiently trained combinations of
8

---

Published as a conference paper at ICLR 2023
shared prompts. Similarly, we also find moderate weight of CPR can lead to the best performance. This indicates our proposed CPR can benefit the query conditional module and thus improve the performance while too large A causes the main objective loss Lg to be ignored.
RELATED WORK
Information retrieval can be generally defined searching relevant documents about short text as an input query. The major challenge of IR comes from the huge amount of candidate documents, which results in the matching function between query and documents having be extremely simple for high efficacy. Therefore, we need to seek superior encoding methods for queries and documents to improve the accuracy of retrieval. Traditional methods such as tf-idf and BM25 rely on term matching and build high-dimension, sparse vector (Yang et al., 2017 Robertson & Zaragoza 2009) for effective retrieval. Although they are effective across various tasks of different domains (Chen al. 2017: Yang et al. 2017), they fail to adapt to more specific tasks and outperformed largely
by the recently popular neural text retriever with sufficient training samples.
Neural text retrievers are based on pre-trained language models such BERT (Devlin et al. 2019)
and can be classified into two types, the dense-vector retrievers (Xiong et al. 2021 and sparse-vector retrievers. For dense-vector retriever (Karpukhin et al., 2020 Gao & Callan 2021.. it encodes both queries and documents into low-dimension vectors and then calculates the relevance scores between them. Although dense vector is more effective to conduct semantic retrieve (Lin & Lin 2022), the compressing process of texts may result in lost of information. In the contrast, sparse-vector retriever (Formal al., 2021b Khattab & Zaharia 2020) encodes both queries and documents into high-dimension sparse vectors and then calculate the concurrence (Nogueira et al. (219a) or top coordinate terms (Formal et 202Tb) of words. Therefore, it can achieve effective lexicon matching, and the varying amount of activating dimensions in vectors relieves the information lost in encoding, while it introduces an additional quantization process to avoid the unstable real-values of vectors.
Although neural retrievers can perform well with moderate amount of data, in real-world applica- tion, the data of target tasks could be considerably searce (Thakur et al. 202T). Hence, zero-shot and few-shot settings retrieval tasks receive more and more attention and various methods have been proposed to improve the model performance under this setting including unsupervised pretrained (Xu
et al., 2022 Wang et al.) 2022), data augmentation (Nogueira et al. [2019b] Thakur et al., 2021 Dai et al. 2022) and enhanced training (Reimers & Gurevych] 2019] Xiong et al. (020). However, there is only primary investigation of methods utilizing transfer learning, and still large room
for improvement. Technically, HYPER is similar to utilizing prefix tuning for IR tasks as /Tm et al.
(2022) and discrete prompt tuning for natural language tasks as [Sanh al. (2022), but goes beyond the comparison of existing mechanisms and focuses on generating dynamically query-conding cunry-conditional prompts, and enabling a neural retriever to process queries of different tasks uniformly. Besides, HYPER is inspired by HyperPrompt (He et al. 2022) that explores prompt-based task-conditioning of self-attention in Transformers. Nonetheless, HYPER dynamically generates the prompt according to every query itself rather than indispensably relying on the task-level information, which enables our model to obtain superior generalization and transferability. Moreover, HYPER employs the prefix-tuning method to utilize the dynamic prompts rather than concatenating prompts into the self-attention layer directly.
5 CONCLUSION
In this paper, we propose to process queries of different tasks uniformly regardless of the differ- ence in query format and varying priorities of query vectors. Specifically, present HYPER, hyper-prompted training mechanism that can be easily combined with any transformer-based neural retrievers. In HYPER, enable the uniform process queries, we propose Query-conditional Prompt Synthesizing (QPS) to dynamically synthesize different parameter combinations for different queries. Moreover, resolve the mode collapse problem of attention scores distribution in QPS, we propose Contrastive Prompt Regularization (CPR) to simultaneously learn representations of hyper-prompts and clustering for different types of queries across different domains or tasks. We conduct extensive experiments which demonstrate our methods can improve the in-domain and out-of-domain zero-shot retrieval performance of a neural retriever significantly. In-depth analyses reveal how our mechanism enables the uniform processing of queries.

---

Published as a conference paper at ICLR 2023
REFERENCES
Tiago Almeida and Sérgio Matos. Frugal neural reranking: evaluation on the covid-19 literature. In Proceedings of the Ist Workshop on NLP for COVID-19 (Purt 2) at EMNLP 2020, Online, December 2020. Association for Computational Linguistics. doi: 10.18655/1122222mmmoooooii
3. URL https: acl BtRLhtttt///claane http://accaantoooo...ooo htt.:pooww...oooo.ooooooooooooo. www.ooid1111111..
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Yinqiong Cai, Yixing Fan, Jiafeng Guo, Fei Sun, Ruqing Zhang, and Xueqi Cheng. Semantic models for the first-stage retrieval: A comprehensive review. CoRR, 10s/2103.0481,, 2021. URL
https://arxiv http://wwwwr//b//////////oe/////
04831
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes, Reading wikipedia answer open- domain questions. CoRR, abs/1704.00051, 2017. URL
00051
Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wor Bei Chang. Promptagator: Few-shot dense retrieval from examples. ArXiv, abs/2209.1175.. 11755, 2022.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Pupers), pp. 4171-4186 2019.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powereed conversational agents. CoRR, abs/1811.012411, 2018. URL
httt:://axxlv.oor///b//11111111 01241
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. SPLADE v2:
Sparse lexical and expansion model for information retrieval. CoRR, abs/2109.10086, 2021a. URL
www.eiii.rgga..//oooeoo / /
10086
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval, 2021b. URL
org/
abs/2109 10086
Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. CoRR, 005/2108.05544 2021. URL https////axxiw..r//aabao../
05540
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. REALM: retrieval- augmented language model pre-training. CoRR, abs/2002.08909) 2020.
Christophe Van Gysel and Maarten de Rijke. Pytrec eval: An extremely fast python interface to trec_eval. CoRR, abs/1805... 880.00197. 2018. URL htttp://aaxxiv.rrg/abba.. .0.5919777 01597
Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Prakash Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. Hyperprompt: Prompt- based lask-conditioning of transformers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 777788660,, PMLR, 2022. m1 mlr. wwtpss/swe//eeeeeeeeeeepp html
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, aby/1606.08415, 2016. URL http: httt:://arxivy http:///xxxwvo.../a/a.sss/////// org/abs.////.ooooooo...........t
08415
10

---

Published as a conference paper at ICLR 2023
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning, pp.
PMLR. 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnic Webber, Trevor CohnI. Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6769-6781, 2020.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Jimmy Huang. Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock. Ji-Rong Wen, and Yiqun Liu (eds.), Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 39 48, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Was Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Patrick S. H. Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for leoeeeeeniee NLP tasks. CoRR, abw/2005.114000 2020. URL URLtttts:://arit arxiv wtwp:/iww.rgg/bb//00...... 11401
Sheng-Chieh Chieh Lin and Jimmy Lin. A dense representation framework for lexical and semantic matching, 2022. URL https:/:: /2t.://2.....o//a///22 09912
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/2110.07600 2021a. htthhtts::/
arxiv. /bgg/bb//// 2110.1111100 07602
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to tne-tunnng universally across scales and tasks. CoRR, abs/2110.07602, 2021b. URL htttt::///aaat http:///axxxvv.ooaaa/////aa wtt.//rg//bba/oooo
07602
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, and Gargi Ghosh. Multi-task retrieval for knowledge-intensiin intensive tasks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACIJIJCNLP 2021, (Volume 1: Long Papers). Virtual Event, August 1-6, 2021, pp. 1098 11111111111111 Association for Computational I Linguisties, 2021. doi: (0.18653/v1/2021..o htw.:///222....logg..... URL httpss://oo..oog/111.BBB33 1tt.:////....11//////////0 accp://222cccccc....a. acl www.aaan...............
Christopher D. Manning and Hinrich Schütze. Foundations of statistical natural language processing. In SGMD, 2002.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and
Li Deng. Ms marco: A numan-generated machine reading comprehension dataset. 2016.
Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. From doc2query to doctttttquery. Online preprint, 2019a.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. CoRR, abs/1904.08377 2019b.
Biswajit Paria, Chih- Suah-Kuan Yeh, Ian En-Hsu Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. Minimizing flops to learn efficient sparse representations. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. httpttttt:://oeenetee ew. net wttt:/orrm..ass/y/eeee wet.

---

Published as a conference paper at ICLR 2023
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thore, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/202 -naacl-main.200
URL https
//aclanthology. org. 2021. naacl main.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.
CoRR, abs/1908. 10084, 2019.
aataaaass/8...0 10084
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and
beyond. Found. Trends Inf. Retr, 3(4) 2009. doi: 10.1561/1500000019 URL https
//doi. htt.l//gg....11/11111.1.0.. 1561/15000000119
Victor Sanh, Alhert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization.
The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL ltttps://openxeviee net/ forum? ?id 9Vrb9DOWI4
Asa Cooper Stickland and lain Murray. Bert and pals: Projected attention layers for efficient adaptation in multi-tast task learning. In International Conference on Machine Learning, pp. 5986- 5995.. PMLR, 2019.
Weng Lam Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Xingjian Zhang, Yuxiao Dong, Jiahua Liu, Maodi Hu, and Jie Tang. Parameter-efficient prompt tuning makes generalized and calibrated neural text retrievers, 2022. httt:htts:///aat
orgg abs/ 2207 07087
Nandan Thakur, Nils Reimers, Andreas Rücklé Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. CoRR, abs/2104.08663, 2021. URL httpp::/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/ ftbs//706603762.. 2017. URL
pettp://arxiie http::////rrrv.oooabbb/////// www.oog...........o..o... /000o///006600000
Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval, 2022. URL URLhttpss:/// / http::::://axxviccoo org. /2000.22200000 2207702578
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. ArXiv, abs/2005. 10242, 2020.
Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul N. Bennett. Zero-shot dense retrieval with momentum adversarial domain invariant representations. CoRR, abs/2110.07581, 2021. URL http:/ttt////iiiiii wrg. waw///////////////////../e.eeeee /1100 07581
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. CoRR, abs/2007 000000007000888, 2020.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httt:p:://ppnzeeiieee net/ for um? (dzzeFrfgr) fg
12

---

Published as a conference paper at ICLR 2023
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Laprador: Unsupervised pretrained dense retriever for zero-shot text retrieval. ArXiv, abx/2203.06169, 2022.
Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Enabling the use of lucene for information retrieval research. In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White (eds.), Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pp. 1253-1256, 2017.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018.
October 31- November 4, 2018, pp.
13

---

Published as a conference paper at ICLR 2023
A FEW-SHOT EVALUATION ON KILT
In Table 4 we continually train models a new task with a few-shot setting. As we can see, our methods consistently outperform several strong baselines, which demonstrates that our method benefits the model more quickly adapting to the different tasks in the same domain by sharing knowledge among tasks.
Table 4: Page-level R-precision on KILT the few-shot setting.
Target Data
NQ
TQA
#Instance
192
384
576
1152
2304
192
384
576
1152
2304
SPLADE
48.19
48.32
48.55
48.97
49.56
51.97
51.32
51.66
53.53
53.18
SPLADE-prefix
49.07
49.40
50.10
50.51
50.42
54.62
54.01
53.37
55.48
56.23
SELADE-HYPERR
50.27
50.24
51.13
51.43
51.96
56.56
55.81
55.07
57.68
58.24
Target Data
FEVER
zsRE
#Instance
192
384
576
1152
2304
192
384
576
1152
2304
SPLADE
73.42
73.61
73.75
74.01
74.72
80.83
81.13
81.24
81.45
81.78
SPLADE-prefix
75.12
75.19
74.86
75.74
76.31
81.30
81.72
81.65
82.09
82.36
78.37
78.44
77.94
78.55
79.08
82.19
82.58
82.63
82.69
83.44
B COMPARISON OF THE TASK-SPECIFIC FINE-TUNING MODEL
We also test the performance of directly fine-tuning the SPLADE on each task of KILT, which results in 7 different retrievers (SPLADE-FT), The results are shown in Table We can find that SPLADE-FT can achieve significantly better AVG score than SPLADE-MT. although multi-task training can bring improvement to 4 out of tasks (a.k.a FEVER, ZsRE, NQ. and TQA). Besides, through incorporating the proposed HYPER, SPLADE-Hypet can achieve comparable AVG score with SPLADÉ-FT and even outperform SPLADE-FT on out tasks (a.k.a. FEVER, zsRE, NQ, TQA, and WoW). Notably, SPLADE-FT train separate models for each task which results in roughly 6 times more parameters than our model.
Table 5:
Comparison of the task-specific fine-tuning model on KILT. SPLADE-prefix means the model trained through prefix-tuning.
Model
FEVER
AY2
zsRE
NQ
HoPo
TQA
WoW
AVG
SPLADE-FT
80.20
55.02
83.97
57.00
51.29
55.40
48.69
61.65
SPLADE-MT
81.23
34.29
84.59
58.17
50.29
60.30
47.38
59.46
SPLADE-prefh
81.56
35.83
83.98
59.25
50.63
61.34
50.13
60.38
SPLADE-HYPER
82.17
38.52
84.23
60.41
51.24
62.76
49.84
61.31
C EFFICIENCY OF QUERY ENCODING
We further investigate how the length of prefixes influences the computation cost of query encoding
and retrieval performance. To this end, we vary the size of the prefix in 15,10.20,50 50, 100, 200} and record both the average time of encoding query and retrieval performance on KILTT. We compare our method with Prefix-tuning and the results are shown in Figure 6 The inference time is measured on machine with Intel Xeon CPU E5-2678. As we can see, our HypeR obtains better average retrieval scores than Prefix-Tunine for all different prefix lengths, and both Prefix-Tuning and HypeR achieve the best performance when the prefix length is 100. Notably, our HypeR costs 4% and 25% more encoding time than Prefix- Prefix-Tuning Tuning when the prefix length is 10 and 100, respectively. We have put the results as a figure in the appendix of our revised manuscript. Thank you again for your constructive suggestions.
14

---

Published as a conference paper at ICLR 2023
61.5
0.150
HypeR_Sooe Sccre
61.0
Prefix
0.125
HypeR_Teee Time
Avg Score
60.5
Prefix Time
0.100
Time
60.0
0.075
0,050
59.5
0.025
59.0
10
20
50
100
200
Prefix Length
Figure 6: Efficiency compraing between HypeR and Prefix- Tuning. The multi-tasking fine-tuning can be viewed as origin.
D CASE STUDY
To provide an intuitive understanding of our methods, we show some similar and dissimilar queries based prompt attention distributions in Table Specifically, belonging to different tasks as base queries from the collection test data of all KILT datasets. Then, for each base query, randomly sample 2 queries from the queries with top 5% similar attention scores. Besides, we also randomly sample 2 queries belonging to different datasets from the queries with the lowest 5% similar attention score, since we find there too many trivial candidates from the same datasets.
randomly sample 3 queries
Table 6: Queries with similar and dissimilar prompt attention distributions.
Query Type
Source Task
Query Content
Case
Which film was made more recently, The Diplomat or Rien que les heures?
Query
Hotpoiqa
Which movie was released more recently, Waking Sleeping Beauty or Mars Needs Moms?
Similar Query
Hotpoiqa
Similar Query 2
Hotpotqa
Who directed film more recently, Don Bluth or Raoul Walsh?
Action Performance to acquire firms [START_ENT TEMPE [END ENT], Ariz. 1996-12-06 Action Performance Cos Inc said Friday has agreed to acquire Motorsport Traditions Ltd and Creative Market- ing & Promotions Inc for about $13 million cash stock. The two firms to be acquired have about $25 million in annual revenues from the design. manufacture and sale and distribution of licensed motorsports products. The deal is expected to close by the end of the year subject to due diligence and other customary closing conditions.
Dissimilar Query
Aidayago2
love American Football, the first game was played between Rutgers and Princcton 1869. Wow had no idea was that recently played! Yep. can actually be traced to carly versions of rugby football also. Who is your favorite team. Probably the New England PAtriots. I always had thing for Tom Brady lol. Did he not deflat footballs something year? ?
Dissimilar Query 2
WoW
Case
Do you shop online for clothes? Yes quite often. allows directly buy goods from a seller over internet without having to leave the comfort of my home. What about you? Yeah same love visiting websites of different retailers directly to see product availability and the best prices. Yeah too. use amazon lot to buy stuff. Like even this computer!
Base Query
WoW
15

---

Published as a conference paper at ICLR 2023
Online shopping is the better experience choose the the product desire. Yes! much prefer shopping online at home on the Internet rather than fighting crowds in an actual store. what did you often purchase? Anything and everything! Housewares, clothing, electronics, you name it! like the convenience of browsing stores my laptop, tablet com- puter and smartphone! yes, it includes lot of choices I especially like the functionality of websites like Amazon for shopping. The have great search feature that allows me to find specific models, brands and items. Such website, improved a lot for customer convenience.
Similar Query 1
WoW
like to shop online, probably bit too much. Lol me too! What's your favorite online store? I love to shop amazon. So you shop online lot too? Where do you shop? Amazon well. also shop a lot online Old Navy and Gap. They have great sales. they really? mainly use only amazon. Do you use Amazon Kindle, the series of e-readers designed and marketed by Amazon. No have never used kindle. My dad has one of those though and looks cool if you love books.
Similar Query 2
WoW
Dissimilar Query
FEVER
Hrithik Roshan was a film star.
When did the theatre open that has Valery Abisalovich Gergiev as it's artistic director?
Dissimilar Query
Hotpotqa
Case 3
Base Query
FEVER
Reliance Industries only works in textiles.
Similar Query 1
FEVER
Sanjjanaa works in the Telegu film industry.
Similar Query 2
FEVER
Rakul Prect Singh mostly works in the Telugu film industry.
Have you ever read anything by John Grisham? Yes, have read his very first novel "A Time to Kill" which was published in June 1989 after he took four years to write it! I havení read anything by him but I remember the movies for both a time kill and the firm.
Dissimilar Query
WoW
Multinational commander going back to east Zaire Jonathan Wright NAIROBI 12-06 The Canadian general in charge of multinational force for eastern Zaire said on Friday was going back to Zaire for more information about the plight of about 165,000 Rwandan refugees adrift in the countryside. Licutenant- conference in Nairobi his main concern was for large group of about 150,000 refugees living off the land in valley about 65 km (40 miles) west of the eastern city of Goma.
Maurice Baril told news
he decided it was necessary and safe for the aircrew, he would not hesitate to order airdrops of food for the refugees, even against the wishes of the government in Kinshasa and the [START ENT]] Zairean [END_ENT] rebels who control much of eastern Zaire, he said. " Tomorrow intention is to go across into eastern Zaire and try to find out for the second time what the situation is on the ground," he said. General Baril saw rebel leader I Laurent Kabila in Goma last week but the rehels told him the crisis already gone home. The rebels deploy on the ground for fear it might help the Zairean army regain control of the area.
Dissimilar Query 2 Aidayago2
going into Rwanda and my
because most of the Rwandan refugees have want the multinational force
16


Reviews:
[{"review_id": "hAeDEIUpd62", "reviewer": "Reviewer_tLxh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors investigate the out-of-domain generalization problem in neural retrieval. They propose a hyper-prompt training mechanism and contrastive prompt regularization to this problem. The hyper-prompt training predicts a distribution over the predefined prompt parameters given the query, and get the task-adaptive prompt embeddings as prefix for the each layer of query encoder to augment the query representation. Contrastive prompt regularization solves the mode collapse problem of attention score distributions. Experimental results show that the proposed approach achieves slight improvement in the multi-task training setting but leads to significant improvement in zero shot setting and on out of domain tasks.", "summary_of_the_review": "In summary, I feel that this is a good work in general, though there are some minor concerns on the definition of “out-of-domain generalization”.", "strength_and_weaknesses": "### Strength\n- The propose approach is novel on designing a hyper-prompt training  approach for neural retrieval out-of-domain generalization.\n- The authors conduct comprehensive experiments and analysis to justify the effectiveness of every component of the proposed approach (hyper-prompt, contrastive prompt regularization). Results are shown on both multi-task and zero shot settings.\n\n### Weakness\n- This work claims to target the out-of-domain generalization for neural retriever but experiments only justify the out-of-dataset generalization. Most retrieval tasks are from the Wikipedia domain.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_UQS4NqKmh", "reviewer": "Reviewer_1Ry7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes HYPER, a mechanism for transfer learning among retrieval tasks in different domains and datasets. For each query, it first dynamically generates a hyper-prompt to synthesize the query by attending to a given number of shared prompts, then encodes the synthesized prompt into prefixes, and employs the prefix-tuning method to train the retriever model in a multi-task setting. Contrastive learning is employed to cluster synthesized prompts based on the datasets they come from. Experiment results show the efficacy of HYPER, especially on 0-shot in-domain and cross-domain retrieval settings.\n", "summary_of_the_review": "The paper addresses a well motivated problem effectively with recent advances in prefix and prompt tuning, and with innovations on conditioning prompts on the query-level instead of task-level. The claims are backed up by extensive experiments and analysis on recent public datasets compared with SOTA systems. ", "strength_and_weaknesses": "Pros:\n- The paper presents an effective approach to address a practical problem of domain adaptation in retrieval, following on the recent popular direction of prompt tuning and prefix tuning.\n- The approach is novel. Compared with previous related work (e.g. Hyper-Prompt), HYPER addresses retrieval task across different domains/datasets, so it dynamically generates/conditions distinct prompts on the query-level instead of task-level.\n- The approaches, model architecture, formulas, and experiments are sound and solid, very clearly explained, and easy to follow. The paper is well-organized and written. \n- Experiments are conducted on popular public datasets of KILT and BEIR, and are compared against the recent SOTA systems in the retrieval field. The results and conclusions are convincing.\n- The discussions and analysis are detailed and insightful. The analysis around prompt attention similarity and task similarity, together with visualization, is interesting and informative.\n\nCons:\n- The approach is employed on query encoder. It would be interesting to explore and discuss about the effect of transfer learning on the document encoder as well. Also, I am a little curious about employing HYPER in retrieval models where the query encoder and document encoder share a large part of parameters, for example, ColBERT.\n- A small suggestion: It would be interesting to add some query examples (maybe in Appendix) with similar and dissimilar prompt attention scores.\n\n\nQuestion:\nPage 3, when describing how to obtain the query representation Q, should it be generalized to any retrieval model's query encoder? Or, maybe mention that this description is specific to a particular model, and the other models follow in a similar way?\n\nMinor editing errors:\nPage 7, 'we calculate the mean...' -> should it be 'we calculate the similarities between the mean...' ?\nPage 9, 'considerably scare' -> 'considerably scarce'\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Pt71k1qE-V", "reviewer": "Reviewer_X25B", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed the idea of learning a dense retriever with a list of learned prefixes. The model attends to prefixes to compute query-dependent prefixes, which will be used to compute final query vectors for retrieval. In order to make the prefixes non-uniform, the paper additionally introduced a contrastive prompt regularization loss.", "summary_of_the_review": "The proposed methods is well motivated and strongly justified by the experiments. The only drawback is that it's not very clear where does the improvement come from (see my question above). But overall, it is a nice paper.", "strength_and_weaknesses": "The idea of computing query-dependent prefix to compute query vectors for retrieval is an appealing idea. The results shows that having learned prefixes does improve models' performance in many downstream tasks. The retrieval performance is also improved in zero-shot setting. Overall, I think this is an interesting paper. Thanks for the excellent work.\n\nMy main concern is the additional computation cost coming from the added prefixes. As discussed in the paper, the length of the prefix is 100. For many retrieval tasks, queries are short (sometimes 20~30 tokens). The added prefix tokens may increase the computation cost by ten times or more. Have you tried measuring the efficiency of your model and how it compares to the baselines?\n\nIf the improvement comes from having more parameters, why is it different from increasing the size of the model? Or, can you append 100 special tokens to queries and let models learn their embeddings.\n\nA few more questions:\n1. Does the number of task affect the size of the prefixes? Do you need more prefixes or longer prefixes if there are, say hundreds of tasks?\n2. Are model's parameters in SPLADE jointly finetuned with parameters of the prefixes?\n\nThere are many paper come out recently in few-shot and zero-shot retrieval. You may want to add some of their numbers to your paper, e.g. Promptagator [1]. The numbers may not be directly comparable, but it's good to have.\n\n[1] Promptagator: Few-shot Dense Retrieval From 8 Examples\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "85zDHjGHZF6", "reviewer": "Reviewer_BVqR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a hyper-prompted training mechanism for large-scale text retrieval across tasks of different domains. ", "summary_of_the_review": "Overall I found this paper clearly written with strong motivation and has extensive and promising experiment results, but the novelty contribution is unclear. Thus I think this paper as marginally above the acceptance threshold and highly recommend the authors to add the discussion and necessary baselines into this paper.", "strength_and_weaknesses": "Strength:\n1. The problem of cross-domain text retrieval that this paper studies is important.\n2. The paper is well written with clear motivation, method description, and experiment setup. \n3. The empirical results are promising and comprehensive, covering two popular benchmarks KILT and BEIR, two representative retrieval methods DPR and SPLADE, and two settings supervised and zero-shot.\n\nWeakness:\nThe main weakness of this paper is the novelty of the proposed HYPER mechanism. It lacks the comparison with other transfer learning mechanisms such as the ones mentioned in [1] and [2]. Although those papers do not conduct experiments on retrieval, it's easy to extend them to the retrieval setting. It seems to me that HYPER is also very general and can be applied to other downstream settings like GLUE. Thus it's necessary to have a discussion with other transfer learning mechanisms. \n\n[1] He, Junxian, et al. \"Towards a unified view of parameter-efficient transfer learning.\" arXiv preprint arXiv:2110.04366 (2021).\n[2] Asai, Akari, et al. \"Attentional Mixtures of Soft Prompt Tuning for Parameter-efficient Multi-task Knowledge Sharing.\" arXiv preprint arXiv:2205.11961 (2022).\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster