Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers
Authors: Weng Lam Tam
Xiao Liu
Kaixuan Ji
Lilong Xue
Jiahua Liu
tao li
Yuxiao Dong
Jie Tang
Venue: EMNLP 2023 Findings (2023)
Keywords: information retrieval
prompt tuning
generalization

Abstract: Prompt tuning attempts to update few task-specific parameters in pre-trained models. It has achieved comparable performance to fine-tuning of the full parameter set on both language understanding and generation tasks. In this work, we study the problem of prompt tuning for neural text retrievers. We introduce parameter-efficient prompt tuning for text retrieval across in-domain, cross-domain, and cross-topic settings.  Through an extensive analysis, we show that the strategy can mitigate the two issues---parameter-inefficiency and weak generalizability---faced by fine-tuning based retrieval methods. Notably, it can significantly improve the out-of-domain zero-shot generalization of the retrieval models. By updating only 0.1% of the model parameters, the prompt tuning strategy can help retrieval models achieve better generalization performance than traditional methods in which all parameters are updated. Finally, to facilitate research on retrievers' cross-topic generalizability, we curate and release an academic  retrieval dataset with 18K query-results pairs in 87 topics, making it the largest topic-specific one to date.

Parameter-Effcient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers
Weng Lam Tam Xiao Liu"*, Kaixuan Jii Lilong Xue Xingjian Zhangi,
Yuxiao Dongi, Jiahua Liu-, Maodi Hu², Jie Tang
TTsinghua University +Meituan eetereeeeeeeeeeeeeeeeeeeeee jietangetsinghua. edu.
[rainatamm7884, www.lii9)@gaaaaaiiia.. com,
Abstract
ParamottesEiiciiney
Zero-shot Generalization
In-Domain
Cross-Doomain (BEIR)
Cross-Topic (OAG-QA)
Prompt tuning attempts to update few task- specific parameters in pre-trained models. It
$2.5
80.6
OpenGA
80.0
50.9
Sciacc
41.8
Geometry
19.9
34
332
has achieved comparable performance to fine- tuning the full parameter set on both lan- guage understanding and generation tasks. In this work, we study the problem of prompt tun- ing for neural text retrievers. We introduce parameter-efficient prompt tuning for text re- trieval across in-domain, cross-domain, and cross-topic settings. Through an extensive anal- ysis, we show that the strategy can mitigate the two issues paraa-trrrre----eeee parrameter-mmfffciieng inefficiency and weak generalizabilit--faced by fine-tuning based re- trieval methods. Notably, it can significantly improve the out-of-domain zero-shot gener- alization of the retrieval models. By updat- ing only 0. % of the model parameters, the prompt tuning strategy can help retrieval mod- els achieve better generalization performance than traditional methods in which all parame- ters updated. Finally, facilitate research on retrievers' cross-topic generalizability. we curate and release an academic retrieval dataset with 18 18K query-results pairs in 87 topics, mak- ing it the largest topic-specific one to date.
Top-28
$5.0
Nbccog
cunoed
100%
6.1%
24.8
31.8
15.4
20
30.1
Fine-taning (fully taned)
PR learning (0.1% parameters tuned)
D
Confidence Calibration
Query-Jength Robustness
Figure 1: For DPR (Karpukhin et al., 2020) trained on OpenQA datasets, PE learning (e-g., P-Tuning v2) offers parameter-efficieng and improved generalization thanks to better calibration and query-length robustness.
et al., 2021), i.e., 100x larger than those used pre- viously (Karpukhin al., 2020).
Mcanwhile, an increasing number of studies have focused on the parameter-efficieng and gen- eralizability challenges of neural methods. In terms of tices (Karpukhin et al., 2020) rely on fine-tuning dual encoders for queries and documents separately and thus cause parameter redundancy (Geigle et al., 2022). Furthermore, fine-tuning the full parameters of pre-trained retriever for multi-lingual (Litschko et al., 2022) cross tross-topic settings can also result in parameter-innffficinnyy Moreover, despite neural approaches' in-domain outperformance, it has been found that their cross-domain generalization can- not match the simple BM25 method (Thakur et al., 2021). Consequently, these issues pose challenges to develop cost efst-effee neural text retrievers.
the common prac-
1 Introduction
Seeking for relevant texts has been a fundamental problem for broad range of natural language pro- cessing (NLP) applications such as open-domain question answering (Chen et al., 2017), retrieval- augmented language modeling (Guu et al., 2020), and fact verification (Thome et al., 201 8). Its re- cent progress has been dominantly favored by the neural approaches (Karpukhin et al., 2020; Khat- tab and Zaharia, 2020), especially the large-scale pre-trained language models with ever-growing pa-
Recently, parameter-efficien (PE) transfer learn- ing, including prompt tuning (Li and Liang, 2021; Liu et al., 2021c; Lester et al., 2021), adapters (Houlsby et al., 2019), and hybrid meth- ods (Hu et al., 2021; Zaken al., 2022), is proved to achieve comparable performance to fine-tuning on language understanding and generation tasks by employing very few task-specific tuning parame- ters. Inspired by this progress, we propose to study whether and how PE learning can benefit neural
rameters. For example, a recent study attempts to leverage models up to 10 billion parameters (Ni
'The first two authors contributed equally. 'Code and data are at ttttps://iiie thwb. COm/THUDM/ P- e-tuning-vie http:teee/min///-Reeeee

---

text retrieval in terms of both parameter-efficinncy and generalizability.
2 Related Work
In this work, we systematically examine a line of mainstream PE methods in in-domain, cross- domain, and cross-topic settings. As expected, most PE approaches perform comparably to fine- tuning on in-domain retrieval. Excitingly, PE prompt tuning (Li and Liang, 2021; Liu et al., 2022) also encourage neural text retrievers general- ize on the cross-domain benchmark BEIR (Thakur al., 2021) and OAG-QA-2 new multi-discipline academic cross-topic retrieval dataset we con- structed. For example, by simply replacing fine- tuning to the parameter-efficient P-Tuning v2 (Liu et 2022), we achieve relative gains ranging from 3.5% to 105.0% on out-of-domain BEIR datasets. Through empirical analyses, attempt to pro- vide an understanding of the better generalization brought by PE prompt tuning. First, PE prompt tun- ing can help empower the neural model with better confidence calibration, which refers to the theoreti- cal principle that a model's predicted probabilities of labels should correspond to the ground-trun truth correctness likelihood (Guo et al., 2017). Second, it encourages better performance on queries with different lengths from in-domain training, demon- strating PÉ methods' generalization capacity to out-of-domain datasets.
Neural Text Retrieval. Text retrievers tradition- ally rely sparse lexical-based inverted index to rank candidate documents containing query terms (e.g., TF-IDF and BM25). They benefit from the simplicity but often suffer from the lexical gap (Berger et al., 2000). Recently, neural text retriev- ers, including dense retrievers (Karpukhin et al.. 2020; Xiong et al., 2021; Hofstätter et al., 2021), late-interaction models (Khattab and Zaharia, 2020; Santhanam al., 2021), and hybrid or re-ranking models (Nogueira et al., 2019; Wang al., 2020b), becomes popular as they can capture the semantic- level query-document similarity thanks to the ad- vance of pre-trained language models (Han et al.. 2021).
Generalization in Text Retrieval. The weaker generalizability of neural retrievers compared conventional lcxical ones has recently arouse con- cerns in the community (Liu et al., 202 Chen et al., 2022), and it results in BEIR, het- crogeneous cross-domain generalization bench- mark (Thakur et al., 2021). While recent works no- tice and employ ideas like bigger pre-trained mod- els (Ni et al., 2021) unsupervised pre-training on large corpus (Izacard et al., 2021) improve scores on BEIR, few of them focus on studying better transferring strategies based on existing ar- chitectures and datasets for out-of-domain general- ization.
To summarize, this work aims to advance the neural text retrievers from three aspects:
Problem: propose leverage PE learn- ing for neural text retrievers with much fewer tuning parameters. We demonstrate that PE prompt tuning can not only perform compara- bly to fine-tuning in-domain but also enable neural retrievers to achieve significant gener- alization advantages over fine-tuning on cross- domain and cross-topic benchmarks.
Parameter-Effiiieet (PE) Learning. Sizes of pre- trained language models are soaring up (Brown et al., 2020), causing great challenges to tradi- tional task transfer based on full-parameter fine- tuning. A recent focus has been on the emerged PE transfer learning, including prompt tuning (Li and Liang, 2021; Liu al., 2021c; Lester et al., 2021), adapters (Houlsby et al., 2019), and hybrid methods (Hu al., 2021; Zaken et al., 2022). They employ very few tuning parameters to achieve fine- tuning comparable transfer performance. Despite abundant research made on problems like language understanding (Houlsby et al., 2019; Liu et al., 2022) and generation (Li and Liang, 2021), how it will impact retrieval remains under-explored.
Understanding: we provide an understand- ing of PE learning's outperformance across domains and topics. Our analysis sug- gests that its generalization advantage largely comes from its confidence-calibratee predic- tion and query-length robustness.
we construct OAG-QA, an academic paper retrieval dataset curated from real-world questions and expert answers, test retriev- ers' cross-topic generalizability. With 22 dis- ciplines and 87 topics, OAG-QA is the largest tne-ggained topic retrieval dataset to date.
Dataset:
3 Challenges in Neural Text Retrieval
The neural text retriever, which leverages pre- trained language models, c.g., BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), as the

---

backbone, has significantly mitigated the lexical gap (Berger al., 2000) in text retrieval and be- come standard component for many NLP applica- tions (Chen et al., 2017; Guu et al., 2020; Petroni et al., 2021). It consists of several different cate- gories and in this work we focus on the following two dominant ones.
ers to achieve good results, it results in sub- stantial parameter redundancy from two aspects. First, training dual-encoders double the size of the parameters to be tuned. The improv- ing strategies, such as parameter sharing (Yan et al., 2021; Geigle et al., 2022), have to sac- rifice the retrieval performance. Second, the cross-lingual (Litschko et al., 2022) and cross- domain (Thakur et al., 2021) transfer may require additional full-parameter tuning on each of the individual tasks and consequently increase the number of parameters by several times.
Dense Retriever (Karpukhin et al., 2020): Dense retrieval learns dual encoders to map queries and documents into dense vector space such that relevant pairs of queries and documents have shorter distances. It usually adopts the inner-dot product for the sake of efficiency as sim(g. = Eq(g)" Ep(p) where F.(+) and Ep(-) are dense encoders that map queries and documents to dense vectors, respectively. A rule- of-thumb training objective is the Noise Con- trastive Error (NCE), which takes the query % and its relevant (positive) document pj and irrelevant (negative) documents Pi,
Weak Generalizability: Though neural retriev- offers advantages domain datasets, e.g., OpenQA datasets (Karpukhin et al., 2020), some of particulaly dense retrievers-Camme generalize well to zero-shot cross-domain bench- marks (Thakur et al., 2021). However, the zero- shot setting is widely adopted in downstream sce- narios, as constructing retrieval training datasets with annotations could be outrageously expen- sive. Such challenge also broadly connects to the generalizability of neural networks.
CCE - llg
esim(gep() Σj= C xim( (i
(1) Late-Interaction Retriever (Khattab and Za- haria, 2020): ColBERT combines the strengths of the bi-encoder and cross-encoder to encode the the query and document at a finer granularity into multi-vector representations. The relevance is es- timated by using the rich yet scalable interaction between the query and document representations. Specifically, the model produces an embedding for every token in queries and documents and compute the relevance using the sum of maxi- mum similarities between vectors of query to- kens and all document tokens as:
In this work, we aim explore the solutions for addressing the above challenges in neural text retrieval. Specifically, we focus on the parameter- efficient transfer learning, which has offered al- ternative strategies for the downstream usage pre-trained models in natural language processing.
Parameter-Efficient Transfer Learning
We introduce the parameter-efficient transfer learn- ing (PE learning) framework and notable tech- niques. Diffcrent from fine-tuning (Devlin et al., 2019), which updates the full parameters of pre- trained models for each target task, PE learning aims achieve comparable performance to fine- tuning by tuning only a small portion of parameters per task (Houlsby et al., 2019; and Liang, 2021; Liu et al., 2022).
sim(q.p)
Σ max EY Eg.
(2)
JEIl Fall
&E||E,11
where Eg and Eg are the sequences embed- dings for query and document d.
Challenges. Neural retrieval approaches, such as dense retrievers and late-interaction models, have achieved outperformance over lexical ones typi- cal open-domain question answering datasets, e.g.. NaturalQuestions (Kwiatkowski et al., 2019). How- ever, recent studies (Litschko et al., 2022; Thakur et al., 2021) unveil some of their inherent limita- tions, posing the following challenges:
4.1 Transformers
The success PE learning largely takes advantages of the Transformer architecture (Vaswani al., 2017). Transformers are composed of stacked lay- ers, each containing a multi-head attention module and a feed-forvee forward network (FFN). The attention function can be written as:
Parameter Inefficiency: Though the full- parameter fine-tuning empowers neural retriev-
OKT
Autention()) softmax
V
(3)
Vihy

---

(CLS) tosen
Proope
[MASK]
(CLS) token [MASK]]
Layer Norm
W
Layer Norm
+b
ReLU
W
Adapter
PLM Module
Wx+b
PLM Module
PLM
PLM
(at) Adapter
(b) BiFit
(c) Lester el al P-Tuning
(6) Prefix Turing & P-Tuning v2
Figure 2: The illustration of four parameter-efficiet methods. The PLM module represents a certain sublayer of PLM. c.g,, the attention or FFN. The components in blue are frozen and the yellow ones are trainable.
where the query Q, key and value V are:
where h2 is the trainable prompt (CF. Figure 2 (b)).
(z) (9 + (g.k,w)
(4)
Prefix-Tuning (Li and Liang, 2021) & P-Tuning v2 (Liu et al., 2022). Prefix-tuning concatenates trainable key and value embeddings of the attention to the prefix on each layer of the language models. Specifically, given the original key vectors K E R and value vectors V E Rixd, the trainable vectors Pey Pa are correspondingly concatenated to K and V. The computation of an attention head becomes:
The multi-head attention performs N heads in par- allel and concatenates their outputs to form the input to FFN where f is an activation function:
FFN(x) [=WW + bi) + b2 (5)
Different PE learning methods attempt modify different modules of Transformer achieve pa- rameter efficiency.
head, (a) Attention(xW")) (PP : (,
V
4.2 Parameter-Efficient Learning Methods
(8) Here the superseript (t) refers to the part of the vectors that correspond to the i-th head. It has been empirically proved comparable to fine-tuning wide range of downstream tasks, including text generation (Li and Liang, 2021), natural language understanding (NLU) and sequence labeling (Liu et al., 2022).
We introduce several emerging learning meth- ods. Figure illustrates the technical differences between them.
Adapters (Houlsby et al., 2019; Pfeiffer et al., 2020). The adapter inserts small modules between Transformer layers, which forms as a bottleneck to limit the amount of parameters in the format of:
Since the retrieval task is more related to NLU, we employ P-Tuming v2's s's implementation, which makes several optimizations on top of prefix-tuning (Cf. Figure 2 (c)).
h - h +
+f f(h down WdoWdown)) W
(6)
where h is the input, W Wdown& Rdxr and Wap Rexd are project matrices, and f(-) is the activation function (Cf. Figure (a)).
In-Domain Parameter-Effiiieng
BitFit (Zaken al., 2022). Each Transformer layer consists of self-attention, FFN, and Layer- Norm operations, all of which have certain bias terms as shown in Eqs 4 and 5. Bit-fit proposes to only tune the bias terms b(-) of the Transformer (Cf. Figure (d)).
In this section, we describe the data and settings we used for the in-domain OpenQA experiments and evaluate the retrieval performance of the parameter- efficient methods introduced above.
Datasets. We follow (Karpukhin et al., 2020) to use five open-QA datasets and their train/test/- valid splits: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), We- bQuestions (WQ) (Berant et al., 2013), Curat- edTREC (TREC) (Baudis and Sedivý, 2015) and SQuAD v (Rajpurkar et al., 2016). We follow (Karpukhin et al., 2020) to use the split text blocks from the English Wikipedia dump the retrieval candidate set, which contains 21,015,324 passages.
Lester et al. & P-Tuming (Liu et al., 2021c). This approach inserts trainable continuous prompts to the input sequences of the Transformer. Given a PLM, e(-) is the input embedding function that maps input tokens to input embeddings. For tem- plate T [[Pos] æ, [P++1m], y} where is the context and y the target, e.g., the [MASK] token, the model's inputs are:
(ho, h,, e(x), h441, kn, e(y)}
(7) Settings. We evaluate the in-domain performance

---

Table 1: In-domain parameter-efftiinnyy The retrievers are multi-task fine-tuned or PE trained on 4 OpenQA
datasets (except for SQuAD", which is excluded from Avg.) following the setting in (Karpukhin al., 2020).
Top-20
Top-100
Retrievers
#Params
Avg.
NQ
Trivia
WQ
TREC SQUAD"
Avg.
NQ
Trivia
WQ
TRFC SQUAD"
BM25
63.0
59.1
66.9
55.0
70.9
68.8
76.4
73.7
76.7
71.1
84.1
80.0
Finc-tuning
100%
800.6
79.4
78.8
75.0
89.1
51.6
86.9
86.0
84.7
82.9
93.9
67.6
P- P-Tuning v2
0,1 1%
80.6
79.5
78.8
75.2
88.8
54.6
87.5
86,6
85.0
83.3
95.1
69.6
Adapter'
0.8%
38.8
37.1
38.8
30.3
49.1
28.3
56.9
53.8
56.4
47.5
70.0
44.1
Lester et al. & P-tuning
0.01% %
78.0
76.7
75.6
72.3
87.5
56.1
85.8
85.0
82.8
82.1
93.1
70.6
BitFit
0.09%
79,9
78.8
77.6
74.8
88.2
56.0
87.2
86.3
84.5
83.3
94.5
71.4
We adopt (Preiffer et al., 2020)'s implementation (CF. Appendix B.1) 1) and tried several syper--arameter combinations.
using the Dense Passage Retrieval (DPR) Model
Cross-Domain and Cross-Topic
proposed by (Karpukhin et al., 2020). We train our DPR model with four different PE learning techniques: Adapters (Houlsby al., 2019), Lester et al. & P-Tuning (Liu et al., 2021c), P-Tuning v2 (Liu et al., 2022) and BitFit (Zaken et al., 2022), which are introduced in Section 4.2, and compare them against the original call-parameter fine-tuned DPR. Following (Karpukhin et al., 2020), we eval- uate in the multi-dataset setting where the training data combines all datasets excluding SQuAD while the testing is done for all datasets.
Generalizability
In this section, we examine the zero-shot general- izability of fine-tuning and PE learning. We take P-Tuning v2 (Liu et al., 2022) as an representative for PE methods, which has the highest average in- domain accuracy Particularly, as previous work seldom looks into the cross-topic generalization, introduce OAG-QA, the largest fine- ane--grined cross-topic retrieval dataset date. On cross- domain evaluation, we adopt the well-acknowledge BEIR (Thakur al., 2021) benchmark.
We use top-k retrieval accuracy as our evalu- ation metrie, which measures the percentage of questions that have at lcast one document contain- ing the answer in the top retrieved documents. In our experiments, we report top-20 and top-100 accuracy following (Karpukhin et al., 2020).
6.1 OAG-QA: Fine-Grained Cross-Topie Scientific Literature Retrieval Dataset
OAG-QA is fine-grained topic-specific pas- sage retrieval dataset constructed by collecting high-quality questions and answers from Online Question-and-Anseee
(Q&A) forums, such as Quora and Stack Exchange. These forums offer people chances to ask questions and receive an- swers from other expert users, potentially with reference to academic papers. These references can be consequently aligned to paper entities with rich metu-informmaton (e.g. abstract, field- of-study (FOS)) in the Open Academic Graph (OAG) (Zhang et al., 2019), the largest publicly available academic entity graph to date.
Results. We identify the best-performed hyper- parameters for each method and the results are shown in Table 1. P-Tuiing v2 and BitFit are com- parable to fine-tuned baseline on all datasets as expected. P-Tuning v2 also performs the best on four in-domain datasets among the tested PE ap- proaches. On the other hand, Lester et al. & P- Tuning performs bit weaker than the fine-tuned baseline. Adapter shows weak performance, but might attributed to the version of implementa- tion (Pfeiffer et al., 2020) (i.e., other versions with different implementation or more tunable parame- ters may be better). The results empirically demon- strate that PE methods significantly cut down necessary tuning parameters to 0.1 % and provide competitive performance in in-domain data.
We collect questions from two influential web- sites: Stack Exchange? in English, and Zhihu in Chinese. On top of the collected pairs of questions and paper titles, we align them to OAG (Zhang et al., 2019; Wang et al., 2020a; Tang et al., 2008) paper ids via public API In terms of topics, disci- plines from Stack Exchange and tags from Zhihu naturally serve as fine-grained topics attached to
Interestingly, we also notice that on the out-of- domain dataset SQuAD, P-Tuning v2, Lester et al. & P-Tuning, and BitFit substantially outperform the fine-tuned counterpart.
fuups://saackexchange.com "htps://wwwwzhhihu.co
service

---

Table 2: Examples of disciplines, topics, and example query-paper pairs (only titles are shown) in OAG-QA.
Disc.
#Topic
Example Topic
#Query
Example query-paper pairs
Neural
Artificial
Q: Can neural networks be used to prove conjectures?
Network
Neural Network
488
Paper: Generating Correctness Proofs with Neural Networks
Quantum
Q: What is the effective potential for photons in X-ray diffraction? Paper: Introduction to the theory of x-ray matter interaction
Photon
125
Mcchanics
Table 3: OAG-QA's statistics and examples. Com-
to involve both binary and graded relevance mea- sures for ranking quality.
pared to existing scientific retrieval dataset (Sci- Fact (Wadden al., 2020), SCIDOCS (Cohan et al., 2020), TREC-COVID (Voorhees et al., 2021)).
Results. Table 4 reports the results of DPR and ColBERT on the 15 datasets of BEIR. For DPR, P-Tuning v2 generalizes much better than the fine- tuned one on all datasets except for MS MARCO and DBPedia. We observe that the datasets where our method improves by more than 5 points, such as Touche-2095 and SciFact, usually consist of long documents with average lengths over 200. We conjecture that the DPR trained on OpenQA has been biased to the 100-word document length in the oridinary setting. In summary, P-Tuning v2 achieves an absolute 5.2% improvement on the fine-tuned baseline on average. Thus, P- P-Tuming v2 greatly improves the out-of-domain generalization of dense retrieval models.
Dataset #Query #Corpus #Disc. #Topic Fabrication
SciFact
1,409
5,183
Crowd-Source
SCIDOCS
22,000
25,657
User Clicks
TREC-COVID
50
171.332
Crowd-Source
OAG-QA 17.948 870,000
22 87
Online Forum
collected questions after post-processing For more construction details, please refer to Appendix A. Consequently, we present OAG-QA (Cf. Ta- ble 3) which consists of 17,948 unique queries from 22 scientific disciplines and 87 fine-grained topics. Given each topic, we sample 10,000 can- didate papers including the groundtruth from the same disciplines as OAG annotates, and take their titles and abstracts as the corpus.
On the other hand, ColBERT trained by P- Tuning v2 also outperforms the fine-tuned Col- BERT on almost all (13/15) datasets. P-Tuning v2 slightly underperforms NQ and Quora where documents are relatively short. For the out-of- domain average scores, P-Tuning v2 outperforms the baseline ColBERT by an absolute gain of 2.4%. Compared to DPR, fine-tuned COIBERT general- izes better, probably because it is trained on the larger and more diverse MS MARCO and its ar- chitecture can be more scalable. But P-Tuiing v2 still gains an advancement generalization over the fine-tuned one. In conclusion, the results show that with similar in-domain performance, P-Tuning v2 can improve zero-shot generalization for cross- domain compared to fine-tuning.
6.2 Zero-Shot Cross-Domain Generalization
Datasets. We adopt Benchmarking-IR (BEIR) pro- posed in (Thakur al., 2021), zero-shot gener- alization benchmark for evaluating retrievers tasks across domains. It consists of zero-shot evalua- tion datasets, (15 out of 18 are available) from 9 retrieval tasks of heterogeneity. The datasets vary from each other in corpus sizes (3.6k 15M documents), queries and documents' lengths, and domains (news articles vs. scientific papers).
Settings. Following (Thakur et al., 2021), we trained the models on one dataset and report the zero-shot performances the other datasets. We choose DPR (Karpukhin al., 2020) from dense re- trievers and ColBERT (Khattab and Zaharia, 2020) from late-interaction models to explore the retrieval effectiveness under PE and full-parameter fine- tuning settings. Following the settings of BEIR, we use the open-sourced Multi-dataset DPR check- point (Karpukhin et al., 2020) and ColBERT model trained on MS MARCO (Nguyen al., 2016).
6.3 Zero-Shot Cross-Topic Generalization
In addition to cross-domain generalization, cross- topic generalization is more pragmatic and mean- ingful challenge for retrieval tasks. For example, in a scientific literature retrieval system, the corpus sizes, abstract lengths, and writing styles would not vary too much. The challenge lies in refining retrievers for more fine-grained fields-of-study.
To obtain comparable evaluation across datasets and tasks in BEIR (Thakur et al., 2021), 1), we use Normalized Cumulative Discount Gain (nDCG@k)
Settings. We use the same trained DPR (Karpukhin et al., 2020) and ColBERT (Khattab and Zaharia,

---

Table 4: Zero-shot cross-domain generalization evalu- ated 14 datasets of BEIR (Thakur et al., 2021). All scores are nDCG @ 10, and those of "FT" are taken from BEIR's report. ("* denotes in-domain datasets; "FT" denotes fine-tuning; "PT2" denotes P-Tuning v2)
Table 5: Zero-shot cross-topic generalization evaluated on 22 disciplines of OAG-QA. All scores are Top-20. ("FT" denotes fine-tuning; "PT2" denotes P-Tuning v2)
Model(-)
Dense
Interactiin
DPR
ColBERT
Topie(4)
M-del(-+)
Lexical
Dense
Late- Interaction
FT
PT2
FT
PT2
BM25
DPR
ColBERT
Geometry
0.154
0.199
0.303
0.323
Datuset([)
Statisties
0.149
0.184
0.289
0.302
FT
PT2
FT
PT2
Algebra
0.194
0.171
0.271
0.267
MS MARCO
0.228
0.177
0.171
0.401
0.414
Calculus
0.145
0.169
0.248
0.259
Number theory
0.136
0.161
0.260
0.256
TREC-COVID
0.656
0.332
0.394
0.677
0.679
Linear algebra
0.227
0.211
0.351
0,345
NFCorpus
0.325
0.189
0.224
0.305
0.327
Astrophysics
0.130
0.160
0.213
0.229
Quantum mechanies
0.134
0.169
0.240
0.245
NQ
0.329
0.474
0.479*
0.524
0.515
Physics
0.205
0.245
0.349
0.360
HotporQA
0.603
0.391
0,416
0,593
0.623
Chemistry
0.157
0,159
0.296
0.300
FIQA
0.236
0.112
0.128
0.317
0.333
Biochemistry
0.301
0.332
0.443
0.463
ArguAna
0,315
0.175
0.214
0.233
0.415
Health Care
0.367
0.388
0.446
0.459
Touche-2020
0.367
0.131
0.207
0,202
0.236
Natural Science
0.306
0.364
0.408
0.410
Psychology
0.214
0.247
0.332
0.362
StuDk
0,299
0.153
0.158
0.350
0.366
Algorithm
0.211
0.244
0.365
0.390
Quora
0.789
0.248
0.509
0.854
0.845
Neural Network
0.176
0,207
0.214
0.245
DBPedia
0.313
0.263
0.254
0.392
0.407
Computer Vision
0.152
0.197
0.264
0.291
SCIDOCS
0.158
0.077
0.099
0.145
0.156
Data Mining
0.139
0,161
0.226
0.231
FEVER
0.753
0.562
0.593
0.771
0.779
Deep Learning
0,143
0.173
0.249
0.271
Machine Learning
0.136
0.187
0.258
0.278
ClimateFEVER
0.213
0.148
0.194
0.184
0.190
NL.P
0.149
0.160
0.234
0.254
SciFact
0.665
0,318
0.436
0.671
0.685
Economics
0.339
0.353
0.321
0.298
Av Avgowlo MS MARCO)
0.430
0.255
0.307
0.444
0.468
Average
0.194
0.220
0.299
0.311
2020) model introduced in 6.2 and conduct zero- shot evaluation. We measure top-20 retrieval ac- curacy on the dataset of each topic and report the average scores over each discipline.
ibration refers to models' ability to provide class probability that corresponds to its likelihood of being true. A calibrated model provide trustful confidence to its prediction, which is particularly important for algorithms deploying in critical real- world scenarios.
Results. Table 5 compares models trained by P- Tuning v2 and fine-tuning using top-20 retrieval accuracy. P-Tuming v2 outperforms fine-tuning in 20/22 topics in DPR and 18/22 topics in ColBERT respectively. Specifically, P-Tuning v2 performs poorly in Algebra and Linear algebra, two fields which contain a large number of mathematical sym- bols, in both DPR and ColBERT at the same time. Overall, average P-Tuning v2 are better than that of baseline, gaining 2.6% and 1.2% absolute improvement over DPR and ColBERT respectively.
Notwithstanding the higher accuracy, mod- ern neural networks are known to be miscali- brated (Guo et al., 2017). Recent literature has also demonstrated that cross-domain calibration is nice proxy for model's out-of-domain generaliz- ability (Wald et al., 2021). To measure a retriever's calibration, we resort to Expected Calibration Error (ECE) proposed in (Naeini et al., 2015) as:
M
Bml
ECE
[T v.) [i]]
7 An Understanding of the Generalization
n
Bm im
(9)
How does PE learning help neural text retrievers to generalize well? While it might be attributed PE learning's 's flatter loss minimum or alleviated catas- trophic forgetting, in this work we investigate other quantifiable reasons, the confidence calibration and query-length robustness.
which bins estimates from n samples within [0. 1J into Bm, a set of M equal-length buckets. Each sample i has its label y/, estimated label ûi and estimated probability pi.
Following prior work (Penha and Hauff, 2021), cast the ranking problem multi-class clas- sification to compute ECE. We take queries with valid top-5 predictions, apply softmax over retrieval scores per query, and turns the ranking into 5-class classification to derive ECE (Cf. Table 6) and cali- bration diagrams (Cf. Figure 3).
7.1 Confidence Calibration
Despite metrics like accuracy are usually the most concerned in machine learning, there are more properties to care about, such as calibration. Cal-

---

Table Expected Calibration Error (ECE) (Nacini et al., 2015) of Fine-tuning (FT) and P-Tuning v2 (PT2) based DPR (Karpukhin al., 2020); smaller the better.
domain
Cross domain
NQ
TQA
WQ
TREC
SQuAD
MS-M
TCovid
NFC
HoPo
FiQA
ArgA
TF2020
CQA
Quora
DBPedia
SCID
FEVER
CREVER
SeiFoot
FT
0,135
0,259
0,219
0,323
0.114
0.156
0,164
0.143
0.01
0.153
0.156
0.141
0.144
0. 0.10
0.128
0.144
0.069
0.135
0.120
0.178
0.084
0.145
0.084
0.051
0.125
0.104
NO
Trivia
ArguAnn
SciFact
Rorledly da brated
Adtect caliinaal
K2
Redectly ca briwed
Aedocdly brated
Rerlctly oa brated
Datu
Ouss
Dyt
Duty
Dat-
Resiore
Resture
Reslune
Figure 3: Calibration diagrams of DPR using P-Tuming v2 and fine-tuning on in-domain OpenQA datasets (c.g., NaturalQuestions and TriviaQA) and cross-domain BEIR datasets (e.g., ArguAna, Quora and SciFact).
Quora
ArguAna
400
NOc0810
0.6
0.30
Finetune
300
0.25
0.5
200
1000
0.20
0.4
100
Pinctune
0.15
14
16
>30
<-100
150
350
400
Query longth
Query longth
Figure 4: NDCG@ 10 (left axis) and #Query (right axis) of P-Tuning v2 (PT2) and Fine-tuning (FT) by query length (splitted into bins) on ArguAna and Quora based on DPR (Karpukhin et al., 2020).
Findings. As shown in Table 6 and Figure 3, we find that P-Tuning v2 based DPR are more cal- ibrated than its fine-tuned counterpart, whatever in-domain cross-domain datasets. The only exception the TREC-COVID dataset in BEIR, which only evaluates on 50 queries and may cause a variance. To conclude, even though fine-tuning and P-T P-Tuning v2 share a similar in-domain perfor- mance, their levels of calibration still vary largely from each other, which accords with observations in (Guo al., 2017) that better accuracy does not mean better calibration property. Such calibra- tion can explain P- P-Tuning v2's generalizability, as (Wald et al., 2021) theoretically proves that supe- rior multi-domain calibration effect to fine-tuning usually leads to better cross-domain generalization.
datasets, Quora and ArguAna from BEIR (Thakur et al., 2021), to justify the hypothesis. The query lengths are derived from splitting plain query texts by white-spaces. For clearer visualization, we split the lengths by equal-sized bins. As shown in Figure 4, when queries are medium-length (30- 100), both P-Tuning v2 and fine-tuning perform comparably. But when queries are either relatively short (in Quora) or long (in ArguAna), P-Tuning v2 generalizes much better than fine-tuning. This indicates that PE learning based (e.g.. P-Tuning v2) neural text retrievers have better robustness against varied query lengths in testing.
Conclusion
We propose to leverage PE prompt tuning for neu- ral text retrieval, which proved for the first time in this problem for comparable performance to full-parameter fine-tuning. Furthermore, PE ap- proaches like P-Tuning v2 improve gross-domain and cross-topic generalization, which fundamen- tally comes from improved confidence calibration and query length robustness as show. Finally, construct and release the largest fine-grained topic-specific academic retrieval dataset OAG-QA, which contains 87 different domains and 17,945 query-pape pairs, to support future research.
7.2 Query-Length Robustness
Mismatched query lengths across datasets is an- other hidden reason. For example, in four OpenQA datasets experiment, most query lengths locate in the interval from 40; while other datasets can have very different query lengths. Fine-tuning changes pre-trained models parameters and may consequently bias text retrievers to certain query lengths; PE methods are free from such worries.
Findings. We present a case study on two typical

---

Limitations
ical chasm: statistical approaches answer findinggiiing..... In SIGIR 2000: Proceedings of the 23rd Annual In- ternational ACM SIGIR Conference Research and Development in Information Retrieval, July 24-28, 2000, Athens, Greece, pages 192-199. ACM.
this section we discuss several potentially unre- solved topics related to this work.
First, despite the superior parameter-effiiencyy of PE learning, a long-standing challenge is that it converges slower and is relatively more sensi- tive to papee--arameters (typically, learning rate) than fine-tuning. We have the same observation in this work and have to bypass the problem by training longer and trying multiple groups of hyper- parameters. It is thus important to design more robust and stable training strategies for prompt tun- ing in the future.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu. Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv, abs/2005.1410 14165.
Second, the OAG-QA dataset requires further exploration. As indicated in Table 7, we purposely leave 20 samples in each fine-graie fine-grained topic for future investigations on the effectiveness of PE learning in few-shot and meta learning settings. Conventionally, these settings require fine-tuning the whole model in each task separately, causing great redundancy. However, PE learning's 's extreme parameter efficiency can come to its rescue. We leave this investigation for future work.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes, 2017. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1870-1879.
Tao Chen, Mingyang Zhang, Jing Lu, Michael Bender- sky, and Mare Najork. 2022. Oft-oo-domain seman- tics to the rescue! sero-shot hybrid retrieval models. In European Conference on Information Retrieval, pages 95-110. Springer.
Third, PE lcarning's calibration and generaliza- tion properties should ideally be applicable to other language tasks, such as text understanding and gen- eration. In this work, we focus on neural text re- trieval, as it usually faces more distribution-shin scenarios. However, many other practical problems also suffer from the challenges of biased training data and generalization, and the application of PE learning them remains largely unexplored.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: document-level representation learning using citation- informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2020, Online, July 5-10, 2020, pages Linguistics.
Association for Computational
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Pruceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
References
Petr Baudis and Jan Sedivy. 2015. Modeling of the question answering task in the yodaqa system. In Ex- perimental IR Meets Multilinguality, Multimodality, and Interaction - 6th International Conference of the CLEF Association, CLEF 2015, Toulouse, France, September 8-11, 2015, Proceedings, volume 9283 of Lecture Notes in Computer Science, pages 222-228. Springer.
Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulié, and Iryna Gurevych. 2022. Retrieve fast, rerank smart: Cooperative and joint approaches for im- proved cross-modal retrieval. Transactions of the Association for Computational Linguistics, 10:503- 521.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2013, 18-21 October 2013. Grand Hyall Seattle, Seattle, Washington, USA. A meeting of SIGDAT, a Special Interest Group of the ACL, pages
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PMLR
ACL.
Adam L. Berger. Rich Caruana. David Cohn, Dayne Freitag, and Vibhu O. Mittal. 2000. Bridging the lex-
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented

---

language model pre-training. In International Con- ference on Machine Learning, pages 3929-3938. PMLR.
Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-We Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguisties, 7:452- 466.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yugi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang. Liang Zhang. et al. 2021. Pre-trained models: Past, present and future. AT Open, 2:225-250.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059.
Sebastian Hofstätter, Sheng-Chich Chich Lin, Jheng- Hong Yang. Jimmy Lin, and Allan Hanbury. 2021. Ef- ficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 113-122.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, pages 4582- 4597. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-effice efficient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Robert Litschko, Ivan Vulié, and Goran Glavaš, 2022. Parameter-efficient neural reranking for cross- lingual and multitingual retrieval. arXiv preprint arXiv:220..02292.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learn- ing Representations.
Binsheng Liu, Hamed Zamani, Xiaolu Lu, and J Shane Culpepper. 2021a. Generalizing discriminative re- trieval models using generative tasks. In Proceedings of the Web Conference 2021, pages 3745-3756.
Gautier Izacard, Mathilde Caron, Lueas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. CoRR, abs/21 12.09118
Linqing Liu, Patrick Lewis, Sebastian Riedel, and Pon- tus Stenetorp. 202 Challenges in generalization in open domain question answering. arXiv preprint arXiv:2109.01156
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61-68.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume Long Papers, pages 1601-1611. Association for Computational Linguistics.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding. Yujie Qian, Zhilin Yang, and Jie Tang. 202 Gpt understands, too. arXiv preprint arXiv:2103.10335
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 7907.11099999
Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39-48. ACM.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
Tri Nguyen, Mir Rosenberg, Xia Song. Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrat- ing neural and symbolic approaches 2016 co-located
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti,

---

with the 30th Annual Conference on Neural Infor- mation Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org.
of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual.
Jianmo Ni. Chen Qu Jing Lu, Zhuyun Dai, Gus- tavo Hernández Abrego, Ji Ma. Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming- wei Chang, and Yinfei Yang. 2021. Large dual encoders are generalizable retrievers. CoRR, abs/21 12.07899.
James Thome, Andreas Vlachos. Christos Christodoulopoolls, and Arpit Mittal. 2018. Fever: large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819.
Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019. From doc2query to docttuttquery. Online preprint, 6.
Gustavo Penha and Claudia Hauff. 2021. On the cal- ibration and uncertainty neural learning to rank models for conversational search. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 160-170.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Tree-covid: constructing a pandemic information trieval test collection. In ACM SIGIR Forum, vol- ume 54, pages 1-12. ACM New York, NY, USA.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick S. H. Lewis. Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2523-2544. Associa- tion for Computational Linguistics.
David Wadden, Shanchuan Lin, Kyle Lo. Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact fiction: Verifying scientific claims. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7534-7550. Association for Computa- tional Linguistics.
Jonas Pfeiffer. Andreas Rücklé Clifton Poth, Aishwarya Kamath, Ivan Vulié. Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP 2020): Systems Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. 2021. On calibration and out-of-domain gen- eralization. Advances in neural information process- ing systems, 34:2215-223 22277.
Kuansan Wang. Zhihong Shen, Chiyuan Huang, Chieh- Han Wu, Yuxiao Dong, and Anshul Kanakia. 2020a. Microsoft academic graph: When experts are not enough. Quant. Sei. Stud., 1(1):396-413.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguisties.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020b. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5788.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021. Col- bertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/2112.01485.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amood Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Jie Tang, Jing Zhang. Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008 Arnetminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 990- 998.
Ming Yan, Chenliang Li, Bin Bi, Wei Wang, and Song- fang Huang. 2021. A unified pretraining framework for passage ranking and expansion. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 35, pages 4563-4563.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: heterogeneous benchmark for zero-shot evaluation

---

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficiet fine- tue-tunnng for transformer-basel masked language-models, In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics ( (Volume 2: Shori Pupers), ACI. 2022, Dublin, Treland, May 22- 27, 2022, pages 1-9. Association for Computational Linguistics.
name in the suffix, take the result as the paper title.
Translation.
Because questions from Zhihu are in Chinese, we use Tencent Cloud for the corpus translation.
Cleaning. Out of consideration for remaining the diversity of questions and difficulty to evaluate the quality of questions academic fields, we just use simple clcaning strategies. For the questions from Stack Exchange, we deleted the questions shorter than 4 words which usually not able to restrict the topic to appropriately sized field for paper retrieval. For the questions from Zhihu, we also just removed the questions manually which are obviously not related academic topics.
Fanjin Zhang, Xiao Liu, Jie Tang. Yuxiao Dong, Peiran Yao, Jie Zhang. Xiaotao Gu, Yan Wang. Bin Shao, Rui Li, and Kuansan Wang. 2019. OAG: toward linking large-scale heterogeneous entity graphs. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 2585 2595. ACM.
A Details of OAG-QA
In this section, we introduced the steps for building our fine-grained cross-topic dataset OAG-QA.
A.3 Alignment.
We align the extracted papers with the OAG paper database (Zhang et al., 2019) to retrieve more infor- mation of papers, especially abstract. The papers which cannot be found in the database or whose corresponding abstract is missing in the database are discarded. Finally we only keep the question- paper pairs with complete title and abstract text.
A.1
Data Collecting
OAG-QA is collected from two widely used web- sites:
Stack Exchange in English and Zhihu in Chinese. Stack Exchange consists of various fo- rums for specific domain, such as data science, physics and chemistry, where questions are marked by fine-graing grained tags by users. Zhihu not divided by domains but questions are also tagged by topics.
A.4 Statisties
Our self-construct dataset OAG-QA composes of 17, 17,948 unique questions from 21 scientific disci- pline and 87 fine-grained topics. We sample 10,000 papers including the groundtruth papers to con- struct a candidate set for each topic. The queries in each topic is divided as a training set of size 20 and a test set with the remaining data. OAG-QA has a two-level hierarchical structure where each topic is under specific discipline. Table 7 shows the statistics of OAG-QA in detail.
A.2 Data Pre-Processing
Paper Extraction and Title Retrieval. We ex- tract the paper from answers by regular expres- sion patterns for paper URLs. So far, we focus on five types of URLS from the answer context: arxiv.org, dl.acm.org, doi.org, researchgate.ner www.nature.com which can indicate publications cited by users. Then we retrieve the titles from the URLs using the the strategies listed below:
arxiv.org: We recognize pdf suffix in the URL and extract arxiv with regular expression, then query in the arxiv API with arxiv id to get the paper title.
B Implementation Details
B.1 Implementation of DPR
Experiment enviroment We conducted our exper- iments the Linux platform, the version of which was 1..000957..e7.x888 __4,,,444 and the GPU version was NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GBJ. After installation of CUDA 11.2, we set basic experiment environment with conda 4.10.1. Our models were implemented using Python 3.8 and Py Torch 1.11.0. We used the trans- formers library (version 4.15.5) for the pre-trained Sungs:/dduuteeeaccccccoceeeeeeee
dlaacm.org: We get the HTML with URL, use the text in "title" label, then delete the website name in the suffix, take the result as the paper title. doi.org: We extract doi with regular expression, then query the doi api with the doi to get the paper title.
researchgate.ner We just split the suffix of URL into words with "_" as the paper title. www.nature.com:: We get the HTML with URL, use the text in "title" label, then delete the website

---

Table 7: Statistics of OAG-QA.
BERT model. When training DPR with Adapter, the addaper-translommeexveesssn 2.2.0) was used.
Discipline
Tople
[iQuery
Train
lest
fQuuery
Original DPR (Karpukhin et al., 2020) We used the open-sourced DPR checkpoint trained on multi- task data with bert-base-uncased model (sequence length: 256). The results are aligned with DPR authors' reported ones in paper.
gecenctry
230
210
198
20
168
dlgchriccJnppllgyy Inpelogy
131
20
111
Goomeryy
differeniaa poomerry
230
20
210
1380
group Shoury
348
20
228
categury
191
171
lopology
140
matlemutical statletics
Statisiis
layes theomu
134
516
probabilit__hoon
238
20
218
Algsheu
ligchra
280
20
250
387
DPR with P-tuning v2 (Liu et al., 2022). For P- tuning v2 training, used batch size of 128 and a sequence length of 256. We trained the question and passage encoders, which are based on bert- based-uncased model, for up to 40 epochs for large datasets (NQ, TriviaQA, SQuAD and Multi-dataset sctting) and 100 epochs for small datasets (TREC, QA) with learning rate of 0.01 and a prefix length of 100 using Adam, linear scheduling with 5% warm-up and dropout rate 0.1.
polynomial
107
87
201
pantial differentia, equation
200
Calcolus
functiornal analyss
127
107
868
hiloer_.space
127
20
1877
real inalysis
172
20
152
number, luvory
274
20
354
Number theory
combinatorics
221
201
150
xine number
225
201
Lincer algebra
matii
Jigebo
220
130
20
20
200
110
350
Istronouyy
108
20
88
astrophysics
101
81
92
cosmology
geaenil_elaatvitt
171
Astrobysicc
special, rebeivity
1575
spocctime
172
20
152
dirk malier
176
20
156
DPR with Lester et al. & P-Tuning (Liu et al., 2021c). Like P-tuning v2, we used bert-based- uncased model as basic model, however, we only applied modification the input and set the param- eters of learning rate 0.01. We tried different prefix length such as 100, 200 to test the perfor- mance of the model.
black_hole
160
20
140
enimupy
127
100
theory
137
quastuse mechanics
467
447
quantum mntanglement
101
quantum, field iheory
295
215
quantum grvvly
154
20
134
quantum, information
190
20
170
Quantum machemics
puniil physic.
347
20
227
2385
photoa
125
105
supersymmetay
245
225
themodynamiis
213
espeimental_hhiie
cunfoemall thewey
101
DPR with BitFit (Zaken et al., 2022). In BitFit training, we use the same values of batch size, se- quence length, dropout rate and learning rate as in P-tuning v2 as well as the same model, bert-based- uncased model. It took 40 epochs to train the model in the same datasets using Adam Optimizer and lin- ear scheduling with 5% warm-up. We fixed all parameters and trained only bias parameters.
grage theoey
104
20
84
classical mecchancs
115
20
95
condensed matter physics
201
151
Physics
151
131
dustmuragnctimm
204
mathenutical slysies
171
151
ogaanic chamistry
352
20
312
chemical snihesis
240
20
220
Chenistry
Inorganic chemusyy
218
20
198
1082
Syyicsl_chemitry
190
20
170
compusaional chemisstr
82
Biocheemistty
bocchemsstry
cell biology
113
291
healih, pare
285
20
268
Heakk Cane
enderinolugy
111
20
91
633
phycologs
224
20
204
DPR with Adapter (Houlsby et al., 2019). In the procedure of training Adapter, we set the Adapter architectures as PřeiffcrConfig style, and except learning rate of 3e-5 and epochs of 50, the pa- rameters and datasets were all same as in Bit- Fit as introduced in the above paragraph. We adopt the implementation of adapter in ADAPTER- TRANSFORMER (Pfeiffer et al., 2020).
Natunil Science
sckenee
193
173
evelutinrry boology
471
451
Psycology
socal psycholugy
segiitive_eerosstee
248
125
20
328
A0
571
Algerill
psph theary
dloocithus
386
189
20
20
366
169
575
Neunal Netwirk
cossitive science
102
82
sompuerr_vas
315
20
295
Compate Vision
ammuter graphics images
68
20
48
661
comoolutional neural network
278
20
258
data
131
111
featue,
130
110
Data Mining
vola_validalon
97
time seits
204
clsser anlyess
92
73
deep Jeaming
372
20
352
Deep Leaming
Aiimistion_Alggriitu
238
20
218
791
renforcemen, Jeaming
181
161
looming
Machine Leaming
huddan mariov model
dhasifier
269
112
20
249
92
1208
linear jearession
244
20
224
NLP
natrrll mntrrillnngggg
meurnnt resral metwork
282
305
20
20
262
255
587
Lecnomics
comnomics
218
Total
17948
1740
16208
17948

---

B.2 Implementation of ColBERT
The model is trained for up 400k iterations.
Original ColBERT (Khattab and Zaharia, 2020) In full-parameter training, We adopt the parameters offered by (Khattab and Zaharia, 2020). We trained ColBERT model with a learning rate of 10 with batch size of 32. We fix the number cm- beddings per query at 32 and follows (Thakur et al., 2021) to set the number of document embeddings 300. The embedding dimension is set as 128.
ColBERT with P-Tuning v2 (Liu et al., 2022). With P-tuning v2, We trained ColBERT from the parameters of gert-based-uncasee for up 400K steps MS MARCO dataset with learning rate of 0.01 and a prefix length of 64. We used a batch size of 32 and fixed the number of embeddings per query at 32 and the number of embeddings per document at 300. The embedding dimension is set to be 128.


Reviews:
[{"review_id": "K75l9fKPA0", "reviewer": "Reviewer_DnA1", "summary": "", "questions": "A.\tIn the introduction, the contributions of this paper are summarized by using number 1, 2, and 3, without the need to adopt “Problem”, “Understanding”, “and Dataset”.\nB.\tThe paper provides extensive experimental evidence to demonstrate the effectiveness of PE learning. However, it lacks an experimental analysis of the model's parameters, I hope the author can supplement the experiment in the follow-up.\nC.\tIn Table 1, the retrieval effect of using Adapter is much lower than that of other methods. What is the reason?\n\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful review and suggestions! We are happy that you like the work, and you may find our detailed responses below:\n\n1. For the experimental analysis of the model’s parameters, we conduct experiments with different PE learning settings. The number of trainable parameters in different training settings and the corresponding retrieval results on in-domain dataset NQ and out-of-domain dataset SciFact are shown in the table below. We train p-tuning v2 models with different lengths of prefix and thus different numbers of trainable parameters. They result in similar scores in both in-domain and out-of-domain datasets and considerably outperform fine-tuned models in out-of-domain data.\n\n|                          | #Prefix |    #Param   | TREC-COVID | NFCorpus |\n|--------------------------|:-------:|:-----------:|:----------:|:--------:|\n| Fine-tune                |    -    | 109,580,544 |    0.677   |   0.318  |\n| P-Tuning v2              |    64   |  1,277,952  |    0.682   |   0.327  |\n|                          |   100   |  1,941,504  |    0.681   |   0.322  |\n|                          |   180   |  3,416,064  |    0.700   |   0.323  |\n|                          |   256   |  4,816,896  |    0.700   |   0.328  |\n|                          |   300   |  5,627,904  |    0.685   |   0.329  |\n| Lester et al. & P-Tuning |    64   |   147,456   |    0.647   |   0.309  |\n\n2. For the low retrieval effect of using Adapter in DPR, compared to P-Tuning methods and BitFit which manipulates the attention-related parameters, (Pfeiffer et al., 2020)'s implementation of adapters (which is also the most common implementation of adapters) is only on the MLP layer. We think that the retrieval training may focus on the improvement of LM's attention patterns rather than the memory pattern (which is known to be stored in MLP's parameters). As a result, the adapter performs poorly on transferring to retrieval tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper demonstrates a high level of creativity and innovation. It provides the first evidence that utilizing PE prompt tuning for neural text retrieval achieves comparable performance to full-parameter fine-tuning. By updating only 0.1% of the model parameters, the prompt tuning strategy can help the retrieval model achieve better generalization performance than traditional methods that update all parameters.\n2.\tThe experiments in this paper are comprehensive and effectively validate the effectiveness of the PE prompt tuning strategy.\n3.\tThis paper curates and releases an academic retrieval dataset with 18K query-results pairs in 87 topics, making it the largest topic-specific one to date. If accepted, this paper can assist NLP researchers in exploring the application of PE prompt tuning to address other practical issues such as biased training data and weak generalization.\n", "reasons_to_reject": "1.\tThe lack of a definite explanation from the author regarding the low retrieval results of the Adapter in Table 1 may cause confusion for readers.\n2.\tThis paper provides substantial experimental evidence to demonstrate the effectiveness of PE learning, but it lacks experimental analysis on the parameters of the P-Tuning v2 model.\n", "questions_to_authors": "A.\tIn the introduction, the contributions of this paper are summarized by using number 1, 2, and 3, without the need to adopt “Problem”, “Understanding”, “and Dataset”.\nB.\tThe paper provides extensive experimental evidence to demonstrate the effectiveness of PE learning. However, it lacks an experimental analysis of the model's parameters, I hope the author can supplement the experiment in the follow-up.\nC.\tIn Table 1, the retrieval effect of using Adapter is much lower than that of other methods. What is the reason?\n\n", "paper_topic_and_main_contributions": "This paper is about prompt tuning for neural text retrievers. In this paper, the authors propose to leverage PE prompt tuning for text re-trieval across in-domain, cross-domain, and cross-topic settings, which is proved for the first time in this problem for comparable performance to full-parameter fine-tuning. And it can alleviate the issues of parameter inefficiency and weak generalization faced by fine-tuning based re-retrieval methods. The main contributions of this paper are as follows:\n1.\tThis paper proposes to leverage PE learning for neural text retrieval. The PE prompt tuning can not only perform in-domain, but also enables neural retrievers to achieve significant generalization advantages over fine-tuning on cross-domain and cross-topic benchmarks.\n2.\tThis paper provides an understanding of PE learning's outperformance across domains and topics. Furthermore, through a large number of experimental analysis, PE approaches like P-Tuning v2 improve cross-domain and cross-topic generalization, which fundamentally comes from improved confidence calibration and query length robustness.\n3.\tThis paper constructs and releases the largest fine-grained topic-specific academic retrieval dataset OAG-QA, which contains 87 different domains and 17,948 query-paper pairs, to support future research.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The author's paper demonstrates a high level of writing proficiency, and the paper reads smoothly."}, "extra_text": {}}, {"review_id": "kX59lRloh7", "reviewer": "Reviewer_deUw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review and appreciating our contributions. Here are our responses:\n\n1. In-domain experiments of ColBERT and other models\n\nThanks for your mentioning. In fact, the in-domain results of ColBERT (which is trained on MS-MARCO) has been reported in the original paper's Table 4 (FT: 0.401, PT-2: 0.414), annotated with \"*\". It shows that parameter-efficient prompt tuning method PT-2 even performs a bit better than fine-tuning on ColBERT in in-domain training.\n\nFor other models, originally we picked DPR and ColBERT as the representatives to cover two typical types of retrievers (Dense & Late-interaction) and demonstrated PE methods' effectiveness on them. We are also conducting experiments on other models including xMoCo [1] and ColBERT v2 [2], and preliminary results show that the conclusion still holds. Due to the short rebuttal period, we haven't completed the whole evaluation and will report them in our paper's future version.\n\nThanks again for your thoughtful evaluations and time commitment. Hope our response could address your concerns!\n\nReferences:\n[1] Yang N, Wei F, Jiao B, et al. xMoCo: Cross momentum contrastive learning for open-domain question answering. EMNLP 2021.\n[2] Santhanam K, Khattab O, Saad-Falcon J, et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper conducted intensive experiments analyzing the generalization ability of parameter-efficient prompt tuning on large pre-trained language models.\n2. This paper provides an empirical understanding which locates the source of the generalization ability of parameter-efficient tuning approaches. This could support the design of future methods.\n3. This paper constructs a cross-topic dataset which facilitates the experiments on models’ generalization ability.", "reasons_to_reject": "In sections 6.2 and 6.3, the authors evaluate the abilities of zero-shot cross-domain generalization and zero-shot cross-topic generalization respectively. These experiments are done with two models, DPR  and ColBERT. But in section 5 on the experiments of in-domain parameter efficiency, only the DPR model is used. Did the authors conduct such experiments on the ColBERT and/ or other models? Is the experiment results consistent with other models?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses the limitations of current parameter-efficient prompt tuning approaches used for neural text retrieval tasks, where machines are required to seek relevant texts based on given documents or questions. Recent studies employ large pre-trained language models, which face challenges on parameter- efficiency and generalizability. The paper examines mainstream parameter-efficient methods in in-domain, cross-domain, and cross-topic settings through intensive experiments and analyses. The results showed that the parameter-efficient prompt tuning strategy provided the models with even better generalization ability than full-parameter tuning. During this research, a cross-topic dataset is constructed to facilitate the experiments on the models’ generalization ability.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Pbe67mxxLb", "reviewer": "Reviewer_TuGb", "summary": "", "questions": "1. Besides OAG-QA, how is the performance of FT and PT2 on other similar datasets (e.g, SCIDOCS)?\n2. Why does fine-tuning could bias the text retriever but the PE one would not, may need some in-depth insights?\n3. What is the value of OAG-QA compared with existing ones besides its size is larger\n", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review and valuable suggestions. Here are our responses:\n\n1. On results besides OAG-QA, for example SCI-DOCS\n\nThanks for your good suggestion. In terms of SCI-DOCS and other datasets, actually we have reported their results as they present in the BEIR benchmark in Table 4. In fact, besides OAG-QA, it is in total 14 different out-of-domain datasets that we have tested. We pick out results of SCI-DOCS, SciFact, and TREC-COVID from Table 4 as below: \n\n| Model (→)   | DPR   |       | ColBERT |       |\n|-------------|-------|-------|---------|-------|\n| Dataset (↓) | FT    | PT2   | FT      | PT2   |\n| SCI-DOCS    | 0.077 | 0.099 | 0.145   | 0.156 |\n| SciFact     | 0.318 | 0.436 | 0.671   | 0.685 |\n| TREC-COVID  | 0.332 | 0.394 | 0.677   | 0.679 |\n\n2. On more in-depth insights on fine-tuning's bias on the text retriever\n\nThanks for your pointing it out. Due to the stringent page limit, we did not post our further analysis and findings in the main content. But we assure you we will update them to our future version. The major findings are:\n+ The bias on the positional encodings: we find it unnecessary to tune positional encodings in the fine-tuning of text encoders for retrieval for generalization. In fact, in our preliminary experiments we find that keeping positional encodings untuned would help for length generalization.\n+ The change of text embeddings and MLP parameters: we find the retrieval task does not introduce much new *language knowledge* in the fine-tuning, but majorly how an LM should change its attention over tokens. As a result, the tuning of word embeddings and MLP parameters could be not that important but to introduce more biases from the training data. It is also verified in our existed experiments, where adapter's performance is much poorer (since the (Pfeiffer et al., 2020)'s default implementation of adapter is only on the MLP part).\n\nWe hope these findings could provide you some in-depth insights on the problem. We will add these findings and corresponding experiments in the future version of the paper.\n\n3. OAG-QA's contributions\n\nIn addition to the size of OAG-QA, OAG-QA has two important contributions:\n+ OAG-QA is categorized into a two-level hierarchical structure. It divided questions into 22 scientific disciplines and 87 fine-grained topics. OAG-QA can offer deeper insights for the evaluation of academic retrieval tasks.\n+ OAG-QA is the first retrieval dataset constructed from online Question-and-Answers forums' provided reference url, such as Quora and Stack Exchange. This method makes the dataset authentic but challenging, and allows researchers to further expand OAG-QA dataset, and construct similar datasets in other domains or languages without human efforts.\n\nIn terms of OAG-QA's quality, we have mentioned that we have manually cleaned the questions created from the pipeline for OAG-QA's high-quality.\n\n4. On the coverage of tested models\n\nThe selection of DPR and ColBERT as our target testing model is under thoughtful consideration. Currently the neural text retrievers are mainly divided into two categories (as we mentioned in Section 3): Dense Retrievers and Late-interaction Retrievers. DPR and ColBERT are the most representative models in these categories, and other models are mostly modified based upon them. Thus, the verification over them could serve as indicative proof of our conclusions.\n\nBut to assure you the solidness of our work, we have been conducting experiments using other models such as xMoCo [1] and ColBERT v2 [2]. Our preliminary results show that our conclusion still holds, and we will update the whole evaluation results in our future version.\n\nThanks again for your thoughtful evaluations and time commitment. If your concerns are further addressed by our responses, would you please consider raising your score to support us? We cannot thank you enough for your help. \n\nReferences: \n\n[1] Yang N, Wei F, Jiao B, et al. xMoCo: Cross momentum contrastive learning for open-domain question answering. EMNLP 2021. \n\n[2] Santhanam K, Khattab O, Saad-Falcon J, et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces PE tuning to neutral text retrievers which shows a new possibility to people working on neutral text retrievers\n2. This paper gives some insights on why PE tuned retrievers have better generalizability on cross-topic retrieval tasks which may interest people in that area\n3. This paper proposed a new dataset which could benefit the NLP community", "reasons_to_reject": "1. This paper lacks noval methods and instead applies existing PE learning methods and compare with existing fine tuned models. Its scope of innovation is limited;\n2. Both PE and FT methods are tested only in DPR model and ColBERT model, whether user's conclusion from this paper can be generalized and served as a reference to a wider range of models/approaches is unclear.\n3. While this paper claims that understanding of PE learning's outperformance is one of the key contributions, it merely conducted some experiments to show that confidence calibration and query-length play an role in the performance gap between PE learning based neural text retriever and fine-tuned one. It does not provide an in-depth explanation on why and how these factors have such a impact;\n4. The dataset proposed by this paper primarily serves its experiments. Its potential value for generalization to other NLP tasks and the quality of the dataset are not clear.\n5. Among all PE methods, only PT2 has shown consistent superior performance compared with fine-tuned one. This raises a crucial question: if we choose different fine-tuned methods, could they surpass PT2? If so, the claim that PE method could have comparable performance to fine-tuning may collapse", "questions_to_authors": "1. Besides OAG-QA, how is the performance of FT and PT2 on other similar datasets (e.g, SCIDOCS)?\n2. Why does fine-tuning could bias the text retriever but the PE one would not, may need some in-depth insights?\n3. What is the value of OAG-QA compared with existing ones besides its size is larger\n", "paper_topic_and_main_contributions": "This paper proposed to apply parameter-efficient (PE) tuning for neutral text retrievers which involves much less tuning parameters. Authors showed that PE tuned text retriever outpeformed fine-tuned one in most of the tasks on most of the datasets. Also, authors found that the neutral text retrievers trained with PE has better generalizability which performs better on cross-domain cross-topic retrieval tasks. Besides that, authors built a new fine-grained topic-specific academic retrieval dataset.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an extensive analysis of various parameter-efficient prompt tuning methods for text retrieval across in-domain, cross-domain, and cross-topic settings and shows that such approaches can mitigate the parameter-inefficiency and weak generalizability issues. Though there is no new approach proposed in this work, the extensive analysis and insights will benefit the research in this line. The authors also contributed a new dataset for academic retrieval. The authors should address the minor concerns raised by reviewers, e.g., the impact of the number of parameters for P-Tuning V2, and clarifications on why only DPR and ColBERT models are chosen.
Decision: Accept-Findings