Topic-DPR: Topic-based Prompts for Dense Passage Retrieval
Authors: Qingfa Xiao
Shuangyin Li
Lei Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Continuous Prompt
Dense Passage Retrieval
Topic Modeling
TL;DR: Topic modeling can improve the effectiveness of dense passage retrieval by aiding in prompt design.

Abstract: Prompt-based learning's efficacy across numerous natural language processing tasks has led to its integration into dense passage retrieval. Prior research has mainly focused on enhancing the semantic understanding of pre-trained language models by optimizing a single vector as a continuous prompt. This approach, however, leads to a semantic space collapse; identical semantic information seeps into all representations, causing their distributions to converge in a restricted region. This hinders differentiation between relevant and irrelevant passages during dense retrieval. To tackle this issue, we present Topic-DPR, a dense passage retrieval model that uses topic-based prompts. Unlike the single prompt method, multiple topic-based prompts are established over a probabilistic simplex and optimized simultaneously through contrastive learning. This encourages representations to align with their topic distributions, improving space uniformity. Furthermore, we introduce a novel positive and negative sampling strategy, leveraging semi-structured data to boost dense retrieval efficiency. Experimental results from two datasets affirm that our method surpasses previous state-of-the-art retrieval techniques.

Topic-DPR: Topic-based Prompts for Dense Passage Retrieval
Qingfa Xiao', Shuangyin Li Lei Chen
South China Normal University
The Hong Kong University of Science and Technology
The Hong Kong University of Science and Technology (Guangzhou) qingfaxiaoêm. scnu. edu. cn, shuangyinli@scmm edu.cn, leichen@cse. ust. hk
Abstract
Deep prompt luning with a single prompt
parsages cillreetngge
Prompt-bases learning's efficacy across numer- ous natural language processing tasks has led to its integration into dense passage retrieval. Prior research has mainly focused on enhancing the semantic understanding of pre-trained lan- guage models by optimizing a single vector as a continuous prompt. This approach, however, leads to semantic space collapse; identical semantic information seeps into all represen- tations, causing their distributions converge in a restricted region. This hinders differentia- tion between relevant and irrelevant passages during dense retrieval. To tackle this issue, present Topic-DPR, dense passage retrieval model that uses topic-based prompts. Unlike the single prompt method, multiple topic-based prompts are established over a probabilistic simplex and optimized simultaneously through contrastive learning. This encourages represen- tations to align with their topic distributions. improving space uniformity. Furthermore, introduce novel positive and negative sam- pling strategy, leveraging semi-structured data boost dense retrieval efficiency. Experimen- tal results from two datasets affirm that our method surpasses previous state-of-the-art re- trieval techniques.
peosages
Contractive learning without prompt
Topic- DPR with multiple prompts
Figure Anisotropic issue of deep prompt tuning with single prompt. Topical information in prompts aids in distinguishing irrelevant passages in our Topic-DPR.
prompt as input to augment the performance of Pre- trained Language Models (PLMs). Prompts, typi- cally discrete text templates with task-specific infor- mation, need explicit definitions. To circumvent lo- cal optima during tuning, researchers (Li and Liang, 2021; Liu al., 2021b) suggested deep prompt tuning that trains single vector as continuous prompt. This approach demonstrated effectiveness and flexibility in text-generation tasks. Inspired by deep prompt tuning, recent research (Tang et al., 2022; Tam et al., 2022) has integrated con- tinuous prompts into retrieval tasks. By adding task aas-sfcccfic semantic information input, these prompts improve PLMs' knowledge utilization and guide PLMs produce more relevant text represen- tations. Consequently, relevant passage represen- tations are likely closer to the query, thus securing higher rankings.
1 Introduction
Dense Passage Retrieval (DPR), due to its efficacy and efficiency, has gained significant attention re- cently (Karpukhin et al., 2020; Lee et al., 2020). DPR encapsulates semantic information of queries and passages within fow-dimensional embedding space and measures relevance using cosine dis- tance.
Prompt-based learning an effective emerging technique for multiple natural language processing tasks (Liu et al., 2021 la; Lester et al., 2021; Wang et al., 2022a). This technique uses a task-specific
However, past research has not fully addressed the limitations of single prompt when dealing with the diverse semantics of comprehensive dataset. The engagement of all passages with a singular prompt induces a uniform semantic shift.
*Corresponding author: Both authors Qingfa Xiao and Shuangyin Li contributed equally to this research.

---

Imagine using single prompt like "Explain the main concept of [QUERY]." " where [QUERY] is the actual query. While this prompt may effectively steer the Pre-trained Language Model (PLM) to generate representations capturing aspects of each discipline, it may overlook the diverse semantics and terminologies across fields. Consequently, it could have difficulty differentiating articles from distinct disciplines. A single prompt might fall short of capturing the nuances within each disci- pline, like subtopics or specialized areas. Employ- ing one prompt for all disciplines may result in a uniform semantic shift and a convergence of pas- sage representations in a restricted region of the embedding space, as depicted in Figure 1. This semantic space collapse (Li et al., 2020; Gao et al.. 2019; Xiao et al., 2023) can blur the distinction between relevant and irrelevant passages, poten- tially masking irrelevant passages amidst relevant ones. Therefore, prompt generation pivotal for this semantically nuanced task. Further analysis is conducted in Section 5.
reflecting the dataset's diverse semantics through data-driven approach. We propose unique prompt generation method that utilizes topic mod- eling to establish the number and initial values of the prompts, which we term topic-based prompts. These prompts are defined within a probabilistic simplex space (Patterson and Teh, 2013), initial- ized using topic model such hierarchical Latent Dirichlet Allocation (hLDA) (Griffiths al., 2003). Moreover, we propose a loss function based on con- trastive learning to preserve the topic-topie relation- ships of these prompts and align their topic distribu- tions within the simplex. The impact of topic- based prompts serves as a pre-guidance for the PLMs, di- recting representations towards diverse sub-topic spaces. For dense retrieval, we consider query sim- ilarities and design tailored loss function cap- ture query-query relationships. We use contrastive learning to maintain query-passage relationships, maximize the similarity between queries and rele- vant passages, and minimize the similarity between irrelevant pairs. Considering the semi-structured nature of the datasets, we also introduce an in batch sampling strategy based on multi-category informa- tion, providing high-qualits positive and negative samples for each query during fine-tuning.
In this paper, we explore the use of multiple con- tinuous prompts to address the anisotropic issue of deep prompt tuning in dense passage retrieval, a persisting challenge. Challenge 1: The effec- tive generation of multiple continuous prompts based on the corpus. A simple approach is par- titioning the dataset into subsets via topic model- ing, each sharing a common topic-based prompt. This strategy allows the distribution of latent top- ics across a probabilistic simplex and unsupervised extraction of semantics (Blei et al., 2003; Li et al., 2022b), enabling the definition and initialization of distinct, interpretable topic-based prompts. Chal- lenge 2: The integration of tapic-based prompts into the Pre-trained Language Models (PLMs). Although our topic-based prompts are defined on a probabilistic simplex using topic modeling, en- suring topical independence, constructing such a simplex and learning topical knowledge within the PLMs' embedding space presents challenge due to inherent model differences. As result, we make the topic-based prompts trainable and adopt contrastive learning (Chen et al., 2020; Gao et al., 2021) for optimizing topical relationships.
The efficacy of our methods is confirmed through comprehensive experiments, emphasizing the rolc of topic-based prompts within the Topic- DPR framework. The key contributions are:
1.
We propose an unsupervised method for con- tinuous prompt generation using topic model- ing, integrating trainable parameters for PLMs adaptation.
2. We introduce Topic-Topic Relation, novel prompt optimization goal. It uses contrastive learning to maintain topical relationships, ad- dressing the anisotropic issue traditional deep prompt tuning.
3. Our framework supports the simultaneous use and fine-tuning of multiple prompts in PLMs, improving passage ranking by producing di- verse semantic text representations.
Related Work
To tackle these challenges, we introduce novel framework, TopR that efficiently incorpo- rates topic-based prompts into dense passage re- trieval. Instcad of artificially inflating the num- ber of prompts, we aim to define a prompt set
2.1 Dense Passage Retrieval
Recent advancements in PLMs such as BERT (De- vlin et al., 2018), Roberta (Liu et al., 2019), and GPT (Brown et al., 2020) have enabled numerous

---

unsupervised techniques to derive dense represen- tations of queries and passages for retrieval. These approaches primarily use a Bi-Encoder structure to embed text in a low-dimensional space and learn similarity relations via contrastive learning, con- trasting traditional sparse retrieval methods like BM25 or DeepCT (Robertson et al., 2009; Dai and Callan, 2019). DPR (Karpukhin et al., 2020) pi- oneered unsupervised dense passage retrieval framework, affirming the feasibility of using dense representations for retrieval independently. This ef- ficient and operational approach was further refined by subsequent studies (Xiong et al., 2020; Gao and Callan, 2021; Ren et al.. 2021; Wang et al., 2022b) that focused on high-quality negative sample min- ing, additional passage relation analysis, and extra training. The essence of these methods is to repre- sent texts in a target space where queries are closer relevant and distant from irrelevant passages.
3 The proposed Topic-DPR
3.1 Problem Setting
Consider a collection of M documents represented as D = {[T, A1, C1), ... (Tar, Aas. Cı)}- where each 3-tuple (Ti, Ar, C,) denotes a docu- ment with title Tj an abstract A,, and set of multi-category information C.. The objective of dense passage retrieval is to find relevant passages A, for given query T, where their multi-categon information sets intersect, denoted C2 n CCC.
3.2 Topic-based Prompts
The principal distinction between our Topic-DPR and other prompt-based dense retrieval methods lies using multiple topic-based prompts to en- hance embedding space uniformity and improve retrieval performance. The idea behind creating Sopic-baset prompts is to assign each document unique prompt that aligns with its semantic and topical diversity. We semantics, defined by the topic distributions within the simplex space, initialize the count and values of the topic-based prompts.
2.2 Prompt-based Learning
As PLMs, such as GPT-3 (Brown et al., 2020), continue to evolve, prompt-based learning (Gu et al., 2021: Lester et al., 2021; Qin and Eisner, 2021; Webson and Pavlick, 2021) has been in- troduced to enhance semantic representation and preserve pre-training knowledge. Hence, for vari- ous downstream tasks, an effective prompt piv- otal. Initially, discrete text templates were manu- ally designed as prompts for specific tasks (Gao et al., 2020; Ponti et al., 2020; Brown et al., 2020), but this could lead to local-optimal issues due to the neural networks' continuous nature. Ad- dressing this, Li and Liang (2021) and Liu et al. (2021b) highlighted the universal effectiveness of well-optimized prompt tuning across various model scales and natural language processing tasks.
We use topic modeling to reveal concealed meanings by extracting topics from a corpus, defined as explained in Appendíx. Topies on probabilistic simplex (Patterson and Teh, 2013), connecting documents and dictionary words via interpretable probabilistic distributions. We employ hierarchical Latent Dirichlet Allocation (hLDA) (Griffiths et al., 2003), a traditional topic modeling approach, to construct the topic-based prompts. hLDA provides comprehensive repre- sentation of the document collection and captures the hierarchical structure among the topics, which is crucial for seizing the corpus's semantic infor- mation diversity.
Recent studies have adapted deep prompt tuning for downstream task representation learning. Prom- CSE (Jiang al., 2022) uses continuous prompts for semantic textual similarity tasks, enhancing universal sentence representations and accommo- dating domain shifts. Tam et al. (2022) introduced parameter-efficien prompt tuning for text retrieval across in-domain, cross cross-domant domain, and cross-topic settings, with P-Tuning v2(Liu et al., 2021b) ex- hibiting superior performance. DPTDR (Tang et al., 2022) incorporates deep prompt tuning into dense passage retrieval for open-domain datasets, achiev- ing exceptional performance with minimal parame- ter tuning.
As shown in Figure 2, hLDA defines hierarchical topic distributions of documents and distributions words, enabling the generation of topic-based prompts from all hidden topies and corresponding topie words. hLDA creates a hierarchical K topic tree with levels; each level comprises multiple nodes, each representing a specific topic. This hierarchical structure allows our method to adapt to varying levels of granularity in the topic space, yielding more targeted retrieval results.
Let A k signify the probabilistic simplex. After uncovering K topics from the corpus, the topic dis- tribution g() of each document d, in A defined

---

Toplc Distributions
Decument Corpus
medeldata. detildddo...ettte.o.... www.w//wwww.ccc boyessan
A1
Prompt Encoder
Au
Continusus Vectore
Liner [fiddienSSte. Middon Sios]
Kg Topic Prompts
Figure 2: The definition of the topic-hased prompts. We model the document corpus using topic modeling obtain topic distributions. Higher-level K, topics from the hierarchy are then inputted into the prompt encoder to construct topic-based based prompts. Notably, the parameters of the linear layers within the encoder will be optimized.
as:
distributions share the same prompt. For the con- trastive learning fine-tuning phase, the PLMs can clearly distinguish simple negative instances with different prompts and focus more on hard negatives with identical prompts.
9 E Ax EAx= (e 00
cx) E R
(I)
where Cj signifies the k-th topic's component of d,. Our topic-based prompts aim to disperse docu- ment representations to alleviate semantic space collapse. For these prompts, maintaining signif- icant semantic differences is vital. We only use higher-level K, topics, subset of K topics, to form these prompts. These high-level K, topics are distinctly unique and suitable for defining prompts. Using hLDA, all documents are assigned to one more topics from the subset in an unsupervised manner, enabling similar documents to share the same topics. This approach enables our method to capture the corpus's inherent topic structure and deliver more accurate and diverse retrieval results. Each topic is (f1, tk.) can be inter- preted as a dictionary subset by the top L words with the highest probabilities in t. defined as g(k)_ = {wi,-.. ... w. We utilize these top L words (topic words) to generate each Topic-based Prompt. We then propose prompt encoder Eg to embed the discrete word distribution (kk), i.e., token ids, into continuous vector V/ (3(k)). assist- ing PLMs in avoiding local optima during prompt optimization. As shown Figure 2, Ee primar- ily comprises a residual network. The embedding layer preserves the topic words' semantic informa- tion, and the linear layer represents the trainable parameters O The prompt encoder generates each vector V22 {V1, .... Vk. } the representation of the topic-based prompt based on topic 4x- Dur- topic- based prompt p() (Vi, ..., Vk.) generated by the topic f(0, where the document has the highest topic component. Documents with similar topic
3.3 Topic-DPR with Contrastive Learning
3.3.1 Deep Prompt Tuning
To incorporate our topic-based prompts into the PLMs, we utilize the P-Tuning V2 (Liu et al., 2021b) methodology to initialize trainable prefix matrix M, dimensions dim (num 2 dim), where dim denotes the hidden size, correspond- ing to our topic-based prompts Figure 2, num refers to the transformer layers count, and 2 repre- sents a key vector K and value vector V. These dimensions specifically support the attention mech- anism (Vaswani et al., 2017), which operates on key-value pairs and needs to align with the trans- former's hidden size and layer structure.
As illustrated in Figure (middle), we encode the title T, and the assigned prompt p(i) as the query & = Attention (M PT)), and the abstract along with its prompt as pas- sage Pi
[M PA.)s Each self-attention computation A; of Attention Iireention[((PP)))) (P(()), PLMs(T,)) can be for- mulated as:
K Kyrompty V Vprompt M((()),
Qinputy Kimput, Vinput PLMssTT),
(2)
K [Kprompt: Kinpus], [Vpromps: Vinpue],
A. Softmaz Cooftaaz(Qimpuaae KVV,
ing retrieval, document d; is assigned
This calculates a weighted sum of input embed- dings, i.e., M((()) and PLMMs(;;,, M((()) and PLM s(A)), based on their contextual relevance. Our approach uses the attention mechanism to

---

Topic Distributions
Deep Prompt Tuning
Contrastive Learning Simplex
margin
positive instancee
negalliee instanoe
Atlention
y
query query
Tibe
Abcovee
Sun lood
hert
Posd
Fomatk
Prefix Matrix
Pretrained Language Model
Figure 3: The Jopic-DPR framework comprises three main components. First, we associate the document titles or abstracts with topic-based prompts based on their topic distributions (left), Secondly, during the deep prompt tuning phase, the prefix matrix houses the parameters for these prompts, and pre-trained language model serves as the encoder for titles or abstracts, with the attention mechanism facilitating inter-layer output interactions (middle). Lastly, we introduce three contrastive learning objectives group relevant queries and passages on simplex for efficient dense retrieval tasks (right).
amalgamate topic-based prompts with PLM em- beddings, enabling focused attention on significant semantic aspects of the input. This facilitates dy- namic adjustment of embeddings based on topic- based prompts, leading to more contextually perti-
where y the temperature hyperparameter, and s(-) denotes the cosine similarity function. We define p(.) as the correlation coefficient of the positive pairs, which is discussed in Section 3.3.3.
Query-Passage Relation. Different from the Eq. 3, the query-passage similarity relation regards the query qi as the center and pushes the negative passages [Pi55 J j=1
nent representations.
Typically, use the first token (CI.S) of out- put vectors as the query or passage representa- tion. Unlike the Prefix-tuning approach, Topic- DPR simultaneously employs and optimizes multi- ple prompts, enhancing the model's ability to cap- ture the corpus's diverse semantics, thereby im- proving retrieval performance.
B farther than the positive pas-
sages { {Pt ==1 function as the negative log- log-likeldhood of the posi- tive passage:
Formally, we optimize the loss
Lock (s (s.. (Pa
3.3.2 Contrastive Learning
Topic-DPR aims to learn the representations of queries and passages such that the similarity be- tween relevant pairs exceeds that between irrele- vant ones. Here, we classify the contrastive loss into three categories: query- and topic-topic relations.
plun vt. log (laip) 11 Eg_ plu.r
(4)
Since the objective of the dense passage retrieval task is to find the relevant passages with a query, we consider that the relation of query-passage sim- ilarity is critical for the Topic-DPR and the Eq.3 is an auxiliary to Eq.4
Query-Query Relation. The objective of learn- ing the similarity in the query-query relation in- crease the distance between the negative query 7; and the query g. while enhancing the similarity be- tween the positive query 4,t and the query 9:- Given a query 4 with m positive queries { [ -= and ت-ز
Topic-Topic Relation. The motivation for opti- mizing multiple topic-based based prompts lies in the fact that a set of diverse prompts can guide the represen- tations of queries and passages toward the desired topic direction more effectively than a fixed prompt. However, with prompts distributed across K, top- ics, the margins between them are still challeng- ing to distinguish using conventional fine-tuning methods. Consequently,
n negative queries { [i.j j=1 we optimize the loss function as the negative log- log-likelihood of the query:
aim to enhance the diversity of these topic-based prompts in the em- bedding space through contrastive learning better match their topic distributions. Given a batch of passages {A we encode them into the PLMs using K₂ topic-based prompts V and generate
loss (a. (at, 1m (o,,
171 E pla. st.a) clannat at ./rr, Σi ((..7)))/""
1
(3)

---

Kg x N passages {pl , }.
PN f. where p Attention[ (V.), (A)) We pro- pose loss function for each prompt V& designed push the other prompts (V2).sh away with the assistance of passages, as formulated below:
(1 -2a)
los81otal
N
loss lossin
ta catdptt opt -
N
loss 10s8/00 [ 2- fo7, for.
N
loss We. VV..VVl 155) 4 (K, 1)N2
(5)
+
K,
loss (V2. (V.)
max (M- s(pi p$) (3.00.
where a hyper-parameter weight losses.
In this function, M is the margin hyper- parameter, signifying the similarity discrepancy among passages across various topies. It is premised on the belief that unique prompts can steer the same text's representation towards mul- tiple topic spaces. Consequently, Pretrained Lan- guage Models (PLMs) can focus relationships among instances with identical prompts and dis- regard distractions from unrelated samples with differing prompts. The additional prompts impose constraints on the PLMs, spreading pertinent in- stances over diverse topic spaces. This approach explains our exclusive use of higher-level topics from hLDA for topic-based prompt definition (Sec- tion 3.2).
Experiments
4.1 Experimental Settings
Datasets We evaluate Topic-DPR's retrieval perfor- mance through experiments on two scientific doc- ument datasets: the arXiv-Article (Clement et al.. 2019) and USPTO-Patent datasets (Li et al., 2022a). For dense passage retrieval, extract titles, ab- stracts, and multi-category information from these cemi-structured datasets, using titles as queries and relevant abstracts as optimal answers. Appendix details these datasets' statisties.
Evaluation Metrics Considering the realistic liter- ature retrieval process and dataset passage count, we use Accuracy (Ace@) I, 10), Mean Reciprocal Rank (MRR @ 100), and Mean Average Precision (MAP@ 10, 50) for performance assessment. Fur- thermore, apply representation analysis tool to gauge the alignment and uniformity of the PLMs embedding space (Wang and Isola, 2020).
3.3.3 In-batch Positives and Negatives
For dense retrieval, identifying positive and nega- tive instances vital for performing loss functions Eq.3 and Eq.4. In our approach, we feed batch of N documents into the PLMs per iteration and sample positives and negatives from this batch. Im- portantly, we employ multi-category information from these documents pinpoint relevant queries passages, aligning with our problem's objec- tive. Queries or passages sharing intersecting multi- category information are considered positive. The correlation coefficient a positive pair (gi. o²)) can be expressed as:
Baselines For a baseline comparison, we em- ploy the standard sparse retrieval method, BM25 (Robertson et al., 2009). The efficacy of our proposed techniques is assessed against DPR and DPTDR, adapted to our datasets. DPR, an advanced dual-encoder method, incorporates trastive learning for dense passage retrieval, while DPTDR, contemporary leading method using deep prompt tuning. employs a continuous prompt to boost PLMs' retrieval efficiency. Due to the ab- sence of specific positive examples for each query, we apply the positive sampling approach across all techniques to guarantee fair comparisons.
pla. a)) |anc
This parallels the positive pair in the query- passage relation. By default, all other queries or passages in the batch are deemed irrelevant.
Implementation Details We initialize our Topic- DPR parameters using two uncased PLMs, BERT- base, and BERT-large, obtained from Hugging- face. All experiments are executed on Sentence- Transformer with an NVIDIA Tesla A100 GPU. Appendix details all the papar-parameters.
3.4 Combined Loss Functions
this section, we combine the three rela- tions presented above to obtain the combined loss function for fine-tuning over each batch of
(((g, pc) W {v

---

Methods
PLM Frozen
Acc@1 1
Acc@ 10
MRR@ 100
MAP@10
MAP@50
BM25
80.20/80.72
96.34/96.70 70
85.98/86.41
31.48/35.38
16.18/20.26
Experiments on BERT-base
DPR
90.98/88.96
98.26/97.87
93.59/92.27
57.86/54.70
49.53/44.53
DPTDR
x
90.75/88.62
97.68/97.73
93.43/92.04
58.57/55.66
50.05/45.71
Topic-DPR
X
91.40/90.29
98.43/98.32
94.21/93.26
61.57/58.08
52.69/48.63
DPTDR
V
88.00/86.11
98.02/97.06
91.82/90.29
50.83/50.04
38.45/37.92
Topic-DPR
89.19/87.41
98.13/97.63
93.01/91.16
53.19/52.68
41.39/40.51
Experiments on BaRT-large
DPR
91.44/89.06
98.68/97.88 .88
94.37/92.42
58.92/56.28
51.44/46.07
DPTDR x
91.12/87.93
98.67/97.61
94.01/91.61
59.62/57.50
52. 16/47.3
Topic-DPR
X
91.98/91.04
98.72/98.50
94.50/93.31
63.01/61.18
54.71/55..29
DPTDR*
90.25/87.32
98.34/97.44
93.35/91.22
54.09/52.89
41.79/41.18
Topic-DPR*
91.03/90.55
98.40/98.40
93.76/92.84
56.83/54.25
45.96/43.32
Table 1: Experimental results the arXiv-Article and USPTO-PPaent datasets in dense passages retrieval tasks
(results presented as arXiv-Article/UUSPTOO Prro-ttten)..... And indicates that only the parameters of prompts can be tuned and the parameters of PLM are frozen.
4.2 Experimental Results
Methods DPR
Aec@ 10
MRR@ 100
MAP@50
98.26
93.59
49.53
Table I presents our experiments' outcomes using BERT-base and BERT-large models the arXiv- Article and USPTO-Patent datasets. In compari- to sparse methods, dense retrieval techniques show significant performance improvement, em- phasizing dense retrieval's s importance. When con- trasting DPTDR with DPR, DPTDR exhibits supe- rior performance in the MAP@ 10 and MAP@50 metrics due to its continuous prompt enhance- ment. Our topic-based prompts in Topic-DPR boost the Acc and MRR metries. Furthermore, Topic-DPR outperforms baseline methods across all metrics. Specifically, our Topic-DPRbase ex- ceeds DPTDRbase by 3.00/2.42 and 2.64/2.98 points in MAP@ 10 and MAP@50, which are vital for large multi-category passage retrieval. Addi- tionally, in the deep prompt tuning setting, our Topic-DPR*. despite slight performance degrada- tion, still maintains comparative performance with only 0.1 %-0.4% of the parameters tuned. The con- sistent enhancements across diverse settings, mod- els, and metrics manifest the robustness and effi- ciency of our Topic-DPR method. This research establishes ToP as an effective deep prompt learning-basee dense retrieval method, setting a new state-of of-thh-rrt for the datasets. Ablation ex- periments are conducted in Appendix.
DPR with random words
96.24
89.19
48.64
DPR with topic words
98.30
93.73
50.19
Topic-DPR
98.43
94.21
52.69
Table 2: Quality analysis of the topic words on the test set of the arXiv-Article dataset. DPR with topic words indicates that each example in the training data has the corresponding topic words added as prompts. DPR with random words indicates that each example in the training data has random topic words added as prompts.
5.1 Quality of Topic Words
To assess whether the topic words extracted from hLDA are helpful for dense passage retrieval tasks, we conducted an experiment that directly used the topic words as prompts to train DPR with BERT- base. Each query was transformed into "[TOPIC WORDS...] + [QUERY]" and each passage transformed into "[TOPIC WORDS... + [PAS- SAGEJ". As shown in Table 2, when noise in- troduced, the model experiences more interference, resulting in decreased performance. This indicates that simply adding random words to the queries and passages is detrimental to the model's ability to dis- cern relevant information. However, with the help of the topical information extracted from hLDA, the model performs better than the original DPR. This suggests that introducing high-quality topic words is beneficial for domain disentanglement, allows the model to better differentiate between various subject areas and focus on the relevant con-
5 Analysis Topic-DPR

---

(a) BERT with 5 topics.
(b) Topic-DPR with 5 topics. (c) BERT with 412 categories. (d) Topic-DPR with 412 cate- gories.
Figure 4: T-SNE visualization ( (Van der Maaten and Hinton, 2008) of the representations from the vanilla BERT-bas bbss and the Topic-DPR-bal -assee. Each point represents one passage under specific label, and each color represents one class. The two types labels we use are the five topics generated by the hL.A and the 412 categories in the USPTO-Patent dataset.
text. By incorporating topic words as prompts, the model is able to generate more diverse and accurate semantic representations, ultimately improving its ability to retrieve relevant passages.
Methods
Align(q.p") Uniform(q)(()) Sim(g.p") /(q.p
BERT
0.73
-0.72/-0.98
63.71 /58.45
DPR
0.47
-2.60/ -2,22
78.90 29.46
DPTDR
0.42
-2.50/ -2.13
79.89 34.57
Topie-DPR
0.38
-3.08/ -3.02
80.67/ 14.81
5.2 Representations with Topic-based Prompts
Table Quality analysis of the representations gener-
ated by different methods in BERT-base. 'hee quality better when all the above numbers are lower except Sim(q.p**..
We visualize the representations with different types of labels and analyze the influence of topic- based prompts intuitively. As shown in Figure 4(a), the passages distributed in Ka topics overlap in the vanilla BERT. As depicted in Figure 4(b), they are arranged in pentagonal shape, resembling the probabilistic simplex of the topic model Aк. Here, each topic is independent and discrete, with clearly distinguishable margins between them. This out- come is primarily due the topic-based prompts, which are initialized by the topic distributions and direct the representations to exhibit significant topi- cal semantics. During the dense retrieval phase, the passages in sub-topic spaces are more finely dif- ferentiated using loss functions Eq.3 and Eq.4. As illustrated in Figure 4(d), passages belonging to the same category cluster together, enabling queries to identify relevant passages with greater accuracy. These observations suggest that our proposed topic- based prompts can encourage PLMs to generate more diverse semantic representations for dense retrieval.
cally, the alignment measures the expected distance between the representations of the relevant pairs (Ir, t+::
Align(z, x²)
(M.. (alaast)
where the f(x the I.2 normalization function. And the uniformity measures the degree of unifor- mity of whole representations Pdata
Uniform(Pam)
log 2.
As demonstrated in Table 3, the results of all dense retrieval methods surpass those of the vanilla BERT. Comparing DPR with DPTDR, the latter exhibits better aligmment performance yet poorer uniformity, with representations from DPR display- ing significant differences average similarity between relevant and irrelevant pairs. This phe- nomenon highlights a shortcoming of deep prompt tuning in dense retrieval, where the influence of a single prompt can lead to anisotropic issues in the embedding space. Furthermore, this observation can explain why DPTDR underperforms DPR in certain metrics, as discussed in Section 4.2. Regard- ing the results of our Topic-DPR, the alignment of representations decreases to 0.38, marginally
5.3 Alignment and Uniformity
We experiment with 5,000 pairs of queries and passages from the USPTO-Pates datasct's 's devel- opment analyze the quality of representations in terms of three metrics, including alignment, uni- formity, and cosine distance. Alignment and uni- formity are two properties to measure the quality representations (Wang and Isola, 2020). Specifi-

---

better than DPTDR. Importantly, our method sub- stantially enhances uniformity, achieving the best scores and indicating greater separation between relevant and irrelevant passages. Thus, our Topic- DPR effectively mitigates the anisotropic issue of the embedding space.
Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 18:117771990.
Ting Chen, Simon Korbblihh, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. ternational conference on machine learning, pages 1597-1607. PMLR.
6 Conclusion
Colin B Clement, Matthew Bierbaum, Kevin P O'Keeffe, and Alexander Alemi. 2019. On the use of arxiv as dataset. arXiv preprint 1eXivv1905.0000
this paper, we examine the limitations of us- ing single task-specific prompt for dense pas- sage retrieval. To address the anisotropic issue in the embedding space, we introduce multiple novel topic-based prompts derived from the corpus's 's se- mantic diversity to improve the uniformity of the space. This approach proves highly effective in identifying and ranking relevant passages during retrieval. We posit that the strategy of generating data-specific continuous prompts may have broader applications in NLP, as these prompts encourage PLMs to represent more diverse semantics.
Zhuyun Dai and Jamie Callan. 2019. Deeper text un- derstanding for ir with contextual neural language modeling. Proceedings of the 42nd International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 985-988.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training deep bidirectional transformers for language understand- ing. arXiv preprint arXiv: 1.1.:0880000003
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv: 1907. 12009.
Limitations
Our method achieves promising performance to en- hance the semantic diversity of representations for dense passage retrieval, but we believe that there are limitations be explored for future works: (1) The topic modeling based the neural net- work may be joint trained with the dense passage retrieval task, where the topies extracted for our topic-based prompts can be determined to comply with the objective of retrieval automatically. (2) The possible metadata of documents, like authors. sentiments and conclusions, can be considered to assist the retrieval of documents further.
Luyu Gao and Jamic Callan. 2021. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. arXiv preprint urXiv:2108.0555
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020.
Making pre-trained language models better few-shot learners. arXiv preprint 20Xi::0012.1522..
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv: 04.8888.
Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. 2003. Hierarchical topic models and the nested chinese restaurant process. Advances in neural information processing systems, 16.
Acknowledgements
This work was supported by National Natural Science Foundation of China (No. 62006083), Natural Science Foundation of Guangdong (2023A1515012073) and National Key Re- search and Development Program of China (2000YFA0712500).
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2021. Ppt: Pre- Pre-trained prompt tuning for few-shot learning, arXiv preprint arXiv: 1ae:22109003333
Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022. Im- proved universal sentence embeddings with prompt- based contrastive learning and energy- based learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3021-3035.
References
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
David M Blei, Andrew Ng. and Michael Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research. 30an))993-1022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2020. Learning dense representations of phrases at scale. arXiv preprint arXiv:2012.1 12624.

---

Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-eftie efficieent prompt tuning. arXiv preprint arXiv:21044 2ee.21444489))
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 3(4)3333-38.
Bohan Li. Hao Zhou, Junxian He. Mingxuan Wang. Yiming Yang, and Lei Li. 2020. On the sentence em- beddings from pre-trained language models. arXiv preprint arXiv:2011.0580
Weng Lam Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Xingjian Zhang, Yuxiao Dong. Jiahua Liu, Maodi Hu, and Jie Tang. 2022. Parameter-efficien prompt tuming makes generalized and calibrated neural text retrievers. arXiv preprint T.Xiv:2207.07005
Huahang Li, Shuangyin Li, Yuncheng Jiang, and Gansen Zhao. 2022a. Copate: A novel contrastive learning framework for patent embeddings. In Pro- ceedings of the 3 ACM International Conference on Information Knowledge Management, pages
Zhengyang Tang, Benyou Wang, and Ting Yao. 2022. Dptdr: Deep prompt tuning for dense passage trieval. arXiv e-prints, pages arXiv-2205,
1104-1113.
Laurens Van Maaten and Geoffrey Hinton, 2008 Visualizing data using t-sne. Journal of machine learning research, 9(11).
Shuangyin Li, Haoyu Luo, Gansen Zhao, Mingdong Tang, and Xiao Liu. 2022b. bi-directional bayesian probabilistic model based hybrid grained semantic matchmaking for web service discovery. World Wide Web, 25(2)445-470
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation, arXiv preprint arXiv: 201.00000000
Pengfei Wang, Xiaocan Zeng, Lu Chen, Fan Ye. Yuren Mao, Junhao Zhu, and Yunjun Gao. 2022a. Promptem: prompt-tuning for low-resource generalized entity matching. arXiv preprint arXiv:2207.04800 0..002.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pre- train, prompt, and predict: systematic survey of prompting methods in natural language processing. arXiv preprint ar arX 13586.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021b. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, arXiv preprint Xrv:: 21 10.00602.
Yifan Wang. Haodi Ma, and Daisy Zhe Wang. 2022b. Lider: An efficient high-dimensional learned index for large-scale dense passage retrieval. arXiv preprint 0rXiv:2205.00970
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Albert Webson and Ellie Pavlick. 2021. Do prompt- based models really understand the meaning their prompts? arXiv preprint arXiv: 210.01277.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1 1907. 1692.
Sam Patterson and Yee Whye Teh. 2013. Stochastic gradient riemannian langevin dynamics the proba- bility simplex. Advances in neural information pro- cessing systems, 26.
Qingfa Xiao, Shuangyin Li, and Lei Chen. 2023. Iden- tical and Fraternal Twins: Srnened Semantic Contrastive Learning of Sentence Representations.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Amoll Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint 2007002007.00800
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. Xcopa: A multilingual dataset for causal common- sense reasoning. arXiv preprint 200033005.00033.
Guanghui Qin and Jason Eisner. 2021. Learning how ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2104.0665 166.9999.
Ruiyang Ren, Shangwen Lv, Yingqi Qu. Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. Pair: Lever- aging passage-centric similarity relation for im- proving dense passage retrieval. arXiv preprint arXiv:2108.0601


Reviews:
[{"review_id": "SXA1yMsCFA", "reviewer": "Reviewer_P8b2", "summary": "", "questions": "1. It would be better to understand the importance of three contrastive loss if there are some ablation study on it. \n2. Why is performance difference between model and baselines lower in Acc@K than MAP@K?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> For Question 1:\n It would be better to understand the importance of three contrastive loss if there are some ablation study on it.\n\nAnswer Q1: \nWe conducted an ablation study on three types of contrastive losses. Due to the length limitation of the paper, we have included them in the appendix. \n\n\n> For Question 2:\nWhy is performance difference between model and baselines lower in Acc@K than MAP@K?\n\nAnswer Q2: \n1. Firstly, for the baseline models, the Acc@K score already lies between 90%-98%, a notably high accuracy level. With such elevated accuracy, achieving any further significant improvements becomes extremely challenging, and a smaller performance gap is anticipated.\n\n2. In retrieval scenarios, Acc@K primarily focuses on the presence of the correct answer within the top K results, without giving due consideration to the specific ordering of these answers. This renders Acc@K a relatively simplistic evaluation metric, unable to holistically represent the overall ranking quality of retrieval results. Consequently, when juxtaposed against baselines, our model finds it challenging to demonstrate a marked difference in Acc@K.\n\n3. In contrast to Acc@K, MAP@K is considerably more stringent. It not only emphasizes the presence of the correct answers but also places significant importance on their order. This ensures that MAP@K provides a more granular assessment of the retrieval system's performance. The substantial improvement we observed in MAP@K underscores the effectiveness and superiority of our proposed method.\n\n\n> For Rejection 2:\nThe novelty of this paper is limited to incorporating an existing topic model with prefix tuning approach.\n\nAnswer R2:\n1. Our approach is effective and non-trivial not only because of the combination of topic modeling and prefix fine-tuning, but also because we designed a contrastive learning training objective for inter-topic relations based on extracted topic prompts and retrieval scenarios, as shown in Formula 5 Topic-Topic Relation. There are two additional experiments that prove this point: Firstly, Section 5.1 demonstrates the effectiveness of topic cues but with some gaps compared to our method due to Topic-Topic Relation; Secondly, the ablation experiment in the appendix also proves the effectiveness of this training objective.\n\n2. According to Table 2, the DPR with topic words outperforms the simple DPR, and they utilize the same loss function. However, there still exists a gap when compared to Topic-DPR. In response to Reviewer ZAsx's question, \"why is the simple DPR with topic words inferior to Topic-DPR? Is the gap attributed to the additional parameters from the trainable matrix M?\", we offer the following analysis:\n\n    2.1 Firstly, the trainable matrix M proves effective for training but without topic words. DPTDR serves as an example of this. It enhances the DPR by introducing additional parameters from M and uses a fixed set of vectors as input, which are randomly determined.\n\n    2.2 Secondly, without designing a specific loss function, the trainable matrix M becomes ineffective or even detrimental when integrating multiple sets of topic words. Firstly, since our trainable matrix M is randomly initialized and hasn't undergone Masked Language Model training, it diverges from DPR w/ topic words. The latter can recognize topic words due to its foundational weights. In contrast, the trainable matrix  M lacks semantic comprehension for multiple sets of topic word vectors. In other words, inputting multiple sets of topic word vectors is equivalent to inputting multiple sets of random vectors. For DPR with trainable matrix M and topic words (i.e. DPTDR with topic words), the trainable matrix M faces greater challenges during fitting, resulting in performance inferior to that of a fixed set of random vectors (i.e. DPTDR). Directly using multiple sets of topic word vectors as inputs for the trainable matrix M doesn't make sense. Similar conclusions can be drawn from the ablation study in the appendix, specifically \"Figure 5, Topic-DPR w/o TTR\", where Topic-DPR without the Topic-Topic Relations training objection is worse than DPTDR. Thus, this performance gap stems from the contrastive learning training loss designed for topic-to-topic relations, represented by equation 5, \"topic-topic relations\". This loss aims to train a topic space that resembles a probability simplex.\n\n3. Topic modeling is an effective and transferable technique, while prefix fine-tuning is currently popular. Therefore, I believe that our work demonstrating the effectiveness of combining these two methods is also necessary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces a novel data-driven way to compute multiple topic-based continuous prompts, that help model to get rid of semantic space collapse caused by using a single prompt. The multiple prompts can capture the distinct diverse semantics information through topic-distribution in the dataset. As a result, the passage and query representation can be well distributed in the latent space, and so query relevant passages can be easier to identify. \n\n2.  The problem is well motivated.", "reasons_to_reject": "1. In the experimental results, Topic-DPR achieve marginal improvement over the performance of simple DPR model, specifically Acc@K. That implies simple way of first extracting topic words from the dataset, then using them in prompts might not be sufficient enough. \n\n2. The novelty of this paper is limited to incorporating an existing topic model with prefix tuning approach. ", "questions_to_authors": "1. It would be better to understand the importance of three contrastive loss if there are some ablation study on it. \n2. Why is performance difference between model and baselines lower in Acc@K than MAP@K?\n", "paper_topic_and_main_contributions": "This paper proposes a continue prompt tuning approach with multiple topic-based prompts in dense passage retrieval to overcome the limitation of single prompt tuning, that help PLM model to capture diverse semantic information of dataset. Hierarchical LDA is used to construct the topic-based prompts, specifically for each topic top L words with highest probabilities have been used to generated the prompt respectively. Authors adopt existing prefix-tuning technique and concatenate topic-based prompts with PLM embeddings in attention computation. Furthermore, to train the model to learn query and passages embeddings, they compute contrastive loss in three aspects: query-query, query-passage, and topic-topic.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HGLpgwgQbV", "reviewer": "Reviewer_ZAsx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank for your hard work and valuable comments. \n\n> For Rejection 1:\nSome important technical details are missing. For example, how negative query pairs and query-passage pairs are selected. If these are provided in the datasets, please specify.\n\nAnswer R1:\nQueries or passages within the same batch that share intersecting multi-category information from arxiv or patent datasets are considered positive. For instance, here are three document samples: A (categories 1, 2, and 3), B (categories 2 and 4), and C (category 5). In this case, we can consider pairs such as (A,A) and (A,B) as positive pairs for A while considering (A,C) as a negative pair. For a fair comparison, we have used the same strategy in all the baselines for comparison.\n\n\n> For Rejection 2:\nVery limited baseline systems (DPR and DPTDR) are compared, and the popularity of the datasets arXiv-Article and USPTO-Patent prevents the authors from comparing more systems. I would suggest evaluating additional (more popular) systems tuning by topics (e.g., TAS-B [1]) and show more convincing results. (Since your topic clustering is unsupervised, why not apply it to more popular datasets?)\n\nAnswer R2:\nThank you, our method of selecting positive samples is similar to TAS-B and we will increase the reference and comparison. From the perspective of the dataset, we used multi-class information as one of the selection criteria for positive samples, which exists in both semi-structured datasets arXiv-Article and USPTO-Patent. From a baseline perspective, since we are innovating from the perspective of prefix fine-tuning, we have chosen DPR and DPTDR as two baselines that are easy to reproduce and directly comparable. For other baseline strategies for sampling positive and negative examples, they can be directly integrated with our framework.\n\n\n> For Rejection 3:\nMissing further ablations to study why simple DPR with topic words are worse, are DPR w/ topic words using the same loss function? Is the gap due to the additional parameters from the trainable matrix M? Showing the two are compared fairly and the difference is made from representation uniformity would be desired, and it will align to the center conclusion of this paper.\n\nAnswer R3:\nAccording to Table 2, the DPR with topic words outperforms the simple DPR, and they utilize the same loss function. However, there still exists a gap when compared to Topic-DPR. In response to your question, \"why is the simple DPR with topic words inferior to Topic-DPR? Is the gap attributed to the additional parameters from the trainable matrix M?\", we offer the following analysis:\n\n1. The trainable matrix M proves effective for training but without topic words. DPTDR serves as an example of this. It enhances the DPR by introducing additional parameters from M and uses a fixed set of vectors as input, which are randomly determined.\n   \n2. Without designing a specific loss function, the trainable matrix M becomes ineffective or even detrimental when integrating multiple sets of topic words. Firstly, since our trainable matrix M is randomly initialized and hasn't undergone Masked Language Model training, it diverges from DPR w/ topic words. The latter can recognize topic words due to its foundational weights. In contrast, the trainable matrix  M lacks semantic comprehension for multiple sets of topic word vectors. In other words, inputting multiple sets of topic word vectors is equivalent to inputting multiple sets of random vectors. For DPR with trainable matrix M and topic words (i.e. DPTDR with topic words), the trainable matrix M faces greater challenges during fitting, resulting in performance inferior to that of a fixed set of random vectors (i.e. DPTDR). Directly using multiple sets of topic word vectors as inputs for the trainable matrix M doesn't make sense. Similar conclusions can be drawn from the ablation study in the appendix, specifically \"Figure 5, Topic-DPR w/o TTR\", where Topic-DPR without the Topic-Topic Relations training objection is worse than DPTDR. Thus, this performance gap stems from the contrastive learning training loss designed for topic-to-topic relations, represented by equation 5, \"topic-topic relations\". This loss aims to train a topic space that resembles a probability simplex.\n\n\n> For Rejection 4:\nNotations may be too verbose to follow, e.g., what is the difference between P(i) and Vk? Why it defines ∆K (in simplex space), but later I do not see it is used. Is beta^(k) in simplex space? In short, the paper needs some refinements on the formulas to make it easier to follow.\n\nAnswer R4:\n1. We will redefine it easier to follow. The matrix P(i) is obtained by multiplying the matrix Vk with the trainable matrix M. The \\theta(i) in ∆K help us define the topical prompts beta^(k) in such simplex space, and then built the simplex space in the bert embedding space with the topic-topic relations training objective. Beta^(k) can be considered as the vector representation of each topic in ∆K. And also beta^(k) bring the initial semantic via word embedding layer, represented by the right of figure 2. We have already used it.\n\n2. From Figure 4, it can be seen that the distribution of vectors conforms to the high-level topical distributions in simplex space ∆K as defined by us, which is closely related to our proposed training method for topic prompts.\n\n3. About the theory of probability simplex, you can refer to this article \"Sam Patterson and Yee Whye Teh. 2013. Stochastic gradient Riemannian Langevin dynamics on the probability simplex. Advances in neural information processing systems, 26.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposed a novel method that not only boosts the retrieval effectiveness, but also justified the representation improvement by alignment and uniformity experiments quantitively. The anisotropic representation is a long-standing issue raised by training deep retrievers. This makes the proposed work more sound for general interests of neural retrievers.\n\nAnother good part of this paper is successful introduction of a traditional topic modeling method for the IR domain. Using contextualized deep representations with topic modeling looks promising. ", "reasons_to_reject": "1) Some important technical details are missing. For example, how negative query pairs and query-passage pairs are selected. If these are provided in the datasets, please specify.\n\n2) Very limited baseline systems (DPR and DPTDR) are compared, and the popularity of the datasets arXiv-Article and USPTO-Patent prevents the authors from comparing more systems. I would suggest evaluating additional (more popular) systems tuning by topics (e.g., TAS-B [1]) and show more convincing results. (Since your topic clustering is unsupervised, why not apply it to more popular datasets?)\n\n3) Missing further ablations to study why simple DPR with topic words are worse, are DPR w/ topic words using the same loss function? Is the gap due to the additional parameters from the trainable matrix M? Showing the two are compared fairly and the difference is made from representation uniformity would be desired, and it will align to the center conclusion of this paper.\n\n4) Notations may be too verbose to follow, e.g., what is the difference between P(i) and Vk? Why it defines ∆K (in simplex space), but later I do not see it is used. Is beta^(k) in simplex space? In short, the paper needs some refinements on the formulas to make it easier to follow.\n\n[1] Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling -- Sebastian Hofstätter et al.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores the use of multiple prompts for more nuanced sub-topics or specialized areas. In particular, the authors adopt topic modeling techniques to generate more fine-grind topics and combine it with deep retrievers which are aware of contextualized representations. The main contributions of this paper include the introduction of topic modeling for training better retrievers by categorizing document topics and alleviating the anisotropic similarity representations generated by deep retrieval models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "J5PwTbDpKW", "reviewer": "Reviewer_q4iy", "summary": "", "questions": "Question A:\nHow can different topic mining models affect passage retrieval? (Robustness to topic modeling.)", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments.\n> For Question A:\nHow can different topic mining models affect passage retrieval? (Robustness to topic modeling.)\n\nAnswer Q-A:\n1. Currently, there are many topic mining models available, including those based on Bayesian theory and neural networks. I have also used LDA and GPT models to extract topics in our framework, and it is undeniable that they each have their own advantages. For neural network-based topic mining models, they are all trained on large-scale domain-specific data sets, making cross-domain transfer difficult. As for the Bayesian-based model, its advantage lies in probability simplex which gives our method great interpretability. Moreover, this model is highly practical and requires only low time and training costs to complete cross-domain training transfer; therefore we choose the Bayesian-based model.\n\n2. Compared with the most popular LDA model, this model requires setting many hyperparameters which may affect our framework's performance more or less. Therefore we choose hLDA (hierarchical LDA) model which inherits the advantages of LDA while having fewer hyperparameters to automatically obtain multiple hierarchical structure topics.\n\n3. From our motivation perspective, we use topic modeling to establish different subspaces for further comparative learning training to cluster data sets more accurately. The hLDA model can automatically extract K high-level topics to help define prompts that meet requirements.\n\n4. In terms of practicality and ease of operation, we use the hLDA model and demonstrate its robustness effects in datasets from different domains.\n\n> For Rejection 1:\nMore baselines are needed to validate the effectiveness of the proposed methods. Only two dense retrieval baselines are included in the experiments and other methods introduced in related work are not selected as baselines in experiments.\n\nAnswer R1:\nThank you for your feedback. We will compare more baselines to improve the traditional prefix-tuning mechanism and enhance retrieval performance. Therefore, we chose these two reproducible and directly comparable baselines. We conducted a fair comparison of the baseline model to demonstrate the advantages of our proposed topical prompt-tuning. As for other baseline strategies for sampling positive and negative examples, they can be easily integrated into our framework.\n\n> For Rejection 2:\nSome related works about dense retrieval for text are not mentioned in this paper, such as Contriever [1], GTR [2] and Instructor [3], etc.\n\nAnswer R2:\nThank you, we will cite these articles in the next version.\n\n> For Rejection 3:\nBERT in Figure 4 is not a good baseline to compare. BERT is not designed to directly represent passages and then has low-quality passage representations. Authors should at least compare the representations from DPR and DPTDR.\n\nAnswer R3:\nWe will add the graphs of DPR and DPTDR in the next version. This experiment aims to demonstrate whether the vector distribution of our model conforms to the probability simplex, which is more intuitive compared with BERT before and after training. Due to numerous categories and limited space in the paper, it is difficult to compare DPR and DRTDR directly, but their results can be observed from the experimental table.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and easy to follow.\n2. Authors propose to aggregate topics into prompts for prompt learning to improve passage retrieval. The motivation of this paper makes sense and is well-described in the paper.\n3. The results shown in Table 1 is promising. This method could achieve non-trivial improvement compared to baselines.\n", "reasons_to_reject": "1. More baselines are needed to validate the effectiveness of the proposed methods. Only two dense retrieval baselines are included in the experiments and other methods introduced in related work are not selected as baselines in experiments.\n2. Some related works about dense retrieval for text are not mentioned in this paper, such as Contriever [1], GTR [2] and Instructor [3], etc.\n[1] Izacard, Gautier, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin and Edouard Grave. “Unsupervised Dense Information Retrieval with Contrastive Learning.” Trans. Mach. Learn. Res. 2022 (2021): n. pag.\n[2] Ni, Jianmo, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang and Yinfei Yang. “Large Dual Encoders Are Generalizable Retrievers.” Conference on Empirical Methods in Natural Language Processing (2021).\n[3] Su, Hongjin, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer and Tao Yu. “One Embedder, Any Task: Instruction-Finetuned Text Embeddings.” Annual Meeting of the Association for Computational Linguistics (2022).\n3. BERT in Figure 4 is not a good baseline to compare. BERT is not designed to directly represent passages and then has low-quality passage representations. Authors should at least compare the representations from DPR and DPTDR.", "questions_to_authors": "Question A:\nHow can different topic mining models affect passage retrieval? (Robustness to topic modeling.)", "paper_topic_and_main_contributions": "This paper proposes topic-aware prompt learning for passage retrieval.\nAuthors recognize that previous works focus on only optimizing a single vector for passage retrieval, which can lead to a semantic space collapse. Hence, in this paper, they propose to use topic-based prompts for passage retrieval. Specifically, they employ a topic model to mine topics in passages. Then these topics will improve the passage retrieval by contrastive learning.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Izacard, Gautier, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin and Edouard Grave. “Unsupervised Dense Information Retrieval with Contrastive Learning.” Trans. Mach. Learn. Res. 2022 (2021): n. pag.\n[2] Ni, Jianmo, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang and Yinfei Yang. “Large Dual Encoders Are Generalizable Retrievers.” Conference on Empirical Methods in Natural Language Processing (2021).\n[3] Su, Hongjin, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer and Tao Yu. “One Embedder, Any Task: Instruction-Finetuned Text Embeddings.” Annual Meeting of the Association for Computational Linguistics (2022)."}}]

Author rebuttal:

Meta-review: This paper presents an interesting idea of topic-based prompt learning for sense passage retrieval. The idea is well motivated. The proposed model incorporates well the idea. The experiments demonstrate the strength of the proposed approach against a single prompt.

The reviewers have found the idea new and interesting. The authors have answered most of the questions raised in the reviews in their rebuttal. It is expected that some of the replies will be added into the final version.

Overall, the paper proposes a new and interesting idea for topic-dependent prompt. The paper may inspire other researchers in the future.
Decision: Accept-Findings