Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search
Authors: Kelong Mao
Zhicheng Dou
Fengran Mo
Jiewen Hou
Haonan Chen
Hongjin Qian
Venue: EMNLP 2023 Findings (2023)
Keywords: Conversational search; passage retrieval; large language models; contextual search intent understanding
TL;DR: A simple yet effective prompting framework that leverages large language models for conversational search with extensive automatic and human evaluations.

Abstract: Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites.  Our findings provide important evidence to better understand and leverage LLMs for conversational search.

Large Language Models Know Your Contextual Search Intent:
A Prompting Framework for Conversational Search
Kelong Maol!2, Zhicheng 2 Fengran Mo Jiewen Hou',
Haonan Chen', Hongjin Qian'
'Gaoling School of Artificial Intelligence, Renmin University of China Engineering Research Center of Next-Generation Search and Recommendation, MOE Université de Montréal, Québec, Canada Institute of Computing Technology, Chinese Academy of Sciences {mkl, douueruc. edu. cn
Abstract
multi-turn conversation with it in natural languages to solve their questions. However, one of the main challenges for this beautiful vision is that the users' queries may contain some linguistic problems (e.g., omissions and coreference) and it becomes much harder to capture their real search intent under the multi-turn conversation context (Dalton et al., 2021; Mao et al., 2022a).
Precisely understanding users' contextual search intent has been important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on lim- ited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversa- tion understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs text- based search intent interpreter to help conver- sational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical re- sponses, and propose aggregate them into an integrated representation that can robustly represent the user's real contextual search in- tent. Extensive automatic evaluations and hu- man evaluations on three widely used conversa- tional search benchmarks, including CAsT-19, CAsT-20,, and CAsT-21, demonstrate the re- markable performance of our simple L.MM framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search. The code is released at https: thub. com/ Syriimaoo/LLMan
To achieve conversational search, an intuitive method known as Conversational Query Rewriting (CQR) involves using rewriting model trans- form the current query into a de-contextualizee form. Subsequently, any ad-hoc search models can be seamlessly applied for retrieval purposes. Given that existing ad-hoc search models can be reused directly, CQR demonstrates substantial prac- tical value for industries in quickly initializing their conversational search engines. Another type of method, Conversational Dense Retrieval (CDR), tries learn conversational dense retriever to en- code the user's 's real search intent and passages into latent representations and performs dense retrieval. In contrast to the two-step CQR method, where the rewriter is difficult to be directly optimized towards search (Yu et al., 2021; Mao et al., 2023a), the con- versational dense retriever naturally learn from session-passage relevance signals.
However, as conversational search sessions are much more diverse and long-tailed (Mao al., 2022b; Dai al., 2022; Mo et al., 2023a), existing CQR and CDR methods trained on limited data still show unsatisfactory performance, especially on more complex conversational search sessions. Many studies (Vakulenko et al., 2021b: Lin et al., 2021a; a; Qian and Dou, 2022; Krasakis et al., 2022) have demonstrated the performance advantages of using de-contextualized human rewrites on ses- sions which have complex response dependency, Also, as reported in the public TREC CAsT 2021 benchmark (Dalton et al., 2022), existing methods still suffer from significant degradation in their ef-
1 Introduction
Conversational search has been expected to be the next generation of search paradigms (Culpepper et al., 2018). It supports search via conversation to provide users with more accurate and intuitive search results and much more user-friendly search experience. Unlike using traditional search en- gines which mainly process keyword queries, users could imagine the conversational search system as a knowledgeable human expert and directly start a
*Corresponding author.

---

fectiveness as conversations become longer.
Context & Question
Recently, large language models (LLMs) have shown amazing capabilities for text generation and conversation understanding (Brown et al.. 2020; Wei et al., 2022; Thoppilan et al., 2022; Zhu et al.. 2023). In the field of information retrieval (IR), LLMs have also been successfully utilized to en- hance relevance modeling via various techniques such as query generation (Bonifacio al., 2022; Dai al., 2023), query expansion (Wang al., 2023a), document prediction (Gao et al., 2022: Mackie et al., 2023), etc. Inspired by the strong performance of LLMs in conversation and IR, we try to investigate how LLMs can be adapted to precisely grasp users' contextual search intent for conversational search.
Search Intent Interpreter (LLM)
REW
RTR
RAR
Rewrite
Rewrite
Rewrite
LLM
N
Response
Response
Query Encoder
SC
MaxProb
ean A Agreeatton
In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLM as a search intent interpreter to facilitate con- versational search. Specifically, we first prompt LLM to generate both short query rewrites and longer hypothetical responses in multiple perspec- tives and then aggregate these generated contents into integrated representation that robustly rep- resents the user's real search intent. Under our framework, we propose three specific prompting methods and aggregation methods, and conduct extensive evaluations on three widely used con- versational search benchmarks, including CAsT- 19 (Dalton et al., 2020), CAsT-20 (Dalton et al., 2021), and CAsT-21 (Dalton et al., 2022)), to com- prehensively investigate the effectiveness of LLMs for conversational search.
Search Intent Vector
Passage Vectors (Offline Encoded)
Figure overview LLM4CS.
versational search field.
We show that additionally generating hypo- thetical responses and properly aggregating multiple generated results are crucial for im- proving search performance.
We demonstrate the exceptional effectiveness of LLMs for conversational search through both automatic and human evaluations, where the best method our LLM4CS achieves remarkable improvements search perfor- mance over state-of-the-art CQR and CDR baselines, surpassing even human rewrites.
In general, our framework has two main advan- tages. First, by leveraging the powerful contextual understanding and generation abilities of large lan- guage models, we show that additionally generat- ing hypothetical responses to explicitly supplement more plausible search intents underlying the short rewrite significantly improve the search perfor- mance. Second, show that properly aggregating multiple rewrites and hypothetical responses can effectively filter incorrect search intents and en- hance the reasonable ones, leading to better search performance and robustness.
Related Work
Conversational Search. Conversational search is an evolving field that involves retrieving relevant information based on multi-turn dialogues with users. To achieve conversational search, two main methods have been developed: conversational query rewriting and conversational dense retrieval. Conversational query rewriting converts the conversational search problem into ad-hoc search problem by reformulating the search session into standalone query rewrite. Existing methods try to select useful tokens from the conversation context (Voskarides et al., 2020; Lin et al., 2021b)
Overall, our main contributions are:
We propose prompting framework and de- sign three tailored prompting methods lever- age large language models for conversational search, which effectively circumvents the se- rious data scarcity problem faced by the con-

---

train a generative rewriter based on the pairs of sessions and rewrites (Lin et al., 2020; Yu et al., 2020: Vakulenko et al., 2021a). To make the rewriting process aware of the downstream retrieval process, some studies propose to adopt reinforcement learning (Wu et al., 2022; Chen et al., 2022) enhance the learning of rewriter with ranking signals (Mao et al., 2023a; Mo et al., 2023a). On the other hand, conversational dense retrieval (Yu et al., 2021) directly encodes the whole conversational search session to perform end-to-end dense retricval. Existing methods mainly try to improve the session representation through context denoising (Mao et al.. 2022a; Krasakis et al., 2022; Mo et al., 2023b; Mao et al., 2023b), data augmentation (Lin et al., 2021a; Mao et al., 2022b; Dai et al., 2022), and hard negative mining (Kim and Kim, 2022).
methods integrated into the framework. Finally, we introduce the retrieval process.
3.1 Task Formulation
We focus on the task of conversational passage retrieval, which is the crucial first step of con- versational search that helps the model access the right evidence knowledge. Given the user query q' and the conversation context C (q' r "r..,""'"''" -------1-1 of the current turn t, where g' and r¹ denote the user query and the system response of the historical i-th turn, our goal is to re- trieve passages that are relevant to satisfy the user's real search intent of the current turn.
3.2 Prompting Methods
The prompt follows the formulation of [Instruc- tion, Demonstrations, Imput], where Input is com- posed of the query of and the conversation con- text C of the current turn t. Figure 4 shows general illustration of the prompt construc- tion.' Specifically, we design and explore three prompting methods, including Rewriting (REW). Rewriting- Then--esspoee (RTR), and Rewriting- And-Response (RAR), in our LLM4CS framework.
IR with LLMs. Due to the revolutionary natu- ral language understanding and generation abili- ties, LLMs are attracting more and more attention from the IR community. LLMs have been lever- aged to enhance the relevance modeling of retrieval through query generation (Bonifacio et al., 2022; Jeronymo et al., 2023; Dai et al., 2023), query ex- pansion (Wang et al., 2023a), document predic- tion (Gao et al.. 2022; Mackie et al., 2023), etc. Besides, Shen et al. (2023) proposed to first use the retriever to enhance the generation of LLM and then the generated content to augment the original search query for better retrieval. Ziems al. (2023) treated LLM as built-in search en- gine to retrieve documents based on the generated URL. There are also some works leveraging LLM to perform re-ranking (Sun et al., 2023; Jiang et al., 2023). Different from previous studies, in this pa- per, we propose the LLM4CS framework that fo- cuses on studying how LLM can be well utilized capture the user's contextual search intent to facilitate conversational search.
3.2.1 Rewriting Prompt (REW)
In this prompting method, we directly treat LLM as a well-trained conversational query rewriter and prompt it to generate rewrites. Only the red part of Figure 4 is enabled. Although straightforward, we show in Section 4.5 that this simple prompting method has been able achieve quite strong search performance compared to existing baselines.
3.2.2 Rewriting- Then-Response (RTR)
Recently, few studies (Mao et al., 2021; Gao et al., 2022; Yu et al., 2023; Mackie et al., 2023) have shown that generating hypothetical responses for search queries can often bring positive improve- ments in retrieval performance. Inspired by them, in addition to prompting LLM to generate rewrites. we continue to utilize the generated rewrites to further prompt LLM to generate hypothetical re- sponses that may contain relevant information to answer the current question. The orange part and the blue part of Figure 4 are enabled. Specifically, incorporate the pre-generated rewrite (i.e., the orange part) into the Input field of the prompt and
3 LLM4CS: Prompting Large Language Models for Conversational Search
In this section, we introduce our LLM4CS frame- work, which leverages LLM text-based search intent interpreter to facilitate conversational search. Figure 1 shows an overview of LLM4CS. In the following, we first describe our task formulation conversational search, and then we elaborate on the specific prompting methods and aggregation
"We put this figure in Appendix A due to the space limita- tion. See our nour-ed code for the full prompt of each prompting method.

---

then prompt LLM generate informative hypo- thetical responses by referring to the rewrite.
(e.g, ANCE (Xiong al., 2021)) to encode each of them into high-dimensional intent vector and aggregate these intent vectors into one final search intent vector s. Specifically, we design and explore the following three aggregation methods, including MaxProb, Self-Consistence (SC), and Mean, in our LLM4CS framework.
3.2.3 Rewriting-And-Rest Response (RAR)
Instead of generating rewrites and hypothetical re- sponses a two-stage manner, we can also gener- ate them all at once with the red part and the blue part of Figure being enabled. We try to explore whether this one-stage generation could lead to bet- ter consistency and accuracy between the generated rewrites and responses, compared with the two-step RTR method.
3.3.1 MaxProb
We directly use the rewrite and the hypothetical response that have the highest generation proba- bilities. Therefore, compared with the other two aggregation methods that will be introduced later, MaxProb is highly efficient since it actually does not require multiple generations.
3.2.4 Incorporating Chain-of-Thouugh
Chain-of-thour -tooughtt (CoT) (Wei et al., 2020) induces the large language models to decompose reason- ing task into multiple intermediate steps which can unlock their stronger reasoning abilities. In this work, we also investigate whether incorporating the chain-of of--- thougtt of reasoming the user's real search intent could improve the quality of rewrite and response generation.
Formally, for REW prompting:
f(i).
For the RTR and RAR prompting methods, we mix the rewrite and hypothetical response vectors:
Specifically, shown in the green part of Fig- for
ure we manually write the each turn of the demonstration, which reflects how humans infer the user's real search intent of the current turn based on the historical conversation context. When generating, we instruct LLM to first generate the -fain-of--thuuhh before generating rewrites (and responses). We investigate the effects of our proposed CoT tailored to the reasoning of contextual search intent in Section 4.6.
[(h:) f(f1)
(2)
2
3.3.2 Self-Consisteney (SC)
The multiple generated rewrites and hypothetical responses may express different search intents but only some of them are correct. To obtain more reasonable and consistent search intent represen- tation, we extend the self-consistency prompting method (Wang et al., 2023b), which was initially designed for reasoning tasks with predetermined answer sets, to our contextual search intent under- standing task, which lacks fixed standard answer. To be specific, we select the intent vector that is the most similar to the cluster center of all intent vectors the final search intent vector, since it represents the most popular search intent overall. Formally, for REW prompting:
3.3 Content Aggregation
After prompting LLM multiple times generate multiple rewrites and hypothetical responses, we then aggregate these generated contents into an integrated representation to represent the user's complete search intent for search. Let us con- sider that we have generated N query rewrites Q = ((1...,(x)) and M hypothetical responses R (f..,..,, fiM)) for each rewrite qi sorted by their generation probabilities from high to l Note that in RAR prompting, the rewrites and the hypothetical responses are always generated in pairs (i.e.. M = 1). While in RTR prompting, one rewrite can have M hypothetical responses since they are generated in a two-stage manner. Next, we utilize a dual well-trained ad-hoc retriever
q' N Σ f((),
(3)
8 = arg max f():) T q",
(4)
f(G))
where 9 is the cluster center vector and denotes the dot product that measures the similarity. For RTR prompting, we first select the intent vector f(ā)) and then select the intent vector f(fxz) from all hypothetical responses generated based on
"That is, the generation probability orders are: P(ğ.)= 2 P(qv) and P(fa) 2 ... 2 P(fax).
"The parameters of the query encoder and the passage encoder are shared.

---

the selected rewrite qx::
Dataset
CAsT-19
CAsT-20
CAsT-21
k = arg max f(a)) ((6)).aa .9"
(5)
# Conversations
20
25
18
# Turns (Sessions)
173
208
157
# Passages/Docs
38M
40M
fg M f(kkj),
(6)
Table I: Statistics of the three CAsT datasets.
2 arg max f(fkg)
T -
(7)
CAsT-21 (Dalton et al., 2022), which are curated by the human experts of TREC Conversational Assis- tance Track (CAsT). Each CAsT dataset has dozens of information-ssekknng conversations comprising hundreds of turns. CAsT-19 and CAsT-20 share the same retrieval corpora while CAsT-21 has a differ- ent one. In contrast, CAsT-20 and CAsT-21 have a more complex session structure than CAsT-19 their questions may refer previous responses. All three datasets provide human rewrites and passage- (9) level (or document-level) relevance judgments la- beled by TREC experts. Table I summarizes the basic dataset statistics.
s f((x) + ffkzz)
(8)
where k and 2 are the finally selected indexes of the rewrite and the response, respectively.
The aggregation for RAR prompting is similar to RTR prompting, but it does not need response selection since there is only one hypothetical re- sponse for each rewrite:
s = f(āx) + ((fa)
2
3.3.3 Mean
We average all the rewrite vectors and the corre- sponding hypothetical response vectors.
Following previous work (Dalton et al., 2020, 2021; Yu et al., 2021; Mao al., 2022a), we adopt Mean Reciprocal Rank (MRR), NDCG@3, and Recall @ 100 as our evaluation metries and calcu- late them using pytrec_eval tool (Van Gysel and (10) de Rijke, 2018). We deem relevance scale 2 as positive for MRR on CAsT-20 and CAsT-21. For CAsT-21, we split the documents into pas- sages and score each document based on its highest- scored passage (i.e., MaxP (Dai and Callan, 2019)). We conduct the statistical significance tests using paired t-tests
For REW prompting:
Σ f(a).
$
For the RTR and RAR prompting methods:
s ΣE N (5) +E N * M)
(11)
0.05 level.
Compared with MaxProb and Self-Consistency, the Mean aggregation comprehensively considers more diverse search intent information from all sources. It leverages the collaborative power to enhance the popular intents, but also supplements plausible intents that missing in single rewrite hypothetical response.
4.2 Implementation details
We use the OpenAl gp13.5-turbo-16k as our LLM. The decoding temperature is set to 0.7. We ran- domly select three conversations from the CAsT- 22 dataset for demonstration. CAsT-22 is new conversational search dataset also proposed by TREC CAST, but only its conversations are re- leased and the relevance judgments have not been made public. Therefore, it cannot be used for eval- uation and we just use for demonstration. For REW prompting, we set N = 5. For RTR prompt- ing, we set N = and M 5. For RAR prompt- ing, we set N 5, and M is naturally set to 1. Following previous studies (Yu et al., 2021; Mao
3.4 Retrieval
All candidate passages are encoded into passage vectors using the same retriever f. At search time, we return the passages that are most similar to the final search intent vector s as the retrieval results.
4 Experiments
4.1 Datasets and Metrics
We carry out extensive experiments on three widely used conversational search datasets: CAsT-19 (Dal-
"Only the tums that have relevance labels counted. htttps://gi //gthbb..htb.t...e/ com/ dal dalddnntonnnnnoooo ton treccastwebe tree/ paster/202. Until the submission deadline of EMNLP 2023.
ton et al., 2020), CAsT-20 (Dalton et al., 2021), and

---

al.. 2022a,b; Mo et al., 2023a), adopt the ANCE (Xiong et al., 2021) checkpoint pre-trained on the MSMARCO dataset as our ad-hoc retriever f- We uniformly truncate the lengths of queries (or rewrites), passages, and hypothetical responses into 64, 256, and 256.
of using different prompting and aggregation meth- ods in Section 4.5 and investigate the effects of the incorporation of COT in Section 4.6.
From Table 2, we observe that LLM4CS outper- forms all the compared baselines in terms of search performance. Specifically, LLM4CS exhibits rel- ative improvement of over 18% compared the second-best results on the more challenging CAsT- 20 and CAsT-21 datasets across all metrics. particular, even compared to using human rewrites, our LLM4CS can still achieve better results on most metrics, except for the Recall @ 100 of CAsT- 19 and NDCG@3 of CAsT-21. These significant improvements, which are unprecedented in prior research, demonstrate the strong superiority our LLM4CS over existing methods and underscore the vast potential of using large language models for conversational search.
4.3 Baselines
We compare our few-shot LLM4CS against the following six conversational search systems:
(1) T5QR (Lin al., 2020): A T5 (Raffel et al., (020)-based conversational query rewriter trained with the human rewrites.
(2) ConyDR (Yu et al., 2021): A conversa- tional dense retriever fine-tuned from an ad-hoc retriever by mimicking the representations of hu- man rewrites.
(3) COTED (Mao et al., 2022a): An improved version of ConvDR (Yu et al., 2021) which incorpo- rates curriculum searming-baseed context denoising objective.
Effects of Different Prompting Methods 4.5
and Aggregation Methods
(4) ZeCo (Krasakis et al., 2022): variant of ColBERT (Khattab and Zaharia, 2020) that matches only the contextualized terms of the cur- rent query with passages to perform zero-shot con- versational search.
We present comparison of NDCG@3 perfor- mance across various prompting and aggregation methods (excluding the incorporation of CoT) Table 3. Our findings are as follows:
(5) CRDR (Qian and Dou, 2022): A conver- sational dense retrieval method where the dense retrieval part enhanced by the distant supervision from query rewriting in a unified framework.
First, the RAR and RTR prompting methods clearly outperform the REW prompting, demon- strating that the generated hypothetical responses can effectively supplement the short query rewrite to improve retrieval performance. However, even the simple REW prompting can also achieve quite competitive performance compared to existing baselines, particularly on the more challenging CAsT-20 and CAsT-22 datasets, where shows sig- nificant superiority (e.g., 0.380 vs. 0.350 on CAsT- 20 and 0.465 vs. 0.385 on CAsT-21). These posi- tive results further highlight the significant advan- tages of utilizing LLM for conversational search. Second, in terms of aggregation methods, both Mean and SC consistently outperform MaxProb. These results indicate that depending solely on the top prediction of the language model may not pro- vide sufficient reliability. Instead, utilizing the col- lective strength of multiple results proves to be a better choice. Additionally, we observe that the Mean aggregation method, which fuses all gen- erated contents into the final search intent vector (Liquation 11), does not consistently outperform SC (e.g., on CAsT-20), which actually only fuses one rewrite and one response (Equation 8). This suggests that taking into account more generations
(6) ConyGQR (Mo et al., 2023a): query refor- mulation framework that combines query rewriting with generative query expansion.
TSQR, CRDR, and ConvGQR are trained on the training sessions of QReCC (Anantha et al., 2021), which is a large-scale conversational question an- swering dataset. The performances of ConvDR and COTED are reported in the few-shot setting using 5-fold cross-validation according to their original papers. We also present the performance of us- ing human rewrites for reference. Note that the same ANCE checkpoint is used to perform dense retrieval for all baselines except ZeCo to ensure fair comparisons.
4.4 Main Results
The overall performance comparisons presented in Table 2. The reported performance of LLM4CS results from the combination of the RAR prompt- ing method, the Mean aggregation method, and our tailored CoT, which shows to be the most effective combination. We thoroughly investigate the effects

---

CAST-19
CAsT-20
CAST-21
System
MRR
NDCG@3
R@ 100
MRR
NDCG@3
R@ 100
MRR
NDCG@3
R@ 100
Conversational
Dense Retrieval
ConvDR
0.740
0.466
0.362
0.510
0.340
0.345
0.573
0.385
0,483
COTED
0.769
0.478
0.367
0.491
0.342
0.340
0.565
0.371
0.485
ZeCo
0.238
0.216
0.176
0.200
0.234
0.267
CRDR
0.765
0.472
0.357
0,501
0.350
0.313
0.474
0.342
0.380
Conversational Query Rewriting
TSQR
0.701
0.417
0.332
0.423
0.299
0.353
0.469
0.330
0.408
ConvGQR
0.708
0.434
0.336
0.465
0.331
0.368
0.433
0.273
0.330
LLMACS
0.776
0.515
0.372
0.6151
0.455
0.489
0.681
0.492
0.614
Human
0.740
0.461
0.381
0.591
0.422
0.465
0.680
0,502
0.590
RI-H +4.9%
+11.7%
-2.4%
+4.1%
+7.8%
+5.2%
+0.1%
-2.0%
+4.1%
RI-2nd-Best
+0.9%
+7.7%
+1.4%
+20.6%
+30.0%
+32.9%
+18.8%
+27.8%
+26.6%
Table Overall performance comparisons. j denotes the results replicated from their original paper. denotes LLM4CS (RAR + Mean + CoTT) significantly outperforms all the compared baselines (except ZeCo) in the < 0.05 level. The best results are bold and the second-bes results are underlined. Human denotes using human rewrites. RI-H and RI-2nd-Bext the relative improvements over Human and the second-best results, respectively.
CAsT-19
CAsT-20
CAsT-21
Aggregation
REW
RTR
RAR
REW
RTR
RAR
REW
RTR
RAR
MaxProh
0.441
0.459
0.464
0.356
0.415
0.430
0.407
0.469
0.462
SC
0.449
0,466
0.476
0.362
0.432
0.444
0,445
0.473
0.469
Mean
0.447
0.464
0.488
0.380
0.425
0.442
0.465
0.481
0.478
Previous SOTA
0.478
0.350
0.385
Human
0.461
0.422
0.502
Table Performance comparisons with respect to NDCG@3 using different prompting and aggregation methods. The best combination each dataset is bold.
CAST-20:
CAST-20: RTR
CAST-20: PAR
0.42
0.45
0.47
#ithout
without
without with CoT
NOC093
0.40
0.388
with CoT
0.100
0.396
0.42
44
with CST
2-427
0.432
3.435
0.435
0.438
0.44B
0.642
0.455
0.41
0.43
D.450
3
MasProb
Mewn
Prx
SC
Mean
MasProb
Mean
CAST21 REW
CAST-21: RTR
CAST-21: RAR
without CaT
5:100
without Cal
without CaT
D.402
with
CaT
NDC029
0.44
0.46
0.48
0.47
3.470
3.471
0/481
NDCG@R
0.47
0.48
0.289
0-774
3.428
0-483
3.407
6.45
0.38
MaxProb
Mean
0.64
MaxProb
SC
Mean
0.88
Mastrob
SC
Mean
Figure 2: NDCG@3 comparisons between incorporating our tailored CoT not across different prompting and aggregation methods on CAsT-20 and CAsT-21 datasets.
may not always be beneficial, and careful se- lection among them could be helpful to achieve improved results.
example to show how our CoT takes effect in Ap- pendix B.I. From the results, observe that:
Incorporating our chain-of-thought into all prompting and aggregation methods generally im- proves search performance. This demonstrates the efficacy of our chain- oiinn-of-hhen thouhh in guiding the large language model towards a correct understanding
4.6 Effects Chain-of-Tie Thought
We show the ablation results of our tailored chain- of-thought in Figure 2. We also provide a real

---

CAST-19 (173 turns)
CAsT-20 (208 tums)
CAST-21 (157 turns)
100
TSOR
TSOR
80-
LLMACS (REW)
LIM4CS
LUM4CS (REW)
Porcent
60-
01.2
Parcent
60-
50.3
40
37.1
41.3
43.2
22.4
20
6.2
175
016
14.
2.6
20
1544
1.9
20
22.1
112
Good-41
Good-A
Badio
BediC
Good-il
Bad-O
DadiC
Good-11
Good-A
Bad-0
BadiC
Figure 3: Human evaluation results for LLM4CS (REW + MaxProb) and T5QR on the three CAsT datasets.
the user's contextual search intent.
fully convey the user's real search intent for CAsT- 19, CAsT-20, and CAsT-21, respectively. In con- trast, the corresponding percentages for T5QR are merely 75.1%, 62.0%, and 58.6%. Such a high rewriting accuracy of LLM4CS further demon- strates the strong ability of LLM for contextual search intent understanding.
In contrast, the improvements are particularly notable for the REW prompting method compared to the RTR and RAR prompting methods. It ap- pears that the introduction of multiple hypothetical responses diminishes the impact of the chain-of- thought. This could be attributed to the fact that including multiple hypothetical responses signifi- cantly boosts the quality and robustness of the final search intent vector, thereby reducing the promi- nence of the thoun-tf-thought in enhancing search performance.
(2) In the case of CAsT-20 and CAsT-21, a sig- nificantly higher percentage of rewrites are labeled as Good-A, in contrast to CAsT-19, where the ma- jority of good rewrites closely resemble the hu- man rewrites. This can be attributed to the higher complexity of the session structure and questions in CAsT-20 and CAsT-21 compared to CAsT-19, which allows for greater freedom in expressing the same search intent.
5 Human Evaluation
The retrieval performance is influenced by the ad- hoe retriever used, which implies that automatic search evaluation metrics may not fully reflect the model's capability to understand contextual search intent. Sometimes, two different rewrites can yield significantly different retrieval scores, even though they both accurately represent the user's s real search intent. To better investigate the contextual search intent understanding ability of LLM, we perform fine- mne-ggrined human evaluation the rewrites generated by our LLM4CS (REW + MaxProb). Specifically, we manually compare each model's rewrite with the corresponding human rewrite and label it with one of the following four categories: (1) Good-If: The model's rewrite is nearly the same as the human rewrite. (2) Good-A: The expression of the model's rewrite is different from the human rewrite but it also successfully conveys the user's s real search intent. (3) Bad-C: the rewrite has coref- erence errors. (4) Bad-O: the rewrite omits impor- tant contextual information or has other types of errors. Furthermore, we apply the same principle to label the rewrites of T5QR for comparison pur- poses. A few examples of such categorization are presented in Appendix B.2.
(3) The rewrites generated by LLM4CS exhibit coreference errors in less than 3% of the cases, whereas T5QR's rewrites contain coreference er- rors in approximately 10% of the cases. This ob- servation highlights the exceptional capability of LLM in addressing coreference issues.
Conclusion
In this paper, we present a simple yet effective prompting framework (i.e., LLM4CS) that lever- ages LLMs for conversational search. Our frame- work generates multiple query rewrites and hypo- thetical responses using tailored prompting meth- ods and aggregates them to robustly represent the user's contextual search intent. Through extensive automatic and human evaluations on three CAsT datasets, we demonstrate its remarkable perfor- mance for conversational search. Our study high- lights the vast potential of LLMs in conversational search and takes an important initial step in ad- vancing this promising direction. Future research will focus on refining and extending the LLM4CS framework to explore better ways of generation facilitate search, improving aggregation techniques, optimizing the Let--vetreeer interaction, and in- corporating reranking strategies.
The results of the human evaluation are shown in Figure 3, where we observe that:
(1) From a human perspective, 85.5%, 89.4%, and 84.8% of the rewrites of LLM4CS success-

---

Limitations
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual.
Our work shows that generating multiple rewrites and hypothetical responses and properly aggregat- ing them can effectively improve search perfor- mance. However, this requires invoking LLM mul- tiple times, resulting in higher time cost for re- trieval. Due the relatively high generation la- tency of LLM, the resulting query latency would be intolerable for users when compared to conven- tional search engines. A promising approach is to design better prompts capable of obtaining all infor- mative content in one generation, thereby signifi- cantly improving query latency. Another limitation is that, similar to the typical disadvantages of CQR methods, the generation process of LLM lacks awareness of the downstream retrieval process. Ex- ploring the utilization of ranking signals to enhance LLM generation would be a compelling direction for future research of conversational search.
Zhiyu Chen, Jie Zhao, Anjie Fang, Besnik Fetahu, Rokhlenko Oleg, and Shervin Malmasi. 2022. Rein- forced question rewriting for conversational question answering.
J Shane Culpepper. Fernando Diaz, and Mark D Smucker. 2018. Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in lorne (swirl 2018). In ACM SIGIR Forum, volume 52, pages 34-90. ACM New York, NY, USA.
Zhuyun Dai and Jamie Callan. 2019. Deeper text un- derstanding for IR with contextual neural language modeling. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, SIGIR 2019, Puris, France, July 21-25, 2019, pages 985-988. ACM.
Acknowledgement
Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, and Kelvin Guu. 2022. Dialeg impainting: Turning documents into dialogs. International Conference on Machine Learning, pages 4558-4586. PMLR.
This work was supported by the National Natural Science Foundation of China No. 62272467, Pub- lic Computing Cloud, Renmin University of China, and Intelligent Social Governance Platform, Major Innovation & Plamning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin Uni- versity of China, and the Outstanding Innovative Talents Cultivation Funded Programs 2024 of Ren- min University of China. The work was partially done at Beijing Key Laboratory of Big Data Man- agement and Analysis Methods.
Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wes Wei Chang. 2023. Promptagator: Few-shot dense retrieval from examples. In 11th International Conference on Learning Representa- tions, ICLR 2023.
Jeffrey Dalton, Chenyan Xiong. and Jamie Callan. 2020. Trec cast 2019: The conversational assistance track overview, In In Proceedings of TREC.
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. Cast 2020: The conversational assistance track overview, In In Proceedings of TREC.
References
Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answer- ing goes conversational via question rewriting. In NAACL-HLT, pages 520-534. Association for Com- putational Linguisties.
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2022. Trec cast 2021: The conversational assistance track overview, In In Proceedings of TREC.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without rele- vance labels. CoRR, abs/2212. 10496..
Luiz Henrique Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Frassetto Nogueira. 2022. In- pars: Data augmentation for information retrieval us- ing large language models. CoRR, abs/2202.051444
Vitor Jeronymo, Luiz Henrique Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto de Alen- car Lotufo, Jakub Zavrel, and Rodrigo Frassetto Nogueira. 2023. Inpars-v2: Large language mod- els as efficient dataset generators for information re- trieval. CoRR, abs/2301.0188)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. CoRR, 0bs/2306.02561.

---

Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 39 40-44..40 ACM.
Yuning Mao, Pengcheng He. Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-saummente retrieval for open- domain question answering. In ACL/JCNLP (1), pages 4089-4100. Association for Computational Linguisties.
Sungdong Kim and Gangwoo Kim. 2022. Saving dense retriever from shortcut dependency in conversational search.
Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, and Juan-Yun Nie. 2023a. ConvGQR: generative query reformulation for conversational search. In ACI, volume ACL 2023. Association for Computational Linguistics.
Antonios Minas Krasakis, Andrew Yates, and Evangelos Kanoulas. 2022. Zero-shot query contextualization for conversational search. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval (SIGIR).
Fengran Mo, Jian- Nic, Kaiyu Huang. Kelong Mao, Yutao Zhu, Peng Li, and Yang Liu. 2023b. Learning to relate to previous turns in conversational search. In 29th ACM SIGKDD Conference On Knowledge Discover and Data Mining (SIGKDD).
Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021a. Contextualized query embeddings for con- versational search. Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing (EMNLP).
Hongjin Qian and Zhicheng Dou. 2022. Explicit query rewriting for conversational dense retrieval. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP).
Sheng-Chieh Chich Lin, Jheng- Yang. Rodrigo Nogucira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2020. Conversational question reformulation via sequence-to to--nccee---suunneee architectures and pretrained language models. arXiv preprint arXiv:2004. 01909.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec. Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res.,
Sheng-Chieh Lin. Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 202 lb. Multi-stage conversational passage retrieval: An approach to fusing term importance estimation and neural query rewriting. ACM Transactions on Information Systems (TOIS), -999.
Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Tianyi Zhou, and Daxin Jiang. 2023. Large lan- guage models are strong zero-shot retriever. CoRR, abs/2304. 14233.
Weiwei Sun, Lingyong Yan, Xinyu Ma. Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agent. CoRR, abs/2234.00952..
Tain Mackie, Shubham Chatterjee, and Jeffrey Dalton. 2023. Generative relevance feedback with large lan- guage models. CoRR, abs/2304.13157
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng. Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin. James Qin, Dehao Chen, Yuanzhong Xu. Zhifeng Chen, Adam Roberts. Maarten Bosma, Yanqi Zhou, Chung- Ching Chang. Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern. Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenhergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John. Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina. Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Agūiera Arcas, Claire Cui, Marian Croak, Fdd H. Chi, and Quoe Le. 2022. Lamda: Language models for dialog applications. CoRR, 03s/2201.082339
Kelong Mao, Zhicheng Dou, Bang Liu, Hongjin Qian, Fengran Mo, Xiangli Wu, Xiaohua Cheng, and Zhao Cao. 2023a. Search-oriented conversational query editing. In ACL (Findings), volume ACL 2023 of Findings ACL. Association for Computational Lin- guistics.
Kelong Mao, Zhicheng Dou, and Hongjin Qian. 2022a. Curriculum contrastive context denoising for few- shot conversational dense retrieval. In Proceedings of the 45th International ACM SIGIR conference on research and development in Information Retrieval (SIGIR).
Kelong Mao, Zhicheng Dou, Hongjin Qian, Fengran Mo, Xiaohua Cheng, and Zhao Cao. 2022b. Con- vtrans: Transforming web search sessions for con- versational dense retrieval. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Kelong Mao, Hongjin Qian, Fengran Mo, Zhicheng Dou, Bang Liu, Xiaohua Cheng, and Zhao Cao. 2023b. Learning denoised and interpretable session representation for conversational search. In Proceed- ings of the ACM Web Conference, pages 3192-3202.
Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu. and Raviteja Anantha. 2021a. Question rewriting for conversational question answering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining ( (WSDM), pages 355-363.

---

Svitlana Vakulenko, Nikos Voskarides, Zhucheng Tu, and Shayne Longpre. 2021b. A comparison ques- tion rewriting methods for conversational passage re- trieval. In ECIR (2), volume 12657 of Lecture Notes in Computer Science, pages 418-424. Springer.
Wenhao Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In 11th International Con- férence on Learning Representations, ICLR 2023.
Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_ eval: An extremely fast python interface to trec_eval. In SIGIR. ACM.
Yutao Zhu, Huaying Yuan, Shuting Wang. Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey, arXiv preprint arXiv:2308.07110
Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, and Maarten de Rijke. 2020. Query reso- lution for conversational search with limited supervi- sion. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 921-930)
Noah Ziems. Wenhao Yu, Zhihan Zhang, and Meng Jiang. 2023. Large language models are built-in au- toregressive search engines. In ACL (Findings), vol- ume ACL 2023 of Findings of ACL. Association for Computational Linguistics.
Liang Wang, Nan Yang, and Furu Wei. 2023a. Query2doc: Query expansion with large language models. CoRR, abs/2303.0767.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In 11th International Conference on Learn- ing Representations, ICIR 2023.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth International Conference Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Jason
Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2020. Chain of thought prompting elicits reasoning large language models. Advances in neural information processing systems.
Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reitter, and Gaurav Singh Tomar. 2022. Conqrr: Conversa- tional query rewriting for retrieval with reinforcement learning.
Lee Xiong. Chenyan Xiong. Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Shi Yu. Jiahua Liu, Jingqin Yang, Chenyan Xiong, Paul Bennett, Jianfeng Gao, and Zhiyuan Liu. 2020. Few- shot generative conversational query rewriting. In Proceedings of the 43rd International ACM SIGIR conference research and development in Informa- tion Retrieval (SIGIR), pages 1933-1936.
Shi Yu, Zhenghao Liu, Chenyan Xiong. Tao Feng. and Zhiyuan Liu. 2021. Few-shot conversational dense retrieval. In Proceedings of the 44th International ACM SIGIR conference on research and development in Information Retrieval (SIGIR).

---

Appendix
A Prompt of LLM4CS
Figure 4 shows general illustration of the prompt of LLM4CS. The prompt consist of three parts, which Instruction, Demonstration, and Input. The red part is for REW prompting, the blue part for the RTR and RAR promptings, and the orange part is for RTR prompting. The green part is for our designed chain-of-thought.
B
Case Study
B.1 Examples of Shtain-ff-Thoughe
The example in Table shows how our CoT takes effect. The CoTf and Our Rewrite fields are gen- erated by LLM4CS (REW CoT). We can find that the generated CoT effectively illustrates the rationale behind the rewriting process. Please refer our anonymously open-sourced repository for more examples.
B.2 Examples of Human Evaluation
An example for each category shown in Table Please refer to our anonymously open-sourced repository for more examples.

---

For an inffommatoo-sccekine dialog, please help reformulate the question into rewrite that can fully express the user's information needs without the need of contexi, but also generate informative response answer the question.
Instruction
will give you several example multi-turn dialogs, where each turn contains question, a response, and rewrite. The rewrite part begins with sentence explaining the reason the rewrite.
Example #1:
Question: What should consider when buying phone?
Rewrite: This is the first turn. So the question should be rewritten as: What should consider when buying phone?
Response: The design of phone and the overall
Demonstration
Question: Cool. Which one would you recommend?
Rewrite: Based on Tum 1, you are inquiring about what should be considered when
question should be rewritten as: Cool. Which smartphone would you recommend for me?
buying phone.
Response: Just because phone has everything...
Example #2:
Your Task (only questions and responses are given):
Context:
Question:
What was the basis of the Watergate scandal?
Response:
Question:
Response:
Input
Current Question: So what happened to Nixon?
Rewrite: So what happened to Nixon after the events of the Watergate scandal?
(Now, you should give me the rewrite and an informative response of the **Current Question bused the ** The output format should always be: Rewrite: SReason. So the question should rewritten as: SRewriteinRespons: SResponse. Go ahead!)
Rewrite: Based all previous turns, Nixon badly involved in the Watergate scandal. So question should be rewritten as: So what happened to Nixon after events the Watergate scandal?
Model Output
Response: With the mounting evidence loss...
Figure 4: A general illustration of the prompt of LLM4CS. The prompt consist of three parts, i.c., Instruction, Demonstration, and Input. The red part is for REW prompting, the blue part is for the RTR and RAR promptings. and the orange part is for RTR prompting. The green part for designed chain-of-thought.

---

CAsT-20 (Conversation #84)
Turn 1
Question: How does seed investment work for startups?
Response: Seed money, sometimes known as seed funding or seed capital, is a form of securities offering in which an investor invests capital in exchange for an equity stake in Human Rewrite: How does seed investment work for startups?
CoT: This is the first turn.
Our Rewrite: How does seed investment work for startups?
Turn 2
Question: Tell me more about angel rounds.
Response: Tell Me More was a National Public Radio interview show hosted by journalist Michel Martin offering sample segments, and soliciting user feedback. It began broadcast- ing on April 30, 2007.
Human Rewrite: Tell me more about angel investment rounds.
CoT: Based on the current question and the previous response, we are comparing angel rounds with crowdfunding.
Our Rewrite: Tell me more about angel rounds in seed investment for startups.
Turn 3
Question: How does it compare with crowdfunding?
Response: For crowdfunding of equity stock purchases, there some research in social
psychology that indicates that ... Managing communications with large number of possibly disappointed investors and supporters can be substantial, and potentially diverting, task. Human Rewrite: How does an angel investment round compare with crowdfunding? CoT: Based on the current question and the previous response, we are comparing angel rounds with crowdfunding.
Our Rewrite: How does angel funding compare crowdfunding for startups?
Turn 4
Question: How much a typical A series round?
Response: OpenGov has raised $47 million from venture capital and angel investors. A $3 million Series A round in 2012 included venture funds 8VC, Founder Collective, Valiant Capital, and number of high-profile...
Human Rewrite: How much is typical A series investment round? talking about different types of funding for startups such as seed investment, angel rounds, and crowdfunding.
CoT: Based the previous turns,
Our Rewrite: How much funding is typically raised in an A series round for startups?
Turn 5
Question: What are the risks of investing?
Angel investors invest in early stage or start-up companies in exchange for an equity ownership interest. Angel... ... Here is what angels particularly care about: The quality, passion, commitment, and integrity of the founders.
Response:
Human Rewrite: What the risks of investing in startups?
CoT: Based on the previous turns, we are discussing different aspects of startup investments.
Our Rewrite: What are the risks associated with investing in startups?
Turn 6
Question: What are the different types of instruments used by VCs?
Response:
Human Rewrite: What are the different types of instruments used by VCs?
CoT: The user is asking about the different types of instruments used by venture capitalists.
Our Rewrite:
What are the various types of instruments employed by venture capitalists
(VCs)?
Table An example show how our CoT takes effect. The CoT and Our Rewrite fields are generated by LLM4CS (REW CoT).

---

Good-H
Question:
How about replacing instead?
Human Rewrite: How much does it cost to replace a garage door opener?
Our Rewrite: What is the cost of replacing a garage door opener?
Good-A
Question: What's important for me to know about their safety?
Human Rewrite: What's important for me know about the safety of smart garage door openers?
Our Rewrite: What are the safety features that I should consider when choosing a garage door opener?
Bad-O
Question:
Are there any famous foods?
Human Rewrite: Are there any famous foods in Washington D.C.?
Our Rewrite: Are there any famous foods?
Bad-C
Question:
What is its main economic activity?
Human Rewrite: What is the main economic activity of Salt Lake City?
Our Rewrite: What is the main economic activity in Utah?
Table 5: Examples of the four categories in human evaluation.


Reviews:
[{"review_id": "XbfIs95COf", "reviewer": "Reviewer_Leu2", "summary": "", "questions": "[Sec. 2] It is essential to clarify the distinction between \"Generative IR\" and \"Text generation for IR\" in the paper. I recommend referencing \"Gen-IR at SIGIR'23\" [https://arxiv.org/pdf/2306.02887.pdf] for more detailed insights on this particular research topic.\n\n[Sec. 3.3.1] The paper mentions that the RAR method produces \"M\" hypothetical responses. However, it does not elaborate on how the response vectors (r_11, r12, ..r1M) are aggregated into a single new vector \"s.\" Providing further explanation in this section would enhance the clarity of the method.\n\n[Sec. 4.5] The comparison between RAR and RTR methods is limited since the RAR experiment seems to produce only one hypothetical response. To make the conclusion more robust, I suggest conducting experiments with the following settings: (i) N=1 vs. N=5(REW) and (ii) N=5, M=1(RTR) vs. N=5, M=5.\n\n[Sec. 5] It is crucial to outline the experimental setup for T5QR, especially regarding its consideration of canonical responses. To ensure fairness, T5QR should be evaluated while taking into account responses in previous turns.\n\n[Overall] Despite the declared limitation on computational costs and efficiency, it would be insightful to provide information about model parameter sizes or approximate numbers (e.g., times, FLOPs, model size, etc.). This information would help readers understand the trade-offs between different methods.\n\n[Overall] To strengthen the paper's contributions, it would be valuable to clarify what the prompting-LLM approach can achieve compared to previous fine-tuned methods. Highlighting the advantages and disadvantages of this approach would offer a better understanding of its practical implications and potential in the field.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explored how LLM can provide fine-grained content/knowledge (e.g., rewritten queries, hypothetical responses) and further validated the effectiveness through benchmark datasets. The author conducted a few experiments with various empirical settings for using prompts; in addition, they conducted a few simple qualitative analyses with human evaluations on LLM-generated content.", "reasons_to_reject": "The paper's proposed methods lack fair and comprehensive comparisons. There is a notable absence of insights and takeaways concerning LLM-generated content, such as qualitative and empirical analyses or correlations between the generated content and improved effectiveness. Moreover, the paper fails to adequately justify the motivation behind using LLM to generate rewrites or responses. The absence of evidence demonstrating how the LLM-generated content outperforms previous rewriting approaches like T5QR is concerning.\n\nAdditionally, the approach of prompting-LM generation appears impractical for Conversational Search, and alternative strategies, such as reranking models, could be explored to improve retrieval effectiveness. For efficiency, supervised sparse retrieval methods could be investigated as well. Furthermore, the paper might benefit from considering how LLM can address more knowledge-intensive tasks like Conversational QA rather than focusing solely on Conversational Search.\n\nIn summary, the paper requires significant improvements in terms of comparative analysis, insightful findings regarding LLM-generated content, and justifications for its approach. Exploring alternative strategies for retrieval effectiveness and efficiency could enhance the paper's contributions to the field. Additionally, expanding the scope of application to knowledge-intensive tasks may lead to more impactful outcomes.", "questions_to_authors": "[Sec. 2] It is essential to clarify the distinction between \"Generative IR\" and \"Text generation for IR\" in the paper. I recommend referencing \"Gen-IR at SIGIR'23\" [https://arxiv.org/pdf/2306.02887.pdf] for more detailed insights on this particular research topic.\n\n[Sec. 3.3.1] The paper mentions that the RAR method produces \"M\" hypothetical responses. However, it does not elaborate on how the response vectors (r_11, r12, ..r1M) are aggregated into a single new vector \"s.\" Providing further explanation in this section would enhance the clarity of the method.\n\n[Sec. 4.5] The comparison between RAR and RTR methods is limited since the RAR experiment seems to produce only one hypothetical response. To make the conclusion more robust, I suggest conducting experiments with the following settings: (i) N=1 vs. N=5(REW) and (ii) N=5, M=1(RTR) vs. N=5, M=5.\n\n[Sec. 5] It is crucial to outline the experimental setup for T5QR, especially regarding its consideration of canonical responses. To ensure fairness, T5QR should be evaluated while taking into account responses in previous turns.\n\n[Overall] Despite the declared limitation on computational costs and efficiency, it would be insightful to provide information about model parameter sizes or approximate numbers (e.g., times, FLOPs, model size, etc.). This information would help readers understand the trade-offs between different methods.\n\n[Overall] To strengthen the paper's contributions, it would be valuable to clarify what the prompting-LLM approach can achieve compared to previous fine-tuned methods. Highlighting the advantages and disadvantages of this approach would offer a better understanding of its practical implications and potential in the field.", "paper_topic_and_main_contributions": "The main contribution of this paper lies in enhancing the effectiveness of Conversational Search through the use of a novel framework based on Language Model Leveraging (LLM). The framework is designed to robustly represent user intents by generating multiple queries and hypothetical responses. Additionally, the paper introduces several approaches for prompting the Language Model Leveraging and aggregating the generated contents in experimental settings.\n\nThe proposed framework is evaluated using TREC CAsT benchmarks, showcasing the efficacy of the methods presented. Overall, the paper's major contribution lies in its empirical applications of employing LLM to enhance the searching capabilities in Conversational AI.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Conversational query reformulation (CQR) \n• [Can You Unpack That? Learning to Rewrite Questions-in-Context] (Elgohary et al., EMNLP-IJCNLP  2019)  \n• [Query and Answer Expansion from Conversation History.] (Yang et al., TREC 2019) Conversational Search \n• [Open-Retrieval Conversational Question Answering] (Qu et al., SIGIR 2020) \n• [Making Information Seeking Easier: An Improved Pipeline for Conversational Search] (Kumar &  Callan, EMNLP 2020) \n• [Improving Conversational Passage Re-ranking with View Ensemble] (Ju et al., SIGIR 2023)\n", "Typos_Grammar_Style_and_Presentation_Improvements": "[Section 3.3] To enhance clarity and ease of understanding, I recommend rearranging the three aggregation methods as follows: \"Max\" (3.3.1) should come first, followed by \"Mean\" (3.3.3), and finally \"Self-consistency\" (3.3.2). This arrangement in an easy-to-hard order might improve the fluency of the section.\n\n[Table 2; Sec. 4.2] It is crucial to ensure consistency in reporting. Many previous works mentioned in the related works section are not reflected in Table 2. To provide a comprehensive comparison, all relevant previous works should be included in Table 2, aligning with the scope and focus of the paper.\n\n\n"}}, {"review_id": "JHl0WR8vDK", "reviewer": "Reviewer_Un6E", "summary": "", "questions": "1. Why are the results for CRDR and ConvGQR on the CAsT-21 dataset not shown in Table 2?\n2. Could you explain why LLM4CS achieves a larger lift on the CAsT-20 dataset compared to the CAsT-19 dataset?\n3. How does the framework proposed in this paper differ from other frameworks that utilize prompts to generate and encode content using LLMs for various tasks?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents a straightforward yet effective prompting framework, LLM4CS, which leverages a Large Language Model for conversational search. The method combines Conversational Query Rewriting (CQR) and Conversational Dense Retrieval (CDR) techniques, rewriting queries to obtain additional information and then encoding it into dense vectors.\n2. LLM4CS demonstrates remarkable performance on three CAsT datasets, particularly excelling on the CAsT-20 and CAsT-21 datasets.\n3. The paper conducts extensive experiments to analyze the impact of different prompting methods and content aggregation techniques.", "reasons_to_reject": "1. The novelty of the proposed framework is somewhat limited. Utilizing pre-trained LLMs to provide additional information may intuitively yield useful results, but the paper lacks novel methods or unexpected findings. It is not entirely clear how the content generated by LLMs leads to performance improvement.\n2. The experiments conducted in the paper may not sufficiently verify the generalizability of the framework. Only one LLM, gpt-3.5-turbo, was utilized. One concern is whether the LLM used (gpt-3.5-turbo) might have seen the dataset used for the experiment, potentially leading to data leakage and affecting the generalizability of the results. It is recommended to conduct experiments with at least two different open-source LLMs with varying parameter scales. Moreover, there should be an analysis of the impact of different text encoders. \n3. The proposed framework seems to be a combination of \"CQR\" and \"CDR,\" i.e. rewriting and then encoding dense vectors. The experiments center around 3 \"CQR\" methods and 3 \"CDR\" methods. However, there is no result of the baseline method that combines CQR and CDR. As far as I know, there is a hybrid version of the baseline method called CRDR, which achieves a result of 0.837 in MRR on the CAsT-19 dataset. I wonder why the results of the hybrid search of CRDR are missing in Table 2.", "questions_to_authors": "1. Why are the results for CRDR and ConvGQR on the CAsT-21 dataset not shown in Table 2?\n2. Could you explain why LLM4CS achieves a larger lift on the CAsT-20 dataset compared to the CAsT-19 dataset?\n3. How does the framework proposed in this paper differ from other frameworks that utilize prompts to generate and encode content using LLMs for various tasks?", "paper_topic_and_main_contributions": "This paper aims to enhance the performance of conversational search by employing a Large Language Model (LLM) to provide additional information in the form of rewritten queries and responses.\n\nThe main contributions of this paper lie in the domain of NLP engineering experiments. The proposed framework, LLM4CS, demonstrates significant improvements in search performance compared to baseline methods. Notably, LLM4CS outperforms baselines on the CAsT-20 and CAsT-21 datasets. However, it is important to acknowledge that the generalizability and robustness of the framework are somewhat limited due to the use of only one LLM in the experiments and the relatively small dataset size.\n\nOverall, while the contributions are valuable, they may be more suitable for a shorter paper rather than a regular long paper.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ujTXTmXUzI", "reviewer": "Reviewer_ARYj", "summary": "", "questions": "1. The anonymized github link is not functional please fix. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. LLM4CS works incredibly well. It outperforms all prior CQR and CDR methods on a wide variety of benchmarks and by a significant margin. \n2. LLM4CS has three methods REW RTR RAR, all of which work better than prior SOTA even before they are aggregated. ", "reasons_to_reject": "1. There is no discussion about the impact of using LLM ( gpt3.5-turbo-16k) for all the LLM4CS prompts before retrieval. While the performance improvements are significant, it is difficult to understand the tradeoffs when it is unclear how slow the LLM4CS system is nor how well it works with a nonclosed source model. Ideally, I would like to see both information on the efficiency of LM4CS relative to other methods and some validation on how model size or scale impacts the performance of LLM4CS\n2. There is no discussion on how the LLM4CS method works with any other forms of retrieval, such as traditional keyword retrieval (e.g. BM25)", "questions_to_authors": "1. The anonymized github link is not functional please fix. ", "paper_topic_and_main_contributions": "The authors introduce LLM4CS and conversation query rewriting and response aggregation framework, which improves the performance of conversational information retrieval by a wide margin. \nThe authors perform an extensive and systematic evaluation on three widely used conversational search benchmarks, CAsT-19, CAsT-20, and CAsT-21, demonstrating how their methodology consistently outperforms previous SOTA by a wide margin, \n\nThe method is simple yet effective and has an LLM that generates Rewriting (REW), Rewriting-Then-Response (RTR), and Rewriting And-Response (RAR) which when paired with an ANCE-based retriever, leads to impressive improvements in retrieval accuracy.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Missing MSMARCO  citation @article{Campos2016MSMA,\n  title={MS MARCO: A Human Generated MAchine Reading COmprehension Dataset},\n  author={Daniel Fernando Campos and Tri Nguyen and Mir Rosenberg and Xia Song and Jianfeng Gao and Saurabh Tiwary and Rangan Majumder and Li Deng and Bhaskar Mitra},\n  journal={ArXiv},\n  year={2016},\n  volume={abs/1611.09268},\n  url={https://api.semanticscholar.org/CorpusID:1289517}\n}\n2."}}]

Author rebuttal:

Meta-review: Reviewers have mixed feelings about this paper.  The third review feels that the simple method is a novel contribution and the first review has a low excitement about this, both with high reviewer confidence.  Ultimately, LLM4CS is a new, if not unexpected, use of large language models and the authors argue that their empirical results are a significant contribution in the rebuttal.
Decision: Accept-Findings