Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
Authors: Fanghua Ye
Meng Fang
Shenghui Li
Emine Yilmaz
Venue: EMNLP 2023 Findings (2023)
Keywords: conversational search
conversational passage retrieval
query rewriting
query reformulation
TL;DR: An novel approach of utilising large language models as query rewriters and rewrite editors to generate informative query rewrites.

Abstract: Query rewriting plays a vital role in enhancing conversational search by transforming context-dependent user queries into standalone forms. Existing approaches primarily leverage human-rewritten queries as labels to train query rewriting models. However, human rewrites may lack sufficient information for optimal retrieval performance. To overcome this limitation, we propose utilizing large language models (LLMs) as query rewriters, enabling the generation of informative query rewrites through well-designed instructions. We define four essential properties for well-formed rewrites and incorporate all of them into the instruction. In addition, we introduce the role of rewrite editors for LLMs when initial query rewrites are available, forming a ``rewrite-then-edit'' process. Furthermore, we propose distilling the rewriting capabilities of LLMs into smaller models to reduce rewriting latency. Our experimental evaluation on the QReCC dataset demonstrates that informative query rewrites can yield substantially improved retrieval performance compared to human rewrites, especially with sparse retrievers.

Enhancing Conversational Search: Large Language Model-Aided
Informative Query Rewriting
Fanghua Ye
Meng Fang
University College London ye.. 19@ucl. ac. uk
University of Liverpool
Fang@laag@li.. senngeliveroool.. ac uk
Shenghui Li
Emine Yilmaz
Uppsala University shenghui. li@it uu.
University College London
emine. yww.azeuclll....m. ac uk
Abstract
What job did Elizabeth have?
Query rewriting plays vital role in enhancing conversational search by transforming context- dependent user queries into standalone forms. Existing approaches primarily leverage human- rewritten queries as labels to train query rewrit- ing models. However, human rewrites may lack sufficient information for optimal retrieval per- formance. To overcome this limitation, we pro- pose utilizing large language models (LLMs) as query rewriters, enabling the generation of in- formative query rewrites through designed instructions. We define four essential proper- ties for well-formed rewrites and incorporate all of them into the instruction. In addition, introduce the role of rewrite editors for LLMss when initial query rewrites are available, form- ing "rewritethen-editt" process. Furthermore, we propose distilling the rewriting capabilities of LLMs into smaller models to reduce rewrit- ing latency. Our experimental evaluation the QReCC dataset demonstrates that informa- tive query rewrites can yield substantially im- proved retrieval performance compared to hu- man rewrites, especially with sparse retrievers.
In what field??
She lecturer midwifery.
Did she de well?
Did Klizabert Blackwell do well?
Elizabeth Blackwell as a lecturer in midwifery? Informartvv sourtth
Human nesrrfe
Figure I: An example showing that human rewrites may overlook valuable contextual information. Specifically, the omission of the phrase "as lecturer in midwifery" makes it challenging for retrieval systems to understand the original query comprehensively.
context modeling by performing query rewriting (Elgohary et al., 2019; Yu et al., 2020; Vakulenko et al., 2021; Wu al., 2022: Mo al., 2023). This approach transforms context-dependen user queries into self-contained queries, thereby allow- ing the utilization of existing off off-the-shelf retrievers that have been extensively validated for standalone queries. For example, the user query "Did she do well?" illustrated in Figure 1 can be rewritten into "Did Elizabeth Blackwell do well as a lecturer in midwifery?" which is context-independent,
1 Introduction
Conversational search has gained significant promi- nence in recent years with the proliferation of digi- tal virtual assistants and chatbots, enabling users to engage multiple rounds interactions to obtain information (Radlinski and Craswell, 2017; Dal- ton et al., 2021; Gao et al., 2023). This emerging search paradigm offers remarkable advantages in assisting users with intricate information needs and complex tasks (Yu al., 2021). However, fun- damental challenge in conversational search lies in accurately determining users' current search intents within the conversational context.
Previous studies (Anantha al., 2021; Vaku- lenko et al., 2021: Qian and Dou, 2022: Hao et al., 2022) predominantly depend on human-rewritten queries as supervised labels to train query rewriting models. Although perform better than the original queries, they may not be informative enough for optimal retrieval per- formance (Chen al., 2022; Wu et al., 2022). This limitation arises from the fact that human rewriters are only concerned with addressing ambiguity is- sues, such as coreference and omission, when trans- forming the original query into a self-contained
queries tend to
An effective approach that has gained increasing attention addresses this challenge of conversational
OOu implementation is available at: https://github com/smarttff/IIIfocC

---

form. Such a simple rewriting strategy may over- look lots of valuable information within the conver- sational context (refer to Figure for an example), which has the potential to enhance the effectiveness of the retriever. As a consequence, existing query rewriting models learned from human rewrites can only achieve sub-optimal performance.
We propose to prompt LLMs as both query rewrit- ers and rewrite editors by providing clear instruc- tions that incorporate all the desirable properties. In addition, we employ distillation techniques to condense the rewriting capabilities of L.MSs into smaller models to improve rewriting efficiency.
We demonstrate the effectiveness of informative query rewriting with two off-the-shelf retrievers (sparse and dense) on the QReCC dataset. Our results show that informative query rewrites can outperform human rewrites, particularly in the context of sparse retrieval.
A straightforward approach to improving the in- formativeness of rewritten queries to provide hu- man annotators with more comprehensive instruc- tions so that they can rewrite the original queries to be not only unambiguous but also informative. However, this approach has several disadvantages, including being expensive, increasing workload for human annotators, and potentially leading to higher inconsistencies among rewrites from differ- ent annotators. Therefore, is necessary explore alternative approaches.
2 Task Formulation
The primary objective of conversational search is to identify relevant passages from a vast collection of passages in response to the current user query. Formally, let Qi and A; be the user query and sys- tem response at turn i, respectively. Furthermore, let X (Q1, A Qt-1, Ar-1) represent the conversational context up to turn t. Then, the task of conversational search can be formulated as re- trieving top-k relevant passages, denoted as Rg, from large passage collection C given the current user query Qt and its associated context XG. This retrieval process is accomplished by a retriever de- fined as f : (Qt, Nc. + R2, where R₂ is a subset of C and l is considerably smaller than the total number of passages in C.
In this paper, we propose the utilization of large language models (LLMs) for query rewriting, lever- aging their impressive capabilities in following in- structions and demonstrations (Brown et al., 2020; Wei et al., 2021; Ouyang et al., 2022: Wei al., 2023). We consider two settings prompt LLMs query rewriters. In the zero-shot learning set- ting, only an instruction provided, while in the few-shot learning setting, both an instruction and a few demonstrations are given. To develop suitable instructions, we first identify four essential proper- ties that characterize a well-formed rewritten query. Then, we design an instruction that incorporates all four properties. However, generating rewrites with all these properties may pose challenges for LLMs due to the intricacy of the instruction (Ouyang et al., 2022: Jang et al., 2023). In view of this, we pro- pose an additional role for LLMS as rewrite editors. Inspired by the fact that humans excel at editing rather than creating from scratch, the purpose of the rewrite editor is to edit initial rewrites provided, forming a newrrie-thennedif process. These initial rewrites can be generated by smaller query rewrit- ing models even by the LLM itself. Furthermore, considering the potential time overhead and high costs associated with LLMs, we suggest distilling their rewriting capabilities into smaller models us- ing their generated rewrites as training labels. Our contributions are summarized as follows:
The unique challenge in conversational search is incorporating the conversational context while retrieving relevant passages, which cannot be di- rectly addressed by cxisting retrievers designed for standalone queries. Additionally, re-training re- trievers tailored for conversational queries can be expensive or even infeasible due to complex sys- tem designs or limited data availability (Wu et al., 2022). To overcome the need for re-training, query rewriting employed an effective solution (Lin et al., 2021c; Mo et al., 2023). Query rewriting involves transforming the context-dependen user query Q: into a self-contained standalone query Q' by extracting relevant information from the con- text X, Consequently, any existing off-the-shelf retrieval systems designed for standalone queries can be leveraged by taking Q', as the input query to find passages that are relevant to the original user query Qr, i.e., f (Q,,C) + Rg-
We are the first introduce the concept of infor- mative conversational query rewriting and metic- ulously identify four desirable properties that a well-crafted rewritten query should possess.
The utilization of query rewriting shifts the chal- lenge of modeling conversational context from the retriever end to the query rewriting model end. As

---

Instruction to prompt LLM as query rewriter
Instruction to prompt LLM as rewrite editor
Given question and context rewrtte that decontextualizes question, edit the rewnie to ereate revised version that fully addresses coreferences omissions the question without changing original meaning the question but providing should not duplicate any previously asked questions the context, If there need edit rewrite, the rewrite
Given a question and its contexi, decontextuaiie: the question by addrassing coreference should rerain its original meaning and be as informative as possible, should not duplicate any previously asked questions the context.
omission issues. The resulting question
information. The rewrite
" demonstrations
ar demonstrations
Who proposed that aloms the basic units matter? John Dalton proposed that each chomical element is compesed of o'atoms a single, unique type, and they combine form more complex structures called chemical compoundd.
Q: Who proposed that atoms the basic units of matier? A: John Dalton proposed that each chemical elemeent composed of atoms of single, unique typc, and they can combine form more complex structures called chemical compounds.
Question: did the proposal come about?
Question:
proposal Rewrite: How did John Dalton's propesal come about?
about?
Revrite: How did John Dalton's proposal that each chemical element composed atoms single unique type, they can combine form more complex structures called chemical compoonnds come about?
How
proposal
chemical element
compased of atoms of single unique type, and they can combine
form more complex structures called chemical compounds come about?
Test instance
Test instance
Q: What job did Elizabeth Blnckwell have?
What job blizabeth Blackwell have?
lecturer midwifery.
A: A lecturer in midwifery.
Question: Did she do well? Rewrite: Did Elizabeth Dlackwell de well as lecturer?
Question: Did do well?
Rewnite:
Edit:
LLM
Did Elizabeth Blackwell do well lecturer?
LLM
Did Elizabeth Blackwell do well lecturer midwifery?
Figure 2: Our proposed approach involves prompting LLMs as query rewriters and rewrite editors through clear and well-designed instructions, along with appropriate demonstrations. In the absence of demonstrations, the LLM functions as a zero-shot query rewriter. We explicitly incorporate the requirement that rewritten queries should be as informative as possible into the instructions generating informative query rewrites.
thus, the effectiveness of retrieval results heavily re- lies on the employed query rewriting models. Only when appropriate rewritten queries are generated off-the-stee shelf retrieval system return highly relevant passages.
that a well-crafted rewritten query should possess:
The rewritten query should pre- serve the meaning of the original query, ensuring that the user's intent remains unchanged.
Correctness:
The rewritten query should be unam- biguous and independent of the conversational context, enabling it to be comprehensible by peo- ple outside the conversational context. This clar- ity can be achieved by addressing coreference and omission issues arising in the original query.
Clarity:
3 Approach
In contrast to relying on human annotators to gener- ate more informative rewrites or developing more complex models to closely replicate existing human rewrites, propose to prompt LLMs to generate informative query rewrites simply by providing clear instructions and appropriate demonstrations, avoiding the requirement for extensive human ef- fort and complicated model designs. Figure illus- trates our proposed approach.
The rewritten query should in- corporate much valuable and relevant informa- tion from the conversational context possible, thereby providing more useful information to the off off-the-shelf retriever.
Informativeness:
3.1 Prompting LLM as Query Rewriter
Nonredundancy: The rewritten query should avoid duplicating any query previously raised in the conversational context, as it is important ensure that the rewritten query only conveys the intent and meaning of the current query.
Recent work (Wei et al., 2021; Ouyang et al., 2022; Peng et al., 2023) has demonstrated the strong ca- pability of LLMs in following given instructions to generate coherent and contextually appropriate text. Inspired by this, it natural consider employing LLMs as query rewriters. Before delving into the details of how we can prompt an LLM as a query rewriter, we first describe the desirable properties
In order to effectively instruct an LLM in gen- crating query rewrites that embody the aforemen- tioned four properties, is essential to formulate

---

appropriate instructions. As an illustrative example, we adopt the following instruction in this work:
the instance (A₁, Qi) as the prompt to the LLM, the rewrite Q₁ is then sampled as follows:
"Given a question and its context, de- contextualize the question by addressing coreference and omission issues. The re- sulting question should retain its original meaning and be as informative as possi- ble, and should not duplicate any previ- ously asked questions in the context." "2
Qf LCM(II|D)
(3)
Note that the query rewrites utilized in the demon- strations should be well-designed, ensuring that they have the aforementioned four properties. Oth- erwise, the LLM may be misled by these demon- strations. For a more detailed description of the demonstrations used in our experiments, please re- fer to Appendix D.
This instruction takes all four desirable properties of good rewritten query into account simultane- ously. Building upon this instruction, we explore two settings to prompt an LLM for query rewriting,
3.2 Prompting LLM as Rewrite Editor
Despite the proficiency of LLMSs following in- structions and demonstrations, recent work (Dong et al., 2022; Liu et al., 2022; Moshach et al., 2023) suggests that they may encounter difficulties when faced with complex tasks or intricate requirements. This limitation highlights that it can be challenging for LLMs to generate query rewrites with all the desirable properties mentioned above. In order address this challenge, we propose an alternative approach in which an LLM is prompted editor whose primary function to edit provided initial rewrites instead of being prompted as a query rewriter who needs to generate query rewrites from scratch. This approach draws inspiration from the observation that humans often find it easier to edit existing content than create it from scratch.
3.1.1 Zero-Shot Learning (ZSL) Setting In the ZSL setting, the LLM is instructed to gener- ate rewritten query Q/ using only the information provided by the current query Qe and its associated conversational context X, without having access to any human-labeled instances. this setting, we entirely rely on the LLM's capability to understand and follow instructions to perform query rewriting. Specifically, we append Xe and Qr to the instruc- tion as the prompt and feed this prompt to the LLM for sampling the rewrite Qi:
rewrite
di (CM()
(1)
where || denotes concatenation. The detailed for- mat of the prompt is shown in Appendix D.
In this work, we adopt the FSL setting prompt LLMs as rewrite editors. addition to the query Q the conversational context X, and the rewrite Q' we introduce an initial rewrite Q for each demon- stration. We represent the concatenation of these augmented demonstrations as:
3.1.2 Few-Shot Learning (FSL) Setting In the FSL setting, the LLM is provided with both the instruction and a small number of demonstra- tions. This type of prompting is commonly referred as in-context learning, which has been shown to be effective in adapting LLMs to new tasks (Brown et al., 2020; Min et al., b022a,b; Wei et al., 2023; Sun et al., 2023; Ram et al., 2023). In this setting, each demonstration consists of query Q, con- versational context X., and a rewrite Q'. We denote the concatenation of these demonstrations
D = Srr,QQ,C, "","".Q"))1 ".""""")1...((("""" Q", Q", QT)).
(4)
For test instance (A, Qi), accompanied by an ini- tial rewrite Qt. we obtain the edited (final) rewrite Q through the following procedure:
(5) where i denotes the modified instruction. Please refer to Figure and Appendix D for details.
Oi LCM
(x), ("."")11 II("",,,,,,,, Q" Q"").
(2)
The initial rewrite can be generated by a small query rewriting model, such as T5QR (Lin et al., 2020; Wu et al., 2022). It can also be generated by LLM, following the prompting method de- scribed in the previous subsection. When an LLM employed as both the query rewriter and rewrite editor, the "rewrite- teeeee-hee--iiiiee process enables the LLM to perform celf-correction (Gou et al., 2023).
where n represents the total number of demonstra- tions. By placing DD between the instruction I and
"Note that in this instruction, we use the term "question" instead of "query". as each query is referred to as a question in the dataset we employed for experimentation. We believe this tiny variation would not significantly impact the quality of generated rewrites. Additionally, other instructions with a similar meaning can also be applicd.

---

3.3 Distillation: LLM as Rewriting Teacher One major obstacle in effectively leveraging LLMs for query rewriting is the substantial demand for memory and computational resources (Hsich et al., 2023), which can further result in significant time overhead. Besides, the cost can be extremely high when there is a lack of in-house models, necessitat- ing the reliance on third-party API services as the only option. To address these issues, we propose to fine-tune small query rewriting model using rewrites generated by an LLM as ground-trus -oanndtruthhho la- bels. In this approach, the LLM assumes the role of a teacher, while the smaller query rewriting model acts as student. The fine-tuning process distills the teacher's i rewriting capabilities into the student. This technique is known knowledge distillation (Gou et al., 2021) and has recently been utilized to distill LLMs for various other tasks (Shridhar et al., 2022: Magister et al., 2022; Marjich et al., 2023). Following previous work (Lin et al., 2020; Wu et al., 2022), we adopt T5 (Raffel et al.. 2020) as the student model (i.e., the small query rewriting model). The input to the model is the concatenation of all utterances in the conversational context X; and the current user query Qt. In order to differenti- ate between user queries and system responses, we prepend a special token <Que> to each user query and a special token <Ans> to each system response. The output of the model is the rewrite Q, which is sampled from the employed LLM. The model is fine-tuned using the standard cross-entropy loss to maximize the likelihood of generating Q/
are sourced from three existing datasets, includ- ing QuAC (Choi al., 2018), Natural Questions (K wiatkowski et al., 2019), and TREC CAsT-19 (Dalton et al., 2020). For ease of differentiation, we refer to these subsets as QuAC-Conv, NQ-Conv, and TREC-Com respectively. Note that TREC- Conv only appears in the test set. For comprehen- sive evaluation, we present experimental results not only on the overall dataset but also on each subset. For additional information and statistics regarding the dataset, please refer to Appendix A.
To evaluate the retrieval results, we adopt mean reciprocal rank (MRR), mean average precision (MAP), and Recall @ 10 (R @ 10) evaluation met- rics. We employ the pytrec_ _ealll toolkit (Van Gy- sel and de Rijke, 2018) for the computation of all metric values.
4.2 Comparison Methods
Since our focus is on the effectiveness of informa- tive query rewrites, two straightforward baseline methods are Original, which uses the user question in its original form as the search query, and Human, which utilizes the query rewritten by a human as the search query. We also include three supervised models as baselines, including T5QR (Lin et al., 2020), which fine-tunes the T5-base model (Raffel et al., 2020) seq2seq query rewriter, ConQRR (Wu al., 2022), which employs reinforcement learning to train query rewriting models by directly optimizing retrieval performance, and ConvGQR (Mo et al., 2023), which combines query rewriting with potential answer generation to improve the informativeness of the search query.
4 Experimental Setup
4.1 Dataset Evaluation Metrics
For our proposed approach, we investigate four variants, namely RW(ZSL), RW(FSL), ED(Self), and ED(TSQR). RWSZLL prompts an LLM as a query rewriter in the ZSL setting, while RW(FSL) prompts an LLM as a query rewriter in the FSL set- ting. By comparison, ED((elf) prompts an LLM as rewrite editor, wherein the initial rewrites are gen- erated by RW(FSL) with the same LLM applied. ED(TSQR) also prompts an LLM as a rewrite edi- tor, but the initial rewrites are generated by TSQR. For simplicity, we only prompt LLMs as rewrite editors in the FSL setting.
Following previous work (Wu et al., 2022; Mo et al., 2023), we leverage QReCC (Anantha et al.. 2021) as our experimental dataset. QRcCC consists of 14K open-domain English conversations with a total of 80K question-ane -nnteer pairs. Each user ques- tion is accompanied by human-rewritten query, and the answers questions within the same con- versation may be distributed across multiple web pages. There are in total 10M web pages with each divided into several passages, leading to a collec- tion of 54M passages². The task of conversational search is to find relevant passages for each question from this large collection and gold passage labels are provided if any. The conversations in QRcCC
4.3 Retrieval Systems
We experiment with two types of off-the-shelf re- trievers to explore the effects of informativeness in query rewrites on conversational search:
'The datasct and passage collection are available at https: // //zenodo. or anggrrccor///// 15890. YZ8kab3MI-O.

---

(II) Our approaches, ED(Self) and ED(TSQR), consistently achieve the best and second-best results on the overall QReCC test set. Notably, they both surpass human rewrites. For example, ED(Self) demonstrates a substan- tial absolute improvement of 9.58 in MRR scores for sparse retrieval compared to human rewrites. (W(FLL) also performs better than human rewrites, while RW(ZSL) fails to show consistent improve- ments over human rewrites. These results empha- size the value of informative query rewriting and in-context demonstrations. (III) The supervised models, T5QR and ConQRR, exhibit worse per- formance than human rewrites, suggesting that relying solely lcarning from human rewrites leads sub-optimal results. Although ConvGQR beats human rewrites in sparse retrieval, its per- formance gain mainly derives from generated po- tential answers rather than more informative query rewrites. (IV) Dense retrieval improvements are QReCC (8209)
QuAC-Conv (6396)
NQ-Conv (1442)
TREC-Come (371)
Query
MRR
MAP
R@ 10
MRR
MAP
R@ 10
MRR
MAP
R@10
MRR
MAP
R@10
Original
9.30
8.87
15,50
9.29
8.84
15.20
9.06
8.64
15.14
10.30
10.27
22.10
Human
39.81
38.45
62.65
40.32
38.98
62.90
40.78
39.05
63.80
27.34
27.04
53.77
Sparse (BM255
TSQR
33.67
32.50
53,68
34.04
32.90
53.83
34.24
32.66
53.92
25.23
24.96
50.13
ConQRR
38,30
60,10
39.50
61.60
37.80
58.00
19.80
43,50
ConvGQR
44.10
64,40
RW(ZSL)
42.63
41.31
60.46
45.43
44.11
63.20
36.43
34.81
54.69
18.50
18.26
35.58
RW(FSL)
46.96
45.53
65.57
49.81
48.38
68.28
41.51
39.71
60.13
19.02
18.86
39.89
ED(Self)
49,39
47.89
67.01
53.01
51.52
70.46
41.57
39.69
59.63
17.43
17.08
36.25
ED(TSQR)
47.93
46.40
66,25
50.67
49.18
68.84
42.69
40.64
60.67
21.04
20.79
43.26
Original
12.12
11.49
18.74
11.34
10.69
17.79
13.11
12.57
19.49
21.76
21.11
32.08
Human
43.15
41.27
66.12
40.67
38.92
64.59
54.01
51.25
73.13
43.74
42.98
65.23
Dense (GTR)
TSQR
37.67
35.93
58.65
35.51
33.88
57.23
46.95
44.47
64.43
38.94
38.16
60.51
ConQRR
41.80
65.10
41.60
65.90
45.30
64.10
32.70
55.20
ConvGQR
42.00
63.50
RW(ZSL)
40.64
38.95
62.28
40.12
38.48
62.47
44.85
42.57
63.58
33.26
32.88
54.09
RW(FSL) 43.89
42.09
66.45
43.50
41.78
66.87
48.60
46.12
68.10
32.37
31.79
52.65
ED(Self) 44.99
43.19
67.34
45.21
43,48
68.30
47 64
45.20
67.27
30.91
30.48
51.03
ED(TSQR) 44.76
42.90
66.64
44.29
42.50
66.65
49.67
47.12
69.22
33.90
33.43
56.47
Table 1: Passage retrieval performance of sparse and dense retrievers with various query rewriting methods the QReCC test set and its three subsets. The best results are shown in bold and the second-bes results are underlined.
BM25 BM25 (Robertson al., 2009) is classic sparse retriever. Following Anantha et al. (2021), we employ Pyserini (Lin et al., 2021a) with hy- parameters k1 0.82 and = 0.68.
5
Experimental Results
5.1 Main Results
Table I presents the retrieval performance of dif- ferent query rewriting methods on the QReCC test set and its subsets. Our key findings summa- rized follows. (I) All query rewriting methods outperform the original query, validating the im- portance of query rewriting.
GTR GTR (Ni et al., 2022) a recently proposed dense retriever'. It has a shared dual-encoder archi- tecture and achieves state-of-the-ar performance on multiple retrieval benchmarks.
4.4 Implementation Details
We adopt ChatGPT (gpt-3.5-tte 5-turbo) provided by OpenAl through their official API as the LLM in our experiments. During inference, we use greedy decoding with a temperature of O. In the FSL set- ting, we utilize four demonstrations (i.e., = 1). We employ Pyserini (Lin et al., 2021a) for sparse retrieval and Faiss (Johnson al., 2019) for dense retrieval. For each user query, we retrieve 100 pas- sages (i.e., k 100). We disregard test instances without valid gold passage labels. As a result, we have 8209 test instances in total, with 6396, 1442, and 371 test instances for QuAC-Conv, NQ-Conv, and TREC-Come respectively. For more implemen- tation details, please refer to Appendix B.
*We use the TS-base version https://huggingfaeet co/ sentence- waanffrrmrs//grr-teee 'platform. opemai. com

---

QuAC-Conv
NQ-Conv
TREC-ConY
Query
MRR
MAP
R@10
Query
AT
%0T
AT
%0T
AT
%0T
Human
39.81
38.45
62.65
Sparse (BM25)
Original
6.75
61.23
6.31
64,41
6,01
74.76
RW(ZSL)
42.63
41.31
60.46
TSQR
9.80
83.63
8.67
85.27
7.31
90.32
RW(ZSL)
41.52
40.20
59.60
RW(ZSI)
16.51
68.71
14.48
73.55
12.53
65.03
RW(FSL)
16.95
76.66
15.74
81.68
15.07
74.11
Dense (GTR)
Human
43.15
41.27
66.12
ED(Self)
22.00
78.
19.82
82.99
21.85
76.22
RW(SSL)
40.64
38.95
62.28
ED(TSQR)
17.93
85.16
15.08
89.28
15.66
87.78
RW(SSL)
39.94
38.28
61.44
Human
10.76
100
8.98
100
7.35
100
Table 3: Ablation study by removing the informative- ness requirement from the instruction in RW(ZS)).
Table 2: Average number of tokens (AT) and the per- centage of overlapping tokens (OT) with human rewrites in queries produced by different rewriting methods.
of informative query rewriting and the effectiveness of our proposed approaches of prompting LLMs query rewriters and rewrite editors. The study also suggests that is crucial take query characteris- tics into account when performing query rewriting with LLMs, which we leave as our future work.
less effective than sparse retrieval. For example, ED(Self) only outperforms human rewrites by 1.84 MRR scores when using the dense retriever GTR. This discrepancy arises due to the need for domain- specific passage and query encoders in dense re- trieval. In our experiments, the GTR model is kept fixed without fine-tuning, which limits the full po- tential of dense retrieval. Besides, ConvGQR also shows inferior dense retrieval performance, further indicating that a fixed general dense retriever can- not fully demonstrate the superiority of informa- tive query rewrites. (V) Breaking down the results by subsets reveals that our proposed approaches can consistently achieve higher performance on the QuAC-Conv subset. They also prevail in terms of MRR and MAP for sparse retrieval and achieve the sccond-bes results for dense retrieval the NQ- approaches are inferior
5.2 Quantitative Analyses of Query Rewrites
The previous results have demonstrated the effec- tiveness of informative query rewrites generated by our proposed approaches in enhancing conversa- tional search. To gain more insights into the quality of these rewrites, we employ the average number of tokens per rewrite as a measurement of infor- mativeness and the percentage of tokens in human rewrites that also appear in the generated rewrites as a measurement of correctness. We assume that rewrite containing more tokens from the corre- sponding human rewrite is more likely be cor- rect. The results are shown in Table 2. We observe that our proposed approaches consistently generate longer rewrites than human rewrites, with ED(Sclf) producing the longest rewrites overall. This im- plies that the rewrites generated by our proposed approaches are more informative. We also observe that T5QR generates shorter rewrites than human rewrites, indicating that relying solely learning from human rewrites fails produce informative rewrites. Moreover, our proposed approaches, al- beit without supervised fine-tuning, achieve rela- tively high correctness compared to human rewrites. For example, more than 76% of tokens in human rewrites are included in the rewrites generated by ED(Self). ED(T5QR) even exhibits higher correct- than T5QR on the QuAC-Conv and NQ-Com subsets. Finally, the longer rewrites and higher per- centage of shared tokens with human rewrites (ex- cept RW on TREC-Conv), compared to the original queries, suggest to some extent that the rewrites generated by our approaches have fair clarity.
Conv subset. However, human rewrites and T5QR on the TREC-Comy subset. One reason is that TREC-Conv contains many obscure questions, making it challenging for an LLM to accurately understand true user needs. It is seen that even human rewrites perform worse on TREC-Conv than on QuAC-Conv and NQ-Conv in sparse retrieval. Additionally, the questions in TREC-Conv are more self-contained and require less rewriting, evidenced by higher ROUGE-1 scores (Lin, 2004) between human rewrites and original questions compared to QuAC-Conv and NQ-Conv. Specifically, the ROUGE-1 scores on TREC-ConY, QuAC-Conv, and NQ-Conv are 60,60, 69.73, and 72.16. respectively. (VI) Both ED(felf) and ED(T5QR) outperform RW(FSL), showing the significance of prompting LLMs as rewrite editors. While ED(T5QR) performs worse than ED(Self) on the overall QReCC test set, excels on the NQ- Conv and TREC- subsets, benefiting from the fact that TSQR is trained with human rewrites.
In summary, this study confirms the importance

---

Human RINFSL) EDISen
Human
Human RWIF
RIVIFBLI EDySeir
RWIFSL)
ED/Seff
EDiSeri
30
20
30-
30
MRR
MAP
RB10
MRR
MAP
R@10
MRR
MAP
R:9:00
MRR
ИАР
Rg10
(a) QReCC
(b) QuAC-Conv
(c) NQ-Conv
TREC-Conv
Figure 3: Distillation results of using BM25 the retriever. The legends indicate the sources of fine-tuning labels.
5.3 Ablation Study
and Craswell, 2017; Rosset et al., 2020). al- lows users to provide and seek clarifications (Xu et al., 2019) and explore multiple aspects of topic, thereby excelling at fulfilling complex informa- tion needs. The primary challenge in conversa- tional search is accurately identifying users' search intents from their contextualized and potentially ambiguous queries (Ye et al., 2022a; Keyvan and Huang, 2022; Ye et al., 2022b; Wang et al., 2023; Owoicho et al., 2023; Zhu et al., 2023).
We conduct an ablation study by removing the in- formativeness requirement from the instruction uti- lized by RW(Z)L) (i.e., removing the phrase "and be as informative as possible"), resulting in mod- ified version denoted as RW(Z)L). Table reports the results. We find that for both sparse and dense retrieval, RWSS)) achieves lower performance across all three evaluation metries than RWSL)) demonstrating that it is valuable to incorporate in- formativeness requirement into the instruction for generating informative query rewrites. Interest- ingly, RW(ZSL) outperforms human rewrites in terms MRR and MAP for sparse retrieval, which again verifies the notion that human rewrites may fail to yield optimal retrieval performance. See Ap- pendix C.3 for ablation results of the other three desirable properties.
Most existing work (Yu al., 2021; Lin et al., 202 lb; Kim and Kim, 2022; Li et al., 2022a; Mao et al.. 2022) addresses this challenge by regarding the concatenation of the current user query with its associated conversational context as a standalone query. However, using this concatenation directly as input to search systems could result in poor trieval performance (Lin et al., 2021b). Moreover, this approach requires training specialized retriev- ers such as dual encoders (Karpukhin al., 2020; Xiong al., 2020; Khattab and Zaharia, 2020), which can be challenging or even impractical many real-world scenarios (Wu et al., 2022).
5.4 Distillation Results
Figure shows the distillation results using BM25 as the retriever. In this study, we sample 10K train- ing instances and employ RW(FSL.) and ED(Self) to generate labels for fine-tuning the T5QR model. For comparison, we include the results with human rewrites as training labels. We find that distillation outperforms using human rewrites labels on the QReCC test set. Notably, distillation with only 10K training instances can achieve superior results than directly utilizing human rewrites as search queries in terms of MRR and MAP. On the QuAC-Conv and NQ-Conv subsets, distillation also consistently demonstrates improved performance. However, for TREC-Com fine-tuning with human rewrites leads better outcomes. Distillation only improves retrieval performance but also reduces time over- head. See Appendix C.4 for latency analyses.
Another line of research addresses this challenge through query rewriting (Elgohary al., 2019; Wu et al., 2022: Qian and Dou, 2022: Yuan et al., 2022: Li et al., 2022b; Mo et al., 2023), which converts the original query into a standalone one. However, these approaches mainly rely on human rewrites to train query rewriting models. As shown in our experiments, human rewrites may lack sufficient informativeness, hence leading sub-optimal per- formance of these rewriting models.
A Alternatively, some studies employ query expan- sion to address this challenge. They select relevant terms from the conversational context (Voskarides et al., 2020; Kumar and Callan, 2020) generate potential answers (Mo et al., 2023) augment the original query. The latter can be seamlessly inte- grated into our approach to leverage the knowledge within LLMs. We leave this study as future work.
6 Related Work
Conversational search addresses users' informa- tion needs through iterative interactions (Radlinski

---

ios where passages are relatively short, the intro- duction of more information in the search query In this work, we propose to prompt LLMs as query rewriters and rewrite editors for informative query rewrite generation. We are the first to introduce the concept of informative query rewriting and iden- tify four properties that characterize well-formed rewrite. We also propose distilling the rewriting ca- pabilities of LLMs into smaller models improve efficiency. Our experiments verify the significance informativeness in query rewrites and the effec- tiveness of using LLMS for generating rewrites.
7 Conclusion
may have a detrimental effect, as it becomes more challenging for retrieval systems to determine the most relevant passages. Conversely, informative query rewriting should prove beneficial in the con- text of long passages document retrieval.
Thirdly, in this work, we have experimented with only one LLM, namely ChatGPT, and therefore our findings may be biased toward this specific model. It is unclear if other LLMs can achieve the same level of performance. Further investigation with more LLMs is worthwhile.
Despite the superb performance achieved by our proposed approach, there are multiple future direc- tions that are worthy of exploration. For example, we can train auxiliary model to decide whether queries generated by prompting LLMs should be preferred to those generated by models that have been finetuned on human rewrites. We can also combine human rewrites and LLM rewrites as la- bels through proper weighting strategy to finetune query rewriting models. Furthermore, in this work, we have used fixed set of demonstrations for all test queries. To achieve the best performance, it is essential to find appropriate demonstrations for each specific query. This would be an effective solution to tackle obscure or complicated queries. Another future direction can be parameter-efficient fine-tuning (e.g., LoRA (Hu et al., 2021)) of LLMs with retrieval performance as feedback. In this way. we will aim to optimize the helpfulness of rewritten queries rather than informativeness.
Ethics Statement
Query rewriting plays a crucial role as an intermedi- ary process in conversational search, facilitating a clearer comprehension of user search intents. This process is beneficial in generating appropriate re- sponses to users. The effectiveness of this approach can be further enhanced through informative query rewriting, resulting in the retrieval of more relevant passages. Nevertheless, is important to acknowl- edge that our proposed approaches are subject the inherent limitations of L.MMs, such as halluci- nations, biases, and toxicity. It is also important to filter out passages that contain offensive text from the passage collection to ensure reliable retrieval results when applying our proposed approaches in practical scenarios.
Acknowledgements
Limitations
We identify three limitations our proposed ap- proach. Firstly, the utilization of LLMs as query rewriters and rewrite editors inevitably suffers from the shortcomings associated with LLMs. Our ex- periments indicate that LLMs do not always fol- low provided instructions, resulting in generated rewrites that fail possess the four desirable prop- erties. For example, these rewrites may contain du- plicate questions from the conversational context, thereby violating the nonredundancy requirement. In Appendix C.5, we present case study demon- strating that the original user query may even be misinterpreted, leading incorrect query rewrites. Secondly, although our experimental results have demonstrated improved retrieval performance, is essential to emphasize that the effectiveness of informative query rewriting is highly dependent on the formatting of the passage collection. In scenar-
This work was funded by the EPSRC Fellowship titled "Task Based Information Retrieval" (grant reference number EP/P024289/1) and the Alan Tur- ing Institute.
References
Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre. Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 520-534, Online. Association for Computational Lin- guistics.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen,

---

al. 2016, Ms marco: A human generated ma- chine reading comprehension dataset. arXiv preprint arXiv: 1610.66.....
Jie Hao, Yang Liu, Xing Fan, Saurabh Gupta, Saleh Soltan, Rakesh Chada, Pradeep Natarajan, Chenlei Guo, and Gokhan Tur. 2022. CGF: Constrained gen- eration framework for query rewriting in conversa- tional AL In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 475-483, Abu Dhabi, UAE. Association for Computational Linguistics.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:187711
Cheng-Yu Yu Hsich, Chun-Liang Li, Chib-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint (rXXii2305.02000
Zhiyu Chen, Jie Zhao, Anjie Fang, Besnik Fetahu, Oleg Rokhlenko, and Shervin Malmasi. 2022. Reinforced question rewriting for conversational question an- swering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 357-370, Abu Dhabi, UAE, Association for Computational Linguistics.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhi Yuanzhi Li. Shean Wang. Lu Wang. Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learn- ing Representations.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages Brussels, Belgium. Association for Com- putational Linguisties.
Joel Jang. Seonghyeon Ye, and Minjoon Seo. 2023. Can large language models truly understand prompts? case study with negated prompts. In Transfer Learning for Natural Language Processing Work- shop, pages 52-62. PMLR.
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. Cast 2020: The conversational assistance track overview.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions Big Data, 733):535-547.
Jeffrey Dalton, Chenyan Xiong, Vaibhav Kumar, and Jamie Callan. 2020. Cast-19: 19: A dataset for conver- sational information seeking. In Proceedings of the 43rd International ACM SIGIR Conference on Re- search and Development in Information Retrieval,
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu. Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6788 6781, Online. Association for Computational Linguistics.
pages 1985-1988.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. survey for in-context learning. arXiv preprint arXiv:2301.00334
Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys, 55(6):1- 40.
Ahmed Elgohary, Denis Peskov, and Jordan Boyd- Graber. 2019. Can you unpack that? learning to rewrite
contexx. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP), pages 5918-5924, Hong Kong, China. Association for Computational Linguistics.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pages 39- 48.
Jianfeng Gao. Chenyan Xiong, Paul Bennett, and Nick Craswell. 2023. Neural Approaches to Conversa- tional Information Retrieval, volume 44. Springer Nature.
Sungdong Kim and Gangwoo Kim. 2022. Saving dense retriever from shortcut dependency in conversational search. In Proceedingy of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10278-10287 Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129: 789-1819.
Vaibhav Kumar and Jamie Callan. 2020, Making in- formation seeking easier: An improved pipeline for conversational search. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3971-39800 Online. Association for Computational Linguistics.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint urXiv:2305.11738

---

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Epstein, Ilaa Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Trunsactions of the Association for Computational Linguisties, 7:453- 466.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. arXiv preprint urXiv:2212.088
Kelong Mao, Zhicheng Dou, and Hongjin Qian. 2022. Curriculum contrastive context denoising for few- shot conversational dense retrieval. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 176-186.
Yongqi Li, Wenjie Li, and Liqiang Nie. 2022a. Dynamic graph reasoning for conversational open-domain question answering. ACM Transactions on Infor- mation Systems (TOIS), 4044):1-24.
Raja Marjieh, Ila Sucholutsky, Pol van Rijn, Nori Ja- coby, and Thomas L Griffiths. 2023. What language reveals about perception: Distilling psychophysi- cal knowledge from large language models. arXiv preprint 203.v22300.01.00
Yonggi Li, Wenjie Li, and Liqiang Nie. 2022b. MM- CoQA: Conversational question answering over text, tables, and images. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics VVoluee 1: Long Papers), pages 4220- 4231, Dublin, Ireland. Association for Computational Linguistics.
Sewon Min. Mike Lewis. Luke Zettlemoyer. and Han- nanch Hajishirzi. 2022a. MetalCI: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2791-2809, Seattle, United States. Association for Computational Linguisties.
Chin Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Sammariza- tion Branches Out, pages 41, Barcelona, Spain. Association for Computational Linguistics.
Jimmy Lin, Xueguang Ma, Sheng-Chier Lin, Jheng- Hong Yang, Ronak Pradeep. and Rodrigo Nogueira. 2021a la. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356- 2362.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis. Hannanch Hajishirzi, and Luke Zettle- moyer. 2022b. Rethinking the role demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-1 11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Sheng-Chich Lin, Jheng-Hong Yang. and Jimmy Lin. 2021b. Contextualized query embeddings for conver- sational search. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1004-1015, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, and Jian-Yun Nic. 2023. Convgqr: Generative query reformulation for conversational search. arXiv preprini arXiv:2305. 75645.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel. Di- etrich Klakow, and Yanai Elazar. 2023. Few-shot fine-tuning vs. in-context learning: fair comparison and evaluation. arXiv preprint arXiv:2305.1690
Sheng-Chie Chich Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, JJuan-Ju Wang, and Jimmy Lin. 2020. Conversational question reformulation via scquence-to-sqqeence architectures and pretrained language models. arXiv preprint arXiv:2004.011 0.9909.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez. Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming- Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9855, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Sheng-Chieh Lin, Jheng- Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2021c. Multi-stage conversational passage retrieval: An approach to fusing term importance estimation and neural query rewriting. ACM Transactions on Information Systems (TOIS), 1994).1229.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta. Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems. 35: 1950-
Paul Owoicho, Ivan Sekulić, Mohammad Aliannejadi. Jeffrey Dalton, and Fabio Crestani. 2023. Exploiting simulated user feedback for conversational search: Ranking, rewriting, and beyond. arXiv preprint arXiv:2304. 13874.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:171..0110..

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval An extremely fast python interface to tree_eval. In The 41st International ACM SIGIR Con- ference on Research Development in Information Retrieval, pages 873-876.
Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, and Maarten de Rijke. 2020. Query reso- lution for conversational search with limited supervi- sion. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pages 921-930.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint 2.Xiv:2304.0327..
Hongjin Qian and Zhicheng Dou. 2022. Explicit query rewriting for conversational dense retrieval. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 4725- 4737, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zhenduo Wang, Zhichao Xu, Qingyao Ai, and Vivek Srikumar. 2023. An in-depth investigation of user response simulation for conversational search. arXiv preprint 2rX0:23004.0944
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652
Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 conference on conference human infor- mation interaction and retrieval, pages 117-126.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tram, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, al. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03844
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21 (1)44555555551.
Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reit- ter, Hannaneh Hajishirzi, Mari Ostendorf, and Gau- rav Singh Tomar. 2022. CONQRR: Conversational query rewriting for retrieval with reinforcement learn- ing. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 10000- 10014, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Brown, and Yoav Shoham. 2023. In-context aelreneal-uummented lan- guage models. arXiv preprint 2rXhv:2302.00000
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- Irieval, 3(4)3333-389
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul Bennett, Junaid Ahmed, and Amoll Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.0000 00808..
Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Cam- pos, Nick Craswell, Saurabh Tiwary, and Paul Ben- nett. 2020. Leading conversational search by sug- gesting useful questions. In Proceedings of the web conference 2020, pages 1160-1170.
Jingjing Xu, Yuechen Wang, Duyu Tang, Nan Duan, Pengcheng Yang. Qi Zeng, Ming Zhou, and Xu Sun. 2019. Asking clarification questions in knowledge- based question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-NCNLLP), pages 1618-1629, Hong Kong, China. Association for Computational Linguistics.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling multi-step reasoning ca- pabilities of large language models into smaller mod- els via semantic decompositions. arXiv preprint arXiv:2212.00193:
Simeng Sun, Yang Liu, Dan Iter, Chenguang Zhu, and Mohit lyyer. 2023. How does in-context learning help prompt tuning? arXiv preprint
Fanghua Ye, Yue Feng, and Emine Yilmaz. 2022a. AS- SIST: Towards label noise-robust dialogue state track- ing. In Findings of the Association for Computa- tional Linguistics: ACL 2022, pages 2719-271 2731, Dublin, Ireland. Association for Computational Lin- guistics.
arXiv:2302.1152 11521.
Svitlana Vakulenko. Shayne Longpre, Zhucheng Tu, and Raviteja Anamtha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM international conference on web search and data mining, pages 355-363.
Fanghua Ye, Xi Wang, Jie Huang, Shenghui Li, Samuel Stern, and Emine Yilmaz. 2022b. MetaASSIST: Robust dialogue state tracking with meta learning.

---

Proceedings the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1197-1169, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Shi Yu, Jiahua Liu, Jingqin Yang. Chenyan Xiong, Paul Bennett, Jianfeng Gao, and Zhiyuan Liu. 2020. Few- shot generative conversational query rewriting. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Informa- tion Retrieval, pages 1933-1936.
Shi Yu, Zhenghao Liu, Chenyan Xiong, Tao Feng, and Zhiyuan Liu. 2021. Few-shot conversational dense retrieval. In Proceedings of the 44th International ACM SIGIR Conference research and development information retrieval, pages 829-838.
Yifei Yuan, Chen Shi, Runze Wang. Liyi Chen, Feijun Jiang, Yuan You, and Wai Lam. 2022. McQueen: benchmark for multimodal conversational query rewrite. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4834-4844, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ri-Rong Wen. 2023. Large language models for information retrieval: survey, arXiv preprint arXiv:238..07107.

---

A Additional Dataset Details
gradient accumulation steps of 2 (i.e., an overall batch size of 32). We employ AdamW (Loshchilov and Hutter, 2017) as the optimizer and creat a linear schedule with warmup to adjust the learning rate dynamically. The peak learning rate is set to le-5, and the warmup ratio is 0.1. The maximum conver- sational context length is restricted to 384, and the maximum output length is set to 64. We use a fixed random seed of 42 and conduct experiments on single TITAN RTX GPU card with 24GB memory. Greedy decoding is used for inference.
The original QReCC dataset is only divided into training set and test set. Following Kim and Kim (2022), we sample 2K conversations from the train- ing set the development set. The statistics are summarized in Table 4. To ensure that the first user question is always self-contained and unambigu- ous, we replace all first questions with their cor- responding human rewrites. This pre-processing aligns with previous work (Anantha et al., 2021; Wu et al., 2022). We note that some questions in the test set lack valid gold passage labels. As these questions are irrelevant for retrieval evaluation, we remove them from the test set. Consequently, our final test set consists of 8,209 questions, with 6,396 for QuAC-Conv, 1,442 for NQ-Conv, and 371 for TREC-Conv. This filtering also helps reduce costs associated with using OpenAl APIs.
To conduct distillation, we first sample 10K ques- tions from the training set and 2K questions from the development set. We then apply RW(FSL) and ED(Self) to generate rewrites as pscudo labels for training the T5QR model. In this experiment, the training settings remain the same as the previous one, except that the number of gradient accumula- tion steps set (i.e., an overall batch size of 16) and the BLEU score is calculated based on the generated pseudo labels rather than human rewrites. We perform testing on the full test set.
Train
Dev
Test
#C
8,823
2,000
2,775
QReCC
#Q
51,928
11,573
16,451
We use Pyserini (Lin et al., 2021a) to construct the sparse index for the BM25 retrieval model, with default hyparameters of k1 0.82 and b 0.68. These values were chosen according to the retrieval performance on MS MARCO (Bajaj et al., 2016), a non-conversational retrieval dataset. We adopt Faiss (Johnson et al., 2019) to build the dense in- dex for the GTR retrieval model. When encoding queries and passages, the maximum length is set to 384 and the dimension of embedding vectors is 768. We utilize cosine similarity between query vector and passage vector to estimate their rele- vance. Building the dense index for 54M passages requires around 320GB of RAM. To efficiently han- dle this, we split the passage collection into 8 parts, conduct retrieval on each part, and subsequently merge the results.
#C 6,008
1,300
1,816
QuAC-Conv
#Q
41,395
8,965
12,389
#C 2,815
700
879
NQ-Cony
#Q
10,533
2,608
3,314
#C
0
80
TREC-Conv
#Q
748
Table 4: Statistics of the QReCC dataset and its three subsets. #C represents the number of conversations and #Q denotes the number questions.
B Additional Implementation Details
Throughout our experiments, we leverage ChatGPT the LLM and set the maximum number of gen- eration tokens to 2,560 for all four variants of our proposed approach, namely RW(ZSL), RW(FLL), ED(Self), and ED(T5QR). ED(Self) utilizes the rewrites generated by RW(FSL) as initial rewrites, while ED(T5QR) takes the rewrites produced by T5QR as initial results. For training, we initialize T5QR with the 15-base checkpoint from Hugging- F and select the best model based on the high- est BLEU score (Papineni et al.,, 2002) with human rewrites the development set. The training pro- cess involves 10 epochs with a batch size of 16 and
Additional Experimental Results
C.1 Pairwise Comparison of Query Rewriting Methods
We provide a more in-depth analysis of different query rewriting methods by comparing the per- formance of every two methods on each instance within the QReCC test set. We leverage recipro- cal rank (RR) as the evaluation metric and mea- sure the ratio of instances where the first method outperforms (win) or achieves equal performance (tie) to the second method. The results are illus-
'huggingface ctttae//ccccc////eeeeeeeesssseee htt.s//rfoorommmmmmmeedddo/////

---

Human
0.0
0.299
0.333
0.276
0.266
0.258
0.40
Human
1.0
0.576
0.37
0.395
0.374
0
409
0.9
0.35
TSQR
0.126
0.21
TSQR
0.576
1.0
0.
0.376
0.391
0.367 0 0.452
RW(ZSL)
0.298
0.363
0.178
0.177
0.191
0.25
RW(ZSL)
0.37
0.376
0.562
0.53 0.51
0.7
0.20
RW(FSL)
0.328
0.396
0.26
0.0
0.114
0.196
RW(FSL)
0.732
0.575
0.15
0.6
ED(Seff) 0.36 0.424
0.292
0.154
0.0
0.213
0.10
ED(Self)
0.374 3.37
0.732
0.583
0.5
EDITSQR)
0.334 0.384 0.299 0.229
0.204
0.0
EDITSQR
0.409 0.452 0.51 0.575
0.583
1.0
0.4
0.00
Human
TSOR
RWIZSLJ
RW(FSL)
EDISeff)
EDITSQR)
Human
T5QR
RW(ZSL]
RW(FSL)
ED(SeIn)
EDITSOR)
Tie
Figure 4: The ratio of cases where query rewriting methods shown in the vertical axis achieve better performance (win) than equal performance (tie) to methods shown in the horizontal axis in terms of the reciprocal rank (RR)
metric on the overall QReCC test set. We adopt BM25 as the retriever in this study.
QReCC
QuAC-Com
NQ-Cony
TREC-Cony
Query
NDCG@3
R@S
R@100
NDCG@3
R@5
R@100
NDCG@3
R@5
RØ 100
NDCG@3
R@s
R@100
Original
7.98
12.06
28.63
8.02
12.10
27.70
7.80
11.14
29.06
7.96
14.82
42.99
Human
35.97
50.71
98.48
36.57
51.21
98.35
36.58
51.65
98.96
23.15
38.54
98.92
Sparse (BM25)
TSQR
30.13
43.09
85.66
30.55
43.57
85,86
30.47
42,86
84.42
21.39
35.58
87.06
ConQRR
88.90 90
90.20
86,70
75.90
ConVGQR
41.00
88.00
RW(ZSL)
39.67
52.00 (M
84.31
42.54
55,02
85.55
33.05
45,30
81.92
15.89
26.82
72.15
RW(FSL)
56.60
46.83
59.59
89.86
37.61
50.85
86.56
16.00
27.36
ED(Self)
46.43
58.86
88.15
50.20
62.55
37.94
50.89
85.72
26.28
66.49
EDXTSQR)
44.85
57.69
89.91
47.80
60.49
91.09
38.77
51.88
87.97
17.71
32.08
77.18
Original
10.90
15.23
27.22
10.09
14.38
25. 64
12.00
15.97
29.26
20.51
27.09
46.50
Human
39.42
54.85
86.86
36.68
52.44
86.46
50.89
65.24
88.54
42.01
56.06
87.24
Dense (GTR)
T5QR
34.37
48.19
79.85
32.06
46.09
79.65
43.98
56.94
79.92
36.79
50.40
83.20
ConQRR
80.90
79.60
ConVGQR
39.10
RW(ZSL)
37.41
52.23
81.03
36.83
51.79
81.53
41.59
55.96
79.89
31.21
45.28
76.95
(W(FSL)
40.62
56.12
84.76
40.17
55.85
85.33
45.41
60.19
84.67
29.87
44.83
75.34
EDXSelf)
41.80
57.11
85.61
42.06
57,50
86.42
44.08
$9.27
84.42
28.44
41.87
76.15
EDXTSQR) 41.49
56.46
85.91
41.02
56.11
86.25
46.39
60.27
85,35
30.56
47.71
82.12
Table 5: Passage retrieval performance of sparse and dense retrievers with various query rewriting methods on the QReCC set and its three subsets. The best results are shown in bold and the second-bes best results are underlined.
trated Figure 4. It can be seen that our proposed approaches, RW(FSL), ED(Self), and ED(T5QR), win more instances than human rewrites. For ex- ample, ED(Self) achieves higher performance in approximately 36% of instances, whereas human rewrites are better in only about 26.6% of instances. It can also be seen that T5QR outperforms human rewrites only 12.6% of instances, again confirm- ing that learning solely from human rewrites is not ideal. Moreover, we observe that ED(T5QR) out- performs T5QR in 38.4 38.4% of instances, while T5QR only wins in 16.3% of instances. This demonstrates that prompting LLMs rewrite editors is effec-
tive in generating higher-quality query rewrites. Among the four variants of our proposed approach, we find that they share relatively high tie ratio. For example, ED(Self) and RW(FSL) achieve equal performance in 73.2% of instances. This is reason- able since ED(Self) takes the rewrites produced by RW(FLL) as initial rewrites.
This analysis reveals that single method pos- sesses superior efficacy over all other methods, thereby indicating considerable potential for fu- ture research. crucial to emphasize that while we assert the limited informativeness of human rewrites in a general sense, it is possible that some

---

Human TSQR
RW(ZSL) RW(FSL)
ED(Seff) ED(TSQR)
Human TSQR
RW(ZSL) RW(FSL)
ED(Seffi
ED(TSQR)
100
100
90
90
80
80
70
70
60
60
50
50
40
40
R@5
R@10
R@20
R@30 R@100
R@5
R@10
R@20
R@30 R@100
(a) QRcCC
(b) QuAC-Conv
Human
RW(ZSL)
ED(Self)
Human T5QR
RW(ZSL)
ED(Self) ED(TSQR)
T5QR
RW(FSL)
ED(TSQR]
RW(FSL)
100
100
90
80
80
70
60
60
40
50
40
20
R@5
R@10
R@20
R@30 R@100
R@5
R@10
R@20
R@30 R@100
(c) NQ-Conv
(d) TREC-Comv
Figure 5: Recall value versus the cutoff rank E (5, 10, 20,30, 30, 100), with BM25 as the retriever.
human rewrites are already informative enough.
are typically considered. Regarding @@ 100, we find that human rewrites consistently demonstrate the best performance. This is rational since human rewrites possess higher guarantee of correctness. Although our proposed approaches are able gen- erate more informative query rewrites, they suffer from a lower guarantee of correctness as they are built upon LLMs. However, it is worth noting that our approach ED(TSQR) achieves the second- results for both sparse and dense retrieval. Concern- ing the performance the three subsets, our find- ings are similar to those presented in Section 5.1. Our approaches consistently achieve the best second- best results in terms of NDCG@3 and R @5 on the QuAC-Conv and NQ-Conv subsets. How- ever, they fall short of human rewrites and T5QR on the TREC-Cony subset. Nevertheless, our ap- proach ED(TSQR) achieves higher performance than ConQRR, even though it employs reinforce- ment learning to optimize the model with retrieval performance the reward.
C.2
Main Results with Different Evaluation Metrics
Here, we expand the range of evaluation metrics to assess the retrieval performance more compre- hensively. Specifically, we utilize normalized dis- counted cumulative gain with cutoff rank of (NDCG@3) and Recall @ (R @5) evaluate the relevance of top-ranked passages. Additionally, we employ Recall 100 (R 100) to consider the rel- evance of lower-ranked passages. The results are presented in Table We find that our proposed approaches can substantially outperform human rewrites and all supervised baselines in terms of NDCG@3 3 and R @ on the QReCC test set. For example, ED(Self) shows absolute improvement of 10.46 in NDCG@3 compared human rewrites, This finding indicates that our approaches effec- tively rank relevant passages higher, which is partic- ularly valuable for downstream tasks such as ques- tion answering, where only the top-ranked passages
We conduct further analysis on the trend of re-

---

Human T5QR
RW(ZSL) RW(FSL)
ED(Seff) ED(TSQR)
Human TSQR
RW(ZSL) RW(FSL)
ED(Setf) ED(TSQR)
90
90
80
80
70
70
60
60
50
R@5
R@10
R@20
R@30 R@100
R@5
R@10
R@20
R@30 R@100
(a) QRcCC
(b) QuAC-Conv
Human
RW(ZSL)
ED(Seif)
Human TSQR
RW(L)))
ED(Sett)
T5QR
RW(FSL)
ED(TSQR)
RW(FSL)
ED(TSQR)
90
90
80
80
70
70
60
60
50
50
R@5
R@10
R@20
R@30 R@100
R@5
R@10
R@20
R@30
R@100
(c) NQ-Conv
(d) TREC-Conv
Figure Recall value versus the cutoff rank E {5, 10, 20, 30, 100), with GTR as the retriever.
call values by varying the cutoff rank k within the range of (5, 10, 20, 30, 100}. Recall, which repre- sents the fraction of relevant passages retrieved, is important metric for evaluating the performance retrieval results, particularly from a holistic point of view. The results of sparse retrieval and dense retrieval are illustrated in Figure and Figure 6, respectively. As anticipated, increasing the value of k leads to higher recall values across all methods, larger number of relevant passages are likely to be included in the results. Notably, proposed approaches RW(FSL), ED(Self), and ED(T5QR) demonstrate commendable performance differ- ent cutoff ranks (i.e., different values k). They can surpass human rewrites when considering small values of k. However, they are inferior to human rewrites when & becomes large. This observation suggests that the quality of rewrites generated by our approaches exhibits higher variability. Even though many rewrites outperform human rewrites, there are also cases where the generated rewrites are significantly inferior. It is worthwhile con-
Query
MRR
MAP
R@D 10
RW(ZSL)
42.63
41.31
60.46
Informativeness
41.52
40.20
59.60
Sparse (BM25)
-orrectness
35.95
34.61
53,59
-Clarity
33.43
32.16
50.87
-Nonredundancy
36.58
35.23
55.02
RW(ZSL)
40.64
38.95
62.28
-Informativeness
39.94
38.28
61.44
Dense (GTR)
-Correctness
38.03
36.28
57.88
-Clarity
35.78
34.11
54.82
-Nonredundancy
38.53
36.82
59.27
Table 6: Ablation study by removing each of the four desirable properties from the instruction in RW(ZSL).
duet further research to optimize the worst-case performance of our proposed approach.
C.3 Additional Ablation Study
In Table 6, we report the ablation results by remov- ing each of the four desirable properties from the instruction RWZZSL). From the table, we can see

---

70
70
Numan
Human
Humen
Human
60
RWIF5D
60-
RIV(FSL)
RWIFSU
50-
50-
EDOSel))
55-
50-
30
30
MAR
MAP
R@100
MRR
MAP
Rg
MAR
MAP
Rig0
MAR
MAP
Rg10
(a) QRcCC
(b) QuAC-Conv
(c) NQ-Conv
(d) TREC-Conv
Figure 7: Distillation results of using GTR as the retriever. The legends indicate the sourees of fine-tuning labels.
that all the four properties contribute the perfor- mance improvement. Removing any one of them will lead performance degradation. particular, we can observe that removing either the correctness or clarity property leads to the most performance drop. This is because these two properties are cru- cial to ensure that the rewrites preserve the mean- ing of the original queries and are self-contained. Since the clarity requirement also contributes to the informativeness of rewritten queries, remov- ing the informativeness requirement only seems to decrease the performance not that much.
T5QR
RW(FSL.
Rewriting Latency
312 (ms/q)
1867 (ms/q)
Table Comparison of rewriting latency in terms of milliseconds per query (ms/q). RW(FSL) is the teacher model, while TSQR the student model.
as the query rewriter and BM25 as the retriever in this study. Table showcases two successful exam- ples. In the first example, adding the information "during his time Chancellor of the University of Chicago" crucial to understanding the original query comprehensively, thereby significantly im- proving the ranking of the relevant passage. In the second example, the added information exhibits a high degree of overlap with the gold passage, which also leads to better retrieval performance. Table 9 showcases one unsuccessful example. In this exam- ple, approach RW(FSL) misinterprets the user query and expands it using wrong contextual infor- mation, resulting in worse retrieval performance.
In summary, this ablation study verifies that our proposed four properties in the instructions are es- sential to the success of prompting LLMs as query rewriters and informative query rewriting is critical for achieving better retrieval performance.
C.4 Additional Distillation Results
The distillation results using GTR as the retrieval model are depicted in Figure 7. It can be observed that distillation outperforms using human rewrites labels by large margin the QReCC test set. On the QuAC-Conv subset, distillation also consis- tently demonstrates superior performance. NQ-Conv subset, distillation surpasses the utiliza- tion of human rewrites as labels when RW(FSL) is employed as the teacher model.
D Prompts
the
The prompts for utilizing LLMs as zero-shot and few-shot query rewriters and few-shot rewrite ed- itors presented in Table 10, Table 11, and Ta- ble 12, respectively.
Distillation offers beneñits not only in enhanc- ing retrieval performance but also in reducing time overhead. The rewriting latency comparison be- tween T5QR and RW(FSL), with RW(FSL) serv- ing as the teacher model, presented in Table The results indicate that the student model T5QR exhibits a significantly higher rewriting speed, ap- proximately six times faster than RW(FSL).
C.5
Case Study
We perform a case study to help understand the im- pact of informative query rewrites on retrieval per- formance more intuitively. We employ RW(FSL)

---

Conversational Context: (id=511_4
Q1: What year was Robert Maynard Hutchins Chancellor the University of Chicago?
A1: Robert Maynard Hutchins served as University of Chicago's Chancellor from 1945 until 1951.
Q2: Did Robert pull any sports out of the schools agenda?
A2: Robert Maynard Hutchins eliminated the University of Chicago's football program, which he
saw as a campus distraction.
Q3: What collegiate conference of sports did he pull the university out of?
Ag: Robert Maynard Hutchins pulled the University of Chicago out of the Big Ten Conference.
Current Question:
Q4: What degree did he make known for two year studies?
Human Rewrite:
Q7: What degree did Robert Maynard Hutchins make known for two year studies? (Rank=56)
Rewrite by RW(FSL):
Q4: What degree program did Robert Maynard Hutchins make known for two year studies during his time as Chancellor of the University of Chicago? (Rank=2)
Gold passage:
Hutchins was able to implement his ideas regarding a two-year, generalist bachelors during his tenure Chicago, and subsequently had designated those studying in depth in field as masters students...
Conversational Context: (id=1875_3
Q1: Where did Wu-Tang Clan's 's name come from?
A₁: Shaolin and Wu Tang is a film that inspired the name of the hip-hop group Wu-Tang Clan.
Q2: When did the group form?
A2: Wu-Tang Clan an American hip hop group formed in the New York City borough of Staten Island in 1992.
Current Question:
Q3: Who were the founding members?
Human Rewrite:
Qj. Who were the founding members of Wu-Tang Clan? (Rank=14)
Rewrite by RW(FSL)):
Ø3: Who were the founding members of the American hip hop group Wu-Tang Clan, which was formed the New York City borough of Staten Island in 1992 and named after the film Shaolin and Wu Tang? (Rank=1)
Gold passage:
Wu-Tang Clan is an American hip hop group formed in the New York City borough of Staten Island in 1992, originally composed of rapper-producer RZA and rappers GZA, OlDirty Bastard, Method Man, Raekwon, Ghostface Killah, Inspectah Deck, U-God and Masta Killa...
Table 8: Two successful examples showing that our proposed approach RW(FSL) can extract valuable contextual information to improve the informativeness of generated query rewrites. We highlight the additional information in generated rewrites compared to human rewrites in blue and the relevant parts in gold passages regarding the added information in red.

---

Conversational Context: (id=617_5)
Q1: What did the singer, Cheryl do for the greater good?
Aj: In 2004, Cheryl's band, Girls Aloud released
of The Pretenders' I'll Stand by You as the
official single for the BBC's charity telethon Children in Need.
Q2: Did she do any other activities for charities?
A2: In March 2009, Cheryl climbed Mount Kilimanjaro in aid of Comic Relief.
Q3: Does she have any charities?
Ag: In February 2011, 1, Cheryl launched her own charitable foundation with The Prince's Trust
following a mecting with The Trust's President, HRH Charles, Prince of Wales.
Q4: Are there any other interesting aspects about this article?
Aq: On 23 January 2015, Cheryl announced the launch of a second charity, once again alongside The
Prince's & Trust
Current Question:
Q5: For what charity?
Human Rewrite:
Q:: What second charity did the singer, Cheryl launch in 2015? (Rank=1)
Rewrite by RW(FSL):
Q3z: For which charity did Cheryl launch her own charitable foundation in February 2011, following meeting with The Trust's President, HRH Charles, Prince of Wales? (Rank=25)
Gold passage:
up with the aim of raising E2 million to build centre, which will support up to 4000 disadvantaged young people her native city of Newcastle. To raise these funds, Cheryl has thus far teamed up with Prizeo March 2015
The charity named Cherylé Trust, and
Table 9:
An unsuccessful example showing that our proposed approach RW(FSL) may misinterpret the original question and thus fail to identify truly relevant contextual information augment the original query.
Given a question and its context, decontextualize the question by addressing coreference and omission issues. The resulting question should retain its original meaning and be as informative as possible, and should not duplicate any previously asked questions in the context.
Context: [Conversational context)
Question: {Question)
Rewrite:
Table 10: Prompt for utilizing LLMs as zero-shot query rewriters.

---

Given question and its context, decontextualize the question by addressing coreference and omission issues. The resulting question should retain its original meaning and be as informative as possible, and should not duplicate any previously asked questions in the context.
Context: [Q: When was Born to Fly released?
A: Sara Evans's third studio album, Born to Fly, was released on October 10, 2000.
Question: Was Born to Fly well received by critics?
Rewrite: Was Born Fly well received by critics?
Context: [Q: When was Keith Carradine born?
A: Keith Ian Carradine was born August 8, 1949.
Q: Is he married?
A: Keith Carradine married Sandra will on February 6, 1982. ]
Question: Do they have any children? Rewrite: Do Keith Carradine and Sandra Will have any children?
Context: [Q: Who proposed that atoms are the basic units of matter?
A: John Dalton proposed that each chemical element is composed of atoms of single, unique type, and they can combine to form more complex structures called chemical compounds. ]
Question: How did the proposal come about?
Rewrite: How did John Dalton's proposal that each chemical element is composed of atoms of single unique type, and they can combine to form more complex structures called chemical compounds come about?
Context: [Q: What is it called when two liquids separate?
A: Decantation is process for the separation of mixtures of immiscible liquids
or of a liquid and a solid mixture such as a suspension. Q: How does the separation occur?
A: The layer closer to the top of the container-the less dense of the two liquids, the liquid from which the precipitate or sediment has settled out-is poured off. ]
Question: Then what happens?
Rewrite: Then what happens after the layer closer to the top of the container is poured off with decantation?
Context: [Conversational context]
Question: (Question)
Rewrite:
Table 11: Prompt for utilizing LLMs as few-shot query rewriters.

---

Given question and its context and rewrite that decontextualizes the question, edit the rewrite to create a revised version that fully addresses coreferences and omissions in the question without changing the original meaning of the question but providing more information. The new rewrite should not duplicate any previously asked questions in the context. If there is need to edit the rewrite, return the rewrite as-is.
Context: [Q:
When was Born to Fly released?
A: Sara Evans's third studio album, Born to Fly, was released on October 10, 2000.
Question: Was Born to Fly well received by critics?
Rewrite: Was Born to Fly well received by critics?
Edit: Was Born to Fly well received by critics?
Context: [Q: When was Keith Carradine born?
A: Keith Ian Carradine was born August 8, 1949.
Q: Is he married?
A: Keith Carradine married Sandra Will on February 6, 1982. ]
Question: Do they have children?
Rewrite: Does Keith Carradine have any children?
Edit: Do Keith Carradine and Sandra Will have any children?
Context: [Q: Who proposed that atoms are the basic units of matter?
A: John Dalton proposed that each chemical element is composed of atoms of single, unique type, and they can combine to form more complex structures called chemical compounds. ]
Question: How did the proposal come about?
Rewrite: How did John Dalton's proposal come about?
Edit: How did John Dalton's proposal that each chemical element is composed of atoms of single unique type, and they can combine to form more complex structures called chemical compounds come about?
Context: [Q: What is it called when two liquids separate?
A: Decantation is process for the separation of mixtures of immiscible liquids
or of a liquid and a solid mixture such as a suspension. Q: How does the separation occur?
A: The layer closer to the top of the container-the less dense of the two liquids, the liquid from which the precipitate or sediment has settled out-is poured off. ]
Question: Then what happens?
Rewrite: Then what happens after the layer closer to the top of the container is poured off?
Edit: Then what happens after the layer closer to the top of the container is poured off with decantation?
Context: [Conversational context)
Question: [Question]
Rewrite: (Initial rewrite)
Edit:
Table 12: Prompt for utilizing LLMs as few-shot editors.


Reviews:
[{"review_id": "RX9f0CBShj", "reviewer": "Reviewer_H9aq", "summary": "", "questions": "Line 413 mentions that “We disregard test instances without valid gold passage labels”. Do you mean using the initial contextualized query to obtain the candidates ? And is this step also applied in other related literature ?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. Please find our responses below.\n \n**Q1:** The novelty of the method is limited, considering Yu et al. SIGIR 2020, where GPT2 is used as the backbone.\n> Thanks for raising this concern. We acknowledge that Yu et al. (SIGIR 2020) is an excellent work on conversational query rewriting. However, its focus is on leveraging ad hoc (non-conversational) search sessions to construct weak supervision data so as to mimic conversational search sessions. While this is a practical way to tackle the issue that human rewrites are expensive to collect, it cannot well address the problem of uninformative query rewriting, especially given that ad hoc search queries are usually short and contain only a few keywords. Besides, real conversational search sessions can be much more complicated than synthetic ones, which also challenges the effectiveness of models trained on these synthetic data.\n> \n>  &nbsp;\n> \n> In contrast, our work focuses on addressing the issue that human rewrites often lack sufficient information to achieve optimal retrieval performance, which is fundamentally different from Yu et al. (SIGIR 2020). Moreover, our contributions are manifold:\n> + We are the first to introduce the concept of informative conversational query rewriting.\n> + We have designed clear and explicit instructions to effectively prompt LLMs as query rewriters and rewrite editors to generate informative query rewrites, which not only solves the problem of high cost associated with human rewriting but also solves the problem that human rewrites are often uninformative enough.\n> + We have presented a comprehensive set of experimental analyses. Impressively, our results show that the generated informative query rewrites can achieve better retrieval performance than human rewrites.\n>\n> We believe these contributions are sufficient to warrant the novelty of our work. We will also release the generated rewrites for future research. Given the limitations of using human rewrites as ground-truth labels, we argue that it does not make much sense to design and train new models to mimic human rewrites. We hope our work will provide valuable insights to the community.\n\n**Q2:** In Section 3.1, four dimensions are defined, while in the ablation study, only informativeness is considered.\n> Thanks for pointing out this issue. We investigate only informativeness in the ablation study for two reasons. Firstly, our main focus in this work is on the informativeness of rewritten queries. The other three dimensions are introduced to ensure that the generated rewritten queries are as accurate as possible (i.e., retain their original meanings and intents). Our motivation is that only when the rewrites remain correct can it make sense to study the impacts of informativeness. Secondly, since we take ChatGPT as the LLM in this work, it is costly to explore the impacts of each dimension. Nonetheless, we have quantitatively analyzed the correctness and clarity of generated rewritten queries in Section 5.2 and Table 2.\n\n**Q3:** Only GPT3.5 is studied and the generalization ability is unknown, considering datasets used in the paper could be a part of the pretraining data of the LLMs.\n> Thanks for this suggestive comment. We agree that it could be a limitation to experiment with only one LLM, i.e., ChatGPT. We have mentioned this point in the Limitation section. Our primary concern is that different LLMs have different capabilities of following instructions to perform rewriting. Other (smaller) LLMs may not achieve the same level of performance as ChatGPT. In other words, the generalization ability is related to the basic capacities of LLMs. However, we do not think that data contamination could affect the generalization ability significantly. It is worth emphasizing that our goal is to prompt LLMs to generate rewrites that are more informative than human rewrites. These informative rewrites are not included in the original dataset. Therefore, even if data contamination happens, LLMs would still struggle to generate informative query rewrites without our proposed instruction and the adopted demonstrations. In fact, we found in our initial trials that even ChatGPT showed inferior performance to human rewrites when we simply set the instruction to *''rewrite the question into a self-contained and contextually independent one''*. This further verifies that the performance improvement is not caused by data contamination.\n\n**Q4:** Line 413 mentions that “We disregard test instances without valid gold passage labels”. Do you mean using the initial contextualized query to obtain the candidates? And is this step also applied in other related literature?\n> Thanks for the question. In our experiments, we removed instances that do not have valid gold passage labels from the test test. This is because when no gold passage labels are provided, it is challenging to evaluate whether the retrieved passages are truly relevant or not. The gold passage labels are provided in the dataset. We do not obtain the labels by ourselves. This preprocessing strategy has also been adopted in ConQRR (refer to footnote 8) [1].\n> \n>  &nbsp;\n> \n> [1] Wu, Zeqiu, et al. \"ConQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning.\" EMNLP 2022, https://aclanthology.org/2022.emnlp-main.679.pdf.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The capability of LLMs for zero/few shot rewriting is studied.\n- The authors show that it is possible to distill small models from LLMs to achieve better performance than using the ground-truth labels on certain tasks.\n- The paper is well-written and easy to follow.\n", "reasons_to_reject": "- The novelty of the method is limited, considering Yu et al. SIGIR 2020, where GPT2 is used as the backbone.\n- In Section 3.1, four dimensions are defined, while in the ablation study, only informativeness is considered.\n- Only GPT3.5 is studied and the generalization ability is unknown, considering datasets used in the paper could be a part of the pretraining data of the LLMs.\n", "questions_to_authors": "Line 413 mentions that “We disregard test instances without valid gold passage labels”. Do you mean using the initial contextualized query to obtain the candidates ? And is this step also applied in other related literature ?", "paper_topic_and_main_contributions": "The paper proposes to use large language models for the task of query rewriting. Four dimensions about reformulations are defined to measure the quality of rewrites. Zero-shot setting and few-shot setting with GPT3.5 are evaluated. Beyond that, a “rewrite-then-edit” process is also evaluated where an initial query is also included during the rewiring process. The authors also evaluate the performance of distilled models using the output from LLMs. The experimental results show that their proposed models can outperform human performance on the QReCC dataset (and on most subsets). Quantitative analysis shows that their proposed methods are more informative and correct.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Major related papers are cited."}, "extra_text": {}}, {"review_id": "kZSxHgzRa4", "reviewer": "Reviewer_uAia", "summary": "", "questions": "see above", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. Please find our responses below.\n \n**Q1:** Further exploration is needed to determine the effect of the informativeness requirement in the instruction (i.e., the resulting question should be as informative as possible) on the generated query. For example, what is the difference between different informative requirements, such as \"the resulting question should be as informative as it can be\" or \"the resulting question should be as helpful as possible\"? Ablation studies show that even removing the informativeness requirement does not significantly decrease performance, which suggests that the effect of informativeness may not be useful. Additionally, although comparing whether including informativeness requirements in RW(ZSL) can show whether the informative requirement is effective, the results of ED(Self) should also be included since it is the main proposed method in the paper.\n> Thanks for the constructive suggestion. We agree that it can be useful to study the difference between different informative requirements. However, it is extremely challenging to measure the informativeness of a rewritten query and determine how much information a rewritten query should contain to achieve the best retrieval performance. The optimal amount of information in a rewritten query should be highly dependent on the query itself and the passages in the collection. For simplicity, we focus in this work on the requirement that the rewritten query should be as informative as it can be. In the Limitation section, we have pointed out that the most informative query rewrite may not be the best one for relevant passage retrieval, especially when passages in the collection are short. Informative query rewriting should prove more beneficial in the context of long passage or document retrieval. We agree that *''the rewritten query should be as helpful as possible''* tends to be a better informative requirement from the perspective of retrieval performance. However, it is challenging to ask an LLM to decide the helpfulness of a rewrite automatically, since the helpfulness can only be assessed after the retrieval process is finished. By comparison, it is much easier to instruct an LLM to generate as informative query rewrites as possible based on the conversational context. In addition, our experimental analyses showed that informative query rewriting could already lead to higher retrieval performance, even outperforming human rewrites.\n>  \n> &nbsp;\n> \n>  Our ablation study showed that removing the informativeness requirement resulted in consistent performance degradation for both sparse and dense retrieval. We have also presented a case study in Appendix C.4, which shows more intuitively how an informative query rewrite can lead to higher retrieval performance. All these analyses validate the importance of the informativeness requirement. However, since we have considered four desirable properties of a well-crafted rewritten query in the instruction, it is reasonable that removing the informativeness requirement does not cause a drastic performance decrease. For example, the clarity property requires that the rewritten query should resolve all coreference and omission issues, which will also contribute to the informativeness of rewritten queries. If we remove the clarity requirement as well, the performance will drop further. \n> \n> &nbsp;\n> \n>  We conducted the ablation study using RW(ZSL) as the method because RW(ZSL) performs rewriting in the zero-shot setting, thus the performance will not be affected by the demonstrations used. It is worth mentioning that we only prompted LLMs as rewrite editors (for both ED(Self) and ED(T5QR)) in the few-shot setting with four demonstrations utilized. These demonstrations will inevitably interfere with the ablation study.\n\n**Q2:** In parts where the subdatasets NQ-Conv and TREC-Conv are used, the claim of emphasizing LLM rewrites over human rewrites seems to be undermined given the lower performances of all the rewrites that are not generated by humans (including LLMs and other baseline models). While the author explains that this is due to the obscure and self-contained characteristics of queries (particularly in TREC-Conv), the research does not continue to address LLMs’ potential applicability for such queries or point out the future direction, which makes the original claim appear only effective under certain conditions.\n> Thanks for this insightful comment. We have pointed out in our paper that it is crucial to take query characteristics into account when performing query rewriting with LLMs (#487-#490). Additionally, there can be many future directions. For example, we can train an auxiliary model to decide whether queries generated by LLMs should be preferred to those generated by models that have been trained on human rewrites. We can also combine human rewrites and LLM rewrites as labels through a proper weighting strategy to finetune query rewriting models. Furthermore, in this work, we have used a fixed set of demonstrations for all test queries. To achieve the best performance, it is essential to find appropriate demonstrations for each specific query. This would be an effective solution to tackle obscure or complicated queries. Another future direction can be parameter-efficient fine-tuning (e.g., LoRA [1]) of LLMs with retrieval performance as feedback. In this way, we will aim to optimize the helpfulness of rewritten queries rather than informativeness. We can add these potential future directions in the camera-ready version. Considering that each future direction requires dedicated research, we cannot address them in a single paper. However, given the excellent performance we have achieved and the thorough analyses we have performed, we believe we have made adequate contributions in this work. We also appreciate your comprehensive summary of our contributions in the Reasons to Accept section. \n>   \n> &nbsp;\n> \n> [1] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685_ (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Comprehensive structure with comparisons against multiple baselines, including original queries, human rewrites, and query rewrites from other models.\n- Introduction of the \"rewrite editor\" idea, leading to consistent and significant enhancement of retrieval performance on a specific dataset.\n- Pioneering study defining four essential properties for instructing LLMs, contributing to the advancement of the field.\n- Exploration of the applicability of using LLM's query rewrites to train smaller models through distillation.\n- Design of experiments to measure the informativeness and correctness of query rewrites, providing valuable insights into the proposed approach's effectiveness.", "reasons_to_reject": "- Further exploration is needed to determine the effect of the informativeness requirement in the instruction (i.e., the resulting question should be as informative as possible) on the generated query. For example, what is the difference between different informative requirements, such as \"the resulting question should be as informative as it can be\" or \"the resulting question should be as helpful as possible\"? Ablation studies show that even removing the informativeness requirement does not significantly decrease performance, which suggests that the effect of informativeness may not be useful. Additionally, although comparing whether including informativeness requirements in RW(ZSL) can show whether the informative requirement is effective, the results of ED(Self) should also be included since it is the main proposed method in the paper.\n- In parts where the datasets NQ-Conv and TREC-Conv are used, the claim of emphasizing LLM rewrites over hunan rewrites seems to be undermined given the lower performances of all the rewrites that are not generated by humans (including LLMs and other baseline models). While the author explains that this is due to the obscure and self-contained characteristics of queries (particularly in TREC-Conv), the research does not continue to address LLMs’ potential applicability for such queries or point out future direction, which makes the original claim appear only effective under certain conditions. \n", "questions_to_authors": "see above", "paper_topic_and_main_contributions": "This paper proposes using LLM to generate and edit query rewrites, addressing the limitation of human rewrites that lack contextual information for optimal retrieval in conversational search. Human rewriters focus on ambiguity but overlook contextual details from previous turns. Leveraging LLM's instruction-following abilities, the research defines desirable properties for query rewrites. However, LLMs may struggle with complex tasks, prompting the introduction of a \"rewrite-then-edit\" process, with initial rewrites generated by smaller models. To reduce memory and computational demands, a distillation method is suggested, fine-tuning smaller models with LLM's generated rewrites as training labels.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none", "Typos_Grammar_Style_and_Presentation_Improvements": "none"}, "extra_text": {}}, {"review_id": "guEnAkyViq", "reviewer": "Reviewer_ve5t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. Please find our responses below.\n \n **Q1:** The novelty of the paper is fairly limited to simply using LLMs to generate silver data, which are then used to train smaller LMs.\n > Thanks for raising this concern. We would like to clarify that **our main focus in this work is on generating informative query rewrites to address the limitation that human rewrites often lack enough informativeness**. To achieve this goal, we have designed clear and explicit instructions to prompt LLMs as query rewriters and rewrite editors. In addition, we have tested LLMs in both zero-shot and few-shot settings to generate informative query rewrites and observed that the generated query rewrites could achieve much higher performance than previous methods, even surpassing human rewrites. Considering that latency might be a concern in some application scenarios, we also investigated distilling the strong rewriting capabilities of LLMs into smaller models by treating the rewrites from LLMs as pseudo labels. It is worth emphasizing that **using LLMs to generate silver data to train smaller LMs is just one contribution among the manifold contributions we have made in this work**. We hope the reviewer could agree with us on this point. Moreover, we will release the generated rewrites to the community for future research, which would further strengthen our contributions. Our work also opens up many future directions, as shown in the response to Q2 of Reviewer uAia. We believe these strong points are sufficient to warrant the novelty of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well written paper and an important task.\n\n- Thorough evaluation and comparison against different competing baselines.\n\n- Valuable insights on how to use LLMs to edit initial rewrites.", "reasons_to_reject": "- The novelty of the paper is fairly limited to simply using LLMs to generate silver data, which are then used to train smaller LMs.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors in this paper propose an approach for conversation query rewriting. The QR task, differently from other works, focuses also on informativeness of the rewritten queries. \n\nThe proposed approach finds out that allowing the LLMs to rewrite then edits the queries. This strategy seems to perform best in terms of retrieval. Finally, the authors distill knowledge from LLMs into smaller LMs to decrease inference latency.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new approach utilizing LLMs for the generation and refinement of query rewrites, addressing a critical limitation inherent in human-generated rewrites – the absence of contextual information required for optimal retrieval in conversational search. Leveraging the instruction-following capabilities of LLMs, this research delineates a set of highly desirable attributes for query rewrites.
Considering that LLMs can face challenges when confronted with intricate tasks, this paper also introduces a "rewrite-then-edit" process.

Although it is essential to underscore the potential impact of this method on the field of conversational search, some reviewers have expressed concerns about the paper's novelty.
Decision: Accept-Findings