InstructoR: Instructing Unsupervised Conversational Dense Retrieval with Large Language Models
Authors: Zhuoran Jin
Pengfei Cao
Yubo Chen
Kang Liu
Jun Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: Conversational Dense Retrieval
Unsupervised Information Retrieval
Large Language Models
TL;DR: To the best of our knowledge, this is the first attempt to utilize LLMs to improve conversational dense retrieval in an unsupervised manner.

Abstract: Compared to traditional single-turn ad-hoc retrieval, conversational retrieval needs to handle the multi-turn conversation and understand the user’s real query intent. However, most existing methods simply fine-tune the pre-trained ad-hoc retriever on limited supervised data, making it challenging for the retriever to fully grasp the entirety of the conversation. In this paper, we find that large language models (LLMs) can accurately discover the user’s query intent from the complex conversation context and provide the supervised signal to instruct the retriever in an unsupervised manner. Therefore, we propose a novel method termed InstructoR to Instruct unsupervised conversational dense Retrieval with LLMs. We design an unsupervised training framework that employs LLMs to estimate the session-passage relevance score as the soft label to guide the retriever's training. Specially, we devise three instructing strategies from context, query and response perspectives to calculate the relevance score more precisely, including conversational retrieval as conversation generation, question rewrite as latent variable and question response as posterior guide. Experimental results show InstructoR can bring significant improvements across various ad-hoc retrievers, even surpassing the current supervised state-of-the-art method. We also demonstrate the effectiveness of our method under low-resource and zero-shot settings. Our code is publicly available at https://github.com/jinzhuoran/InstructoR/.

INSTRUCTOR: Instructing Unsupervised Conversational Dense Retrieval with Large Language Models
INSTSSCTION
Zhuoran Jin 12 Pengfei Caol:2, Yubo Chen 1.2., Kang Liu Jun Zhao
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 2 The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China {zhuoran.jing pengfei. cao, yubo. chen, kliu, jzhao)enlpr. ia.ac.
Abstract
through multi-turn conversations. However, they still struggle to provide highly accurate and up- to-date responses. To mitigate the problems, one promising solution (Izacard and Grave, 2021) is to employ a retrieve-then -eee-trraennggrtteeeee-- pipeline. As illus- trated in Figure (a)), to answer the user's conver- sational query "Did he like collecting things?" in the conversation, the pipeline first retrieves small of reference passages about "Nicolas-Clade Fabri de Peirese" via conversational retrieval and then generates the response conditioned on these passages along with the ongoing conversation.
Compared to traditional single-turn ad-hoc re- trieval, conversational retrieval needs handle the multi-turn conversation and understand the user's real query intent. However, most existing methods simply fine-tune the pre-trained ad- hoc retriever on limited supervised data, mak- ing challenging for the retriever to fully grasp the entirety of the conversation. In this paper, we find that large language models (LLMs) can accurately discover user's query intent from the complex conversation context and provide the supervised signal to instruct the retriever in an unsupervised manner. Therefore, we pro- pose a novel method termed INSTRUCTOR Instruct unsupervised cOnversational dense Retrieval with LLMs. We design an unsuper- vised training framework that employs LLMs to estimate the session- passage relevance score as the soft label to guide the retriever's training. Specially, devise three instructing strategies from context, query and response perspectives calculate the relevance score more precisely, including conversational retrieval as conversa- tion generation, question rewrite as latent vari- able and question response as posterior guide. Experimental results show INSTRUCTOR can bring significant improvements across various ad-hoc retrievers, even surpassing the current supervised state-of-thed tfet--aat---rt method. We also demonstrate the effectiveness our method under ww-resource and zero-shot settings. Our code is publicly available at G
Conversational retrieval, which focuses on re- trieving relevant passages from a large passage col- lection based on the given conversation session, has gained considerable attention from researchers. Unlike traditional single-turn ad-hoc retrieval, con- versational retrieval needs to deal with the com- plex multi-turn conversation session which con- sists of both the historical conversation context and the user's current query. In contrast to standalone queries, conversational queries may contain more complicated linguistic phenomena, such as omis- sions, coreferences, and ambiguities shown in Fig- ure (a). Besides, topic-switching behaviour is nat- ural in information-ssckingg conversations (Adlakha et al., 2022), introducing a significant amount of noise in the conversation context that is unrelated the user's query. Therefore, conversational retrieval is much more challenging than ad-hoc retricval.
1 Introduction
Existing methods can be mainly categorized into conversational query rewriting and conversational dense retrieval. Conversational query rewriting methods (Yu et al., 2020; Lin et al., 2020) utilize the rewriting models to explicitly reformulate the con- versational queries into de-contextualized rewrites and then perform ad-hoc retrieval. While such a two-stage approach offers strong interpretability, its rewriting stage poses difficulties in direct op- timization for retrieval performance and leads to increased retrieval latency (Mao et al., 2022a).
The development of generative language models (Brown et al., 2020; OpenAI, 2023) has sparked paradigm shift in information-ssekingg transition- ing from using search engines to interacting with conversational assistants (e.g., ChatGPT). Conver- sational assistants offer a more flexible and user- friendly experience by addressing users' queries
httttp:://ithuli ctte ccmm/jinehhuorrn/ITssttttta "Corresponding author.

---

( Canssseeiim Cuntect O:: W bat 3:l region?
Refenmnc Passegss
Toptc: Flll nejon
Nirols Claade Fibd Poinsc: Intelladtoe and colltckn kmn His oimegensierza Mallashe arnns light on the personally of Malhahhe's ooabial are...
Large Larguage
A rejion of intendallrr atnmi hydrogg that loniood
Modal
Calcclizoo
Cc. When was it observed
Tagso Seaxing
Relevarze
Ozzidoo
4g. 1610
Bersevar
Soor
Arai by nilore?
Caspuctinn
Clasde Fulari de Prinse.
Toppi: Peirese
B
Generator
Puttover
Retricrp
Top-I Passagis
In?
Ceraterace
Ru Friechh astatomer, arzgqary and serust
Generato
User's Chrmeer Queryy
Converator Sestioo
Amidiaa''s ssid cillatonn
Conserator:
Paraae Collection
4:t to osllactnng Bhings?
Data
(a) Au esamppe of retrieve Uhen gearcale pipeline cumnerssiin somratoo
(1) Our welbod. INSTRUCTOR.
Figure 1: The illustration of a serrieve-then eeneratt pipeline and our proposed insupervised method.
Recently, conversational dense retrieval methods (Lin et al., 2021; Mao et al., 2023b) have shown bet- ter retrieval effectiveness. Most existing supervised methods leverage the pre-trained ad-hoc retriever to encode the entire conversation session into dense embedding and then fine-tune the retriever on con- versational retrieval data. However, they still face the following two challenges: (1) Fine-tuning the retriever often requires large number of labeled session-passage pairs. In practical applications, an- notating session-passage pairs is much more dif- ficult than collecting conversation data. Hence, there is a need to train conversational dense re- triever in an unsupervised manner, i.e., without using pession-passage pairs; (2) Simply compress- ing the session into one single vector may mix irrelevant context and neglect crucial information, making it hard for the retriever to comprehend the user's query in the complex conversation.
retrieves top- K relevant passages via ad-hoc re- trievers, then generates session- session-passage relevance score with frozen LLMs to rerank the top-K pas- sages, and finally uses relevance scores to guide the iterative training of conversational retrieval.
The core of INSTRUCTOR lies in how utilize LLMs to calculate relevance scores. Since the gen- crative process requires LLMs to focus on every token in the session and passage, we approximate the relevance score based on the LLM's conditional generation log likelihood. To obtain more accu- rate score, we propose three instructing strategies from the perspectives of context, query, and re- sponse within the conversation, outlined below: (I) Conversational Retrieval as Conversation Generation: To avoid LLMs being distracted by lengthy and irrelevant context, we decouple conver- sational dense retrieval into instructional generation subtasks, including conditional context generation and question generation; (2) Question Rewrite as Latent Variable: To resolve the user's ambiguous query, model question rewrites generated by black-box LLMs as latent variables implicitly guide the retriever training, showing superior feetiveness and efficiency compared to the explicit usage of question rewrites for retrieval; (3) Ques- tion Response as Posterior Guide: To capture relevance precisely, we find that question response can provide the relevant signal when the passages related to the session unknown. Therefore, we treat question responses the posterior guide further enhance the retriever. We conduct exten- sive experiments on four datasets and prove all three strategics can bring significant improvements across various ad-hoc retrievers. Moreover, our method surpasses the current supervised state-of- the-art art model without using labeled training data.
In this paper, we propose a novel method termed + INSTRUCTOR to Instruct unsupervised cOnversational dense Retrieval with large language models (LLMs). The main insight of our INSTRUC- TOR is that the powerful LLMs can help us train ad- hoc retrievers for conversational retrieval without supervision. On the one hand, some research (Liu al., 2022: Meng et al., 2022; Dai et al., 2022) has demonstrated that LLMs can generate high-quality annotated data with only few or even without demonstration examples. On the other hand, we find that LLMs have powerful linguistic capabili- ties (Liang et al., 2022) and can accurately grasp the user's intent from the complex and noisy conver- sation context. Based on these findings, we unleash the power of LLMs judge the relevance between sessions and passages without requiring any train- ing. Specifically,, we design three-stage unsuper- vised training framework for INSTRUCTOR shown in Figure ((b). Given a session, INSTRUCTOR first
Our contributions are summarized as follows:
We propose a novel method called INSTRUC-

---

TOR to instruct unsupervised conversational dense retrieval with LLMs. To the best of our knowledge, this is the first attempt to utilize LLMs to empower conversational retrieval. We devise three instructing strategies to calcu- late dession-passage relevance score, includ- ing conversational retrieval as conversation generation, question rewrite latent variable and question response as posterior guide.
2.3 LLM Likelihood as Relevance Score
Given text pair (s,d), we employ generative LLMs to compute the relevance score score | without any training examples, which approximates the probability of retrieving based on Since the generative process requires LLMs to focus on and explain every token in 8 and d, we estimate the relevance score score(d 8) as the generation likelihood of d conditioned on 8:
We demonstrate INSTRUCTOR can bring con- sistent and significant improvements across various ad-hoe retrievers, with an average Re- call 100 improvement of 9.0% and 34.1 % on QReCC and TopiOCQA, even surpassing the current supervised wate-of-the-an method by 3.2% and 9.7%. We also evaluate our method under low-resource and zero-shot settings.
score(d 8) p(d 8) logp(d
(3)
logp (d+ I, 1,
m
where is the task sask-specific generation instruction, and m is the sequence length of We collectively refer to those resource resource-intensive LLMs (>100B) which allow limited access through commercial APIs, as black-box LLMs (e.g., ChatGPT and GPT- 4). Conversely, we refer to those cost-effective LLMs (<10B) that can be called millions of times during training to calculate score (d s), white- box LLMs (e.g., Flan- T5 (Chung al., 2022) and GPT-Neo (Andonian et al., 2021)).
2 Preliminary
2.1 Task Definition
We formulate the conversational retrieval task as finding the relevant passage d from large pas- sage collection 'D based on the given conver- sation session St = {(--144), where 4-1 = {g1, 01, Q2,
.... ....................... 4 denotes the histor- ical conversation context consisting of t 1 question- response pairs (g.a), and qi denotes the user's query of 1-th turn. Considering the length limitation of the retriever's input, we represent the input format of the session 8 as:
Methodology: INSTRUCTOR
In this paper, we consider a more realistic setting which is to train a conversational dense retriever in an unsupervised manner, i.e., without using session- passage pairs. We only have certain number of conversation sessions and large passage collec- tion during the training. Our INSTRUCTOR uses the frozen LLMs' conditional generation log likeli- hood as the supervised signal to guide the training of the retriever without direct supervision. Accord- ing to Sachan et al. (2022b), we adopt an unsu- pervised training framework to distill knowledge from LLM to retriever. Moreover, we devise three instructing strategies from diverse perspectives in the conversation to calculate the relevance score of session- passosge pairs more accurately.
8t (CLS) @ge Dat @a.aa @g [SFP], (1)
where [CLS] and [SEP] are special tokens, @ de- notes concatenation. We demonstrate that this re- concatenation works well in Appendix A. We denote the query rewrite of qe as rt. For simplicity, we omit the subscript the rest of this paper.
2.2 Embedding Similarity as Retrieval Score
We adopt the pre-trained ad-hoc retriever the strong initial conversational retriever, which uses a dual-encoder architecture where sessions and pas- sages are encoded independently. Given a session s and a passage d, we encode them with session en- coder fa and passage encoder fa respectively. The retrieval score sim(s, d) between a session and a passage di is the dot product of the embeddings:
3.1 Unsupervised Training Framework
In the unsupervised training framework of IN- STRUCTOR, adopt three-stage approach instruct the ad-hoc retriever with LLM shown in Figure 2. For fast passage retrieval, we apply the passage encoder fa to the passage collection and build passage index I. We froze fa to avoid rebuilding the index during the training.
sim(s, d) (fa(s), Sa(d))
(2)
Despite the dual-encoder's efficiency, the limited model parameters and embedding dimensions hin- der its ability to comprehend sessions and passages.

---

[,(2)
Knowlnsgee Distilation
Passage 1 Encoder
Where do mas huroons typhoios and oydisoss origiezae?
KL(P
by Heresphere
How interoiian danaliee?
P(x|ss
P(z
nyt Typhoun anven typiton and sape byption.
score(s (s)
Nartieen lieniigheee and climate Aorticiicc North
sim
Large Language Model
by
conditus typoson enginsted
olloatt
Top-X
Passages
its diinee chancleroe coid winkers cnel
Use''s Carrer Quey
Co. Witel land lorms tos
Session Encoder
Presage
In reguzze??
Instructing
Index.
(((s)
Figure 2: The unsupervised training framework of INSTRUCTOR. Red dotted arrows indicate the gradient flow.
Compute Top-K Retrieval Probability, Given conversation session 8, first retrieve the top-K passages Z with the retriever (student) and com- pute the retrieval probability of passage 2 Z:
To calculate the relevance score more pre- cisely, will detail our proposed three instructing strategies from context, qutery and response per- spectives within the conversation.
exp (sim (s, 2) /T) Eiez exp (sim (s, 2') /T) (4)
Conversational Retrieval as Conversation Gen- eration (INTTRUCTORRRCG). We find that we calculate p(2 s) by directly generating 2 condi- tioned on with white-box LLMs, it causes LLMs to be distracted by the lengthy and noisy conversa- tion context and unable focus on the current query q. Therefore, we reformulate conversational retrieval as a task of generating conversation ses- sion s conditioned on given passage z, which can be further decoupled into two instructional genera- tion subtasks based on Bayes' Theorem:
s,
s,
where denotes temperature hyperparameter.
Generate Top-K Relevance Probability. Since there are no supervised labels for sessions and pas- sages, we use the LLM (instructor) to generate the relevance probability of passage conditioned on session as the soft supervised signal:
exp (score (= 8))
(2 8, 2, Evez exp (score (z" s))
(5)
score, (s) Re p(2 s) x log c,
We provide three different strategies to derive the score(z s) formula in Section 3.2.
c(2;
= log p(c,g)
p(z)p(c z)p(g 2,
Distill Knowledge from LLM to Retriever. We
= log (c)p(a|e) & logp(c 22, + logp(a 19, 2, c),
train the retriever by minimizing the KL divergence
between the relevance probability P (= 8, 2) and the retrieval probability (z s, 2):
where log p(2), logp(c) and logp(q | e) are as- sumed as constants and can ignored. As shown in Figure 3(a), we first prompt LLMSs with an in- struction I "Please create a conversation." to generate the conversation context e condition on the retrieved passage then prompt LLMs with another instruction 1 "Please continue write question:" generate the current question q condi- tion on the passage z and the ongoing conversation context Hence, we can combine log p(c rg 2) and logp(q 12,c, 2, c) together as the estimate relevance score score, between s and
P (2 2)
C C= P 2, log P (z s, s,
(6)
2E2
We only update the parameters of session encoder during the training.
3.2 Instructing Strategies
this section, measure the relevance score of retrieving passage according the session s via score(z s), which can be estimated as the LLM''s generation log likelihood p(2 s) of conditioned

---

given session 8. Then we treat R as the references for white-box LLMs compule log f" 2) during the training. In practical implementation. we pre-generate the rewrites for all training ses- sions, thereby avoiding redundant calls to black- box LLMs. Our strategy greatly reduces the cost of employing black-box LLMs and demonstrates superior effectiveness and efficiency compared to the explicit usage of question rewrites for retrieval.
Phooe (1).2)
LLM
2.5.5
White Dox L1M
log ple 2.a)
Floase contaue wnta qustoo:
(a) Conrersational Retneval Conversation Generation.
Placa maniy helle guasion
c.9
S
Black Box
11M
12
pir (.g)
2.
Sc0ee, (z. 5)
VV hilb Rox
ILM
RR IR litt
log pir (1)
Question Response as Posterior Guide (INSTRUCTORORPc). We find that p(= s, a) can serve as more potent supervised signal than p(= s), as demonstrated in Appendix A. Here, refers to the gold response is generated by black-box language models. Therefore, we argue that question response a can provide additional relevance judgement when the passages related to the question are unknown under the unsupervised setting. To precisely identify relevant passages, we incorporate question response generation task into INSTRUCTOR as the posterior guide:
E
Firese deteiled queeton:
(b) Question Rewnte as Latent Vartable.
VV hits Rox MLM
Please creats conversatio: log pie II.
X,5.9,3
White
1LM
Plazse cvstinse queaion pig (......)))
soore In
White 11M
log pla Wi A..0) Please regood the asi quostoo:
(c) Question Response as Posterior Guide.
Figure 3: Three instructing strategies of INSTRUCTOR.
scorea(2 s) pp> 8) x logp(z c,
Question Rewrite as Latent Variable (INTRRCCTORORIN). We reveal that con- versational query rewriting exhibits outstanding performance in topic-switching scenarios illus- trated in Appendix A. To solve conversational retrieval from the query perspective, we utilize both black-box and white-box LLMs to rewrite question q. As shown in Table 4, we find that black-bor box LLMS have a much higher rewrite quality than white-box LLMs, indicating that black-box LLMs have more powerful linguistic capabilities to understand the conversations. However, suffering from limited API usage, we cannot completely use black-box LLMs in INSTRUCTOReRCG. To augment the retriever with the ability of black-box LLMs to tackle the user's ambiguous query 9. we model a query rewrite r as a latent variable and marginalizing over all possible rewrites R:
log cl
a(c,q,a)
-log p(z)p(e 2)p(g z, c)p(a 2,c,g) (9)
p(c)p(q c)p(a c,4)
x b(c 12, =) + p(g 12 z,
+ log 1 [. 2, c,
where log p(a c, q) is irrelevnn to 2, so it is elim- inated. As shown in Figure 3(C), we prompt LLMs with instruction /2,c "Please respond to the last question:" to generate the response for the last question q. considering the reference passage and the conversation session s. log p(a I2 2, C, 4) indicates that more informative passage 2 can assist LLMs in generating response a more effortlessly.
4 Experimental Setup
4.1 Datasets and Evaluation
score, s) s p(= 8) logp(z r)p(( (.
We evaluate our INSTRUCTOR in three experimental settings. The first is the unsupervised setting conducted on two popular conversational retrieval datasets: QReCC (Anantha et al., 2021) and TopiOCQA (Adlakha et al., 2022). QReCC is large-scale dataset for open-domain conversa- tional question answering (ODCQA) with human- rewritten query rewrites. TopiOCQA is a more chal- lenging dataset for ODCQA under topic-switching
Datasets.
=log p(r =))())p(( c.q)
p(r)
(8)
logp(r c.) logp(r 2)
plr C. log Ig =)).
FER
As shown in Figure 3(b), we first instruct black-box LLMs to generate a set of potential rewrites r E R with confidence logp(r log: C, q) based on the

---

QReCC
TopiOCQA
Retriever
MRR
NDCG@3
R@
R@ 100
MRR
NDCG@3
R@
R 100
Unsupervised Approaches
RewriteContieer
19.7
17.0
33.6
56.4
12.4
10.3
26.2
52.4
33.8
30.9
52.5
70.4
30.0
28.3
51.3
72.3
HyyDEConneve
40.4
37.3
63.0
82.9
24.1
22.0
45.2
69.8
HyDE CyyDEECnnerssmiieeee
43.3
40.5
84.6
35.3
34.1
59.0
77.8
query2doC Cmiiivee
7.4
19.1
12.2
9.9
24.4
52.1
query 2er.yddocconneeee
29.4
27.1
45.3
62.8
25.3
23.2
45.9
71.3
DPR
28.7
26.3
43.0
60.7
10.7
9.8
18.2
33.0
w/ INSTRUCTORcRce
34,5
31.8 15.5
55.1
73.5
20.2
18.8
35.4 717.2
59.7
w/ INSTRUCTOR
RQULY
30.1
26.4 10.1
52.3
75.4 114.7
21.8 111.1
20.1
39.8
64.5
w/ INSTRUCTORQuNG
36.2 17.5
33.3 17.0
57.0 114.0
75.6 114.9
29.9 719.22
28.3 118.3
50.3 132.1
72.6
ANCE
41.4
38.9
61.2
74.7
11.6
10.2
21.8
40.1
w/ INSTRUCTORcRUG
42.0 19.8
39.1 10:2
64.6 73-4
83.0 18.3
18.2 76.6
16.8 76.6
33.3 111.5
56.4 716.3
w/ INSTRUCTORgRLY
42.7 11.3
40,2 11.3
72.2
13.7
17.9 16.3
"5.6
34.8 113.0
119.6
w/ INSTRUCTORgRNG
43.5 12.1
40.5 11.6
66.7 15.5
85.6 110.
25.3 113.7
23.7 113.5
45.1 123.3
128.9
Contriever
41.5
38.7
63.8
84.3
6.9
5.3
14.5
37.5
w/ INSTRUCTOReRca
48.7
45.9 17:2
73.8 110.0
90.9 16.6
16.4 19.5
13.7 '84
34.0 119.5
67.5 130.0
w/ INSTRUCTORoRLV
43.6
40.4
69.0
90.7
18.6
15.9 130.6
37.7
71.8
w/ INSTRUCTORORNI
50.0
47.2
74.5
90.4 76.1
27.4
25.2
51.3 136.8
78.1
Contriever-nsmarco
47.6
45.0
70.5
88.9
16.2
14.3
29.4
56.0
w/ INSTRUCTORRRRG
50.0 50.0 12.4
47.2 1212
74,9 744
92.1 73.2
23.9 17.7
21.6 77.3
44.4 715.0
73.7 717.7
w/ INSTRUCTORQRIV
44.1 13.5
40.7 14.3
70.6 10.1
92.0 13.1
28.6 1124
26.0 111.7
52.3 122.9
80.2 124.2
w/ INSTRUCTORORRG
52,9 15.3
50.4 13.4
77.7 77.2
92.9 14.0
38.5 722.3
37.0 722.7
62.1 192.7
83.2 727.2
Supervised Approaches
DP
35.0
32.4
54.9
75.2
31.6
29.8
53.3
76.3
PNCE"T
48.2
45.3
70.6
88.3
32.6
31.3
53.7
73.3
Pontriever
48.7
74.7
34.4
32,9
78.5
Contriovee-msmrroo
50.9
48.1
76.0
92.2
35.0
33.6
78.9
SPLADEF11
50.0
46.6
69.9
87.8
30.7
29.5
52.1
72.0
ConvDR!
38.5
35.7
58.2
77.8
27.2
26.4
43.5
61.1
LeCoRBt
51.1
48.5
73.9
89.7
32.0
31.4
$4.3
73.5
Table 1: Experimental results of unsupervised setting on QRcCC and TopiOCQA. The best results are highlighted in bold. indicates the results taken from corresponding original papers.
scenarios. The second is the low-resource setting, we sample different numbers of conversation data from TopiOCQA. The third is the zero-shot setting, train the models on the TopiOCQA training set, then directly evaluate them on two conversational search test sets: CAsT-19 (Dalton al., 2020) and CAsT-20 (Dalton et al., 2021). More details about these four datasets are shown in Appendix C.
trained on MS MARCO (Nguyen et al., 2016); (3) Contriever (Izacard et al., 2022): An unsuper- vised retriever trained with contrastive learning; (4) Contriever-msmarco. Contriever fine-tuned MS MARCO. Except ANCE initialized with RoBERTa (Liu et al., 2019), all other retrievers are initialized with BERT (Devlin et al., 2019).
Large Language Models. We adopt Flan-T5 XI. (3B) as the white-box LLM and gpt-3. 2pt--355-20 5-turbo as the black-box LLM. More details about hyper- parameter settings can be found in Appendix E.
Evaluation Metries. In line with prior studies (Mao et al., 2023b), we employ four metrics (%): MRR, NDCG@3, Recall @ 10, and Recall @ 100 to comprehensively evaluate the retrieval perfor- mance. We use pytrec_ewe toolkit (Van Gysel and de Rijke, 2018) calculate these metries.
4.3 Baselines
Unsupervised Methods. We compare INSTRUC- TOR with three unsupervised methods that also adopt LLMs: (1) Rewrite: We an LLM to refor- mulate the query, then perform ad-hoc retrieval: (2) HyDE (Gao et al., 2022): An unsupervised method without training the retriever, which adopts an LLM to generate hypothetical documents for the query, then retrieves real documents with hypothetical
4.2 Implementation Details
Ad-hoc Retrievers. We adopt four representative ad-hoc retrievers as initial retrievers in INSTRUC- TOR: (1) DPR (Karpukhin al., 2020): retriever trained on Natural Questions (Kwiatkowski et al., 2019); (2) ANCE (Xiong et al., 2021): )): A retriever

---

tion session and discover the user's query intent. (2) Compared to other unsupervised methods that employ LLMs to generate query rewrites or hypothetical documents during inference time, our trainable method achieves superior performance. We argue that using LLMs to dynamically guide the training of retrievers enables better transition from ad-hoc retrieval to conversational retrieval. (3) Without using any labeled training data, IN- STRUCTOR surpasses the current supervised state- of-the-art method LeCoRE, with Recall @ 100 im- provements of 3.2% and 9.7% on QReCC and Topi- OCQA. Furthermore, parable or even better performance than direct fine- tuning on supervised data. This proves that LLMs have the zero-shot ability to measure text relevance. (4) Experimental results validate the effective- ness of all three proposed instructing strategies. IN- STRUCTORCRCG. which solely relies white-box LLMs, gains consistent improvements on QReCC and TopiOCQA. INSTRUCTOR which incor- porates more powerful black-box LLMs, improves significantly on TopiOCQA, but not so much on QReCC. These findings align with our insights pre- sented in Appendix A. Notably, INSTRUCTORQRrG achieves the best retrieval performance, demonstrat- ing the efficacy of utilizing question responses posterior guidance for the retrievers
1101
Realllgth
60
Caetia BEGARA INSTUCTI
55
Cartern ELESIRO PSTBUCTURa
0
200
500
1000
2000
5000
10000
20000
Number of Training Samples
method can achieve com-
Figure 4: Experimental results of low-resource setting on TopiOCQA.
documents; (3) query2doc (Wang et al., 2023): A query expansion approach, which expands the orig- inal query with LLM's generated documents. For a fair comparison, we all adopt gpt-3. 3pt-3.5-turbe 5-turbo as the LLM, Contriever and Contriever-msmarco the ad-hoc retrievers to reproduce these three methods on the conversational retrieval task.
Supervised Methods. We also compare our INSTRUCTOR with several supervised methods: (1) FT: We fine-tune DPR, ANCE, Contriever and Contriever-msmarco supervised data; (2) SPLADE (Formal al., 2021): sparse lexical- based retriever fine-tuned on supervised data; (3) ConvDR (Yu et al., 2021): ANCE fine-tuned on
5.2 Low-Resource Conversational Retrieval
We perform an evaluation of wwwrresuucee conver- sational retrieval on TopiOCQA, using different numbers of conversation data for training to sim- ulate the low-resource scenario. As illustrated in Figure 4, our method can better utilize small num- ber of training samples than directly fine-tuning the retriever. Only needing 200 samples can bring a 10%-15% performance improvement to the ad- hoc retriever. Besides, INSTRUCTORgRIV is data- efficient, requiring only 10,000 training samples (costing $5) to achieve excellent performance.
supervised data using knowledge distillation be-
tween the query rewrite representation and the la- tent session representation; (4) LeCoRE (Mao et al., 2023b): The current supervised state-of-the-art method, which extends SPLADE with two well- matched multi-level denoising methods.
5 Experimental Results
5.1 Unsupervised Conversational Retrieval
The experimental results of the unsupervised set- ting on QReCC and TopiOCQA are presented in Table 1, where we have the following observations: (1) INSTRUCTOR can bring consistent and signif- icant improvements across various ad-hoc retriev- ers in Recall @ 100, with an average improvement of 9.0% on QReCC and 34.1% % on TopiOCQA. Re- markable improvement TopiOCQA shows that our method is effective under challenging topic- switching scenarios, indicating that LLMs can help ad-hoc retrievers understand the complex conversa-
5.3 Zero-Shot Conversational Retrieval
We also conduct zero-shot conversational retrieval to evaluate the transferability. We first train the retrievers on the TopiOCQA training set and then directly evaluate them on CAsT test sets. Accord- ing to Table 3, note the following key findings: (1) INSTRUCTOR exhibits an average MRR im- provement 14.5% on CAsT-19 and 18.3% on CAsT-20, highlighting its strong generalization.

---

CAsT-11
CAsT-20
Retriever
MRR
NDCG@3
R
R
MRR
NDCG@3
R@
R@
w/o Fine-toning
RewrifSComiiver
40.6
24.5
6.4
26.1
21.7
12.0
7.5
25.1
Rewrile Comrreeenieieennmmaaa
63.6
48.2
12.2
43.8
39.6
25.7
14.6
43.5
HyDE Cmiiever
54.7
38.9
9.3
33.2
44.8
33.2
16.0
44.4
Hy Coniiver
Consttver Esmmano
61.5
46.3
41.4
51.1
36.5
$3.6
query2doc 2doc Contricue
47.8
32.4
27.6
29.1
query2doc (o Contrieee
65.2
50.6
12.0
41.8
55.9
40.4
20.1
52.2
Supervised Fine -uninn
DPR
40.9
26.4
5.2
16.4
22.5
ANCEFT
56.2
25.3
42.5
26.5
Contriever
53.7
38.4
8.2
23.7
35.5
20.8
22.5
Contriever-msmarce FT
55.7
41.2
9.0
25.7
36.8
21.8
8.6
25.5
Unsupervised Fine-tuning
30.1
17.5
15.6
INSTRUCTORgarG
120.4
32.9 115.4
7.2 13.4
24.2 16.6
115.5
18:2
15.1
115.0
ANCE
63.1
48.7
8.8
29.9
26.5
15.7
29.9
w/ INSTRUCTORgRrG
61.2 11.3
46.6 12.1
9.7 10.3
34.4
43.7 "17.2
29.6 113.9
13.6 13.9
40.8 110.9
Contriever
25,2
12.7
3.1
13.8
18.8
9.3
8.2
25.2
w/ INSTRUCTORgurG
57.7
39.8
9.0 9.0
33.9 "20.1
36.9
22.5
14.0 15.0
41.5 116.3
Contriecer-msmarvo
63.6
46.5
10.8
41.3
27.3
15.9
12.4
39.4
w/ InstRucToRgero
70.4
55.1
13.2 1244
45.4
51.7 "24.d
32.8 116.9
18.7 16.3
54.0
Table 2: Experimental results of zero-shot setting on CAsT-19 and CAsT-20. The best results in bold.
40
vil INSTRUCTTOR
Noxral Rotucice Sperifl Iuanntion
ANCE
$4.8
56.1
56.4
57.5
-75
70
67.8
DRR
59.1
50.5
59.7
69.3
901
65
V111
$6.2
Contricoer
6633
67.4
67.5
65.8
60
0.0
46
-011
0.0
72.4
73.5
73.7
75.2
10
TS-XC- edapt
FLAN XL Faithu::5
LLM
Rotilovrr
Figure 5: Recall @ 100 of INST on Topi-
Figure Recall 100 INSTRUCTOR on Topi- OCQA with different instructions.
OCQA with different ad-hoe retrievers and LLMs.
(2) Directly fine-tuning the retrievers on the su- pervised data shows poor out-of- -ifttiibiitttto gener- alization. It suggests that conversational retrieval requires the retrievers comprehend the conversa- tion rather than relying on retrieval shortcuts.
(2) TO: T5-XL.-Im-adapt with instruction tuning;
(3) Flan-T5 XL: A more powerful instruction-tune
T5: (4) FastChat-T5: An open-source chatbot fine- tuned from Flan-T5 XL. As illustrated in Figure 5, instraction-tuned LLMs are better instructors for trievers. In addition, conversational LLMs exhibit enhanced dialogue understanding capabilities and may be more suitable for conversational retrieval.
(3) Unsupervised methods with LLMs like HyDE and query2doc perform well in zero-shot scenarios, indicating the considerable potential of LLMs zero-shot information retrieval.
Investigation of Different Instructions. We prompt LLMs with three types of instructions: (1) Empty Instruction: We use meaningless space as the instruction; (2) Normal Instruction: We use " "Please write a text:" as the instruction; (3) Spe-
5.4 Discussion and Analysis
Selection Ad-hoc Retrievers and White-box LLMs. We choose four white-box LLMs (3B): (1) X------mmadapp An improved version of T5;

---

No-switch
Switch
Retriever
MRR
R@ 100
MRR
I00
MRR
R@ 100
MRR
R@ 100
Contriever
6.9
60.5
42.9
w/ INSTRUCTORcRCG
16,4
67.5
17.7
80.5
19.3
71.2
8.8
54.2
w/ RRyy
18.6
71.8
19.0
81.0
21.7
75.0
11.2
61.0
w/ INsTRUCTORoRrG
27.4
78.1
27.2
85.4
27.7
79.1
20.4
71.8
Contriever-msmarrco
16.2
56.0
36.2
89.3
17.9
60.7
6.1
34.1
w/ RNS
23.9
73.7
38.3
89.3
27.0
75.1
15.5
60.5
w/ ONSTRUCTORgRLV
28.6
80.2
34.8
84.9
31.1
84.1
19.2
65.9
w/ INsT
38.5
83.2
46.4
87.8
39,5
84.6
33.5
78.4
Table 3: Experimental results of different question types on TopiOCQA, All includes First, No-switehe and Switch types. The best results are in bold.
We carefully write instructions for different generation tasks in Appendix D. As de- picted in Figure 6, well-designed instructions can help LLMs to calculate relevance scores accurately.
cific Instruction:
85
Scaling with Number of LLM Parameters. We examine the effect of LLM sizes on retrieval perfor- mance improvements. Figure 7 shows the scaling of performance with LLM parameters, including FLAN-T5 small (80M), FLAN-T5 base (250M), FLAN-T5 large (780M) and FLAN-T5 XL (3B). We note that the size of FLAN-T5 is critical for IN- STRUCTORCROG- So we guess that using a larger FLAN-T5 XXL (11B) or Vicuna (13B) will further improve the retriever's performance. Compared with INSIRUCTORCRCG. INSTRUCTORQRLV has better retrieval performance due the use of black- box LLM ChatGPT. Overall, the performances of proposed three instructing strategies continue improve as the Flan-T5 parameters increase.
TT
Recalt
Contriooee-mmmmnn [NS] RUCTORRRC
Commeras-mmmatee w: INSTRUCTORRgI
75
Contriever-mmmmn
70
0,08B
0.25B
0.788
3B
Flan-T5 Parameters
Figure 7: Experimental results with different size of Flan- T5 TopiOCQA.
Analysis Different Question Types. Follow- ing the previous study (Kim and Kim, 2022), we
Due to the limited space, more comprehensive analysis is shown in Appendix F, G, H, and
define three question types, first, no-switch, and switch. The first question is the first question in con- versation without any history. The no-switch and switch questions be distinguished by whether dž contains similar or same topics as dř-1. where the de gold passage at turn and 1. We conduct experiments on TopiOCQA to analyze the impact of our approach on different question types. As shown in Table 1, we find that powerful ad-hoc retrievers like Contriever can solve the first prob- lem very well, but can hardly handle the switch question. After the guidance of our InstructoR, Contriever achieves the most significant perfor- mance improvement for the switch problem, indi- cating that L.LM-ggeerteed supervision signals can help the retriever understand the complex conver- sation session and discover the user's query intent.
Conclusion
In this paper, propose novel method termed INSTRUCTOR to instruct unsupervised conver- sational dense retrieval with LLMs. INSTRUCTOR calculates the session session-passage relevance score with frozen LLMss as the supervised signal to guide the ad-hoc retriever's training. To estimate the rele- vance score more precisely, we devise three strate- gies from diverse perspectives: conversational re- trieval as conversation generation, question rewrite as latent variable and question response posterior guide. Experimental results show that INSTRUC- TOR can bring significant improvements across var- ious ad-hoe retrievers, even surpassing the current supervised state-of taee--of-hheette t-thh-art art method.

---

Limitations
Sciences (Grant No.XDA27020100) the Youth In-
novation Promotion Association CAS, and Yunnan Provincial Major Science and Technology Special Plan Projects (N0.202222AD080004)
For further study, we conclude some limitations of our work as follows:
Limited by computing resources (4 NVIDIA
Geforce RTX A6000 GPUs), we can only adopt a woite-box LLM (e.g., Flan-T5 XL) with a maximum parameter size of 3B as the instructor. However, we acknowledge that scaling up the LLM's parameters beyond this limit, such as by utilizing more powerful model like Vicuna (13B) (Chiang et al., 2023) on A100 GPUs, has the potential to further enhance the overall retrieval performance.
References
Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Sule- man, Harm de Vries, and Siva Reddy. 2022. Topi- OCQA: Open-domain Conversational Question An- swering with Topic Switching. Trunsactions of the Association for Computational Linguistics, 10:468- 483.
We estimate the relevance score as the condi- tional generation log likelihood to provide a passage-level supervised signal. In the future, will explore more fine-grain grained relevance estimation method to provide token-level su- pervision signals.
Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520-534, Online. Association for Computational Lin- guistics.
Pro-
Due to the large capability gap between ad- hoc retriever and LLM, this results in the dis- crepancy of predictions between the retriever and a stronger LLM may tend to be severer. How to mitigate the large discrepancy of pre- dictions remains to be studied.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler. Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. 2021. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch.
Ethics Statement
To ensure the reproducibility of our paper, will release all source codes, all generated rewrites, and all trained checkpoints upon the acceptance of this paper. We conduct experiments with pub- licly available conversational retrieval datasets and open-source white-box LLMs. Since our method leverages LLMs to instruct the training of the re- trievers, it is important to note that the retrieval re- sults obtained may contain biases from the L.LMs. Additionally, we also employ black-box LLMS which can be accessed through OpenAl APIs. How- ever, when considering real-world applications, is crucial to recognize the potential risks associ- ated with uploading privacy-sensitive conversation data. These factors should be taken into careful consideration for future research and work.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Zhiyu Chen, Jie Zhao, Anjie Fang, Besnik Fetahu, Oleg Rokhlenko, and Shervin Malmasi. 2022. Reinforced question rewriting for conversational question an- swering. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing: Industry Track, pages 357-370, Abu Dhabi, UAE. Association for Computational Linguistics.
Acknowledgements
This work is supported by the National Key Re- search and Development Program of China (No. 2020AAA0106400), the National Natural Science Foundation of China (No. 61976211, 62176257). This work is also supported by the Strategic Pri- ority Research Program of Chinese Academy of
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph F, Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.

---

Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Com- putational Linguistics.
NLP. pages 94-99, Dublin, Ireland. Association for Computational Linguistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski. Armand Joulin, and Edouard Grave. 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Trunsactions on Machine Learning Research.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint ar 22x1:2200.11406
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Zhuyun Dai, Vincent Y Zhao, Ji Ma, Luan, Jianmo Ni. Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wa Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint arXiv:2209.11 1155.
Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expan- sion by prompting large language models. arXiv preprint arxiv:2305.03633
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020. Trec cast 2019: The conversational assistance track overview. arXiv preprint arXiv:2003. 13624.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Fdduno, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. Cast 2020: The conversational assistance track overview. In In Proceedings of TREC.
Jacob Devlin, King-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). As- sociation for Computational Linguistics.
Sungdong Kim and Gangwoo Kim. 2022. Saving dense retriever from shortcut dependency in conversational search. In Pruceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 10278- 10287, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Thibault Formal, Benjamin Piwowarski. and Stéphane Clinchant. 2021. Splade: Sparse lexical and expan- sion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 21, page 2288-2292 New York, NY, USA. Association for Computing Machinery.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Ilaia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones. Matthew Kelcey, Ming Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retricval without rele- vance labels. arXiv preprini arXiv:2213 12.11 10496.
Percy Liang. Rishi Bommasani. Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint a.X11:2211.09110.
Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng, Zheng Zhang. Jianzhu Yao, Lei Liu, Xiaoyan Zhu, and Minlie Huang. 2023. Eva2.0: In- vestigating open-domain chinese dialogue systems with large-scale pre-training. Machine Intelligence Research, 20(2):207-219
Sheng-Chiel Lin, Jheng- Hong Yang, and Jimmy Lin. 2021. Contextualized query embeddings for conver- sational search. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing. pages 1004-1015, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nam Hai Le, Thomas Gerald, Thibault Formal, Jian-Yun Nic, Benjamin Piwowarski, and Laure Soulier. 2023, Cosplade: Contextualizing splade for conversational information retrieval. In European Conference on Information Retrieval, pages 537-552. Springer.
Sheng-Chieh Lin, theng-Hong, Yang. Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2020. Conversational question reformulation via gqueecc-to-sequenee architectures and pretrained language models. arXiv preprint arXiv:2004.01909
Etsuko Ishii, Yan Xu, Samuel Cahyawijaya, and Bryan Wilie. 2022. Can question rewriting help conversa- tional question answering? In Proceedings of the Third Workhhp on Insights from Negative Results in

---

Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi 2022. WANLI: Worker and Al collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730- 277444 Curran Associates, Inc.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Dangi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprin! arXiv: 1907.1 1692..
Hongjin Qian and Zhicheng Dou. 2022. Explicit query rewriting for conversational dense retrieval. In Pro- ceedings of the 2022 Conference Empirical Meth- ods Natural Language Processing, pages 4725- 4737, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kelong Mao, Zhicheng Dou, Haonan Chen, Fengran Mo, and Hongjin Qian. 2023a. Large language mod- els know your contextual search intent: A prompting framework for conversational search. arXiv preprint arXiv:2303.06575
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang. et al. 2023. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint 230.22306.17663
Kelong Mao, Zhicheng Dou, and Hongjin Qian. 2022a. Curriculum contrastive context denoising for few- shot conversational dense retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 186-186, New York, NY, USA. As- sociation for Computing Machinery.
Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md Arafat Sultan, and Christopher Potts. 2023. Udapdr: Unsupervised domain adaptation via Ilm prompting and distillation of rerankers. arXiv preprint arXh::2303.00807
Kelong Mao, Zhicheng Dou, Hongjin Qian. Fengran Mo, Xiaohua Cheng, and Zhao Cao. 2022b. Con- v Vrans: Transforming web search sessions for con- versational dense retrieval. In Proceedings of the 2022 Conference on Empirical Methods Natu- ral Language Processing, pages 2935-2946, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022a. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 3781-3797, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Kelong Mao, Hongjin Qian, Fengran Mo. Zhicheng Dou, Bang Liu, Xiaohua Cheng, and Zhao Cao. 2023b. Learning denoised and interpretable session representation for conversational search. In Proceed- ings of the ACM Web Conference 2023, pages 3193- 3202.
Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. 2022b. Questions are all you need to train dense passage retriever. Transactions of the Association for Computational Linguistics.
Yu Meng. Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language mod- els: Towards zero-shot language understanding. In Advances in Neural Information Processing Systems.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai. Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. International Conference on Learning Representations.
Fengran Mo. Kelong Mao. Yutao Zhu Yihong Wu, Kaiyu Huang, and Jian-Yun Nie. 2023a. Convgqr: Generative query reformulation for conversational search. arXiv preprint arXiv:2305.15645.
Fengran Mo, Jian-Yun Nie, Kaiyu Huang. Kelong Mao, Yutao Zhu, Peng Li, and Yang Liu. 2023h. Learning to relate to previous turns in conversational search. arXiv preprint 230.0:2306.02553.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. choice, 2640:660.
OpenAI. 2023. Gpt-4 technical report.

---

Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Tianyi Zhou, and Daxin Jiang. 2023. Large language models are strong zero-shot retriever. arXiv preprint arXiv:2304. 14233.
Radev, Eduardo González Ponferrada, Efrat Lev- kovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hicu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar CronzalezDDios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing. Joydeep Bhattacharjee. Khalid Al- mubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan. Loubna Ben allal, Lu- dovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Saško, Max Huang. Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Moham- mad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen. Omar Espejel, Ona de Gibert, Paulo Villegas, Pe- ter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani. Roberto Luis López. Rui Ribeiro. Salomey Osei. Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silber- berg, Suhas Pai, Sydney Zink, Tiago Timponi Tor- rent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina. Veronika Laippala, Violette Lepercq. Vrinda Prabhu, Zaid Alyafeai, Zeerak Ta- lat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Tasar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajy- oti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang. Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Sai- ful Bari, Maged S. Al-shaibani, Matteo Manica, Ni- hal Nayak, Ryan Teehan. Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim. Tali Bers, Thibault Fevry, Trishala Necraj. Ur- mish Thakker, Vikas Raunak, Xiangru Tang, Zheng- Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh. Adam Roberts, Hyung Won Chung. Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayaman, Hatim Bourfoune, Jared Casper. Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette. Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, San- chit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil. Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lover- ing, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva. Ekaterina Voloshina. Eli Bog- danov, Genta Indra Winata, Hailey Schoelkopf, Jan- Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Na- joung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewiez. Verena Rieser, Vitaly Protasov.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301. 12652..
Tian-Xiang Sun, Xiang-Yang Liu, Xi-Peng Qiu, and Xuan-Jing Huang. 2022. Paradigm shift in natural language processing. Machine Intelligence Research,
1963):169-183.
Weiwei Sun, Lingyong Yan, Xinyu Ma. Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large lan- guage models as re-ranking agent. arXiv preprint arXiv:2304.09542.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume I Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971.
Svitlana Vakulenko, Shayne Longpre. Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM '21, page 355-363, New York, NY, USA. Association for Computing Machinery.
Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_evard eval: An extremely fast python interface to tree_ trec_eval. In The 41 International ACM SIGIR Con- ference Research & Development in Information Retrieval, pages 873 876.
Liang Wang. Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678
BigScience Workshop, :, Teven Le Seao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné. Alexandra Sasha Luc- cioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Vil- lanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Maiie Major. Iz Belt- agy. Huu Nguyen, Lucile Saulnier, Samson Tan, Pe- dro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir

---

Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger. Zdenčk Kasner, Al- ice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdol- lahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Car- los Muñoz Ferrandis, Daniel McDuff, Danish Con- tractor, David Lansky, Davis David, Douwe Kiela, Duong Nguyen, Edward Tan, Emi Baylor, Ez- inwanne Ozoani, Fatima Mirza, Frankline Onon- iwu, Habib Rezanejad, Hessie Jones, Indrani Bhat- tacharya, Irene Solaiman, Irina Sedenko, Isar Ne- jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim El- badri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok. Nafis Abrar, Nazneen Ra- jani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An. Rasmus Kromann. Ryan Hao, Samira Al- izadeh, Sarmad Shubber, Silas Wang, Sourav Roy. Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Lee Yoyo Yang. Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda--Escladaa, Ayush Singh, Benjamin Beilharz, Bo Wang. Caio Brito. Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang. John Giorgi, Jonas Golde, Jose David Posada, Karthik Ranga- sai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato. Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pamics, Maria Castillo, Mari- anna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg. Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Ki- blawi, Simon Ott, Since Sang-aroonsiri. Srishti Ku- mar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojeiech Kusa, Ya- nis Labrak, Yash Shailesh Bajaj. Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint 2211.05700.
Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In International Conference on Learning Representations.
Shi Yu, Jiahua Liu, Jingqin Yang, Chenyan Xiong, Paul Bennett, Jianfeng Gao, and Zhiyuan Liu. 2020. Few- shot generative conversational query rewriting. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR '20, page 1933-1936, New York, NY, USA. Association for Computing Machin- ery.
Shi Yu, Zhenghao Liu, Chenyan Xiong, Tao Feng, and Zhiyuan Liu, 2021. Few-shot conversational dense retrieval. In Proceedings of the 44th Inter- national ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '21, page 829-838, New York, NY, USA. Association for Computing Machinery.
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh Inter- national Conference on Learning Representations.
Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reit- ter, Hannaneh Hajishirzi, Mari Ostendorf, and Gau- rav Singh Tomar. 2022. CONQRR: Conversational query rewriting for retrieval with reinforcement learn- ing. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing. pages 10000-10014, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang,

---

sourievver-ssarce A Looking Deeper into Conversational Retrieval
The experimental results on QReCC and Topi- OCQA are presented in Table 4, where have the following observations:
In this paper, we conduct an in- in-depth investigation of conversational retrieval on QReCC and Topi- OCQA datasets. We employ four ad-hoc retrievers: DPR, ANCE, Contriever and without fine-tuning them on the conversational re- trieval data. To analyze the influence of different input formats on conversational retrieval perfor- mance, we conduct experiments with 12 kinds of input formats:
(1) We provide evidence that there exists re- trieval shortcut conversational retrieval. Follow- ing Kim and Kim (2022), we define the shorteut in conversational search as where gold passage d' can be predicted in top-K predictions even without the current question We find that just using the conversation context gains good performance in QReCC, which suggests that the ad-hoc retrievers cannot really understand the user's query intent, but rely on spurious lexical cues predict relevant passages. However, the performance of ad-hoc re- trievers will drop sharply TopiOCQA. This is because there are fewer retrieval shortcuts in the scene of topic switching.
(1) Question: the question Or at the current turn f;
(2) Context: the historical conversation context before turn t;
(3) Context&Question: the sequential concatena- tion of the context C and question gr;
(2) We reveal that conversational query rewrit- ing is not a silver bullet for conversational retrieval. Query rewrites are not as good as directly using the entire conversation sessions for retricval in QReCC Nevertheless, experimental results on Top- iOCQA demonstrate query rewriting works well in the scene of topic switching.
(4) Question&Contextt the reverse concatenation of the context Q-1 and question gr:
(5) Answer: the response a, to question dr:
(6) Lnswerr&Qustion&Cooneeee
the reverse con- catenation of the context 6 1. question & and response a;;
(3) We argue that black-box LLMs have more powerful linguistic capabilities than white-box L.Ms. We provide several query rewriting exam- ples in Table 5, 6, 7, 8, and 10. Black-box LLM ChatGPT achieves compa-co or even bet- ter query rewriting quality in QReCC. Sometimes ChatGPT is too smart and will answer questions redundantly (e.g., Figure 8) or refuse to rewrite un- certain questions (e.g., g., Figure 10). For white-box LLMs, the effect of query rewriting continues to im- prove as the LLM parameters increase. However, TOpp may not have been sufficiently fine-tuned with instruction data, so is difficult to follow our instructions to generate rewrites. It is worth mentioning that Vicuna which is an open-source chatbot trained by fine-tuning LLaMA (Touvron et al., 2023) achieves the closest effect ChatGPT.
(7) Flan-T5 XL Rewrite: query rewrite r, of question 4t generated by Flan-T5 XL2 (3B) (Chung et al., 2022);
(8) Flan-TS XXL Rewrite: a query rewrite r; of question de generated by Flan-T5 XXL (11B) (Chung et al., 2022);
(9) TOpp Rewrite: query rewrite Tt of ques- tion q generated by TOpp* (11B) (Sanh et al., 2022);
(10) Vicuna Rewrite: a query rewrite r of question 4 generated by Vicuna (13B);
(11) ChatGPT Rewrite: a query rewrite r; of ques- tion q generated by gpt-3. 5..33.5-uurr"
(4) We find that question responses can provide the relevant signal when the gold passages are known. Compared with Question&Context using Answer& Derreer&uueestinooooooooe Context the input format has significant improvement on QReCC and Topi- OCQA. Inspire by this, we treat question responses as the posterior guide to further enhance the train- ing of ad-hoc retrievers.
(12) Manual Rewrite: a human-rewritten query rewrite r of question qi only annotated in QReCC dataset.
hhtops://huginngfaee http:/ac...oooooeeeeaaatttt
hnttps://hugginge co/ ho/goooll/flannn-e t5- xx-x
'https://hugginggaac http://ggigngaacccooooeeeeeeeeee cw/bbggcceenc//eppp////o//
"https://githhbbcc wtwp:/ttuub.coommmssss///sse htw.://.mmmmmmss//sssccaaaaaa
(5) We find a trick that the reverse concatenation of conversation sessions leads to better retrieval
'https://platfon openai cww/docs/modelll
gpt-3-5

---

performance. Context&Question shows better re- sults than CouestionCContes on all four retrievers. Due to the limited context of the ad-hoc retrievers, We use reverse concatenation to avoid losing the important information the latest conversation.
use contrastive learning to train the conversational query encoder for context denoising. Recently, Mao et al. (2023b) propose sparse lexical-based conversational retriever two well-matched multi- level denoising methods.
B Related Work
B.2 LLM for Information Retrieval
B.1 Conversational Retrieval
Recently, some research (Sachan et al., 2022a,b; Yu et al., 2023; Saad-Falcon et al., 2023; Jagerman
Different from traditional single-turn ad-hoc re- trieval, conversational retrieval needs to deal with the multi-turn conversation context and understand the user's query intent. To solve conversational retrieval, existing methods are mainly divided into two categories: conversational query rewriting (Yu et al., 2020; Lin al., 2020; Vakulenko et al., 2021; Qian and Dou, 2022; Mo al., 2023a,b; Mao et al., 2023a) and conversational dense retrieval (Yu et al., 2021; Lin et al., 2021; Mao et al., 2022a; Ishii et al., 2022; Mao et al., 20226, 2023b; Hai Le et al., 2023).
et al., 2023; Sun et al., 2023; Shen et al., 2023; Shi et al., 2023; Mao et al., 2023a; Qin et al., 2023) has focused on applying LLMs (Brown et al., 2020; Sun et al., 2022; Gu et al., 2023; Workshop et al., 2022; Ouyang et al., 2022; OpenAI, 2023) in in- formation retrieval tasks. Most of these works use LLMs to generate query-related documents to ex- pand the original query in few-shot or zero-shot ad-hoe retrieval scenarios. Dai et al. (2022) prompt LLMs to generate synthetic task-specific training data for few-shot retrieval. Sachan et al. (2022a) use LLMs as the zero-shot reranker to improve pas- sage retrieval in open domain question answering. Given query, Gao et al. (2022) first use LLMs generate the hypothetical document, then ground the generated document the actual corpus. Wang et al. (2023) first generate pseudo documents by few-shot prompting LLMs, and then expands the query with generated pseudo documents. Sun et al. (2023) evaluate the capabilities of ChatGPT and GPT-4 on various passage reranking benchmarks. In this paper, we consider more realistic setting which train a conversational dense retriever in an unsupervised manner, i.e., without using session- passage pairs. We only have a certain number of conversation sessions and large passage collec- tion during the training. To the best of our knowl- edge, this is the first attempt to leverage LLMs to guide the training of unsupervised conversational retrieval.
Conversational query rewriting methods uti- lize the rewriting models explicitly reformulate the conversational queries into de-contextualized rewrites and then perform standard ad-hoc retrieval. Yu et al. (2020) first large amounts of ad hoc search sessions to generate weak supervision data, then fine-tune GPT-2 to rewrite conversational queries. Lin et al. (2020) demonstrate the effectiveness of T5 to rewrite queries. For better QA performance, Chen al. (2022) propose using QA feedback to supervise the rewriting model with reinforcement learning. Wu et al. (2022) also adopt reinforcement learning to directly optimize the rewritten query towards retrieval performance. Recently, Qian and Dou (2022) propose a unified framework for query rewriting and context modelling. Mao et al. (2023a) presents a prompting framework called LLMCS that leverages LLMs to perform few-shot conversa- tional query rewriting for conversational retrieval. Conversational dense retrieval methods usually fine-tune the ad-hoc retrievers on a large number of labeled session-passee passage pairs. Due to the lack of supervised data, Yu et al. (2021) fine-tune the ad-hoc search retriever on conversational search data using knowledge distillation for few-shot re- trieval. Kim and Kim (2022) train the dense re- triever with hard negatives effectively mitigates the heavy shortcut dependency. Lin et al. (2021) leverage external datasets to produce more pseudo- relevance labels for conversational search to over- come the lack of training data. Mao et al. (2022a)
C Dataset Details
Table 11 shows the statistics of QReCC, Topi- OCQA, CAsT-19, and CAsT-20 datasets.
QReCC is the first large-scale open-domain conversational QA dataset that contains human- annotated question rewrites. QReCC contains con- versations from QuAC (Choi al.. 2018), TREC CAsT and Natural Questions (Kwiatkowski et al., 2019). In this paper, we focus on solving the con- versational retrieval task.

---

is an open-domain conversational QA dataset with topic switches based Wikipedia. On average, a conversation in TopiOCQA has 13 -nnwer turns and involves topics. In contrast to QReCC, TopiOCQA does not provide human-annotated query rewrites.
TopiOCQA
FT Question Generation
Passage: (passage).
Please write detailed question based the given passage:
Question: (question).
CAsT-19 and CAsT-20 are two conversational scarch datasets relcased by TREC Conversational Assistance Track (CAsT). Due to the limited num- ber of conversations contained in them, they often used as the evaluation datasets.
D.3 Question Response as Posterior Guide
T 22,.9' Response Generation
Passage: (passage).
D Instruction Templates
Conversation context: [context).
Please answer the following question based on the given passage and conversation con-
D.1 Conversational Retrieval as Conversation Generation
text:
Question: (question).
1 Context Generation
Answer: (answer).
Passage: {passage).
Please create conversation (consisting of several questions and answers) based on the given passage:
E Parameter Settings
Our implementation based on HuggingFacc'ss Transformers?, Sentence Transformers", Mega- tron and PyTorch For ad-hoc retrievers, we adopt DPR" ANCE Contriever!, and Contriever-msmaroo In unsupervised train- ing framework, retrieve the top-K 32 pas- sages train the ad-hoc retriever. We the value of the temperature hyperparameter using cross- validation. We use the Adam algorithm optimize retriever parameters. The learning rate is initial- ized as le-6 with warmup and linear decay. The batch size is set to 4. For INSTRUCTORQRLV we use the manual query rewrites in QReCC and gen- erated query rewrites in TopiOCQA. We sample 15,000 conversations in the TopiOCQA training and use gpt-3. 5-turbo generate | 3 possible rewrites for each conversation with a tem- perature is 0.7. To ensure rewriting quality, we use regular expressions to filter out those rewrites
Conversation context: [context).
12,c: Conversational Query Generation
Passage: (passage).
Conversation context: (context).
Please complete the conversation with question based on the given passage and conversation context:
Question: (question).
D.2 Question Rewrite as Latent Variable
Tg Query Rewrite Generation
You are a helpful assistant that need to understand the following conversation.
/httpss//gitie thub. http://tww....ooooonnnggeeeaeaee
"https://ww/ sbert. net/
Conversation context: (context). Question: (question).
"nttpss//huugiingfacee
To better retrieve the relevant passages, please reformulate the last question of the given conversation context into complete and de-contextullized question rewrite:
"nttpss//huugiingaccce
"nttpss:/huggiingface
contriever-msmarco//

---

85
75
Codaleva
NDC083
Corslever-usmrre
Recall
Contievar INSTRUCTOROGUY
70
Comiriivaremmmenn w/ INSTRUCTO Rore
Conniever msmareo
65
Commiera-ommaenne Commirrasmmmeee w: INSTRUCTORRU
Contrieva--mmmaee INSTRUCTOR
Comiever-mmmanco INS
15
60
2
16
32
32
Top Retrieved Passages
Retrieved Passages
Figure 8: 3DCG@3 of different strategies with different numbers K of retrieved passages on TopiOCQA.
Figure 9: Recall@ @ 100 of different strategies with differ- ent numbers of retrieved passages on TopiOCQA.
that follow our instructions incorrectly. We adopt the gold responses our main experiments for INSTRUCTORguRC. We also use gpt-3. 3pt-3.5-tuns 5-turbo to generate query responses with a temperature is 0.3. All experiments are conducted with 4 NVIDIA GeForce RTX A6000 GPUs.
because larger R will contain incorrect rewrites, introducing more noise. Hence, we set [R] to a moderate value of 3 in main experiments.
Vinuca Rewrite
ChacIPT Rewrite
792
79.6
78.3
F
Effect of Top-K Retrieved Passages
77.1
75.4
75.3
78.1
We conduct experiments with different numbers (K 2, 4, 8, 16, 32) of retrieved passages during training. As shown Figure 8 and 9, we find that a smaller number of retrieved passages leads to better NDCG@3 performance, and a larger number of retrieved passages produces better Recall @ 100 performance. But retrieving too small passages, like K = 2, doesn't yield reasonable results. In this paper, set K 32 which offers reasonable middle ground, while K 16 may lead better performance.
75
33.1
Realts8100
65
Nomber [R] of Poosible Generated Rewrites
Figure 10: Recall 100 of Contriever-msmarco w/ IN- STRUCTO with Vicuna Rewrite and ChatGPT Rewrite on TopiOCQA.
G Ablation of Question Rewrites
We conduct an ablation study on INSTRUC- TORØRLV with Vicuna Rewrite and ChatGPT Rewrite. Vicuna Rewrite denotes the query rewrites generated by Vicuna, and ChatGPT Rewrite denotes the query rewrites generated by opt-3.5-turbe As shown in Figure 10, ChatGPT Rewrite performs better than Vicuna Rewrite be- cause of the higher rewriting quality ChatGPT. Besides, we find that the number |R| of possible rewrites is not the higher the better. This may be
H Ablation of Question Responses
We conduct an ablation study INSTRUC- ToRqRPG with Gold Response and Generated Re- sponse. Gold Response denotes the real response in the original conversation, and Generated Response denotes the response generated by 3pt-3.5-turbae As illustrated in Figure 11, Generated Response works better than Gold Response when there are a

---

small number of training samples. However, when there are enough training samples, Gold Response has better performance than Generated Response, especially in NDCG@3. Hence, We adopt the Gold Response for ORs in our main ex- periments.
Performanc
Resporse
Recalig 100 of Gold Responsc
40
NDCG & Genaaaed Kispanie
Gold Resoree
2,00
500
1000
2000
5000
10000
Number of Training Samples
RecaligN
Figure 11: Recall@ 100 and NDCG@3 of Contriever- msmarco w/ RRNG with Manual Response and Generated Response on TopiOCQA.
I Recall N of Different Strategies
Cool anno TEC.ERiae
We also present the @ of different in- structing strategies with different numbers N (1,3, 5, 10, 20, 50, 100) of retrieved passages in Figure 12. We can find that montriever-msmarco and RNS are the best combination.
ASIT ENSTRACTTO
Continat (USTE
10
20
50
100
Number of Retrieved Passsagss
Figure 12: Recall@ of different strategies with differ- numbers N of retrieved passages TopiOCQA.

---

QReCC
TopiOCQA
Retriever
Input Format
MRR
NDCG@3
R@10
R@ 100
MRR
NDCG@3
R@10
R@ 100
Question
5.5
4.8
8.4
14.3
3.7
3.3
6.7
12.2
Context
24.9
23.1
37.2
51.4
7.4
6.9
12.6
23.5
Context&Question
44.1
61.5
10.6
35.0
Question&Context
28.7
26.3
43.0
60.7
10.7
9.8
18.2
33.0
Answer
24.5
22.9
34.8
51.1
15.5
15.1
21.5
30.0
Auswer&&Qeetion&&Cooo
36.7
34.3
53.3
70.5
14.5
13.4
24.2
40.2
DPR
Flan-115 Rewrite
10.7
19.1
31.5
11.7
10.7
21.5
35.0
Flan-T5 XXL Rewrite
13.4
11.8
20.9
33.9
16.1
15.4
27.7
41.7
TOpp Rewrite
12.6
11.1
20.1
32,4
8.6
7.8
15.2
25.8
Vicuna Rewrite
22.2
35.9
14.3
13.2
25.8
ChatGPT Rewrite
21.6
19.3
33.7
51.5
15.4
14.3
27.6
46.2
Manual Rewrite
21.7
19.5
33.1
51.0
Question
20.1
3.7
3.4
7.2
Context
30.9
29.0
47.1
58.8
6.1
5.4
10.7
21.8
Context& Question
38,2
35.7
57.6
72.3
8.7
7.8
15.6
29.9
Question&Conteet
41.4
38.9
61.2
74.7
10.2
21.8
Answer
40.5
38.6
54.0
63.9
10.5
9.9
16.1
26.6
Answer& Cerrrrr&uusstinuuuuutttet
54.0
51.6
72.9
83.5
23.2
21.7
37.6
55.8
ANCE
Flan-T5 XL Rewrite
22.3
20.4
34.4
46.1
14,0
13.3
25.0
36.4
Flan-115 XXL Rewrite
20.5
32.1
43.4
15.9
14.7
27.8
'Tppp Rewrite
19.9
18.2
30.9
41.4
9.0
8.4
16.1
25.7
Vicuna Rewrite
28.7
26.6
43.8
57.3
19.6
18.4
34.6
51.9
ChatGPT Rewrite
32.5
29.9
49.3
63.5
22.7
21.7
39.3
58.0
Manual Rewrite
33.7
31.2
50.2
65.3
Question
6.1
5.0
11.2
20.3
1.5
1.1
3.4
9.7
Context
35.2
69.9
5.8
4.8
11.7
Context& Question
41.2
38.4
63.2
83.4
5.8
4.1
12.3
34.1
Question&CContett
41.5
38.7
63.8
84.3
6.9
5.3
14.5
37.5
Answer
56.5
54.8
83.4
21.4
20.5
29.8
42.2
Answer&
58.9
57.0
80.3
92.4
19.0
16.9
34.7
60.7
Contriever
Flan-T5 XL Rewrite
16.1
13.7
28.4
49.6
5.8
4.6
12.5
29.2
Flan-T5 XXL Rewrite
15.5
13.1
27.5
49.2
7.3
5.8
15.0
35.3
TOpp Rewrite
53.7
4.9
3.8
11.1
Vicuna Rewrite
23.3
20.5
40.4
65.5
9.8
8.0
21.4
45.6
ChatGPT Rewrite
28.7
25.5
48.6
74,9
12.4
10.3
26.2
52.4
Manual Rewrite
26.9
23.5
46.3
74.6
Question
11.5
10.3
18.4
27.1
5.1
4.7
9.2
16.2
Context
36.6
34.6
55.0
70.4
7.8
6.8
14.0
29.8
Context& Question
42.2
86.6
11.6
10.1
20.6
Duestion&Connten
47.6
45.0
70.5
88.9
16,2
14.3
29.4
56.0
Answer
56.3
54.7
71.4
82.5
25.5
24.9
34.5
47.1
Cuswerr&uuctiioo&CCnnee
63.0
61.2
82.4
93.8
32.1
30.6
52.4
Contriever
Flan-TS XI. Rewrite
29.6
66.1
17.8
16.9
30.6
43.6
Flan-T5 XXL. Rewrite
27.1
24.4
43.9
62.7
20.0
19.1
34.2
48.2
TOpp Rewrite
26.5
24.2
42.4
59.3
11.8
10.9
19.9
32.4
Vicuna Rewrite
79.4
24.5
45.0
ChatGPT Rewrite
44.0
40.8
67.0
87.1
30.0
28.3
51.3
72.3
Manual Rewrite
44.1
40.7
67.6
88.6
Table 4: Experimental results of different input formats and ad-hoc retrievers QReCC and TopiOCQA.

---

41 What happened to Martha and the Vandellas during motown major
hit years?
aj: Following their signing to Motown's Gordy imprint in 1962, the Vandellas struck gold with their second release.
92: What happened after that?
02: Martha and the Vandellas' second hit, Heat Wave, became phenom-
enal record for the group.
T3: Can you tell me how long did it stay on the charts?
az: Martha and the Vandellas' Heat Wave, became phenomenal record,
hitting number one on the R&B singles chart for five weeks.
q4: What were some of their major hits?
a4: Martha and the Vandellas' string of hits included Come and Get
These Memories, Heat Wave, and Dancing in the Street.
Qs: Were they popular among lot of people?
Conversation Session
a5: Martha and the Vandellas struck gold with their second release, titled
Come and Get These Memories.
96: What other things took place after that?
ag: Heat Wave became their first million-seller and eventually got the
group their only Grammy Award nomination.
47: Did they go on to continue to be successful?
07: Martha and the Vandellas' success continued with their second Top
Ten single and third Top 40 single, Quicksand.
98 Any other important aspects regarding their major hit years?
ag: Annette, who was pregnant with her first child and get married,
chose leave her singing career behind by 1964.
(g: Did the group grow apart after she left?
ag: Betty Kelley, formerly of the Velvelettes, was brought in shortly afterward.
910: Did she do well with joining the group?
Betty Kelley, formerly of the Velvelettes, was brought in shortly
Response:
Flan-T5 XL Rewrite
Flan-T5 XXL Rewrite
Can you tell me how long it stayed on the charts?
Martha and the Vandellas' second hit, Heat Wave, became a phenomenal record.
TOpp Rewrite
Question: What happened to Martha and the Vandellas after Annette left the group and Betty Kelley joined?
Vicuna Rewrite
ChatGPT Rewrite
How did Betty Kelley perform after joining Martha and the Vandellas?
Did Betty Kelley do well with joining the group Martha and the Vandel- las?
Manual Rewrite
Kelley was let go after reportedly missing shows, as well as getting into altercations with Reeves.
Gold Response
Table 5: QReCC example 1. Red denotes the current user's query. Blue denotes the correct rewritten content.

---

Q1 When was the expansion for the Arapaho?
a1 Once established, the Arapaho began to expand on the plains through trade, warfare, and alliances with other plains tribes. Around 181 the Arapaho made an alliance with the Cheyenne.
Q2: What expanded?
02: Their strong alliance with the Cheyenne allowed the Arapaho to greatly expand their hunting territory.
93: What was special about the expansion?
az: Conflict with the allied Comanche and Kiowa ended in 1840 when
Conversation Session
the two large tribes made peace with the Arapaho and Southern Cheyenne and became their allies.
Q4: Who did they trade with?
a4: Along the upper Missouri River, the Arapaho actively traded with
the farming villages of the Arikara, Mandan, and Hidatsa.
45: Who did they go to war with?
a5: By 1826, the Lakota, Dakota, Cheyenne, and Arapaho pushed the Kiowa (Niciilechhineenolu@bbb; Kiowa tribe: Niciiheihiitenn) and in- vading Comanche to the south.
96 What types of things did they trade?
Flan-T5 XL Rewrite
Response:
What types of things did the Arapaho trade?
Flan-T5 XXL Rewrite
Can you tell me how long it stayed on the charts?
TOpp Rewrite
What did the Arapaho trade with the Arikara, Mandan, and Hidatsa.
Question: What did the Arapaho trade with the farming villages along the upper Missouri Riversnnesspons: Along the upper Missouri River, the Arapaho actively traded with the farming villages of the Arikara, Mandan, and Hidatsa.
Vicuna Rewrite
What were the items or goods that the Arapaho actively traded with the farming villages of the Arikara, Mandan, and Hidatsa along the upper Missouri River?
ChatGPT Rewrite
Manual Rewrite
What types of things did the Arapaho trade?
Gold Response
They traded meat and hides for corn, squash, and beans.
Table 6: QRcCC example 2, Red denotes the current user's query. Blue denotes the correct rewritten content.

---

Q1: Tell me about William Dampier's second circumnavigation?
ay: William Dampier was appointed commander of the 26-gun ship St George, with a crew of 120 men.
02: Where did he set sail from?
02: William Dampier set sail on 11 September 1703 from Kinsale, Ire-
land.
ę3: Did he encounter pirates?
0z: UNANSWERABLE.
Conversation Session
44: Where did they go from Kinsale?
a4: The two ships made a storm-tossed passage round Cape Horn, arriv-
ing at the Juan Fernlu00e Inde Islands off the coast of Chile in February 1704.
45: What did they do there?
05: While watering and provisioning there, they sighted heavily armed French merchantman, which they engaged in a seven-hour battle but were driven off.
96 Did they fight?
Flan-T5 XL Rewrite
Response: Did William Dampier fight with the French?
Flan-T5 XXL Rewrite
What was the name of the French merchantman?
TOpp Rewrite
Did they fight?
Question rewrite:ln What happened during the seven-hour battle between the St George and heavily armed French merchantman that they en- gaged off the coast of Chile?
Vicuna Rewrite
Did William Dampier and his crew engage in a battle with a French merchantman while at the Juan Fern'u00e 1 Islands?
ChatGPT Rewrite
Manual Rewrite
Did William Dempier and his crew fight in Chile?
Gold Response
Yes, they engaged in a seven-hour battle but were driven off.
Table 7: QReCC example 3. Red denotes the current user's query. Blue denotes the correct rewritten content.

---

91 Does callic baby die in season episode 18?
01 No.
92: Who plays the character mentioned above?
02: Sara Ramirez.
43 Apart from acting, does she have a career in any other profession?
a3: She is singer and songwriter.
44: Name some of her songs?
Conversation Session
a4: Silent Night.
45: What is the significance of the above song?
ag: It is a popular Christmas carol.
4: Who has written it?
ag: Joseph Mohr.
47: The above mentioned episode is from which series?
a7: Grey's ' Anatomy.
98 Name some characters of it.
Flan-T5 XL Rewrite
Response: what are some characters in the show grey's anatomy?
Flan-T5 XXL. Rewrite
What is the name of the character who dies in season 7 episode 18?
TOpp Rewrite
Turn 3: Sara Ramirez is singer and songwriter. What is the significance
Question: Can you give me the names of some characters of the TV series Grey's 's Anatomy?
Vicuna Rewrite
Rewrite: Can you provide some names of the characters from the TV series Grey's s Anatomy? 'nResponse: Meredith Grey, Derek Shepherd, Miranda Bailey, Alex Karev, Cristina Yang, and many more.
ChatGPT Rewrite
Grey's Anatomy:
Introduction. Justin Chambers, who played Dr. Alex Karev, left the show during this season. \"Grey's Anatomy is the longest-running scripted primetime show currently airing ABC, and the longest scripted primetime series carried by ABC. As of February 28, 2019, it was the longest rumning American primetime medical drama series. The series was most recently renewed for its seventeenth sea- son. Its success catapulted longtime series regulars including Pompco, Dempsey and Oh to worldwide recognition; they were among the top 5 highest-earning television actors in 2013. While the show's ratings have fallen over the course of its run (it was once among the overall top 10 shows in the United States), it is still one of the highest-rated shows among the 18lu201349 349 demographic, and the #3 drama all of broadcast television in the United States.
Gold Passage
Gold Response
Meredith Grey, Alex Karev, Miranda Bailey and Richard Webber.
Table 8: TopiOCQA example 1. Red denotes the current user's query. Blue denotes the correct rewritten content.

---

Q1 Which was the new building constructed at the start of the 1900s?
01: It transformed its society into the world's first industrialised nation.
22- Which age its history be traced back to?
02: 780,000 years ago.
03 Wha htt are some of the rivers found in this country?
03: Thames, Mersey, Tyne, Severn etc.
Conversation Session
q4: Any well-known museums here?
04: London's ' British Museum.
45 Where is this museum located?
05: Bloomsbury area of London, United Kingdom.
9: Who was the founder?
ag: Sir Hans Sloane.
27: Which was the new building constructed the start of the 1900s?
Response: what was the building constructed at the start of the 1900s?
Flan-T5 XL Rewrite
Flan-T5 XXL Rewrite
What the new building constructed at the start of the 1900s?
The British Museum is located in the Bloomsbury area of London, United Kingdom.
TOpp Rewrite
Question rewrite:An What was the name of the building that was con- structed the start of the 1900s the British Museum?
Vicuna Rewrite
Can you provide information about the new building constructed at the start of the 1900s in the British Museum located in the Bloomsbury area of London, United Kingdom?
ChatGPT Rewrite
British Museum:
New century, building (1900u20131925), Morgan had also acquired major part of Sir John Evans's 's coin collection, which later sold to the museum by his son John Pierpont Morgan Junior in 1915. In 1918, because of the threat of wartime bombing, some objects were evacuated via the London Post Office Railway to Holborn. the National Library of Wales (Aberystwyth) and a country house near Malvern. On the return of antiquities from wartime storage in 1919 some objects were found to have deteriorated. A conservation laboratory was set up in May 1920 and became permanent department in 1931. It is today the oldest in continuous existence.
Gold Passage
Gold Response
A conservation laboratory.
Table 9: TopiOCQA example 2. Red denotes the current user's query. Blue denotes the correct rewritten content.

---

91: Which of the following the ultimate result of evolution?
a1: UNANSWERABLE.
42 What is evolution?
02: It is change the heritable characteristics of biological populations
over successive generations.
23 Who was the first to study this?
az: Charles Darwin and Alfred Russel Wallace.
44: When was the first one born?
a4 Shrewsbury, Shropshire.
T5: Did he suffer from any illness?
05: He had "an uncomfortable palpitation of the heart".
qe: Did he publish anything?
Conversation Session
ag: Yes, his theory of evolution with compelling evidence in his 1859
book "On the Origin of Species".
47: What does modern synthesis state about evolution?
a7: It explained patterns observed across species in populations, through fossil transitions in palaeontology, and complex cellular mechanisms in developmental biology.
1s: What is the last type of biology about?
ag: It is the study the process by which animals and plants grow and develop.
19: Does it have any model for organisms?
0g:
910: Where has the study of evolution been applied to?
Flan-T5 XL Rewrite
Response:
What has the study of evolution been applied to?
Flan-T5 XXL Rewrite
What is the ultimate result of evolution?
Change the heritable characteristics of biological populations over successive generations.
TOpp Rewrite
Question rewrite:lnWhal are some areas where the study evolution has been applied?
Vicuna Rewrite
Please provide complete and de-contextualized question rewrite for the last question in the conversation.
ChatGPT Rewrite
Evolution: Applications. Concepts and models used evolutionary biology, such as natural selection, have many applications. Artificial selection is the intentional selection of traits isms. This has been used for thousands of years in the domestication of plants and animals. More recently, such selection has become vital part of genetic engineering, with selectable markers such antibiotic resistance genes being used to manipulate DNA. Proteins with valuable properties have evolved by repeated rounds of mutation and selection (for example modified enzymes and new antibodies) directed evolution. Understanding the changes that have occurred during an organism' 's evolution can reveal the genes needed to construct parts of the body, genes which may be involved in human genetic disorders.
population of organ-
Gold Passage
process called
Gold Response
Domestication plants and animals, genetic engineering, etc.
Table 10: TopiOCQA example 3. Red denotes the current user's query. Blue denotes the correct rewritten content.

---

Dataset
# Conversations
Turns
# Passages
Train
10,823
63,501
QReCC
54M (54,573,064)
Test
2,775
16,451
Train
3,509
45,450
TopiOCQA
25M (25,700,592)
Test
205
2,514
CAsT-19
Test Test
50
479
38M (38,429,852)
CAsT-20
25
216
Table 11: Statistics of conversational retrieval datasets.


Reviews:
[{"review_id": "ce66DuSrBw", "reviewer": "Reviewer_WnM3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews. Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! \n\n> **Response to Rejection 1**: My major concern is that you might be able to achieve impressive results with weak retrieval supervision from the conversation data, without the need to query LLMs as instructors. The training data used, namely QReCC and TopiOCQA, comes with both question and answer turns. You can apply the same method used in CONQRR [1] to derive \"gold\" passages and fine-tune retrievers with those weak labels. My guess is that the performance can already be good enough, which may also explain why QRPG gives the highest score as it uses the next answer turns. Also, you'll need to use hard negatives in retriever training as that's widely acknowledged to boost performance.\n\n**We would like to clarify the differences between InstructoR and the heuristics-based matching method employed in CONQRR and highlight the advantages of our method**. \n\nTo annotate pseudo-gold passages as the weak retrieval supervision, CONQRR heuristically maps the answer $a_{t}$ to one passage with the **highest span overlap score**, and then designates it as the \"gold\" passage $d_{t}^{*}$ for the question $q_{t}$. Since the answers may be hidden in the original passages, this heuristics-based matching method has been widely adopted in open-domain QA [2]. The main differences and advantages between our InstructoR and this heuristics-based matching method are as follows:\n- In our training framework, the supervised labels generated by LLMs are **soft** (0 - 1). However, the labels obtained by heuristics rule are **hard** (0 or 1). For conversational retrieval, the relevance between session and passage may not be just 0 or 1. So **our method adopts more fine-grained labels**, achieving significant improvements across various ad-hoc retrievers. **It is worth mentioning that our method achieves even better performance than using real gold labels, which cannot be achieved using pseudo-gold labels**.\n- Our method employs LLMs to read sessions and passages and estimate the relevance scores, **which can better understand the relevances between sessions and passages than rule-based matching**. For example, it is difficult to find positive passages for boolean and abstractive answers using only string matching functions [3]. Retrievers can be affected by the noise introduced by the heuristics-based matching method.\n- While QRPG is only one of the three strategies we propose, **CRCG and QRLV strategies can still work well without using additional question responses**. Our method considers a more challenging unsupervised scenario, and provides effective solutions. **We also explore the use of LLM-generated responses in QRPG and demonstrate its effectiveness in Appendix I**.\n- Our method iteratively retrieves top-K passages and then scores them with LLM during the training stage. In contrast, the heuristics-based matching method involves dataset preprocessing before the training stage. **The retrieved top-K passages used for training can be considered as the hard samples [4] you mentioned, thus resulting in an improvement in our performance**.\n\n[1] CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning, Wu et al. 2022.\n\n[2] Dense Passage Retrieval for Open-Domain Question Answering, Karpukhin et al. 2020.\n\n[3] Neural Ranking with Weak Supervision for Open-Domain Question Answering: A Survey, Shen et al. 2023.\n\n[4] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval, Xiong et al. 2021.\n\n> **Response to Rejection 2**: I don't agree with claim that \"annotating session-passage pairs is much more difficult than collecting conversation data\". The datasets you use are collected by having annotators search for passages first before writing agent utterances. How could annotating passages more difficult than labeling conversation data?\n\nWe agree with your concerns in this regard. We want to explain why we claim that \"annotating session-passage pairs is much more difficult than collecting conversation data\".\n\n- We observe that a significant portion, specifically 53.4%, of the conversational questions present in the QReCC training set lacks corresponding labeled passages. Furthermore, it's worth noting that the CAsT datasets exclusively provide test sets without corresponding training sets. So we think **it is time-consuming and labor-intensive to label the source of the question responses**.\n\n- In practical applications, we can **collect rather than label** a large amount of public dialogue data through various sources (e.g., social media, chat platform and online forum), but the responses in these dialogues are often not clearly annotated with the sources (i.e., the supported passages).\n- In addition, some recent studies [5] have tried to use LLMs to automatically construct conversation data without human annotations, and **there will be no session-passage annotations for such synthetic conversations**. We think it is an interesting direction how to train a conversational retriever on such automatically synthesized conversation data. This could be a further extension of our work in the future.\n\n[5] q2d: Turning Questions into Dialogs to Teach Models How to Search, Bitton et al. 2023.\n\n> **Response to Rejection 3**: You are using LLM generation probability to calculate the document relevance, which is then used as training signals for retrievers. Why don't you simply use LLM generation prob to rank passages? You may get even better results than fine-tuning the retriever.\n\nThanks for your valuable suggestion! We indeed explored the utilization of the LLM (Flan-T5 XL) generation probability to rerank the retrieved top-1000 passages. As shown in **Table 1**, we find that **the performance of LLM Rerank approach merely exhibits a marginal improvement compared to our CRCG strategy. However, it significantly lags behind the results achieved by both the QRLV and QRPG strategies**. We think this result is reasonable for the following reasons:\n\n- Our InstructoR adopts a trainable framework which iteratively mines top-K hard samples to **train the retriever in an unsupervised manner**. However, LLM rerank can only rank the passages **in a zero-shot manner**. Besides, using hard samples in the retriever training can boost performance.\n- During the training stage, QRLV models question rewrites generated by black-box LLMs as latent variables, and QRPG treats question responses as the posterior guide. **These useful supervised signals are not accessible to LLM Rerank method at the inference stage**.  So QRLV and QRPG strategies have achieved better performance than LLM Rerank.\n- LLM Rerank can only rank top-K (K<10000) retrieved passages at the reranking stage. However, it is difficult to handle **millions of passages** at the retrieval stage. Compared with LLM Rerank, **our method is more efficient with less latency**. Because our method only needs to distill knowledge from LLM into the retriever at the training stage, and does not need to use LLM at the inference stage. \n\n**Table 1:** The performance of LLM Rerank and InstructoR on TopiOCQA.\n\n| Retriever                            | MRR  | NDCG@3 | R@10 | R@100 |\n| ------------------------------------ | :--: | :----: | :--: | :---: |\n| Contriever                           | 6.9  |  5.3   | 14.5 | 37.5  |\n| &nbsp;&nbsp;&nbsp;w/ LLM Rerank      | 17.0 |  14.3  | 34.4 | 70.6  |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG | 16.4 |  13.7  | 34.0 | 67.5  |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV | 18.6 |  15.9  | 37.7 | 71.8  |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG | 27.4 |  25.2  | 51.3 | 78.1  |\n| Contriever-msmarco                   | 16.2 |  14.3  | 29.4 | 56.0  |\n| &nbsp;&nbsp;&nbsp;w/ LLM Rerank      | 24.7 |  22.1  | 47.0 | 76.6  |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG | 23.9 |  21.6  | 44.4 | 73.7  |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV | 28.6 |  26.0  | 52.3 | 80.2  |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG | **38.5** |  **37.0**  | **62.1** | **83.2**  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. They propose a novel framework to fine-tune the retriever model to handle conversational queries.\n3. They did extensive experiments to show improvements of their approach on three different benchmarks.", "reasons_to_reject": "1. My major concern is that you might be able to achieve impressive results with weak retrieval supervision from the conversation data, without the need to query LLMs as instructors. The training data used, namely QReCC and TopiOCQA, comes with both question and answer turns. You can apply the same method used in CONQRR (Wu et al., 2022) to derive \"gold\" passages and fine-tune retrievers with those weak labels. My guess is that the performance can already be good enough, which may also explain why QRPG gives the highest score as it uses the next answer turns. Also, you'll need to use hard negatives in retriever training as that's widely acknowledged to boost performance.\n\n2. I don't agree with claim that \"annotating session-passage pairs is much more difficult than collecting conversation data\". The datasets you use are collected by having annotators search for passages first before writing agent utterances. How could annotating passages more difficult than labeling conversation data?\n\n3. You are using LLM generation probability to calculate the document relevance, which is then used as training signals for retrievers. Why don't you simply use LLM generation prob to rank passages? You may get even better results than fine-tuning the retriever.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the problem of conversational passage retrieval. The authors propose a method, named Instructor, to use frozen LLMs to provide retriever training signals. Specifically, the relevance score between a passage and a conversational query (i.e., conversation history + current user question) is calculated as the generation probability of the passage by a LLM, conditioning on the conversational query. The authors propose three ways to calculate such generation probabilities. Then they train the retriever with the KL loss between LM generation probabilities and passage-query similarities from the retriever.\n\nThey conduct experiments on three conversational retrieval benchmarks and achiever significant improvements with such training guidance from LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6VbSd6EedP", "reviewer": "Reviewer_UCrL", "summary": "", "questions": "1. Can this framework designed without LLM? (i.e. How should we think is this method flexible/feasible or not) and how can we ensure the reproduction as the current LLM normally with fast version iteration. If not, is it still a flexible framework?\n\n2. What is the cost and the latency of using LLM, especially on the two big QReCC and TopiOCQA datasets? Please show statistic information.\n\n3. How can we control the generated text by LLM will not bring noise to the query encoder training and how can we analysis via experiments? In other words, how to evaluate the quality of the generated text by LLM for IR evaluation. \n\n4. How to consider the comparison with the manually rewritten query and the query reformulation approaches, especially on the cast datasets.\n\n5. For Eq. 7, why we can assume log p(z), log p(c) and log p(q | c) as instants and what is the relation between eq. 7 and the quality of the generated text?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews. Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! \n\n> **Response to Rejection 1**: The approach relies heavily on large proprietary language models, which have limitations like bias and lack of transparency and there are limited error analysis to understand cases where the method fails. This could be important since the LLM is used as a black-box model and the quality of generated results are without guarantee. Meanwhile, the workflow details (e.g. the concrete prompt and the reformulated query) for the proposed three strategies should be much clearer.\n\nWe agree with your concerns in this regard. We think that **our paper is an early exploration of LLM in information retrieval, aiming to utilize LLMs to improve conversational dense retrieval in an unsupervised manner**. \n\nWhile current LLMs have some limitations like bias and lack of transparency, we believe that this will be gradually solved in the future. In our work, we are cautious about these limitations. **To make our method clearer and more transparent, we provide the workflow details of our method in Appendix**. As shown in Appendix E, we provide all the instruction templates/prompts used in our strategies for reproduction. As shown in Table 4, 5, 6, 7, 8, and 9, we provide several reformulated query examples to better understand where LLM successes and fails. We can find that Black-box LLM ChatGPT achieves human-comparable or even better query rewriting quality. Sometimes ChatGPT is too smart and will answer questions redundantly (e.g., Table 7) or refuse to rewrite uncertain questions (e.g., Table 9). For white-box LLMs, the effect of query rewriting continues to improve as the LLM parameters increase. However, T0pp may not have been sufficiently fine-tuned with instruction data, so it is difficult to follow our instructions to generate rewrites. It is worth mentioning that Vicuna which is an open-source chatbot trained by fine-tuning LLaMA achieves the closest effect to ChatGPT.\n\nFurthermore, our work, as the first stage of retrieval-augmented methods, aims to better retrieve supported passages based on the current conversations. Because the retrieved passages can reduce the hallucinations and bias of LLMs, **we think that our work is a step towards improvement in these areas**. We agree the controllable generation of reformulated queries is very important, we leave it in future work.\n\n> **Response to Rejection 2**: There is no detailed analysis on how the unsupervised training objectives affect the retriever representations and no comparison with query reformulation methods. Besides, the cost and the latency of using LLM should also be indicated.\n\nThanks for your valuable suggestion! We believe that **our unsupervised training objectives will lead to better representations of questions, especially topic-switching questions**. To analysis how the unsupervised training objectives affect the retriever representations, we define three question types, *first*, *no-switch*, and *switch* following the previous study [1]. The *first* question is literally first question of conversation without any history. The *no-switch* and *switch* questions can be distinguished by whether $d_{t}$ contains similar or same topics as $d_{t+1}$, where the $d_{t}$ is a gold passage at turn $t$ and $t > 1$. We conduct experiments on the topic-switching dataset TopiOCQA to analysis how the quality of LLM-generated supervision signals varies across different types of questions. As shown in **Table 1**, we find that **powerful ad-hoc retrievers like Contriever can solve the *first* problem very well, but can hardly handle the *switch* question**. **After the guidance of our InstructoR, Contriever achieves the most significant performance improvement for the *switch* problem**, indicating that LLM-generated supervision signals can help the retriever understand the complex conversation session and discover the user’s query intent in topic-switching scenarios. We think your suggestion is well, and we will add it in the revised version.\n\n**Table 1:** The performance of different question types (All, First, No-switch and Switch).\n\n| Retriever                             | MRR (All) | NDCG@3 (All) | R@10 (All) | R@100 (All) | MRR (First) | NDCG@3 (First) | R@10 (First) | R@100 (First) | MRR (No-switch) | NDCG@3 (No-switch) | R@10 (No-switch) | R@100 (No-switch) | MRR (Switch) | NDCG@3 (Switch) | R@10 (Switch) | R@100 (Switch) |\n| ------------------------------------- | :-------: | :----------: | :--------: | :---------: | :---------: | :------------: | :----------: | :-----------: | :-------------: | :----------------: | :--------------: | :---------------: | :----------: | :-------------: | :-----------: | :------------: |\n| Contriever                            |    6.9    |     5.3      |    14.5    |    37.5     |     8.4     |      5.1       |     23.4     |     60.5      |       8.6       |        6.8         |       17.9       |       42.9        |     2.2      |       1.6       |      3.6      |      17.1      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG  |   16.4    |     13.7     |    34.0    |    67.5     |    17.7     |      14.7      |     39.5     |     80.5      |      19.3       |        16.8        |       38.5       |       71.2        |     8.8      |       5.8       |     21.6      |      54.2      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV  |   18.6    |     15.9     |    37.7    |    71.8     |    19.0     |      15.9      |     43.4     |     81.0      |      21.7       |        18.8        |       42.6       |       75.0        |     11.2     |       8.7       |     24.0      |      61.0      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG  |   27.4    |     25.2     |    51.3    |    78.1     |    27.2     |      25.7      |     55.6     |     85.4      |      27.7       |        25.2        |       52.8       |       79.1        |     20.4     |      17.9       |     41.4      |      71.8      |\n| Contriever-msmarco                    |   16.2    |     14.3     |    29.4    |    56.0     |    36.2     |      33.2      |     61.0     |     **89.3**      |      17.9       |        15.7        |       32.3       |       60.7        |     6.1      |       4.9       |     12.4      |      34.1      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG  |   23.9    |     21.6     |    44.4    |    73.7     |    38.3     |      35.5      |    **66.3**     |     **89.3**      |      27.0       |        25.1        |       47.7       |       75.1        |     15.5     |      13.7       |     28.9      |      60.5      |\n| &nbsp;&nbsp;&nbsp;w/ InstructorR QRLV |   28.6    |     26.0     |    52.3    |    80.2     |    34.8     |      33.2      |     63.9     |     84.9      |      31.1       |        28.5        |       56.2       |       84.1        |     19.2     |      16.9       |     36.7      |      65.9      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG  |   **38.5**    |     **37.0**     |    **62.1**    |    **83.2**     |    **46.4**     |      **45.5**      |     65.9     |     87.8      |      **39.5**       |       **37.7**        |       **64.0**       |       **84.6**        |     **33.5**     |      **32.5**       |     **56.0**      |      **78.4**      |\n\nBesides, we also add more comparison with latest query reformulation methods in **Response to Question 4**.\n\nAccording to your suggestion, we discuss the cost and the latency of using LLM in **Response to Question 2**.\n\n[1] Saving Dense Retriever from Shortcut Dependency in Conversational Search, Kim et al. 2022.\n\n> **Response to Rejection 3**: Lack of control of injecting noise (bring by the generated text from LLM) and the analysis of how these expansion terms will influence the retrieval results. This is important in terms of the query analysis and avoid the harmful terms generation.\n\n**We would like to first clarify that our method mostly uses LLM to measure the relevance between input and output, and does not really need to generate text**. We try to avoid the bias and noise brought by the generated text as much as possible. Instead, we use the more transparent LLM generation probability as the retrieval guide. We only use LLM to generate query rewrites as the training data in the first step of QRLV strategy. Besides, **to ensure rewriting quality, we use regular expressions to filter out those rewrites that follow our instructions incorrectly as mentioned in Appendix D**. \n\n> **Response to Question 1**: Can this framework designed without LLM? (i.e. How should we think is this method flexible/feasible or not) and how can we ensure the reproduction as the current LLM normally with fast version iteration. If not, is it still a flexible framework?\n\nYes, **our proposed framework is flexible and scalable**. In Appendix F, we examine the effect of LLM sizes on retrieval performance improvements. We find that using a small language model such as Flan-T5 base (250M) can also achieve good results. In Section 5.4, we analysis the selection of different types of LLM and find that our method is robust, with various LLMs leading to consistent improvements.\n\nTo ensure the reproduction, we adopt the open-source models (e.g., Flan-T5) as the white-box LLMs. Considering that ChatGPT has a fast version iteration, we will release our generated rewrites for further research. **We believe the value of our work is to demonstrate the potential of utilizing LLMs to improve conversational dense retrieval in an unsupervised manner**.\n\n> **Response to Question 2**: What is the cost and the latency of using LLM, especially on the two big QReCC and TopiOCQA datasets? Please show statistic information.\n\nThanks for your valuable comments! **We first analyze the cost of our method from memory and compute costs**.\n\n- Memory cost: The main memory cost of our method consists of three components: the passage index, the retriever, and the LLM. We compute the memory cost as follows: $C_{memory}=d\\times|D|+2\\times\\theta_{retriever}+\\theta_{LLM}$, where $d=768$ is the dimension of the session/passage embeddings, $|D|$ is the total number of passages,  $\\theta_{retriever}=110M$ is the parameter size of the retriever, and $\\theta_{LLM}=\\{80M,250M,780M,3B\\}$ is the parameter size of the LLM. As with most retrieval methods, our memory cost is mainly in $d\\times|D|$. We take Flan-T5 XL ($\\theta_{LLM}=3B$ ) as an example and provide memory consumption on QReCC and TopiOCQA. As shown in **Table 2**, our method is friendly to most university labs, which **can easily run on a single machine with 2-4 GPUs**.\n\n**Table 2:** The memory cost on QReCC and TopiOCQA.\n\n|             | **QReCC** | **TopiOCQA** |\n| :---------- | :-------: | :----------: |\n| Memory Cost |  44.69B   |    22.42B    |\n| GPU Cost    |  4*A6000  |   2*A6000    |\n\nCompute cost: The main compute cost of our method is concentrated at the training stage. We need to emphasize that our method only needs to use LLMs during the training stage, so **InstructoR is computationally efficient during the inference phase**. We use the FLOPs-per-token estimates to calculate the compute cost: $C_{compute}=l_{session}\\times\\theta_{retriever}+|D|\\times(d+d-1)+K\\times(l_{session}+l_{passage})\\times\\theta_{LLM}$, where $l_{session}=200$ is the number of tokens per session, $l_{passage}=100$ is the number of tokens per passage, and $K=\\{8,16,32\\}$ is the number of retrieved top-K passages. We provide the specific compute cost of our method in **Table 3**. In our implementation, our method requires only a few hours for training. We also report the cost of calling ChatGPT API in our paper (line 480), which only costs $5 to generate 10,000 query rewrites.\n\n**Table 3:** The compute cost on QReCC and TopiOCQA.\n\n|                  | **QReCC** | **TopiOCQA** |\n| ---------------- | :-------: | :----------: |\n| FLOPs Per Sample | 2.890*e13 |  2.886*e13   |\n| Compute Cost     | 1.835*e18 |  1.311*e18   |\n\nTo further reduce the memory and compute costs, we can use cheaper inference (e.g., LLM.fp16 and LLM.int8) and smaller LLM (e.g., Flan-T5 base and Flan-T5 large). We also examine the effect of LLM sizes on retrieval performance improvements in Appendix F. We find that using a small language model such as Flan-T5 base (250M) can also achieve good results.\n\n**We then discuss the latency of our method**. Our method only needs to distill knowledge from LLM into the retriever at the training stage, and does not need to use LLM at the inference stage. Hence, **our method is efficient in inference and has the same low latency as other dense retrieval methods**. \n\n> **Response to Question 3**: How can we control the generated text by LLM will not bring noise to the query encoder training and how can we analysis via experiments? In other words, how to evaluate the quality of the generated text by LLM for IR evaluation.\n\nPlease refer to **Response to Rejection 3**.\n\n> **Response to Question 4**: How to consider the comparison with the manually rewritten query and the query reformulation approaches, especially on the cast datasets.\n\nThanks for your advice! We add the comparison with the manually rewritten query and the query reformulation approaches on TopiOCQA and CAsT datasets. To evaluate the effectiveness of our method, we compare it with several latest query reformulation baselines, including: \n\n- T5QR [2]: A strong T5-based QR model.\n- CONQRR [3]: A reinforcement-learning framework for T5-based QR model.\n- ConvGQR [4]: The current SOTA query reformulation method which integrates query rewriting and query expansion toward generating more effective search queries through a new knowledge infusion mechanism.\n- LLM Rewrite: A zero-shot ChatGPT-based model.\n- Human-Rewritten: Using manually rewritten query for retrieval.\n\nExperimental results validate the effectiveness of all three proposed instructing strategies on all the four datasets. **Our method surpasses the current supervised state-of-the-art query reformulation method ConvGQR**. Furthermore, **our method can achieve comparable or even better performance than Human-Rewritten**.\n\n**Table 4:** The performance on QReCC.\n\n| **Method**                           | **MRR**  | **NDCG@3** | **R@10** | **R@100** |\n| ------------------------------------ | :------: | :--------: | :------: | :-------: |\n| T5QR + ANCE                          |   34.5   |    31.8    |   53.1   |   72.8    |\n| T5QR + BM25                          |   33.4   |    30.2    |   53.8   |   86.1    |\n| CONQRR + ANCE                        |   41.8   |     -      |   65.1   |   84.7    |\n| CONQRR + BM25                        |   38.3   |     -      |   60.1   |   88.9    |\n| ConvGQR + ANCE                       |   42.0   |    39.1    |   63.5   |   81.8    |\n| ConvGQR + BM25                       |   44.1   |    41.0    |   54.4   |   88.0    |\n| LLM Rewrite + Contriever             |   19.7   |    17.0    |   33.6   |   56.4    |\n| LLM Rewrite + Contriever-msmarco     |   33.8   |    30.9    |   52.5   |   70.4    |\n| Human-Rewritten + ANCE               |   38.4   |    35.6    |   58.6   |   78.1    |\n| Human-Rewritten + BM25               |   39.7   |    36.2    |   62.5   | **98.5**  |\n| ANCE                                 |   41.4   |    38.9    |   61.2   |   74.7    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG |   42.0   |    39.1    |   64.6   |   83.0    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV |   42.7   |    40.2    |   63.4   |   78.4    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG |   43.5   |    40.5    |   66.7   |   85.6    |\n| Contriever-msmarco                   |   47.6   |    45.0    |   70.5   |   88.9    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG |   50.0   |    47.2    |   74.9   |   92.1    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV |   44.1   |    40.7    |   70.5   |   92.0    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG | **51.9** |  **49.1**  | **76.4** | **92.4**  |\n\n**Table 5:** The performance on TopiOCQA.\n\n| **Method**                           | **MRR**  | **NDCG@3** | **R@10** | **R@100** |\n| ------------------------------------ | :------: | :--------: | :------: | :-------: |\n| T5QR + ANCE                          |   23.0   |    22.2    |   37.6   |   54.4    |\n| T5QR + BM25                          |   11.3   |    9.8     |   22.1   |   44.7    |\n| CONQRR + ANCE                        |    -     |     -      |    -     |     -     |\n| CONQRR + BM25                        |    -     |     -      |    -     |     -     |\n| ConvGQR + ANCE                       |   25.6   |    24.3    |   41.8   |   58.8    |\n| ConvGQR + BM25                       |   12.4   |    10.7    |   23.8   |   45.6    |\n| LLM Rewrite + Contriever             |   12.4   |    10.3    |   26.2   |   52.4    |\n| LLM Rewrite + Contriever-msmarco     |   30.0   |    28.3    |   51.3   |   72.3    |\n| Human-Rewritten + ANCE               |    -     |     -      |    -     |     -     |\n| Human-Rewritten + BM25               |    -     |     -      |    -     |     -     |\n| ANCE                                 |   11.6   |    10.2    |   21.8   |   40.1    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG |   18.2   |    16.8    |   33.3   |   56.4    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV |   17.9   |    15.8    |   34.8   |   59.7    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG |   25.3   |    23.7    |   45.1   |   69.0    |\n| Contriever-msmarco                   |   16.2   |    14.3    |   29.4   |   56.0    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG |   23.9   |    21.6    |   44.4   |   73.7    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV |   28.6   |    26.0    |   52.3   |   80.2    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG | **38.5** |  **37.0**  | **62.1** | **83.2**  |\n\n**Table 6:** The performance on CAsT-19.\n\n| **Method**                       | **MRR**  | **NDCG@3** | **R@10** | **R@100** |\n| -------------------------------- | :------: | :--------: | :------: | :-------: |\n| T5QR                             |   70.1   |    41.7    |    -     |     -     |\n| ConvGQR                          |   70.8   |    43.4    |    -     |     -     |\n| LLM Rewrite + Contriever         |   40.6   |    24.5    |   6.4    |   26.1    |\n| LLM Rewrite + Contriever-msmarco |   63.6   |    48.2    |   12.2   |   43.8    |\n| Human-Rewritten                  | **74.0** |    46.1    |    -     |     -     |\n| Contriever                       |   25.2   |    12.7    |   3.1    |   13.8    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR  |   57.7   |    39.8    |   9.0    |   33.9    |\n| Contriever-msmarco               |   63.6   |    46.5    |   10.8   |   41.3    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR  | **70.4** |  **55.1**  | **13.2** | **45.4**  |\n\n**Table 7:** The performance on CAsT-20.\n\n| **Method**                       | **MRR**  | **NDCG@3** | **R@10** | **R@100** |\n| -------------------------------- | :------: | :--------: | :------: | :-------: |\n| T5QR                             |   42.3   |    29.9    |    -     |     -     |\n| ConvGQR                          |   46.5   |    **33.1**    |    -     |     -     |\n| LLM Rewrite + Contriever         |   21.7   |    12.0    |   7.5    |   25.1    |\n| LLM Rewrite + Contriever-msmarco |   39.6   |    25.7    |   14.6   |   43.5    |\n| Human-Rewritten                  | **59.1** |  **42.2**  |    -     |     -     |\n| Contriever                       |   18.8   |    9.3     |   8.2    |   25.2    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR  |   36.9   |    22.5    |   14.0   |   41.5    |\n| Contriever-msmarco               |   27.3   |    15.9    |   12.4   |   39.4    |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR  | **51.7** |  32.8  | **18.7** | **54.0**  |\n\n[2] Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models, Wu et al. 2020.\n\n[3] CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning, Wu et al. 2022.\n\n[4] ConvGQR: Generative Query Reformulation for Conversational Search, Mo et al. 2023.\n\n> **Response to Question 5**: For Eq. 7, why we can assume $\\log p(z)$, $\\log p(c)$ and $\\log p(q \\mid  c)$ as instants and what is the relation between Eq. 7 and the quality of the generated text?\n\nWe are sorry for making the confusions. In this paper, we assume that the passage prior $p(z)$ is uniform for all $z \\in \\mathcal{Z}$, and the conversation context prior $p(c)$ is uniform for all $c \\in \\mathcal{C}$. Since $c$ and $q$ happen at the same time, $p(q\\mid c)=1$. Hence, we assume $\\log p(z)$, $\\log p(c)$ and $\\log p(q \\mid  c)$ as constants. Except for $\\log p(r\\mid I_{c,q}^{r},c,q)$ in Eq. 8, we use black-box LLMs to generate rewrites. We usually use LLMs to measure the relevance between input and output, not really generate text.\n\n> **Missing References**: The query reformulation methods could be included to be discussed and some recent publications could be added to the related work later to make the literature review more complete.\n\nWe really appreciate your suggestions on the details of our paper writing! Based on your suggestions, we now add the discussions of missing references [2, 3, 4, 5, 6, 7, 8] in the related work section. We also compare our method with the latest query reformulation methods in the experiment section.\n\n[5] Open-retrieval Conversational Question Answering, Qu et al. 2020.\n\n[6] Explicit Query Rewriting for Conversational Dense Retrieval, Qian et al. 2022.\n\n[7] Learning to Relate to Previous Turns in Conversational Search, Mo et al. 2023.\n\n[8] CoSPLADE: Contextualizing SPLADE for Conversational Information Retrieval, Le et al. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Addresses an important practical problem that lacking of conversational retrieval training data and proposes a creative method for leveraging large language models to provide training signal without any data annotation.\n\n2. Thorough and rigorous experiments across diverse settings like low-resource and zero-shot. Surpassing supervised methods is a notable result.\n\nOverall, the novel unsupervised learning approach, thorough experiments, and analyses make this a valuable contribution.", "reasons_to_reject": "1. The approach relies heavily on large proprietary language models, which have limitations like bias and lack of transparency and there are limited error analysis to understand cases where the method fails. This could be important since the LLM is used as a black-box model and the quality of generated results are without guarantee. Meanwhile, the workflow details (e.g. the concrete prompt and the reformulated query) for the proposed three strategies should be much clearer.\n\n2. There is no detailed analysis on how the unsupervised training objectives affect the retriever representations and no comparison with query reformulation methods. Besides, the cost and the latency of using LLM should also be indicated.\n\n3. Lack of control of injecting noise (bring by the generated text from LLM) and the analysis of how these expansion terms will influence the retrieval results. This is important in terms of the query analysis and avoid the harmful terms generation.\n\nThe main risks relate to overselling the generality of the approach and glossing over potential issues with language models. But the paper seems reasonably cautious about limitations. Overall the methodology appears solid despite some aspects needing deeper analysis.", "questions_to_authors": "1. Can this framework designed without LLM? (i.e. How should we think is this method flexible/feasible or not) and how can we ensure the reproduction as the current LLM normally with fast version iteration. If not, is it still a flexible framework?\n\n2. What is the cost and the latency of using LLM, especially on the two big QReCC and TopiOCQA datasets? Please show statistic information.\n\n3. How can we control the generated text by LLM will not bring noise to the query encoder training and how can we analysis via experiments? In other words, how to evaluate the quality of the generated text by LLM for IR evaluation. \n\n4. How to consider the comparison with the manually rewritten query and the query reformulation approaches, especially on the cast datasets.\n\n5. For Eq. 7, why we can assume log p(z), log p(c) and log p(q | c) as instants and what is the relation between eq. 7 and the quality of the generated text?", "paper_topic_and_main_contributions": "This paper proposes a new method called INSTRUCTOR for training conversational dense retrievers in an unsupervised manner, without needing labeled session-passage pairs. The key problem it addresses is the lack of training data for conversational retrieval, where annotating session-passage relevance is difficult. Most existing methods rely on supervised fine-tuning.\n\nThe main contributions are:\n\n1. An unsupervised training framework where large language models estimate session-passage relevance scores to guide retriever training.\n\n2. Three strategies to more accurately calculate relevance using the language model: modeling retrieval as conversation generation, question rewriting as a latent variable, and using question responses as a posterior guide.\n\n3. Experiments showing INSTRUCTOR significantly improves various ad-hoc retrievers like DPR and ANCE, even surpassing supervised methods.\n\nIn summary, the paper proposes a novel unsupervised approach for training conversational retrievers by instructing them with large language models. The method and thorough experiments are the key contributions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Open-retrieval conversational question answering. Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W Bruce Croft, Mohit Iyyer. (SIGIR 2020)\n\n2. Explicit Query Rewriting for Conversational Dense Retrieval. Hongjin Qian, Zhicheng Dou. (EMNLP 2022)\n\n3. ConvGQR: Generative Query Reformulation for Conversational Search. Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, Jian-Yun Nie (ACL 2023)\n\n4. Learning to Relate to Previous Turns in Conversational Search. Fengran Mo, Jian-Yun Nie, Kaiyu Huang, Kelong Mao, Yutao Zhu, Peng Li, Yang Liu (SIGKDD 2023)\n\n5. CoSPLADE: Contextualizing SPLADE for Conversational Information Retrieval. Nam Hai Le, Thomas Gerald, Thibault Formal, Jian-Yun Nie, Benjamin Piwowarski, Laure Soulier. (ECIR 2023)\n\nThe query reformulation methods could be included to be discussed and some recent publications could be added to the related work later to make the literature review more complete."}}, {"review_id": "tbXSUJZoMg", "reviewer": "Reviewer_nZaB", "summary": "", "questions": "Q1: How do you compute the confidence $\\log p(r|I^{r}_{c,q},c,q)$ by ChatGPT? It seems that it cannot return the generation probability through API. Even if it is text-davinci, it can only provide the probability of the first five tokens. How do you deal with this problem?\n\nQ2: The proposed method is applied to several ad-hoc dense retriever. Is it also possible to apply it for conversational dense retrieval model?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews. Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! \n\n> **Response to Rejection 1**: Distilling knowledge from LLM to retriever, or more generally, distilling knowledge from reranker to retriever is not a new idea [1, 2].\n\n**We first clarify the differences and advantages between our InstructoR and these methods that also use LLMs for retrieval**. \n\n- Both RankGPT [1] and PRP [2] are very inspiring work, which use LLM to rank top-K passages at the reranking stage. However, it is difficult to handle millions of passages at the retrieval stage. Compared with these methods, **our method is more efficient with less latency**. Because our method only needs to distill knowledge from LLM into the retriever at the training stage, and does not need to use LLM at the inference stage.\n- Besides, there are some interesting work [3, 4] that leverage LLM as dataset generator, and train task-specific retrievers based on the generated data. Different from these outstanding work, our method does not need to use LLM to generate training data, but unleashes the power of LLM to judge the relevance between sessions and passages. **We try to avoid the bias and noise brought by the generated text as much as possible**.\n- To the best of our knowledge, **this is the first attempt to utilize LLMs to improve conversational dense retrieval in an unsupervised manner**. Specially, we devise three instructing strategies from context, query and response perspectives in the conversation. All three strategies are tailored for conversational dense retrieval. Furthermore, our method provides a feasible idea to combine white-box LLMs and black-box LLMs for distilling knowledge from LLM to retriever. \n\n[1] Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent, Sun et al. 2023.\n\n[2] Large language models are effective text rankers with pairwise ranking prompting, Qin et al., 2023.\n\n[3] Promptagator: Few-shot Dense Retrieval From 8 Examples, Dai et al. 2022.\n\n[4] InPars: Data Augmentation for Information Retrieval using Large Language Models, Luiz et al., 2022.\n\n> **Response to Rejection 2**: Missing some recent literature in the related work part.\n\nWe really appreciate your suggestions on the details of our paper writing! Based on your suggestions, we now add the discussions of missing references [5, 6, 7] in the related work section. We also compare our method with the latest query reformulation methods (e.g., ConvGQR) in the experiment section.\n\n[5] Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search, Mao et al., 2023.\n\n[6] ConvGQR: Generative Query Reformulation for Conversational Search, Mo et al., 2023.\n\n[7] Learning to Relate to Previous Turns in Conversational Search, Mo et al., 2023.\n\n> **Response to Question 1**: How do you compute the confidence $\\log p(r\\mid I_{c,q}^{r},c,q)$ by ChatGPT? It seems that it cannot return the generation probability through API. Even if it is text-davinci, it can only provide the probability of the first five tokens. How do you deal with this problem?\n\nWe are sorry for making the confusions. In practical implementation, we first prompt ChatGPT to generate the query rewrites, then use Flan-T5 to compute the confidence $\\log p(r\\mid I_{c,q}^{r},c,q)$. We find that what plays a key role in Eq. 8 is $\\log p(r \\mid I_{z}^{r}, z)$.\n\n> **Response to Question 2**: The proposed method is applied to several ad-hoc dense retriever. Is it also possible to apply it for conversational dense retrieval model?\n\nYes, **our method can also be applied to conversational dense retrievers**. **To be consistent with previous work and achieve a fair comparison, we use the widely used ad-hoc retrievers as the initialization**. Experimental results show InstructoR can bring significant improvements across various ad-hoc retrievers, even surpassing the supervised state-of-the-art conversational dense retrieval method. In addition, we consider the more realistic setting where session-passage pairs are not available, so we train an ad-hoc retriever in an unsupervised way to obtain a conversational retriever.\n\n> **Missing References**\n\nWe really appreciate your suggestions on the details of our paper writing! Based on your suggestions, we now add the discussions of missing references [1, 2, 3, 4, 5, 6, 7] in the related work section. We also compare our method with the latest query reformulation methods in the experiment section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method leverages LLMs for knowledge distillation, which is a good application of LLMs on real research.\n2. The experiments are sufficient and solid. The results show significant improvement over existing studies.\n3. The paper is well-written and easy to understand.\n", "reasons_to_reject": "1. Distilling knowledge from LLM to retriever, or more generally, distilling knowledge from reranker to retriever is not a new idea [1, 2]. \n2. Missing some recent literature in the related work part.\n\nReference:\n\n1. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent, Sun et al. 2023\n\n2. Large language models are effective text rankers with pairwise ranking prompting, Qin et al., 2023\n", "questions_to_authors": "Q1: How do you compute the confidence $\\log p(r|I^{r}_{c,q},c,q)$ by ChatGPT? It seems that it cannot return the generation probability through API. Even if it is text-davinci, it can only provide the probability of the first five tokens. How do you deal with this problem?\n\nQ2: The proposed method is applied to several ad-hoc dense retriever. Is it also possible to apply it for conversational dense retrieval model?\n", "paper_topic_and_main_contributions": "This paper proposes a new unsupervised method “Instructor” for training conversational dense retrieval models. Instructor leverages the knowledge from large language models (LLMs) to generate supervision signals to instruct the retriever. To distill the knowledge from the LLMs to the retriever, the authors design three methods to calculate the relevance score more precisely. Sufficient experimental results on four datasets under various settings demonstrate the effectiveness of the proposed method. Overall, the paper is well-written and easy to understand. The experiments look very solid. Though the technical depth is not so deep, and the distillation idea is not very novel, I suggest a weak accept on this paper.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search, Mao et al., 2023\n\n[2] ConvGQR: Generative Query Reformulation for Conversational Search, Mo et al., 2023\n\n[3] Learning to Relate to Previous Turns in Conversational Search, Mo et al., 2023\n"}}, {"review_id": "1YqlJFHjZM", "reviewer": "Reviewer_3sLe", "summary": "", "questions": "Can you provide more analysis into how the quality of LLM-generated supervision signals varies across different types of conversations or questions? Any patterns in where it works better/worse?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews. Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! \n\n> **Response to Rejection 1**: Limited analysis on how different types of conversations/questions affect the quality of LLM supervision.\n\nThanks for your valuable advice! **We conduct analysis on how different types of conversations/questions affect the quality of LLM supervision**. Following the previous study [1], we define three question types, *first*, *no-switch*, and *switch*. The *first* question is literally first question of conversation without any history. The *no-switch* and *switch* questions can be distinguished by whether $d_{t}$ contains similar or same topics as $d_{t+1}$, where the $d_{t}$ is a gold passage at turn $t$ and $t > 1$. We conduct experiments on the topic-switching dataset TopiOCQA to analysis how the quality of LLM-generated supervision signals varies across different types of questions. As shown in **Table 1**, we find that **powerful ad-hoc retrievers like Contriever can solve the *first* problem very well, but can hardly handle the *switch* question**. **After the guidance of our InstructoR, Contriever achieves the most significant performance improvement for the *switch* problem**, indicating that LLM-generated supervision signals can help the retriever understand the complex conversation session and discover the user’s query intent in topic-switching scenarios. We think your suggestion is well, and we will add it in the revised version.\n\n**Table 1:** The performance of different question types (All, First, No-switch and Switch).\n\n| Retriever          | MRR (All) | NDCG@3 (All) | R@10 (All) | R@100 (All) | MRR (First) | NDCG@3 (First) | R@10 (First) | R@100 (First) | MRR (No-switch) | NDCG@3 (No-switch) | R@10 (No-switch) | R@100 (No-switch) | MRR (Switch) | NDCG@3 (Switch) | R@10 (Switch) | R@100 (Switch) |\n| ------------------ | :-------: | :----------: | :--------: | :---------: | :---------: | :------------: | :----------: | :-----------: | :-------------: | :----------------: | :--------------: | :---------------: | :----------: | :-------------: | :-----------: | :------------: |\n| Contriever         |    6.9    |     5.3      |    14.5    |    37.5     |     8.4     |      5.1       |     23.4     |     60.5      |       8.6       |        6.8         |       17.9       |       42.9        |     2.2      |       1.6       |      3.6      |      17.1      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG |   16.4    |     13.7     |    34.0    |    67.5     |    17.7     |      14.7      |     39.5     |     80.5      |      19.3       |        16.8        |       38.5       |       71.2        |     8.8      |       5.8       |     21.6      |      54.2      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV |   18.6    |     15.9     |    37.7    |    71.8     |    19.0     |      15.9      |     43.4     |     81.0      |      21.7       |        18.8        |       42.6       |       75.0        |     11.2     |       8.7       |     24.0      |      61.0      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG |   27.4    |     25.2     |    51.3    |    78.1     |    27.2     |      25.7      |     55.6     |     85.4      |      27.7       |        25.2        |       52.8       |       79.1        |     20.4     |      17.9       |     41.4      |      71.8      |\n| Contriever-msmarco |   16.2    |     14.3     |    29.4    |    56.0     |    36.2     |      33.2      |     61.0     |     **89.3**      |      17.9       |        15.7        |       32.3       |       60.7        |     6.1      |       4.9       |     12.4      |      34.1      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR CRCG |   23.9    |     21.6     |    44.4    |    73.7     |    38.3     |      35.5      |     **66.3**     |     **89.3**      |      27.0       |        25.1        |       47.7       |       75.1        |     15.5     |      13.7       |     28.9      |      60.5      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRLV |   28.6    |     26.0     |    52.3    |    80.2     |    34.8     |      33.2      |     63.9     |     84.9      |      31.1       |        28.5        |       56.2       |       84.1        |     19.2     |      16.9       |     36.7      |      65.9      |\n| &nbsp;&nbsp;&nbsp;w/ InstructoR QRPG |   **38.5**    |     **37.0**     |    **62.1**    |    **83.2**     |    **46.4**     |      **45.5**      |     65.9     |     87.8      |      **39.5**       |        **37.7**        |       **64.0**       |       **84.6**        |     **33.5**     |      **32.5**       |     **56.0**      |      **78.4**      |\n\n[1] Saving Dense Retriever from Shortcut Dependency in Conversational Search, Kim et al. 2022.\n\n> **Response to Rejection 2**: Unclear how the approach scales as the dataset size increases. Memory and compute costs?\n\nThanks for your valuable comments! We will add the following discussion in the Appendix.\n\nOur proposed framework is flexible and scalable, and we analyze it from memory and compute costs.\n\n- Memory cost: The main memory cost of our method consists of three components: the passage index, the retriever, and the LLM. We compute the memory cost as follows: $C_{memory}=d\\times|D|+2\\times\\theta_{retriever}+\\theta_{LLM}$, where $d=768$ is the dimension of the session/passage embeddings, $|D|$ is the total number of passages,  $\\theta_{retriever}=110M$ is the parameter size of the retriever, and $\\theta_{LLM}=\\{80M,250M,780M,3B\\}$ is the parameter size of the LLM. As with most retrieval methods, our memory cost is mainly in $d\\times|D|$. We take Flan-T5 XL ($\\theta_{LLM}=3B$ ) as an example and provide memory consumption on QReCC and TopiOCQA. As shown in **Table 2**, our method is friendly to most university labs, which can easily run on a single machine with 2-4 GPUs.\n\n**Table 2:** The memory cost on QReCC and TopiOCQA.\n|             | **QReCC** | **TopiOCQA** |\n| :---------- | :-------: | :----------: |\n| Memory Cost |  44.69B   |    22.42B    |\n| GPU Cost    |  4*A6000  |   2*A6000    |\n\n- Compute cost: The main compute cost of our method is concentrated at the training stage. We need to emphasize that our method only needs to use LLMs during the training stage, so InstructoR is computationally efficient during the inference phase. We use the FLOPs-per-token estimates to calculate the compute cost: $C_{compute}=l_{session}\\times\\theta_{retriever}+|D|\\times(d+d-1)+K\\times(l_{session}+l_{passage})\\times\\theta_{LLM}$, where $l_{session}=200$ is the number of tokens per session, $l_{passage}=100$ is the number of tokens per passage, and $K=\\{8,16,32\\}$ is the number of retrieved top-K passages. We provide the specific compute cost of our method in **Table 3**. In our implementation, our method requires only a few hours for training. We also report the cost of calling ChatGPT API in our paper (line 480), which only costs $5 to generate 10,000 query rewrites.\n\n**Table 3:** The compute cost on QReCC and TopiOCQA.\n|                  | **QReCC** | **TopiOCQA** |\n| ---------------- | :-------: | :----------: |\n| FLOPs Per Sample | 2.890*e13 |  2.886*e13   |\n| Compute Cost     | 1.835*e18 |  1.311*e18   |\n\nTo further reduce the memory and compute costs, we can use cheaper inference (e.g., LLM.fp16 and LLM.int8) and smaller LLM (e.g., Flan-T5 base and Flan-T5 large). We also examine the effect of LLM sizes on retrieval performance improvements in Appendix F. We find that using a small language model such as Flan-T5 base (250M) can also achieve good results.\n\n**We then discuss the latency of our method**. Our method only needs to distill knowledge from LLM into the retriever at the training stage, and does not need to use LLM at the inference stage. Hence, **our method is efficient in inference and has the same low latency as other dense retrieval methods**. \n\n> **Response to Question 1**: Can you provide more analysis into how the quality of LLM-generated supervision signals varies across different types of conversations or questions? Any patterns in where it works better/worse?\n\nPlease refer to **Response to Rejection 1**.\n\n> **Writing Comments**: Typos Grammar Style And Presentation Improvements\n\nWe really appreciate your suggestions on the details of our paper writing! Based on your suggestions, we now revise typos and use sub-section headings for each strategy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tackles an important problem of unsupervised conversational retrieval training. The way using LLMs to provide training signals is a novel in the conversational context (although it has been validated in ad-hoc search), and the three strategies provide nice ways to leverage LLMs from different perspectives of the conversation. This can enable building conversational retrievers without labeled data.\n\n2. Thorough ablation studies analyzing the effect of different proposed strategies.\n\n3. Strong empirical results, surpassing supervised approaches on two datasets.\n\n4. Well-written paper with sufficient details to reproduce the approach.", "reasons_to_reject": "\n1. Limited analysis on how different types of conversations/questions affect the quality of LLM supervision.\n\n2. Unclear how the approach scales as the dataset size increases. Memory and compute costs?\n", "questions_to_authors": "Can you provide more analysis into how the quality of LLM-generated supervision signals varies across different types of conversations or questions? Any patterns in where it works better/worse?", "paper_topic_and_main_contributions": "This paper proposes a novel method called INSTRUCTOR for unsupervised conversational dense retrieval. The key idea is to leverage large language models (LLMs) to provide supervised signals to guide the training of retrievers, without requiring any labeled data. The main contributions are:\n\n1. An unsupervised training framework where LLMs generate soft session-passage relevance scores to instruct the retrievers.\n\n2. Three strategies to accurately estimate relevance from different perspectives: conversational retrieval as conversation generation, question rewrite as latent variable, and question response as posterior guide.\n\nThe paper addresses the lack of labeled data for conversational retrieval, and demonstrates how large pretrained models can be leveraged for unsupervised learning in this setting.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 143: \"discover\" -> \"discovering\"\n\nLine 289: Extra space before period\n\nSection 3.2: Consider using sub-section headings for each strategy\n\nFigure 1: Increase font size of text in the boxes\n\nFigure 3: Improve spacing around \"+\" symbols"}}]

Author rebuttal:

Meta-review: This paper proposes an alternative way to use LLMs for conversational IR. It generates relevance scores for candidate passages, and use the scores to instruct dense retrievers. Although there is some similarity between this method and the previous utilizations of LLM for generating supervision signals for dense retrievers, this study provides an alternative and interesting method.

The authors have answered most of the questions of the reviewers in their rebuttal. Overall, this is a valuable piece of work that enriches the current literature on conversational IR.
Decision: Accept-Findings