STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents (2024)

Abstract: Equipping a conversational search engine with strategies regarding when to ask clarification questions is becoming increasingly important across various domains. Attributing to the context understanding capability of LLMs and their access to domain-specific sources of knowledge, LLM-based clarification strategies feature rapid transfer to various domains in a posthoc manner. However, they still struggle to deliver promising performance on unseen domains, struggling to achieve effective domain transferability. We take the first step to investigate this issue and existing methods tend to produce one-size-fits-all strategies across diverse domains, limiting their search effectiveness. In response, we introduce a novel method, called STYLE, to achieve effective domain transferability. Our experimental results indicate that STYLE bears strong domain transferability, resulting in an average search performance improvement of ∼10% on four unseen domains.

heading: 1 Introduction
text: Recent research in conversational search systems has highlighted the potential of utilizing large language models (LLMs) to address ambiguities present in user queries (Kuhn et al., 2022; Zhang and Choi, 2023). A key focus has been on investigating the strategies regarding when to ask clarification questions during conversations (Aliannejadi et al., 2021). When developing a search system for a specific domain without prior training, its clarification strategy may be limited, since ambiguities are influenced by the domain-specific background knowledge (Ferrari et al., 2017). For example, financial terminology may be perceived as ambiguous by the conversational search systems that are not confident with the finance jargon. To address
* Corresponding author.
this, recent studies (Zhang and Choi, 2023; Zhang et al., 2023) resort to LLMs to feature rapid transfer to various domains in a post-hoc manner, such as finance services (Deng et al., 2023a) and movie recommendation (Fan et al., 2023). This may be attributed to the context-understanding capability of LLMs and their access to domain-specific sources of knowledge (Zhang et al., 2023). However, empirical evidence suggests that the effectiveness of LLM-based methods remains suboptimal in unseen domains, with notable difficulties in achieving effective domain transferability (Deng et al., 2023b). This motivates us to rethink: What impedes the domain transferability of LLM-based methods regarding the clarification strategy?
To investigate this, we conduct an in-depth analysis to examine off-the-shelf LLM-based methods (detailed in Section 2). Our findings reveal that while LLM-based methods, armed with domainspecific knowledge sources, somewhat develop strategies that could work on unseen domains, they often produce a one-size-fits-all strategy, such as asking increasingly more questions as the conversation advances. Such strategies typically perform less effectively compared to tailored strategies trained on domain-specific data. This highlights the limited ability of LLM-based methods to adapt to unseen domains effectively. The challenge lies in the fact that strong domain transferability can not be achieved by training on single-domain data. Meanwhile, the mismatched distribution of domainspecific representations poses a significant obstacle to effective domain transfer. Affected by the above challenges, existing methods are accustomed to addressing ambiguities solely in a single domain (Rahmani et al., 2023; Aliannejadi et al., 2019), potentially leading to models becoming brittle when faced with unseen domains.
To this end, we propose a novel method,
10633
called STYLE, which features rapid tranSfer To previouslY unseen domains via tailored stratEgies in a post-hoc manner. STYLE comprises the domain-invariant strategy planner (DISP) in the conversational search engine and multi-domain training (MDT) paradigm. Specifically, the DISP is configured to extract domain-invariant information, mitigating the mismatch in the distribution of domain-specific representations and ensuring robustness across domains. We further leverage a retrieval-augmented paradigm to obtain documents that match the user queries in conjunction with their matching scores. The matching score reflects the retrieval quality and confidence of the retrieval module, which is highly correlated to the ambiguous level of the user queries and avoids the introduction of domain-specific semantic representation. Moreover, instead of relying on one solitary domain, MDT encourages the domain transferability of DISP by training it across multiple diverse domains. This is inspired by the population-based training (Long et al., 2023), which suggests that the generalization of a collaborative agent to held-out populations can be improved by training larger and more diverse populations (Charakorn et al., 2020). As such, STYLE enhances domain transferability and adapts its strategies for various domains.
We conduct experiments to evaluate STYLE using four domain-specific benchmark datasets in conversational search, including e-commerce, movies, and books. Our findings demonstrate that STYLE consistently surpasses existing LLM-based baselines in terms of domain transferability, resulting in a significant average performance improvement of ∼10%. Further analysis reveals that STYLE tailors its strategies to diversify them in different domains, which laid the foundation for its effectiveness. In summary, we analyze and address
the inadequacies of existing methods in determining when to ask clarification questions in scenarios involving previously unseen domains. This paves the way to advance both practical applications and academic research in this area. We present three key contributions:
• We verify and highlight that the one-size-fits-all strategies impedes the domain transferability of existing LLM-based methods when deciding on the timing to pose clarification questions.
• We present a new method STYLE to improve domain transferability in a post-hoc manner. It includes a domain-invariant strategy planner (DISP) in the search engine and a multi-domain training (MDT) paradigm.
• We experimentally show that STYLE bears strong domain transferability, resulting in an average search performance improvement of ∼10% on four unseen domains.

heading: 2 Preliminary Analysis
text: Previous analyses indicate that current methods’ effectiveness of clarification strategies is compromised when applied in unseen domains (Deng et al., 2023a), yet the underlying cause remains unclear. This section aims to investigate the factors that may hinder the domain transferability of LLM-based methods. Our analysis involves two factors: search performance, and strategy characteristics.

heading: 2.1 Experimental Setup.
text: Baselines & Datasets. We compare the in-domaintrained model with LLM-based methods, including LLMs with zero-shot, few-shot, and CoT prompts. Here, the in-domain-trained model means the model is well-trained on data from the same domain as the test set, which contains a BERT-based
encoder and the two-layer fully connected network. We consider four domain-specific datasets, detailed in Appendix A. Evaluation metrics. Following previous work (Chen et al., 2023), we evaluate search performance using SR@5. To analyze strategy characteristics, considering the variation of domain knowledge in different search domains, diverse clarification strategies may be required, we introduce strategy diversity as a metric. Specifically, as the strategy module decides whether to ask for clarification during each conversation turn, we analyze the diversity of strategies based on the strategy trajectories, each formed by multi-turn actions. Take LLM with CoT as an example, Figure 2 (C) reveals its strategy trajectories on different domains, represented as tr1, tr2, tr3, tr4. To evaluate its strategy diversity, we calculate the average similarities between pairs of trajectories. A lower similarity score indicates a greater variety of strategies. More information on strategy diversity can be found in Appendix A. It’s important to note that the strategy diversity of the in-domain-trained model is assessed based on trajectories from their variances trained on its own specific domain. Investigation Analysis. As illustrated in Figure 2(A), the in-domain-trained model outperforms LLM-based methods in terms of search performance. Additionally, in-depth analysis in Figure 2 (B)1 reveals that the in-domain-trained model excels in both strategy diversity and search performance, whereas LLM-based methods often adopt a one-size-fits-all strategy with limited diversity across different domains (cf. Section 4.3 for details). Taking Figure 2 (C) for example, LLM with CoT shows a tendency to ask increasingly more questions as the conversation advances. This indi-
1More results are in Appendix A.
cates that the in-domain-trained model has diverse tailored strategies for various domains, while LLMbased methods lack the flexibility to produce diverse strategies tailored to unseen domains.

heading: 3 The Method
text: To achieve effective domain transferability when deciding on when to seek clarification, we propose STYLE. It includes a domain-invariant strategy planner (DISP) and multi-domain training paradigm (MDT). We provide the problem formulation in Section 3.1. The overall architecture of STYLE is introduced in section 3.2, with detailed explanations of DISP and MDT provided in section 3.3 and section 3.4, respectively.

heading: 3.1 Problem Formulation
text: Retrieval-based Conversational Search. We focus on the retrieval setting since it is one of the most common paradigms (Gao et al., 2022). Formally, for a user ui, there exists a document di in the collection D that aligns with the user’s intent. The interaction commences with the user’s initial query q1. At each turn t, when the user presents a query qt, the conversation history Ht = {q1,m1, ..., qt−1,mt−1, qt} is formed, where qt−1 and mt−1 represent the user’s query and the system’s response at turn t− 1. Given Ht, the system initially retrieves a subset of documents Dt ⊂ D. Subsequently, based on Ht and Dt, the system generates a response mt by either posing a clarification question cqt to the user or displaying the top x retrieved documents in Dt. This iterative process continues until the system presents di to the users or reaches the maximum number of turns T . MDP Environment. The conversational search process is often formulated as a Markov Decision Process (MDP) (Chen et al., 2023). At turn t, con-
sidering qt, Ht, and Dt, the system chooses an action at ∈ A from a set of clarification strategies A. The goal is to learn a strategy π that maximizes the expected total rewards across the observed conversation episodes. This is formulated as: π∗ = argmaxπϵΠ E [∑T t=0 r(st, at) ] , where st represents the state comprising Ht and Dt, r(·) is the immediate reward, denoted as rt.

heading: 3.2 Overall Architecture
text: As illustrated in Figure 1, STYLE includes domaininvariant strategy planner (DISP) in the conversational search engine and multi-domain training paradigm (MDT). In Figure 1(b), STYLE initially trains DISP across various domains using MDT. Subsequently, the well-trained DISP can be introduced into unseen domains in a post-hoc manner. During the inference at conversation turn t, as depicted in Figure 1(a), the LLM-based retriever identifies documents Dt that closely match the user query within the conversation context Ht. Following this, based on Ht and Dt, the DISP decides whether to ask the user a clarification question by generating an action at using domain-invariant information. If the action at suggests asking, our conversational search engine utilizes the LLM-based generator to create a clarification question cqt+1 through few-shot CoT, considering the conversation context and the retrieved documents. Otherwise, our search engine presents x retrieved documents to the user.

heading: 3.3 Domain-Invariant Strategy Planner
text: To mitigate the discrepancy in the distribution of domain-specific representations, we propose DISP, implemented by a two-layer fully connected network. DISP is configured to extract domaininvariant representation that is general and structural, thereby enhancing its robustness to the domain transfer. The domain-invariant representation used in DISP is the concatenation of encoded conversation context and retrieved documents, and the ranking scores of retrieval results. This information captures the conversation state, the matching degree between user queries, the domain-specific knowledge from retrieved documents, and the confidence and quality of the retrieval module.
In particular, We utilize a fixed BERT (Devlin et al., 2019) to encode Ht and Dt into representations Ht and Dt, which remain unchanged during the training process. Then, we concatenate Ht and
Dt with the ranking scores score1:kt of k retrieved documents assigned to each document in Dt by the retrieval module since these scores can be indicative of the retrieval results’ relevance or the retrieval system’s confidence while relatively independent from the domain knowledge distributions.
Finally, such domain-invariant information plays the role of the state st and is fed into the DISP to yield action at, formulated below:
value = MLP ( Ht ⊕Dt ⊕ score1:kt ) , (1)
at =
{ ask, value ≥ 0.5
answer, value < 0.5 (2)
When DISP chooses to ask, the search engine generates and provides a clarification question to the user. Otherwise, the engine provides the x retrieved documents with the highest ranking scores as answers to the user.

heading: 3.4 Multi-Domain Training
text: To encourage the domain transferability of DISP, STYLE involves MDT. Inspired by the populationbased training (Long et al., 2023), the MDT trains DISP on multiple domains (e.g., e-commerce, web), which enhances the generalization of the DISP to tailor its strategies to unseen domains.
Taking inspiration from population-based training, we engage in training the DISP using a diverse set of domain datasets. With n distinct domain-specific datasets represented as B = {B1, B2, ...Bn}, we randomly select a subset as the training data for each epoch. This training set is designed to expose the planner to an assortment of strategies relevant to different domains, thereby bolstering its capacity to tailor its strategy in novel scenarios. Upon completion of training, we retain the refined parameters of the planner, allowing it to make efficient inferences on any unseen domain B∗ (B∗ /∈ B) effectively. In MDT, we engage in interactive reinforcement learning using an LLMbased user simulator as described in prior research (Deng et al., 2023b). Each sample includes a user ui seeking a specific document di along with intent
details d∗i . We leverage d ∗ i and role instructions to formulate the user prompt Puser (Details are in Appendix F.3). When the system presents a statement mt+1 to user ui, the user responds with qt+1 as follows:
qt+1 = LLM (Puser (d ∗ i ) ,mt+1, Ht) . (3)
Once we receive the response qt+1, we calculate the reward rt based on predefined criteria. Subsequently, we employ the dueling Q-network for training, which can be expressed as follows, where yt = Q ∗(st, at) and Q∗ indicates DISP in this case.
yt = Est+1 [ rt + γmax
a∈A Q∗(st+1, at+1)|st, at
] .
(4)

heading: 4 Experiments
text: We conduct experiments to evaluate the domain transferability of STYLE and analyze reasons behind its success. We study three research questions:
• RQ1: Can STYLE effectively transfer to the unseen domain without domain-specific training?
• RQ2: Does STYLE produce strategies tailored to different domains?
• RQ3: Why is STYLE effective in dealing with unseen domains?

heading: 4.1 Experimental Settings
text: Domain-specific Datasets. We evaluate STYLE using domain-specific benchmark datasets in conversational search across different domains. Building on prior research (Aliannejadi et al., 2019; Wang and Ai, 2022; Owoicho et al., 2023), we consider four datasets: ClariQ (Aliannejadi et al., 2021) which gathers ambiguous web search queries, FaqAnt (Chen et al., 2023), focusing on a conversational FAQ task in the financial domain, MSDialog (Qu et al., 2018), which records ambiguous queries about Microsoft products, Opendialkg (Moon et al., 2019), which contains ambiguous questions regarding movies and books. Data statistics2 are in Table 1. Importantly, these datasets contain unambiguous queries that are utilized to assess the strategy module’s ability to correctly determine when it is unnecessary to ask questions. To simulate unseen domains, we conduct held-out evaluation by training STYLE on three datasets and reserving one as the unseen domain dataset.
2Data processing details can be found in Appendix C.
Baselines. We compare STYLE with the following two classes of baselines3, which allow us to observe the in-depth insight brought by asking clarification questions. Implementation details for each method can be found in Appendix F.
• Retreval-based conversational search models always provide answers to users without asking clarification questions. This includes 1) BM25: a statistics-based method, 2) senBERT (Reimers and Gurevych, 2019) uses siamese and triplet BERT to encode the input, 3) monoBERT (Nogueira and Cho, 2019): a BERT-based crossencoder re-ranker, and 4) ChatSearch (Sun et al., 2023), a ChatGPT-based retrieval method with the SOTA performance.
• LLM-based methods opt to present either the retrieved documents or clarification questions to the user. This includes 1) ClarSim (Zhang and Choi, 2023) determines when to inquire using uncertainty modeling through self-questioning, 2) CLAM (Kuhn et al., 2022) identifies when to ask and generates questions through few-shot incontext learning, 3) CLAMzeroShot utilizes similar prompt following (Kuhn et al., 2022) except using zero-shot learning, and 4) ProCoT (Deng et al., 2023a) detects ambiguity and generates questions using few-shot CoT.
Evaluation Metrics. Considering that good clarification strategies lead to better performance of the search engines, we mainly focus on evaluating the efficiency and effectiveness of the search engines. Following previous works (Lei et al., 2020; Deng et al., 2023b; Chen et al., 2023), We adopt the Recall@5, average turn (AvgT) and the success rate at turn k (SR@k) as the automatic evaluation metrics.

heading: 4.2 Evaluation on unseen domains (RQ1)
text: We evaluate the domain transferability by assessing the conversational search performance on unseen domains. The results are displayed in Table 2, with a comprehensive analysis provided below.
3Notably, we omit supervised methods (Zhang and Choi, 2023; Aliannejadi et al., 2020) as they fail to transfer to unseen domains in a post-hoc manner.
STYLE exhibits superior performance on unseen domains, showcasing strong domain transferability. As illustrated in Table 2, STYLE achieves the most accurate search results in the fewest turns across unseen domains. On average, it outperforms the leading LLM-based baseline (ProCoT) by a significant ∼10% margin in SR@5 across all domains. With the highest SR@5 scores, STYLE also maintains a lead of over ∼5% in AvgT compared to the baselines in most domains. Furthermore, STYLE performs well even in domains where clarification questions play a less crucial role. Notably, on MSDialog, ChatSearch outperforms all LLM-based methods, suggesting that asking clarification questions may not be essential. Nevertheless, STYLE still surpasses ChatSearch, underscoring its robust transferability in unseen domains.

heading: 4.3 Strategy characterises analysis (RQ2)
text: This section verifies whether STYLE produces tailored strategies necessary for effective transfer to unseen domains. To achieve this, we compare STYLE with its counterpart STYLEinDomain4, which is trained on the corresponding in-domain dataset. Importantly, strategies contained in STYLEinDomain are carefully tailored to the specific domain via customized training. Subsequently, we visualize the strategies of various methods in Figure 4 and evaluate how closely they align with
4We report its performance in Appendix D and Table 6.
STYLEinDomain in Table 3.
LLM-based baselines prefer size-fits-all strategies across various domains. As depicted in Figure 4, the trajectory of strategies for ProCoT remains consistent across various domains, showing a tendency to ask more questions as the conversation progresses. Similarly, CLAM follows a uniform strategy, maintaining a consistent likelihood of asking questions at each turn across all domains. Quantitatively speaking, as shown in Table 3, the strategies of LLM-based baselines exhibit
limited alignment with the in-domain strategies of STYLEinDomain, failing to tailor their strategies to different domains. STYLE produces diverse and tailored strategies to different domains. As shown in Figure 4, STYLE demonstrates the highest level of strategy diversity compared to LLM-based baselines. It also showcases clarification strategies that closely align with those of STYLEinDomain. For instance, on Opendialkg, both STYLE and STYLEinDomain tend to introduce clarification questions early in the conversation, gradually reducing the frequency of asking as the conversation progresses. This trend is consistent across the other datasets as well. The quantitative findings in Table 3 further confirm the superior alignment of STYLE with STYLEinDomain. In conclusion, STYLE customizes strategies to meet diverse requirements across different domains.

heading: 4.4 Characteristics of STYLE (RQ3)
text: In this section, we investigate reasons behind the effective domain transferability of STYLE. To accomplish this, we measure the benefits of ask-
ing clarification questions at each turn (we term it asking benefits). In particular, a good clarification question posed at an appropriate time would assist the search module in retrieving documents. Thus, we calculate the ranking change of target documents after the user answers this clarification question5. The outcomes are presented in Figure 5. The domain transferability of STYLE stems from its diverse strategies tailored to suit different domain needs. As depicted in Figure 5, all methods demonstrate diverse gains from asking questions across various domains. Given the fluctuating asking benefits across domains, an effective approach must adapt its strategy to meet the distinct requirements of each domain. In particular, CLAM sticks to a consistent probability of asking, regardless of the conversation turn. When the asking benefits show a notable decrease after the 2nd turn on MSDialog and FaqAnt, CLAM still performs a consistent probability and fails to adjust its strategy to adapt to the fluctuations of asking benefits. In contrast, STYLE showcases precise control over its strategy, adjusting to fluctuations in gains on a turn-by-turn basis. Notably, its asking benefits decrease gradually on ClariQ, prompting a reduction in the probability of asking questions as the benefits diminish. On MSDialog, where the asking benefits remain notably low (around -20), STYLE strategically limits its asking probability to a minimum level across all turns. Consequently, STYLE
5More details can be found in Appendix B.
exhibits more tailored strategies compared to other baselines, demonstrating better transferability by employing diverse strategies customized to meet specific domain needs. This enables STYLE to maximize the benefits of asking clarification questions in various domains, thereby enhancing its performance in unseen domains.

heading: 4.5 Ablation Study
text: We conduct ablations to ascertain the contribution of each module in STYLE. The results are provided in Table 4. Detailed findings are outlined below. DISP. Table 4, row(a), shows that not including the DISP leads to the biggest decrease. This highlights its significance in STYLE. Training Sources of MDT. A reduction in the variety of datasets employed for training, as shown in rows (b-c) of Table 4, leads to a noteworthy reduction in STYLE’s performance. Considering the diverse origins of these datasets, the results reinforce the necessity of training the STYLE on sufficiently diverse domains to ensure transferability. Domain-invariant Input of DISP. The removal of the input information for the DISP, documented in rows (d-e) of Table 4, provides further insights. Specifically, row (d) reveals that excluding retrieved documents Dt from the input diminishes performance across most domains. Additionally, as shown in row (e), the absence of document scores score1:kt undermines the performance in all tested domains. The retrieval score is domain-invariant information and potentially indicative of both the retrieval model’s confidence and the relevance of the documents retrieved, thereby aiding DISP in making more informed decisions. Prompt Design. The CoT utilized by the LLMbased Generator indirectly influences the LLMbased Retriever’s outputs by shaping the conversational context, thereby also impacting DISP’s
effectiveness. In row (f), the prompt design for generating questions was altered from CoT to a more straightforward in-context learning approach. The comparative analysis indicates that while utilizing CoT is beneficial, STYLE still retains superior performance even after CoT’s removal, affirming the robustness of the STYLE.

heading: 5 Related Work
text: Determining when to ask clarification questions is critical in conversational search engines (Rahmani et al., 2023; Wang and Ai, 2021; Deng et al., 2022; Aliannejadi et al., 2021), which resolves the ambiguous user query (Aliannejadi et al., 2020). Supervised methods rely on extensive data annotation and training to adapt to specific domain requirements (Keyvan and Huang, 2022; Rahmani et al., 2023). Building on the success of Large Language Models (LLMs), developers have created strategies for determining when to ask questions based on LLMs. These strategies use LLMs to identify if a user query is ambiguous and generate questions through in-context learning (Kuhn et al., 2022; Zhang and Choi, 2023) and chain-of-thought methods (Deng et al., 2023a). This approach enables efficient development across various domains in a post-doc manner. However, their performance is still not satisfactory (Deng et al., 2023b,a), especially when applied to unseen domains. For the first time, We reveal that they struggle to establish tailored strategies when transferred to various unseen domains and propose the use of STYLE to address this challenge.

heading: 6 Conclusion
text: In this paper, we verify and highlight that one-sizefits-all strategies impede the transferability of existing LLM-based methods in unseen domains. To
tackle this limitation, we present STYLE, featuring a domain-invariant strategy planner (DISP) and a multi-domain training paradigm (MDT), to execute tailored strategies in conversational search engines in a post-hoc manner. We conduct a comprehensive set of experiments utilizing four benchmark datasets, each representing distinct domains. The empirical evidence from the results confirms the effectiveness of STYLE. Through detailed analysis, we ascertain that tailored strategies form the basis for efficient domain transferability, elucidating the efficacy of STYLE. With STYLE, we lay the foundation for research on effective model customization.
Limitation
First, owing to the multifaceted nature of conversational search, which encompasses question answering (QA), retrieval, and recommendation scenarios, a thorough analytical study of the clarification question module across all search settings would provide a more comprehensive understanding. However, such an expansive investigation would significantly amplify the experimental workload and diverge from the central research question. Consequently, our focus is confined to conversational retrieval, to extend our research to encompass other forms of conversational search in future work. In addition, since we focus on verifying whether STYLE performs well in unseen domains, multiple unseen datasets are required as test sets for each out-domain trained model. However, due to the heavy experimental workload, for each out-ofdomain training model, we only use one dataset at a time as the unseen domain test set. In the future, we will consider using multiple datasets simultaneously as the test set.

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China (No. 62272330); in part by the Fundamental Research Funds for the Central Universities (No. YJ202219).

heading: A Details of the Experimental Investigation
text: In this section, we present the details of the experimental investigation as shown in section 2. Baselines & Dataset Firstly, we select three representative LLM-based methods as baselines, which are LLM with ZeroShot, LLM with FewShot (CLAM) (Kuhn et al., 2022), and LLM with CoT (ProCoT) (Deng et al., 2023a). Then, We consider four domain-specific datasets, which are: ClariQ in the Web domain (Aliannejadi et al., 2021), FaqAnt in the e-commerce domain (Chen et al., 2023), MSDialog in the Microsoft products domain (Qu et al., 2018), and Opendialkg in the Movies/Books domain (Moon et al., 2019). Strategy Diversity. Firstly, we introduce the approach for quantifying strategy diversity. As the strategy module decides whether to ask a clarification question during each conversation turn, we analyze the diversity of strategies based on the strategy trajectories, each formed by multi-turn actions.
Take LLM with CoT as an example, Figure 6 reveals its strategy trajectories on different domains, represented as tr1, tr2, tr3, tr4. To evaluate its strategy diversity, we calculate the average similarities between pairs of trajectories. To this end, we employ Dynamic Time Warping (DTW) distance, a widely recognized metric for gauging the similarity of sequences. A greater DTW distance between trajectories indicates a lower sequence similarity, leading to a higher level of strategic diversity. Consequently, the strategy diversity of LLM with CoT
is quantified by the average DTW distance between pairs of the four trajectories:
dtw (tr1, tr2) + dtw (tr1, tr3) + ...+ dtw (tr3, tr4)
6 . (5)
It’s important to note that the strategy diversity of the in-domain-trained model is assessed based on trajectories from their variances trained on its own specific domain. Overall Analysis. We obtain the strategy diversity and performance of each method on four datasets across different domains, as shown in Table 5. Table 5 reveals that the in-domain-trained model outperforms LLM-based methods in search performance on all domains. Meanwhile, the in-domaintrained model excels in both strategy diversity and search performance, whereas LLM-based methods often adopt a one-size-fits-all strategy with limited diversity across different domains as shown in Table 5. Additionally, there is a strong correlation (Pearson’s correlation fluctuates between 0.9060 and 0.9923.) between strategy diversity and search performance in all domains.
This indicates that the in-domain-trained model has diverse tailored strategies for various domains, while LLM-based methods lack the flexibility to produce diverse strategies tailored to unseen domains.

heading: B Asking Benefit Calculation
text: In this section, we present the details of calculating the benefit of asking a clarification question cqt at turn t. Considering that a good clarification question helps the search engine retrieve the user’s desired information, we calculate the ranking change of the user’s desired document di after the user answers the question cqt. Ideally, if the
system asks the user a good question at an appropriate time, the user’s answer to this question will assist the search module in retrieving di accurately. We formulate the benefit of the question cqt as:
gaincqt = rankt (di)− rankt+1 (di) , (6)
where di indicates the user’s desired document. rankt (di) indicates the rank of di at turn t and rankt+1 (di) indicates its rank at turn t + 1 after the user answer cqt. After calculating the benefit of each clarification question, we can measure the average benefit per question that can be achieved at each turn of the conversation.

heading: C Data Processing
text: We perform the experiments on four datasets: ClariQ (Aliannejadi et al., 2021), FaqAnt (Chen et al., 2023), MSDialog (Qu et al., 2018) and Opendialkg (Moon et al., 2019). To obtain data that is challenging and fits our setting, we perform the data processing on these datasets. We built our data in a format that fit our setting and increased the proportion of ambiguous queries in our data. More specially, we need to obtain the data in the format of ( ui, di, d ∗ i , q ini i ) . ui is the user ID. di indicates the ground truth document that matches the user intent. d∗i is the intent information of the user and qinii is the first-turn query of the user. In this section, we explain the process of data processing. ClariQ contains conversations between the user and the search agent. Each conversation features an initial query qinii of user ui, an ambiguity classification label of the query (1: ambiguous/0: not ambiguous), a clarification question cqi posed to the user, and a corresponding facet that aligns with the user’s intent. Initially, we consider this facet to be the user’s ground truth document di. Subsequently, we construct the intent information d∗i . We employ ChatGPT to rephrase di, modifying certain key terms and enriching the context to obtain d∗i . Finally, to augment the complexity of our task,
we removed a portion of conversations where the ambiguity label of query qinii is 0 to ensure a preponderance of ambiguous queries. After the above process, we obtain a set of 1000 conversations in ClariQ for the experiment.
FaqAnt contains conversations between the user and a finance customer service agent. Each conversation includes an initial query qinii of user ui, an ambiguity label of the query (1: ambiguous/0: not ambiguous), and the FAQ question-answer pair which matches the user intent. We first take the user’s desired question-answer pair as the ground truth document di. We then construct the refined intent information d∗i by applying ChatGPT to paraphrase di. Finally, to ensure that our task is challenging and contains enough ambiguous queries, we removed a portion of conversations following our operation on ClariQ. After the above process, we obtain a set of 3380 conversations.
MSDialog encompasses question-answering conversations sourced from the Microsoft forum, featuring discussions with multiple participants. This includes the user ui with their initial query qinii , as well as responses from various Microsoft human agents. First, we need the ground truth document di that matches the user intent. Fortunately, in MSDialog, there is a binary label for each turn in the conversation marking whether this turn is acknowledged as the right answer to the user’s intent. Following prior research (Wang and Ai, 2022), we determine the ground truth document di by selecting the response with the highest vote count. Then, we construct the intent information d∗i by utilizing ChatGPT to rephrase di into d∗i . Finally, to maintain a dataset rich in ambiguous queries, we removed a portion of conversations that the di can be simply retrieved by BM25 given the first turn query qinii . This yields an experimental set comprising 1948 conversations.
Opendialkg consists of dialogues wherein a user seeks a recommendation or opinion from an agent
on topics such as movies, music, or books, initiated with a query qinii . Originally, Opendialkg was utilized for conversational reasoning and knowledge graph entity prediction tasks. Aligning with the methodologies of prior studies (Wang and Ai, 2021, 2022), our approach concentrates on conversation retrieval without leveraging knowledge graph-centric models. To establish the ground truth document di, each conversation undergoes human review. Subsequently, we generate the intent information d∗i by utilizing ChatGPT to rearticulate di into a rephrased version, d∗i . This method results in an experimental dataset of 1507 conversations.
D In-domain Training Analysis
In this section, we first demonstrate whether our method can outperform existing supervised methods in unseen domains when training data for unseen domains are available for both our method and existing methods. Then, we assess the capability of our method to maintain superior performance over existing methods when training data for unseen domains is only accessible to the supervised methods.
By conducting the comparison against welltrained supervised baselines, we save the need to compare with numerous domain adaptation methods (Tran et al., 2019), considering that supervised baselines with domain-specific training data surpass domain adaptation methods that lack such data. It is worth noticing that we only consider the training data (query-document pairs) for the document retrieval model to be available, as annotating the training data for supervised clarification question models does not match our setting.
As shown in Table 6, given sufficient training data in unseen domains, our method STYLEinDomain significantly outperforms the supervised retrieval method. This illustrates that our method has a higher performance upper bound when extensive in-domain training data is at
hand. Moreover, to our supervised, our method surpasses existing supervised methods in search performance, even when in-domain training data is only available for existing methods. This indicates that our method does not rely on domain-specific training and has robust transferability, which enables its efficient application across unseen domains.

heading: E Human Evaluation of Clarification Question
text: To rigorously evaluate the quality of the clarification questions produced by each method, we performed a human evaluation. Our comparison focused on STYLE and the top-performing LLMbased method, i.e., ProCoT and CLAM (Deng et al., 2023a; Kuhn et al., 2022). From the clarification questions generated by each method, we randomly selected 100 instances, ensuring that the conversation context, the documents retrieved, and the user intent information were included in the sample. For the evaluation, we enlisted three independent raters to assess the sampled questions based on two specific criteria:
• Helpfulness. This criterion gauges if the posed question is informative and has the potential to elicit valuable information from the user.
• Intent Consistency. This measures the degree to which the question includes elements (such as keywords) that are pertinent to the user’s intent.
To ascertain the level of consensus among the evaluators, we calculated the inter-rater reliability using Fleiss’ Kappa (Fleiss and Cohen, 1973). The results revealed a Kappa score of 0.517 for Helpfulness and 0.782 for Intent Consistency, indicating a moderate to substantial agreement among the raters. Figure 7 presents the outcomes of our
human evaluation, which demonstrate that the clarification questions conceived by STYLE outperform those from ProCoT and CLAM in terms of both Helpfulness and Intent Consistency. This suggests that STYLE’s generated questions are superior in both providing informative content and aligning with the user’s intended purpose.
F Implementation Details
F.1 Parameters of our method
We split the dataset by 6:1:1 for training, validation, and testing. During the training, we randomly sample data from datasets in multiple domains. We set the maximum turn T as 10 and the number of training episodes as 1800. We set the size of the experience buffer in DQN as 10000 and the sample size as 32. We set the learning rate as 1e-4 with Adam’s optimizer. The discount factor γ is 0.99. We adopt the reward of a successful search as 1.0 and the punishment of exceeding the maximum turns as -0.5. We set the number of presented documents x to users as 5. We set the number of layers in the BERT-based encoder in section 3.3 as 3.
F.2 Baseline Implementation
Retreval-based conversational search models. We consider the representative retrieval-based baselines including BM25, sentenceBERT (Reimers and Gurevych, 2019), monoBERT (Nogueira and Cho, 2019), and ChatSearch (Sun et al., 2023). The BERT-based baselines (sentenceBERT and monoBERT) are initialized by the publicly available checkpoint pre-trained from the huggingface6 based on an open-domain corpus, such as MS MARCO7. We then fine-tune these methods on the same training source as our method, which is not in the same domain as the test set. During training, we set the learning rate as 5e-5 and utilized
6https://huggingface.co/OpenMatch/cocodr-basemsmarco
7https://microsoft.github.io/MSMARCO-QuestionAnswering/
the same number of training data following our method. We set the epoch to 15 and the batch size to 16 using the AdamW optimizer For the LLMbased retrieval method (ChatSearch), we use the permutation generation prompt as described in the corresponding paper (Sun et al., 2023). LLM-based methods. We construct the LLMbased method based on gpt-3.5-turbo. For the implementation of CLAM, we adhere to the prompts outlined in the paper (Kuhn et al., 2022) and undertake the task of clarification need prediction and question generation utilizing few-shot in-context learning, as demonstrated in their work. In the case of ClarSim, we diverge from the original approach described in the original paper (Zhang and Choi, 2023) because their method necessitates the entropy information from the output of the decoder and the Intended Interpretation, which our models and dataset do not provide. Alternatively, we apply the Self-Ask strategy as indicated in their publication, prompting LLMs to make a decision—represented by the outputs "Yes" or "No"—on whether to pose a question. For ProCoT (Deng et al., 2023a), it depends on a grounded document, which our dataset lacks. To adjust for this, we replace the grounded document with documents we have retrieved and execute a similar inquiry strategy using few-shot Chain of Thought (CoT) prompts. Lastly, when it comes to CLAMzeroShot, we dispense with the few-shot examples and employ the same prompts used in CLAM but through a zero-shot in-context learning framework. The prompts of LLM-based methods are presented in Figure 8 & 9.
F.3 Implementation of User Simulators Evaluating a conversational system presents a significant challenge due to the multi-turn nature of user-system conversations (Huang et al., 2023; Zamani et al., 2023). In this paper, we resort to the user simulator. The simulated users are expected to play their assigned role as information seekers to communicate with the search engines. Previous studies have leveraged LLMs as user simulators and demonstrated their good performance (Deng et al., 2023b; Sekulić et al., 2022; Wang et al., 2023). Therefore, we utilize the ChatGPT to construct the user simulator based on gpt-3.5-turbo. Given a user ui with the intent information d∗i , we utilize d∗i and some role instructions to formulate the user prompt Puser as shown in Figure 10. When the search engine asks the user a clarification ques-
tion, we send Puser into the ChatGPT and obtain the user’s answer to this question. When the search engine provides the retrieved documents to users, we simply ask the user to give a positive or negative response depending on whether the user’s desired document d∗i is in the provided document.

heading: G Runtime Analysis
text: In this section, we conduct runtime analysis to verify that our method STYLE does not incur additional runtime overhead. Specifically, we evaluate the average running time of different methods within each turn. As shown in Table 7, STYLE requires less time per turn. This is because we use the lightweight model (DISP) as the strategy module, instead of the parameter-intensive LLMs commonly employed in prior approaches. Consequently, our added component significantly reduces the time needed during execution.

heading: Future Work
text: Consequently, our focus is confined to conversational retrieval, to extend our research to encompass other forms of conversational search in future work. In addition, since we focus on verifying whether STYLE performs well in unseen domains, multiple unseen datasets are required as test sets for each out-domain trained model. However, due to the heavy experimental workload, for each out-ofdomain training model, we only use one dataset at a time as the unseen domain test set. In the future, we will consider using multiple datasets simultaneously as the test set