Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions

## Abstract

Abstract Large language models (LLMs) must often respond to highly ambiguous user requests. In such cases, the LLM’s best response may be to ask a clarifying question to elicit more information. We observe existing LLMs often respond by presupposing a single interpretation of such ambiguous requests, frustrating users who intended a different interpretation. We speculate this is caused by current preference data labeling practice, where LLM responses are evaluated only on their prior contexts. To address this, we propose to assign preference labels by simulating their expected outcomes in the future turns. This allows LLMs to learn to ask clarifying questions when it can generate responses that are tailored to each user interpretation in future turns. In experiments on open-domain QA, we compare systems that trained using our proposed preference labeling methods against standard methods, which assign preferences based on only prior context. We evaluate systems based on their ability to ask clarifying questions that can recover each user’s interpretation and expected answer, and find that our training with our proposed method trains LLMs to ask clarifying questions with a 5% improvement in F1 measured against the answer set from different interpretations of each query.

## 1 Introduction

Ambiguity is a hallmark of natural language that enables concise communication by allowing speakers to exclude details that are inferable from the context (e.g., conversational, temporal, geographical) (Piantadosi et al., ). At times, however, the speaker’s intent is unclear despite the context, and further interaction is necessary to clarify their intent. Asking clarifying questions is particularly important for large language models (LLMs), which are tasked with serving a wide audience, often without access to the personalized context available in human interactions. In this work, we develop LLMs that can ask clarifying questions to resolve ambiguity in their users’ requests.

State-of-the-art LLMs (OpenAI, ; Gemini Team, ) often do not ask clarifying questions when presented with an ambiguous request, and instead respond directly by assuming the user’s intent (see an example in Figure 1). We speculate that this tendency stems from a shortcoming in their RLHF training pipelines, which utilize annotated preference data (Ouyang et al., ).
In standard preference data collection, annotators are given a conversation history and are tasked with ranking options for the next assistant turn (Bai et al., ; Wang et al., ). These annotation schemes only consider preferences over single-turns of interaction, making it difficult for annotators to assess the utility of a clarifying question. Furthermore, this can lead to biases where annotators prefer responses with complete but presumptuous answers over incomplete clarifying questions.

We propose an alternative method for annotating double-turn preferences over clarifying questions where annotators interact with the LLM by providing their responses to clarifying questions and observe the LLM’s subsequent responses. Each annotator then assigns preferences based on whether the completed interaction successfully fulfilled the request. When comparing multiple clarifying questions and direct answer responses, each annotator’s preferences are aggregated to identify the response that maximizes preference across all annotators. We depict our proposed double-turn preference annotation method and compare against standard methods in Figure 1.

To demonstrate the benefits of training LLMs to ask clarifying questions with double-turn preferences, we experiment on open-domain QA (Chen et al., ). We develop an automatic evaluation framework for evaluating clarifying questions that use simulated user interactions (Section 2). We evaluate systems on two axis: efficiency measured by the number of model turns and effectiveness measured by F1 score between the predicted answer set for each user and their expected gold answer.

Figure: Figure 1: Depiction of our interaction scenario and preference labeling schemes. We aim to build an LLM that can interact with user to generate the final answer set $R$, containing an answer for each user, for the input query $x$. In this example, we include two responses from state-of-the-art LLMs ([A] from GPT-4 and [B] from Gemini, full responses in Appendix A), which both presuppose an interpretation of the word football. We also include two clarifying responses ([C] and [D]) where [C] correctly disambiguates the two intended interpretations across all users. We depict two different ways to assign preferences on LLM’s initial output (Single-Turn/Double-Turn).
Refer to caption: x1.png

In our experiments, we demonstrate that training systems with double-turn preferences outperforms training with standard preferences annotation methods (Zhu et al., ), resulting in consistent 4-5% improvement in F1 score over three different base LLMs (Dubey et al., ; Mesnard et al., ). We also demonstrate that double-turn preferences can be used to train systems that judiciously determine if clarification is needed, or if the user’s question can be directly answered without additional interaction.

We summarize our contributions below:

- •
We establish an automatic framework for evaluating double-turn interaction involving clarifying question. Our framework includes a user simulation model and automatic evaluation metrics measuring the system’s ability to predict outputs that are tailored to each user efficiently.
- •
We develop a double-turn preference annotation method for labeling preferences over clarifying question and direct response that utilizes the outcomes in later turns of interaction.
- •
Our results on Open-Domain QA datasets (Kwiatkowski et al., ; Min et al., ) demonstrate that training systems to ask clarifying questions using our double-turn preferences improves QA performance over training with standard single-turn preferences. We also show that training with double-turn preferences can teach models to judiciously decide when to ask for clarification.

Our evaluation paradigm and preference labeling scheme can further support future research in developing interactive LLM assistants. We release all code and data at [https://github.com/mikejqzhang/clarifying_questions](https://github.com/mikejqzhang/clarifying_questions).

## 2 Task: QA with simulated user interaction

We chose to study open-domain QA, as it is both prevalent in real user-assistant interactions (Zhao et al., ) and the scope of necessary clarification questions is not immediately identifiable from the input or the task. For simplicity, we choose a well-studied short factoid QA setting (Kwiatkowski et al., ; Min et al., ) where an answer to the question consists of a few tokens. Below, we describe our task and evaluation metrics. Figure 1 contains an example of task scenario.

### 2.1 Interaction Scenario

#### Setting / Goal

We assume an input query $x$, which can be potentially ambiguous, and a set of $k$ users $u_{1},...u_{k}$ who posed the input query $x$. Each $i$-th user $u_{i}$ has an disambiguated intent for query ($x_{i}$) and an expected gold answer ($y_{i}$). Our goal is to build a system $M$ that returns an answer set $R$ that can satisfy as many users as possible (by providing the expected answer $y_{i}$ for each user $u_{i}$) with the minimum number of interactions (measured by the number of model turns).

We will have a simple function $\phi(M(x))$ that returns True if $M(x)$ is a clarifying question, and False otherwise. We further assume an oracle user simulation model $\psi(x,M(x),y_{i})=a_{i}$, that takes first turn input $x$, model’s initial output $M(x)$, and $i$-th user’s targeted answer $y_{i}$, that generates a response to the model’s initial response $M(x)$.

#### Dialogue Turns

In the initial turn, user poses the input query $x$. In the second turn, given the input query $x$, the system $M$ is to generate its initial response ($r_{\text{init}}$). Depending on whether $r_{\text{init}}$ is a question, it will trigger different dialogue trajectories.

- •
Double-turn Scenario If the initial response is a clarifying question ($\phi(r_{\text{init}})$=True), we continue the conversation with the user simulation model. For each of $i$-th user, the evaluation system generates a third turn ($a_{i}$) with the user simulation model $\psi(x,r_{\text{init}},y_{i})=a_{i}$. For the fourth turn, the system will take each of the $k$ interaction traces $(x,r_{\text{init}},a_{i})$ to generate the fourth turn response for $i$-th user $M(x,r_{\text{init}},a_{i})=r_{\text{next}}^{i}$. The final output answer set from the system will be a set of $k$ answers $R=\{r_{\text{next}}^{1},r_{\text{next}}^{2},...r_{\text{next}}^{k}\}$, where each $r_{\text{next}}^{i}$ is a response for $i$-th user.
- •
Single-turn Scenario Otherwise if $\phi(r_{\text{init}})$=False, we will consider the model output $R=\{r_{\text{init}}\}$ as the final prediction. To allow a model that does not generate clarifying questions to generate multiple answers, we sample multiple answers from the model, up to $k$ answers. The final answer set in this case will be $R=\{r_{\text{init}}^{1},...r_{\text{init}}^{k}\}$.

Existing open-domain QA models mostly generates an single answer to the initial input query $x$, even when the query is ambiguous. Such systems that predict an answer without first confirming the user’s intent may mislead users with incorrect answers and fail to serve the entirety of their user population. In this work, we explore methods for QA-with-Clarification, where models can ask the user a clarifying question $q$ and observe the user’s response $a$ before predicting an answer. We will also build a system that sometimes poses a clarifying question and sometimes directly answers, judiciously determine when clarifying questions is useful (Zhang & Choi, ).

#### Evaluations

We evaluate the system in two axis: efficiency and effectiveness. To measure efficiency, we simply measure the average number of total turns from the system. For unambiguous inputs, asking clarifying question is unnecessary. If the system can satisfy users without incurring additional interaction, such system should be preferred.

To measure effectiveness, we use F1 between the model generated answer set ($R$) and the gold reference answer set ($\{y_{1},..y_{k}\})$, which provides us a holistic view of the precision and recall of our systems. When evaluating QA-with-Clarification systems, we additionally enforce that answers must be correct for their corresponding clarifying answer. We hold QA-with-Clarifications to this higher standard, as it aligns with our goal of to developing systems that can identify different user interpretations of a query and accurately predict corresponding answers.

### 2.2 Evaluation Framework Implementation

#### Data

We will leverage existing open-domain QA datasets (Kwiatkowski et al., ; Min et al., ) where each query is paired with annotated answers from multiple annotators.

#### Identifying clarifying question ϕ ⁢ ( ⋅ ) italic-ϕ ⋅ \phi(\cdot) italic_ϕ ( ⋅ )

We will use a very simple method, which takes the input string and check whether it starts with “Clarifying Question:”.

#### User Simulation Model ψ 𝜓 \psi italic_ψ for Turn 3

We use GPT-4 as our oracle user simulation model ($\psi$). We prompt GPT-4 with the input query $x$, clarifying question $q$, and target answer $y_{i}$ and task it with generating a corresponding clarifying answer $((x,q,y_{i})\rightarrow a_{i})$.
We prompt GPT-4 to abstain from providing a clarifying answer $a_{i}$ if it judges that none exists, in which case we count the resulting target answer prediction $r^{i}_{\text{next}}$ as incorrect. We further ensure that the generated clarifying answers do not leak the target answer by removing instances where the target answer $y_{i}$ appears in the predicted clarifying answer $a_{i}$, treating such examples the same as abstains from GPT-4. We provide the prompt used in Appendix B.

## 3 Fine-tuning LLMs to ask Clarifying Questions

We present our approach to build a model that can engage with users for a multi-turn interaction. Specifically, we focus on building a LLM $M$ which can ask clarifying question when it can satisfy more users in the future turns.

Following the standard RLHF training pipeline (Ouyang et al., ), we first construct an instruction tuning dataset focusing on clarifying responses as desired outputs. We then use this constructed dataset for supervised finetuning (SFT), before performing preference learning between two possible responses. We first describe our method for constructing this instruction tuning dataset before introducing our annotation scheme for assigning preferences over clarifying questions.

### 3.1 Preference Data Generation from User-LLM Interactions

To train LLMs to generate useful clarifying questions and identify when to ask them, we propose a novel method for annotating preferences based on full user-LLM interactions. Our method for labeling preferences deviates from standard single-turn annotation methods in two ways. First, we simulate additional interaction with the user and derive reward from the LLM’s final response after the additional turn. Second, we rely on multiple annotators for each example. While some prior work has explored using multiple annotator judgments in single-turn preference datasets (Wang et al., ; Kopf et al., ), these works aggregate annotator judgments via majority choice to generate a single preference label per example. This practice, however, can lead to adverse outcomes where systems whose outputs are catered a toward a single, majority accepted response (Fleisig et al., ; Santy et al., ). In contrast, we use preferences across multiple annotators without removing individual annotator judgments, helping us identify whether a clarifying question or direct-answer response successfully accommodates the range of different annotator’s interpretations.

Figure: Figure 2: Depiction of our preference annotation method. Here, simulated users provide their responses to model-generated clarifying questions and determine preference based on which clarifying question or direct-answer responses lead to their expected answer. We then aggregate preferences across users by selecting the response that is preferred by the most users while minimizing the number of user interactions turns.
Refer to caption: x2.png

In our double-turn annotation scheme (depicted in Figure 2), annotators are provided an input query $x$ with several candidate clarifying question $q$ and direct-answer responses $\widehat{y}$. Annotators are tasked with responding to each clarifying question by providing the clarifying answer that best matches their own interpretation. We then show annotators the model’s final output prediction, given each clarification. Annotators then assign their preferences over clarifying questions and direct-answer responses based on whether the final output matched their expected interpretation.
We then aggregate these preferences to identify which response was preferred by the most annotators.
To prevent systems from asking unnecessary clarifying questions, ties between direct-answer and clarifying question responses are broken by favoring the direct-answer response. The final aggregated preferences can then be used in conjunction with standard RL methods from human preferences for training.
While this annotation scheme can be applied to human annotators, in this work we use simulate user interactions for annotation.

#### Simulating User Interaction

To simulate different user interpretations of the input query, we utilize datasets consisting of an input query $x$ and a set of correct answer outputs $\{y_{1},...,y_{k}\}$. For each query, we use the set of gold outputs to simulate $k$ different users, one for each answer in $\{a_{1},...,a_{k}\}$. We then simulate the behaviors of each of these users and their responses to a proposed clarifying question $q$ using a we use a trained user-simulator model, which takes as input query $x$, clarifying question $q$, and the user’s annotated answer $y_{i}$ and is trained to predict the user’s clarifying answer $(x,q,y_{i})\rightarrow a_{i}$.
While we use prompted GPT-4 to simulate clarifying answers from users for evaluation, during training we use this trained user-simulator model to reduce costs and to ensure that our systems do not overfit to a particular user-simulator method.

#### Answer Prediction / Assigning Preferences

Conditioning on the simulated clarifying answer, our LLM model predicts the output $\widehat{y_{i}}$, $(x,q,a_{i})\rightarrow\widehat{y_{i}}$.
As a last step, we aggregate annotator preferences so that clarifying questions are ranked by their EM accuracy, evaluated against each annotator’s expected answer: $\sum_{i=1}^{k}\text{EM}(\widehat{y_{i}},y_{i})/k$. To rank direct-answer responses, which only predict a single answer $\widehat{y}$, we similarly evaluate the EM accuracy of the predicted answer $\widehat{y}$ against each annotator’s expected answer. To prevent systems from asking unnecessary clarifying questions, we favor the direct-answer response in the case of ties. We then use these aggregated rankings to determine the pairwise preference labels between responses for training.

## 4 Experiments

#### Models

We use Llama2-7b (Touvron et al., ), Gemma-7b (Mesnard et al., ), and Llama3-8b (Dubey et al., ) as our base LLMs. We forego using the instruction-tuned models due to data leakage concerns, as such systems are finetuned on the entirety of NQ-Open. For training and inference, we use 8-bit quantization (Dettmers et al., ) with LoRA (Hu et al., ; Dettmers et al., ) (training details in Appendix B).

#### Data

We perform our experiments on the NaturalQuestions (NQ-Open) (Kwiatkowski et al., ; Lee et al., ) and AmbigQA (Min et al., ) datasets. In both of these datasets, each input query $x$ are associated with a set of possible output answers $\{y_{1},\dots,y_{k}\}$. We describe each dataset below, and provide details in Appendix A.

- •
NQ-Open is comprised of questions from Google search queries by real users. Each query is annotated with answers from Wikipedia by up to five annotators; many questions (about 10% of train and 42% of development set) in the dataset contain multiple distinct answers based on each annotator’s judgment in interpreting the query and selecting the best answer span.
The answers sets identified by annotators for such examples can stem from a variety of ambiguities, from formatting to ambiguities resulting in semantically distinct answers (examples in Appendix A).
- •
AmbigQA consists of a subset of queries from NQ-Open which have been additionally annotated with whether or not the input is ambiguous. If the input is ambiguous, annotators then provide the additional answers to all possible interpretations of the query. This process identifies about 56% of all queries in NQ-Open as ambiguous, and recovers 2.6x more answers to ambiguous questions than the base NQ-Open annotations alone. We provide additional comparisons in the answer sets identified by each dataset in Appendix A.

We evaluate on AmbigQA test set (n=1960). In addition to evaluating over the Full test set, we additionally report performance on splits of Unambiguous questions (788 questions with one answer each) and Ambiguous questions (1,172 questions averaging 3.7 answers each).

#### Supervised Finetuning (SFT) Dataset Generation

To generate examples for SFT training, we prompt GPT-4 with an input query $x$ and a set of feasible answers $Y=\{y_{1},\dots,y_{k}\}$ and task it with proposing a clarifying question $q$ that will help determine which answer the questioner expects. Our prompt also tasks GPT-4 with predicting responses to the clarifying question for each one of the feasible answers $\{(a_{1},y_{1}),\dots,(a_{k},y_{k})\}$, if it determines one exists. We use the same GPT-4 prompt to construct all examples SFT training, and introduce two methods for generating feasible answer sets $Y$ for a given query $x$.

- •
Answers from Human Annotations: We use the ambiguous queries from the AmbigQA training and development splits and their annotated answers as the feasible answer set $Y_{\text{human}}$. Identifying ambiguities and labeling all possible answers is a challenging task for annotators, and thus this data can be expensive to collect. Furthermore, we hypothesize that what is ambiguous for an LLM often deviates from what is ambiguous for humans.
- •
Answers from Model Predictions: We build feasible answer sets $Y_{\text{model}}$ to for queries in the NQ-Open training set using answer candidates proposed by a base LLM. Prior work has demonstrated that base LLMs are well calibrated in open-domain QA tasks with few-shot prompting (Kadavath et al., ; Cole et al., ). Drawing on these observations, we construct a feasible answer set $Y_{\text{model}}$ by sampling 5-shot predictions from Llama-2-7b (Touvron et al., ). For each query, we a sample 5-shot prompt and generate its greedy answer with temperature $T=0.0$ and sample an answer with $T=1.0$. We repeat the process 10 times per query and remove repeated answers to generate a set of model-identified feasible answers. We further filter examples where none of the feasible answers match any annotated answers from NQ-Open.

For each method of generating feasible answer sets ($Y_{\text{human}}$ and $Y_{\text{model}}$), we generate an SFT dataset of 4,400 input query and clarifying question pairs $(x,q)$, which we split into training (4,000) and development (400) splits. Between both datasets, this gives us a total of 19,807 $(x,q,a_{i},y_{i})$ examples. In Appendix A, we include examples and the exact prompts.

Figure: Figure 3: Illustration of models in our study (right) and the data used for training them (left). Ans-After-Clarify SFT model is used to generate responses for the fourth turn, User-simulator SFT model is used to generate responses for the third turn. All other models generate responses for the 2nd turn.
Refer to caption: x3.png

### 4.1 Compared Systems

Figure 3 summarizes all models used in our experiments. We have two types of models, model that will be used at the second turn (which we refer as Turn 2 model) and model that will be used at fourth turn (which we refer as Turn 4 model). The model that will be used at the second turn can ask clarification question or directly answer the question. All models that are capable of asking clarification questions are paired with Turn 4 model. We use a separate model for performing this latter step to ensure that differences in performance only reflect the value of the clarifying question, and explore training joint models for both steps in Section 5. We describe each of these below.

#### Direct Ans SFT

This model is trained to predict the answer given the input query $(x)\rightarrow y$. We fine-tune base LLM with next-token prediction loss on the full set of question answer pairs (i.e., each $(x,y_{i})$) in our generated SFT training sets. We use this model under two inference settings: (1) Greedy where we take the single greedy answer prediction and (2) Sampled where we sample a set predicted answers from the LLM. We sample 20 answers from the model and select the top $k$ most frequent answers, where $k$ matches the gold number of annotated answers. In rare cases ($<2\%$), there are less than $k$ unique answers within the 20 samples.

#### Clarify SFT

This model is trained to ask clarifying question to input query. We finetune base LLMs on our SFT datasets constructed with both human annotations and model predicted answers. In Section 5, we provide ablation experiments training over each dataset to compare their utility.

#### Clarify DPO (Starling) / Clarify DPO (Ours)

These two models are each fine-tuned from Clarify SFT model with using DPO (Rafailov et al., ), a method for optimizing LLMs on preference data that eliminates the need for training an intermediate reward model.
They are provided the same response pairs, which are sampled from Clarify SFT model. Specifically, we sample six candidate clarifying questions (one with $T=0.0$ and five with $T=1.0$) and form preference pairs among them. The only difference is how to assign reward. For Clarify DPO (Starling) model, we use the Starling-RM-7B-alpha reward model (Zhu et al., ) which was finetuned from Llama2-7B-Chat on over 3.8M single-turn preferences between the outputs from a variety of different LLMs. For Clarify DPO (Ours) model, we use the preference assigned by simulating double-turn preferences using our user-simulator model. We construct this model by finetuning the base LLM on all $(x,q,y_{i})\rightarrow a_{i}$ pairs.

#### Clarify-or-Direct Ans DPO (Ours)

We first build Clarify-or-Direct Ans SFT model by training on the union of two datasets used to train Direct Ans SFT and Clarify SFT. We then further train this model over by double-turn preferences over clarifying question and direct-answer responses with DPO learning objective. We perform DPO training over pairwise comparisons from the same six candidate clarifying questions generated for our Clarify DPO models above and the single greedy sampled direct-answer response generated from our Direct Ans SFT model above.

## 5 Results

**Table 1: Main results. Our method (Clarify DPO (Ours)) yields the optimal results in Answer F1 for all base models at the cost of always asking clarifying questions. Our method that selectively asks clarifying questions (Clarify-or-Direct-Ans DPO (Ours)) strikes a balance between efficiency (Avg Turns) and effectiveness (Answer F1). Bold results denote statistically significant differences from all other methods with the same base model (over $N=10,000$ samples with $p<0.01$).**
| Base | Turn 2 Model | Inference | Avg Turns ($\downarrow$) | Answer F1 ($\uparrow$) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Unambig | Ambig | Total |  |  |  |  |
| Llama2<br>(7b) | Direct-Ans SFT | Greedy | 1 | 0.254 | 0.168 | 0.211 |
| Direct-Ans SFT | Sampled | 1 | 0.250 | 0.172 | 0.214 |  |
| Clarify SFT | Greedy | 2 | 0.310 | 0.216 | 0.259 |  |
| Clarify DPO (Starling) | Greedy | 2 | 0.310 | 0.257 | 0.283 |  |
| Clarify DPO (Ours) | Greedy | 2 | 0.383 | 0.282 | 0.328 |  |
| Clarify-or-Direct-Ans DPO (Ours) | Greedy | 1.56 | 0.289 | 0.211 | 0.243 |  |
| Gemma<br>(7b) | Direct-Ans SFT | Greedy | 1 | 0.254 | 0.168 | 0.211 |
| Direct-Ans SFT | Sampled | 1 | 0.250 | 0.172 | 0.214 |  |
| Clarify SFT | Greedy | 2 | 0.357 | 0.236 | 0.288 |  |
| Clarify DPO (Starling) | Greedy | 2 | 0.339 | 0.257 | 0.295 |  |
| Clarify DPO (Ours) | Greedy | 2 | 0.407 | 0.286 | 0.339 |  |
| Clarify-or-Direct-Ans DPO (Ours) | Greedy | 1.61 | 0.282 | 0.222 | 0.246 |  |
| Llama3<br>(8b) | Direct-Ans | Greedy | 1 | 0.312 | 0.192 | 0.248 |
| Direct-Ans | Sampled | 1 | 0.282 | 0.202 | 0.247 |  |
| Clarify SFT | Greedy | 2 | 0.376 | 0.265 | 0.315 |  |
| Clarify DPO (Starling) | Greedy | 2 | 0.362 | 0.267 | 0.309 |  |
| Clarify DPO (Ours) | Greedy | 2 | 0.429 | 0.317 | 0.365 |  |
| Clarify-or-Direct-Ans DPO (Ours) | Greedy | 1.57 | 0.352 | 0.251 | 0.291 |  |

We report our main results in Table 1. Overall, systems that are equipped to ask clarifying questions outperform Direct-Ans models in effectiveness (Answer F1). Clarify-or-Answer methods strike a balance between effectiveness and efficiency. We further investigate the behaviors of Clarify-or-Answer methods in Section 5.1.

We also find that training with double-turn preference labeling scheme consistently yields the best answer F1 overall. We observe mixed results from training with single-turn RLHF annotations from Starling: yielding positive results with Llama2 as the base LLM and minor increases/decreases in performance for Gemma and Llama3 when evaluated over both ambiguous and unambiguous queries. These results support our claim that single-turn RLHF annotation struggles to identify useful clarifying questions. In contrast, training with double-turn preferences, where clarifying questions are assessed based on their later outcomes, demonstrates consistent improvements in the LLM’s ability to generate useful clarifying questions, on both ambiguous and unambiguous queries. This demonstrates that clarifying questions not only help models disambiguate user intents for ambiguous queries, but also can help models recover correct answers in general even for unambiguous queries. We include examples of generated clarifying questions in Appendix C.

**Table 2: Ablations comparing training Clarify DPO (Ours) systems using SFT training on clarifying questions generated from $Y_{\text{human}}$ or $Y_{\text{model}}$ answer sets or both.**
| Base Model | SFT Dataset | Unambig | Ambig | Full |
| --- | --- | --- | --- | --- |
| Llama2<br>(7b) | Human | 0.288 | 0.236 | 0.259 |
| Model | 0.299 | 0.264 | 0.283 |  |
| Both | 0.383 | 0.282 | 0.328 |  |
| Gemma<br>(7b) | Human | 0.379 | 0.274 | 0.323 |
| Model | 0.391 | 0.283 | 0.329 |  |
| Both | 0.407 | 0.286 | 0.339 |  |
| Llama3<br>(8b) | Human | 0.401 | 0.309 | 0.349 |
| Model | 0.423 | 0.312 | 0.364 |  |
| Both | 0.429 | 0.317 | 0.365 |  |

#### Ablations: Comparing Clairifying Question Generated with Human Annotated Answers vs. Model Predicted Answers

In our main experiments, we performed SFT training on the union of clarifying questions generated from model-identified $(Y_{\text{model}})$ and human-identified $(Y_{\text{human}})$ feasible answer sets. Is one of them more useful than the other? In Table 2, we compare performance using SFT examples constructed from our $Y_{\text{model}}$ and $Y_{\text{human}}$ answer sets, after RLHF training with double-turn preference. Looking first at our results using Llama2, we see clear improvements when using our model-ambiguity SFT dataset over using our human-ambiguity dataset alone. Surprisingly, we find that these improvements even hold when evaluated against the answer sets from AmbigQA, despite the fact that our human-ambiguity SFT dataset was generated to distinguish between answers labeled in the AmbigQA training split.

Next, looking at our results using Gemma and LLama3, we find that there are only minor overall improvements from training when our model-ambiguity SFT dataset. Recall that our model-ambiguity dataset was using feasible answer sets $Y_{\text{model}}$ from few-shot Llama2 predictions. These results, therefore, suggest that constructing a model-ambiguity dataset that is specific to the base model improves the resultant clarifying question quality.

#### A Joint Model for Asking Clarifying Questions and Predicting Answers

So far, our full systems have two separate LLMs. Here, we experiment with using a single LLM for both the clarification $(x\rightarrow q)$ and answer $(x,q,a_{i}\rightarrow y_{i})$ conversation turns. To create this joint model, we take inspiration from prior works demonstrating that averaging the weights of multiple finetuned models is a simple and effective approach to joining the capabilities of finetuned models into a single LLM (Wortsman et al., ). We merge the LoRA parameter updates from our clarification model (trained with preferences from full interactions) and the answer prediction models used in our main results (Table 1).

**Table 3: Results comparing F1 using separate clarifying question and answering models and using a single joint model to perform both tasks.**
| Base LLM | Cl. & Ans.<br>Models | Unambig | Ambig | Full |
| --- | --- | --- | --- | --- |
| Llama2<br>(7b) | Separate | 0.383 | 0.282 | 0.328 |
| Joint | 0.324 | 0.250 | 0.287 |  |
| Gemma<br>(7b) | Separate | 0.407 | 0.286 | 0.339 |
| Joint | 0.362 | 0.274 | 0.314 |  |
| Llama3<br>(8b) | Separate | 0.429 | 0.317 | 0.365 |
| Joint | 0.410 | 0.292 | 0.346 |  |

We report our results from these experiments in Table 3, where we compare performance against using separate LLMs for each conversation turn. We see that overall performance degrades when using a joint model for both turns; however, we are still able to retain most of the performance gains over Direct-Ans and QA-with-Clarification methods trained with single-turn preferences. Future work may explore alternative training methods that mitigate the performance degradation, such as using mixture-of-experts methods for LLMs (Jiang et al., ) or improved multi-turn learning algorithms (Zhou et al., ).

### 5.1 Determining When Clarifying Questions are Necessary

In this section, we further investigate our system’s ability to identify when clarifying questions are necessary. To evaluate our Clarify-or-Answer model’s predictions, we report Direct-Answer Accuracy: Did the the system correctly decide to directly answer the question when the question was unambiguous and the greedy predicted direct-answer was correct. To determine whether the greedy predicted direct-answer was correct, we decode our Clarify-or-Answer system’s greedy direct-answer prediction for each input question.

We compare our models predictions (C-or-A-Pred) against the random baseline for determining whether to directly-answer or ask a clarifying question. To compute this, we fix the percent of direct-answer responses (DA%) to the model’s clarify-or-answer predictions and randomly sample DA% of predictions to directly answer, and predict clarifying questions for the remaining input queries.

**Table 4: Results for Clarify-or-Answer methods. For each model, we include the percent of questions directly answered (DA%), and evaluate using F1, and Direct-Answer Acc.**
| Model | Method | F1 | DA Acc |
| --- | --- | --- | --- |
| Llama2<br>(DA 44%) | Random | 0.234 | 0.554 |
| C-or-A Pred | 0.243 | 0.619 |  |
| Gemma<br>(DA 38%) | Random | 0.237 | 0.604 |
| C-or-A Pred | 0.246 | 0.643 |  |
| Llama3<br>(DA 43%) | Random | 0.280 | 0.549 |
| C-or-A Pred | 0.365 | 0.595 |  |

#### Results

We report our results from our Clarify-or-Answer systems in Table 4. Our Clarify-or-Answer system’s predictions, however, are able to consistently outperform random baseline, with the biggest improvement being from the Direct-Answer accuracy metric. Prior work has demonstrated that the similar task of classifying ambiguous from unambiguous questions is a challenging task, with the best LLM-based methods achieving 58% accuracy (Cole et al., ). While our task is slightly different, determining if the question is ambiguous or the direct-answer response is incorrect, we find that this task is similarly difficult for LLM systems. The relatively large gains on Direct-Answer accuracy compared to the F1, indicates that we can expect performance on end-task metrics to further improve with better clarifying question generations.

## 6 Related Work

Ambiguity in NLP
Ambiguity has been studied extensively in NLP in across a variety of tasks. In machine translation, works have studied instances where sentences in a source language have multiple valid translations in a target language (Fernandes et al., ; Lopes et al., ; Voita et al., ). In natural language inference (NLI), Nie et al. () collected a dataset of highly cross-annotated examples that contain high disagreement in annotator judgments. Liu et al. () then found that these disagreements are often the result of ambiguity in the input. Recent work has studied various sources of ambiguity, from entity-linking (Lee et al., ), co-reference ambiguities (Yuan et al., ), to temporal and geographical contexts (Zhang & Choi, ).

Uncertainty in LLMs
Several recent works have studied calibration in LLMs, looking specifically at the effects of RLHF training. Specifically, whereas likelihoods for base LLMs (i.e., pretrained) tend to be well-calibrated for a variety of tasks (Cole et al., ; Kadavath et al., ), this behavior is lost after RLHF finetuning (OpenAI, ). Additionally, Zhou et al. () investigated uncertainty-expression via epistemic markers generated by LLMs and found evidence that standard single-turn RLHF training encourages over-confidence in LLM responses. Band et al. () trained an LLM to generate long-form text that includes confidence statements about its own generation. Zhang & Choi () studied when uncertainty in predictions are caused by lack of knowledge or by lack of clarification.

Clarifying Question Generation
Prior works for training LLMs to ask clarifying questions have focused on settings where the types of clarification required are fixed by the task (i.e., there is fixed set of relevant input features (Wei et al., ; Kuhn et al., ))
or the input (e.g., reading-comprehension tasks where the context contains multiple correct answers (Guo et al., )).
Such works have studied clarifying question generation for a variety of tasks, including gauging social and moral situations (Pyatkin et al., ) and image classification (Yu et al., ). Andukuri et al. () trained LLMs to ask clarifying questions to resolve task ambiguity, where a single request may have multiple different indented tasks depending on the user. In this work, the authors simulate ideal responses from a fixed pool of simulated personas, and task models with asking clarifying questions to generate a tailored response. Hong et al. () also uses LLM-simulated dialogues for training. Recent work has also noted the scarcity of clarifying questions in responses from existing LLMs, and have studied promoting methods for eliciting such responses with greater frequency (Deng et al., ; Shaikh et al., ). Li et al. () found that clarifying questions can be useful for learning individualized preferences.

## 7 Limitations and Future Work

Throughout this work, we only consider systems for one or two turn interactions and does not consider cases where asking additional clarifying questions might be helpful after observing the user’s clarifying answer. Future work might explore methods for extending our evaluation framework and double-turn preference annotation methods to accommodate general multi-turn interactions with users. Furthermore, the systems explored in this work also do not model dialogue acts (Stolcke et al., ) outside of predicting a single answer or asking a clarifying question. At times, it might be more appropriate for LLMs to generate responses for different behaviors (e.g., Overton responses containing multiple answers if there are only a few possible interpretations. Long vs. short answers for complex queries. Abstaining in lieu of predicting erroneous answers). Future work might consider how clarifying questions should be used in conjunction with other strategies.

Finally, throughout this work we rely on simulated user interactions for labeling double-turn preferences and for evaluation. While future work might consider applying our evaluation framework and double-turn preference annotation scheme to human annotators, there are several challenging hurdles. In particular, annotating a clarifying answer to an input query, clarifying question, and known expected answer can be challenging, particularly in open-domain QA where queries often ask about niche topics. Furthermore, relying on annotators to simply provide their best own answers to clarifying questions may lead issues in coverage of possible query interpretations. These challenges indicate that relying on human evaluators for clarifying questions can be costly and its reliability is highly dependent on the pool of annotators.

## 8 Conclusion

We propose a method for training LLMs to ask clarifying questions with double-turn preferences. To evaluate our systems, we develop an automatic framework for evaluating systems that ask clarifying questions using user simulated user interactions. Our results on Open-Domain QA demonstrate that training systems to ask clarifying questions with double-turn preferences improves QA performance over training with standard single-turn preference labels. We also demonstrate that training with double-turn preferences be used to train models to judiciously decide when to ask for clarification.

## Acknowledgments

We thank the UT NLP community for providing feedback throughout the project. This work was in part supported by Cisco
Research. Any opinions, findings and conclusions,
or recommendations expressed in this material are
those of the authors and do not necessarily reflect
the views of Cisco Research. The work is partially supported by a gift from Google Research. This work has taken place in part in the Rewarding Lab at UT Austin. The Rewarding Lab is supported by NSF (IIS-2402650), ONR (N00014-22-1-2204), EA Ventures, Bosch, and UT Austin’s Good Systems grand challenge.

## Appendix A Dataset Details

In Table 5, we provide examples from AmbigQA and NQ-Open with differing answer sets. These examples highlight instances where relying on a multiple annotators to provide the answer versus relying on a single annotator to provide all possible answers yielded differing answer sets.

**Table 5: QA examples differing labeled answer sets form NQ-Open and AmbigQA.**
| Question | NQ-Open Annotations | AmbigQA Annotations |  |
| --- | --- | --- | --- |
| Answers | Is Ambiguous? | Answers |  |
| Who is singing in the | {“Ray Charles”, | No | {“Jamie Foxx”} |
| background of gold digger? | “Jamie Foxx”} |  |  |
| How much does a 2004 | {“1,162 - 1,315 kg”, | No | {“2,562–2,900 lb”} |
| chevy cavalier weight? | “2,562 - 2,900 lb” } |  |  |
| What season does meredith | {“Season 7”} | Yes | {“Season 5”, |
| and derek get married |  |  | “Season 7”} |
| in grey’s anatomy? |  |  |  |
| When did the soviet union | {“29 August 1949”} | Yes | {“29 August 1949”, |
| test its first atomic bomb? |  |  | “7:00 a.m”} |
| When did the soviet union | {“29 August 1949”} | Yes | {“29 August 1949”, |
| test its first atomic bomb? |  |  | “7:00 a.m”} |

### A.1 Human and Model Ambiguity SFT and RLHF Training Dataset Details

In Table 7, we report dataset statistics for or two SFT datasets. Note that $(x,q)$ pairs are used for SFT training for our clarifying question asking $(x\rightarrow q)$ systems. Additionally, the $(x,q,a,y)$ pairs are used for training our joint and separate systems for predicting QA answers given clarification $(x,q,a\rightarrow y)$ as well as our user simulator models $(x,q,y\rightarrow a)$. We also use these pairs for training our direct QA systems $(x\rightarrow y)$, ignoring the clarifying question and answer. In Table 8, we include several examples from each dataset, highlighting the differences in the ambiguities identified in each.

For RLHF training, we use examples from the NQ-Open training set after removing examples used to generate our SFT datasets from NQ-Open. This leaves us with 70,904 remaining input questions, which we split into training and develpent splits (64,584 and 6,320).

**Table 6: LLM responses to the input query. All responses were generated on May 21st, 2024.**
| Input Query: Who is the highest paid football player? |
| --- |
| GPT-4 Response (104 words): |
| As of 2024, the highest-paid football players are Cristiano Ronaldo and Karim Benzema, both earning around €200 million per year. Ronaldo plays for Al Nassr, and Benzema is with Al Ittihad, both in the Saudi Pro League. |
| Following them, N’Golo Kanté and Neymar each earn approximately €100 million annually, also playing in the Saudi Pro League for Al Ittihad and Al Hilal, respectively. Kylian Mbappé is the highest-paid player outside the Saudi Pro League, with a salary of €70 million at Paris Saint-Germain. |
| These figures reflect the significant financial influence of the Saudi Pro League, which has attracted many top players with lucrative contracts. |
| Gemini Response (30 words): |
| The highest-paid football player in the NFL for 2024 is Joe Burrow, the quarterback for the Cincinnati Bengals. He earns a whopping $55 million per year according to his contract. |
| Claude-3 Sonnet Response (112 words): |
| As of 2023, the highest paid football (soccer) player in the world is Kylian Mbappé of Paris Saint-Germain. His reported annual salary is around €72 million (approximately $78 million). |
| Some other top highest-paid football players in 2023 include: |
| 2. Lionel Messi (Paris Saint-Germain) - Around €41 million per year |
| 3. Neymar (Paris Saint-Germain) - Around €36.5 million per year |
| 4. Kevin De Bruyne (Manchester City) - Around €25 million per year |
| 5. Robert Lewandowski (Barcelona) - Around €23.5 million per year |
| However, player salaries can fluctuate frequently with new contracts, bonuses and endorsements. Mbappé’s new deal with PSG from 2022 made him the highest paid at the reported €72 million annual wages. |

**Table 7: SFT Dataset Statistics.**
| SFT Dataset | Train | Dev |  |  |
| --- | --- | --- | --- | --- |
| $(x,q)$ | $(x,q,a,y)$ | $(x,q)$ | $(x,q,a,y)$ |  |
| Human-Identified Ambiguity | 4000 | 8781 | 400 | 889 |
| Model-Identified Ambiguity | 4000 | 9189 | 400 | 948 |

**Table 8: Randomly sampled Model-Ambiguity SFT dataset examples.**
| Question $(x)$ | Clarifying Question $(q)$ | Clarifying Answers $(a_{i})$ | Answers $(y_{i})$ |
| --- | --- | --- | --- |
| where were the olympic games held in greece | Are you asking about the ancient Olympic Games or the modern Olympic Games? | Ancient Olympic Games / Modern Olympic Games | Olympia / Athens |
| who won the red army or the white army | Are you asking about the Russian Civil War or a specific battle? | Russian Civil War / Specific battle | The Red Army / the white army |
| who wrote ob la di ob la da | Are you asking about the primary composer or the co-writer? | Primary composer / Co-writer | Paul McCartney / John Lennon |
| original singer of how am i supposed to live without you | Are you asking about the original singer who first recorded the song or the singer who popularized it? | The original singer who first recorded the song / The singer who popularized the song | Laura Branigan / Michael Bolton |

**Table 9: Randomly sampled Human-Ambiguity SFT dataset examples.**
| Question $(x)$ | Clarifying Question $(q)$ | Clarifying Answers $(a_{i})$ | Answers $(y_{i})$ |
| --- | --- | --- | --- |
| Who played thanos in gardians of the galaxy? | Are you asking about the actor who provided the voice for Thanos, or the actor who physically portrayed him on screen? | The actor who provided the voice for Thanos. / The actor who physically portrayed him on screen. | Isaac C. Singleton Jr. / Sean Gunn |
| Where in the bible is the story of samuel? | Are you asking for the specific book and verses where the story of Samuel is found, or are you asking for the general section of the Bible where the story is located? | I’m asking for the specific book and verses. / I’m asking for the general section of the Bible. | 1 Samuel 1:1–28 / Hebrew Bible |
| Most amount of points scored in an nba game? | Are you asking about the most points scored by a single player in a game or the highest combined score in a game? | Most points scored by a single player / Highest combined score in a game | 100 / 370 |
| Who has the highest instagram followers in the world? | Are you asking about the individual with the most followers on Instagram, or the account with the most followers overall? | The individual with the most followers / The account with the most followers overall | Cristiano Ronaldo / Instagram |

**Table 10: Prompt used for generating our Human and Model Ambiguity SFT datasets with GPT-4.**
| You will be given a question and several possible responses. |
| --- |
| Possible responses may be correct due to an ambiguity in the question or when the question was asked. |
| Provide a clarifying question to determine which possible response is correct. |
| If one or zero of the possible responses are correct, respond with “None”. |
| Provide your output in the following format: |
| Clarifying Question: [Clarifying Question] |
| 1. Clarifying Answer: [First Clarifying Answer] |
| 1. Response: [Copied Response for First Clarifying Answer] |
| 2. Clarifying Answer: [Second Clarifying Answer] |
| 2. Response: [Copied Response for Second Clarifying Answer] |
| … |
| Question: {question} |
| Possible Responses: |
| {answers} |

## Appendix B Experimental Details

We perform all experiments on a single machine with 8 A40 (48GB) GPUs using the transformers library (Wolf et al., ) and the AdamW optimizer (all training runs completed within 24 hours). During SFT training, we train all models with a learning rate of 5e-5 and batch size of 32. Training was performed for up to 5 epochs, evaluating on our development set after each epoch and selecting using the best performing checkpoint.

For DPO training, we merge LoRA checkpoints from our SFT-only baseline and train using a KL regularization factor of 0.1 and a learning rate of 5e-6 in all experiments. For training was performed for up to 2 epochs until loss converges on development data, selecting the best performing checkpoint on development data. For Llama2-7b based methods, we train with a batch size of 32, evaluating every 750 steps. For Gemma-7b based methods we train with a batch size of 16, evaluating every 1500 steps.

For performing additional SFT training to create our joint models for asking clarifying questions and answering with clarification, we merge LoRA checkpoints again and use the same hyperparameters used for standard SFT training listed above.

## Appendix C Example Outputs

We include examples of models successfully using clarifying questions to predict the target answers for ambiguous (in Table 13) and unambiguous test queries (in Table 12).

## Appendix D Licensing

The transformers library is licensed under Apache License 2.0. AmbigQA does not list any license; however NaturalQuestions, the dataset which it is based on, is under Apache License 2.0. Llama2 is licensed under the LLAMA 2 Community License Agreement. Gemma is licensed under the Gemma Terms of Use. We license all our created assets under CC BY-NC-SA 4.0.

**Table 11: GPT-4 prompt used for simulating user clarifying answers during evaluation. All evaluations were carried out using the GPT-4 version gpt-4-0613.**
| Pretend that you are a user asking an AI assistant a question. |
| --- |
| In response to your question, the AI assistant has asked you a clarifying question to help it determine which answer you expect. |
| For each of the following expected answers, provide a clarifying answer to the AI assistant’s clarifying question that indicates you are expecting that answer. |
| All clarifying answers should be a concise sentence or phrase and should not contain the expected answer. |
| If there is no possible clarifying answer, respond with “None.” instead. |
| Provid your response in the following format: |
| Clarifying Answer 1: [Clarifying Answer for Expected Answer 1] |
| Clarifying Answer 2: [Clarifying Answer for Expected Answer 2] |
| … |
| Question: {question} |
| Clarifying Question: {clarify_q} |
| {answers} |

**Table 12: Example predictions where asking clarifying questions helped recover the correct answer to an unambiguous test query. Clarifying question predictions are taken from our LLama2-7B-based Clarify DPO (Ours) model trained with double-turn preferences. We also include the predicted LLama2-7B-based Direct-Ans SFT model predictions.**
| Question $(x)$ | Pred Clarifying Question $(q)$ | GPT-4 Clarifying Answers $(a_{i})$ | Clarify DPO Pred Ans. $(\widehat{y_{i}})$ | Direct-Ans SFT Pred Ans. $(\widehat{y_{i}})$ | Gold Ans $(y_{i})$ |
| --- | --- | --- | --- | --- | --- |
| When does telltale’s guardians of the galaxy episode 3 come out? | Which platform are you asking about for Telltale’s Guardians of the Galaxy Episode 3? | I’m asking about the release date on all platforms. | August 22 , 2017 | October 17, 2017 | August 22 , 2017 |
| When did gods not dead 3 come out | Which “God’s Not Dead 3” are you referring to? The first, second, or third installment? | I’m asking about the release date on all platforms. | March 30 , 2018 | April 2, 2018 | March 30 , 2018 |
| When did the sea of monsters movie come out? | Which “Percy Jackson and the Sea of Monsters” movie are you asking about? The one released in 2013 or the one released in 2022? | The one released in 2013. | August 7 , 2013 | March 26, 2010 | August 7 , 2013 |
| Who won the nobel prize for peace in 2014? | Which individual or organization won the Nobel Peace Prize in 2014? | The individuals who won | Kailash Satyarthi and Malala Yousafzai | Kailash Satyarthi | Kailash Satyarthi and Malala Yousafzai |
| How many islands are included in the federated states of micronesia? | Which specific islands are you referring to? | I’m referring to all the islands and islets that make up the country. | 607 islands and islet | 600 | 607 islands and islet |

**Table 13: QA-with-Clarification predictions on unambiguous test queries from our LLama2-7B model trained with double-turn preferences. We include examples where clarifying questions helped the model recover multiple annotated answers.**
| Question $(x)$ | Pred Clarifying Question $(q)$ | GPT-4 Clarifying Answers $(a_{i})$ | Pred & Gold Ans. $(\widehat{y_{i}},\widehat{y_{i}})$ |
| --- | --- | --- | --- |
| How many starbucks ? | As of which date are you | As of the end of 2018. | (28,218, 28,218) |
| are there around the | asking about the number | As of the end of 2004. | (6,000, 4,962) |
| world | of Starbucks stores around | As of the end of 2016. | (23,768, 23,768) |
|  | the world? | As of the end of 2017. | (25,000, 30,000) |
| Who sang the original | Which version of “Do You | The version that was a | (The Contours, The Contours) |
| do you love me? | Love Me” are you referring | hit in the 1960s. |  |
|  | to? The original song by |  |  |
|  | The Contours or the cover | The version from the | (Topol, Topol) |
|  | by The Partridge Family | musical Fiddler on the Roof. |  |
| When did the queen | Which specific queen are you | Queen Victoria | (20 June 1837, 20 June 1837) |
| became queen of | referring to? | Queen Elizabeth II | (6 February 1952, 6 February 1952) |
| england? |  | Queen Anne | (8 March 1702, 1 May 1707) |

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: Would it be helpful & possible to include some more informative comparisons and evaluation metrics (see "Weaknesses" above)?

On annotation reliability: I wasn't clear whether the additional annotation described for AmbigQA was done as part of this work, or previously by other authors and described elsewhere ("his process identifies about 56% of all queries in NQ-Open as ambiguous" etc.).

Table 2 doesn't say what metric it uses - presumably F1?

Some minor typos:
083 two axis->axes
097 citation should be citep not citet
110 of [the] task scenario
187 an additional turn[s]
406 Cla[i]rifying
188 observ[ing] the LLM's 
206 simulate[d] user
soundness: 3
strengths: The topic addressed -- clarification behaviour by LLMs, and how to make their behaviour more accurate & reliable by clarifying where required -- is very topical and of great interest to the NLP community.

The paper proposes a fairly simple and intuitive method, describes it clearly, and evaluates it sensibly (mostly -- see below), and results seem good in that they show reliably increased accuracy.
summary: The paper proposes a method for training LLMs in question-answering tasks to learn to ask clarification questions where appropriate given query ambiguity.
weaknesses: The evaluations and baselines compared to could be stronger or more meaningful in some cases. In the main experiment, one of the main takeaways from the results seems to be that answer accuracy is increased if a CQ is always asked, although this comes at the cost of more turns. This raises the question of how well a system would do if given a very simple fixed prompt which forced a CQ as the first response, and then an answer based on the resulting three-turn context - but without going through the extra training steps given here. I expect it wouldn't be quite as good as the best system here, but it would be very instructive to know how much difference there was.

In the experiment of sec 5.1 (determining whether CQs are necessary), performance is compared to a random baseline (sampling a random fixed percentage of direct answers vs CQs), but there's no direct evaluation of the main question here: how often a CQ was asked after an ambiguous question (i.e. accuracy of identifying when a CQ is required). In fact, the question is perhaps better: how often was a CQ asked after a question which was ambiguous, and where the possible answers differ (if the answers are the same, a CQ provides no benefit, even if the question is nominally ambiguous).

The data on possible CQs and answers, including answer correctness/preference, is mostly drawn from LLM-based simulations, but there doesn't seem to be any validation of how accurate or realistic this is.

There's no discussion of how ambiguity in a question actually relates to differences in answers. Ambiguity sometimes doesn't matter in practice: e.g. a variant of the first question in Table 8 (appendix) could be "what country were the first olympic games held in"; then, although there is ambiguity about whether the query concerns the modern or ancient games, that doesn't affect the correct answer which would be Greece in both cases. It would be interesting to know how common this is, and whether the behaviour of the models reflects (or should reflect) this difference.

The biggest weakness I perceived is the lack of relation to other relevant work. There's discussion of related work with LLMs and ambiguity, and LLMs being trained to ask CQs in different contexts; but there's no discussion of pre-LLM work that seems relevant. For instance, there is a large body of work in training dialogue systems to respond suitably to ambiguous input (including asking clarification questions), much via reinforcement learning. Closer to the domain here, if usually less close in terms of method, is the body of work in interactive information retrieval (including using clarifying questions to refine or further specify search queries or QA questions). Some discussion of how the method here resembles (or doesn't resemble) those would be very helpful.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How do you consider scenarios in which clarifying questions are undesirable behavior? Relatedly, how do you propose to balance clarifying questions with behaviors such as hedging, or presenting information for all possible intents?

2. Does the motivating problem of LLMs not asking clarifying questions get solved by prefixing with a simple instruction such as "ask a clarifying question if ambiguous"? For instance, I tried this with the example from Figure 1 and Gemini asks a clarifying question. To that end, such systems may be tailored to one type of behavior over another by design?

3. Have you considered multiple types of ambiguity (e.g. contextual versus lexical)?

4. Is the proposed approach intended to improve general LLM usage, or for this one particular task? if the latter, is this task with basic response outputs (e.g. single token) representative of real-world usage?

5. L161 is the model specifically fine-tuned to output "Clarifying Question:" (i.e. all generated responses should be accompanied by an intermediate dialogue act token)?
soundness: 3
strengths: ***Significance of setup***

The proposed setting is important to investigate. As LLMs are increasingly tuned for use in dialogue applications it is important to consider how to better address the needs of users, and disambiguation is certainly a reasonable idea.

***Ablations and analysis***

The paper presents experiments comparing the proposed approach on multiple base LLMs, and performs multiple reasonable ablations. The paper also considers and interesting examination on the effect of obtaining human annotations versus model predictions. 

***Clarity***

Overall the paper is clear and the experimental details are helpful.
summary: The paper proposes an approach which constructs preference data which are determined by examining (up to 2) simulated future dialogue turns. The proposed approach is evaluated in an open-domain QA dataset, NaturalQuestions, and a related derivative dataset, AmbigQA. The authors compare their approach against a standard single-turn reward model and present their results in terms of Token-based F1 for the QA task. The authors also present a set of experiments on ambiguity recognition.
weaknesses: ***Concerns regarding generalizability***

The focus and main contribution of the paper is that it considers a lookahead approach for future conversation turns in order to optimizing preferences in question-answering settings. However, the approach is only defined for at most two future conversation turns. It is not clear how 1. this may be generalized to longer rollouts and 2. whether this approach would remain effective in such a setting.

***Scope of experiments***

Relatedly, the domains examined are limited. The approach is validated on NaturalQA and AmbigQA. AmbigQA is an extension of NaturalQA, and as a result it is not clear nor trivial how the proposed approach would generalize to any other more complex and realistic domains (e.g., those requiring complex rollouts, or task-oriented settings).

***Limited baselines***

The baselines in the main experiments are somewhat limited. If I understand correctly, Direct Answer SFT and Clarify SFT both do not seem like fair baselines in a task where there is mixed ambiguity. How does the proposed approach compare against SFT on both types of responses? Moreover, the proposed approach focuses on using RL tuning combined with future turns. How does it compare against directly including those future turn trajectories? It would also be helpful to compare against other existing RL-based approaches for ambiguity detection/action planning (e.g. [1]).

There are also no baselines for the ambiguity recognition experiments. Does the proposed approach outperform prompting? The objective performance of the proposed model here seems rather low. If the model asks questions unnecessarily, it may lead to degraded user experience even if it successfully elicits additional information to solve the downstream task / provides additional context to condition on for answer generation.

*Minor Points*

typos:
L187 "their answer the proposed"
L300 "construct all examples SFT training"
L306 "to for queries"

References

[1] Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents, ICLR 2024

---

confidence: 3
contribution: 4
initial_rating: 5
presentation: 2
questions: - Line 164: Please clarify “We prompt GPT-4 to abstain from providing a clarifying answer a_i if it judges that none exists, in which case we count the resulting target answer prediction r_next^i as incorrect.”
- Line 245: What is “EM accuracy”?
- Lines 299: What does “if it determines one exists” mean and when can this happen?
- Line 312: Please clarify “generate its greedy answer with temperature T = 0.0 and sample an answer with T = 1.0”.
- Line 324: Which of the models below is used for the Turn 4 model?
- Lines 330—359: The purpose of each model and how they are used in the two-step pipeline is unclear.
- Line 342: What is the purpose of the Clarify DPO models?
- Line 347: More details about the reward model and the RL training are needed.
- Line 432: Why not training a model on the multi-turn dataset?
- Line 459: Why is the random baseline appropriate for comparison?
- Line 461: How did you set the value of DA?
soundness: 3
strengths: - Asking clarifying questions for ambiguous queries is a valuable trait for QA systems, and the paper’s proposed methods are well-motivated.
- The approach for assessing the quality of clarifying questions based on reachability to correct answers is novel, interesting, and well-reasoned.
summary: This paper addresses the challenge of asking clarifying questions for ambiguous user queries. To determine the optimal timing and content of a clarifying question, one of the core contributions of this paper is a dataset of multi-turn QA dialogues that include such questions. The paper also presents a method for evaluating the quality of clarifying questions. In this method, multiple candidate clarifying questions are tested using a user simulator, with the quality of each question assessed based on the aggregate reachability to correct answers. Models trained on this dataset in various settings demonstrate improved QA accuracy on both ambiguous and unambiguous queries.
weaknesses: - The primary weakness of this paper, in my opinion, is the lack of clarity in the method sections (2–4). One contributing factor is the use of terms without clear definitions. For example, in line 196, the term “annotators” is used, which seems to imply human annotators, while the preceding text suggests a “simulator” is involved. Additionally, the description of user simulation from line 208 onwards does not clearly explain the origins of *q* and *a_i*, making it challenging to follow. Although some of these ambiguities are clarified later in the paper, I would recommend reorganizing the structure to reduce the distance between concepts and their realizations. Further questions and suggestions are provided in the Questions section below.
- The paper contains numerous ungrammatical and broken sentences, which significantly impact readability. For instance, line 187 ("providing their answer the proposed"), line 191 ("catered a toward"), line 206 ("we use simulate user interactions"), line 212 ("using a we use"), line 300 ("examples SFT training"), line 307 ("to for queries"), and line 312 ("we a sample"), among others. Additionally, citations are incorrectly formatted on lines 087 and 097. I will raise my score if the authors thoroughly address the grammatical and formatting issues and their revisions in their rebuttal.
- Some experimental settings and design choices lack sufficient motivation. For example, why is asking clarifying questions preferable to simply providing multiple answers in one response (e.g., Q: "Where were the Olympic Games held in Greece?" → A: "The ancient Olympic Games were held in Olympia, and the modern Olympic Games were held in Athens.")? Moreover, there is no discussion on efficiency or how humans might perceive these clarifying questions, and a human evaluation might be valuable.
- There is no experiment involving training a model directly on multi-turn QA dialogues. Instead, the authors only experiment with combining the weights of two separate models. Since a multi-turn dataset is available, training and evaluating a model on this dataset should be feasible.
- In section 5.1, the comparison with the random baseline is questionable. A more effective comparison would evaluate whether the model asks clarifying questions when a query is ambiguous and refrains when unnecessary (e.g., based on the ambiguous and unambiguous splits of the dataset or cases where annotators provided consistent answers).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: The presentation of the paper has significant room for improvement. I have identified several typos and areas for enhancement:

- **Line 16**: Can you specify whether it is *evaluated*? I think it should be *trained*, (or both)?
- **Line 212**: "using a we use a trained user-simulator model" – this sentence does not parse.
- **Lines 224–226**: The green box is confusing. Why are q1 and q2 being preferred simultaneously?
- **Line 186**: “by providing their answer the proposed clarifying question” – this sentence does not parse.
- **Line 206**: "in this work we use simulate user interactions for annotation." – this sentence does not parse.

**Generic Questions:**

- Is your human annotation IRB-approved?
- In the methods section, can you write the loss function for DPO? This would make your paper more self-contained and eliminate the need for readers to refer to the original DPO paper.
- Can you provide more details on the evidence supporting the hypothesis that model responses are evaluated only on their prior contexts (Line 17)? This is a very interesting hypothesis (perhaps the central hypothesis of the paper), so additional justification would be beneficial.
- Line 305 presents an interesting hypothesis: "We hypothesize that what is ambiguous for an LLM often deviates from what is ambiguous for humans." Has this hypothesis been proven?
- Can you specify the input format for the LLaMA model? If my understanding is correct, only the Chat version includes the system and user specifications.
- What is the computation time and the number of training samples used?


Building on all my comments, I'd vote for a 5/10 score now :)
soundness: 3
strengths: - The paper provides a thorough scientific investigation, building upon several interesting hypotheses about LLM responses by training (evaluating) on prior data (Line 17) and speculating about the differences between LLM ambiguity and human ambiguity (Line 305).
- The paper introduces a novel pipeline for eliciting human preferences in multiple rounds of conversation, even though I have some reservations regarding its operation.
-  The experiments are very thorough, considering different variations of LLMs. However, the study lacks comparisons with non-LLM methods.
summary: This paper examines handling ambiguity, an often overlooked aspect of large language models (LLMs) in multi-round conversations with users. The authors first develop a pipeline to collect human preferences in a two-round setting and then apply Direct Preference Optimization (DPO) to several open-source LLMs to enhance their performance.
weaknesses: ## Weakness 1: Unclear Relationship Between Human Annotators and the User Simulator

The paper appears to ask humans to only grade the final prediction of the model (the final step as shown in Figure 2). It is unclear whether their grading aligns with their judgment of the clarifying questions. It would be more appropriate to set a predefined user goal at the beginning and have human annotators grade the final answer based on its consistency with the original goal and the arrangement of questions and answers in the intermediate steps.

I recommend that the authors refer to the conversational information-seeking literature in the IR community. For example, the paper [1] adopts a user simulator that maintains an initial item of interest (e.g., iPhone 16 128GB). During the question-answering stages, the user simulator responds to questions in line with the item's specifications (e.g., "How much storage do you want?" "128GB"). In this paper, it is unclear whether the human's final grading considers an initial goal and whether the grading assesses both the answer quality and adherence to that goal.

This is my major concern, and I hope the authors can provide further justification during the response period. Thank you!

## Weakness 2: Lack of Baselines Beyond LLM Fine-Tuning

While the paper includes numerous ablation studies and comparisons with other LLM tuning baselines, additional comparisons are needed to justify the method's effectiveness. I recommend the following:

- **Unsupervised Beam Search**: This approach allows LLMs to search the space of possible clarifying questions (implicitly) [2].
- **Chain-of-Thought**: Prompting LLMs to explicitly reason about possible ambiguities.
- **Non-LLM Baseline Methods**: Including methods for asking clarifying questions that do not rely on LLMs, as explored in various works [3, 4].

I hope the authors can try to add these results or explain why these methods are not suitable for comparison. Thanks!

## Weakness 3: Need for More Insightful Analysis

- **Knowledge Taxonomy**: A taxonomy for knowledge types is needed to make the dataset more insightful. What types of knowledge are more desirable for clarifying questions? If the original dataset does not provide this, consider latent word clustering or LDA?
- **Error Accumulation**: Correct me if I am wrong, but I do not see an analysis of how an incorrect clarifying question may lead to error accumulation in the final answer.

I hope the authors can discuss with me if these research questions are helpful and whether they plan to append them. 

# References
1. [Conversational Recommender System](https://arxiv.org/abs/1806.03277) SIGIR 2018  
2. [Self-Evaluation Guided Beam Search for Reasoning](https://arxiv.org/abs/2305.00633) NeurIPS 2023  
3. [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://arxiv.org/abs/1907.06554) SIGIR 2019  
4. [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://aclanthology.org/2023.findings-emnlp.772/) EMNLP 2023

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. What is $a_i$ in line 120? It seems that it has a different meaning from ($a_i$) in line 129.
2. Are the input query $x$ and the user intent ($x_i$) supposed to be different forms?
3. Is the user-simulator model for preference data construction different from the the user simulation model in Section 2.1? 
4. What not apply Clarify-or-Direct-Ans SFT as a baseline?
5. How is the original performance of these LLMs?
6. How about just using in-context learning for enabling the clarification capability?
7. Is there any existing method that can be applied as baselines, instead of all your self-designed baselines?
soundness: 2
strengths: 1. This work studies an important question in the era of LLMs, i.e., teach LLMs to ask clarification questions
2. Propose a novel approach to construct preference data including clarifications by simulating future conversations
3. Experimental results show that the proposed method outperforms other standard baselines.
summary: This paper focuses on the problem of having LLMs ask clarification questions when the user request is ambiguous or uncertain. The authors propose a new approach to construct preference data by simulating future conversations. Experiments are conducted in two open-domain QA datasets. The results show that the proposed method outperforms other standard baselines.
weaknesses: 1. The presentation of the proposed method is a bit hard to follow. There is a lot of symbols and variables in the text, which is very messy and not well defined. More details in the questions that need to be clarified by the authors.
2. The information of annotators for preference data construction is also not clear. 
3. The experimental settings are not extensive and reasonable enough, only comparing on self-designed baselines.  
4. Several recent studies on LLM-based clarifications are not discussed or compared, such as [1-3].

Minors: The usage of \cite{} is incorrect.

[1] STaR-GATE: Teaching Language Models to Ask Clarifying Questions
[2] STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents
[3] Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training

Meta-review: This paper proposes to ask clarifying questions to the users to elicit more information from them in the case of ambiguous questions for LLM-based question answering. Experimental results show 5% improvement in F-measure with an LLM fine-tuned to ask clarifying questions. Reviewers highlight the simplicity and meaningfulness of the idea as the strength of the work. The weaknesses listed include many questions that aim to clarify experiments and decisions made in the work, and its positioning with respect to earlier work.

Decision: Accept (Poster)