Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions
Authors: Michael JQ Zhang
W. Bradley Knox
Eunsol Choi
Venue: ICLR 2025 Poster (2025)
Keywords: Clarifying Questions
QA
Ambiguity
RLHF
Primary area: foundation or frontier models, including LLMs
TL;DR: We RLHF train LLMs to ask clarifying questions in response to ambiguous requests by assigning preferences based on their expected outcomes in future turns.

Abstract: Large language models (LLMs) must often respond to highly ambiguous user requests. In such cases, the LLM's best response may be to ask a clarifying question to elicit more information. Existing LLMs often respond by presupposing a single interpretation of such ambiguous requests, frustrating users who intended a different interpretation. We speculate this is caused by current preference data labeling practice, where LLM responses are evaluated only on their prior contexts. To address this, we assign preference labels by simulating their expected outcomes in future turns. This allows LLMs to learn to ask clarifying questions when it can generate responses that are tailored to each user interpretation in future turns. On open-domain QA datasets with multiple annotations, we evaluate systems based on their ability to ask clarifying questions to recover each user's interpretation and expected answer. We compare systems trained using our proposed preference labeling methods against standard methods, which assign preferences based on only prior context. Our method achieves a 5% improvement in F1 measured against the answer set from different interpretations of each query, showing the value of modeling future conversation turns. We further demonstrate that our method can be used to train models to judiciously determine when to ask clarifying questions, directly answering the question when clarification is unnecessary. In our experiments, we find that our method achives a 3% improvement in accuracy of such judgments over existing methods.

Published as a conference paper at ICLR 2025
MODELING FUTURE CONVERSATION TURNS TO TEACH LLMS TO ASK CLARIFYING QUESTIONS
Michael J.Q. Zhang"
W. Bradley Knox", Eunsol Choi
New York University The University of Texas at Austin
mi michael cchhellzhang@@nyu.co
ABSTRACT
Large language models (LLMSs) must often respond to highly ambiguous user re- quests. In such cases, the LLM's best response may be to ask clarifying question to elicit more information. Existing LLMs often respond by presupposing a single interpretation of such ambiguous requests, frustrating users who intended differ- ent interpretation. We speculate this is caused by current preference data labeling practice, where LLM responses evaluated only their prior contexts. To ad- dress this, we assign preference labels by simulating their expected outcomes future turns. This allows LLMs to learn to ask clarifying questions when it generate responses that are tailored to each user interpretation in future turns. On open-domain QA datasets with multiple annotations, we evaluate systems based on their ability to ask clarifying questions to recover each user's s interpretation and expected answer. We compare systems trained using our proposed preference la- beling methods against standard methods, which assign preferences based on only prior context. Our method achieves a 5% improvement in measured against the answer set from different interpretations of cach query, showing the value of mod- cling future conversation turns. We further demonstrate that our method can be used to train models to judiciously determine when to ask clarifying questions, directly answering the question when clarification unnecessary. In our exper- iments, find that method achives 3% improvement in accuracy of such judgments over existing methods.
INTRODUCTION
Ambiguity is hallmark of natural language that enables concise communication by allowing speak- ers to exclude details that are inferable from the context (e.g., conversational, temporal, geographi- cal) (Piantadosi et al.) 2012). At times, however, the speaker's intent is unclear despite the context, and further interaction is necessary to clarify their intent. Asking clarifying questions is particularly important for large language models (LLMs), which tasked with serving a wide audience, often without access to the personalized context available in human interactions. In this work, we develop LLMs that can ask clarifying questions to resolve ambiguity in their users' requests.
State-of-the-art LLMs (OpenAl 2023 Gemini Team 2024) often do not ask clarifying questions when presented with an ambiguous request, and instead respond directly by assuming the user's intent (see an example in Figure Tp. We speculate that this tendency stems from a shortcoming in their RLHF training pipelines, which utilize annotated preference data (Ouyang et al.) (022). In standard preference data collection, annotators are given a conversation history and are tasked with ranking options for the next assistant turn (Bai et al. 2022, Wang et al., 2024). These annotation schemes only consider preferences over single-turns of interaction, making it difficult for annotators to assess the utility a clarifying question. Furthermore, this can lead to biases where annotators prefer responses with complete but presumptuous answers over incomplete clarifying questions.
We propose an alternative method for annotating double-turn preferences over clarifying questions where annotators interact with the LLM by providing their responses clarifying questions and ob- serving the LLM's subsequent responses. Each annotator then assigns preferences based on whether the completed interaction successfully fulfilled the request. When comparing multiple clarifying

---

Published as a conference paper at ICLR 2025
[Trrn ||
User's Input Query: x
Users [1,",,",}}
User Intends ask: Who is the highest paid foothall player t
Who highest paid Booballl player?
User u, Intends to ask: Who the highest paid football player + Expects Answer: y Jor Rammay
Expects Answer: y Cristanno Ronalde
User u, Intends to ask: Who is the highest paid foothall player e Expects Answer: y, Cristinno Ronalda
Turn 21 LLM (M) Prediets
Single-Ture Turn Responses: M(x) F
[A] Direct- Answer (
[8] Direci-Anseer (+))
[C] Clarifying Question (Q-rr-()7)) Are you nsking about American Foutball or Soccer?
[D] Charifying Question ?):
As 2014, the
The highest
T... Are you taking about speciffic time period?
highost-puid fouiball player
football player in the NFL for
are Cristian Ronaldo...
2024 is loe Burnow.....
( False
Kir = -False
(r = True
nr. = True
IAI
IBJ
[C]
[D]
xr.) = True, then continue interaction
Single- Turn
Preferences
User
X
A
X
Majority Best Respouse: [A]
Users 3,3
X X
[uru Users Respond
a--Aoou
- About C
Currently
Clarifying Answers: y, (
AAbout +
a, Currently
d Curreatly
[Turn LLMs Prediet Next
pf Ine Barmun.
Response: M(x, "ix. a)=
Cristiame Romaldu...
Cristiano Ronablo.
[AI
[B]
[C]
[D]
Evaluation Metrics
[A]
[BB]
[C]
[D]
Preferences
Double-Turn
User
X V
V
X
Response: [C]
Majority Best
Efficiency of Turns)
2
Users 2,
(R, 33,3)
0.8
0.5
1.0
0.5
Figure 1: Our interaction scenario and preference labeling schemes. We aim to build an LLM that can interact with user generate the final answer set R, containing an answer for each user, for the input query æ. In this example, we include two responses from state-of-the of-the-art LLMs ([A] from GPT-4 and [B] from Gemini, full responses in Appendix [ which both presuppose an interpreta- tion of the word football. We also include two clarifying responses ([C] and [DJ) where [C] correctly disambiguates the two intended interpretations across all users. We depict two ways to assign pref- erences on LLM's 's initial output, single-turn and our proposed double-turn.
questions and direct answer responses, each annotator's preferences are aggregated to identify the response that maximizes preference across annotators. We depict our proposed double-turn pref- erence annotation method and compare against standard methods in Figure
To demonstrate the benefits of training LLMS to ask clarifying questions with double-turn prefer- ences, we experiment open-domain QA (Chen et 2017). We develop automatic evaluation framework for evaluating clarifying questions that use simulated user interactions (Section 2 We evaluate systems two axes: efficiency measured by the number of model turns and effectiveness measured by FI score between the predicted answer set for each user and their expected gold answer.
Our experimental results show that training systems with double-ture preferences outperforms train- ing with standard preferences annotation methods (Zhu et al. 2023) resulting in consistent 4-5% improvement in F1 score over three base LLMs (Dubey et al.
2024 Mesnard et al.) 2024). We also demonstrate that double-turn preferences can be used to train systems that determine if clarification needed, if the user's question can be answered without additional interaction.
We summarize our contributions below:
We establish automatic framework for evaluating double-turn interaction involving clarifying question. Our framework includes a user simulation model and automatic evaluation metrics measuring the system's 's ability to predict outputs that are tailored to each user efficiently.
We develop double-turn preference annotation method for labeling preferences over clarifying question and direct response that utilizes the outcomes in later turns of interaction.
Our results Open- Open-Domain QA datasets (Kwiatkowski et al. 2019, Min et al. (020)) demon- strate that training systems to ask clarifying questions using our double-turn preferences improves QA performance over training with standard single-tune turge--trrnnnne preferences. We also show that training with double-turn preferences can teach models to judiciously decide when to ask for clarification.
Our evaluation paradigm and preference labeling scheme further support future research in de- veloping interactive LLM assistants. We release all code and data at hhtps:: / h/g/tttth com/ mi mike jiieanggagg//// ari fyi
2

---

Published as a conference paper at ICLR 2025
2 TASK: QA WITH SIMULATED USER INTERACTION
We chose to study open-domain QA, as it is both prevalent in real user-assistant interactions ZZhao jet al., 2024) and the scope of necessary clarification questions is not immediately identifiable from the input or the task. For simplicity, we choose a well-studied short factoid QA setting (Kwiatkowski et al. 2019; Min et al. (020)) where an answer to the question consists few tokens. Below, we describe our task and evaluation metrics. F contains example of task scenario.
2.1 INTERACTION SCENARIO
Setting Goal:
We assume an input query æ, which can be potentially ambiguous, and set of users Up, who posed the input query z. Each i-th user u; has an disambiguated intent for query (xr,) and an expected gold answer (y,). Our goal to build a system M that returns an answer set R that can satisfy as many users as possible (by providing the expected answer ys for each user u.) with the minimum number of interactions (measured by the number of model turns).
We will have simple function of (x)) that returns True if M (x:) is clarifying question, and False otherwise. We further assume an oracle user simulation model vi( Tenas v:) = a, that takes first turn input x, model's initial output M Mxx,, and i-th user's targeted answer yı, that generates response to the model's initial response (x).
In the initial turn, user poses the input query In the second turn, given the input query æ, the system M is to generate its initial response (Tini). Depending on whether Tini is a question, it will trigger different dialogue trajectories.
Dialogue Turns:
Double-turn Scenario If the initial response is a clarifying question (d(rim))=True) we continue the conversation with the user simulation model. For each of i-th user, the evaluation system generates a third turn (a.) with the user simulation model 10 Finit: y1) ag- For the fourth turn, the system will take each of the k interaction traces (xr, Tinit: a4) generate the fourth turn response for i-th user M (x, Tinit: 0;) = = Tnext- The final output answer set from the system will be a set of i answers R = {Freett 2 Thex:e....nnee where each next is a response for i-th user. Single-turn Scenario Otherwise if (((imin==alsse as the final prediction. To allow a model that does not generate clarifying questions to generate multiple answers, we sample multiple answers from the model, up k: answers. The final answer set in this case will be R (rimi (r.m..... m.
will consider the model output R (rinit
Existing QA models mostly generates single answer to the initial input query x, even when the query is ambiguous. Such systems that predict an answer without first confirming the user's intent may mislead users with incorrect answers and fail to serve the entirety of their user population. In this work, we explore methods for QA-with- wiith-Clarification, where models can ask the user a clarify- ing question q and observe the user's response before predicting answer. system can also sometimes poses a clarifying question and sometimes directly answer (Zhang & Choi 2023a).
We evaluate the system in axis: efficiency and effectiveness. To measure effi- ciency, we simply measure the average number of total turns from the system. For unambiguous inputs, asking clarifying question is unnecessary. If the system can satisfy users without incurring additional interaction, such system should be preferred.
Evaluations:
To measure effectiveness, reference answer set ((31,..)))) When evaluating enforce that answers must be correct for their corresponding clarifying answer. We hold QA-with- Clarifications to this higher standard, as it aligns with our goal of to developing systems that can identify different user interpretations of a query and accurately predict corresponding answers.
F1 between the model generated answer set (R) and the gold
wwwithhharriiiaatiinn systems, we additionally
2.2
EVALUATION FRAMEWORK IMPLEMENTATION
Data: We will leverage existing open-domain QA datasets (Kwiatkowski et al.) 2019 Min et al. 2020) where each query is paired with annotated answers from multiple annotators.
Identifying clarifying question o(-): We will use a very simple method, which takes the input string and check whether it starts with "Clarifying Question:".
3

---

Published as a conference paper at ICLR 2025
User Simulation Model a for Turn 3: We use GPT- as our oracle user simulation model (1)). We prompt GPT-4 with the input query 2. clarifying question q. and target answer Vi and task with generating corresponding clarifying answer q. a;). We prompt GPT-4 to abstain from providing clarifying answer a; if it judges that none exists. This happens when the clarifying question does not address the ambiguity in the query, hence count the resulting target answer prediction next as incorrect. We further ensure that the generated clarifying answers do not leak the target answer by removing instances where the target answer y: appears in the predicted clarifying answer (2 treating such examples the same as abstains from GPT-4 (full prompt in Appendix (C)
FINE-TUNING LLMS TO ASK CLARIFYING QUESTIONS
We present our approach to build a model that can engage with users for multi-turn interaction, ask clarifying question when it can satisfy more users in the future turns. Following the standard RI training pipeline (Ouyang et al. (202)) we first construct an instruction tuning dataset focusing on clarifying responses as desired outputs. We then use this constructed dataset for supervised finetun- ing (SFT), before performing preference learning between two possible responses. We first describe our method for constructing this instruction tuning dataset before introducing our annotation scheme for assigning preferences over clarifying questions.
3.1 PREFERENCE DATA GENERATION FROM USER-LLM INTERACTIONS
Our method for labeling preferences deviates from standard annotation methods (Bai et al. 2022 Wang et al. 2024) in two ways. First, we simulate an additional interaction with the user and derive reward from the LLM's final response after the additional turn. Second, we expect multiple anno- tators for each example. By using multiple human annotators to identify a set of expected answers, simulate each annotator's interactions with the LLM and their preferences. While prior work has explored using multiple annotator judgments in single-turn preference datasets (Wang et al., [024] Kopf al. 2023). these works aggregate annotator judgments via majority choice generate single preference label per example. This practice, however, can lead to adverse outcomes where systems whose outputs are catered toward a single, majority accepted response (Fleisig al. [023] Santy et al.] 2023). In contrast, we use preferences across multiple annotators without removing individual annotator judgments, helping us identify whether a clarifying question or direct-answer response successfully accommodates the range of different annotator's 's interpretations.
In our annotation scheme (depicted in Figure D. annotators are provided an input query æ with sev- eral candidate clarifying question g and direct-answer responses y. They respond to each clarifying question by providing the clarifying answer that corresponds to their interpretation. We then show annotators the model's 's final output prediction, given each clarification. Annotators then assign their preferences over clarifying questions and direct-answer responses based on whether the final out- put matched their expected interpretation. We then aggregate these preferences to identify which response was preferred by the most annotators.
To prevent systems from asking unnecessary clarifying questions, ties between direct-answer and clarifying question responses are broken by favoring the direct-answer response. The final aggre- gated preferences ences for training. While this annotation scheme can be applied to human annotators, in this work we use simulated user interactions for annotation, simulating different users' interactions from existing datasets with multiple annotations (discussed below in Section
then be used in conjunction with standard RL methods from human prefer-
Simulating User Interaction To simulate different user interpretations of the input query. we utilize datasets consisting of an input query x and a set of correct answer outputs (y1,.., Mt.). For each query, we use the set of gold outputs to simulate & different users, one for each answer in (a1...,,y' }x. which are identified by multiple human annotators (we discuss how these annotations are sourced in Sec We then simulate the behaviors of each of these users and their responses to proposed clarifying question g using a trained user-simulator model, which takes input query I, clarifying question q. and the user's 's annotated answer Yi and is trained to predict the user's clarifying answer (x. 9. y1) a,, While we use prompted GPT-4 to simulate clarifying answers from users for evaluation, during training we use this trained user-simulator model to reduce costs and to ensure that our systems do not overfit particular user-simulator method.
4

---

Published as a conference paper at ICLR 2025
[T1] Users ask Inpet Quety (c)
[T3] Simulate each Clarifying Question -y User's Clarifying g.J,) -ta,
[12] LLM generales
[T4] LLM predicts
Users Judge [12)
Output:
Options by compariing EM(J,-))
or Disect-Answer (x) -
Ux 9,)
Inpur Query
x= How manyy
User Clarifying All sports.
Ouiput: 8,-121
(Uxpects y, lend to 121
46)
modals did the van the 2016 olymptes?
Clarifying Question: qf Which sport are you usking abowl?
to
User Clarifying, a.-All All sports.
Output: F:-12/
User 2 (Expeets y, .22)
User Expected Answer:
q' lead to
g' lead to 112
User 2 Expected y
Clarifying Question: Are osking about all medalss or medalls?
User Clarifying, Gold Medals.
Output: 8,746
Aggresated Rankings
User prefers q"
User Clarifying a,-All Medals.
Ourput: 8:-121
Both Users prefer g" peeferred overall
12/
How many bridges in the city pittsburgh?
Clarifying Question: - Are you asking about all bridges only ones curren! use??
User Clarifying A: a, All briddges.
Outputt 8,4466
User (Expects y,
122)
lead to 0,= 446
lead to 444
User Expected Answer:
User 2 Clarifying A a,- bridges.
Output: y 4446
(Expects y-
q' lend to 6,46
446
Direct-Answer:
to 9 -166-
8 446
Expocted
Appresuring Rankings
Answer: y,- 446
Both Users prefer and Ties favor the Direct-Answer Response. y is preferred overall
Figure 2: Depiction of preference annotation method. Here, simulated users provide their responses model-generated clarifying questions and determine preference based on which clarifying question or direct- answer responses lead to their expected answer. We then aggregate preferences across users by selecting the response that is preferred by the most users while minimizing the number of user interactions turns.
Answer Prediction / Assigning Preferences Conditioning on the simulated clarifying answer, our LLM model predicts the output yi, (z,q.a,) yi- As a last step, we aggregate annotator preferences so that clarifying questions are ranked by their EM accuracy, evaluated against each annotator's expected answer: Σ EM(U: /tk/ To rank direct-answer responses, which only predict single answer J. similarly evaluate the EM accuracy of the predicted answer against each annotator's expected answer. To prevent systems from asking unnecessary clarifying questions, favor the direct-answer response in the case of ties. We then use these aggregated rankings to determine the pairwise preference labels between responses for training.
4 EXPERIMENTAL SETTINGS
Models We use Llama2-7b Touvron et al. 2023), Gemma-7b (Mesnard et al. 2024), and Llama3- 8b (Dubey et al. 2024) as our base LLMs. We forego using the instruction-tuned models due to data leakage concerns, as such systems are finetuned on the entirety of NQ-Open. For training and inference, use 8-bit quantization (Dettmers et al. 2022 with LoRA (Hu et al. 2022 Dettmers ct al. 2024) (training details in Appendix
Data We perform our experiments on the NaturalQuestions (NQ-Open) (Kwiatkowski al. 2019
Lee et al. 2019) and AmbigQA (Min et al. 2020) datasets. In both of these datasets, each input query are associated with a possible output answers (.I,...,, y. We describe each dataset below, and provide details in Appendix A
NQ-Open is comprised of questions from Google search queries by real users. Each query is annotated with answers from Wikipedia by up to five annotators: many questions (about 10% of train and 42% of development set) in the dataset contain multiple distinct answers based on each annotator's judgment in interpreting the query and selecting the best answer span. The answers sets identified by annotators for such examples can stem from a variety formatting to ambiguities resulting in semantically distinct answers (examples in Appendix AA). AmbigQA additionally annotates a subset of NQ-Open whether the question is ambiguous. the input ambiguous, annotators then provide the additional answers other possible interpreta- tions of the query. This process identifies about 56% of all queries in NQ-Open as ambiguous, and recovers 2.6x answers to ambiguous questions.
ambiguities, from
5

---

Published as a conference paper at ICLR 2025
Supervised Finc- Tuning Data
Doable vS.
Clarity DPO (Likelibood, Match)
Clarify: (x) -
Clarity SFT
Direct y
Stagle turn searan
Clarify DPO (RM)
Aln-Affer-laarify:
Base LLM
Direct Ans SFT
(A. q. 0) yy
(x. q. u
Charify-r--Direee Ans SFT
sourieg
Charify-r--iiee Direct DPO
Responses Preference Learning
Clarity Responses
Clantyy SFT Medel
Aler-Cla-ll
Usur-Simuater
SFT
Answer Responses
(x) y From Direct Model
Illustration of models in our study (right) and the data used for training them (left). Ans-After- Clarify SFT model is used to generate responses for the fourth turn, User-simulator SFT model is used to generate responses for the third turn. All other models generate responses for the 2nd turn.
Figure 3:
We evaluate on AmbigQA test set (n=1960). addition to evaluating over the Full test set, we ad- ditionally report performance on splits of Unambiguous questions (788 questions with one answer each) and Ambiguous questions (1,172 questions averaging 3.7 answers each).
Supervised Finetuning (SFT) Dataset Generation To generate examples for SFT training, we prompt GPT-4 with an input query x and a set of feasible answers Y (y:....: ysk and task with proposing clarifying question q that will help determine which answer the questioner expects. Our prompt also tasks GPT-4 with predicting responses to the clarifying question for each one of the feasible answers (1e,,1)...-. ..... (ax, yk yk)}, if it determines one exists. We the same GPT-4 prompt to construct all examples SFT training, and introduce two methods for generating feasible answer sets Y for given query œ.
Answers from Human Amootatioss: We use the ambiguous queries from the AmbigQA training and development splits and their annotated answers as the feasible answer Yuman- Identifying ambiguities and labeling all possible answers is a challenging task for annotators, and thus this data can be expensive to collect. Furthermore, we hypothesize that what is ambiguous for an LLM
often deviates from what is ambiguous for humans.
We build feasible answer sets Ymodel for queries in the NQ- Open training set using answer candidates proposed by base LLM. Prior work has demonstrated that base LLMs are well calibrated in open-domain QA tasks with few-shot prompting (Kadavath) et al. 2022 Cole et al. (002)). Drawing on these observations, we construct feasible answer set model by sampling 5-shot predictions from Llama-2-7b (Touvron et al. 2023). For each query, we sample 5-shot prompt and generate its greedy answer with temperature T 0.0 and sample an answer with T 1.0. We repeat the process 10 times per query and remove repeated answers to generate set of model-identified feasible answers. We further filter examples where none of the feasible answers match any annotated answers from NQ-Open.
Answers from Model Predictions:
For each method of generating feasible answer sets human and Ymodet), we generate an SFT dataset of 4,400 input query and clarifying question pairs (Ir, g). which we split into training (4,000) and de- velopment (400) splits. Between both datasets, this gives us a total of 19,807 (x,, q, 04y y:) examples. In Appendix A we include examples and the exact prompts.
4.1 COMPARED SYSTEMS
Figure 3 summarizes all models used in our experiments. We have two types of models, model that will be used at the second turn (which we refer as Turn 2 model) and model that will be used at fourth turn (which we refer Turn 4 model and notate
Sans-After-Clarffyy model in the figure). The model that will be used at the second turn can ask clarification question or directly answer the question. All models that are capable of asking clarification questions paired with the same Turn model, which is trained over all generated SFT examples. We use a separate model for performing this latter step to ensure that differences in performance only reflect the value of the

---

Published as a conference paper at ICLR 2025
clarifying question, and explore training joint models for both steps in Section S We describe each of these below.
Direct Ans SET This model is trained to predict the answer given the imput query (x) y. We fine-tune base LLM with next-token prediction loss on the full set of question answer pairs (i.c., each (x, v:)) in our generated SFT training sets. We use this model under two inference settings: (1) Greedy where we take the single greedy answer prediction and (2) Sampled where we sample predicted answers from the LLM. We sample 20 answers from the model and select the top most frequent answers, where matches the gold number of annotated answers
Clarify SFT This model is trained to ask clarifying question to input query. We finetune base LLMs on our SFT datasets constructed with both human annotations and model predicted answers. In Section 5 we provide ablation experiments training over each dataset to compare their utility.
Clarify DPO (RM, Likelihood, Match) These models are each fine-tuned from Clarify SFT model with using DPO (Rafailov al., 2023), method for optimizing LLMs preference data. It uses the following loss for updating the policy LLM Ty based on the base reference policy Tref. prompt h, and preferred/reeetedd responses (Tp: r,):
Te(Tplh) 3 log (ryl(1
To (r(
(To, Frafi pp Tr, h) = logo log
Fref Fref(r, 1h)
where 33 is a hyperparametter and (r|hh the predicted probability of generating given the prompt h. These models are provided the same response pairs sampled from Clarify SFT model only differing in how reward assigned.
RM: This uses off-the-sheeff reward model to rank clarifying questions. We use the Starling- RMM7B-alppa reward model (Zhu et al., 2023) which was finetuned on over 3.8M single-turn preferences between the outputs from variety of LLMs.
Likelihood: In this method, clarifying questions are ranked by the summed likelihoods of gener- ating each gold answer after observing the simulated users' clarifying answers. Using likelihood to score questions has been proposed in prior work studying clarifying question generation to identify different user personas (Andukuri et al.) (024)..
Match: We rank clarifying questions by whether the predicted answer matches each user's gold answer after observing each simulated user's clarifying answer, averaged over all simulated users.
Clarify-or-Direet Ans SFT / DPO We train Clarify -Drrreect Ans SFT model the union of all data used to train our Clarify SFT and Direct-Ans SFT models above. To train our Clarify-or-DDe -D-rrccct Ans DPO model, wefurther train our Clarify-or-Direct Ans SFT model above using double-turn preferences over clarifying question and direct-answer responses with the DPO learning objective. To rank responses, we use answer matching as depicted in Figure 2 pairwise comparisons from the same six candidate clarifying questions generated for our Clarify DPO models above and the single greedy sampled direct-answer response generated from our Direct Ans SFT model above.
We perform DPO training over
RESULTS
We report main results in Table [ capturing the number of user turns in the conversation (# L) and Answer F1. Overall, systems that are equipped to ask clarifying questions outperform Direct- Ans models in effectiveness (Answer F1). Clarify-or-Answer methods strike balance between effectiveness and efficiency.
Training with our double-turn preference labeling scheme consistently achives the best answer Fl, with our match method yielding the best results overall. We observe mixed results from training with single-tur turn RLHF annotations from the Starling RM: yielding positive results with Llama2 as the base LLM and minor increases/decreesse in performance for Gemma and Llama3 when evalu- ated over both ambiguous and unambiguous queries. These results support our claim that single-turn
In rare cases (< 2%), there are less than unique answers within the 20 samples.
2 Wee sample six candidate clarifying questions (one with T = 0.0 and five with T 1.0) and form prefer- ence pairs among them.
7

---

Published as a conference paper at ICLR 2025
Table I: Main results, separated by systems that never clarify (# Turns = 1), always clarify (# Turns 2), or can do both (# Turns E [1, 2]). The shaded blocks represent systems that simulate future turns. Our method (Clarify DPO w/ Match) yields the optimal results in Answer Fl at the cost of always asking clarifying questions. Our method that selectively asks clarifying questions (Clarify- or-Direct-Ans DPO strikes a balance between efficiency (# Turns) and effectiveness (Answer FI). Bold results denote statistically significant differences from all other methods with the same base model (over N 10, 000 samples with < 0.01).
Llama2 Answer F1 (T)
Llama3 Answer Fl (1)
Gemma Answer FI (1) Unamb / Amb / All
# (L)
(1)
Unamb / Amb / All
Unamb/ / Amb / All
(1)
Direct-Ans SFT
w/ Greedy
1
25.4/ 16.//11./2/1 21.1
!
31.2/ 24.8
26.1 / 16.8 / 21.1
w/ Sampled
1
25.0 17.0/172/214
1
28.21
24.7
1
23.7 17.9 21.4
Clarify SFT
2
31.0/21.6/ 25,9
2
37.6/ 26.5 31.5
2
35.7 23.6 28.8
Clarify DPO
w/ RM
2
31.0 28.3
2
36.2/ 30.9
2
/ 29.5
w/ Likelihood
2
30.2/239//72 27.2
2
43.5/29.65 359
37.3/ 26.3/26.87 31.5
w/ Match
2
38.3 / 28.2
32.8
2
42.9/ 3.17 / 36.5
2
40.7 28.6 33,9
Clarify-or caee--r--Direeee Direct-Ans
SFT
1.12
25.6 / 1
1.40
28.2
1.43
22.3 / 19.0 20.3
DPO
1.56
28.9/21.1 24.3
1.57
35.2
25.1 29.1
1.61
28.2
22.2 24.6
Table 2: Answer Fl results comparing training Clarify DPO (Ours) systems using SFT training on clarifying questions generated from Y human or Y model answer sets or both.
Llama2 (7b)
Llama3 (7b)
Gemma (7b) Amb
SET Data
Unamb
Amb
All
Unamb
Amb
All
Unamb
All
Human Model
28.8
23.6
25.9
40.1
30.9
34.9
37.9
27.4
32.3
29.9
26.4
28.3
42.3
31.2
36.4
39.1
28.3
32.9
Both 38.3
28.2
32.8
42.9
31.7
36.5
40.7
28.6
33.9
RLHF annotation struggles to identify useful clarifying questions. In contrast, training with double- turn preferences, where clarifying questions are assessed based on their later outcomes, demonstrates consistent improvements in the LLM's ability to generate useful clarifying questions, on both am- biguous and unambiguous queries. This demonstrates that clarifying questions not only help models disambiguate user intents for ambiguous queries, but also can help models recover correct answers in general even for unambiguous queries. We include such example generations in Appendix D
Ablations: Comparing Clarifying Question Generated with Human Annotated Answers Model Predicted Answers So far, we performed SFT training on the union of clarifying questions generated from model-identified (Ymode)) and human-identified (Y (Ynuman) feasible answer sets. Is one of them more useful than the other? In Table 2
we compare performance using SFT examples constructed from our Y mede! and human answer sets, after RLHF training with double-turn preference.
With Llama2 base model, we see clear improvements when using our model-ambiguity SFT dataset over using our human- ambiguutt dataset alone. These improvements even hold when evaluated against the answer sets from AmbigQA, despite the fact that our human-anbiguity SFT dataset was generated to distinguish between answers labeled in the AmbigQA training split. However, on Gemma and LLama3, we find only minor improvements from training with model-ambigut dataset. The model-ambiguity dataset was using feasible answer sets Ymodel from Llama2 predic- tions. This gap suggests that constructing ambiiutmyy dataset that is specifie to the base model improves the resultant clarifying question quality.
SFT
A Joint Model for Asking we experiment with using sin- gle LLM for both the clarifi- cation (r
Clarifying Questions and Predicting Answers Here, Table 3: Answer Fl results comparing separate versus joint clar- ifying question and answering models (CI/ Ans).
q) and answer y:) conversation
(x,q, v. 0; turns. To create this joint model, we take inspiration from prior works demonstrating that aver- aging the weights of multiple finetuned models is simple and effective approach joining the capabilities of finetuned mod-
Gemma (7b) Unamb Amb All Unamb Amb All Unamb Amb All
Llama2 (7b)
Llama3 (7b)
Separate Joint
38.3 28.2
32.8
42.9 31.7 36.5
40.7 28.6 33.9
32.4
25.0 28.7
41.0 29.2 34.6
36,2 27.4 31.4
8

---

Published as a conference paper at ICLR 2025
We merge the LoRA parameter updates from our clarification model (trained with preferences from full interactions) and the answer prediction mod- els used in our main results (Table I
els into single LLM (Wortsman al., 2022).
Table 3 compares the joint model against using separate LLMs for each conversation turn. Overall performance degrades when using joint model for both turns; however, we are still able to retain most of the performance gains over Direct-Ans and QA-witt-CCarification methods trained with single-turn preferences. Future work may explore alternative training methods that mitigate the performance degradation, such as using mixture-of-experts methods for LLMs (Jiang et al., 2024) or improved multi-tum learning algorithms (Zhou al. 2024b).
5.1 DETERMINING WHEN CLARIFYING QUESTIONS ARE NECESSARY
We further investigate system's ' ability to identify when clarifying questions necessary. To evaluate our Clarify-or-Answe model's predictions, we report Direct-Answer Accuracy: Did the the system correctly decide to directly answer the question when the question unambiguous and the greedy predicted direct-answer was correct. To determine whether the greedy predicted direct- answer was correct, we decode our Clarify-or-Answer system's greedy direct-answer prediction for each input question. We also report Ambig Acc: the accuracy of a system's 's clarify-vs-answer predictions on the human ambiguous-ves wwa.oouuss-ssuumuummee vs-unambiguuus labels.
We compare our models predictions (C-or-A- Pred) against a random baseline and two addi- tional methods, ProCot and PPDPP. To com- pute the random baseline, we fix the per- cent of direct-answer responses (DA%) to our model's clarify-or-answer predictions and ran- domly sample DA% of predictions to directly answer, and predict clarifying questions for the remaining queries. ProCot[Deng et al. (2023a) utilizes a CoT prompt to determine whether not the input is ambiguous and a clarifying
Table Additional results for Clarify-or- method (Llama2 model). We include the percent of questions directly answered (DA%) and evalu- ate using F1, Direct-Answer (DA) Acc., and Am- big (A) Acc.
Method (DA%)
Ans Fl
DA Acc 1 A Acc 1
Random (44%)
23.4
55.4
52.1
Ours (44%) Ours w/ ProCoT (35%)
24.3
61.9
53.7
15.6
60.1
49.2
Ours w/ PPDPP (46%)
23.9
57.7
59.0
question is necessary. In practice, we use the instruction-tuned variants of each base model to make these predictions,
we find irregular behavior when applying the CoT prompt non-instruction tuned models. PPDPP Deng et al. (2023b) proposes a method for training an auxiliary model to predict how models should respond. Here, we train an additional copy of the base model make the binary prediction of whether asking clarifying question is necessary for the system recover more all answers. ProCot and PPDPP do not train models to what questions to ask, and are only designed to predict whether clarifying is necessary. To evaluate these baselines, we pass their predictions to our trained Clarify-or- Answer model, which then generate the clarifying question of direct answer according to the baseline's 's predicitons. We include implementation details in Appendix B
-nnswer
We report our results with Llama2 in Table f (results with other base models Appendix Bb. Our Clarify-or-nnswer system's 's predictions consistently outperforms all baselines across all metrics cept Ambig Accuracy. Despite using a stronger base model (instruction tuned variant instead of base model), the prompt-based method (ProCot Prompt) performs the worst, as it is not trained in-domain. Training a separate model for ambiguity prediction (PPDPP) performs well, showing almost 10% higher ambiguity accuracy than our full model. However, our method still outperforms this model in answer accuracy. This demonstrates that our method (1) more frequently asks clarifying questions when it's useful do so; and (2) avoids asking for clarification on ambiguous questions when the model is not able to successfully use the clarifying question. The relatively large gains on Direct- Answer accuracy compared to F1 indicate that we can expect performance on end-task metrics to further improve with better clarifying question generation.
6 RELATED WORK
Ambiguity in NLP Ambiguity has been studied extensively in NLP in across a variety of tasks.
In machine translation, works have studied instances where sentences in a source language have multiple valid translations in target language (Fernandes et al. 2021 Lopes et al. 2020 Voita et al.)
9

---

Published as a conference paper at ICLR 2025
2019). In natural language inference (NLD, Nie al. (2020) collected dataset of highly cross-
annotated examples that contain high disagreement in annotator judgments. Liu et al. (2023) then found that these disagreements
often the result of ambiguity in the input. Recent work has studied
various sources of ambiguity, from entity-linking (Lee et al., 2024), co-reference ambiguities Yuan et al. 2023), to temporal and geographical contexts (Zhang & Choi 2021).
Uncertainty in LLMs Several recent works have studied calibration in LLMs, looking specifically the effects of RLHF training. Specifically, whereas likelihoods for base LLMs (i.e., pretrained) tend to be well-calibrated for a variety of tasks (Cole et al. 2023 Kadavath ct al. 2022). this be- havior lost after RLHF finetuning (OpenAll 2023). Additionally, Zhou et al. (2024a) investigated via epistemic markers generated by LLMs and found evidence that standard single-turn RL training encourages over-confidence in L.LM responses. Band al. (2024) trained an LLM to generate long-form text that includes confidence statements about its own generation. Zhang & Choil (2023b) studied when uncertainty predictions are caused by lack of knowledge or by lack of clarification.
eecertainty-expressiiie
Clarifying Question Prior works for training LLMs to ask clarifying questions have focused on settings where the types of clarification required are fixed by the task (i.e., there is fixed set of rele- vant imput features (Wei et al. 2018; Kuhn et al.) 2022) or the input (c.g.. conpreencmmpinnnniionn tasks where the context contains multiple correct answers (Guo et al. 202I)). Such works have studied clarifying question generation for a variety of tasks, including gauging social and moral situations (Pyatkin al. 2022) and image classification YYu et al. 2019).
Closely related to our work, Andukuri et al. (2024) trained LLMs to ask clarifying questions to resolve task ambiguity, where a single request may have multiple different indented tasks depending on the user, after user simulation. Compared to their experimental setting of simulated user persona for open-ended tasks, we address real-world open domain question answering task. Furthermore, we leverage preference optimization framework while they focus on generating supervised fine- tuning data by filtering low-quality clarifying questions. [Hong al. (2023) also uses LLM--smulated dialogues for training. Recent work has also noted the scarcity of clarifying questions in responses from existing LLMs, and have studied promoting methods for eliciting such responses with greater frequency (Deng et al., [203aa] SSaiikh et al. 2023). Li et al. (2023) found that clarifying questions can be useful for learning individualized preferences.
7 LIMITATIONS AND FUTURE WORK
We only consider systems for one or two turn interactions and does not consider cases where asking additional clarifying questions might be helpful after observing the user's clarifying answer. Future work might explore methods for extending our evaluation framework and double-turn preference annotation methods to accommodate general multi-turn interactions with users. Furthermore, the systems explored in this work also do not model dialogue acts (Stolcke et al. 2000) outside of predicting a single answer or asking clarifying question. At times, it might be more appropriate for LLMs to generate responses for different behaviors (e.g., Overton responses containing multi- ple answers if there are only a few possible interpretations. Long vs. short answers for complex queries. Abstaining in lieu of predicting erroneous answers). Future work might consider how clar- ifying questions should be used in conjunction with other strategies. Another important direction extending double-turn preference method for other tasks. Our evaluation framework is intended to be easily adaptable, only requiring (1) multiple annotator labels for expected outputs and (2) an changing end-task evaluation metric for the new setting.
8 CONCLUSION
We propose a method for training LLMs to ask clarifying questions with double-turn preferences and an an automatic framework for evaluating systems that ask clarifying questions using user simulated user interactions. Our QA results demonstrate that training systems to ask clarifying questions with double-turn preferences improves performance over training with standard single-turn preference labels and can be used to train models judiciously decide when to ask for clarification.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the UT NLP community for providing feedback throughout the project. This work was in part supported by Cisco Research. Any opinions, findings and conclusions, or recommenda- tions expressed in this material are those of the authors and do not necessarily reflect the views of Cisco Research. The work is partially supported by gift from Google Research. This work has taken place in part in the Rewarding Lab at UT Austin. The Rewarding Lab is supported by NSF (IIS-2402650), ONR (N00014-22-1-22044) EA Ventures, Bosch, UT Austin's 's Good Systems grand challenge, and Open Philanthropy. This work was done in part while the first and last author was visiting the Simons Institute for the Theory of Computing.
REFERENCES
Chinmaya Andukuri, PhiPphl Fränken, Tobias Gerstenberg, and Noah D Goodman. Star-gate: Teaching language models ask clarifying questions. ArXiv, 2024. URL https://arxiv.
hoc/pdf/2403.0 19154
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hemandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, 2022. URL http:////wwwwwww..aaasoso//aaa/
058
Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. Linguistic calibration of language models. ArXiv, abs/2404.00474, 2024. URL httpp::/appp http//pp...ematttcccccccchlllall org/
CorpusID: ID:
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open- domain questions. 2017. URL http:////aaaaaaoooc////////...0. 0005.
Jeremy R Cole, Michael J.Q. Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Yang Deng, Wenqiang Lei, Lizi Liao, and Tat seng Chua. Prompting and evaluating large language models for proactive dialogues: Clarification,, target-guided, and non-collaboration. In Findings of the Association for Computational Linguisties: EMNLP, 2023a. URL https://arxxt
hxq/pdf/2333 200.E//235.162222
Yang Deng, Wenxuan Zhang, Wai Lam, See- Kiong Ng, and Tat-Seng Chua. Plug-and-plaz policy planner for large language model powered dialogue agents. ArXiv, abs/2311.0026. 2023b. URL
https:/ //api semant insmmnttccchoooo ar. www.orrg/oooo wttg//orpussTii ID: 833170
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix mul-
tiplication for transformers scale. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. URL hattps://arxiv.oret www.ariww.....ooaaooooooa 90t3:/00/.///00..333333
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2024. URL
/2305. 14314
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. ArXiv, 2024. URL/ttts:://arae http://axxxx..g//...///// wwtpooog///p/.......
21783
Patrick Fermandes. Kayo Yin, Emmy Liu, André FT Martins, and Graham Neubig. When does translation require context? data-driven, multilingual exploration. Proceedings of the An- mual Meeting of the Association for Computational Linguistics (ACL). 2021. URL https: w/arxiww.rggppff////
11

---

Published as a conference paper at ICLR 2025
Eve Fleisig, Rediet Abebe, and Dan Klein. When the majority is wrong: Modeling annotator dis- agreement for subjective tasks. In Conference on Empirical Methods in Natural Language Pro- cessing, 2023. htttt:/eeeaa...emaaa.cccccoo..r. wwt.:/awww...cccoo......oo.oraaa
Google Gemini Team. Gemini 1.5: Unlocking multimodal understandinnn across millions of tokens of context. 2024.
05530
Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. Abg-coQA: Clarifying ambiguity in conversational question answering. 3rd Conference on Automated Knowledge Base Construc- tion, 2021. URL
Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via rl on imagined conversations. arXiv preprint arXiv:23/1.055560 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https: /arxiv org/ org/pdf/
2106. 09685
Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour. Guillaume Lample, L élio Renard Lavaud, Louis Ternon, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. ArXiv, 2024. URL 04088
https://arxiv>
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et Language mod- cls (mostly) know what they know. ArXiv, 2022. URL /tttp:://axx.vvorrsppff///// 05221
Andreas Kopf, Yannic Kilcher, Dimitri von Rutte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Rich'an ard Nagyfi, ES Shahul, Sameer Suri, David Glushkov, Arnay Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment. ArXiv, abs/2304.07327, 2023. URL htttt:::// /tts:///ppi.. semant www.hoaaasccccca......m org Corpus_D:25817943
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Clam: Selective clarification for ambiguous questions with generative language models. arXiv preprint arXivv22/2.07776,, 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrow. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics (TACL), 2019. URL https: tttto:://accattholoogyo
oxq/019-1026. pdf
Kenton Lee, Ming-Wer Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019, URL /RLnttpp://ccciioorrr /1900 00..9.666000000
Yoonsang Lee, Xi Yc, and Eunsol Choi. Ambigdocs: Reasoning across documents on differ- ent entities under the same name. 2024. URL https::/apppp http:/pp...emaattcssccchlolooeee
Corpus 1D:22222807
Belinda Z. Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. Eliciting human preferences with language models. ArXiv, 2023.
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We're afraid language models aren't modeling ambiguity. Proceedings of the Conference on Empirical Methods in Natural Language Process- http:/:///axwwvvao.a ing (EMNLP), 2023. httpp///..//////////ooooooo///// 14399. f
12

---

Published as a conference paper at ICLR 2025
António V Lopes, M Amin Farajian, Rachel Bawden, Michael Zhang, and André FT Martins. Document-level neural mt: A systematic comparison. 2020.
Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L'eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Am'elie H' H'eliou, Andrea Tacchetti, Anna Bulanova, Antonia Pa- terson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-hhoo, Cl' ement Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptin Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vladimir Feinberg, Woj- ciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong. Tris Brian Warkentin, Ludovic Peran, Minh Giang, CI' Cl'ement Farabet, Oriol Vinyals, Jeffrey Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Ar- mand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. ArXiv, 2024. URL pdf 08295
Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski,
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL
10645.pdf
Yixin Nie, Xiang Zhou, and Mohit Bansal. What we learn from collective human opinions on natural language inference data? Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL
03532. fd
OpenAl Gpt-4 technical report. ArXiv, 2023. URL https://axxivvorr///tt/// 08774.pdE
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. URL wttppttt///ete //tpsi////xvv... org wwwg//pd... /pff/ 2203. 02155
Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cognition, 122(3):280-29) 2012.
Valentina Pyatkin, Jena Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chan- dra Bhagavatula. Clarifydelphi: Reinforeed clarification questions with defeasibility rewards for social and moral situations. ArXiv, 2022. /tall//ttae//aavi.ooo///////////
2040.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), 2023. URL
https://arxiv: org. abs/ 2305 18290
Sebastin Santy, Jenny T' Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. Niposition- ality: Characterizing design biases of datasets and models. In Annual Meeting of the Associa- tion for Computational Linguistics, 2023. URL https:://// //tp://pp.... www.etiiscccooooo........e org/
Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding or guesswork? large language models are presumptive grounders. arXiv preprint 1rXiv:2311.09144 2023.
13

---

Published as a conference paper at ICLR 2025
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor. Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computationnal Lingusstees, 2000.
URL htRlhttps:// //ap::///ll antho org. -
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09228. 2023.
Elena Voita, Rico Sennrich, and Ivan Titov. When a good translation is wrong in context: Context-
aware machine translation improves on deixis, ellipsis, and lexical cohesion. Proceedings of the Anmual Meeting of the Association for Computational Linguistics (ACL), 2019. URL https:
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang,
Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024.
Wei Wei, Quoe Le, Andrew Dai, and Jia Li. Airdialogue: An environment for goal-oriented dialogue research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3844-3854, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the ural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2020. StR.https:// http://axvvv.. /rg/
nat-
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965 -23998.. PMLR, 2022.
Lili Yu, Howard Chen, Sida Wang. Yoav Artzi, and Tao Lei. Interactive classification by asking informative questions. ArXiv, 2019. Utthttps: http://:///.a.x....o//////////// wwtp://gg/ppppeoe 03598
Yuewei Yuan, Chaitanya Malaviya, and Mark Yatskar. Ambicoref: Evaluating human and model
sensitivity to ambiguous coreference. Findings of the Association for Computational Linguistics: EACL, 2023. Uttp::
wwt.or/////.... 2302 00762
Michael J.Q. Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. URLpttts: / arxiv org/ /tfpg//pf/////// pttpff//10000 20w... 0610 157 pdf.
Michael J.Q. Zhang and Eunsol Choi. Clarify when necessary: Resolving ambiguity through inter-
action with Ims. ArXiv, abs/2311 1.09469, 2023a. URL
Corg/Corpus TD: 26522144 1465
Michael J.Q. Zhang and Eunsol Choi. Clarify when necessary: Resolving ambiguity through inter- action with Ims. ArXiv, 2023b. URL htt.ss///aaxx...rr///////111 http:///2211..049...2
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild. International Conference on Learning Representations
(ICLR), 2024. URLpttps:: https///axxxvvoooe.e wwwpw/wwwwwp//////////// p 01470
Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. ArXiv, 2024a. URL httpt:t//r/rxiiv
org/pdt 2201.40111000 06730
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-tumn rl, 2024b. URL
T9446
14

---

Published as a conference paper at ICLR 2025
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b 7b: Improving Ilm
helpfulness & harmlessness with rlaif, 2023. URL https: /ttp://staannnn.../.e as be berkeley. edd//]
15

---

Published as a conference paper at ICLR 2025
A DATASET DETAILS
In Table we provide examples from AmbigQA and NQ-Open with differing answer sets. These examples highlight instances where relying on a multiple annotators to provide the answer versus relying on single annotator to provide all possible answers yielded differing answer sets.
A.1 HUMAN AND MODEL AMBIGUITY SET AND RLHF TRAINING DATASET DETAILS
In Table 8 report dataset statisties for or two SFT datasets. Note that (I, q) pairs used for SFT training for our clarifying question asking (x - q) systems. Additionally, the (æ, q. o, y) pairs used for training our joint and separate systems for predicting QA answers given clarification (z.q.0 q. a - y) as well as our user simulator models (I, - a). We also use these pairs for training our direct QA systems (x - y). ignoring the clarifying question and answer. Table we include several examples from each dataset, highlighting the differences in the ambiguities identified each.
For RLHF training, we use examples from the NQ-Open training set after removing examples used generate our SFT datasets from NQ-Open. This leaves us with 70,904 remaining input questions, which we split into training and develpent splits (64,584 38444448 and 6,320).
B ADDITIONAL EXPERIMENTS
B.1 CLARIFY-OR- BASELINES
Here, we describe the additional baselines introduced in Section 5.1
Proactive chain-of-though (ProCoT) prompt Deng et al. (2023a presents a chain-of-thought prompt method for instructing LLMs first determine whether not an input question is ambigu- ous before generating a direct-answer or clarifying question response. The need for instruction- following abilities for this bascline necessitates using the instruction-tunee variants of each base model. Therefore, performance of this approach may be inflated due to the stronger base model.
Plug-and-Play Dialogue Policy Planner (PPDPP) Deng et (023b) proposes a SFT and RL methods for training separate model predicting the proper dialogue act for the next assistant turn. Because their method not designed train systems to generate responses, only to decide which type response to predict, system trained over the SFT training sets over the gold dialogue act: clarify if the question ambiguous or directly answer otherwise. We do not include their RL method, the authors report comparable performance with and without additional RL training. We found training to be hyper parameter sensitive, with the suggested hyperparameter training systems always select single option. Using this trained system, we then use its predictions to determine whether to predict a clarifying question direct-answer response using Marify-or-Diree Ans DPO model.
train a separate PPDPP model from our base LLM via SFT. This
B.2 ADDITIONAL RESULTS
In Table we report the additional results comparing different methods for determining when clar- ifying questions are necessary with additional base models (Llama3 and Gemma). We find that the ProCot Chain-of-Thought prompt yields degenerate behavior when used with the instruction tuned Gemma and Llama3 models, failing to ever predict to directly answer question. When looking at the other methods, we find similar trends to the results from Llama2 discussed in Sc.
C EXPERIMENTAL DETAILS
We perform all experiments on single machine with 8 A40 (48GB) GPUs using the transformers library (Wolf et al. 2020) and the AdamW optimizer (all training runs completed within 24 hours). During SFT training, was performed for up to 5 epochs, evaluating on our development set after each epoch and selecting using the best performing checkpoint.
train all models with a learning rate of 5e-5 and batch size of 32. Training
16

---

Published as a conference paper at ICLR 2025
NQ-Open Annotations Answers
AmbigQA Annotations
Question
Is Ambiguous?
Answers
Who is singing in the background gold digger?
("Ray Charles". "Jamie Foxx")
No
("Jamie Foxx")
How much does 2004 chevy cavalier weight?
{11,162 - 1,315 kg".
No
122,562-2990) lb")
"2,562 2,900 lb")
What season does meredith and derek get married in grey's 's anatomy?
{"Season 7")
Yes
Season 5" Season 7")
When did the soviet union test its first atomic bomb? When did the soviet union test its first atomic bomb?
('29 August 1949")
Yes ('29 August 1949", "7:00 a.m")
('29 August 1949")
Yes
('29 August 1949", "7:00 a.m")
Table 5: QA examples differing labeled answer sets form NQ-Open and AmbigQA.
Table Llama3 Results for Clarify-or-Anse Answer methods. For each model, we include the percent of questions directly answered (DA%), and evaluate using F1, Disect-Answer Acc., and Ambig Acc.
Base Model
Method (DA%)
Ans Fl
Direct-Answer Acc
Ambig Acc
Random (43%)
55.6
51.1
Llama3
Ours (43%)
25.1
59.2
54.0
PPDPP (60%)
22.2
45.3
53.4
Random (38%)
23.6
60.0
52.1
Gemma
Ours (38%)
24.6
64.1
54.3
PPDPP (54%)
21.7
50.8
56.7
For DPO training, we merge LoRA checkpoints from SFT-only baseline and train using KL regularization factor 0.1 and a learning rate of 5e-6 in all experiments. For training was per- formed for up 2 epochs until loss converges development data, selecting the best performing checkpoint on development data. For Llama2-7b based methods, we train with a batch size of 32, evaluating every 750 steps. For Gemma-7b based methods we train with batch size of 16, evaluat- ing every 1500 steps.
For performing additional SFT training create our joint models for asking clarifying questions and answering with clarification, we merge LoRA checkpoints again and use the same hyperparameters used for standard SFT training listed above.
D EXAMPLE OUTPUTS
We include examples of models successfully using clarifying questions to predict the target answers for ambiguous (in Table 14) and unambiguous test queries (in Table 13
E LICENSING
The transformers library is licensed under Apache License 2.0. AmbigQA does not list any license; however NaturalQuestions, the dataset which it is based on, under Apache License 2.0. Llama2 is licensed under the LLAMA Community License Agreement. Gemma is licensed under the Gemma Terms of Use. We license all our created assets under CC BY-NC-SA 4.0.
17

---

Published as a conference paper at ICLR 2025
Input Query: Who is the highest paid football player?
GPT-4 Response (104 words):
As of 2024, the highest-paid football players are Cristiano Ronaldo and Karim Benzema, both earning around E000 million per year. Ronaldo plays for Al Nassr, and Benzema is with Al Ittihad, both in the Saudi Pro League.
Following them, NoGolo Kanté and Neymar each earn approximately E100 million annually, also playing in the Saudi Pro League for Al Ittihad and Al Hilal, respectively. Kylian Mbappé is the highest-paid player outside the Saudi Pro League, with a salary of E70 million at Paris Saint-Germain.
These figures reflect the significant financial influence of the Saudi Pro League, which has at- tracted many top players with lucrative contracts.
Gemini Response (30 words):
The highest-paid football player in the NFL for 2024 is Joe Burrow, the quarterback for the Cincinnati Bengals. He earns a whopping $55 million per year according to his contract.
Claude-3 Sonnet Response (112 words):
As of 2023, the highest paid football (soccer) player in the world is Kylian Mhappé of Paris Saint-Germain. His reported anmual salary is around E72 million (approximately $78 million). Some other top highest-paid football players in 2023 include:
Lionel Messi (Paris Saint- Seiit-Germain)-. Around E41 million per year
3. Neymar (Paris Saint-Germain) Around E36. million per year Kevin De Bruyne (Manchester City) Around E25 million per year
5. Robert Lewandowski (Barcelona) Around E23. million per year
However, player salaries can fluctuate frequently with new contracts, bonuses and endorsements. Mbappé's new deal with PSG from 2022 made him the highest paid at the reported E72 million annual wages.
Table 7: LLM responses to the input query. All responses were generated on May 21st, 2024.
Train
Dev
SFT Dataset
(x, q)
(x, q, a, y)
(x, q)
(æ, q, a, y)
Human-ldentified Ambiguity
4000
8781
400
889
Model-Identified Ambiguity
4000
9189
400
948
Table 8: SET Dataset Statisties.
18

---

Published as a conference paper at ICLR 2025
Question (x)
Clarifying Question (4)
Clarifying Answers (a,)
Answers
(v))
where the olympic games held greece
Are you asking about the cient Olympic Games or
Ancient Olympic Games
Olympia /
Modern Olympic Games
Athens
modern Olympic Games?
who won red army or the white army
Are you asking about the Rus- sian Civil War or a specific bat- tle?
Russian Civil battle
War
Specific
The Red
Army / the
white army
who wrote ob la ob la da
Are you asking about the pri- mary composer or the writer?
Primary composer Co-writer
Paul Mc-
Cartney
John
Lennon
original singer of how am supposed to live without you
Are you asking about the origi- singer who recorded song or the singer who popular- ized it?
The original singer who first recorded the song singer who popularized the song
Laura
Branigan
Michael
Bolton
Table 9; Randomly sampled Model-Ambiguity SFT dataset examples.
Question (x)
Clarifying Question (q)
Clarifying Answers (a.)
Answers
(v))
Who played thanos in gardians of the galaxy?
Are you asking about the tor who provided the voice Thanos, or the actor who physi- cally portrayed him on screen?
The actor who provided the voice for Thanos. The actor who physically portrayed him on screen.
Isaac C.
Singleton
Jt. / Sean
Gunn
Where in the bible is the story of samuel?
Are you asking for the specific book and verses where the story of Samuel is found, are you asking for the general section of Bible where the story is lo- cated?
I'm asking for the specific book and verses. I'm asking for the general section of the Bible.
Samuel
1:1-28
/
Hebrew
Bible
Most amount points scored in an nba game?
Are you asking about the most points scored by a single player in game or the highest com- bined score in game?
Most points scored by a sin- gle player Highest combined score in game
100 370
Who has the highest instagram followers the world?
Are you asking about the in- dividual with the most follow- ers on Instagram, or the account with the most followers over- all?
The individual with the most followers / The account with the most followers overall
Cristiano
Ronaldo /
Instagram
Table 10: Randomly sampled Human-Ambiguin SFT dataset examples.
19

---

Published as a conference paper at ICLR 2025
You will be given question and several possible responses. Possible responses may be correct due to an ambiguity in the question or when the question was asked.
Provide clarifying question to determine which possible response is correct. If one zero of the possible responses correct, respond with "None". Provide your output in following format:
Clarifying Question: [Clarifying Question]
1. Clarifying Answer: [First Clarifying Answer]
1. Response: [Copied Response for First Clarifying Answer]
2. Clarifying Answer: [Second Clarifying Answerl
2. Response: [Copied Response for Second Clarifying Answer]
Question: {question}
Possible Responses:
{answers}
Table 11: Prompt used for generating our Human and Model Ambiguity SFT datasets with GPT-4.
Pretend that you In response to your question, the Al assistant has asked you clarifying question to help determine which answer you expect.
asking Al assistant question.
For cach of the following expected answers, provide a clarifying answer to the Al assistant's clarifying question that indicates you expecting that answer. All clarifying answers should be concise sentence phrase and should contain the expected answer.
If there is no possible clarifying answer, respond with "None." instead.
Provid your response in the following format:
Clarifying Answer [Clarifying Answer for Expected Answer Clarifying Answer [Clarifying Answer for Expected Answer 2]
Question: {question] Clarifying Question: (clarify .4} {answers)
Table 12: GPT-4 prompt used for simulating user clarifying answers during evaluation. All evalua- tions carried out using the GPT-4 version gpt-4-0613.
20

---

Published as a conference paper at ICLR 2025
Table 13: Example predictions where asking clarifying questions helped recover the correct answer to an unambiguous test query. Clarifying question predictions are taken from our LI LLamm2-7hh 1ama2-7B-basee Clarify DPO (Ours) model trained with double-turn preferences. We also include the predicted ALLama2-B--basee Direct-Ans SFT model predictions.
Question (#)
Pred Clarifying Question (q)
GPT4 Clarifying An-
Clarify DPO
Direct-
Gold Ans
swers (a,)
(v:)
Pred Ans.
Prod Ans.
(Wi)
When does telitall's ss guardians the galaxy episode come out?
Which platform are you asking about for Tell- tale's Guardians of the Galaxy Epiinde
I'm asking about the re- date plat- forms.
August
October
August 22, 2017
22 2017
2017
When did gods not dead
Which "God's Not Dead you referring to? The lirst, second, or third
I'm asking about the re- lease date plat- forms.
March 30 2018
April 2018
2,
March 30 .0018
When did the sea mon- movie out?
Which "Percy Jackson and the Sea of Mon- sters" movie are you ask- ing about? The re- leased in 2013 or the ene released
The one released 2013.
August 7, 2013
March 26, 2010
August 7, 2013
Who wan nohel prize for peace 2014?
Which individual or ar- ganization the Nobel Peace Prize in 2014?
The individuals who won
Kailash Saryarthi and
Kailash Saryanth
Kailash Saryarti and Malala
Malala Yousafzai
Yousafrai
How many islands in- cluded in federated states of micronesia?
Which specilic islands are you referring to?
I'm referring all the is- lands and islets that make up the country.
607
is- and
600
607 lands and
is-
lands
islet
islet
Table 14:
predictions on unambiguous test queries from our LLama2-7BB model trained with double-turn preferences. We include examples where clarifying questions helped the model recover multiple annotated answers.
Question (x)
Clarifying Question
GPT44 Clarilying
(a²)
Pred Gold (3i Vi)
many starbucks there around the world
As which date are you asking about the number Starbucks stores around the world?
the 201
(28.218, 28.218)
of the end of 2004.
(6,000, 4,962)
As of the cod of 2016.
(23,768 23,768)
(25,000,
sang original do you love me!
version "Do You Love Me" you referring to? The original song by The Contours the
The version that was hit in the 1960s.
(The Contours, The Contours)
The version from the musical Fiddler on the Roof.
(Topol, Topol)
by The Partridge Family
When did the queen becarne queen england?
Which specific queen are referring to?
Queen Victoria
(20 June 1837, 20 June 1837)
Queen Elizabeth
(6 February 1952, February 1952)
Queen Anne
(8 March 1702. May 1707)
21


Reviews:
[{"review_id": "Wlxkt9ADgw", "reviewer": "Reviewer_jxRi", "summary": "The paper proposes a method for training LLMs in question-answering tasks to learn to ask clarification questions where appropriate given query ambiguity.", "questions": "Would it be helpful & possible to include some more informative comparisons and evaluation metrics (see \"Weaknesses\" above)?\n\nOn annotation reliability: I wasn't clear whether the additional annotation described for AmbigQA was done as part of this work, or previously by other authors and described elsewhere (\"his process identifies about 56% of all queries in NQ-Open as ambiguous\" etc.).\n\nTable 2 doesn't say what metric it uses - presumably F1?\n\nSome minor typos:\n083 two axis->axes\n097 citation should be citep not citet\n110 of [the] task scenario\n187 an additional turn[s]\n406 Cla[i]rifying\n188 observ[ing] the LLM's \n206 simulate[d] user", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic addressed -- clarification behaviour by LLMs, and how to make their behaviour more accurate & reliable by clarifying where required -- is very topical and of great interest to the NLP community.\n\nThe paper proposes a fairly simple and intuitive method, describes it clearly, and evaluates it sensibly (mostly -- see below), and results seem good in that they show reliably increased accuracy.", "weaknesses": "The evaluations and baselines compared to could be stronger or more meaningful in some cases. In the main experiment, one of the main takeaways from the results seems to be that answer accuracy is increased if a CQ is always asked, although this comes at the cost of more turns. This raises the question of how well a system would do if given a very simple fixed prompt which forced a CQ as the first response, and then an answer based on the resulting three-turn context - but without going through the extra training steps given here. I expect it wouldn't be quite as good as the best system here, but it would be very instructive to know how much difference there was.\n\nIn the experiment of sec 5.1 (determining whether CQs are necessary), performance is compared to a random baseline (sampling a random fixed percentage of direct answers vs CQs), but there's no direct evaluation of the main question here: how often a CQ was asked after an ambiguous question (i.e. accuracy of identifying when a CQ is required). In fact, the question is perhaps better: how often was a CQ asked after a question which was ambiguous, and where the possible answers differ (if the answers are the same, a CQ provides no benefit, even if the question is nominally ambiguous).\n\nThe data on possible CQs and answers, including answer correctness/preference, is mostly drawn from LLM-based simulations, but there doesn't seem to be any validation of how accurate or realistic this is.\n\nThere's no discussion of how ambiguity in a question actually relates to differences in answers. Ambiguity sometimes doesn't matter in practice: e.g. a variant of the first question in Table 8 (appendix) could be \"what country were the first olympic games held in\"; then, although there is ambiguity about whether the query concerns the modern or ancient games, that doesn't affect the correct answer which would be Greece in both cases. It would be interesting to know how common this is, and whether the behaviour of the models reflects (or should reflect) this difference.\n\nThe biggest weakness I perceived is the lack of relation to other relevant work. There's discussion of related work with LLMs and ambiguity, and LLMs being trained to ask CQs in different contexts; but there's no discussion of pre-LLM work that seems relevant. For instance, there is a large body of work in training dialogue systems to respond suitably to ambiguous input (including asking clarification questions), much via reinforcement learning. Closer to the domain here, if usually less close in terms of method, is the body of work in interactive information retrieval (including using clarifying questions to refine or further specify search queries or QA questions). Some discussion of how the method here resembles (or doesn't resemble) those would be very helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3o91y3Q0oW", "reviewer": "Reviewer_UjMF", "summary": "The paper proposes an approach which constructs preference data which are determined by examining (up to 2) simulated future dialogue turns. The proposed approach is evaluated in an open-domain QA dataset, NaturalQuestions, and a related derivative dataset, AmbigQA. The authors compare their approach against a standard single-turn reward model and present their results in terms of Token-based F1 for the QA task. The authors also present a set of experiments on ambiguity recognition.", "questions": "1. How do you consider scenarios in which clarifying questions are undesirable behavior? Relatedly, how do you propose to balance clarifying questions with behaviors such as hedging, or presenting information for all possible intents?\n\n2. Does the motivating problem of LLMs not asking clarifying questions get solved by prefixing with a simple instruction such as \"ask a clarifying question if ambiguous\"? For instance, I tried this with the example from Figure 1 and Gemini asks a clarifying question. To that end, such systems may be tailored to one type of behavior over another by design?\n\n3. Have you considered multiple types of ambiguity (e.g. contextual versus lexical)?\n\n4. Is the proposed approach intended to improve general LLM usage, or for this one particular task? if the latter, is this task with basic response outputs (e.g. single token) representative of real-world usage?\n\n5. L161 is the model specifically fine-tuned to output \"Clarifying Question:\" (i.e. all generated responses should be accompanied by an intermediate dialogue act token)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "***Significance of setup***\n\nThe proposed setting is important to investigate. As LLMs are increasingly tuned for use in dialogue applications it is important to consider how to better address the needs of users, and disambiguation is certainly a reasonable idea.\n\n***Ablations and analysis***\n\nThe paper presents experiments comparing the proposed approach on multiple base LLMs, and performs multiple reasonable ablations. The paper also considers and interesting examination on the effect of obtaining human annotations versus model predictions. \n\n***Clarity***\n\nOverall the paper is clear and the experimental details are helpful.", "weaknesses": "***Concerns regarding generalizability***\n\nThe focus and main contribution of the paper is that it considers a lookahead approach for future conversation turns in order to optimizing preferences in question-answering settings. However, the approach is only defined for at most two future conversation turns. It is not clear how 1. this may be generalized to longer rollouts and 2. whether this approach would remain effective in such a setting.\n\n***Scope of experiments***\n\nRelatedly, the domains examined are limited. The approach is validated on NaturalQA and AmbigQA. AmbigQA is an extension of NaturalQA, and as a result it is not clear nor trivial how the proposed approach would generalize to any other more complex and realistic domains (e.g., those requiring complex rollouts, or task-oriented settings).\n\n***Limited baselines***\n\nThe baselines in the main experiments are somewhat limited. If I understand correctly, Direct Answer SFT and Clarify SFT both do not seem like fair baselines in a task where there is mixed ambiguity. How does the proposed approach compare against SFT on both types of responses? Moreover, the proposed approach focuses on using RL tuning combined with future turns. How does it compare against directly including those future turn trajectories? It would also be helpful to compare against other existing RL-based approaches for ambiguity detection/action planning (e.g. [1]).\n\nThere are also no baselines for the ambiguity recognition experiments. Does the proposed approach outperform prompting? The objective performance of the proposed model here seems rather low. If the model asks questions unnecessarily, it may lead to degraded user experience even if it successfully elicits additional information to solve the downstream task / provides additional context to condition on for answer generation.\n\n*Minor Points*\n\ntypos:\nL187 \"their answer the proposed\"\nL300 \"construct all examples SFT training\"\nL306 \"to for queries\"\n\nReferences\n\n[1] Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents, ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rkCM9xgCh0", "reviewer": "Reviewer_5Axj", "summary": "This paper addresses the challenge of asking clarifying questions for ambiguous user queries. To determine the optimal timing and content of a clarifying question, one of the core contributions of this paper is a dataset of multi-turn QA dialogues that include such questions. The paper also presents a method for evaluating the quality of clarifying questions. In this method, multiple candidate clarifying questions are tested using a user simulator, with the quality of each question assessed based on the aggregate reachability to correct answers. Models trained on this dataset in various settings demonstrate improved QA accuracy on both ambiguous and unambiguous queries.", "questions": "- Line 164: Please clarify “We prompt GPT-4 to abstain from providing a clarifying answer a_i if it judges that none exists, in which case we count the resulting target answer prediction r_next^i as incorrect.”\n- Line 245: What is “EM accuracy”?\n- Lines 299: What does “if it determines one exists” mean and when can this happen?\n- Line 312: Please clarify “generate its greedy answer with temperature T = 0.0 and sample an answer with T = 1.0”.\n- Line 324: Which of the models below is used for the Turn 4 model?\n- Lines 330—359: The purpose of each model and how they are used in the two-step pipeline is unclear.\n- Line 342: What is the purpose of the Clarify DPO models?\n- Line 347: More details about the reward model and the RL training are needed.\n- Line 432: Why not training a model on the multi-turn dataset?\n- Line 459: Why is the random baseline appropriate for comparison?\n- Line 461: How did you set the value of DA?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- Asking clarifying questions for ambiguous queries is a valuable trait for QA systems, and the paper’s proposed methods are well-motivated.\n- The approach for assessing the quality of clarifying questions based on reachability to correct answers is novel, interesting, and well-reasoned.", "weaknesses": "- The primary weakness of this paper, in my opinion, is the lack of clarity in the method sections (2–4). One contributing factor is the use of terms without clear definitions. For example, in line 196, the term “annotators” is used, which seems to imply human annotators, while the preceding text suggests a “simulator” is involved. Additionally, the description of user simulation from line 208 onwards does not clearly explain the origins of *q* and *a_i*, making it challenging to follow. Although some of these ambiguities are clarified later in the paper, I would recommend reorganizing the structure to reduce the distance between concepts and their realizations. Further questions and suggestions are provided in the Questions section below.\n- The paper contains numerous ungrammatical and broken sentences, which significantly impact readability. For instance, line 187 (\"providing their answer the proposed\"), line 191 (\"catered a toward\"), line 206 (\"we use simulate user interactions\"), line 212 (\"using a we use\"), line 300 (\"examples SFT training\"), line 307 (\"to for queries\"), and line 312 (\"we a sample\"), among others. Additionally, citations are incorrectly formatted on lines 087 and 097. I will raise my score if the authors thoroughly address the grammatical and formatting issues and their revisions in their rebuttal.\n- Some experimental settings and design choices lack sufficient motivation. For example, why is asking clarifying questions preferable to simply providing multiple answers in one response (e.g., Q: \"Where were the Olympic Games held in Greece?\" → A: \"The ancient Olympic Games were held in Olympia, and the modern Olympic Games were held in Athens.\")? Moreover, there is no discussion on efficiency or how humans might perceive these clarifying questions, and a human evaluation might be valuable.\n- There is no experiment involving training a model directly on multi-turn QA dialogues. Instead, the authors only experiment with combining the weights of two separate models. Since a multi-turn dataset is available, training and evaluating a model on this dataset should be feasible.\n- In section 5.1, the comparison with the random baseline is questionable. A more effective comparison would evaluate whether the model asks clarifying questions when a query is ambiguous and refrains when unnecessary (e.g., based on the ambiguous and unambiguous splits of the dataset or cases where annotators provided consistent answers).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GLBcBWABpj", "reviewer": "Reviewer_kypX", "summary": "This paper examines handling ambiguity, an often overlooked aspect of large language models (LLMs) in multi-round conversations with users. The authors first develop a pipeline to collect human preferences in a two-round setting and then apply Direct Preference Optimization (DPO) to several open-source LLMs to enhance their performance.", "questions": "The presentation of the paper has significant room for improvement. I have identified several typos and areas for enhancement:\n\n- **Line 16**: Can you specify whether it is *evaluated*? I think it should be *trained*, (or both)?\n- **Line 212**: \"using a we use a trained user-simulator model\" – this sentence does not parse.\n- **Lines 224–226**: The green box is confusing. Why are q1 and q2 being preferred simultaneously?\n- **Line 186**: “by providing their answer the proposed clarifying question” – this sentence does not parse.\n- **Line 206**: \"in this work we use simulate user interactions for annotation.\" – this sentence does not parse.\n\n**Generic Questions:**\n\n- Is your human annotation IRB-approved?\n- In the methods section, can you write the loss function for DPO? This would make your paper more self-contained and eliminate the need for readers to refer to the original DPO paper.\n- Can you provide more details on the evidence supporting the hypothesis that model responses are evaluated only on their prior contexts (Line 17)? This is a very interesting hypothesis (perhaps the central hypothesis of the paper), so additional justification would be beneficial.\n- Line 305 presents an interesting hypothesis: \"We hypothesize that what is ambiguous for an LLM often deviates from what is ambiguous for humans.\" Has this hypothesis been proven?\n- Can you specify the input format for the LLaMA model? If my understanding is correct, only the Chat version includes the system and user specifications.\n- What is the computation time and the number of training samples used?\n\n\nBuilding on all my comments, I'd vote for a 5/10 score now :)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper provides a thorough scientific investigation, building upon several interesting hypotheses about LLM responses by training (evaluating) on prior data (Line 17) and speculating about the differences between LLM ambiguity and human ambiguity (Line 305).\n- The paper introduces a novel pipeline for eliciting human preferences in multiple rounds of conversation, even though I have some reservations regarding its operation.\n-  The experiments are very thorough, considering different variations of LLMs. However, the study lacks comparisons with non-LLM methods.", "weaknesses": "## Weakness 1: Unclear Relationship Between Human Annotators and the User Simulator\n\nThe paper appears to ask humans to only grade the final prediction of the model (the final step as shown in Figure 2). It is unclear whether their grading aligns with their judgment of the clarifying questions. It would be more appropriate to set a predefined user goal at the beginning and have human annotators grade the final answer based on its consistency with the original goal and the arrangement of questions and answers in the intermediate steps.\n\nI recommend that the authors refer to the conversational information-seeking literature in the IR community. For example, the paper [1] adopts a user simulator that maintains an initial item of interest (e.g., iPhone 16 128GB). During the question-answering stages, the user simulator responds to questions in line with the item's specifications (e.g., \"How much storage do you want?\" \"128GB\"). In this paper, it is unclear whether the human's final grading considers an initial goal and whether the grading assesses both the answer quality and adherence to that goal.\n\nThis is my major concern, and I hope the authors can provide further justification during the response period. Thank you!\n\n## Weakness 2: Lack of Baselines Beyond LLM Fine-Tuning\n\nWhile the paper includes numerous ablation studies and comparisons with other LLM tuning baselines, additional comparisons are needed to justify the method's effectiveness. I recommend the following:\n\n- **Unsupervised Beam Search**: This approach allows LLMs to search the space of possible clarifying questions (implicitly) [2].\n- **Chain-of-Thought**: Prompting LLMs to explicitly reason about possible ambiguities.\n- **Non-LLM Baseline Methods**: Including methods for asking clarifying questions that do not rely on LLMs, as explored in various works [3, 4].\n\nI hope the authors can try to add these results or explain why these methods are not suitable for comparison. Thanks!\n\n## Weakness 3: Need for More Insightful Analysis\n\n- **Knowledge Taxonomy**: A taxonomy for knowledge types is needed to make the dataset more insightful. What types of knowledge are more desirable for clarifying questions? If the original dataset does not provide this, consider latent word clustering or LDA?\n- **Error Accumulation**: Correct me if I am wrong, but I do not see an analysis of how an incorrect clarifying question may lead to error accumulation in the final answer.\n\nI hope the authors can discuss with me if these research questions are helpful and whether they plan to append them. \n\n# References\n1. [Conversational Recommender System](https://arxiv.org/abs/1806.03277) SIGIR 2018  \n2. [Self-Evaluation Guided Beam Search for Reasoning](https://arxiv.org/abs/2305.00633) NeurIPS 2023  \n3. [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://arxiv.org/abs/1907.06554) SIGIR 2019  \n4. [Asking Clarification Questions to Handle Ambiguity in Open-Domain QA](https://aclanthology.org/2023.findings-emnlp.772/) EMNLP 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9sMlBvNgAf", "reviewer": "Reviewer_vGfa", "summary": "This paper focuses on the problem of having LLMs ask clarification questions when the user request is ambiguous or uncertain. The authors propose a new approach to construct preference data by simulating future conversations. Experiments are conducted in two open-domain QA datasets. The results show that the proposed method outperforms other standard baselines.", "questions": "1. What is $a_i$ in line 120? It seems that it has a different meaning from ($a_i$) in line 129.\n2. Are the input query $x$ and the user intent ($x_i$) supposed to be different forms?\n3. Is the user-simulator model for preference data construction different from the the user simulation model in Section 2.1? \n4. What not apply Clarify-or-Direct-Ans SFT as a baseline?\n5. How is the original performance of these LLMs?\n6. How about just using in-context learning for enabling the clarification capability?\n7. Is there any existing method that can be applied as baselines, instead of all your self-designed baselines?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. This work studies an important question in the era of LLMs, i.e., teach LLMs to ask clarification questions\n2. Propose a novel approach to construct preference data including clarifications by simulating future conversations\n3. Experimental results show that the proposed method outperforms other standard baselines.", "weaknesses": "1. The presentation of the proposed method is a bit hard to follow. There is a lot of symbols and variables in the text, which is very messy and not well defined. More details in the questions that need to be clarified by the authors.\n2. The information of annotators for preference data construction is also not clear. \n3. The experimental settings are not extensive and reasonable enough, only comparing on self-designed baselines.  \n4. Several recent studies on LLM-based clarifications are not discussed or compared, such as [1-3].\n\nMinors: The usage of \\cite{} is incorrect.\n\n[1] STaR-GATE: Teaching Language Models to Ask Clarifying Questions\n[2] STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents\n[3] Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)