Interactive Large Language Models for Reliable Answering under Incomplete Context
Authors: Jing-Cheng Pang
Heng-Bo Fan
Pengyuan Wang
Jia-Hao Xiao
Nan Tang
Si-Hang Yang
Chengxing Jia
Ming-Kun Xie
Xiang Chen
Sheng-Jun Huang
Yang Yu
Venue: Accepted by TMLR ()

Abstract: The rise of large language models (LLMs) has revolutionized the way humans interact with artificial intelligence systems. However, their reliability in sensitive applications—such as personal consultations or clinical decision-making—remains limited. A critical shortfall lies in LLMs’ inherent lack of interactivity: these models generate responses even when essential context or domain-specific knowledge is absent, risking inaccurate or misleading outputs. A potential approach to mitigate this issue is to enable LLMs to pose clarifying questions, thereby uncovering the missing information required to provide accurate responses. However, previous methods often tend to greedily prompt LLMs to ask questions. This burdens the user to respond to potentially irrelevant questions and makes the system less flexible. In this paper, we introduce LaMSeI (Language Model with Selective Interaction) method, which enhances LLMs’ ability to judge when interaction is necessary under ambiguous or incomplete contexts. The motivation of LaMSeI is to measure the level of LLMs’ uncertainty about the user query, and interacts with user only when the uncertainty is high. Additionally, we incorporate active learning techniques to select the most informative questions from question candidates, for effectively uncovering the missing context. Our empirical studies, across various challenging question answering benchmarks, where LLMs are posed queries with incomplete context, demonstrate the effectiveness of LaMSeI. The method improves answer accuracy from 31.9% to 50.9%, outperforming other leading question-answering frameworks. Moreover, in experiments involving human participants, LaMSeI consistently generates answers superior to or comparable to baselines in more than 82% of the cases. Moreover, we verify the performance of LaMSeI on various LLMs, such as LLAMA2, LLAMA3, Vicuna and GPT-3.5, highlighting its capability to improve interactive language models.

Published in Transactions on Machine Learning Research (07/2025)
Interactive Large Language Models for Reliable Answering under Incomplete Context
OingCh Pangl.* Heng-Bo Fan**, Pengyuan Wang Jia-Hao Xiao Nan Tang! Sihang Yangl-, Chengxing Jiat'3, Ming-Kun Xie², Xiang Chen', Sheng-Jun Huang', Yang Yu National Key Laboratory for Novel Software Technology, Nanjing University, China & School of Artificial Intelligence, Nanjing University, China
College of Computer Science and Bechnology/Artifcen Intelligence, Nanjing University of Aeronautics and Astronautics, China Poliixiraai
Reviewed on OpenReview:
Abstract
The rise of large language models (LLMs) has revolutionized the way humans interact with artificial intelligence systems. However, their reliability in sensitive applications such as personal consultations or clinical decision-making remains limited. critical shortfall lies LLMs' inherent lack interactivity: these models generate responses even when essential context or domain-speeific knowledge absent, risking inaccurate or misleading outputs. A potential approach to mitigate this issue to enable LLMs to pose clarifying questions, thereby uncovering the missing information required to provide accurate responses. However, previous methods often tend to greedily prompt LLMs to ask questions. This burdens the user to respond to potentially irrelevant questions and makes the system less flexible. In this paper, we introduce LaMSeI (Language Model with Selective Interaction) method, which enhances LLMs' ability to judge when interaction is necessary under ambiguous or incomplete contexts. The motivation of LaMSeI is measure the level of LLMs uncertainty about the user query, and interacts with user only when the uncertainty is high. Additionally, incorporate active learning techniques to select the most informative questions from question candidates, for effectively uncovering the missing context. Our empirical studies, across various challenging question answering benchmarks, where LLMs are posed queries with incomplete context, demonstrate the effectiveness of LaMSeL. The method improves answer accuracy from 31.9% to 50.9%, performmgi other leading question-answert Moreover, experiments involving human participants, LaMSeT consistently generates answers superior to comparable baselines in more than 82% of the cases. Moreover, we verify the performance of LaMSel various LLMs, such LLAMA2, LLAMA3, Vieuna and GPT-3.5, highlighting its capability to improve interactive language models.
frameworks.
1 Introduction
Recent advances in large language models (LLMs) AOpenAl 2023 Team 2024a Du et al. 2022 DeepSeek-At 2023 Google Deepmind 2025) lift the curtain new era in human-machin interaction. These language models, pre-trained on massive corpora text 'Gao et al. 2021 are designed to comprehend and respond user queries with human-like proficiency. One of the most impressive applications of LLMs the chatbot, which generates responses to various user queries in natural language. However, effective dialog requires sufficient contextual information, which is often omitted when user interact with LLM chatbots. For instance, patients may inadvertently neglect to mention certain symptoms when consulting LLMs about their health,
* "Equal contribution. Corresponding: huangej@mmaa.eduuu edd..c.

---

Published in Transactions on Machine Learning Research (07/2025)
I've been feeling really anzious Jately notsure why.
I've been feeling realyy ausious litely and I'm not sure why.
I'm sorry hear that you'le to expertenccigg anxietty. Have theme boen any changes life that might have contributed to your ledings of acxirtyy
your
Andiyt can be caused by a vartet factors, intluainn simess, genetics brain chemistey, traumalic eventa, or envirumential facturs, It's important to consrult with bealthuure professional for a pessonal evaluation.
Yes, I've monntly started a now job, and very stressful.
Starting a new job can cortaially le stresfful experience and could cuntribute feelings anziety. nightt be helptul explore stress reduction techntiqess
(A) Direct answering
(B) Interactive answering
Figure 1: An illustration of interactive LLMs, which seeks for more dialog context to provide helpful answers. (A) LLMs directly generate answer to the user query. (B) L.MS interact with the user to seek more context.
as depicted in Fig. [ (A). On the other hand, LLMs are trained to respond to various questions and often attempt to provide answers even when context is lacking or their own knowledge is insufficient, which can frequently lead to low-quality responses.
To mitigate the issuc of incomplete context information, two principal strategics have been investigated: Aarrieval-Augmennle Generation (RAG (Lewis et al.) 2020)) and asking clarifying questions (Zhao & Dou 2024 Andukuri et al. 2024 Tix & Binsted 2024). RAG introduces the concept of augmenting LLMs with external information sources, such as knowledge bascs or scarch engines (Press et al. 2023). However, these RAG-basod methods primarily focus on the the limited knowledge of LLMs and do not adequately address the absence of user context. (Wang al. 2023a). Except for RAG, another solution is mulli-lurn clarificalion, which asks users clarifying questions to obtain additional context. For example, MulClari-LLM, method (Zhao & Dou 2024 uses a rulc-based system to generate scries of clarifying questions in a multiple-choice format, with the aim of clarifying user intent. STaR-GATE (Andukuri et al. 2024) fine-tunes the LLM to ask clarifying questions that are likely to elicit high-quality responses. While these methods effectively enhance the answer quality, they typically operate under a manner that encourage LLMs to ask more clarifying questions, which are often unrelated, resulting in an increasing burden on users to answer them. This calls for method that enables LLMs to ask clarifying questions in a more flexible and adaptive manner.
In this paper, we address the challenge of bridging the information gap between LLMs and users by introducing the LaMSeI (Language Model with Selective Interaction) method. Fig. | (B) illustrates the concept of interactive LLMs. We consider two pivotal questions for enhancing LLM-user interaction: when should LLMs interact with the users, and what questions should LLMs ask to acquire additional context? Firstly, we suggest that LLMs should interact with the LLMs only when they are highly uncertain about the user query. To achieve this, LaMSel estimates the LLM's uncertainty regarding the query by sampling multiple responses and calculating their variation. Under high uncertainty, LaMSel prompts the model to generate question candidates regarding the user query, enabling LLMs to pose effective clarifying questions. These candidate questions are not equally effective in bridging the information gap. To deal with this problem, we adopt the active learning technique to select the most informative clarifying questions, aiming to narrow the gap between LLMs and users. As a result, by answering these questions, the user enables the LLM to acquire the missing contextual information.
The main contributions are as follows: Firstly, we underline the critical need for interactive LLMs that flexibly interact with the users to uncover the missing context from the users. We suggest that the clarifying questions should be posed with caution to reduce the users' workload in answering them, which has been neglected by previous works. Secondly, we proposc a selective interaction mechanism for LLMs using active learning techniques, which aims to select the most informative questions to interact with the user. To our knowledge, this is the first work to combine active learning techniques and LLMs to handle user queries with incomplete context. This technique enables LLMs to dynamically obtain query-specific information. Lastly, we conduct comprehensive experiments to evaluate the algorithms' performance on user queries with imcomplete context, demonstrating that. LaMSel consistently outperforms existing Q & A frameworks on
2

---

Published in Transactions on Machine Learning Research (07/2025)
various LLMs. Furthermore, we evaluate LaMSeI with wide range of backbone LLMs, demonstrating its efficacy in improving LLM's comprehension of user query, thus enhancing the quality of model response.
2
Related Work
Handling user input with incomplete information. Previous studics have explored constructing AI systems that better understand user intention, under incomplete information in the user query (Griol el al.j 2025 Yang 2025). A promising approach is interact with users by posing questions for clarification or information (Aliannejadi et al. 2019 Shi et al.) 2022 Zamani et al. 2020 Stoyanchev et al. 2014)). Such systems have been effectively implemented in domains like hotel reservation services (Bemile et al.. 2014), where they prompt users with specific inquiries to verify and complete booking details. The field of natural language processing has also shown an interest in generating clarifying questions, particularly in response to ambiguous queries. For example, Zamani et al.] (2020) utilizes a set of question templates to address the issue of ambiguous web search queries. [Stoyanchev et al. (014)) studies the task of selecting clarifying questions from a set of human-generated questions for open-domain information seeking. A separate investigation (Coden et al., 2015) focuses on asking clarifying questions for entity disambiguation, phrased as "Did you mean A B?' However, this method is limited to entity disambiguation and does not apply to broader range of queries, such as those with multiple facets. MEDIQ (Li et al. 2024) proposes a dynamic medical consultation framework to simulate the medical consultation process, where doctor repeatedly asks the palient questions for more information. Though various research studies seek user clarification, they generate clarifying questions by rule or prompt LLMs to pose excessive questions. This paper proposes that we should conduct selective intcraction and only interact when necessary, improving the overall system efficacy.
Enhancing LLM answer. Research on improving LLM response has altracted considerable attention Zamani et al. 2020 Hu et al. 2025 Related methods can be divided into two categories: invasive methods that train LLM to generate high-quality response (Pang et al. 2024), and non-invasive methods that improve generation by simply providing in-context demonstrations of the task (Brown et al. 2020), or increasing the model's reasoning ability through various answer refinement techniques (Wei et al. 2022 Wang et al., 2023b h. However, a considerable limitation in these prompt-based methods is that they cannot elicit knowledge absent from training data, which leads to hallucinated response (Huang et al. 2023). especially in specialized domains OpenAA 2023). A related line of approaches improves model response by allowing the model to access external knowledge or information. These proposed methods, known as Retrioval-Augmenter Generation (RAG) Izacard et al. 2023 Wu et al. 2025 Mialon et al.) 2023 Jiang et al. 2023). aim to incorporate LLM with external knowledge to enrich its final response. A retriever is commonly used for searching related model. Despite their potential, RaG-based mothods primarily address the challenges of outdated knowledge rather than resolving specific, ambiguous user queries. Unlike them, this paper focuses on empowering LLMs to seek for the missing context by actively interacting with the user, thereby improving the the answers.
content based on keywords (Lewis et al., 2020) and is jointly fine-tuned with a sequence-to-sequer
Active learning. Active learning (Huang al., 2010 Cohn et al. 1996) is machine learning approach that seeks for an effective sampling stralegy to select the most valuable examples and inquire an oracle for their labels. The objective of active learning is to maximize model performance while minimizing cost for oracle annotation. Existing approaches can be divided into two main categories: the informativeness-bait
methods and the representativeness based methods. The former group samples examples close to the decision boundary to reduce model uncertainty YYuu et al.) 2014: Yan & Huang 2018). (Huang al., 2016 Zhang et al., 2017) explored a variation for neural networks using gradient information as the metric of informativeness in text classification tasks. At the same time, the representativeness- bassd methods constrain the chosen data points to be distinct from each other or conform them to the data distribution (Roy & McCallum 2001 Li ct al. 2020). N-gram or word counts can be regarded as a mcasure of density distribution and standards for sampling practical examples (Ambati 2012 Zhang & Plank] 2021). Preference for instances with more unseen n- a-gramss (Erdmann et al. 2019 is another approach to selecting more representative samples. Although initially designed for supervised learning, active lcarning has shown to be able to assist LLMs in selecting in-context demonstrations (Margatina et al., 2023). In this work, LaMSeI utilizes active learning as a tool to select the most informative clarifying questions to pose to the user.
3

---

Published in Transactions on Machine Learning Research (07 (025)
Answer
(A)
Var(A)
low uncertainty
Answer generation
LLM
Answer
calculation
Variation
Uoe ques
Clatfire quartons
repease
Answer list jindigation Lis
2. Strees or Ausizty...
high unvertainly
3 AAppoddicio:: This mal-- cal Lonapenryy clansteriedd...
Clarifying questions
E User
Any eyssptens-
Active interaction
LLM
x (x:2:((())
User
response U
Question candidates
8
My stomach hurts little, what's going an?
LLM
User query X
Dataflow operation
Conditional path
Figure Illustration of LaMSel method. (A) Overall workflow: the user query X processed by the uncortainty estimation module. Conditioned the estimation result, LLM interact the user directly generates the response. (B) Uncertainty estimation module evaluates the LLM's level of uncertainty regarding query X. (C) In the active interaction module, LLM actively interacts the user by posing clarilying questions. After receiving the user's feedback, LaMSel updates the user query to incorporate this new information and ro-cstimates the unccrtainty. (D) The answer generation module generates the answer to a user query and supplementary user response (if there is user-inleraction).
Uncertainty estimation. Uncertainty estimation [Lakhhmnnaaaaaaan et al. 2017 Xiong et al. 2023 Wang et al. 2025 Lewis 1995 Loquercio et al., 2020 Malinin & Gales 2018) is an important area of research, which measures how confident a neural network is about the predictions it makes. It distinguishes between two main types of uncertainty: aleatoric and epistemic (Abdar et al. 2021 Blundell et al., 2015). Aleatoric uncertainty arises from the inherent noise and randomness present the data itself, which cannot be reduced even with more data. On the other hand, epistemic uncertainty is due the model's lack of knowledge and be potentially reduced the model learns from more data through improved model architectures. In this paper, focus on the later uncertainly, LLM's uncertainty comes from its lack of user context. To estimate these uncertainties, various techniques have been developed. Bayesian Neural Networks (BNNs (Goan & Fookes 2020)) offer a principled statistical framework by placing priors over the model weights, thus allowing for direct estimation of uncertainty. Monte Carlo Dropout (Gal & Ghahramani 2016) is a practical method that approximates Bayesian inference by using dropout at test. time to sample from the weight distributions and thus capture uncertainty. Ensemble methods (Lakshminarayanan et al. (207)), where multiple models are trained and their predictions are aggregated, can also be used to gauge the variability in predictions a proxy for uncertainyy. However, the training of multiple models can be infeasible in the context of constrained computational resources when constrained to black-box model. An alternative approach is to generate multiple samples of model outputs and assess the extent of their divergence (Chen & Mueller 2024). In our study, LaMSel estimates LLM uncertainty by sampling a multitude of responses, with particular emphasis on the semantic divergence of these responses.
3 Method
This section introduces our primary contribution, Language Model Selective Interaction (LaMSel), which equips LLM with the ability to interact with the user for uncovering the missing user context. We describe the problem formulation (Sec. 3.1), the details of LaMSel (Sec. 3.2). and practical implementation of LaMSeT (Sec. 3.3 in the following subsections.
4

---

Published in Transactions on Machine Learning Research (07 (025))
3.1 Problem Formulation
We consider common scenario where the is having a conversation with the LLM M, which takes the user query X as input and outputs the answer based on the user query; Y = =M()). We expect LLM to generate helpful response to the user query.
However, the user query may unintentionally leaves out some critical context/inne /infttmatioonnnnn that LLM cannot access, resulting in the LLM generating less helpful answers. To complete the missed piece of context, LLM actively interacts with the user with a set of clarifying questions Q. Then, the user will provide the feedback U()) to the questions as supplemental clarification the initial user query. The interaction above can be repeated several times to ensure that the LLM accurately grasps the user's intent and obtains enough context. to answer the query. Finally, the LLM outputs the answer based on the integrated original user query and interaction content.
3.2 Language Model Selective Interaction
The overall workflow of LaMSel is presented in Fig. 2 Given a user query X, LaMSel evaluates LLM's uncertainty regarding the query. If the uncertainty is high, LaMSel inquires the user to clarify the ambiguity in the initial user query and then augments the user query with user feedback as additional information. Then, LaMSel re-estimates the uncertainty regarding the updated query. This process is repeated until the LLM's uncertainty about user query is salisfactorily reduced. LaMSel consists of three key components: (1) uncertainty estimation that evaluates LLM's uncertainty about user query X, (2) active interaction that asks user clarifying questions to uncover the missed context, and (3) answer generation that generates the answer based on the integrated information. We will elaborate on these three components in the following.
3.2.1 Uncertainty Estimation by Multiple Answers Sampling
The uncertainty estimation module evaluates LLM's uncertainty about a user query X and determines whether to interact with the user actively. The previous study verifies that the model's uncertainty to an input. can be estimated from an ensemble of the model outputs (Lakshminarayaaan et al. 2017 Thurin et al. 2019). In our sctting, the model output is in the form of text. The uncertainty can be reflected by the semantic consistency of multiple answers. To achieve this, LaMSeI samples multiple answers A2. ... Ar], where each answer A; .M() is generated by the LIM under appropriate temperature parameters. To estimate the uncertainty, LaMSel applies embedding model to convert the array of answers into corresponding text embeddings (E1, E2, Er). Here, we use a embedding model rather than LLM embeddings because these third-party embedding models are often optimized for general-purpose semantic similarity tasks and have been extensively tested for their ability to capture nuanced relationships between texts, which is critical for accurately estimating the uncertainly of our LLM's responses. A high variance in these embeddings indicates greater uncertainty by the LLM regarding the query, whereas a low variance suggests confidence. The variation of the answer is calculated follows:
K
T EE E+12
Var(A)
1
-
(1)
K
T
k=
where K is the dimensionality of the embeddings, T is the number of sampled responses, D denotes the value of the i-th answer for the k-tth dimension, and E* = across all T embeddings.
E F the mean value for the k-th dimension
Subsequently, LaMSel operates distinct pathways, conditioned on the uncertainty estimation outcome. If the uncertainty is low, i.c., Var(A) < 8 (ō denotes the active interaction threshold), LaMSeI outputs the answer A = M(X). Otherwise, in cases of higher uncertainty, it indicates that LLM requires more clarification from the user for better understanding of the user query. should be noted that low variance does not guarantee absolute correctness, as uncertainty may still exist certain cases. The paper uses the low variance threshold as heuristic to guide the LLM to interact with users only when necessary, thereby improving the efficiency and reliability of the model's responses.
5

---

Published in Transactions on Machine Learning Research (07 (025)
3.2.2 Active interaction with Selective Clarifying Questions
Active Interaction Module (AIM) enables LLM to interact with the user, ask clarifying questions, and better understand the user's intent. The primary focus on how to formulate useful clarifying questions. A direct yet effective approach is leveraging LLM to generate the questions (Arora et al. 2023). Specifically, AIM prompts the L.M to generate set of questions Q regarding the user's input X. The specific prompts used in the experiments are presented in Appendix C Nevertheless, questions generated this way may not be uniformly helpful and could increase the user's burden by raising too many questions simultancously. To mitigate this, it essential to filter and present only the most informative questions to users. To achieve this goal, we actively select most useful clarifying questions to enhance the LLM's understanding of user intent. Given a set of N potential questions 2, the goal is to select a subset of M (M I N) clarifying questions Q =(Q).. We explore two prevalent active learning strategies for this selection process, denoted as S.
1. Similarity-based sampling strategy focuses identifying questions that closcly align with the user query. To achieve this, LaMSel first extracts all questions and user query embeddings. Cosine similarity is then employed to measure the degree of semantic correspondence among embeddings, allowing for selecting the top M questions that exhibit the greatest relevance to the user's query. The underlying rationale behind this strategy the intuition that a higher degree of similarity correlates with a richer provision of pertinent information to address the user's query.
2. Diversity-based sampling strategy aims to capture a broad spectrum of questions. This is accomplished by encoding all questions within the set Q using an embedding model. Following this, the K- K-Means clustering algorithm (MacQueen et al. 1967) applied to categorize the question embeddings into M clusters. From each cluster, one question is chosen at random. This method ensures that the selected questions represent diverse and extensive range of information, which enhance the comprehensiveness of the questions gathered.
The system presents the user with this set of clarifying questions Q The user responds to these questions successively, providing clarification, denoted as U()). Following this, the original user query is augmented
to include the user's clarifications, resulting in an updated query: X + [X; Ô U(]),, which we call it interaction-qugmentee query. This augmentation draws upon the directional stimulus prompt technique (Li et al. 2023b), which has been shown to improve the model generation. Subsequently, LaMSel re-estimates LLM's uncertainty regarding the updated user query. This uncertainty estimation and active interaction process continues iteratively until the uncertainty falls below the active interaction threshold 8 or the maximum number of iterations is reached.
3.2.3 Answer Generation with Interaction-augmentte Query
With the interactton--augmeneee query induced by AIM, the LLM can generate the answer to this query enriched with the additional context of user clarifications. Note that LaMSeI does necessarily inquire the user if the LLM's uncertainty about the initial user query has already been low. For clarity, the answer generation process is described here and in Fig. 2 (D) under the condition that user interaction occurs.
3.3 Practical Implementation
In the design of LaMSeI, we present an iterative interaction between the user and the LLM. While this iteralive process can be laborious and lime-consuming due to the requirement for the user to respond to clarifying questions, our practical implementation streamlines this to a single user interaction iteration. This refinement is sufficient to elucidate the user's initial query. We present this singlo-iteration implementation process in Algorithm 1 detailed in Appendix B We apply 2022 to convert text to embeddings.
saabedding-ada-aee model (Greene et al.

---

Published in Transactions on Machine Learning Research (07/2025)
Table 1: Experiment results on various Q&A datasets. LaMSel outperforms oracle method that takes supporting facts imput on certain datasets. Ace stands for the accuracy evaluated by ChatGPT. Excluding Oracle's results, the (nearly) best results are highlighted in bold.
Method
Datasss
HotpotQA
StrategyQA
ZW
MuSiQue
HIR
F1
EM
Acc.
BM
Ace
F1
EM
Acc
57:2
57.8
14.7
20.8
CoT
32.6
41.6
48.1
66.7
66.9
96.9
31.0
32.5
49.6
5.5
12.8
19.8
17.5
22.0
24.6
SelF-ask
27.7
38.0
58.6
34.3
63.2
34.3
42.5
49.2
54.0
15.0
27.0
28. 5
10.1
30.
16.8
Self-ask (web)
16.2
25.4
47.9
33.1
63.3
38.1
28.7
36.3
51.1
19.0
20.2
27. 8
6.0
10.7
25.6
CRUD RAG
37.6
37.0
45.0
18.3
23.1
25.0
35.0
Oraele contant
59.4
86.9
72.4
78.2
78.2
78.1
60.5
72.0
83.5
21.0
33.5
36.
29.4
62.5
42.6
LaMSal
47.5
66.4
51.1
Coo
49.1
69.6
71.7
71.7
71.7
61.1
73.0
18.5
27.7
31.5
36.6
46.1
4 Experiment
In this section, we evaluate the efficacy of LaMSeI through extensive experiments on diverse, challenging
datasets. The objective is to validate LaMSel's capacity lo improve LLM's understanding of user queries and to improve the quality of the responses provided. We aim to answer the following essential questions: (1) How does LaMSel perform in comparison to current answer generation methods across different themes of user query (Sec. 4.2 4.5
(2) Can LaMSeI gain useful information from the selective interaction (Sec. 4.3)? (3)
Can LaMSeT be integrated with various LLMs (Sec. 4.4h² and (4) What is the impact of each component and parameter on the performance of LaMSel (Sec. 4.7 We begin with introducing the experimental setting.
4.1 Experimental Setup
In this subsection, we first roduce how to construct user interaction framework to conduct our experiments. Then, we detail the datasets used the experiments, followed by the introduction of the baseline methods. Finally, we present implementation details and the evaluation settings.
Setup for LLM-user interaction. Our experiments require a procedure wherein an user interacts with the LLM. An ideal setup would involve human user engaging with the LLM, responding to its clarifying questions. However, this setup is impractical due to the high costs and labor-intensive nature of human participation. Follow existing work (Li et al.) 2024), we employ GPT-4 as proxy for human user in the majority of our experiments. We use datasets in which each problem comprises a query paired with supporting facts that serve as a context for the query. The nature of these datasets is particularly suited for our experiments, as they are designed to simulate the absence of user-provided context. Note that ensure comprehensive evaluation, we also conduct experiments with actual human participation, which are described in Sec. 4.5
Dataset. We conduct experiments on five challenging Q& A datasets alongside a dataset dedicated to meeting summarization. Tab. 2 shows some examples from these datasets. In our experiments, all methods, if not specifically marked, are agnostic to the supporting facts. These datasets are: (1) HotpotQA (Yang et al. 2018) is collected on the English Wikipedia. Each question in the dataset comes with the two gold paragraphs, as well as list of sentences that crowdworkers identify as supporting facts necessary to answer the question; (2) Strategy (Geva al.) 2021) consists of questions similar to that in HotpotQA, but the answer format is limited to True False; (3) 2WikiMulttHHpQQZ (Ho et al.) 2020 uses structured and unstructured data and introduces the evidence information containing reasoning path for multi- questions; (4) MuSiQue (Trivedi et al. 2022 is multi-hop QA datasct with 2-4 hop questions using seed questions from five single-hop dalasets; (5) IRRC [eerguson et al. 2020) is a dataset for incomplete information reading comprehension, providing only partial information to answer them, with the missing information occurring one or more linked documents; (6) QMSum (Zhong al. 2021 contains dialogue
histories of multi-donain meeting. The user queries LLM to answer specific questions about the meeting, while some parts of the dialogue masked.

---

Published in Transactions on Machine Learning Research (07/2025)
Table 2: Examples user query, supporting facts, and correct answer for the datasets used in the experiments.
HotpotQA
StrategyQA QA
User query
Were Up and The Watercolor released in the same year?
Are more people today related to Genghis Khan than Julius Caesar?
Supporting facts
Up and The Watercolor are two films. Up was released in
Compare the number of their offispring- Julius Caesar had three children. Genghis Khan had sixteen children...
Label
Yes
True
Baseline. We compare LaMSel with five representative baseline methods that are widely applied in the community, (1) Direct Generation (DG) directly generates the answer with the deterministic output of the LLM; (2) Greedy asking (GA) is bascline that greedily asks the user N clarifying questions. It directly prompls the LLM to generate N clarifying questions to pose lo the user. (2) Chain of Thought (CoT) (Wei et al. 2022) generates the answer by prompting the LLM to reason the results through a series of intermediate reasoning steps. We use the Let's think step by step' prompt with few-shot demonstrations
in the experiments; (3) Self-ask (Press et al.) 2023) poses and responds to the self-generated follow-up questions, refining its understanding before providing the initial query's response; (4) Self-ask (Web) extends the Self-ask method by integrating a web-search API, specifically the Bing search incorporate external knowledge into its answering process; (5) CRUD-RAG (Lyu et al. 2024 uses RAG to retrieve relevant paragraphs sentences from vector databases. Here, we use all all-mpnet-base-T model from teitencc-trrassormmee library (Reimers & Gurevych| 2019) to extract embeddings and compute cosine similarity. (7) Oracle generates the answer directly but is provided ground-truth supporting facts.
Evaluation and Metrics. We evaluate various methods on the first 400 questions from the training set across five Q& datasots. To ensure fair comparison, we employ in-context demonstration for all mothods, selecting two random examples from each dataset to serve as demoonstratioss. In this way, the LLMs are able to generate answers in standardized format. We evaluate performance using exact. match (EM) and Fl scores across the Q&A datasets. Following Iter-RetGen (Shao et al. 2023 also evaluate answer accuracy (Acc) using the 3. t-rbr model, as direct calculation of answer accuracy is not feasible on these datasets. For the QMSum dataset, which contains long lexl, we apply gpt-4 to evaluate the answer preference over the DG method.
Implementation Details. We utilize ChatGPT (OpenAl 2023) as the primary model for our experiments, popular model in the field. To reduce human costs in providing feedback to elarifying questions, we deploy ChatGPT in dual capacities: experiment. We present the queries from the dataset to the LLM, while the supporting facts are provided to only the GPT-4 model. In this way, the user query needs more context and information to be answered. To ensure the experiment's integrity, we restrict GPT-4 from accessing the original user questions when it answers the clarifying questions. We also consider LaMSel+Cool method, which equips LaMSel with Let 's think slep by step prompl and incorporate remonsttationn to provide a fair and compelitive comparison. The experiments are conducted with 2 NVIDIA 3090 and AMD EPYC 9654 96-Core processor. More detailed information regarding the experimental setting, such prompts and parameters, is available in Appendix
LLM (GPT-3.5) and as pseudo-human interlocutor (GPT-4) in the
4.2 Main Results
Performance on Queries with Incomplete Context. Tab. 1 presents the comparative performance of LaMSel against baseline methods across five Q&& datasets. Overall, LaMSeI consistently surpasses the bascline mothods, achieving an average answer accuracy of 50.9%, which marks a significant improvement compared to the basic Q&A framework DG (31.9%). This result underscores LaMSel's superiority in improving model's answer on queries with incomplete context. The CoT method, specifically designed to enhance the reasoning capabilities of LLM, also falls short of LaMSel's performance. The limitation of CoT
I Bing search APT accessible htttt::////ee http://www.ttaataatto/////aaaaaa search-ar

---

Published in Transactions on Machine Learning Research (07 (07/2025)
for our setting lies in that it completely relies on the LLM's embedded knowledge, which does not actively seek out additional context that may be absent in the user's query. Similarly, the Self-ask strategy, which also leverages the LLM's embedded knowledge, cannot address the gaps in information presented by the user, resulting in performance disparity with LaMSeI. Self-ask (web) is a typical RAG-based method that seeks information from the web. The experiment. results of Self-ask (web) suggest that the external knowledge base does not demonstrate an advantage dealing with ambiguous user queries, as indicated by close performance to the DG method. CRUD-RAG is another RAG-based method that builds a vector database and leverages the retriever to retrieve relevant content for specific queries. The result reveals that directly retrieving related information and then using it as context information is inferior to LaMSeI 's iterative form. Notably, LaMSel achieves comparable performance with the Oracle method on HotpotQA and StrategyQA and outperforms it on IRCC. These results provide strong evidence that LaMSel elfectively acquires valuable information from the user, improves the model's comprehension of user queries, and leads to improved performance.
50
BS Sct
63.0
23.4
13.6
40
Rate
30
IS Set
63.6
21.9
14.4
48
20
29
23
TS Sct
61.8
20.3
17.9
10
Win
Tie Outcome
Lose
Win
Lose
(a) GPT-4 Evaluation
(b) Human-participateed Experiment.
Figure 3: Comparative and analytical results for LaMSel on the QMSun datasel: (a) GPT-4-based evaluation showing LaMSel's win rate against DG on three subsets of QMSun. (b) Human win rate of LaMSel output against DG. In this experiment, LaMSel interacts with a human participant, who accessos to the supporting facts.
GPT-4 evaluation on QMSum. We conduct experiments on the QMSum dataset, wherein the language model is asked summarize specific perspectives based on the meeting's dialogue history. Assessing the quality of the summaries generated by the model is a significant challenge, traditional reference-based metrics like BLEU and ROUGE often correlate poorly with human judgment. In our study, we employ GPT-4 as frferenco-free metric for evaluation, which has demonstrated high degree of alignment with human assessments (Liu et al.] 2023). Fig. 3(a) illustrates that LaMSel surpasses the DG method by a margin exceeding 60% in win rate and 20% in tie rate. These results underscore the efficacy of LaMSeI, which, through active interaction, garners valuable information from users to accurately respond queries, thereby demonstrating the effectiveness of approach.
4.3 Performance of Selective Interaction
Comparison with greedy asking. To verify whether the selective interaction LaMSeT effectively selects meaningful questions to interact with the user, we conduct experiments on QMSum to compare LaMSel with greedy asking (GA) method that greedily interacts with users with five clarifying questions by prompts LLM to generate the questions. The results are presented in Tab. where each score represents the method's winning rate over direct gencration method. There are two main observations: (1) LaMSeI outperforms GA method, although asking only three clarifying questions during the interaction, as opposed to GA's 's five questions. This results demonstrate that LaMSel can pose effective clarifying questions through selective interaction and active lcarning question sclection.
(2) All these intoractive-LLM method improve performance over DG method, achieving more than 60% winning rate over direct answer generation. This results highlight the importance of interacting with users to seek for more context.

---

Published in Transactions on Machine Learning Research (07/2025)
Table Comparison of LaMSel with greedy asking. GA method greedily asks five clarifying questions before answering, while LaMSeI asks only three questions when interacting. The score in the table represents the method's winning rate over DG method.
GA
LaMSel (Diversity) LaMSel (Similarity)
Score
63.04
63.04
65.76
Analysis on model outputs. Previous results have demonstrated LaMSel enhances the capability of LLM to generate improved responses. To investigate the source of this enhancement and its inffuence on the LLM's understanding on user queries, we conduct a case study analyzing the running process of LaMSel method, which covers successful case and failed cases, as shown in Tab. g in Appendix. The analysis yields several critical insights. Firstly, LaMSel can raise effective clarifying questions
seek clarification for the user
query. For instance, Q1 in the success case, LaMSeI inquires about the specific actions that lead to injuries, aligning precisely with the supporting facts. Secondly, applying active learning strategies filters the most informative questions from the question candidates. Overall, the quality of clarifying questions directly determines whether LaMSel successfully resolves ambiguily and generates accurate answers. In the success case, the selected questions (e.g., 'Are there specific movements in waltz /slam dancing that could lead to injurics?') preciscly target the core of the user query ('Is waltz less injurious?"). These questions align with the supporting facts (e.g,, slam dance involves 'collisions") ) enabling LaMSel extract critical context about injury mechanisms and produce correct answer. Conversely, in the failure case, the selected questions (e-g., "Other income sources for Billie Eilish?' or "Average Porsche price?") are misaligned with the query's key uncertainty ("Can she afford it?"). They overlook the decisive evidence in the supporting facts, lcading to an incorrect answer despite sufficient information being present. This contrast confirms that high-quality questions must directly address the information gap implied by the query and supporting context; irrelevant or overly broad questions fail to reduce uncertainty and degrade answer accuracy. Future work could better leverage this feature to generate clarify questions.
Table 4: Experiments on various open-sourced LLMs with with/withon LaMSeT technique. LaMSel consistently improves the performance of different LLM backbones.
Dataset
HotpotQA
StrategyQA
2Wiki*
MuSiQue
IIRC
Method
EM
EM
Acc
EM
Acc
EM
Acc
EM
Acc
LLaMA2-7B-DG
25.3
50.3
60.6
15.2
37.0
1.3
13.7
15.3
LLaMAA-7B-CoT
15.8
28.3
56.1
67.4
20.0
29.3
2.0
14.0
16.3
LLMMA2-BB-LMMal
8.7
19.8
41.6
43.6
25.6
69.7
3.0
23.5
12.0
21.3
LLMMA2-BB-LaMS--CoT
21.3
33.6
51.4
51.1
39.7
62.0
10.7
32.5
12.3
22.3
Vieuma-BB-DG
2.0
19.0
50.5
47.6
8.1
8.7
1.0
7.0
6.5
Vicuna-BB-oo"
12.5
29.1
56.6
58.6
12.0
13.5
1.5
10.0
9.2
17.5
Seuua--B-LaMir
27.3
55.9
57.6
59.1
18.7
28.2
7.7
24.0
19.3
35.1
29.1
56.9
62.4
63.9
33.2
31.0
7.9
25.2
21.5
$8.1
LLaMA3-8B-DC
15.9
22.0
65.5
64.0
18.3
36.5
3.0
16.7
15.4
LLAMA3-8BBCoT
17.8
40.8
59.0
59.1
29.6
36.2
4.1
17.7
13.1
18.5
50.5
66.4
62.1
61.2
49.5
58.6
11.0
32.0
19.3
25.4
MaaMA--BP LaMMell-Co
52.6
67.2
79.5
76.0
51.3
65.1
15.2
32.7
31.6
40.1
Vicuna- Vicuma-I3B-DG
12.0
36.8
56.8
57.8
10.0
32.3
9.0
15.0
3.5
9.5
Vicuna- 13B-CoT
20.0
38.1
60.3
60.9
30.8
40.5
11.5
17.2
5.5
11.1
Micuma-13B-Lan
34.8
60.1
55.8
56.1
12.3
58.5
10.0
33.0
17.0
24.1
Vieuma-13B-LaMSe-CorT
33.8
60.6
65.6
67.1
32.5
63.3
12.0
30.2
18.8
24.2
4.4 Applicability on Different LLMs
We verify the applicability of LaMSel by assessing its performance on different LLMs. We conduct experiments
with LLaMA2-7B (Touvron et al. 2023). Vicuna-7B (Zheng al. 2023b), L.MMA3-BB (Tam 2024a) and
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Vicuna- 13B. These models are all from the open-sourced version. The experimental setup is consistent with the main experiments, with the results shown in Tab. 1 The results demonstrate that. LaMSel surpasses the DG and CoT methods on two datasets. This suggests that LaMSel is well-suited to models with fewer parameters. Besides, LaMSel can improve the performance a wide range LLMs, justifying the applicability of the proposed method.
4.5 Experiments with Human Participant
In addition to experiments employing GPT-4 as a simulated human interlocutor, we also conduct human- participated experiments where actual human interaction is integrated. The experiment is conducted using the QMSum dataset, from which 100 user queries are randomly selected for evaluation. We invite five participants to the experiment, each accounting for 20 queries. Participants respond to clarifying questions posed by the LLM, which subsequently generates answers informed by the human response. Subsequently, another participant compares these anonymous answers generated by LaMSel and DG and annotates preference for one. The experimental results are illustrated in Fig. 3(b) The human evaluation results reveal that LaMSeT perforosms or is comparable to DG in 77% of the instances. This result demonstrates LaMSeT's superior ability to comprehend user input during human interaction and verifies the feasibility applying LaMSeI in real application.
Table Performance of LaMSeT on user query with different levels of context masking. We use GPT-4 with evaluation prompt (Fig. assess the response quality of LaMScI against DG.
ES
1S
TS
Mask Rate
Win
Lose
Tie
Win
Lose
Tie
Win
Lose
Tie
0
47.86
31.21
17.90
48.13
36.90
14.97
55.28
30.08
14.63
0.3
55.14
23.67
21.22
58.70
31.52
9.78
48.96
34.38
16.67
0.5
63.04
23.35
13.62
63.64
21.93
14.44
61.79
20.33
17.89
0.7
66.17
21.05
20.54
71.20
13.91
15.71
62.12
25.00
12.88
4.6 User Query with Less Context
LaMSel actively seeks clarification from users to gain additional context regarding their queries. To evaluate this capability for context-seekinne seeking. we conduct experiments on the QMSum dataset, with varying masking ratio applied to the dialogue history. We implement the context mask by directly masking the tokens of support facts that are exposed to the LLM. As the masking rate increases, the availability of contextual information to the LLM correspondingly decreases, allowing LaMSeI to demonstrate stronger relative gains by actively seeking missing information. As the experimental results are shown in Tab. 5 when larger proportions of context are masked, LaMSel demonstrate more significant advantage over the DG method. This improved performance is attributed to LaMSel's ability to seek useful context from the user, which mitigates the challenges of insufficient contextual data. Furthermore, is observed that LaMSeI maintains its efficacy in eliciting additional information to answer the user query, even when the full context is available, thereby enriching its responses to user queries.
4.7 Ablation Study
In this subsection, we conduct an ablation study to investigate the influence of different components of LaMSeI the algorithm performance, as the results presented in Fig.
Number of clarifying questions. The number of clarifying questions posed to the user, denoted as M, is an essential parameter in LaMSel framework. These ablation results are presented in Fig. 4 Overall, there is a clear improvement in LaMSel's performance correlating with an increase in the number of clarifying questions M. This trend substantiates the efficacy of LaMSel's clarifying questions mitigating the contextual gap between the LLM and the user. We observe that M 3 is sufficient achieve notable performance gains. This performance improvement occurs because the L.MSs can pose more questions, gathering additional
11

---

Published in Transactions on Machine Learning Research (07 (025)
70
Diversity Similarity
Diversity Similarity
Diversity Similarity
Win Rate vs. DG
Win Rute vx. DG
60
60
Win Rate vs.
60
50
40
4
3 (b) Results on IS
4
3
Number Claritying Questions
Number of Clarifying Questions
Number of Clarifying Questions
(a) Rosults on ES
(c) Results on TS
Figure 4: Ablation study on the impact of the number of clarifying questions and active learning strategies on (a) ES dataset, (b) IS dataset, and (c) TS dataset. The values in the figure represent the win rate of LaMSel against DG method.
pertinent context to address the user's query. While larger M potentially contributes to superior performance, it also imposes heavier demand the user provide leedback. Therefore, it essential to balance user experience with the quality of the model's responses. Thus, recommend adopting moderate value of M, such 3, for practical applications of LaMSel.
Active Learning Selection Strategy. LaMSel employs active learning strategy to select informative questions to inquire the user. We present the experimental results of LaMSel using distinct active learning selection strategies, as illustrated in Fig. 3 Overall, the similarity strategy, which prioritizes selecting clarifying questions that are semantically similar the user's query, outperforms the diversity strategies. These findings suggest that focus diversity may not be as effective in sifting out the most informative questions, and that the similarity of the questions to the user's query is more critical factor.
Active Interaction threshold. In LaMSel, the active interaction threshold, denoted as is important in determining the instances when the LLM actively interacts with the user. To evaluate the impact of on LaMSel, we conduct experiments with different values of 6. The results in Tab. c indicate an enhanced performance of LaMSeT at lower 8 values, which correlates with a higher propensity for actively inquiring users. Notably, at more significant threshold, e.g., ô 0.015, the performance of LaMSel approximates that of the DG method, attributed to the reduced frequency of active intcractions. This observation encourages to set relatively low d value to enable active interaction users by the LLM.
ES
IS
TS
5
Win
Lose
Tic
Win
Losc
Tic
Win
Losc
Tic
0.005
49.42
15.56
35.02
42.78
16.57
40.65
40.65
12.20
47.15
0.010
23.57
10.72
65.71
20.32
6.95
72.73
19.51
5.69
74.80
0.015
11.28
2.33
86.39
10.16
3.74
86.10
8.94
4.87
86.19
Table Ablation study on the offect of the active inquiry threshold on the QMSum dataset. We use GPT-4 assess the response quality of LaMSel against DG.
with evaluation prompt (Fig. G
5 Conclusion and Limitation
This paper highlights the need for LLMs to be interactive to better understand user's intent. To achieve this, propose novel LaMSel method, which empowers LLMs to selectively interact with users, effectively handling user queries with incomplete context. Comprehensive experiments demonstrate that LaMSel clearly improves LLM's grasp of user intent, uncovers the missed context and leads to more reliable model response. Despite the superior performance, LaMSel has certain limitations. First, LaMSel straightforwardly prompts LLM to generate a set of question candidates. As shown in our case study Sec. 4.3] this process may only sometimes yield sufficient and informative questions even after the active learning process. As potential improvement, there has been learning-basee based method (Komelli et al. 2022 that shows promising results in
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
generating practical clarifying questions by training query generator. Second, the current embedding-based method for assessing the LLM's uncertainty about a user's query assumes that the embedding model is able to accurately captures the semantic information of the original text. An alternative could involve estimating the model's uncertainty based on the likelihood of the generated response. Lastly, our experiments utilize sampled questions from existing Q&A datasets to mimic user queries, which may only partially represent. actual user interactions the real world. It would be interesting to evaluate LaMSeI's performance when facing more realistic user queries and interacting with humans. We encourage future research to explore these interesting fields and develop more effective approach for interactive LLMs.
Acknowledgement
This work is supported by the NSFC (U2441285, 62222605). The authors extend their appreciation anonymous reviewers for providing valuable comments during the peer-review process.
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavanzadehi, Paul W. Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncortainty quantification in doep learning: Techniques, applications and challenges. Information Fusion, 76:243 297, 2021.
Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. Asking clarifying questions in open-domain information seeemattoo--eeking conversations. In SIGIR, 2019.
Vamshi Ambati. Active learning and crowdsourcing for machine translation in low resource scenarios. PhD thosis, Carnegie Mellon University, 2012.
Chinmaya Andukuri, Jan-Philipe Fränken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions. arviv, /tt///243.1955// 2024.
Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Ré. Ask me anything: simple strategy for prompting language models. In ICLR, 2023.
Richard Bemile, Akwasi Achampong, and Emmanuel Danquah. Online hotel reservation system. International Journal of Innovative Science, Engineering & Technology, 199):583-56) 588, 2014.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural networks. Xiv, abs/ has/1505.05424, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In ACL, 2024.
Anni Coden, Daniel Gruhl, Neal Lewis, and Pablo N. Mendes. Did you mean a or b? supporting clarification dialog for entity disambiguation. In SumPre, 2015.
David A Cohn, Zoubin Ghahramani, and Michael T Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129 145, 1996.
DeepSeek-Al. Deepseek-r1: Incentivizing reasoning capability in lmns via reinforcement learning, arXie, abs/22501.12945 2025.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling In ACL, 2022.
13

---

Published in Transactions on Machine Learning Research (07 (025)
Alexander Erdmann, David Joseph Wrisley, Christopher Brown, Sophie Cohen-Bodénéss, Micha Elsner, Yukun Feng, Brian Joseph, Béatrice Joyeux-Prunel, and Marie-Catherine de Marneffe. Practical, efficient, and customizable active learning for named entity recognition in the digital humanities. In ACL, 2019.
James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. IRRC: A dataset of incomplete information reading comprehension questions. In EMNLP, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dalaset of diverse text for language modeling, arXiv, abs/2101.00027,, 2021.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? A question answering benchmark withimplicit reasoning strategics. Transactions of the Association for Compulational Linguislics, 9:346 361, 2021.
Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. arXie, abs/2006.11 2024, 2020.
Google Deepmind. Introducing gemina 3: The most capable model you can run on single gpu or tpu. 2025.
Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. New and improved embedding model, 2022. http:://ppaaaa..ccomol/llaaaarre
Maxime Griot, Coralie Hemptinne, Jean Vanderdonckt, and Demet Yuksel. Large language models lack essential metacognition for reliable medical reasoning. Nature Communicationss, 16(1):642, 2025.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In COLING, 2020.
Yinghao Hu, Leilei Gan, Wenyi Xiao, Kun Kuang, and Fei Wu. Fine-tuning large language models for improving factuality in legal question answering. In COLING, 2025.
Jiaji Huang, Rewon Child, Vinay Rao, Hairong Liu, Sanjeev Satheesh, and Adam Coates, Active learning for speech recognition: the power of gradients. arXiv, abs/ 0t///1112.022266 2016.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv, abs/2311.052322 2023.
Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou Active learning by querying informative and reprosentative examples. In NewrIPS, 2010.
Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yii Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24:251:1 251:43, 2023.
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In EMNLP, 2023.
Mojtaba Komcili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue gencration. In ACL, 2022.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
David D. Lewis. sequential algorithm for training text classifiers: Corrigendum and additional data. SIGIR Forwm, 29(2):13-14 19, 1995.
14

---

Published in Transactions on Machine Learning Research (07 (225))
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petromi, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive NLP tasks. In NewrIPS, 2020.
Changsheng Li, Handong Ma, Zhao Kang, Ye Yuan, Xiao Yu Zhang, and Guoren Wang. On deep unsupervised active learning. arxiv, abs/2200. 13959, 2020.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction luning. arXiv preprinl ar Xiv:2308 12032, 2023a.
Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. MEDIQ: question-asking Ilms for adaptive and reliable clinical reasoning, arXiv, abs/2406.0082 00922, 2024.
Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. Guiding large language models via directional stimulus prompting. In NeurIPS, 2023b.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NL. evaluation using gpt-4 with better human alignment. In EMNLP, 2023.
Antonio Loquercio, Mattia Segd, and Davide Scaramuzza. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 5(2):3153 3160, 2020.
Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. Crud-rag: A comprehensive chinese benchmark for retrieval-angnented generation of large language models. arx, abs/2401.17044 17043, 2024.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In BSMSP, 1967.
Andrey Malinin and Mark J. F. Gales. Predictive uncertainty estimation via prior networks. In NewrIPS, 2018.
Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yi Yu. Active learning principles for in-context learning with large language models. In EMNLP, 2023.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Fdouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey, arXiv, abs/2302.07742, 2023.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. arXiv, abs/2004.10645, 2020.
OpenAl. GPT-4 technical report. arXiv, abs/2303.08774, 2023.
Jing- Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-improvement by reinforcement lcarning contemplation. In ICLR, 2024.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In EMNLP, 2023.
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP-IJCNLP? 2019.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. In ICML, 2001.
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlic Huang, Nan Duan, and Weizhu Chen. Enhancing retricval- augmented large language models with iterative retrieval- -eenerrtnnnnnoionnn synergy. arXiv, abs /hbs//200... 15294, 2023.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Zhengxiang Shi, Yue Feng, and Aldo Lipani. Learning to execute actions ask clarification questions. In NAACT, 2022.
Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. Towards natural clarification questions in dialogue systems. In AISB, 2014.
Meta Team. Introducing meta Ilama 3: The most capable openly available Ilm to date. https://ar meta com/blog/meta-11 -ateettaaaaaaa/////// 2024a.
Qwen Team. Qwen2.5 technical report. arXiu, abs/2412.1511 2024b.
Julien Thurin, Romain Brossier, and Ludovic Métivier. Ensemble- massemble-basd uncertainty estimation in full waveform inversion. Geophysical Journal International, 219(3):1613 1635, 2019.
Bernadette Tix and Kim Binsted. Better results through anbiguity resolution: Large language models that ask clarifying questions. In HCT, 2024.
Hugo Touvron, Louis Martin, and et al. Llama 2: Open foundation and fine-tuned chat models. arXie, abs/2307.09223 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguisties, 10: 539-554, 2022.
Ante Wang, Linfeng Song, Ge Xu, and Jinsong Su. Domaīn adaptation for conversational query production with the RAG model feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), EMNLP, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023b.
Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Yue Zhang, Ren Wang, Xiaoshuang Shi, and Kaidi Xu. Word-sequence entropy: Towards uncertainty estimation free-form medical question answering applications and beyond. Engineering Applications of Artificial Intelligence, 139:109553, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. NewrIPS, 2022.
Chenyuan WU, Ninglu Shao, Zheng Liu, Shitao Xiao, Chaozhuo Li, Chen Zhang, Senzhang Wang, and Defu Lian. Lighter and better: Towards flexible context adaptation for retrieval augmented generation. In WSDM, 2025.
Miao Xiong, Zhiyuan HU, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can 1mms express their uncertainty? an empirical evaluation of confidence elicitation 1lms. arXiv, abs//3066.10000 2023.
Yifan Yan and Sheng-Jun Huang. Cost-effective active learning for hierarchical multi-label classification. In IJCAI, 2018.
Janghoon Yang. Context information can be more important than reasoning for time series forecasting with a large language model. arXiv, abs/2502.05669, 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning, Hotpotqa: A dataset for diverse, explainable multi-hop question answering, In EMNLP, 2018.
Xinge You, Ruxin Wang, and Dacheng Tao. Diverse expected gradient active learning for relative attributes.
IEEE transactions on image processing, 23(7):3203 3217, 2014.
Hamed Zamani, Susan T. Dumais, Nick Craswell, Paul N. Bennett, and Gord Lueck. Generating clarifying questions for information retrieval. In WWW, 2020.
16

---

Published in Transactions on Machine Learning Research (07/2025)
Mike Zhang and Barbara Plank. Cartography active learning. arXiv, abs/2109.04288,, 2021.
Ye Zhang, Matthew Lease, and Byron C. Wallace. Active discriminative text representation learning. In AAAI, 2017.
Ziliang Zhao and Zhicheng Dou. Generating multi-turn clarification for web information seeking. In WWW, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv, abs/2306.05665, 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging Ilm-as-a- judge with mt- mt-bench and chatbot arena. arXiv, abs/2306.0500 05685, 2023b.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir R. Radev. Qmsum: A new benchmark for query-basod multi-domain meeting summarization. In NAACL-HLT, 2021.
A Discussion
A.1 Usage of Datasets in Experiments
In the current research, we systematically conduct experiments using a range well-established datasets assess the performance of LaMSeI. These experiments involve creating scenarios where user queries are ambiguous by withholding supporting facts. However, these datasets may only partially represent the complexity and nuances of real-life conversational scenarios where users pose diverse and often intricate questions. To bridge this gap and enrich the experimental outcomes, future research could incorporate questions drawn from everyday human interactions. Additionally, exploring alternative datasets encompassing more extensive chatting histories could yicld more comprchensive insights. Besides, the current experimental framework primarily utilizes GPT-4 OpenAl 2023 as a simulated conversational partner, offering responses to clarification questions. While simulating real-world scenarios by providing GPT-4 access to supporting facts, this setup may only partially capture the conversational style typical of human interactions. The initial phase of our research involved preliminary assessment of LaMSel's competency in engaging with human participants. This provided valuable insights into its interactive capabilities. For more robust evaluation, it would be beneficial to expand the scope of the experiments to include a more comprehensive array of human participants. Engaging these participants in conversations with LaMSel, utilizing their unique inquiries and feedback, would offer a more authentic and varied perspective on the model's interactive performance and capabilitics.
A.2 Active Learning for LLMs
Active learning Huang et al. (2010) is acclaimed for its efficacy in augmenting model performance, particularly in scenarios where obtaining a label is laborious and costly Margatina et al. (2023). Prior research combines active learning and L.Mss, mainly focusing on selecting in-context. demonstrations for LLMss Unlike them, this study, LaMSel leverages active learning techniques by selecting the most informative clarifying questions to inquire user. This strategy enables LLMs to learn from relevant and valuable responses. The utility of active learning extends to its capacity for tailoring interactions based on user feedback, fostering dynamic and uscr-focused engagement.
Nevertheless, there are some challenges to involving active learning techniques. A primary concern is the initial selection and crafting of clarifying questions, which must be sufficiently broad to encompass the user's potential intent while remaining focused enough to steer the model toward constructive clarifications. Furthermore, there exists challenge concerning the potential bias inherent in the queries chosen by the active learning algorithm. Such bias could inadvertently direct the model's learning trajectory in specific,
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
perhaps unintended direction. Addressing this issue, future enhancements might involve the development of more advanced question-generatioo methods. These methods should possess deeper understanding of the muances and context of user queries and incorporate more diverse array of data sources to counteract bias. Additionally, the evolution of active learning to accommodate more intricate, multi-turn interactions could improve the richness and quality of dialogues between users and the LLM. Such advancements would likely result in more sophisticated and precise responses from the LLM, thereby enhancing the overall effectiveness and user experience.
Practical Algorithm of LaMSel
In Sec. 3.2 the LaMSeI method is delineated through an iterative framework. Nonetheless, in practical applications, this iterative process of querying the user to diminish the uncertainty associated with the LLM's undorstanding of the user query may be time-intensive and oncrous. To address these challenges, we introduce a streamlined version of LaMSel, implemented with singular inquiry iteration. The details of this practical implementation are outlined in Algorithm !
Algorithm Practical Implementation of LaMSel
Require: User input X, active learning selection strategy S, active inquiry threshold 6, number of clarifying questions M
Sample a set of answers to user query: {4: = M(X)}
Calculate the variation of the answers Var(A) [
if Var(A) < 8 then
// Low uncertainty
Generate the answer directly: Y = M(X)
else
// Active inquiry
Generate a set of clarifying questions Q
Select questions from the set with active learning strategy: Q SQQ) Inquire the user and get the foodback (())
Generate the answer Y
end if
Return: Answer to user query Y
C More Experiment Details
This section provides more details of the experiments, including prompts, syper-parameters, and more examples of the datasets.
C.1 Prompts
We present the prompts used in the experiments in three topics: prompts used for LaMSel (Fig. prompts based on Zheng et al. (2023a) and prompts for baselines, which depicted in Fig. 6 and Fig. respectively.
5
evaluation
C.2 Soperr-prameters
Tab. presents hyper- parameearssserss used in our experiments. To implement baseline methods, we utilize their official implementation released hyper-parameters.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Prompts for LaMAI
eGerrat clarifying questions for Q&A datasets)
Now you need to ask some questions about the query to help complete the question better. Note that the questions you pose should be useful for assistant to answer the original user query. List your questions order and separate them by" ;
The user query is: {USER QUERY) J. Your posed questions are:
enerate clarifying questions for QMSum)
Here is the dialog history of a multi- turm conversation: (DIA HISTORY),
User query: (USER QUERY) }. Do you have any questions to address the user query?
If so, please list and separate them by Y. Your posed questions are:
(A nswer clarifying questions for GPT-4)
Supporting facts: (SUPPORTING FA CT.. User query: {USER QUERY Please answer the following questions based on the provided supporting facts and user query. You should make full use the supporting facts to answer the questions, and your answer must be helpful.
Questions are: (CLARIFY ING QUESTIONS)
nAnswer generation)
Question: (USER QUERY] J. Y ou should answer the user query based on the hints (a set of question- ansiiorn- pairs),
Please answer the question directly without explanation. Give the answer following the formats of the given examples.
Example question: AMPLM QUESTION). Example answer: AEXAM ANSWER).
Hint 1: CCLARIFYING QUESTION 1}: ER 1}
Hint ICLARIFYING QUESTION 2}: (ANSW ER 2}
3:nt (CLARIFY ING QUESTION 3):
(ANSW ER 3}
Figure 5: Prompts for LaMSel.
General Prompts- GPT-4 Evaluation
Please act as impartial judge and evaluate the quality of the responses to the user task displayed below. Y ou should rate these two answers based on which one is closer to the ground truth answer. Y our evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any positional biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation.
Please provide your explanation, and output your final verdict by strictly following this format:
V Vrriic:: [explanations]
Choice: A, B or C.
"A if answer A is better, "B" answer better and "C" for a tie.
Here is the ground truth answer: (GROUND TRUTH ANSWER).
Two answer candidates:
[A]: (ANSW ER1] [B]: (ANSWER2).
Figure 6: Prompt for GPT-4 evaluation.
19

---

Published in Transactions on Machine Learning Research (07/2025)
Prompts for Baseline eethods
(Direct Generation)
Please answer the question directly without explanation. Just give the answer and do NOT need to give the explanation. ou should follow the formats following examples. Question: Which film has the director bom first, Captain Phantom or Brasileirinho (Film)? Answer. Captain Phantom
Question: {TASK)
(Direct Generation Cot)
Y ou should follow the formats of following examples. First think step by step, then give your own answer. So the answer is: [your answer].
Note that [your answer] should be short and concise, and do NOT need the ex expaanation.
(Direct Generation with Knowledge- Oracle)
Here is the dialog history of multi-turn turlti-tum conversation: (DIALOG HISTORY); }.
User query: (USER QUERY) }. Do you have any questions to address the user query? so, please list and separate them by our posed questions are:
sk)
Question: Who lived longer, Theodor Haecker or Harry Vaughan W Watkins?
Are follow up questions needed here: ess.
Follow up: How old was Theodor Haecker when he died?
Intermediate answer: Theodor Haecker was 65 years old when he died.
Follow up: How old was Harry Vaughan W Watkins when died?
Intermediate answer: Harry V V'augha W Wakkins was 69 years old when he died.
So the final answer is: Harry V Vaughan Watkins.
Are follow up questions needed here: (Here, you can say Y Follow up:...", ;, like demonstrations mentioned above, 'No. the final answer is:... to provide the answer directly)
Figure 7: Prompts for baseline methods.
Name
Value
Num. of clarifying questions M
3
0.005
temperature for uncertainty estimation
0.5
top_ P presence penalty
sample strategy
diversity
Num. of demonstration
2
Table Hyper-parameters used in the experiments.
20

---

Published in Transactions on Machine Learning Research (07 (07/2025)
C.3 Examples of Datasets
We present examples from datasets in Tab. 8 The table shows that each lask consists of a user query and corresponding supporting facts or user intent. LaMSel needs to propose the most valuable questions based on the context to seek clarification from the user to respond with a more refined answer.
User query
Supporting facts / User intent
Label answer
HotpotQA
Musician and satirist Al- Goertz wrote a song about the "The Simpsons' character Milhouse, who Matt Groening named af- ter who?
'Lisa. Marie Simpson is a fic- tional character in the ani- mated television series, The Simpsons. She is the middle child and most intelligent of the Simpson family'...
President Richard Nixon
Strategy QA
Could Lil Wayne legally operate a vehicle on his own at the beginning of his career?
Lil Wayne's career began in 1995, at the age of 12, when he was signed by Birdman and joined Cash Money Records as the youngest member of the label...
False
2WikiMultHHopQA
Are director of film Move (1970 Film) and director of film Méditerranée (1963 Film) from the same coun- try?
Move is 1970 American com- edy film.. and directed by Stuart Rosenberg. The screen- play was written by...
No
Musique
What is the highest point in the country where Bugabula found?
Bugabula one of the five tra- ditional... is located in the Kamuli District. Iran consists of the Iranian Plateau...
1400 meters
IRC
How old was Hokutoumi when he defeated Jingaku Takashi by making him stumble out of the dohyo?
He came from the same area of Japan as future stable-mates Sakahoko and Terao. He was fond of kendo at school. He joined Izutsu stable 1977...
27 years
Table Examples of user query, supporting facts and correct answer for the datasets used in the experiments.
21

---

Published in Transactions on Machine Learning Research (07/2025)
Table 9: case study on the model output for LaMSel, with a success and failure cases. More examples are in Appendix.
User query
Success Is waltz less injurious than slam dance?
Fasture Can Billie Eilish afford Porsche?
Supporting facts
The waltz is a rhythmic dance performed in triple time by a couple. A slam dance is a type of dance in which leaping dancers collide against each other.
Billie Eilish is a famoous female singer. Billie Bilish is 18 years old and has a net worth of $25 Million. A Porsche Boxster is car that starts at $50,000. $25,000,000 is greater than $59,000.
Questions before selection
21: What are the potential injuries associated with slam dancing? Q2: Are there any specific movements or techniques in waltz that could lead to injuries? (10 questions in total)
Q1: Does Billie Eilish have any other expensive assets or investments? Q2: Docs Billie Eilish have any en- dorsement deals or sponsorships that could contribute to her ability to afford a Porsche? ...(10 questions in total)
Questions after selection
Ộ1: Are there any specific movements or tech- niques in waltx that could lead to injuries? Q2: Are there any specific movements or techniques in slam dancing that could lead to injuries?
Ô1: Are there any known sources of income for Billie Filish besides her music career? Q2: What is the average price range for Porsche?
Output
/ Label
True/ / True
False
True
22

---

Published in Transactions on Machine Learning Research (07 (07/)025)
D Additional Results
In this section, we present additional experimental results that are omitted in the main text due to the space limitation.
D.1 Experiments with More Models
To better verify the applicibility of LaMSeI, we conduct experiments with Qwen-2.5-2 2-5 and Swen-22.5144 models Team (2024b) on 2WikiMultHHoQAA and MuSiQue datasets, the results shown in Tab. 10 Overall, LaMSel can also improve these advanced models' performance, outperforming baselines such as DG and CoT, confirming LaMSel's elfectiveness more recent models.
Table 10: Experiments on Qwen models, which are advanced models published recently.
Dataset
2WikiMultHHopQA
MuSiQue
Method
EM
Acc
EM
Acc
17.5
21.5
3.0
9.6
Owen2.5-7B-CCo
6.5
11.5
3.5
11.4
Owen2.5-7BB-LaN
46.0
50.3
12.1
21.8
Qwen225-77BlaaMSee--Co
59.0
63.9
15.5
25.8
Qwen2.5-14BBDDG
2.0
7.0
11.0
2.7
Qwen2.5-14BB-oll
25.5
30.4
16.5
19.3
Oyvon2.5114-LaaMteel
51.0
55.4
18.5
17.8
Qven2.55-44BLaMMaalCCo
51.5
55.7
16.5
17.0
D.2 Ablation Study on Model Temperature
The temperature parameter plays crucial role in controlling the randomness and diversity of the generated responses from the LLM. We have added an ablation experiment to study the influence of temperature, using the model of Q Instruct. As the results shown in the Tab. I1 a higher temperature would lead to more diverse and random responses, potentially increasing the uncertainty estimation. However, performance changes become less pronounced when the temperature exceods a threshold (c.g., > 0.3).
Table 11: Ablation study on model temperature. The experiments are conducted with Qwen2.5-7B Instruct. Treen2.t-BBInstruut...
Temperature
2WikdMulciHopQA MuSiQue
0.1
31.7
20.7
0.3
49.0
21.1
0.5
50.3
21.8
0.7
57.0
20.0
0.9
57.0
21.0
D.3 Results on AmbigQA Dataset
We further conduct experiments with AmbigQA dataset Min al. (2020), which aligns closcly with our focus on clarifying user intent through selective interaction, as the results shown in the Tab. 12 The results demonstrate the effectiveness of LaMSel method on AmbigQA dataset.
23

---

Published in Transactions on Machine Learning Research (07/2025)
Table 12: Experimental results on AmbigQA dataset.
Method
AmbigQA
DG
46.0
CoT
42.5
LaMSel
47.5
LaMScI+CoT
51.0
D.4 Combine LaMSel with Perplexity Measurement
LaMSel implements uncertainty estimation module via multiple answers sampling, which shares similarities with perplexity measurement. In our study, we opted for multiple answer sampling due to its ability to directly reffect the semantic divergence of model outputs, which aligns well with our research focus on semantic-level uncertainty in language models. Despite that, perplexity-based methods could also be valuable and worthy of further exploration. Thus, we conduct experiment with perplexity-based methods, as shown in Tab. [3] The results show that perplexity measurement can also be incorporated LaMSel method, and improves the final answer of LLM compared DG method.
Table 13: Experiments of LaMSel with perplexity measurement for uncertainty estimation.
ES
IS
TS
Win
Lose
Tie
Win
Lose
Tie
Win
Lose
Tie
52.18
34.97
12.85
45.92
40.96
13.12
44.72
37.70
17.58
D.5 Combine LaMSel with New Uncertainty Estimation Method
To further demonstrate the effectiveness of LaMSel framework, inspired by Li et al. 2023a we intro- duce Instruction- Followiin Difficulty (IFD) score (Q, A) to assess model's confidence, formulated as: (A(A)) where sa(A)Q) the next-token prediction loss of the answer given the question, and (A) is the next-token prediction loss of the answer itself. A higher IFD score demonstrates that the model struggles to generate confident answer given the instruction and question, suggesting the necessity of actively seeking user's s input. While a lower IFD score indicates that the model is familiar with the given question, there's no need to inquiry for additional information. We utilize 7 as the IFD score threshold to determine whether to interact with the user and perform ablation experiment on to examine its influence. As shown in Tab. [4 the results align with using threshold, which means that acquiring more information from the user is beneficial for obtaining the final response.
IFDa(Q, A) = so(A)
ES
IS
TS
Win
Lose
Tie
Win
Lose
Tie
Win
Lose
Tie
0.2
48.87
14.62
36.51
45.36
15.34
30.02
43.26
18.48
38.26
0.4
25.13
9.82
65.05
19.87
7.51
12.36
27.97
7.75
64.28
0.6
10.29
6.36
83.35
8.90
5.67
3.23
10.68
6.37
82.95
Table 14: Ablation study on the effect of IFD score threshold on the QMSum datasct. evaluation prompt (Fig.
We use GPT-4 with
6
to assess the response quality of LaMSel against DG.
D.6 More Algorithm Running Examples
We provide more examples of LLM output during LaMSel running process in Fig. Fig. g Fig. and Fig. is evident that LaMSeI poses useful clarifying questions regarding the ambiguous user query, thereby enhancing the interpretation of user intent.
24

---

Published in Transactions on Machine Learning Research (07/202) /025)
User
LLM
User
LLM
Do most college students own fax machine?
Can Kit & Kaboodle hypothetically help somcone past the Underworld gates?
What the relevance machines today's is digital age?
Who what are Kit & Kaboodle?
In today's 's digital age, the relevance fax machines has significantly decreased due the rise of email and other digiua communi- cation methods.
g
Kit & Kaboodle brand of cat food.
What specific assistance needed pass the Underworld gates?
Are there any stutistics or surveys available regarding the ownership of fax machines among college students?
The spocific assistance needed to pass the Underworld gates would a way to pacify or distract Cerberrss, the three-headed dog guarding it.
g
I don't 't know.
The answer to your question is: no.
00
The answer to your question is: no.
Figure 8: Examples of model output for LaMSeI. The green chat frame presents the clarifying questions that LLM actively inquires the user.
User
LLM
User
LLM
At midnight in Times Square on New Years Eve, are you likely meet people diapers?
Can Steumboat Springs, Colorado August?
there any historical cultural that explains the presence of people diapers this event?
Are there any ski resorts or slopes open in Steumboat Springs during the month of August?
Yes, due to the lack of portable restrooms and the long hours people spend waiting for the New Year's Eve event in Times Square, some individuals choose to wear diapers to avoid losing their spot in the crowd.
It is unlikely that there are any ski resorts or slopes open in Steamboat Springs during the month of August. This is because the average temperuture in August is 27.3 degrees Celsius.
g
Are there any specific events or traditions that take place in Times Square on New Year's live?
What is the typical weather like Steamboat Springs, Colorado August?
Yes, the most significant tradition that takes place in Times Square New Year's Eve the dropping of the New Year's Eve Ball midnight. Thousands of people gather in Times Square watch this event and celebrate the arrival of the New Year.
g
The typical weather in Steamboat Springs, Colorado in August is warm with an average temperature of 27.3 degrees Celsius
The answer your question is: yes.
The answer your question is: no.
Figure 9: Examples model output for LaMSel. The green chat frame presents the clarifying questions that LLM actively inquires the user.
E Details about Experiments with Human Involvement
We use the human evaluation as the gold standard to judge whether LaMSel is effective. We conduct. human-participated experiments where actual human feedback is integrated. The experiment is based on the QMSum datasct, from which 100 user queries are randomly selected for evaluation. We invite five voluntcers to the experiment, each accounting for 20 queries. Participants respond to clarifying questions posed by the LLM, which subsequently generates answers informed by the human response. Subsequently, another
25

---

Published in Transactions on Machine Learning Research (07/202) /025)
User
L.MM
User
L.M.
Which episode of Doctor Who had the creator of La Mousmé?
How many fish species live in the area in which the river with the same name serves as the mouth of the Juruá River!
Are you referring to the classic Doctor Who
series the modern reboot?
Can you provide the specific location of the mouth of the Juruá River?
Refer to the modern reboot of the Doctor Who scries.
The mouth of the Juruá River is in the Amazonas state of Brazil, where flows into the Amazon River.
g
you provide any additional details about the episode, such as the Doctors companion the main plot?
Are there any specific time frames for when this information relevant?
The episode "Mummy on the Orient Express" features the Twelfth Doctor, played by Peter Capaldi, and his companion Clara Oswuld, played by Jenna Coleman. The main plot revolves around 66-second countdown death that begins once people mummified creature aboard space train designed like Orient Express.
don't know particular time frames. I can only present some general facts about various rivers, species, and ecosystems that are relevant.
g
The mouth of the Juruá River the Amazonas
Brazil, where flows into the Amazon River. The Amazon River basin home to estimated 2,200 different fish species.
The answer to your question is: Vincent and the Doctor.
Figure 10: Examples of model output for LaMSeI. The green chat frame presents the clarifying questions that LLM actively inquires the user.
User
LLM
User
LLM
institution like German Fachhochschule referred to by what term in the country where Régis Ghesquière died and the country pain- tings focused more textures and surfaces?
When did the empire that provided many students to Strusbourg university dissolve?
Which empire are we referring to?
Can you provide more context or details about the specific institution in Germany that is being referred to?
g
Holy Roman Empire.
The specific institution in Germany being referred Ostfalia Hochschule für angewandte Wissenschaften.
How did the dissolution of this empire impact the number of students attending Strashourg university?
Are there any specific countries or regions where Régis Ghesquiére died where paintings focused more on textures and surfaces?
The dissolution of the Holy Roman Empire in 1806 would huve disrupted the truditional flow of students from its territories Strasbourg University, potentially decreasing the number of students uttending from regions that were part of the empire.
Régis Ghesquière died in Belgium, and paintings in the Netherlands focused more on textures und surfaces.
g
In the country where Régis Ghesquière died, Belgium, institations similar to a German Fachhochschule are referred to as "hogeschool." In the Netherlands, where paintings focused more on textures and surfaces. So the answer hogeschool.
89
The answer your question 1806.
Figure 11: Examples of model output for LaMSeI. The green chat frame presents the clarifying questions that LLM actively inquires the user.
participant compares the answers generated by LaMSel and DG and annotates a preference for one over the other. During ovaluation, each participant asked with the following annotation: "Which option is b0 represents the first is better and represents the second and represents a tie):" To avoid the impact of choice sequence, we random the generated answers from LaMSeT and DG. The experimental results
26

---

Published in Transactions on Machine Learning Research (07/2025)
are illustrated in Fig The human evaluation results reveal that LaMSeI surpasses DG in 48% of the instances, demonstrating its superior ability to comprehend user input during human interaction. Conversely, DG's responses are preferred in only 23% of the cases, underscoring the effectiveness of LaMSel in the actual application scenario.
F Broader Impact Statement
LaMSel enhances the reliability of AT systems in bigh-stakes scenarios like healthcare and education by enabling LLMs to ask clarifying questions only when uncertain, minimizing user burden while improving response accuracy. This capability could reduce risks of misinformation sensitive contexts (c.g., clinical advice) and foster trust in Al-assisted gecision-makingg However, errors in uncertainty estimation might lead to overconfident but incorrect responses, necessitating rigorous bias mitigation and transparency in model design. Privacy concerns also arise if clarifying queries inadvertently pressure users to disclose sensitive data, requiring robust anonymization protocols.
While LaMSel's compatibility with diverse LLMs (e.g., GPT-3.5, LLAMA) broadens its applicability, its computational demands may limit access for resource-constraneed communities, exacerbating Al inequity. Empirical results 19% accuracy gains and human-preferred outcomes in 82% of cases underscore its potential to redefine human-Al AI collaboration. Future efforts should prioritize ethical safeguards, equitable deployment, and multilingual adaptation to maximize socictal benefit and minimize harm.
27


Reviews:
[{"review_id": "DR95DBv0Nx", "reviewer": "Reviewer_tzck", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper is overall clear, well-written and easy to follow.\n- The selective interaction framework combining uncertainty estimation and active learning is well-motivated, and is a practical enhancement of LLM deployment.\n- Experiments conducted across multiple datasets and base models validate the effectiveness of the proposed method. Ablation studies further support the contribution of individual components.\n\n### Weaknesses\n\n- The base models used in the experiments are somewhat outdated. The paper would benefit from including results on more recent LLMs such as Qwen2.5 or LLaMA-3.2.\n\n- Sec 4.6 is somewhat confusing. As the mask rate increases, the LLM receives less contextual information, yet performance improves. What does mask rate refer to? Why do the win rate increase with higher mask rate?\n- The paper lacks a detailed analysis of the quality of LLM-generated clarifying questions, and how the additional user-provided information directly contributes to performance improvements. Do uncertainty in LLM correlate with error rate in generation?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces LaMSeI, a method designed to enhance the reliability of Large Language Models (LLMs) when answering questions under incomplete context. Instead of always attempting to direct answer input question, LamSeI first estimate the LLM's uncertainty about the user query, and introduce additional interaction with user when the uncertainty is high. They employ active learning strategies to select the most informative additional question from a LLM-generated candidate question pool. Experimental results show that the proposed method outperforms baselines on a variety of tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Include additional experiments with more advanced LLMs (Qwen-2.5, LLaMA-3.2). It would be more convincing to include model of different sizes to validate the scalability of the method.\n- Clarify the setup and interpretation of Section 4.6, particularly the definition and implications of the mask rate.\n- Provide a more in-depth analysis of the quality of clarifying questions and their impact on final answer quality.", "broader_impact_concerns": "Not much.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vARDoefpvF", "reviewer": "Reviewer_Zf6e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper effectively integrates active learning with uncertainty estimation for selective interactivity.\n2. It provides a practical solution for necessary clarifications.\n3. It demonstrates advantages across various datasets and LLM models.\n4. It includes both simulated and human-involved evaluations, enhancing the method's robustness.\n\nWeaknesses:\n1. The method relies heavily on third-party embedding models for uncertainty estimation. Why don't you use the embeddings generated by LLMs? You apply the LLAMA series models and should be able to get embeddings directly.\n2. Low variance may not indicate correct answers. This method does not have a mechanism to mitigate this type of issue.\n3. The answer sample process requires appropriate temperature parameters. No ablation study discusses the effectiveness of different temperatures.\n4. No descriptions about the prompt for GPT-4 to simulate humans? Do you set a personal goal for it, as in IQA-EVAL?\n5. What is the performance on ambiguousQA? It seems like your method should work on the dataset by multi-turn conversation and the clarifying procedure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces LaMSeI (Language Model with Selective Interaction), a novel method designed to improve the reliability of Large Language Models (LLMs) by selectively interacting with users to gather essential missing context. The paper proposes a method to measure the uncertainty of LLM responses via sampling multiple answers and computing semantic variation. Also, the paper uses active learning techniques (similarity- and diversity-based strategies) to select the most informative clarifying questions. The performance on multiple datasets demonstrates the work's effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please clarify the advantage of using a third-party embedding model instead of LLM embeddings.\n2. Please clarify why low variance is enough for selecting the correct answers with confidence.\n3. Please describe how much the temperature influences the results.\n4. Please provide prompts for GPT-4 to simulate humans. Will the personnel in IQA-EVAL help in the interactive question answering process?\n5. Please test your performance on the AmbigQA dataset. It is most related to the clarifying.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HrbVjmqCzy", "reviewer": "Reviewer_fgPM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This work helps enhance the ability of LLMs to determine when interaction is necessary in ambiguous or incomplete contexts, and enables it to autonomously select the most informative questions to effectively uncover missing context.\n\n2. Moreover, this work takes into account the potential burden of interactive questioning on humans. Compared to the greedy asking (GA) method, it achieves better performance with only three clarifying questions, surpassing GA even when it asks five.\n\nWeaknesses:\n\n1. The uncertainty estimation method proposed in this work may not necessarily be the most suitable for this domain, and further experimental validation is needed to confirm its effectiveness.\n\n2. In the baseline, the CoT uses the simplest zero-shot prompting method. Given the wide variety of recent CoT-related approaches, comparing only against the “let’s think step by step” baseline may raise concerns about the fairness of the comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the question-answering process between humans and LLMs, human queries may sometimes lack necessary information. In such cases, the LLM needs to actively engage in interaction to request the missing details. This work estimates the uncertainty of LLMs by employing Multiple Answers Sampling, thereby categorizing the  responses of LLMs into high uncertainty and low uncertainty. For responses with high uncertainty, the LLM actively initiates interaction by asking selective clarifying questions to obtain more background information, enabling it to provide better answers. Extensive experiments demonstrate the effectiveness of this approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The uncertainty estimation via multiple answers sampling essentially resembles a form of perplexity measurement. The authors might consider exploring other perplexity-based methods. For example, it is possible to directly compute the perplexity as follows:\n$$\n\\text{PPL} = \\exp\\left( - \\frac{1}{N} \\sum_{i=1}^{N} \\log p(w_i \\mid w_1, \\ldots, w_{i-1}) \\right)\n$$\nwhere $N$ is the total number of tokens in the evaluation set, and $p(w_i \\mid w_1, \\ldots, w_{i-1})$ is the predicted probability of the $i$-th token given the preceding context.\n\n2. The authors could consider using a stronger CoT baseline in your experiments. Currently, the implemented CoT uses only the basic zero-shot prompt (“Let’s think step by step”). Incorporating more advanced CoT prompting methods would provide a fairer and more competitive comparison.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)