Adaptive Elicitation of Latent Information Using Natural Language
Authors: Jimmy Wang
Thomas P Zollo
Richard Zemel
Hongseok Namkoong
Venue: ICML 2025 poster (2025)
Keywords: Uncertainty Quantification
Natural Language
Adaptive Survey Design
Primary area: applications->language_speech_and_dialog
TL;DR: We introduce a novel meta-learning and planning algorithm that can use LLMs to quantify uncertainty over natural language inputs, and ask questions to elicit information that reduces this uncertainty.

Abstract: Eliciting information to reduce uncertainty about a latent entity is a critical task in many application domains, e.g., assessing individual student learning outcomes, diagnosing underlying diseases, or learning user preferences. Though natural language is a powerful medium for this purpose, large language models (LLMs) and existing fine-tuning algorithms lack mechanisms for strategically gathering information to refine their own understanding of the latent entity. To harness the generalization power and world knowledge of LLMs in developing effective information-gathering strategies, we propose an adaptive elicitation framework that actively reduces uncertainty on the latent entity. Since probabilistic modeling of an abstract latent entity is difficult, our framework adopts a predictive view of uncertainty, using a meta-learned language model to simulate future observations and enable scalable uncertainty quantification over complex natural language. Through autoregressive forward simulation, our model quantifies how new questions reduce epistemic uncertainty, enabling the development of sophisticated information-gathering strategies to choose the most informative next queries. In experiments on the 20 questions game, dynamic opinion polling, and adaptive student assessment, our method consistently outperforms baselines in identifying critical unknowns and improving downstream predictions, illustrating the promise of strategic information gathering in natural language settings.

Adaptive Elicitation of Latent Information Using Natural Language
Jimmy Wang" Thomas Zollo
Richard Zemel
Hongseok Namkoong'
Abstract
reduce uncertainty about new environment or problem instance. For example, before an optimal lesson plan can be prepared for particular student, information must first be gathered about their underlying skills and abilities. Simi- larly, a patient's 's health status must be quickly assessed upon intake, while an online service secking retention aims to gain a fast understanding of a new customer's 's preferences.
Eliciting information to reduce uncertainty about latent entity is a critical task in many appli- cation domains, e.g., assessing individual stu- dent learning outcomes, diagnosing underlying diseases, or learning user preferences. Though natural language is a powerful medium for this purpose, large language models (LLMs) and ex- isting fine-tunine tinettuning algorithms lack mechanisms for strategically gathering information to refine their own understanding of the latent entity. To har- ness the generalization power and world knowl- edge of LLMs in developing effective informa- tion gathering strategies, we propose an adaptive elicitation framework that actively reduces un- certainty on the latent entity. Since probabilistic modeling of abstract latent entity is difficult, our framework adopts predictive view of un- certainty, using a meta-learned language model to simulate future observations and enable scal- able uncertainty quantification over complex nat- ural language. Through autoregressive forward simulation, our model quantifies how new ques- tions reduce epistemic uncertainty, enabling the development of sophisticated information gather- ing strategies to choose the most informative next queries. In experiments on the twenty questions game, dynamic opinion polling, and adaptive stu- dent assessment, our method consistently outper- forms baselines in identifying critical unknowns and improving downstream predictions, illustrat- ing the promise of strategic information gathering in natural language settings.
Notably, in these (and many other) cases, the object of in- terest is latent, meaning it cannot be directly measured or observed but can only be queried indirectly. This makes gathering information particularly challenging, as it requires carefully designed strategies to infer the latent entity's 's char- acteristics through indirect signals. To achieve efficiency, these strategies must be adaptive, dynamically tailoring sub- sequent queries based on the information gained so far. In the context of student assessment, adaptive approach might start with broad math questions covering multiple skills. If the student gets question wrong, the system would then drill down into each relevant skill individually, asking questions of varying difficulty to determine the limits of their proficiency. By progressively refining its queries in this way, the system efficiently maps out the student's knowledge boundaries and thus reduces uncertainty about their individual skill profile (see Figure 6).
As natural language is a particularly powerful and flexible medium for eliciting such latent information, one might assume that modern large language models (LLMs) (Brown et al., 2020; Bai et al., 2022; DeepSeek-AL al., 2025) could be helpful in such dynamic information gathering ef- forts. However, LLMs and existing fine-tuning algorithms often treat uncertainty passively, and lack mechanisms for strategically gathering information to refine their own un- derstanding of the latent entity. While existing L.LMs are often trained to instill as much static world knowledge as possible (Hendrycks al., 2021), this world knowledge can- not directly be used reduce uncertainty about new, unseen individuals that the model has little information about.
1. Introduction
The performance of many valuable services and systems depends on the ability to efficiently elicit information and
To harness the generalization power and world knowledge of LLMs to address the renewed uncertainty that arises when- ever a new environment or individual is encountered, we introduce an adaptive elicitation ieceitationframmeewe framework that uses natural language to actively reduce uncertainty by simulating future responses. Crucially, our approach leverages meta-learning,
'Columbia University. Correspondence Jimmy Thomas Zollo
Wang <jw4209ecluumba..ddu. <1px2105@collmmii.cdu.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Adaptive Elicitation of Latent Information Using Natural Language
whereby the model is trained diverse historical question- answer trajectories spanning many latent entities. This meta- trained foundation enables the model to handle new, unseen entities- such as brand-new student- wwhose responses are initially unknown and thus create fresh epistemic uncer- tainty. By aligning a language model's perplexity objective with the goal of predicting all possible (yet unobserved) answers to the questions we might ask, we transform the challenge of directly modeling latent entity into a sim- pler and more scalable problem of predicting masked future observations (Ye et al., 2024; Fong et al., 2023). As the model observes each new answer from the individual, it systematically sharpens its beliefss, distinguishing between uncertainty it can reduce with further data (epistemic) and the inherent noisiness or variability that remains (aleatoric). Our framework enables wide range of exciting and im- pactful applications, e.g., constructing a dynamic diagnostic questionnaire that maximizes the information gained about a patient's 's health or generating a personalized set of test questions that yield the most insight into a student's 's learning needs (see Figure 1).
(I) Meta-learns a predictive language model from histor- ical question-answer data. (2) Uses this model quan- tify uncertainty about future or unobserved answers from new, unseen latent entities, using autoregressive forward simulation to efficiently distinguish between epistemic and aleatoric uncertainty. (3) Accurately quantifies and sharp- ens beliefs given new information, and dynamically adapts question selection to elicit information that optimally re- duces uncertainty. A key advantage of our method is that we can apply our meta-learning method directly on existing pre-trained LLMs,, augmenting uncertainty quantification with internet-scale world knowledge.
2.1. Problem Formulation
We consider an unobservable latent entity U E u (e.g.. a student's skill profile or patient's 's health status). We query U by posing a question X E X (in natural language) and observing an answer E y drawn from
Answer Y ~ Q--1 Question X, Latent U),
(1)
where Q is the ground truth distribution. Our two primary goals are to: (1) Quantify our uncertainty about U based on observed question question-answer pairs. (2) Reduce that uncertainty by adaptively choosing which questions X to ask next.
Contribution In the remainder of this paper, we intro- duce our novel framework for latent uncertainty reduction using natural language and demonstrate its effectiveness across several key applications. Our work contributes a key conceptual and algorithmic insight to the accelerating field of I.LMs: by obviating the need for directly modeling distribution over the latent entity and instead employing a predictive view of uncertainty, we enable the development of adaptive information gathering strategies that naturally scale with L.LM performance, improving as models become more capable. Our adaptive elicitation framework can be applied directly on top of existing LLMs, enabling the use of internet-scale linguistic knowledge to comprehend uncer- tainty. Through experiments on tasks such as dynamic opin- ion polling and adaptive student assessments, we illustrate the versatility and significant potential of our framework to enable more efficient and targeted information elicitation in critical domains and applications. Overall, we aim to lay the foundation for future research into rigorous uncertainty quantification and adaptive eeciiion-mmainn using LLMs, highlighting the promise of active, context-awan aware strategies in solving real-world problems.
Pre- Fre-Trainig We assume that we have an abundance of historical data, where a model can learn from past trajecto- ries to inform adaptive elicitation about new, unseen individ- uals from which we wish to gather data. We have collec- tion of entities U E Utaiin For each latent entity U we have access to sequence of questions and answers ( (Xi:7 v(!)) produced by it. Then our historical pre-training data con- sists of: Dirain == (t) an online tutoring service may have an abundance of data about previous students that they may utilize.
I:T 1:T U E Umiin J. For example,
Test-Time Adaptive Selection After pre-training, we wish to quantify and reduce uncertainty about new, unseen, latent entities Unew (e.g. a new student or patient). For each Unew have T rounds where tions and receive responses. At each ( 1= 0, 1,..T - 1, the model can adaptively choose question X1+1 E X based on previous feedback H4 == (X Une U and receives an answer Y441 E V1+1-
sequentially ask ques-
1:t
2. Adaptive Elicitation Framework
We will evaluate the model on its ability to predict unob- served answers Y1+ generated by the latent entity to any future questions XT+1:00- For example, we may conduct survey where we can only include 5 questions, but we wish to know the answer to 1000 additional questions. Being able to predict + requires the ability to adaptively collect relevant information Y1.r, which in turn requires the ability to both quantify and reduce remaining uncertainty about
In this section, we present an approach to uncertainty quan- tification and adaptive question selection in natural language settings where the latent entity cannot be directly modeled. We adopt a predictive view of uncertainty: rather than speci- fying a direct prior or complete model of the latent entity, we focus on how well the model can predict and quantify un- certainty over future observations of that entity. Our method:
2

---

Adaptive Elicitation of Latent Information Using Natural Language
the entity New We detail approach in the following sections.
Example: Adaptive Student Assessment
Our Algorithm
2.2. Quantifying Uncertainty Using Predictive Model
Select Next Test Question
(Question)
Traditional approaches model uncertainty may try to model U directly (e.g. by assigning probability distri- bution over structured latent space) (Blei et al., 2003; Salakhutdinov & Mnih, 2007). However, specifying such models for complex human-generated responses can be both restrictive and infeasible. For example, it is unclear how to define an explicit parametric model to represent an individ- ual's 's political opinions. Instead, we adopt predictive view of uncertainty that avoids the need to directly model latent variables. This approach allows to train autoregressive models directly in the space of natural language, enabling flexible and scalable modeling of its full complexity.
Quantity Uncertainty in Latent Skills
Y (AAswer))
Our algorithm can adaptively elicit information from latent entity via natural language interaction. For example, in assessing new student, the system may ask questions areas where the student's abilities are not yet known, to maximize the information gained from each question and efficiently reduce un- certainty about the student's 's individual skill profile.
Figure
Define the epistemic uncertainty to be uncertainty that can be reduced with more information, and the aleatoric uncer- tainty to be uncertainty due random noise or variation that cannot be reduced by observing more data. Our key observation is that if we were observe infinite data Y1:o produced by the entity, all epistemic uncertainty about the entity would disappear. Intuitively, if a teacher could ob- serve a student's answers to a very large set of questions, that teacher could probably completely predict a student's ss future answers with errors only due to aleatoric variation. Examples of this aleatoric uncertainty could include random noise intrinsic uncertainty in the student's 's own decision process. This idea is in line with classical views that treat latent variables as unobserved data (Rubin, 1976; Lindley, 1965; Hill, 1968; Dawid, 1984), as well as more modern treatments (Fong et al., 2023; Ye et al., 2024; Zhang et al., 2024).
Once we have this estimate, we can adaptively choose ques- tions that reduce the greatest amount uncertainty about the missing data Y1+1:00 (see Section 2.4 for more details). Notice that this approach to uncertainty quantification works directly in the space of observables X1:00, Y1: and does not require any explicit modeling of the latent U. If we have predictive distribution over future answers Y441:00 given previous observations X1 Y1.t for every t,
P(Ynoo Xist Yi.) = P(Future Answers Current Info),
then we can exactly calculate the entropy term in Equa- tion (2). Thus, we directly train an autoregressive predictive model pa (Y X1.,, Yıs) by optimizing the parame- ters f E θ in order to model these quantities.
Under this view, epistemic uncertainty is naturally the certainty due missing data: specifically, the uncertainty about unobserved future responses Yt+ given the current information Yia- Thus, our objective is to provide accu- rate uncertainty estimates over missing data Y In order to quantify uncertainty about 1 we first build off the notion of entropy. Given distribution P with den- sity p(-), the entropy and conditional entropy over an swer Y E y are defined as HY) EyEy p(y) log p(y), H HY 1-) E p(y I- jov J..
2.3. Meta-Learning an Autoregressive Predictive Model
To obtain the most accurate estimates of uncertainty, the ideal strategy would be to use the ground-truth answer distri- bution Q in Equation (1) as the predictive distribution over future observations. In particular, we aim to approximate the conditional distribution (Y,++:000) X1:t> Yı.)) induced by Q, which represents the true distribution over future answers given past interactions. Under this setup, the corresponding conditional entropy HQ X1:t. Yis)) would yield exact measures of uncertainty. Since Q is unknown in prac- tice, our goal becomes to approximate the conditional Q as closely as possible by training a model po in order to produce reliable uncertainty estimates.
Next, let
P(Y+1 = y X1i,, Xx41 = x) = pp(y) X1., Yia), which
also induces the conditional entropy Hp({++1 X1a, Yi.). Using this notation, we are interested in measuring the uncertainty over missing data
To approximate Q. we assume access to historical data from collection of latent entities Merain Each entity U Utaain is associated with sequence of question-answer (U)
Uncertainty (Future Answers Current Info) (Y X1a, Y1.).
pairs {(x(U) 1. v }}. where Y Q( X,U) and U is
3

---

Adaptive Elicitation of Latent Information Using Natural Language
sampled from a prior distribution. Our first step is to meta- train autoregressive language model po this historical data consisting of sequences of questions and answers from various latent entities
Training. For training, we process each sequence of ques- y-(d)) corresponding to a latent entity U by sequentially arranging them into one long nat- ural language string (x("), , x )..). We as- sume that the probability of a response to a question is inde- pendent of the ordering of the earlier questions and answers, and we randomly permute the order of the question/answer pairs within each entity's sequence during training. Then we optimize language model to predict each answer Y, conditioned on the current question Xr and previous ob- servations 14 To do so, we apply gradient mask that masks out tokens which do not correspond to any Y,. We use stochastic gradient descent procedures to optimize the training loss.
tions and answers {XIN 1:N
Duain == (X
IT y!U) : U E )
In the student assessment example, Dirain may be a historical dataset of past students, each with an associated sets of test questions and answers. For simplicity, we assume that each sequence is of length T, but our framework is agnostic to differing sequence lengths.
Objective Define a sequence of previous observations H4: := {X1a, Yi.ch. We train our autoregressive language model po. parameterized by O, to output one-step proba- bilities over future answers conditioned on previous observa- tions (i.e., question/answer pairs) po (Y++1l H4, X2+1 x),, inducing joint distribution over future outcomes
2.4. Adaptive Question Selection by Future Simulation
Having trained a predictive model Po from historical data, we can use this model to quantify uncertainty about future observations generated by the latent entity, and take actions to reduce said uncertainty. Given a new latent entity Unew- may also be interested in different targets of uncertainty: for example, the full sequence of future answers (Z Y1:50), specific subset of questions, or the answer to a particular question (Z = =Y. To make our notation more general, we notate Z as the object we wish to understand. Our goal to reduce uncertainty about by sequentially choosing questions X that are most informative i..., those that will optimally reduce the model's 's uncertainty about Z (See Figure 6). For example tutor may be interested in understanding which questions will reveal the most about the student's 's understanding.
Po OY 1:0 Hr, Xe+1 = I1+1, Xe+2 = I2+2,,..)
Do
II (Y xs).
The training objective for our model is then to optimize the joint log iikelihood/marginal likelihood of the observed sequence within the historical dataset
{ Uuraiill
maxago
log po H4-1, X- ar))
UEu 1=1
(4)
Setup As described in Section 2.1, we operate in an adap- tive setting at test time. At each round t 0, 1, 1,... T 1. we may select a question X Xe+1 based on the inter- action history far, H4 == (XUnnew y!!!" ), and receive an answer Y++1 E D/+1-
After optimizing our model po. we can now use it to approx- imate the uncertainty estimates HY | X1st, Yi:r) A Hp (Y X1.t, Y1.). To show that optimizing this objective is optimal for approximating Q, we first note that maximizing the objective in Equation (4) is equiva- lent to optimizing empirical version of the cross entropy Eo llg po
To quantify informativeness, we define the information gain from a question- answerr pair (Xt-1. Y(41) as:
Expanding this loss, we can see that
0 maxEo llog pe (Yi.r (1r)]
IG, (X++1, Y(+1)) ( H2/H U (X++1, Yo+1))
maxx
( - DxI. (Q(lp))
(5)
g
Eg flo (Yii. (
This measure quantifies the reduction in entropy about Z after observing a new interaction, by quantifying the difference between the current uncertainty Hpo (Z H4) and the uncertainty after observing (X++1, Y++1), H(Z YY+1)). Since we do not yet know Y.+1 when
where DKL (Q[Q)) 0. An equivalent interpretation of max- imizing the joint log likelihood is that we are minimizing the KI. divergence between Po and Q, leading to accurate downstream uncertainty estimates. By optimizing this ob- jective over historical data, our aim is for the model to learn meta-learn structures and patterns that will be useful for adaptive testing over new, unseen entities.
H; U choosing 41+11 we can instead quantify the expected reduc- tion in uncertainty by simulating potential answers using our meta-learned model po. This idea leads to the Expected
4

---

Adaptive Elicitation of Latent Information Using Natural Language
Information Gain (EIG) (Chaloner & Verdinelli, 1995):
in Section B, where Proposition B.3 bounds the performance gap between full combinatorial planning approach and the greedy selection procedure.
EIG, (Z;
(6)
Hyg (2) H - p (Z| Hi U (ze+1, Y++))),
Lookahead / Monte Carlo Planning. To account for multi-step effects (e.g., a question that might not imme- diately reduce much uncertainty but paves the way for more informative follow-ups) and to better approximate the com- binatorial quantity in Equation (8), we can apply standard Monte Carlo Tree Search (MCTS) techniques from rein- forcement learning (Browne al., 2012; Silver al., 2016). We use simple instantiation with strong empirical perfor- mance and leave more complex variants to future explo- ration.
where we use our meta-learned model p.(.) to simulate Z and Y141 ~ ps (-H, X1+1 = 21+1) in the expectation. To
calculate the EIG for multiple choices of z, we have
(Z; (Ie-1, 2142, ... [(+K))=
t-
(7)
Ho (2 Ho) E(Hp, pa (Z) Hi U (1,, Y.)I. i=t+1
where we autoregressively simulate Y++1++ from our meta-learned model. This quantity naturally quantifies the amount of epistemic uncertainty we expect to reduce by choosing a set of questions. If the EIG is very small, then this implies that the reduction in entropy is small and there- fore this information is not informative. This could be because there is lot of aleatoric uncertainty, such that the information gathered is very noisy, or due to the fact that the information gathered is not relevant to the object of interest.
With an MCTS policy π MCTS we sample entire simulated question-answer sequences using the meta-learned model Pe up to depth d to estimate the cumulative information gain. In order to simulate future responses, we use T Tgreddy select questions and the Information Gain (5) as proxy rewards. Starting at time i, we first calculate F.Gr: (Z; X2+1:00) for each æ E Nr, and choose the top K questions. For each of the K questions, we then simulate N futures up to depth d. For each sample path i [N], we receive reward (x) IG, (Z: (X y-( x(f (411+a)). where questions are sequentially selected using 7 groody and answers are simulated using the meta-learned model pg. Finally, the MCTS policy chooses an action as
Question Selection Policies To select the optimal ques- tion at each time 4, we would ideally like to choose a ques- tion selecting policy π. : H + X, where Xe+1 xx-( such that we maximize
argmnax Ex~r()) 1x)))
(8)
N
MCTS
1
IG, (z: (X:
where EIG, Z; x(+1:)) is defined in Equation (7). To ap- proximate this quantity, we use our meta-leared model Pe to autoregressively simulate possible future answers Y=+1. ~ Pa. and the chosen policy क to simulate ques- tion choices X1+1.T ~ x(-). Through autoregressive future simulation, we can optimize for our question selection pol- icy æ. While it is possible to calculate the discrete optimal Ier that maximizes this objective, it can be intractable as simulating XI+1.T, Y++1:T is combinatorial in the number of steps. Instead, we introduce two procedures that show strong practical performance while having feasible compu- tational cost.
π
argmax xEX,
N
Though more expensive computationally, we find that π MCTS can find better long-horizon query strategies.
3. Experiments
To rigorously benchmark adaptive information gathering strategies for LLMs, we require datasets that (i) capture diverse latent entities or hidden factors, (ii) provide many possible queries about these entities, and (iii) for each entity, link some queries to corresponding ground-truth answers. Such an experimental setup allows us to assess the abil- ity of an LLM and/or particular algorithm to strategically select questions in order to reduce uncertainty about the latent entity. Ideally, each dataset reflects the real-world complexities of human-generated responses, while still pro- viding enough structure for robust evaluation of different query selection policies. In practice, a large pool of possi- ble questions with many ground truth answers is essential, since it allows an adaptive strategy to actively and deeply explore the latent entity along many dimensions, while still leaving unobserved data for evaluation. Then, each latent entity (e.g., a survey respondent's political stance, a stu-
Greedy Selection. A simple question selection policy groody is first enumerate the candidate questions E {z1, (xi,,.,, Then for each zj. calculate the one-step ex- pected information gain EIGr,+1 2,)) Finally, choose the I that maximizes this quantity. Concretely,
greedy == argmax El:r)-(Z;zz). n
Although greedy, this policy often performs well in practice and is computationally simpler than globally optimal plan- ning. We establish the theoretical validity of this procedure
5

---

Adaptive Elicitation of Latent Information Using Natural Language
dent's hidden skill profile, or the identity of secret object) can be progressively unveiled by observing how it answers newly selected queries. Such design criteria enable con- trolled, quantitative evaluations of LLMs under interactive, information gathering scenarios.
given specific questions and objects. Our dataset is publicly available, ! including the complete set of objects, curated questions, generated answers, and relevant metadata.
3.2. Other Datasets
Our experiments focus on three applications: the "Twenty Questions" game (using our novel and publicly available dataset, described below), opinion polling, and student as- sessment. In each scenario, the objective is to adaptively select questions that reveal as much information possi- ble with respect to separate (though potentially overlap- ping) set of target questions. Questions are chosen one at a time, and each new question-answen pair is appended to the LLM's context before proceeding. For every experiment, we start with a dataset containing& collection of latent entities U, each associated with set of questions X and answers y To train our meta-learned model ps, we split each dataset by groups of latent entities into training, validation, and test sets. We first meta-learn po on question-answer pairs corresponding to the training entities, after which we evaluate how effectively the model quantifies and reduces epistemic uncertainty about observations generated by test entities. Further details regarding the datasets, training pro- cedure, baseline comparisons, and evaluation metrics are provided below.
OpinionQA (Santurkar et al., 2023) Originally created to evaluate the alignment of LLM opinions to those of 60 US demographic groups, this dataset contains 1498 multiple choice political questions answered by a diverse collection of survey respondents. These questions target various polit- ical issues ranging from abortion to automation. For each question X, the multiple choice answer corresponds the observable feedback Y, and the survey respondent's latent political preference corresponds to the unobservable U.
EEDI Tutoring Dataset (Wang et al., 2020) EEDI is an online educational and tutoring platform that serves mil- lions of students around the globe. This dataset includes a collection of 938 math questions focusing on various areas such as algebra, number theory, and geometry, as well as individual responses from many students. Each question is multiple choice question with four answers that includes visual diagram well as associated text. The student's 's true mathematical ability U generates the student's 's answer Y to the math question X.
3.1. Twenty Questions Dataset
3.3. Meta-Train Trainiri Details
The classic "Twenty Questions" game epitomizes core goal of reducing uncertainty about hidden entity through targeted queries. Specifically, the object (such as an animal, a musical instrument, or any of a wide range of other con- cepts) serves as the latent factor U that cannot be directly observed. player (or model) must identify U by posing sequence of questions and observing the corresponding answers (c.g.. "no," "maybe," or "yes"). Hence, the game in- herently captures the essence of characterizing a latent entity by uncovering how it generates answer to different possible questions. To operationalize this game for benchmarking, we construct a novel "Twenty Questions" dataset from a curated set of objects in the THINGS database (Hebart et al., potential hidden entity. For each
We first split the training datasets by entity into train, val- idation, and test with 70%. 15%, 15% split. To meta- train our model, we initialize a pre-trained Llama-3.1-2 model in FP16 precision and use LoRA (Hu et al., 2021) to finetune our model with parameters a 24, rank= 8, and dropout= 0.1. Additional details are shown in Ap- pendix C.I
3.4. Baselines
Here we describe the baselines to which we compare our al- gorithm; each consists of an approach to model fine-tuning, and an approach to question selection. As in our method, each chosen question-answer pair is appended to the LLM context for predicting unseen answers.
2019), each serving object, we produce diverse set of candidate questions (e.g., "Does it have four legs?", "Is it edible?", "Is it used for entertainment?") together with the corresponding answers, generated by a top-quality L.M (Claude-3.5-Some -.-nnne).. In total, the dataset contains 800 objects, each with answers to a set of 1200 questions. By treating each object as a distinct latent entity, we capture a broad spectrum of scenar- ios, ranging from everyday items ("banana," " "telephone") to more uncommon concepts ("violin," "canoe"). We note that the absolute correctness of Claude's 's answers is not cru- cial, as our goal is for our model to learn the underlying aata-generating process governing which answers appear,
Base LLM First, we consider a simple baseline. For an LLM we use Llama-3.1 1-8,, from which our meta-trained model initialized, with no additional fine-tuning; question selection is performed randomly.
In-Context Tuning (ICT) Next, we consider typical in-context learning (ICL) baseline. First, we meta-train the
htttpa:/huugiingee cot.facaccc/aataattsst.... namkoong- wankoonng1ab/bine 1ab wttp//eeetyuuustiiim.
6

---

Adaptive Elicitation of Latent Information Using Natural Language
EEDI
OpinionQA
Twenty Questions
0.6
0.8
Accuracy
0.5
0.60
Base LLM
0.55
ICT
0.4
0.6
Ours
0.50
Perplexity
0.70
0.7
0.9
0.65
0.6
0.60
0.8
0
1
2
3
4 0
1
2
3
2
3
4
# Questions Selected
# Questions Selected
# Questions Selected
Figure 2. Accuracy (top) and perplexity (bottom) of our adaptive elicitation framework compared to baseline methods across three datasets: OpinionQA, EEDI student assessment, and Twenty Questions. The x-axis represents the number of questions selected. Our method works best to gather information and accurately characterize the latent each case. Each plot is the average of 10,000 simulations across unseen entities.
model via In-Context-Tunne Tuning (Chen et al., 2022), where the objective is to predict the label for query example given some number of in-context support examples. Then, questions are selected based on embedding similarity to the target questions that we aim to answer (Liu et al., 2021). We use the same model and parameters as described in Section 3.3, and we use Tibbaba-NLP/gte-laraa our embedding model.
latent entity? (2) Do we generate reasonable posterior prob- ability updates and reduce uncertainty as more information is gathered? (3) When is this adaptive procedure particularly helpful, and when is advanced planning (i.e., MCTS) most important? (4) How crucial is our training procedure for producing actionable uncertainty quantification? (5) Does the performance of our framework improve with better underlying L.M? Throughout, we connect these findings to the paper's broader motivation: the importance of adaptive strategies to eliciting information efficiently in real-world scenarios.
as
3.5. Evaluation
To evaluate how well each method can ask targeted ques- tions to reduce uncertainty about the latent entity, we per- form 10,000 trials, where on each trial we randomly select an entity and apply our algorithm (and baselines). For each trial and its corresponding entity, we randomly select a pool of N candidate questions from which the methods can se- quentially choose questions to ask, and randomly select K held-out target questions. The objective is to sequentially choose optimal questions from the candidate questions to reduce the most uncertainty about the held-out target ques- tions for the entity. In our experiments, we choose N = 20 and K = 5, but we include ablations that vary these quan- tities in Section 4.5. We evaluate the performance on the target questions with four metrics: (1) Accuracy, (2) Per- plexity (Jelinek et al., 1977), (3) Expected Calibration Error (Guo et al., 2017), and (4) Brier Score (Brier, 1950).
4.1. Overall Gains from Adaptive Elicitation
Overall results for our method and 2 baselines across all 3 datasets are shown in Figure 2. The top row of plots record accuracy on the target questions, while the bottom row record perplexity (or negative ligeliieldhood loss). The Base LLM is omitted on bottom for ease of visualization. Our framework is applied using the greedy EIG strategy. In both figures, the X-axis records the number of questions that have been selected so far.
Across all datasets and both metrics, our algorithm most effectively characterizes the latent entity by predicting the answers to target questions (we show similar results for Brier Score in Figure 7). Further, our algorithm consistently im- proves its characterization more information is gathered, whereas gathering more questions based on embedding dis- tance does not always help. Overall, our adaptive elicitation framework proves effective in gathering information and reducing uncertainty across 3 diverse domains.
4. Results and Discussion
In this section, we empirically study the following questions: (1) Can our framework be used to adaptively select ques- tions to reduce uncertainty and elicit information about the
7

---

Adaptive Elicitation of Latent Information Using Natural Language
OpinionQA
1 Questions Selected
4 Questions Selected
8 Questions Selected
0.70
Avg. Confidence Avg. Accuracy
1.0
Perfect Calibration
1.0
1.0
0.65
Accuracy
0.8
0.8
0.8
0.6
0.6
0.6
0.60
0.4
0.55
0.2
0.2
0.2
ECE, 0.0055
ECE₂ = 0.0058
ECE, =
0.50
0.0
0.0
0.0
8
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Questions Selected
Confidence
Confidence
Confidence
Figure 3. Reliability diagrams comparing confidence and accuracy after different numbers of selected questions (and observed answers). Our model maintains well-calibrated uncertainty estimates, increasing both confidence and accuracy as more questions are asked.
4.2. Uncertainty Quantification
hypothesize that adaptive strategies are especially important in characterizing features of the latent which are relatively rare in the population. As a concrete example, while many students may have overlapping weaknesses (e.g., many get the same test question wrong), it can be harder to learn that particular student is struggling in an area where other stu- dents generally do not. An adaptive strategy could help by selecting test question that most find easy but this student may answer incorrectly.
A comeessone of our approach is using predictive perplexity as indicator of uncertainty to guide the adaptive strategy; this makes sense only if our model's probabilities correctly reflect confidence about unseen data. To assess this, we examine calibration, or the extent to which the model's confidence reflects its prediction accuracy.
For each dataset, we plot reliability diagrams (Guo al., 2017) of confidence accuracy. where perfect calibration line, and record Expected Calibration Error
To investigate this hypothesis, we specify two different sub- groups of questions as targets by running an evaluation where for each target question in the subgroup, the en- tity's 's answer must have probability less than either 50% ("medium") or 30% ("hard") across the population. We use our
lies on the (ECE). Both the reliability diagram and ECE are produced by separating predictions into 10 bins by confidence, and comparing the average confidence and accuracy for each bin. Results are shown after 1, 4, and 8 questions are selected, and the far left subfigure displays overall average confidence and accuracy for each setting.
model with random, EIG, and MCTS ques- tion selection, and record results after N questions have been selected. Results are shown in Figure 4. For each question subgroup (as well as all questions from the previous exper- iment), we record on the y-axis the relative accuracy gain from using EIG or MCTS, compared to random selection.
Results for OpinionQA are shown in Figure 3, while EEDI and Twenty Questions are shown in Appendix Figures 8 and 9. For all 3 datasets, we observe that the predicted prob- abilities lie close to the diagonal of perfect calibration-our model's confidence aligns well with actual accuracy. As more questions are observed, the model's average confi- dence (and accuracy) both go up, confirming that uncer- tainty diminishes in an intuitive way. In the motivating scenario, this means that by asking just a few strategically chosen questions, the model not only improves its predictions but also becomes more certain in them. For a high-stakes application such as medical diag- nostics or skill placement exams, it is crucial know when model has enough data to be sure in its predictions, versus when it is still uncertain; these calibration results confirm our framework performs well in this sense.
First, we notice that the more advanced MCTS planning strategy outperforms EIG in all cases, and both always out- perform random. Intuitively, while greedy strategy picks the single next question that locally maximizes immediate information gain, it may miss questions whose short-term yield seems small but that pave the way for far more re- vealing follow-up queries. By looking multiple steps ahead, MCTS better accounts for how each query reshapes future options, often enabling it to find more globally optimal ques- tioning strategies. This means that given a good model for uncertainty quantification, we can improve our results by spending more compute, indicating good scaling behavior in our algorithm.
4.3. When is Adaptivity Most Helpful?
Next, we observe trends across different subgroups of ques- tions. In all 3 example applications, adaptivity and planning have massive impact on the ability to answer hard ques- tions compared to random question selection. For EEDI and Twenty Questions the percent gain over random with
Having established that our adaptive question selection method is generally effective at quantifying uncertainty and eliciting information about some latent, we next examine when such a procedure is most helpful. In particular, we
8

---

Adaptive Elicitation of Latent Information Using Natural Language
EEDI
OpinionQA
Twenty Questions
% Gain vs. Random
20
MCTS
15
20
20
5
0
all
medium
hard
all
medium Questions Type
hard
all
medium Questions Type
hard
Questions Type
Figure 4. Relative accuracy gain from adaptive question selection (EIG and MCTS) over random selection for different subsets of target questions: all, medium difficulty (answer observed in < 50% of the population), and hard (answer observed in < 30%).. Adaptivity provides the greatest benefits when identifying rare latent traits, demonstrating when strategic question selection is advantageous.
Training Method Ablation Twenty Questions
selecting questions with either random selection or the EIG strategy. To measure what is gained from planning, record the ratio of target question accuracy with planning that with random selection (a value above 1 indicates some accuracy gain from planning).
Accuracy, EIG vs. Random
1.3
all
medium
hard
Results are shown in Figure First, we see that planning performs poorly using the Base L.LM, reducing accuracy almost 15% on hard questions compared random question selection. The ICT model performance is largely unchanged by planning, across all 3 question types. On the other hand, our model's performance is greatly improved when ques- tion selection is guided by planning, highlighting that our training procedure is essential to enable such strategic infor- mation gathering with LLMs.
0.8
Base LLM
ICT
Ours
Figure Comparison performance gains from planning (EIG-
based selection) using different models.
EIG or MCTS is more than 10x higher for hard questions than for all questions; for OpinionQA, it is 5x higher. We thus have strong evidence that our adaptive information elic- itation strategy is most important when characterizing the latent features which are most atypical with respect to the population. If the latent entity exhibits atypical behavior (a student struggles with concept that most find easy, an opinion respondent holds rare viewpoint), an adaptive method can target precisely those concepts that discriminate such cases. Conversely, random or fixed questionnaires fail to unearth those nuances within a limited query budget.
4.5. Other Ablations
We first ablate the number of candidate and target questions to choose from. Our experiments were run with the models being able to select from questions in order accurately predict 5 targets. In Table 2 in Appendix E, find that method gains more accuracy the question bank becomes larger. In Table 1, we find that performance stays roughly the same as the number of target questions changes. Finally, we study the effect of the base model for our meta-training procedure. We test GPT2, Llama-3.2-1BB and Llama-3.1 1- 8B, and find Table that performance increases as the model is larger.
4.4. Training Ablation
Our results in Figure 2 confirm the effectiveness of our end-to-end adaptive elicitation framework, while Figure demonstrates the significant gains from planning-based question selection over random selection. Now, we turn understanding the remaining component-metre training by evaluating how planning performs when ap- plied to our model versus the ICT model and base L.M. We use the Twenty Questions dataset, and the same splits of all, medium, and hard questions as the previous experiment. For each setting and each of underlying models, (Base. ICT, and ours), we record the accuracy on target questions after
5. Conclusion
In this work, we propose an adaptive elicitation framework, based on the missing-data view, that actively reduces un- certainty on latent entity by simulating counterfactual rich body of literature on topics re-
responses. There lated to latent entity modeling and its applications, of which we only able to give a limited overview here. See Ap- pendix A for a thorough discussion of related research.

---

Adaptive Elicctation of Latent Information Using Natural Language
Acknowledgements
Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. arxiv.org/abs......o wwt./b/g////oooooeoo..ooeeeeeee/ 2005.000500000 14165.
We thank ONR Grant N00014-23-1-2436 for its generous support. This work supported by the funds provided by the National Science Foundation and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence).
Browne, C. B., Powley, E.,, Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., and Colton, S. A survey of monte carlo tree search methods. IEEE Transactions on Computa- tional Intelligence and Al in Games, 4(1):1-43, 2012. doi: 10111///AGG...012222220
Impact Statement
The broader impact of this work includes significant po- tential for enhancing human-centered systems and services that require adaptive decision-making based incomplete information. Our approach enables more targeted and per- sonalized interactions, which could improve educational tools, assistive technologies, and personalized healthcare assessments. However, the ethical implications of such a framework must be carefully considered. Adaptive elicita- tion systems can shape user experiences, influence decision- making, and reinforce biases present in training data. We acknowledge the importance of ongoing scrutiny ensure that such systems are deployed responsibly, particularly in high-stakes applications like healthcare and education.
Castricato, L., Lile, N., Rafailov, R., Franken, J.-P., and Finn, C. Persona: A reproducible testhed for pluralistic alignment. arXiv preprint, arX1v.2407.11738, 2024. URL https: https///xxxvvv.or.aaa/b http:///////////733/.//..
Chaloner, K. and Verdinelli, I. Bayesian Experimental De- sign: A Review. Statistical Science, 10(3):273 304, 1995. doi: 10. 10.1214/s/111770099333 //doi. org 10. 1e/1122/////////995000//
URL https:
Chapelle, O. and Li, L. An empirical evaluation of thompson sampling. In Advances in Neural Information Processing Systems 24 (NeurIPS 2011), pp. 2249-2257. NeurlPS, 2011.
References
Agrawal, S. and Goyal, N. Thompson sampling for contex- tual bandits with linear payoffs. In International confer- ence on machine learning, pp. 127-135. PMLR, 2013.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15084-15097, Curran Associates, Inc., 2021. URL
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds Z., Hernan- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, Training a helpful and harmless assistant with rein- forcement learning from human feedback, 2022. URL https: / arviv. org/ abs/ / 4204 05862.
cc/paper /itpppee_fLle/ppppe///////// neurips 0F44662220cb0002225331062666633 pdf.
Chen, Y., Zhong, R., Zha, S., Karypis, G., and He, H. Meta- learning via language model in-context tuning, 2022. URL https:://////. /ttp://aaxvvvv/.v org hts//bb//21100 07814.
Blei, D. M., Ng. A. Y., and Jordan, M. I. Latent dirichlet al- location. J. Mach. Learn. Res., (unull)99331102) March 2003. ISSN 1532-4435.
Corbett, A. T. and Anderson, J. R. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4):253-278. 1994.
Boutilier, C., Zemel, R. S., and Marlin, B. Active col- laborative filtering. In Proceedings of the Nineteenth Conference on Uncertainty in Artificial Intelligence (UAI 2003), Toronto, ON, Canada, 2003.
Dawid, A. P. Statistical theory: The prequential approach.
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
Journal of the Royal Statistical Society, Series A, 147: 278 198-292, 1984.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G.,
de Torre, J. The generalized dina model framework. Psychometrika, 76(2):179-199, 2011. doi: 10.1007/ sl -11336-001-2207-7
10

---

Adaptive Elicitation of Latent Information Using Natural Language
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J. Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang. C., Ruan,
Fong, E., Holmes, C., and Walker, S. G. Martingale poste- rior distributions. Journal of the Royal Statistical Society, Series B, 2023.
Frazier, P. L. A tutorial on bayesian optimization. arXiv preprint, arXiv:1807 0077002111 2018.
C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F. Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H. Wang, H., Ding. H., Xin. H., Gao, H., Qu, H., Li, H. Guo, J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang. I.., Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M.. Tang, M., Li, M., Wang. M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen, Q., Du, Q., Ge, R., Zhang. R., Pan, R., Wang. R., Chen, R. J., Jin, R. L.. Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang. S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X. Li, X., Su, X., Lin, X., Li, X. Q. Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang,
Frazier, P. I., Powell, W. B., and Dayanik. S. A knowledge- gradient policy for sequential information collection. SIAM Journal on Control and Optimization, 47(5):2410+ 2439, 2008. doi: 10.1 1377070063424
Ghavamzadeh, M., Mannor, S., Pineau, J., and Tamar, A. Bayesian reinforcement learning: A survey. Foundations and Trends Machine Learning, 8(5-6):359 483, 2015.
Glushkova, T., Zerva, C.,, Rei, R., and Martins, A. F. T. Uncertainty-awe aware machine translation evaluation. In Findings of the Association for Computational Linguis- tics: EMNLP 2021. Association for Computational Lin- guistics, 2021. doi: 10.18653/v1/0021 htwg://o2...iiiinn---mmi.. 330. URL http::/t////e wtt.::.oo..og............ org 18653/v1 2021 findi mewddiinss-mmmmmm 330.
X. Shan, X., Li, Y. K., Wang. Y. Q., Wei, Y. X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang. Y., Shi, Y., Xiong, Y., He, Y.. Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y. Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng,
Gonzalez, J., Osborne, M., and Lawrence, N. Glasses: Re- lieving the myopia of bayesian optimisation. In Proceed- ings of the 19th International Conference on Artificial Intelligence and Statistics, 2016.
Grover, A., Markov, T., Attia, P., Jin, N., Perkins, N., Cheong, B., Chen, M., Yang, Z., Harris, S., Chuch, W. and Ermon, S. Best arm identification in multi-armed bandits with delayed feedback. In Proceedings the 21st International Conference on Artificial Intelligence and Statistics, pp. 833-842. PMLR, 2018.
Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren, Z.,, Sha, Z., Fu, Z.,, Xu, 7.,, Xie, Z., Zhang, 7.,, Hao,
Z. Ma. Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li,
Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https: //arii////.vv.////a wwt. abs 2501 12948.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks, 2017.
Diao, S., Wang, P., Lin, Y., and Zhang, T. Active prompting with chain-of-thour of-thoughtt for large language models, 2023.
Handa, K., Gal, Y., Pavlick, E., Goodman, N., Andreas, J., Tamkin, A., and Li, B. Z. Bayesian preference elicitation with language models, 2024. URL https: //arxiv. org/ abs/ / 22003 05534.
Du, Y., Yang, M., Florence, P., Xia, F., Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, B., Kaclbling, L., Zeng, A., and Tompson, J. Video language planning. In Proceedings of the International Confer- ence on Learning Representations (ICLR). ICLR, 2024. Google Deepmind, Massachusetts Institute of Technol- ogy, UC Berkeley.
Hebart, M. N., Dickter, A. H., Kidder, A., Kwok, W. Y.,
Corriveau, A., Van Wicklin, C., and Baker, C. I. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLOS ONE, 14(10):1-24, 10 2019. doi: 10.1371//jurnal. pone.0223792. URL https: http://ooioooooo 13711..11// journal .ooee..pppo.......mm... pone. 0223792.
Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. Shifting attention to rele- vance: Towards the predictive uncertainty quantification of free-form large language models, 2024.
Hendrycks, D.,, Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Embretson, S. E. and Reise, S. P. Item Response Theory. Psychology Press, New York, NY, 2013.

---

Adaptive Elicitation of Latent Information Using Natural Language
Hill, B. M. Posterior distribution of percentiles: Bayes' theorem for sampling from population. Journal of the American Statistical Association, 63(322))677-691, 1968.
S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J.,
Kernion, J., Kravec, S., Lovitt, I.., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language models (mostly) know what they know, 2022.
Hou, B., Liu, Y., Qian, K., Andreas, J., Chang, S., and Zhang, Y. Decomposing uncertainty for large language models through input clarification ensembling, 2024.
Kirk, H. R... Whitefield, A., Röttger, P., Bean, A., Mar- gatina, K., Ciro, J., Mosquera, R., Bartolo, M., Williams, A., He, H., Vidgen, B., and Hale, S. A. The prism alignment project: What participatory, representative and individualised human feedback reveals about the sub- jective and multicultural alignment of large language models. arXiv preprint, arXiv:2404 2024. URL https: http://aaxvvvvv. wttp/io.orgg....... 1tt.:/.ab./2.411.66/0
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z.,, Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https: aarivv. org/ org/abs/ 22w.. 09685.
Hu, Z., Liu, C., Feng, X., Zhao, Y., Ng. S.-K., Luu, A. T., He, J., Koh, P. W., and Hooi, B. Uncertainty of thoughts: Uncertainty-awar planning enhances information seek- ing in large language models, 2024.
Kuhn, L., Gal, Y., and Farquhar, S. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natu- ral language generation, 2023.
Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettle- moyer, L., Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via post-hoe parameter merging. arXiv preprint, arXiv:23 11564, 2023. URL https:// arxiv. org/ aws/aas///a 11564.
Lattimore, T. and Szepesvári, C. Bandit algorithms. Cam bridge, 2019.
Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. In-context decision- making from supervised pretraining. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023. URL https: http:///ppeeiiieeeeeeeeeeeeeeeee nnt.//
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. Ad- vances in neural information processing systems, vol- ume 34, pp. 1273-1286, 2021.
Lee, K.-H., Nachum, O., Yang. M., Lee, L., Freeman, D., Xu, W., Guadarrama, S., Fischer, I., Jang, E., Michalewski, H., and Mordatch, I. Multi-game deci- sion transformers. In Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022.
Jelinek, F., Mercer, R. L., Bahl, L. R., and Baker, J. M. Perplexity-a measure the difficulty of speech recogni- tion tasks. Journal of the Acoustical Society of America, 62, 1977. URL https: / api. iimmantcccchol...
Li, J., Mchrabi, N., Peris, C., Goyal, P., Chang, K.-W., Galstyan, A., Zemel, R., and Gupta, R. On the steer- ability of large language models toward data-driven per- sonas, 2024a. URL https: 2311. 04978.
Jiang, S., Jiang, D., Balandat, M., Karrer, B., Gardner, J., and Garnett, R. Efficient nonmyopic bayesian optimiza- tion via one-shot multi-step trees. In Advances in Neural Information Processing Systems 20, 2020.
Jones, D. R., Schonlau, M., and Welch, W. J. Efficient global optimization of expensive box functions. Journal of Globul Optimization, 13(4) 1998. ISSN 1573- URL http:
Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual-bandii approach to personalized news article recommendation. In Proceedings of the 19th Interna- tional Conference World Wide Web (WWW 2010), pp. 661-670. ACM, 2010.
2916. doi: 10. 10.10033/A:10000306445111 //dx wtt.:/o///1..112221100003361
Jun, K.-S., Jamieson, K., Nowak, R., and Zhu, X. Top arm identification in multi- aame bandits with batch arm pulls. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 139-148. PMLR, 2016.
Li, X., Lipton, Z. C.,, and Leqi, L. Personalized language modeling from personalized human feedback. arXiv preprint, arXivv2402.05133, 2024b. URL. https:// abt/ 2210.00133..
Lin, L.,, Bai, Y., and Mei, S. Transformers as decision mak- ers: Provable in-context reinforcement learning via super- vised pretraining. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL https: / //openrevies ew net forum? d=yN4Wv1 7ss3.
Kadavath, S., Conerly. T., Askell, A., Henighan, T., Drain, D., Perez, E.,, Schiefer, N., Hatfield-Dodds Z., DasSarma, N., Tran-Johnson, E., Johnston, S., Eh-Sk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman,
12

---

Adaptive Elicctation of Latent Information Using Natural Language
Lin, Z., Liu, J. Z.,, and Shang, J. Towards collabora- tive neural-symbolic graph semantic parsing via certainty. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Findings of the Association for Com- putational Linguisties: ACI. 2022, pp. 4160-4173, Dublin, Ireland, May 2022. Association for Computa- tional Linguistics. doi:
Osband, Asghari, S. M., Roy, B. V., McAleese, N., Aslanides, J., and Irving, G. Fine-tuning language models via epistemic neural networks, 2023a.
Osband, L,, Wen, Z., Asghari, S. M., Dwaracherla, V., Ibrahimi, M., Lu, X., and Van Roy, B. Approximate thompson sampling via epistemic neural networks. In Un- certainty in Artificial Intelligence, pp. 1586-1595 PMLR, 2023b.
328. URL https: / httpaaaaaaannttoo..oooaoaooo www.://oooooyy../222222./. org/2022. Eindings-acl.33 3288
Park, J., Zou, C. Q.Q Shaw, A., Hill, B. M., Cai, C., Morris, M. R., Willer, R., Liang, P., and Bernstein, M. S. Genera- tive agent simulations of 1,000 people. 2024. Preprint.
Lin, Z., Trivedi, S., and Sun, Generating with confidence: Uncertainty quantification for black-box large language models, 2024b.
Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, J., and Sohl-Dickstein, Deep knowledge trac- ing. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M.. and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https::/ neurips.. cc f /cc/sppe__fLlee/ppprr/////////// 2ac21222775560334422111888333333 pdf.
Lindley, D. V. Introduction to Probability and Statistics from a Bayesian Viewpoint. Cambridge University Press, 1965.
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt- 32, 2021. URL https: httpaa:///aavvv.aa wwtpaawwwwwww...////o//aaataaaaa 06804.
Liu, Q., Zhuang, Y., Bi, H., Huang, Z., Huang, W., Li, J., Yu, J., Liu, Z., Hu, Z., Hong, Y., Pardos, Z. A., Ma, H., Zhu, M.. Wang, S., and Chen, E. Survey of computerized adaptive testing: A machine learning perspective. arXiv April 2024. URL https: preprint abtgg//bb///// 2404 00712.
Poddar, S., Wan, Y., Ivision, H., Gupta, A., and Jaques, N. Personalizing reinforcement learning from human feedback with variational preference learning. 2024.
Qin, C., Wen, Z., Lu, X., and Van Roy, B. An analysis of en- semble sampling. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 11602-21614 Curran Associates, Inc., 2022.
//arxiv. org/
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization, 2019. BRLhhtt::///axtt org/ abs/
05101.
Rafferty, A. N., Brunskill, E., Griffiths, T. L., and Shafto, P.
Malinin, A. and Gales, M. Uncertainty estimation in autore- gressive structured prediction, 2021.
Faster teaching by pomdp planning. In Artificial Intelli- gence in Education, pp. 280-287. Springer, 2011.
Nemhauser, G. L., Wolsey, L. A., and Fisher, M. L. An analysis of approximations for maximizing submodular set functions. Mathematical Programming, 14:265-294, 1978.
Riquelme, C., Tucker, G., and Snoek, J. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference on Learning Representations, 2018.
Osband, I., Blundell, C., Pritzel, A., and Roy, B. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, volume 29, 2016.
Rubin, D. B. Inference and missing data. Biometrika, 63 (3):581-592, 1976. ISSN 00063444, 14643510. URL http: /wwp::///www.. j jttorr org/ http://taaeeeee333 /tt.sor..rr//taaaee/3333333 www.storrrrrooo..oooeeeeeeeoeeee
Osband, I., Aslanides, J., and Cassirer, A. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, 2018.
Russo, D. Simple bayesian algorithms for best-arm identifi-
cation. Operations Research, 68(6):1655-144) 2020.
Russo, D.,, Van Roy, B., Kazerouni, A., Osband. I., and Wen, Z. A tutorial on thompson sampling. 2020.
Osband, I., Wen, Z., Asghari, S. M., Dwaracherla, V., Lu, X., Ibrahimi, M. Lawson, D., Hao, B., O'Donoghue, B., and Van Roy, B. The neural testbed: Evaluating joint predictions. In Advances in Neural Information Processing Systems, volume 35, pp. 2255-12565, 2022.
Ryan, E. G., Drovandi, C. C., McGree, J. M., and Pettitt,
A. N. A review of modern computational algorithms for bayesian optimal design. International Statistics Review, 84(1):128-155 2016.
13

---

Adaptive Elicitation of Latent Information Using Natural Language
Salakhutdinov, R. and Mnih, A. Probabilistic matrix fac- torization. In Proceedings of the 21 International Conference on Neural Information Processing Systems, NIPS'07, pp. 1257-1264, Red Hook, NY, USA, 2007. Curran Associates Inc. ISBN 9781 7781605603520.
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., and Schuurmans, D. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129, 2023.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate prob- lem solving with large language models. arXiv preprint 2rX5. 10601, 2023.
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17555 2023.
Si, C., Gan, Z., Yang, Z., Wang. S., Wang, J., Boyd-Graben Graber, J., and Wang, I., Prompting gpt-3 to be reliable, 2023.
Ye, N., Yang, H., Siah, A., and Namkoong, Exchangeable sequence models can naturally quantify uncertainty over latent concepts. arXiv preprint arXiv:2408. 000000003300... 2024. Available at
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, I.., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D.. Nham, J., Kalchbrenner, N.. Sutskever, I., Lillicrap. T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 January 2016. doi: 10. 00.1038/nauure166611
033307.
Zhang, K. W., Cai, T. T., Namkoong, H., and Russo, D.
Posterior sampling via autoregressive generation. arXiv preprint arXiv:2405.19466, 2024. Published on arXiv, 8 October 2024.
Zollo. T. Siah, A. T. Ye, N., Li, A., and Namkoong. H. Personalllm: Tailoring 1lms to individual pref- erences, 2025. URL https: /arxivv.vvvv org//bs/ 2409.20296.
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., and Adams, R. Scalable bayesian optimization using deep neural net- works. In International conference on machine learning, pp. 2171-2180. PMLR, 2015.
Torre, J. L. Dina model and parameter estimation: A di- dactic. Journal of Educational and Behavioral Statistics, 34(1):115-13) 2009.
Wang, X., Wei, J., Schuurmans, D., Lc. Q. Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint 2rXiv:2033.11771 2022.
Wang, Z., Lamb, A., Saveliev, E., Cameron, P., Zaykov, Y., Hermández-Lobato, J. M., Turner, R. E., Baraniuk, R. G., Barton, C., Jones, S. P., Woodhead, S., and Zhang, C. Diagnostic questions: The neurips 2020 education challenge. arXiv preprint arXiv:2007. 12061, 2020.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 248377 2022.
Wen, Z. Osband, I. Qin, C., Lu, X., Ibrahimi, M., Dwaracherla, V., Asghari, M., and Roy, B. V. From predictions to decisions: The importance of joint pre- dictive distributions. arXiv preprint, arXiv:2107.09224, 2021.
Yadkori, Y. A., Kuzborskij. I., György, A., and Szepesvári, C. To believe or not believe your Ilm, 2024.
14

---

Adaptive Elicitation of Latent Information Using Natural Language
Static
juice bottle holds 3.9 liters. share equally among 13 friends, how many liters does each get?
A milk carton has 1.25 liters. How many liters per portion we pour into even glasses?
water bottle has 0.60 liters. Pour it equally into cups. How many liters per cup?
If T'immy $9.75 total from his 13 aunts and each gave the same, much did give?
If Rose gets $1,50 total from her 3 aunts and each gave same, did each give?
Skill Level
Skill Level
Skill Level
Division
Decimals
Division Decimals
Division
Decimals
Adaptive
makes S1375 per 11-hour work day, how much does she make per hour?
John apples share with friends, how many does each get?
A Juice bottle holds 3.9 liters. If share equally among 13 friends, how many liters does each get?
If aunts each give Rose 50.50 Christmas, much does she have?
If aunts ench give Timmy $0.75 Christmas, much does he have?
Skill Level
Skill Level
Skill Level
Division Decimals
Division Decimals
Division Decimals
Figure An example of how adaptive elicitation (bottom) can improve over static strategics (top) new student assessment. Each question asked to the student is marked green (answered correctly) red (answered incorrectly). Once student answers incorrectly, the adaptive strategy is able search over the high-dimensional question space, and present the student with series more granular examples that resolve uncertainty about specitic abilities the high--dieensooal space representing the student's latent abilities. In this case, active question selection reveals that the student is strong division, but struggles with decimal points. static assessment, on the other hand, fails to resolve this uncertainty.
Reproducibility
Our code is at http://///..tooo///////aaoo//aa/
A. Related Work
Latent Uncertainty Modeling and Decision Making Traditional works that attempt to model latent uncertainty to make robust decisions often pose explicit Bayesian models that directly specify latent parameter. Classical multi-armed bandits such Thompson Sampling-basee based methods (Russo, 2020; Grover al., 2018; Chapelle & Li, 2011: Agrawal & Goyal, 2013; Lattimore & Szepesvári, 2019; Jun et al., 2016; Li et al., 2010) specify Bayesian model as a prior, which can be used to draw explicit posterior samples. Examples of Bayesian models include Gaussian and Bernoulli distributions, as well as Bayesian linear or logistic regression (Russo et al., 2020). Bayesian Optimal Experimental Design (BOED) methods (Ghavamzadeh et al., 2015; Chaloner & Verdinelli, 1995; Ryan et al., 2016) follow similar paradigm by quantifying information gain using explicit Bayesian models make sequential decisions.
15

---

Adaptive Elicitation of Latent Information Using Natural Language
Other lines of work include more sophisticated probabilistic modeling. Active collaborative filtering methods (Boutilier al., 2003) specify probabilistic models of user preference data. Bayesian Optimization techniques (Frazier et al., 2008; Frazier, 2018; Gonzalez et al., 2016; Jiang et al., 2020; Jones et al., 1998) apply acquisition functions such as Expected Improvement (Jones et al., 1998) and Knowledge Gradient (Frazier et al., 2008) on top of Gaussian Processes. While these traditional methods are statistically principled, the need to specify explicit models means they often struggle to model very high-dimensional spaces such as the space of natural language. For example, (Frazier, 2018) mentions that Bayesian Optimization methods are best suited to dimensionality V 20, while natural language embeddings are often thousands of dimensions.
To overcome these limitations, recent works have focused on representing uncertainty over high dimensions augmented by the representation power of neural networks. One line of works explicitly uses neural network representations to represent the underlying uncertainty. where decision making algorithms are applied on top of Bayesian Neural Networks (BNNs) or the last layer representation (Snoek et al., 2015; Riquelme et al., 2018; 8; Osband et al., 2016; 2018; Piech et al., 2015). Another strand of works uses ensembles to represent the underlying uncertainty (Qin et al., 2022), or more efficient variants such as Epistemic Neural Networks (Wen et al., 2021; Osband et al., 2022; 2023b). We offer a different approach by directly modeling the uncertainty surrounding future predictions using language models. Methods that explicitly represent uncertainty in terms of ensembles or through specified parameters still struggle to operate in the discrete, high-dimensional space of natural language, whereas our perspective is able to directly represent uncertainty in natural language predictions. Additionally, our meta-learning method does not require new architectures and can be applied on top of powerful pre-trained language models, allowing the use of internet-scale language understanding in comprehending uncertainty.
Computerized Adaptive Testing Our work is closely related to Computerized Adaptive Testing (CAT) methods, form of educational testing that adapts student's s ability level. Classical CAT methods attempt to capture student's latent ability level through simple parametrized models. Item Response Theory, also known Latent Response Theory, includes family of simple mathematical models such as a one parameter logistic regression or Item Response Function to model a student's latent ability (Liu et al., 2024; Embretson & Reise, 2013). The Diagnostic Classification Model (DCM) is designed to measure proficiency across a wide array of specific knowledge concepts. A prominent example includes the DINA method (Torre, 2009; de la Torre, 2011) which uses a probabilistic binary matrix model to represent these concepts. Knowledge Tracing techniques. which train machine learning methods to model the latent knowledge of students as they interact with coursework, traditionally use Hidden Markov Models (HMMs) (Corbett & Anderson, 1994) or Partially Observable Markov Decision Processes (POMDPs) (Rafferty et al., 201 to model the latent state. More modern treatments use deep learning models such as a Recurrent Neural Network (Piech et al., 2015) to represent latent knowledge.
Reinforcement Learning with Sequence Models number of works propose to train or use powerful pre-trained models in order to solve complex reinforcement learning (RL) tasks, focusing on how these models can make decisions using vast amounts of offline data (Janner et al., 2021; Yang et al., 2023; Chen et al., 2021; Du al., 2024; Lee et al., 2022). Another line of works show that using meta-learned sequence models to predict the next action can approximate standard bandit algorithms (Lin et al., 2024a; Lee et al., 2023; Zhang et al., 2024). We extend these ideas to natural language while focusing on how our meta-learned model can quantify uncertainty to make a decision.
Uncertainty Quantification over Natural Language. There has been a recent class of works focusing on developing uncertainty measures to augment the reliability of model responses. (Kuhn al., 2023; Lin et al., 2024b; Malinin & Gales, 2021; Duan et al., 2024) focus on predictive entropy measures with off-the-shelf language models, while other approaches focus on self- self-consistency in the generation space (Lin et al., 2022; Si et al., 2023; Kadavath et al., 2022; Diao et al., 2023). Another class of works focuses instead on detecting epistemic uncertainty from aleatoric uncertainty in model outputs (Yadkori et al., 2024; Osband et al., 2023a; Hou et al., 2024; Glushkova et al., 2021). Our meta-learning uncertainty quantification framework is complementary to these works, as these measures are designed to be applied on top of pre-trained foundation models.
Planning and Information Gathering with LLMs Our work is related to Uncertainty of Thoughts (UoT) (Hu et al., 2024) and OPEN (Handa et al., 2024). While these methods build elicitation procedures on top of off-the-shelf language models, we use a meta-learning procedure in order to accurately quantify uncertainty over new environments. Other works introduce methods to enhance general reasoning or planning capabilities by using natural language reasoning steps (Wei et al., 2022; Wang et al., 2022; Yao et al., 2023).
16

---

Adaptive Elicitation of Latent Information Using Natural Language
Personalization with Language Models With the recent success of Large Language Models (LLMs), natural question is whether these models be tailored and personalize to various users. There has been nascent series of works that propose benchmarks and methods for personalized language modeling. (Zollo al., 2025; Kirk et al., 2024; Castricato et al., 2024) propose new testbeds and evaluation criteria that target various dimensions of personalization through synthetic and real data. (Jang et al., 2023) proposes to merge model parameters to personalize models, while (Li et al., 2024b; Poddar et al., 2024) propose new personalized fine-tuning and Reinforcement Learning from Human Feedback (RLHF) techniques. In the context of opinion polling, (Li et al., 2024a) steers model outputs to different personas by using embeddings from collaborative filtering, while (Park et al., 2024) demonstrates that language models can successfully be adapted individual responses.
B. Theoretical Validity
We now detail two propositions that provide insight into our meta-learning framework. The first proposition lower bounds performance of our model in terms of the performance in the ideal simulation as well as the difference between the true and simulated distribution. The second proposition quantifies the performance gap between a greedy and full combinatorial method, showing the losses one might expect to incur.
First, let Xr- (#1,..., .T). Define X' to be the optimal set of questions X that maximizes the log likelihood of the object of interest Z under the meta-learned distribution pa,
X == argmaxx
po [r)]].
24
This is the set of questions would ask if we could perfectly optimize in our simulated environment. Next, let q be the true distribution and pa our meta-learned model. Using these definitions, we state our first proposition Proposition B.1. For any po.
Eg Blogogo po (Z) x²) 2 Ep, po x*) Ens [log (2 xx))xx(((22) po x))).
(9)
[ollogo X² represents the cross entropy of Z between py and the ground truth distribution q. The lower bound first involves Ep, PoggoopP22
Po (Zl ), which is the likelihood of the data under our simulated distribution. The second term involves Po ())
both the likelihood under the simulated distribution Ep, [log" pa Z X²'] as well as the distance between q and p through
)). This bound tells us that if pa has high likelihood under the simulator and has little distance to g, then we are guaranteed to achieve good test-time performance. However, since the difference x² is scaled by the likelihood, simply having high likelihood in the simulated distribution is not enough. In fact, (4(pp))) is large, then having high likelihood in the simulation can exacerbate this error in the second term. We prove this in Appendix B.2.
Bound for Greedy Policy Next, we bound the performance gap between the questions collected from the optimal policy X with those collected from the greedy policy
greddy
argmax Ep, Dlogpo po (z)...;;-1U
where Xgreedy greedy
greedy 27
First we detine the concept of submodularity:
Definition B.2 (Submodularity). f 2 R is submodular if X CY C Ω and V2 & Y we have
((x - f(x) 2 fY (2)) f(Y)
Then we state our proposition:
Proposition B.3. Under the assumption that the entropy over Z produced by the meta-learned model po is submodular,
1 (IG
Epu [og po (Z xx*) - EEpu [log po &greedy) )
We prove this statement in Appendix B.3. This proposition states that the difference in achieving the optimal log likelihood under the simulated environment and using the greedy strategy is bounded by (Z; We can then use this bound
17

---

Adaptive Elicitation of Latent Information Using Natural Language
and substitute in Proposition B.1 quantify the performance lower bound for the greedy policy, In this proposition, we have assume submodularity of our meta-learned model because it is not guaranteed in practice due training instabilities errors. Empirically we find that the entropy of our meta-learned model behaves submodularly, as evidenced by the perplexity graphs in Figure 2.
B.1. Summary
Our framework provides a data-driven, natural-language based alternative to parametric modeling of latent entities. It proceeds by: (1) Meta-training a language model on diverse prestion-answerr sequences, (2) Interpreting the model's predictive distribution over future answers as an uncertainty measure about new entities, and (3) Iteratively selecting questions to optimally reduce that uncertainty. We show theoretically that our procedure gives strong performance, even under simple and efficient greedy planning strategy. Next, we explore the empirical performance our framework in a series of adaptive information gathering scenarios.
In this section, show the theoretical validity of using greedy procedure to select actions with the highest expected information gain. We wish to quantify and reduce our uncertainty about some object Z by choosing the optimal questions X to query the latent entity U. First we define the expected information gain of asking a set of questions X₁ == (II, F2) S X,
(I; (Z) TE, (Z U Y.)]
Note that each Y, in the history is a random variable, and we use our meta-learned model po to simulate possible answers Y, ~ 1, X, = xx). Similarly, Z ~ >>p(-) as well. Ultimately, our goal to choose a set designs X; = 3 that yields the largest amount of information gain possible. First set notation, define q to be the ground truth underlying question and answer distribution. Let p be the distribution induced by the meta-learned model, and p(Z(X) be the conditional distribution over Z after marginalizing out the feedback Y1:1 corresponding to the questions Xr where Y1:1 comes from the ground truth distribution such that Y1a+ ~ q(-).
First, define X to be the optimal set of questions X that maximizes the log likelihood of the object of interest Z under the meta-learned distribution P.
X argmax Ep p(Z x,)]-
X
B.2. Proof of Proposition B.1
We restate our proposition for clarity.
E. flo p(Z x²)) Epllogp(2 x²)]
Elllog p(Z) x*))xx(((Z)) pl x))).
Proof. We can decompose the difference between the cross entropy term E. [ogpp( X²))] and entropy term log p( X²)) as
El x²)) E, [log x²']]
= f (p(=)=*-) - log x=
/ p(= 1x*))
9(2)
log logp(> (z xx)p(= x x))dz
(fa- p(= X²) )2p(z xz) (f (1og2(ote xx()) 12*)
((2)
E,,loo" p(Z X 1)2 plz 1x))).
where the second to last line follows from the Schwart inequality and the last line follows from the definition of the x2 divergence. Then by flipping the sign, we obtain the stated inequality.
18

---

Adaptive Elicitation of Latent Information Using Natural Language
B.3. Proof of Proposition B.3
We first define submodularity for clarity:
Definition B.4 (Submodularity), f 22 + R is submodular if YY C Ω2 and Va & Y we have
S(YU ((X)
(()
In order to show that the greedy procedure is able to perform close to the optimal solution, we rely on the following assumption, that the entropy calculated from our meta-learned model is submodular:
Assumption B.5 (Submodularity of Entropy). Let H, S Hi. Then for any (X++1. Y++1) $ Hi,
H(Z He U (X Y44)) ( H4) H(Z Hi U (X (44)) H(Z Hy).
We first show that the Expected Information Gain (EIG) is submodular. If the entropy submodular, then the Information Gain also submodular. Define the Information Gain as
f(H.) IG())=IG(2::)) H.) H(Z) (Z H.).
Then, for any history set H, and any additional observation (X++1: Y(+1) & He, the marginal gain of adding (X++1 Y441 given by
J(H U ((X+11, Y++))) ((H,) = H()) H H4 U ((X (o1)))++11)]]]])) - [1 -(F())
H(2 Hii
= H H4)) H(Z Hi U
Now, consider two history sets H, C Hj and the same observation (Xt-1, Y(+1) & H/. By our submodularity assumption on the entropy, have
H H4) - H | H4 U (( M(+1)) 2 HZ H4) - H(Z | H4 ((xe).
In terms of the Information Gain function, this inequality becomes
f(He U ((X+11 Y(41)) - f(H)) 2 f(H ((x+11, - f(H)).
Thus, by definition the Information Gain is submodular. Since this is true for all X,41, Y141, then the Expected Information Gain (EIG) is also submodular. Then by (Nemhauser et al., 1978), we have that
EIG Agreedy) 2 (1 EIG(Z)
implying that
EIG Z; (Z: - EIG(Z; 1 EIG(Z:X").
(10)
Finally, to prove the bound we can note that
Epllog p(Z X²)] - Epllog p(Z Agreedy) ) = Ey|EEz p( | - Ey E2 [log p(Z Kgreedy, Vgreedy I
= [Z X" x²))") Ey[H( Xgreeet ; ) )]
[Y xx,""") H(2)+ + +H(Z) - Ey[H( Rgreedy V Vercody) )
EIG(Z; Kgresty) EIG(Z;
=EIG(Z;X*),
where the last line follows from an application Equation (10). This completes the proof.
19

---

Adaptive Elicitation of Latent Information Using Natural Language
EEDI
OpinionQA
Twenty Questions
0.26
ICT Ours
0.145
0.100
Brier Brier Score
0.25
0.140
0.095
0:24
0.135
0.22
0.130
0.080
0.125
0.075
2 # Questions Selected
0
2
3
4
1
2
Questions Selected
# Questions Selected
Figure Brier score results in overall setting across datasets.
EEDI
1 Questions Selected
4 Questions Selected
8 Questions Selected
0.80
1.0
1.0
1.0
Confidence Avg. Accuracy
Perfect Calibration
0.75
Accuracy
0.8
0.8
0.8
0.65
0.2
0.2
0.2
ECE₂ 0.021
ECE₂ = 0.0191
ECE; = 0.0217
0.60
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Questions Selected
Confidence
Confidence
Confidence
Figure 8. Calibration results with FEDI.
C. Experiment Details
C.1. Meta-Training Details
We initialize the AdamW (Loshchilov Hutter, 2019) optimizer with learning rate of 0.0001 and B = (0.9, 0.95), weight decay of 0.1, and we use linear warmup for the learning rate after which we use cosine scheduler. We train our model for 10, 000 epochs with batch size of 4 and block size of 1024, after which we take the checkpoint with the lowest validation loss.
D. Experiment Results
Here we include additional experiment results. Figure 7 shows results for the overall experiments with the Brier Score metric. Figure shows calibration results for EEDI, and Figure shows calibration results for Twenty Questions.
E. Ablations
Additional ablations are shown in Table 1 (number of target questions), Table 2 (number of candidate questions), Table 3
(base model).
Table Ablating number target questions on EEDI; accuracy conditioned on Questions
Accuracy
10
20
Base
0.6042
0.6005
0.6066
0,5987
Ictx
0.6269
0.6278
0.6295
0.6255
Ours
0.6759
0.6784
0.6871
0.6832
20

---

Adaptive Elicitation of Latent Information Using Natural Language
Twenty Questions
1 Questions Selected
Questions Selected
8 Questions Selected
Avg. Confidence Accuracy
Perfect Calibration
0.85
Accuracy
0.6
0.6
0.6
0.80
0.4
0.4
0.4
0.2
0.2
0.2
ECE, 0.0048
ECE, 0.007
ECE; 0.0071
0.70
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
# Questions Selected
Confidence
Confidence
Confidence
Figure 9, Calibration results with Twenty Questions.
Table 2. Ablating number of candidate questions OpinionQA; accuracy conditioned Questions
Accuracy
10
15
20
25
Base
0.4030
0.4042
0.4089
0.4093
Ictx
0.4988
0.4993
0.5023
0,5009
Ours
0.5933
0.5953
0,5987
0.6068
Table 3. Ablating base model: Twenty Questions accuracy using our framework conditioned on 4 questions
GPT2
Llama-3.2-1B
Llama-3.1-8B
Accuracy 0.5201
0.6131
0.7382
21


Reviews:
[{"review_id": "x8AHWzrRtO", "reviewer": "Reviewer_BfZB", "summary": "This paper introduces a framework for adaptive elicitation of latent information using LLMs to optimize question selection based on predictive uncertainty. Instead of explicitly modeling latent variables (e.g., knowledge levels or user preferences), the method quantifies uncertainty via LLM perplexity and simulated future responses to guide information gain-based question selection. The approach, tested on opinion polling, student assessment, and the Twenty Questions game, outperforms baselines by efficiently reducing uncertainty. The key contributions include uncertainty-driven question selection (via Greedy EIG and MCTS) and a general-purpose adaptive questioning framework applicable to diverse domains like education and healthcare.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and consideration taken in reviewing our paper, and for recognizing our contribution to the application of uncertainty-driven question selection in important domains like education and healthcare.  Below, we respond to your particular concerns.\n\n**[Q1: Modeling the latent, relationship to perplexity, and separation of epistemic + aleatoric uncertainty]**\n\nMost latent entities have no obvious physical meaning (e.g., academic proficiency or political opinions), and can only be observed through noisy observations (e.g., student answers where they guess if they're unsure).\nOur framework avoids the need to perform such direct modeling, allowing us to build a framework that is able to be applied on existing LLMs to use their internet-scale pretrained knowledge to make robust, adaptive decisions.  \n\nOur formulation of uncertainty quantification derives from a large body of work on inference with missing data (e.g. [1-2]), and we will include a description of this relationship and our characterization of\nepistemic and aleatoric uncertainty in the paper.  Under this view, knowing $U$ is defined as being able to predict any answer $Y$, with errors only due to random, aleatoric variation. Observing the infinite sequence $(X_{1:\\infty}, Y_{1:\\infty})$ reduces all epistemic uncertainty, leaving only aleatoric uncertainty: $H(Y \\mid U) = H(Y \\mid X_{1:\\infty}, Y_{1:\\infty})$. Then, epistemic uncertainty in $U$ naturally corresponds to uncertainty in the *missing data* $Y_{t+1:\\infty}$ given history $Y_{1:t}$. The joint likelihood $P(Y_{t+1:\\infty} \\mid X_{1:t}, Y_{1:t})$ exactly quantifies uncertainty about the missing data $Y_{t+1:\\infty}$ which shows it is the correct objective. Under the special condition of exchangeability, this is precise: there exists a function $\\theta(X_{1:\\infty}, Y_{1:\\infty})$, such that $H(Y \\mid U) = H(Y \\mid \\theta(X_{1:\\infty}, Y_{1:\\infty}))$ exactly.      \n\nSimulating future trajectories allows us to quantify which actions will reduce the most epistemic uncertainty. To quantify uncertainty about a target $Z$, we calculate\n    $$\\text{Epistemic Uncertainty}(X_{1:t}, Y_{1:t}) = (\\text{Total}) H(Z \\mid X_{1:t}, Y_{1:t}) - (\\text{Aleatoric}) \\mathbb{E}[H(Z \\mid X_{1:\\infty}. Y_{1:\\infty})]$$ If we could infinitely simulate $Y \\sim p_{\\theta}$, we can exactly calculate epistemic/aleatoric uncertainty. In practice, we will simulate finite time steps to approximate this quantity. \n\nIn response to this concern, we will work to clarify these discussions in our revised paper.\n\n**[Q2: Data requirements]**\n\nWe appreciate the reviewer's feedback about issues such as dataset imbalance or insufficient data. We will include a grounded discussion of the fundamental limitations of our approach---also shared by other approaches---in the camera ready version. While LLMs may learn average-case behavior, our meta-learning approach is designed to personalize to individuals by understanding what data to gather about each specific individual. A key advantage of our method is we can include any/all user information for the LLM to condition on for sufficient personalization. \n\n**[Q3: Complex Reasoning]**\n\nOur work takes on a different approach to reasoning by integrating explicit uncertainty quantification and estimation with the natural language capabilities of LLMs. Our method can be complementary to reasoning models such as OpenAI's O1 and DeepSeek-R1 by explicitly integrating reasoning with our uncertainty estimation capabilities. Such ideas are beyond the scope of this work, as we aim to first demonstrate the validity of our UQ framework. \n\n**[Q4: Computational Expense]**\n\nWhile MCTS can be a bottleneck in settings with large question spaces or strict real-time constraints, it remains practical for many applications including those we study. In education, for example, computation can occur during a student’s response time, causing only minor lag. In contrast, real-time tasks like diagnostics or dialogue may require faster methods such as greedy selection. Our work offers a new perspective on uncertainty-guided questioning, and we hope it inspires more efficient approaches (e.g., via distillation). \n\nWe also note that methods like embedding-based methods will require a linear scan of the possible questions, yielding a complexity similar to our greedy method. Furthermore, methods that directly use LLMs to generate questions and responses require forward passes in the number of tokens in each question and responses, while our method only requires one forward pass in this respect. We will include a detailed discussion of complexity/tradeoffs in the camera-ready version. \n\n[1] Edwin Fong, Chris Holmes, and Stephen G Walker. Martingale posterior distributions. Journal of the Royal Statistical Society, Series B, 2023.\n\n[2] Naimeng Ye and Hongseok Namkoong. Exchangeable sequence models quantify uncertainty over latent concepts. arXiv preprint arXiv:2408.03307, 2024.", "claims_and_evidence": "The claim can be supported by the paper", "methods_and_evaluation_criteria": "Since the framework does not explicitly define or model the latent variable U, it remains unclear what the model has actually learned about the underlying information structure. This lack of interpretability makes it difficult to trust the system’s reasoning process, especially in high-stakes applications like education assessment or medical diagnosis. A more structured approach, such as probabilistic graphical models (e.g., Variational Autoencoders, Hidden Markov Models), could improve transparency.\n\nThe paper primarily uses PPL as the main measure of uncertainty, but PPL only reflects how well a language model predicts text sequences, not the true epistemic uncertainty about the latent variable. In scenarios where the dataset is imbalanced, the model may simply reinforce dominant patterns—for example, assuming that all patients have high blood pressure because it is the most common label in the data. This can lead to overconfident yet incorrect predictions, which undermines the reliability of uncertainty-based question selection.", "theoretical_claims": "Correct", "experimental_designs_or_analyses": "The method relies on high-quality training data, but it is unclear where such data can be sourced across different domains like education and healthcare. Real-world datasets are often noisy, biased, or incomplete, which can affect the reliability of the model’s predictions. Additionally, LLMs inherently learn statistical patterns at the population level, making them effective for broad generalizations but less suited for capturing individual-specific nuances due to the lack of explicit memory or personalization.\n\nThe framework is evaluated only on Q&A-style tasks such as OpinionQA, student assessment, and the Twenty Questions game, which focus primarily on eliciting missing information rather than complex decision-making. This raises concerns about how well the method generalizes to more challenging tasks, such as medical diagnosis, scientific research, or multi-step reasoning, where information acquisition needs to be integrated with reasoning and planning.\n\nThe approach requires multiple forward passes through the LLM to compute uncertainty via response sampling, which is computationally expensive. This makes it impractical for real-time applications, such as interactive tutoring or live medical diagnostics, where response speed is crucial. Reducing reliance on repeated sampling, optimizing inference efficiency, or leveraging smaller models for uncertainty estimation could help mitigate this issue.", "relation_to_broader_scientific_literature": "The use of perplexity (PPL) to quantify uncertainty relates to prior work on uncertainty-aware machine learning, particularly in Bayesian deep learning and active learning. Traditional methods, such as Monte Carlo Dropout (Gal & Ghahramani, 2016) and Deep Ensembles (Lakshminarayanan et al., 2017), explicitly model uncertainty by estimating the variance of predictions. In contrast, this paper proposes a novel use of LLM perplexity as a proxy for predictive uncertainty, which aligns with prior research on entropy-based uncertainty estimation in NLP (e.g., Maaløe et al., 2019). However, unlike standard Bayesian methods, this approach does not explicitly separate aleatoric and epistemic uncertainty, which may limit its robustness.\n\nThe idea of selecting optimal questions to reduce uncertainty aligns with Bayesian Experimental Design (Chaloner & Verdinelli, 1995) and active learning methods (Settles, 2010), which aim to iteratively collect informative data. Prior work in educational assessment (Reich, 2012) and personalized learning (Piech et al., 2015) has explored similar strategies for adaptively selecting test questions to estimate student knowledge. This paper extends these concepts by using large language models to dynamically generate and select questions based on expected information gain (EIG). The approach is conceptually similar to Pólya tree priors (Hanson, 2006) in Bayesian adaptive testing but replaces probabilistic models with LLM-driven heuristics.\n\nRecent studies have explored using LLMs as agents for reasoning and interactive decision-making (e.g., Brown et al., 2020 (GPT-3); Wei et al., 2022 (Chain-of-Thought Reasoning)). This paper contributes to this line of research by demonstrating how LLMs can be used for adaptive information gathering, a capability related to human-like cognitive strategies for active inference (Friston, 2010) and rational metareasoning (Griffiths et al., 2019). Unlike previous work on prompting-based reasoning, this study frames LLMs as active participants in information elicitation, rather than passive responders.\n\nThe use of Monte Carlo Tree Search (MCTS) for multi-step question selection connects this work to reinforcement learning (Silver et al., 2016, AlphaGo) and Bayesian optimization for decision-making (Snoek et al., 2012). While MCTS has been widely applied to game playing and planning problems, its application to adaptive questioning with LLMs is relatively novel. However, this paper does not fully integrate reinforcement learning techniques, which distinguishes it from related work in LLM self-improvement (Schick et al., 2023).", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "See Methods And Evaluation Criteria and Experimental Designs Or Analyses.", "other_comments_or_suggestions": "See Methods And Evaluation Criteria and Experimental Designs Or Analyses.", "questions_for_authors": "See Methods And Evaluation Criteria and Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qfe87munLc", "reviewer": "Reviewer_nfGQ", "summary": "This paper introduces an adaptive elicitation framework for actively reducing uncertainty about latent entities, using adaptive query selection and simulated counterfactual responses. It leverages a meta-trained LLM to quantify uncertainty about future or unobserved answers via simulation, then iteratively selects queries that maximize expected information gain. Experiments on dynamic opinion polling, adaptive student assessment, and a structured 20 Questions game demonstrate that the method significantly improves accuracy and uncertainty reduction.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort in reviewing our paper, and their thoughtful feedback.  We are pleased that the reviewer appreciated the importance of the topic as well as the novelty of our approach.  Below please see our responses to the particular concerns raised.\n\n**[Q1: Sensitivity to embedding model]**\nAs per the reviewer's suggestion, we ran additional experiments to ablate the importance of the embedding model. We tested 5 embedding models: stella_en_1.5B_v5, Qwen2-7B-instruct, e5-mistral-7b-instruct , OpenAI text-embedding-3-large, and AlibabaNLP/gte-large-en-v1.5 (which we used in our original paper), and measured their accuracy after conditioning on 3 in-context questions for the EEDI and OpinionQA dataset. We set the number of targets to be 5 and number of possible questions to ask as 20, which mirrors the setup in our paper. Here are the experiment results, where numbers reported are for each entry is an average over 10,000 different runs: \n\n| Embedding Model                         | EEDI Accuracy | OpinionQA Accuracy |\n|----------------------------------------|---------------|--------------------|\n| AlibabaNLP/gte-large-en-v1.5           | 0.621         | 0.505              |\n| stella_en_1.5B_v5                      | 0.618         | 0.510              |\n| gte-Qwen2-7B-instruct                  | 0.625         | 0.501              |\n| e5-mistral-7b-instruct                 | 0.626         | 0.506              |\n| OpenAI text-embedding-3-large          | 0.631         | 0.515              |\n\nWe find that while there is a bit of variation in the embedding model performance. For EEDI, we notice that the larger 7B models and OpenAI text-embedding-3-large have slightly higher performance. For OpinionQA, the performance is more varied. However, none of the embedding models make the In-Context Tuning + Embedding baseline outperform our method. Our method achieves 0.678 accuracy on EEDI and 0.597 accuracy on OpinionQA, as reported in Figure 3. \n\n**[Q2: Computational Complexity]**\n\nIn terms of analysis, let $n$ be the number of possible questions to ask and $m$ be the number of possible answers to each question. The computational complexity of the EIG method at each time step $t$ is $O(m\\cdot n)$. This is because we need to calculate the conditional entropy of each answer for each feasible question in the set. To calculate the complexity of MCTS, let $z$ be the number of trajectories we simulate and $d$ be the depth of simulation. Then MCTS is $O(z\\cdot n\\cdot m\\cdot d) $, as we have $z$ simulations up to depth $d$ for each question, and at each depth we need to perform an EIG calculation. We will provide a detailed computational complexity analysis in our final draft.\n\nTo compare the complexity of our method to other approaches,  we note that other common methods like embedding-based methods will require a linear scan of the possible questions, yielding a complexity similar to our greedy method. Furthermore, methods that directly use LLMs to generate questions and responses require forward passes in the number of tokens in each question and responses, while our method only requires one forward pass in this respect. Applying techniques like MCTS on top of using LLMs to generate responses will incur complexity $O(z\\cdot n\\cdot m\\cdot d)$ multipled by complexity of generating responses.\n\nWe acknowledge that the complexity of the MCTS approach can be a bottleneck in scenarios with an extremely large question space or stringent real-time constraints. However, we also want to highlight that there are also many important applications (including some that we consider) where this should not be prohibitive.\n\nFor example, in educational settings the extra computation can be performed during the student’s response time, as the policy is updated while the student works on the current question—thus introducing a tolerable lag. Conversely, in real-time applications such as live diagnostics or interactive dialogue systems, the computational overhead of MCTS might be prohibitive, and more efficient strategies (e.g., greedy selection) may be preferred. We view our work as introducing a new conceptual perspective on uncertainty-guided questioning, and we hope that future work will build on our approach to develop more computationally efficient solutions (e.g., via distillation). \n\nIn response to the reviewer's concern, we will include a more detailed discussion of these trade-offs and complexity considerations in the camera-ready version.\n\n**[Q3: Related work on latent entity modeling]**\n\nWe agree with the reviewer that a review surrounding latent entity modeling would be of major benefit. We will make sure to include this discussion in the camera-ready version of the paper.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence, including empirical results across three different tasks (Opinion QA, EEDI tutoring, and 20 questions), and the theoretical analysis on the expected information gain.", "methods_and_evaluation_criteria": "The use of LLMs meta-trained with historical data containing latent entities is reasonable and well-motivated. Predictive perplexity as a measure of uncertainty makes sense, as it provides a practical and scalable way to approximate epistemic uncertainty without requiring explicit latent variable modeling. For the evaluation, the selected datasets cover diverse domains, and the authors select common metrics for uncertainty quantification, such as ECE and Brier score.", "theoretical_claims": "I did not check the proofs in the Appendix due to time constraint.", "experimental_designs_or_analyses": "* Validity of the experimental design is checked.\n\nThe baselines (in-context tuning, base LLM with random question selection) use the same backbone model or meta-training setting as in the main experiment, which is a fair comparison. The split is done with latent entities instead of just QA pairs, which avoids data leakage. A consistent improvement in performance and a reduction in uncertainty are clear across all three datasets and evaluation metrics, showing the effectiveness of the adaptive elicitation framework. The ablation studies are extensive, including comparisons between adaptive (EIG, MCTS) and random selection for different subsets of difficulty, the comparisons of performance gains from planning using different models, and the comparisons of different number of target questions.", "relation_to_broader_scientific_literature": "The paper builds on previous RL research demonstrating that pretrained models can be adapted for decision-making tasks with offline data, and meta-learned sequence models approximate bandit algorithms, but instead focuses on the natural language setting and uncertainty quantification in decision making. It is complimentary and aligns with the works on uncertainty quantification over natural language. It also fit into the category of planning and information gathering with LLMs,. However, unlike prior methods that rely on off-the-shelf models for uncertainty estimation, it uses a meta-learning procedure.", "essential_references_not_discussed": "To my knowledge, there are no essential related works that are missed.", "other_strengths_and_weaknesses": "Strengths:\n\n* Uncertainty-aware decision-making in LLMs is an important topic, and the specific focus of eliciting latent entities with meta-training and adaptive question selection appears a novel contribution.\n\n* The selected datasets are from various domains, and effectively demonstrate the framework’s potential for latent entity elicitation.\n\n* Code is available\n\n* The paper is well-presented and easy to follow\n\nWeaknesses:\n\n* Lacking analysis and comparison on the overall complexity of the method (especially for variants with MCTS)", "other_comments_or_suggestions": "* It would be helpful if the authors could outline \"the rich body of literature on topics related to latent entity modeling\" in the appendix.", "questions_for_authors": "* Is the performance sensitive to the choice of the embedding model? Have you tried such ablation with different embedding models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fmC6zUKgSv", "reviewer": "Reviewer_hSjM", "summary": "Authors consider a meta-learning QA scenario in which each dataset contains an unobservable latent variable (e.g., medical notes with an unseen clinical diagnosis). They propose an iterative and adaptive framework designed to reduce uncertainty around these latent variables.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and thorough review, and their recognition that our claims are well-supported by experiments. We response below to the concerns. \n\n**[Q1: Use of \"counterfactual response\"]**\n\nWe apologize for the confusing use of the term \"counterfactual response\" in the paper. \"Counterfactual\" has no reference to its usage in the causal inference literature and a better term to use would be \"simulated response\". We will replace this in the final draft. \n\n**[Q2: Predictive perplexity / Joint likelihood]**\n\nOur formulation of uncertainty quantification derives from a large body of work on inference with missing data (e.g. [1-2]).  Under this view, knowing $U$ is defined as being able to predict any answer $Y$, with errors only due to random, aleatoric variation. Observing the infinite sequence $(X_{1:\\infty}, Y_{1:\\infty})$ reduces all epistemic uncertainty, leaving only aleatoric uncertainty: $H(Y \\mid U) = H(Y \\mid X_{1:\\infty}, Y_{1:\\infty})$. Then, epistemic uncertainty in $U$ naturally corresponds to uncertainty in the *missing data* $Y_{t+1:\\infty}$ given history $Y_{1:t}$. The joint likelihood $P(Y_{t+1:\\infty} \\mid X_{1:t}, Y_{1:t})$ exactly quantifies uncertainty about the missing data $Y_{t+1:\\infty}$ which shows it is the correct objective. Under the special condition of exchangeability, this is precise: there exists $\\theta(X_{1:\\infty}, Y_{1:\\infty})$, such that $H(Y \\mid U) = H(Y \\mid \\theta(X_{1:\\infty}, Y_{1:\\infty}))$.\n\nWe will include a detailed description in the paper. \n\n**[Q3: Simulation to quantify epistemic/aleatoric?]** \n\nSimulating future trajectories allows us to quantify which actions will reduce the most epistemic uncertainty. To quantify uncertainty about a target $Z$ we calculate,\n    $$\\text{Epistemic Uncertainty}(Z | X_{1:t}, Y_{1:t}) = (\\text{Total}) H(Z \\mid X_{1:t}, Y_{1:t}) - (\\text{Aleatoric}) \\mathbb{E}[H(Z \\mid X_{1:\\infty}. Y_{1:\\infty})].$$ In practice, we approximate this quantity using simulation to calculate existing epistemic/aleatoric uncertainty. The EIG then quantifies the reduction in epistemic uncertainty, which we also calculate by simulation. \n\n**[Q4: Exchangability assumption]**\nWe agree with the reviewer that the exchangeability assumption may be too strong in practice. We will outline these fundamental limitations in our camera-ready version. We also emphasize that exchangeability mainly serves as inspiration for our practical UQ framework, alongside our strong empirical results. \n\n**[Q5: Computational analysis]**\nDefine $n$ as the number of questions to ask and $m$ be the number of possible answers.  The computational complexity of the EIG method at each time step $t$ is $O(m\\cdot n)$ because we need to calculate the conditional entropy of each answer for each feasible question in the set. The complexity of MCTS is $O(z\\cdot n\\cdot m\\cdot d)$, where $z$ is the number of simulations and $d$ be the depth of simulation, since there are $z\\cdot d$ EIG calculations. We will include a detailed discussion of complexity analysis in the camera-ready version.\n\n**[Q6: Proof of A4]**\n\nSteps (5-7) in the proof outline a telescoping expansion of the KL divergence term. The telescoping sum consists of three terms. (5) is the difference between the ground truth distribution $q$ and $p$ conditioned on the optimal information $X_{p_{\\theta}}^{\\ast}$. (6) is the difference of $p(Y_{t:T} | X_{p_{\\theta}}^{*})$ when $Y_{t:T}$ is simulated from $q$ versus $p$. Finally (7) measures the difference of conditioning on the optimal information $X_{p_{\\theta}}^{\\ast}$ and $X_{greedy}$. \n\nRegarding Fact A.2, we apologize as we made a typo. Fact A.2 refers to the fact that for any $S \\in \\mathcal{S}, 0\\leq H(S) \\leq \\log |S|$ on lines 671-674. The term $\\log (|Y| (T-t))$ comes from bounding $H(Y_{t:T} | X_{p_{\\theta}}^{*})$ on line 685. Because $Y_{t:T}$ is exchangeable and the ordering does not matter, then the cardinality of $Y_{t:T}$ is $|Y|(t-T)$.\n\n**[Q7: Realistic use cases]**\nThank you for this suggestion, we agree that incorporating more realistic use cases could strengthen the evaluation, especially in high-impact areas like medical diagnosis.  While 20 Questions is synthetic, we highlight that EEDI and OpinionQA are real-world datasets in personalized tutoring and opinion polling, important domains where a technique like ours might make significant impact. \n\n**[Q8: Related work/qualitative examples]**\nWe thank the reviewer for pointing out missing related work; we will include these and additional references in our final version. We will also include examples of questions and answers in the paper, and show how uncertainty changes in practice.  \n\n[1] Edwin Fong, Chris Holmes, and Stephen G Walker. Martingale posterior distributions. Journal of the Royal Statistical Society, Series B, 2023.\n\n[2] Naimeng Ye and Hongseok Namkoong. Exchangeable sequence models quantify uncertainty over latent concepts. arXiv preprint arXiv:2408.03307, 2024.", "claims_and_evidence": "**Claim 1.** \"We introduce an adaptive elicitation framework that uses natural language to actively reduce uncertainty on the latent entity by simulating counterfactual responses\"\n\nYes, the framework is introduced in Sec. 2. I am not sure that the responses are actually counterfactual in this framework. See “Questions For Authors”.\n\n**Claim 2.** \"Our approach enables the model to identify epistemic uncertainty and facilitate sophisticated information gathering strategies as it updates its understanding of the latent ...\"\n\nEven though the method provides some form of uncertainty over a latent entity, I don’t see clear justification that predictive perplexity estimates epistemic uncertainty over a latent entity.\n\n**Claim 3**. Experimentally, “we illustrate the versatility and significant potential of our framework to enable more efficient and targeted information elicitation in critical domains and applications.”\n\nThis claim is well-supported by the experiments.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are logical. However, incorporating a realistic use case would make the argument more compelling.", "theoretical_claims": "1. The assumption of exchangeability is too strong. See “Other Comments and Suggestions” for further details.\n\n2. The proof of A4 was difficult to follow. Specifically:\n- Steps (5-7) require more detailed explanations.\n- The definition and role of fact A2 are unclear.\n- The bound $\\log⁡(∣Y∣(T−t))log(∣Y∣(T−t))$ (LL680) is not well-explained.\n- It is unclear why assumption A2 is referred to as a fact.\n\nAs a result, I was only able to fully understand the bound of the third term.", "experimental_designs_or_analyses": "I think the datasets and baselines are well-selected. The only issue I see is the lack of computational efficiency analysis.", "relation_to_broader_scientific_literature": "I think the paper discusses well the relation to broader scientific literature. The work nicely expands works on LLM planning and uncertainty quantification to a an important application of reducing uncertainty over a latent entity in a dialogue.", "essential_references_not_discussed": "I think more relevant works should be mentioned in related work section, for instance:\n\n### Planning with LLMs\nLanguage Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents https://arxiv.org/pdf/2201.07207\n\nReAct: Synergizing Reasoning and Acting in Language Models https://arxiv.org/pdf/2210.03629\n\n### UQ for LLMs\nKernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities https://arxiv.org/abs/2405.20003\n\nReducing Conversational Agents’ Overconfidence Through Linguistic Calibration https://aclanthology.org/2022.tacl-1.50/", "other_strengths_and_weaknesses": "### Strength:\n- Original work and important topic,\n- The method is valid and supported by the experimental results\n\n### Weakness:\n- Some assumptions (exchangeability) are not well discussed, and are not realistic\n- The method in my opinion could be motivated better, e.g., \n    * Why optimizing the joint log likelihood/marginal likelihood is optimal?\n    * Why is simulation needed to quantify uncertainty?", "other_comments_or_suggestions": "Section 2: the method explanation in the beginning was unclear,\n\nThe exchangeability assumption is overly strong and known to be incorrect based on findings from cognitive psychology, such as order effects, framing, and anchoring effects. It is important to discuss and emphasize that this assumption is not realistic.\n\nIt would be useful to provide examples of questions and answers in the paper, and show how the uncertainty changes in practice.", "questions_for_authors": "I'm puzzled by the term \"counterfactual response\" in the paper. Could you clarify its meaning? I don't see how the responses generated by your framework qualify as counterfactual. Are you using \"counterfactual\" in a different sense than its common usage in causal inference literature?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dSXAQP7QNi", "reviewer": "Reviewer_54sR", "summary": "This paper studies the problem of adaptively selecting queries to reduce uncertainty on a latent entity. They propose to fine-tune an LLM for meta-learning the task of question answering with a latent entity (e.g. the 20 questions game with different hidden entities). This approach allows for measuring uncertainty of the latent entity as the LLM’s predictive uncertainty in predicting answers to future questions. This approach is shown to be effective in producing calibrated uncertainty estimates which allows for better uncertainty-guided question selection.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback.  We are encouraged that the reviewer found the problem setting to be well motivated, and our experiments and results to be presented clearly.  Please see our responses to the particular concerns raised.\n\n\n**[Q1: Calculating confidence]**\n\nConfidence is calculated in the typical way as the softmax response for the predicted answer.  In response to this concern, we will clarify this calculation in our revised paper, as well as clearly specify when we use related notions of entropy or variability. \n\n**[Q2: Computational Expense]**\nIn terms of analysis, we provide a brief overview for the reviewer. Let $n$ be the number of possible questions to ask and $m$ be the number of possible answers to each question. The computational complexity of the EIG method at each time step $t$ is $O(m\\cdot n)$. This is because we need to calculate the conditional entropy of each answer for each feasible question in the set. To calculate the complexity of MCTS, let $z$ be the number of trajectories we simulate and $d$ be the depth of simulation. Then MCTS is $O(z\\cdot n\\cdot m\\cdot d) $, as we have $z$ simulations up to depth $d$ for each question, and at each depth we need to perform an EIG calculation. We will provide a detailed computational complexity analysis in our final draft.\n\nTo compare the complexity of our method to other approaches,  we note that other common methods like embedding-based methods will require a linear scan of the possible questions, yielding a complexity similar to our greedy method. Furthermore, methods that directly use LLMs to generate questions and responses require forward passes in the number of tokens in each question and responses, while our method only requires one forward pass in this respect. Applying MCTS on top of using LLMs to generate responses (as in UoT) will incur complexity $O(z\\cdot n\\cdot m\\cdot d)$ multiplied by complexity of generating responses.\n\nWe acknowledge the complexity of the MCTS approach can be a bottleneck in scenarios with an extremely large question space or stringent real-time constraints. However, we also want to highlight that there are also many important applications (including some that we consider) where this should not be prohibitive.\n\nFor example, in educational settings the extra computation can be performed during the student’s response time, as the policy is updated while the student works on the current question—thus introducing a tolerable lag. Conversely, in real-time applications such as live diagnostics or interactive dialogue systems, the computational overhead of MCTS might be prohibitive, and more efficient strategies (e.g., greedy selection) may be preferred. We view our work as introducing a new conceptual perspective on uncertainty-guided questioning, and we hope that future work will build on our approach to develop more computationally efficient solutions.\n\n\n\n**[Q3: Comparison to other methods]**\n\nWhile there exists a growing body of literature on UQ in LLMs, much of this work is focused on different settings and types of uncertainty than we address here.  In particular, much of this growing body of work is focused on short-form generation and QA style tasks [e.g., 1-5].  These works are primarily focused on improving the reliability in off-the-shelf LLMs, by quantifying existing uncertainty in some answer to a response to some question. This is a fundamentally different task than quantifying how observing the answer to some question will affect your uncertainty in other question/answers pairs, and using these estimates to make adaptive decisions to reduce uncertainty.\n\nWith respect to Uncertainty of Thoughts (UoT), we did not compare as a baseline because UoT uses the LLM to *generate* potential questions, while our setting involves choosing potential questions from a given question bank. If we adapt UoT to our setting, it would be equivalent to applying the EIG measure to the base LLM. We compare this in the ablation in Figure 6, and show that adapted UoT (EIG + base LLM) performs worse than random selection. We hypothesize that this worse performance is due to the specialized nature of datasets like OpinionQA and EEDI which may not be in the training set of the base LLM, pointing to the necessity of our meta-learning procedure. \n\n- [1] Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation\thttps://arxiv.org/abs/2302.09664\n- [2] Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models\thttps://arxiv.org/abs/2307.01379\n- [3] Language Models (Mostly) Know What They Know\thttps://arxiv.org/abs/2207.05221\n- [4] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback\thttps://arxiv.org/abs/2305.14975\n- [5] INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection\thttps://arxiv.org/abs/2402.03744", "claims_and_evidence": "The claim that the adaptive query selection strategy is better than a random selection strategy is supported by results in Figure 3 as well as Figure 5.\n\nThe claim that the meta-learning method effectively learns to model uncertainty is supported by Figure 4 where the ECE is clearly very low. It’s unclear though how exactly the confidence is calculated in this case. It is stated in Section 2.3 that the “variability in these simulated futures” is treated as a measure of uncertainty, but it is not mentioned anywhere what measure of variability is used and how uncertainty is converted to confidence. I believe some notion of entropy is used, but more details about how the entropy is computed would help clarify this point.\n\nThe claim that their method works for general latent information elicitation settings is supported by their experiments across three diverse applications.", "methods_and_evaluation_criteria": "The proposed methods and evaluation metrics make sense.", "theoretical_claims": "I looked over the proof of Theorem A.4 which bounds the performance gap of the greedy approach and an optimal planning method. The proof is relatively straightforward, although I did not look into all of the details.", "experimental_designs_or_analyses": "The experiment in Figure 3 and the analysis for how well the proposed method can perform adaptive elicitation is valid. The examination of calibration to determine effectiveness of their measure of uncertainty is also a valid approach. Their further analysis on how well their method performs on the most challenging questions is very valuable and provides useful insight into where their approach works best. Finally, the ablation on the underlying model in their framework is a valid way to show that the meta-training method is vital to their method’s success.", "relation_to_broader_scientific_literature": "The method presented in this paper for fine-tuning an LLM for the task of meta-learning next action prediction has been studied in prior work in reinforcement learning, but this paper uses the idea for LLM uncertainty quantification. The problem setting in this paper has also been studied in prior work such as Uncertainty of Thoughts which similarly tries to determine the best questions to ask to elicit the most information, but this paper shows that fine-tuning an LLM for meta-learning is actually useful for this setting.", "essential_references_not_discussed": "The Uncertainty of Thoughts paper is mentioned in the related work, but it is unclear why it is not directly compared against in any of the experiments.", "other_strengths_and_weaknesses": "Other strengths:\n- The problem considered is well motivated as highly significant.\n- The experiments and results are presented in a clear manner.\n\nOther weaknesses:\n- The experiments seem to be lacking on baselines. It is well supported that the proposed LLM fine-tuning method is useful for uncertainty quantification, and the selection method is useful for the question selection task, but it is unclear how this approach fairs compared to the other methods which address these problems.\n- A discussion on the computational expense of the method would be helpful since it seems like if there are thousands of candidate questions, then even the greedy method could be very slow.", "other_comments_or_suggestions": "- Line 36 left: The sentence is either missing punctuation or generally does not make sense.\n- Line 402 right: “setti” -> “setting”\n- Line 652 is missing a closing parenthesis.", "questions_for_authors": "1. Why are existing LLM uncertainty quantification methods not comparable baselines?\n2. How is confidence, as shown in Figure 4, calculated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new framework for eliciting latent information using specifically fine-tuned LLMs, which can be used to reduce uncertainty in various follow-up applications. The reviewers agree that the studied problem is novel and important, and the proposed approach is well supported. During the rebuttal period, the authors addressed most of the concerns. Overall, I think this paper would be a good addition to the community.
Decision: Accept (poster)