ICML 2025

Adaptive Elicitation of Latent Information Using Natural Language
Jimmy Wang * 1 Thomas Zollo * 1 Richard Zemel 1 Hongseok Namkoong 1
Abstract
Eliciting information to reduce uncertainty about
a latent entity is a critical task in many appli-
cation domains, e.g., assessing individual stu-
dent learning outcomes, diagnosing underlying
diseases, or learning user preferences. Though
natural language is a powerful medium for this
purpose, large language models (LLMs) and ex-
isting fine-tuning algorithms lack mechanisms for
strategically gathering information to refine their
own understanding of the latent entity. To har-
ness the generalization power and world knowl-
edge of LLMs in developing effective informa-
tion gathering strategies, we propose an adaptive
elicitation framework that actively reduces un-
certainty on the latent entity. Since probabilistic
modeling of an abstract latent entity is difficult,
our framework adopts a predictive view of un-
certainty, using a meta-learned language model
to simulate future observations and enable scal-
able uncertainty quantification over complex nat-
ural language. Through autoregressive forward
simulation, our model quantifies how new ques-
tions reduce epistemic uncertainty, enabling the
development of sophisticated information gather-
ing strategies to choose the most informative next
queries. In experiments on the twenty questions
game, dynamic opinion polling, and adaptive stu-
dent assessment, our method consistently outper-
forms baselines in identifying critical unknowns
and improving downstream predictions, illustrat-
ing the promise of strategic information gathering
in natural language settings.
1. Introduction
The performance of many valuable services and systems
depends on the ability to efficiently elicit information and
1Columbia University.
Correspondence to:
Jimmy
Wang
<jw4209@columbia.edu>,
Thomas
Zollo
<tpz2105@columbia.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vanco...