Title: Conformal Information Pursuit for Interactively Guiding Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Interactive Question Answering

Uncertainty Quantification

Twenty Questions
Abstract: A significant use case of instruction-finetuned Large Language Models (LLMs) is to solve question-answering tasks interactively. In this setting, an LLM agent is tasked with making a prediction by sequentially querying relevant information from the user, as opposed to a single-turn conversation. This paper explores sequential querying strategies that aim to minimize the expected number of queries. One such strategy is Information Pursuit (IP), a greedy algorithm that at each iteration selects the query that maximizes information gain or equivalently minimizes uncertainty. However, obtaining accurate estimates of mutual information or conditional entropy for LLMs is very difficult in practice due to over- or under-confident LLM probabilities, which leads to suboptimal query selection and predictive performance. To better estimate the uncertainty at each iteration, we propose *Conformal Information Pursuit (C-IP)*, an alternative approach to sequential information gain based on conformal prediction sets. More specifically, C-IP leverages a relationship between prediction sets and conditional entropy at each iteration to estimate uncertainty based on the average size of conformal prediction sets. In contrast to conditional entropy, we find that conformal prediction sets are a distribution-free and robust method of measuring uncertainty. Experiments with 20 Questions show that C-IP obtains better predictive performance and shorter query-answer chains compared to previous approaches to IP and uncertainty-based chain-of-thought methods. Furthermore, extending to an interactive medical setting between a doctor and a patient on the MediQ dataset, C-IP achieves competitive performance with direct single-turn prediction while offering greater interpretability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nC-IP provides a mathematically rigorous solution to uncertainty quantification in sequential querying, using conformal prediction sets to address fundamental LLM calibration issues. While offering strong theoretical guarantees and demonstrated performance improvements, the approach has limited scope beyond sequential questioning scenarios and does not directly enhance reasoning quality. The contribution is significant within its domain, with clear potential for broader application and integration with complementary reasoning frameworks.\n\nSTRENGTHS AND WEAKNESSES:\nA. Strengths of this work:\n\nA.1. Rigorous Theoretical Foundation: The work is grounded in conformal prediction theory, providing mathematically principled uncertainty quantification with formal coverage guarantees. Notably, the approach is distribution-free and does not require calibrated probability estimates, addressing a fundamental limitation of traditional entropy-based methods.\n\nA.2 Demonstrated Practical Effectiveness: C-IP shows consistent performance improvements across multiple domains (20 Questions, medical QA), with clear mathematical stopping criteria and computational efficiency for discrete classification problems. The method provides robust uncertainty estimates even when LLM probabilities are miscalibrated.\n\nB. Weaknesses of this work:\n\nB.1 Limited Scope and Applicability:\n\nNarrow focus on sequential query scenarios limits broader applicability\nPrimarily designed for discrete classification problems\nHeavy dependence on query set quality and availability\nNo mechanisms for improving reasoning quality or argument coherence\n\nB.2 Fundamental Methodological Gaps:\n\nDoes not address core LLM limitations (bias, hallucination, error propagation)\nStatic approach lacks adaptive mechanisms for evolving problem complexity\nLimited to uncertainty quantification without enhancing underlying reasoning processes\n\nOverall Assessment: C-IP makes a significant contribution to uncertainty quantification in sequential querying through rigorous mathematical foundations. However, its impact could be substantially enhanced through integration with complementary reasoning frameworks like EVINCE [1], which addresses the broader challenges of LLM reliability and reasoning quality that C-IP does not tackle.\n\n[1] Optimizing Multi-LLM Dialogues Through Information-Theoretic Control, E. Chang, August 2024.\nhttps://www.researchgate.net/publication/383425368_Optimizing_Multi-LLM_Dialogues_Through_Information-Theoretic_Control\n\nQUESTIONS:\nGood work.  A couple of questions...\n\n1. Your method's effectiveness appears heavily dependent on the quality and comprehensiveness of the available query set Q. In real-world scenarios where optimal queries may not be pre-known or where the query space is naturally limited, how robust is C-IP to suboptimal query sets? Could you provide guidance on when practitioners should expect the method to fail or underperform?\n\n2. While C-IP effectively addresses uncertainty quantification, it doesn't directly improve the quality of reasoning or handle LLM hallucinations. Given the demonstrated complementarity with approaches like EVINCE that focus on reasoning quality, have you considered how C-IP could be integrated into broader multi-LLM reasoning frameworks? What would be the technical challenges and potential benefits of such integration?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper an Information Pursuit strategy is applied to settings where LLMs can interactively query the user to make a prediction, e.g. in a medical diagnosis setting. Such strategies typically maximize the information gain (or equivalently the conditional entropy) of each query. While LLMs output probabilities over their outcome, these probabilities are known to be miscalibrated and therefore do not serve as a reliable measure of uncertainty need to estimates the conditional entropy. The paper therefore proposes to leverage the upper bound on the entropy base on the cardinality of prediction sets (see Correira et al.), in order to circumvent the miscalibration issue.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n* the approach is theoretically sound as it is based on a mathematical relation between the entropy and the cardinality of prediction sets.\n* reducing the number of questions in such interactive LLM-settings is an important research aim and the suggested approach is as far as I know an original combination of the Information Pursuit, Prediction sets and LLMs. \n* clear motivation and easy-to-follow introduction of the background material on Information pursuit and conformal prediction.\n\n**Weaknesses:**\n*  it is not fully clear to me if there are performance gains in the application setting in Section 5 or not\n\n**Minor comment:**\n* The legend in Figure 4 is very small and difficult to read.\n* It is fairly uncommon to have the Related Work Section in the Appendix only. Usually explaining how the work builds upon the existing literature is an integral part of a paper.\n\nQUESTIONS:\n* For the results in Figure 2: How is \"uncertainty\" defined and measured?\n---\n\n**An explanation regarding the following two questions would likely make me increase my score:**\n\n---\n* Since the LLM predictions are miscalibrated, the sampled of histories from LLM simulations should also be biased and therefore the predictions sets, too, might be biased. Is it some kind of \"averaging out\" of biases that makes the construction work well in the end nevertheless?\n---\n* Section 5.2 says:\n> The performance of C-IP and the baselines for the three specialities is shown in Figure 4. C-IP consistently obtains higher test accuracy at each iteration than IP for IM and P while achieving comparable performance for N. \n\nIn Figure 4 on the **IM dataset**, to me it looks like that C-IP has lower accuracy for 1,2,5,6 and 7 iterations and higher accuracy for 3,4, 9 and 10 iterations. On the **P dataset**, it looks like C-IP performs better for 2,3,4,7,9 and 10 iterations and worse for 1,5,6 and 8 iterations. Am I misreading Figure 4? Otherwise, I would suggest weakening the statement \"consistently obtains higher test accuracy\" to \"achieves comparable performance\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper introduces conformal information pursuit, which extends the information pursuit framework for guiding LLMs in sequential question selection. Specifically, C-IP uses conformal prediction sets to estimate uncertainty in a distribution-free manner for subsequent query selection. The method is evaluated on the 20 Questions and MediQ datasets.

STRENGTHS AND WEAKNESSES:
Strengths

- The proposed framework is easy to implement.

- The study is well-motivated by the open-ended nature of queries and answers in conversational settings.

Weaknesses

1. Lack of Theoretical Contribution

- The study relies heavily on theoretical analysis from prior work [18, 64], with limited novel theoretical development.

2. Insufficient Theoretical Justification for History Sampling and Multi-Round Settings

- The construction of prediction sets across dialogue histories involves marginalization steps that lack theoretical grounding.

- Additionally, the paper proposes two potential methods for history sampling, but neither is supported by formal theoretical analysis.

3. Limited Baselines and Marginal Improvements

- There is no comparison to learned query policies, calibration-based methods, or reinforcement learning-based strategies.

- The performance improvements appear modest and primarily limited to baseline comparisons, with gains mostly in early stages.

4. Why Not Calibrate the LLM Instead?

- The paper assumes that LLM outputs are too miscalibrated for entropy-based selection, but makes no effort to calibrate the LLM’s own confidence scores.

5. Similarity to Prior Work

- The proposed method appears highly similar to [64].

QUESTIONS:
See above.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at interactive multi-turn dialogues where an LLM is able to ask questions and obtain information before giving a final prediction (e.g., as in the 20 Questions game). The strategy the authors pursue is motivated by \"information pursuit\" (or choosing to ask queries that maximize information gain at each step), but via the use of conformal prediction sets instead of measures of conditional entropy that rely on estimated probabilities from the LLM (which are unreliable). Specifically, the approach is to minimize prediction set size at each iteration --- which the authors connect to an upper bound on the standard entropy. Empirically, the authors show good performance on 20Q + MediQ at getting good predictive accuracy within a small number of iterations.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written, and the motivation (reducing uncertainty set size) is well-motivated. The experiments also indicate that the proposed method is effective. However, I also have a number of questions (see below), and at a high level, while I buy that a measure that is less susceptible to estimation error like conditional entropy is, I am not fully convinced of the need for _conformal_ prediction and calibration. Specifically, the primary advantage of conformal prediction---the formal coverage guarantee---is not a central feature of the algorithm's goal, which is to minimize uncertainty for query selection, not to produce prediction sets with valid coverage (and even so, the benefits of the marginal conformal guarantees are somewhat dubious here --- which is made even more marginal here due to tractability issues). For me, this raises a bit of a question of whether the complexity of the conformal framework is justified over simpler, alternative uses of the calibration data (which does not seem to be used for the baselines).\n\nMore concretely, the proposed C-IP method uses the average size of a conformal prediction set as a proxy for uncertainty. The size of the conformal sets also depend on the LLM probabilities, but in a more discretized way, i.e., the count of how many outputs have confidence > $\\tau$. $\\alpha$ doesn't seem to be used in any important way here, so searching over $\\alpha$ hyper-parameters I believe could also be viewed as just measuring the average number of labels per X that are above some confidence threshold. I would be curious to compare this to other, simple baselines that use metrics like this --- like the average top-label confidence, the average difference between the top label's confidence and the second-to-top label's confidence, or if the confidence score is noisy, the average accuracy of the top-label (so, ranking queries by their conditional accuracy).\n\nQUESTIONS:\n- I'm a bit confused by L169, where the claim is made the the uncertainty of Y given data X is upper bounded by the expected size of the prediction set. I think the more precise claim is + some additional constants that depend on $\\alpha$ and $|\\mathcal{Y}|$ (which, looks like it can be quite large, unless $\\alpha$ is near $0$ --- for instance, for $\\alpha$ near 1 this bound is vacuous, as it bounds the conditional entropy by the entropy of a uniform distribution over $\\mathcal{Y}$). It's also weird to me to related the conditional entropy to the expected set size, marginal over $X$ --- this is then a bound that is no longer dependent on the input example?\n\n- I'm also a bit unclear on how $E_X[ \\mathcal{C}(q(X)) \\vert \\vert \\mathcal{H}]$ is computed. Are the same set of query trajectories applied to the calibration set for all $X$? In the open query set, what happens when $q$ does not apply to some $X_i \\in \\mathcal{D}_\\mathrm{cal}$?\n\n- The calibration set is used quite extensively for the CP algorithm. Have you tried also using it for the baselines, e.g., to calibrate the confidence of the LLMs by histogram binning or platt scaling?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}