Introduction
Any practical Question Answering (QA) system must be able to assess its own confidence so that it can (1) avoid making up incorrect, incomplete, or misleading answers, and (2) request clarification to resolve ambiguity arising from unclear questions or missing context.
In recognition of this desirable property, recent work has addressed confidence estimation of Large Language Models (LLMs) by evaluating QA tasks in unambiguous settings Jiang et al. ( 2021 ); Kadavath et al. ( 2022 ) . In these experiments, questions are posed to the model with multiple-choice answers so the probability of each answer can be computed.
Here we extend the study of confidence to practical scenarios, requiring free-form text answers to arbitrary questions which may be underspecified or ambiguous.

We describe a range of experiments to help disentangle uncertainty about the world from uncertainty about the question. While we focus on the question answering capabilities of large pretrained language models, these two forms of uncertainty can be more clearly distinguished in an idealized scenario in which a symbolic question-answering system first converts the question into a formal denotation (such as an SQL query or lambda expression), which is then applied to a knowledge base (e.g., Zelle and Mooney, 1996 ) . The system may be uncertain about the meaning of the question — denotational uncertainty — if the question is underspecified because of assumed background knowledge or context. For any given denotation, the system may also be uncertain about the correct answer — epistemic uncertainty — if the knowledge base is incomplete 1 1 1 Gruber et al. ( 2023 ) present an overview of uncertainty in Machine Learning, making a related distinction between epistemic and aleatoric uncertainty. We do not address aleatoric uncertainty in this work. . In Figure 1 , denotational uncertainty is shown in the upper fork and epistemic uncertainty is shown in the lower set of forks.

The most competitive systems today do not construct explicit denotations of questions. As a result, it can be difficult to clearly separate these two forms of uncertainty. Nonetheless, our general approach is to attempt to first approximately disambiguate the question (in natural language) and then selectively answer user questions with sufficiently high confidence.
This allows the model to more effectively represent ambiguous user inputs.
This scheme is represented in Figure 1 .
Further, we argue that repeated sampling within our disambiguate-then-answer framework provides reliable confidence estimates of the model.

We summarize our contributions as follows:

We reframe the discussion of model confidence with respect to denotational and epistemic uncertainty and address each in turn. Our experiments suggest that the answer probabilities given by current LLMs are somewhat helpful for assessing epistemic uncertainty but cannot detect denotational uncertainty.

We present two simple and effective approaches for measuring confidence under all types of uncertainty, based on answer counts within repeated samples from the LLM. These approaches are better calibrated than the model’s answer probabilities and a self-verification prompt, and are especially effective for ambiguous questions and for instruction-tuned model variants.