Measuring LLM Confidence through Stable Explanations

## Abstract

Abstract In many critical machine learning (ML) applications it is essential for a model to indicate when it is uncertain about a prediction. While large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, their overconfidence in incorrect responses is still a well-documented failure mode. Traditional methods for ML uncertainty quantification can be difficult to directly adapt to LLMs due to the computational cost of implementation and closed-source nature of many models. A variety of black-box methods have recently been proposed, but these often rely on heuristics such as self-verbalized confidence. We instead propose a framework for measuring an LLM’s uncertainty with respect to the distribution of generated explanations for an answer. While utilizing explanations is not a new idea in and of itself, by interpreting each possible model+explanation pair as a test-time classifier we can calculate a posterior answer distribution over the most likely of these classifiers. We demonstrate how a specific instance of this framework using explanation entailment as our classifier likelihood improves confidence score metrics (in particular AURC and AUROC) over baselines across five different datasets. We believe these results indicate that our framework is a promising way of quantifying uncertainty in LLMs.

## 1 Introduction

Large language models (LLMs) are known to at times confidently provide wrong answers, which can greatly mislead non-expert users of the model [, ]. In the some cases an LLM may even ‘hallucinate’ facts all together [, ]. Although scaling generally improves factual accuracy, past work has shown that even the largest models can give incorrect answers to certain types of questions [].

To prevent these misleading scenarios, one intuitive approach is to have the model also report its confidence (or uncertainty) in the accuracy of its own response. This task, known as *uncertainty quantification*, has a vast associated literature [, ]. In its most naive form, this can entail taking the softmax of prediction logits to calculate a ‘distribution’ over answers. However in most cases there is no guarantee that this metric should correspond to the actual probability of correctness on a new datum. Empirically this mismatch has been demonstrated for LLM token logits [, ].

One might instead hope that by probing the model (e.g. through its weights or activations) one could infer a measure of confidence that somehow aligns with our expectations. However, full access to a large language model is often infeasible due to a combination of proprietary restrictions and computational expense. Recently a range of ‘black-box’ approaches have been proposed that avoid the need for access to internal model information [, , ]. These approaches typically rely on custom prompting strategies to elicit self-verbalized (linguistic) confidence or generate multiple variations of a response (consistency). While empirically promising, these methods are heuristic and still return overconfident responses in many cases.

We reason that the main issue with existing uncertainty quantification methods for LLMs stems from the underlying inductive assumption that test and training data are sampled from the *same distribution*. Unfortunately, this is rarely the case, meaning any uncertainty quantification strategy that is well-calibrated on one dataset is not guaranteed to be calibrated on new test data. However, an LLM offers a unique opportunity to adjust its decision boundary transductively at test-time via intermediate generated text (explanations). While inserting random text would likely lead to a high-entropy decision distribution, adding relevant facts or logical step-by-step reasoning serves to ‘stabilize’ the sampled answers around an isolated minimum.
Indeed, prompts inducing chain of thought (CoT) reasoning have already shown to improve model accuracy in this manner [, ]. However, more recent work has shown that even CoT explanations can be biased and may not correspond with the correct answer []. If we could somehow distinguish between ‘stable’ and ‘unstable’ explanations then we would know to what extent to trust their corresponding answer distributions.

In this work we propose a method for generating confidence scores from the distribution of LLM-generated explanations for an answer. This method, which we call stable explanations confidence, can be thought of as computing the posterior predictive distribution by transductive marginalization over test-time classifiers. We illustrate the usefulness of these scores on two common uncertainty quantification tasks: calibration, in which we measure how close confidence is to empirical accuracy, and selective uncertainty, in which we determine how well the scores can discriminate between correct and incorrect predictions. We compare to other recently proposed methods across five datasets of different scope and complexity (CommonsenseQA, TruthfulQA, MedQA, MMLU Professional Law, MMLU Conceptual Physics) using two popular LLMs (GPT-3.5 [] and GPT4 []). We find that our method on average outperforms baselines on the selective uncertainty task (measured via AUROC and AURC), particularly for more complex question-answering problems.

## 2 Related Work

In this section we first summarize the uncertainty quantification problem in machine learning. We then highlight key challenges in the natural language generation setting and the ‘confidence gap’ of existing LLM models. Lastly we discuss exisiting approaches for LLM uncertainty quantification and methods for their evaluation.

### 2.1 Uncertainty Quantification in Machine Learning

Defining and reasoning about uncertainty has been a long-standing problem in different disciplines including philosophy, statistics, and economics.
Many formal representations with unique properties have been proposed, (e.g. Dempster-Shafer belief functions, ranking functions, etc. []), but in the machine learning setting uncertainty quantification typically relies on the standard language of probability measures. For a classification task we can think of the sequential training data-label pairs $\mathcal{D}:=\{(x_{i},y_{i})\}_{i=1}^{N}$ as the model’s source of knowledge about the world. Given some test datum $x_{N+1}$, we would like the model to both make a prediction $\hat{y}_{N+1}$ and provide a ‘useful’ confidence score $r_{N+1}\in[0,1]$. Useful confidence scores should allow models to express their belief in the accuracy of a prediction, and is called well-calibrated if on average predictions with confidence $r=0.XX$ are correct close to $XX\%$ of the time. If the classification task also specifies cases for which it is better to return no prediction than a wrong one, we can imagine creating some selection rule using confidence scores to determine whether to trust the classifier’s prediction. We will formalize these two related goals later when discussing evaluation metrics in Section˜4.1.

Uncertainty quantification methods differ from one another based on their assumptions about where uncertainty is coming from. Sources of uncertainty are traditionally categorized into two broad classes: *epistemic* uncertainty arising from the agent’s incomplete knowledge of the world, and *aleatoric* uncertainty inherent to the data generating process (e.g. the flip of a coin). In reality, definitions vary among the machine learning community [] and most methods do not fit neatly into either category.
In this work we discuss a few of most common methods based on the underlying assumptions placed on the *test data*. We make this distinction because without this fundamental assumption it is impossible to know *anything* about the test distribution from training data. Note that for a full discussion and taxonomy of the numerous uncertainty quantification methods in machine learning we refer to a suvery paper such as [, ].

#### Related Training and Test Worlds.

Most uncertainty quantification methods rely on the fundamental assumption that the test data comes from the same distribution as the training set. Under this type of assumption Bayesian approaches such as Bayesian Neural Networks (BNNs) are popular. BNNs measure epistemic uncertainty through a posterior on the learned weights, which can be reduced as more data is recieved [, ]. Another popular method is that of conformal prediction, which introduces a somewhat dual notion of the conformal set. Under a slightly weaker exchangibility assumption (i.e. that the joint distribution remains the same under permutations of the training and test data), the conformal set of predictions is guaranteed to contain the true label with error probability less than some $\epsilon$ []. Weaker predictive models result in larger conformal sets, and so set size can be taken as an indicator for higher model uncertainty. Other methods include looking at the robustness of predictions under semantic-preserving transformations of the input, as mentioned in [].

#### Different Training and Test Worlds.

Small and large differences between training and test distributions are typically denoted as *distribution shift* and *out-of-distribution* respectively []. In this setting methods like prior networks attempt to capture the specific notion of this distributional uncertainty through and additional prior over predictive distributions and training explicitly on a loss objective [].

### 2.2 Uncertainty Quantification in LLMs

Recently much attention has been devoted to measuring uncertainty specifically in LLMs [, ]. Since LLMs are generative models, uncertainty may be measured with respect to an infinite set of text sequences as opposed to a fixed number of classification labels []. Many works, however, use multiple choice question answering tasks to evaluate LLMs using standard classification methodologies [, ], and we will follow a similar approach in this work.
Issues with using token logits directly to compute confidence are well known. Recent works [, , ] show that larger models are typically better calibrated on multiple choice datasets than smaller ones, but are still sensitive to question reformulations as well as typical RLHF training strategies. Another recent work [] notes that language models fail to identify unanswerable questions at a higher rate than humans.

At a high level, existing techniques for LLM confidence elicitation can be classified as either white-box, requiring access to internal model weights and token probabilities, or black-box, using only samples from the model []. We choose to summarize inference time interventions below, as training time interventions are often computationally expensive and require strict inductive assumptions.

White-box Methods. Access to the last activation layer of the LLM (token logits) admits calculating token and token sequence probabilities via the softmax function. One can incorporate text sequence probabilities to implement conformal prediction [] methods, or adjust them based on semantic importance of individual tokens to improve calibration []. Surrogate models can also serve as an effective substitute if access the original model is restricted-access []. Internal activations can also be observed to determine if certain feature directions are more or less truthful [, ].

Black-box Methods. Black-box confidence typically uses one or both of the following approaches: Sample+aggregate methods involve analyzing the distributions of multiple responses sampled from the model []. Responses can be generated in a variety of ways, such as using chain-of-thought prompting [], asking for multiple answers in a single response [], or perturbing the question in-between samples []. Confidence can be found by observing the frequency with which answers occur, or by averaging over other metrics []. Self-evaluation methods use customized prompts in order for the model to generate its own confidence estimates in natural language []. These methods can also be augmented with chain-of-thought or other more complex reasoning steps []. Much effort has been put into analyzing how changes in prompt (e.g. by including few-shot examples) affects these confidences [, ].

## 3 Stable Explanations

Given a question, we would like to assign a confidence value to an answer based on how plausible its associated explanations are. Intuitively, humans are confident in an answer when likely explanations exist for it and no other answers have reasonable explanations. However, the space of explanations (variable-length token sequences) is infinite and hard to work with directly. To overcome this, we will first approximate this distribution by sampling a set of explanations from the LLM conditioned on the question, and then reweight based on their logical consistency with the question desscrption. Afterwards we can compute the degree to which explanations support each answer.
We can view these two steps as estimating the conditional likelihood of the explanation given the question, and the conditional answer distribution of the test-time model parameterized by this explanation. These two components will allow us to compute a posterior predictive distribution in a Bayesian fashion. We formalize each step in the following subsections, and summarize the complete method in Algorithm˜1.

Figure: Algorithm 1 Stable Explanation Confidence Calculation

#### Preliminaries.

Consider a multiple choice question $q:=\{x_{1},\dots,x_{t}\}=x^{t}$ consisting of a sequence of tokens in some alphabet $x_{j}\in\mathcal{A}$, and a set of possible answers $a\in S\subseteq\mathcal{A}$ which are also some subset of tokens in the same alphabet. We will designate $\phi$ as an LLM, which will take any variable length token sequence as input and output a token logit vector of size $|\mathcal{A}|$. We use $\phi(s_{1},s_{2})$ to denote the direct concatenation of two token sequences in the LLM input, and $\phi(\text{prompt}(s))$ to denote adding prompt instructions to the input. Lasrly, $s\sim\phi$ will be used to denote sampling a token sequence from the LLM.

### 3.1 Answer Likelihood Conditioned on Explanations

In its default configuration, providing a question to an LLM $\phi$ without further input can be used to find an answer:

$$ $\displaystyle~\underset{S}{\rm argmax}~{\phi(q,\{~\})}=a$ (1) $$

One can also naively compute a ‘probability distribution’ over possible answers by taking the softmax of token logits produced by the model. We will denote this calculation as

$$ $\displaystyle p_{\phi}(a|q):=\text{softmax}(\phi(q,\{~\}))_{i},$ (2) $$

where $i$ denotes the logit index of $a$. However, these default token probabilities have been shown to be miscalibrated and sensitive to variations in the input [, ].
Next, we formally say that explanations, like questions, are also variable length sequences of tokens $e\in\mathcal{A}^{\tau}$ located between question and answer. If the model generates these explanations (like in the chain-of-thought reasoning paradigm []) then the sequences can be thought of as a possible trajectory from the question to an answer. While the set of possible trajectories is infinite, we can group explanations into equivalence classes by noting that two semantically identical explanations must support the same answers [, ]. This notion leads us to the following idea: characterize the distribution of explanations by looking at the *new answers* they lead to.

$$ $\displaystyle~\underset{S}{\rm argmax}~{\phi(q,e)}=a^{\prime}$ (3) $$

This idea is related to the semantic entropy method of [], but here we use the next token distribution $p_{\phi}(a|q,e)$ instead of a pairwise explanation similarity to ‘cluster’ explanations. If we can enumerate all likely explanations, we can calculate the posterior answer probability as follows

$$ $\displaystyle\hat{p}(a|q)=\sum_{e}p_{\phi}(a|e,q)p(e|q)$ (4) $$

A key detail omitted so far is how to efficiently approximate the distribution of all ‘stable’ explanations. We will see in the following subsection that this can be achieved using only the LLM $\phi$.

### 3.2 Determining Likely Explanations

A naive method for estimating $\hat{p}(e|q)$ would be to sample explanations using a modified prompt (e.g. using a CoT ‘think step-by-step’ approach). Indeed, a number of consistency-based question-answering methods work by sampling and then aggregating explanations and answers in this manner [, ].
However, due to the way LLMs are trained, this distribution does not necessarily represent the probability that an explanation *actually explains* the data in the question [, ].
To combat this, we enforce logical consistency by checking the entailment probability of our sampled explanations ($q\models e$), which can be approximated by using the LLM and a modified prompt $\phi_{entail}(q,e)$ []. We then reweight sampled explanations using this entailment probability:

$$ $\hat{p}(e|q):=\frac{\phi_{ent.}(q,e)}{\sum_{e^{\prime}\in E}\phi_{ent.}(q,e^{\prime})}$ (5) $$

We reason that enforcing logical structure prevents trusting explanations that ‘overfit’ to the test datum. For example while an explanation such as ‘the answer is always (a)’ is syntactically correct and may result in a confidently correct answer for our test question, it would prove a useless classifier on previous training data.
While we use entailment probability in our main results, an exploration of alternative explanation plausibility calculations can be found in Section˜B.4.

## 4 Experiments

To gain insight into the usefulness of LLM-sampled explanations we first examine differences in distributions of explanations conditioned on correct vs. incorrect answers (see Figure˜1) and find explanation entailment (Section˜3.2) can help distinguish between the two.
We then conduct a series of experiments to compare our proposed stable explanation confidence (Algorithm˜1) with exisiting approaches across a set of five benchmark datasets and discuss our findings below.

### 4.1 Setup

#### Evaluation Method.

How do we know whether a proposed confidence metric is useful or not?
In line with previous works [, , , ] there are typically two tasks that uncertainty metrics are evaluated on. The first is confidence calibration, where the goal is to produce confidence scores approximating the empirical probability that the model answers the question correctly. Expected calibration error (ECE) [] attempts to estimate this using differences between the average confidence and accuracy for a group of similarly scored answers, however ECE can be misleading (see Section˜5). We still include this metric in our reports for ease of comparison with previous work. The second related task is typically called selective uncertainty (also known as failure prediction). Here the goal is to create a binary classifier using confidence scores that predict when the model should return ‘I don’t know’ instead of its original prediction. A variety of classifier metrics can be used, depending on how one chooses to penalize false positive (overconfident) and false negative (underconfident) predictions. In this work we use two of the most common metrics: area under the reciever-operator curve (AUROC) [], and area under the risk-coverage curve (AURC)[]. Uninformative (i.e. completely random) confidence scores will have a worst-case AUROC of $0.5$ and an worst-case AURC equal to average model accuracy. The best possible value for both AUROC and AURC is $1.0$. We include formal definitions for each of these metrics in Appendix˜A.

#### Datasets and Models.

We evaluate our method using five standard question answering datasets covering a variety of reasoning tasks: CommonsenseQA (CSQA) [], TruthfulQA [], MedQA [], MMLU Professional Law, and MMLU Conceptual Physics []. Besides covering a range of topics, these datasets also vary largely in their complexity. As seen in Table˜1, the average length of an MMLU law question is almost ten times that of the average CSQA question. Shorter questions typically resemble more traditional classification tasks (e.g. ‘Something that has a long and sharp blade is a? ’ from CSQA), while longer questions typically include descriptions of a specific scenario that require more complex reasoning. We test both methods and baselines on snapshots of two state-of-the-art models GPT-3.5-turbo [] and GPT-4-turbo []. Further data and model details can be found in Appendix˜B.

#### Compared Metrics.

We use four different baselines for comparison purposes. Token probabilities for each answer can be produced by taking the softmax over the models logit vector and are one of the most commonly used confidence metrics during model evaluation [, ]. Linguistic and Top-k methods both ask the model for a verbalized confidence estimate directly, the former prompting the model for a single answer and confidence estimate while the later asks for the $k$-best guesses and associated confidences [, ]. Lastly the sef-consistency method samples multiple responses from the model and approximates confidence via the relative frequency of parsed answers. Here we use a particular variant of this method, CoT-Consistency [], which uses a zero-shot chain-of-thought prompt to generate responses, and which has been shown to outperform the vanilla method []. We use the similar prompts to those selected in previous work for comparison purposes, the details of which can be found in Section˜B.1.

**Table 1: Average question length and accuracy for each of the datasets tested in this work. One can observe a weak correlation between question length and difficulty, as typically longer questions describe more complex scenarios and logical structure.**
| Dataset | Avg. Question Length (# Chars) | GPT-3.5 Accuracy | GPT-4 Accuracy |
| --- | --- | --- | --- |
| CSQA | 151 | 0.79 | 0.84 |
| TruthQA | 329 | 0.54 | 0.85 |
| MedQA | 916 | 0.59 | 0.82 |
| MMLU Law | 1233 | 0.46 | 0.64 |
| MMLU Physics | 172 | 0.57 | 0.92 |

### 4.2 Likely Explanations Not Always Correct

We first illustrate how explanation likelihood, as measured via conditional token log probability, does not always correspond with the correctness of the supported answer. These results align with previous findings differentiating syntactically vs. semantically correct model responses [, ], and help us to motivate using entailment probability in our method. First recall that the length-normalized conditional log-likelihood for sequence $x^{t}$ given sequence $s$ is defined as

$$ $\displaystyle LL(x^{t}|s):=\frac{1}{t}\sum_{i=1}^{t}\log(P_{\phi}(x_{i}|s,x_{1},x_{2},\dots,x_{i-1})),$ (6) $$

which can also be thought of as the average token logit value. Higher log-likelihood of explanations should mean higher chance of being sampled by the LLM. We can observe in Figure˜1 two distributions of explanations: one set (in blue) is conditioned on answers we know a priori are correct, the second set (in red) is conditioned on incorrect responses. The model prompt for each set is the same and is given in Section˜B.1. We see that while the mean log-likelihood for correct explanations is slightly higher than that of incorrect explanations, the two distributions are hard to distinguish. In contrast there is clearly a distinct tail for the distribution of incorrect explanations measured via entailment probability. This result suggests that we may be able to discount certain explanations sampled by the LLM but that are well written but logically ‘unstable’, hence improving our confidence score.

Figure: Figure 1: Empirical distribution of explanation log likelihoods (top left) and explanation entailment probabilities (top right) generated for the TruthQA dataset using token logits from GPT3.5-Turbo. Red denotes explanations generated by conditioning on the incorrect answer and blue denotes explanations justifying the correct answer. While mean likelihood for the two explanation distributions are different, there is significant overlap. In contrast the tail of the incorrect explanation distribution is distinct when using entailment probability. The example explanation (lower) suggests we can use this entailment measure to distinguish semantically unlikely explanations in cases where likelihood fails.
Refer to caption: x1.png

### 4.3 Stable Confidence Improves Selective Uncertainty

For each dataset we evaluate our stability method using both a simple explanation prompt and explicit chain-of-thought explanation thought (‘think step by step’) inspired by [] (see Section˜B.1).
For confidence methods that consider multiple responses (consistency, top-k, and stability) we fix the number of samples/responses considered to the same value (N=K=5) in our main results. We further analyze the effect of changing sample size in Appendix˜B.

When testing on the GPT-3.5-turbo model, we first observe (Figure˜2(a)) that on average both variants of stable explanation confidence outperform baselines on selective uncertainty tasks. Average AURC is 0.784 vs. next best of 0.761, while average AUROC is 0.802 vs. 0.789. Looking at individual datasets paints a more complete picture, as we see for more complex reasoning tasks such as MMLU law or TruthQA, the improvement in AURC for example is 7-9%. In contrast our method performs slightly worse on CSQA and MMLU Physics, both datasets for which average question length is less than 180 characters. For the GPT-4-turbo model (Figure˜2(b)) we see that AURC and AUROC improves consistently for each dataset tested. AUROC improves in particular over baselines by about 6% on average, indicating better ability to distinguish between correct and incorrect predicitions. ECE is roughly the same as baselines in this case.

Figure: (a) Confidence Elicitation Strategies on GPT-3.5-turbo.

### 4.4 Ablation Study

We perform an ablation study in an attempt to isolate the effect of the two key components of our stable explanation method. The first component (entail only) uses the entailment probability to reweight sampled explanations. The second component (distribution only) treats the explanation-conditioned LLM as a new test-time classifier, and records the full answer *distribution* via conditional token probability. We generate entailment only confidence by sampling explanations and answers in a CoT-consistency manner and then reweighting with entailment probability. Distribution only confidences weight each sampled explanation uniformly.
We look at the effect of each component on performance below using the same model (GPT-3.5-Turbo) across all datasets. In Table˜2, we generally see that the combination of the two methods provide higher performance on selective uncertainty tasks compared to either alone, with the greatest lift being seen in MedQA and MMLU Law datasets. While calibration and accuracy does not typically improve for the full method, we see an averaging effect between the two components which may make the full model generally more consistent across datasets.

**Table 2: Ablation Study isolating the effects of entailment reweighting and explanation-conditioned answer distributions. Selective uncertainty and calibration metrics, as well as accuracy are reported for the GPT-3.5-Turbo model. Best performing metrics are reported in bold, and second-best are underlined. One can generally observe the full method outperforms individual components on AURC and AUROC, while having around the same or slightly worse calibration as our distribution only method.**
|  | Stability Entail Only |  |  | Stability Distr. Only |  |  | Stability Full |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | Acc. $\uparrow$ | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | Acc. $\uparrow$ | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | Acc. $\uparrow$ |
| CSQA | 0.882 | 0.708 | 0.21 | 0.7 | \ul0.899 | 0.783 | \ul0.131 | \ul0.784 | 0.901 | \ul0.779 | 0.123 | 0.796 |
| TruthQA | 0.739 | 0.818 | 0.19 | 0.668 | \ul0.79 | 0.859 | \ul0.196 | \ul0.656 | 0.801 | \ul0.853 | 0.21 | 0.644 |
| MedQA | \ul0.74 | 0.762 | 0.186 | 0.62 | 0.735 | \ul0.778 | 0.16 | 0.688 | 0.784 | 0.798 | \ul0.169 | \ul0.633 |
| MMLU Law | 0.626 | 0.733 | \ul0.198 | 0.528 | \ul0.655 | \ul0.774 | 0.196 | 0.568 | 0.67 | 0.792 | 0.213 | \ul0.556 |
| MMLU Physics | 0.777 | 0.812 | 0.146 | 0.668 | \ul0.79 | 0.832 | \ul0.164 | 0.723 | 0.792 | 0.834 | 0.186 | \ul0.719 |

## 5 Discussion

In this study, we propose a framework for eliciting confidences from large language models (LLMs) by estimating the distribution of semantically likely explanations, which can be thought of as a set of conditional classifiers. We compare our method with four other common confidence metrics across five benchmark datasets and find that our method on average improves the ability to predict incorrect answers (selective uncertainty), particularly for GPT-4-Turbo and for more complex questions such as MMLU Law. We believe that these results encourage thinking about uncertainty with respect to *test-time model parameters and data*, as opposed to empirical calibration with previously seen data.

#### Alternate Perspectives.

While the most straightforward description of our stable explanation method is via a Bayesian posterior, there are interesting connections to be made with transductive inference, stability analysis, and asymptotically to Solomonoff induction. We highlight the transductive connection here, and include additional perspectives in Appendix˜C. Transductive learning optimizes a classifier at inference-time based on a combination of training and test data, typically by fine-tuning some classifier parameter based on an explicit loss objective [, , ]. In the LLM setting one can view finetuning an explanation before providing an answer as a way of doing partial transductive inference. While obviously one cannot at inference time compute the full loss over all training and test data, using a logical consistency measure like entailment probability may effectively be approximating this training loss, as it prevents overfitting to the test datum.

#### Calibration

With regards to performance of calibration (ECE) task not being at the state-of-the-art, we stress that calibration metrics rely on the inductive hypothesis that training, test, and calibration data are all drawn from the same distribution, which is nether verifiable nor falsifiable at test-time. Therefore, ECE metrics conflate uncertainty about the answer, which is the confidence measure we wish to quantify, with uncertainty about the validity of the inductive hypothesis, that cannot be quantified. Additionally previous work such as [] have demonstrated bias in the metric depending on accuracy and binning strategy. For this reason we indicate the ECE metric in red in the tables, but include the results nonetheless for transparency and ease of comparison.

#### Limitations and Future Work

A notable exception to the observed trend of improved selective uncertainty occurs when making stable confidence predictions on simpler questions (e.g. average question lengths of CSQA and MMLU Conceptual Physics are less than half of others). We hypothesize that when questions resemble classical inductive classification tasks, the advantage of our test-time computation is less evident. Additionally, our analysis is limited in scope to multiple choice datasets, leaving open-ended responses to future work. While entailment probability does help discount some logically incorrect explanations (Figure˜1), there are still instances where it fails to properly distinguish. We test some alternatives to explanation faithfulness in Section˜B.4, but further exploration is needed. Efficiently sampling *high quality* explanations remains an open question as well. Our method adjusts the given explanation distribution based on plausibility, but better explanations may still exist that are not sampled by the LLM. One possible solution could involve using our entailment probability measure as a way to accept or reject incoming samples, increasing complexity but ensuring higher quality.

## Appendix A Evaluation of Uncertainty Metrics

In this section we provide formal definitions for each of the confidence evaluation metrics used. Consider the paired dataset $(x_{i},y_{i})\in\mathcal{D}$ where each datapoint $x_{i}$ has associated label $y_{i}$. Each $y_{i}$ takes on one value in the discrete set $\mathcal{Y}:=\{1,2,\dots,\ell\}$. Now our chosen prediction model $\phi$ outputs a prediction $\hat{y}_{i}:=\phi(x_{i})$ and our confidence function $f$ produces a score $f(x_{i},\hat{y}_{i})=r_{i}\in[0,1]$. We use the indicator variable $c_{i}$ to denote whether the prediction is correct ($c_{i}:=\mathbf{1}(y_{i}=\hat{y}_{i})$). Lastly we define the full sequence of predictions $\hat{Y}$ and confidence predictions $R$ on dataset $\mathcal{D}$ of size $N$ as

$$ $\displaystyle\hat{Y}$ $\displaystyle:=\{\hat{y}_{i}=\phi(x_{i})\mid x_{i}\in\mathcal{D}\}$ (7) $\displaystyle R$ $\displaystyle:=\left\{r_{i}=f(x_{i},\phi(x_{i}))\mid x_{i}\in\mathcal{D}\right\}$ (8) $$

#### Expected Calibration Error (ECE)

To calculate expected calibration error, we first group our data into $M$ partitions based on confidence interval. We denote the set of indices in each partition as:

$$ $\displaystyle B_{m}:=\left\{i\>|\>i\in N,\>\frac{(m-1)}{M}<r_{i}\leq\frac{m}{M}\right\}$ (9) $$

Next, the empirical accuracy and average confidence functions for each partition are defined as

$$ $\displaystyle Acc(B_{m}):=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}c_{i},\quad Conf(B_{m}):=\frac{1}{|B_{m}|}\sum_{i\in B_{m}}r_{i}$ (10) $$

Then the ECE is defined as the following weighted average:

$$ $\displaystyle\text{ECE}(R,\hat{Y},M):=\sum_{m\in M}\frac{|B_{m}|}{M}|Acc(B_{m})-Conf(B_{m})|$ (11) $$

The lower this error is, the better calibrated the model should be (with respect to the data distribution). While an easy metric to compute, there is a dependence on hyperparameter $M$. Another well known issue with ECE is that when accuracy is very high, simply giving a high constant confidence estimate will result in very low calibration error [, ]. Despite these drawbacks, we still choose to report the ECE metric as it is intuitive and serves as a common reference point with previous work.

#### Area Under the Risk-Coverage Curve (AURC)

For now, assume that $r_{i}\neq r_{j}~\forall i\neq j$. Define the subset $R_{\geq r_{i}}$ as

$$ $\displaystyle R_{\geq r_{i}}:=\{r\in R\mid r\geq r_{i}\}$ (12) $$

We now say that the ordering map $\sigma:\{1,\dots,N\}\rightarrow\{1,\dots,N\}$ is the function that returns the dataset index $i$ of the $k$th largest element in $R$. Formally:

$$ $\sigma(k):=i\quad s.t.~|R_{\geq r_{i}}|=k$ (13) $$

To summarize so far, this ordering essentially gives us the dataset index of the $k$th most confident prediction. We can now finally define subsets of our most confident predictions as

$$ $\hat{Y}_{K}:=\{\hat{y}_{\sigma(k)}\mid k\in\{1,\dots,K\}\}$ (14) $$

The risk-coverage curve will measure the tradeoff between the size of $\hat{Y}_{K}$ and the accuracy. For each coverage level $h:=K/N\in[0,1]$, we plot the accuracy $Acc(\hat{Y}_{K})\in[0,1]$ to obtain the curve. Naturally $h=1\implies K=N$ and so the loss is simply the average model accuracy for the entire dataset. If our confidence measure is a good one, we expect higher accuracy when restricting our evaluation to a smaller subset of the most confident answers. Formally, the area under the risc-coverage curve (AURC) is is

$$ $\text{AURC}(R,\hat{Y}):=\sum_{K=1}^{N}Acc(\hat{Y}_{K})\frac{K}{N}$ (15) $$

#### Area Under the Receiver Operator Curve (AUROC)

For any binary classification problem, the receiver operator curve looks at the tradeoff between false positive rate $\alpha$ (plotted on the x-axis) and true positive rate $\beta$ (y-axis), based on retaining only predictions with scores above some threshold $t$. We denote a thresholded set of predictions as $\hat{Y}_{t}:=\{y_{i}\in\mathcal{D}\mid r_{i}>t\}$, and $t_{\alpha}$ as the threshold such that $\text{FP}(\hat{Y}_{t_{\alpha}})=\alpha$. If we have built a perfect classifier of correct and incorrect predictions, there should exist a threshold $t_{0}$ for which $\hat{Y}_{t_{0}}$ contains all of the predictions the model got right and none of which it got wrong. This would correspond to a true positive rate of $\beta=1.0$ for all false positive levels $\alpha\in[0,1]$. Conversely, if confidence metrics were generated at random, any $X_{t}$ is likely to contain just as many false positives and true positives, and so the ROC curve will resemble a diagonal line. Therefore we would like the area under the reciever operator curve to be as closer to 1 as possible. Formally, this area is written as

$$ $\text{AUROC}(R,\hat{Y}):=\int_{0}^{1}\text{TP}(\hat{Y}_{t_{\alpha}})d\alpha,$ (16) $$

## Appendix B Experimental Details

In this section we discuss the implementation details of LLM prompts, dataset characteristics, and evaluation methods. We also include additional experiments examining the effect of explanation hyperparameters.

### B.1 Prompts

In this section we provide the prompts used for each confidence elicitation method. Text in red represents substitutions that are made to the prompt at inference time, for example adding the text of the specific multiple choice question. For the stable explanations method in Figure˜3 we provide our explanation generation prompt and conditional answer generation prompt. We use the response from this first prompt to generate our default question explanations (discarding the answer that comes after). We then use the logits from the second prompt conditioned on explanations as the posterior answer distribution for that explanation. The entailment probability prompt used is the same as in []. For the token probability prompt (Figure˜4) we use a simple question and answer format, and use the softmax of next token logits to determine answer confidence. For the linguistic confidence prompt in Figure˜5 we follow [] best prompt choice and parse the returned response for answer and confidence value. For chain-of-thought consistency confidence we use a zero-shot modified version of the prompt from [] (Figure˜6) to generate multiple explanations and answers (discarding explanations and taking a majority vote over returned answers). We also explore using this prompt to generate explanations (discarding answers instead) for our CoT-stability confidence metric. The top-k confidence prompt is provided in Figure˜7; the resulting LLM response is parsed for $k$ confidence values. Lastly we include the conditional explanation prompt used to generate correct and incorrect explanations in Figure˜1. Unless otherwise noted, temperature for all generated explanations is set to Temp=0.7 for both stable explanations and CoT-consistency method.

Figure: Figure 3: Stable Explanation Prompts
Refer to caption: figures/stability_prompt_v2.png

Figure: Figure 4: Token Probability Prompt
Refer to caption: figures/token_prob_prompt.png

Figure: Figure 5: Linguistic Confidence Prompt
Refer to caption: figures/lingusitic_prompt.png

Figure: Figure 6: Chain of Thought Explanation Prompt
Refer to caption: figures/cot_prompt.png

Figure: Figure 7: Top-K Confidence Prompt
Refer to caption: figures/topk_prompt.png

Figure: Figure 8: Conditional explanation prompt used to generate explanations in Figure˜1
Refer to caption: figures/conditional_prompt.png

### B.2 Dataset Details

We can observe in Table˜3 that the QA datasets with longer questions typically are harder for the model to answer correctly. We see that our method, like many other sample+aggregate based answering methods generally has higher accuracy than the baseline model []. This accuracy boost is less pronounced however for GPT-4.

For GPT-3.5-Turbo results we generate confidence scores for 250 questions per dataset (or maximum dataset size if smaller). Due to computational cost we only use 100 questions per dataset when testing on GPT-4-Turbo. We use validation splits for CSQA, TruthQA, and test splits for MedQA and MMLU datasets.

**Table 3: Comparing accuracy for default model predictions vs. most confident stability predictions across benchmark datasets. One can observe a clear improvement in accuracy for both GPT-3.5 and GPT-4.**
| Method | Avg. Question Length | GPT-3.5 Acc. | GPT-3.5 Stability Acc. | GPT-4 Acc. | GPT-4 Stability Acc. |
| --- | --- | --- | --- | --- | --- |
| CSQA | 151 | 0.79 | 0.80 | 0.84 | 0.88 |
| TruthQA | 329 | 0.54 | 0.64 | 0.85 | 0.91 |
| MedQA | 916 | 0.59 | 0.68 | 0.82 | 0.84 |
| MMLU Law | 1233 | 0.46 | 0.56 | 0.64 | 0.67 |
| MMLU Physics | 172 | 0.57 | 0.72 | 0.92 | 0.92 |

### B.3 Evaluation Details

When evaluating confidence methods, it is important to note that performance implicitly depends on the prediction set $\hat{Y}$. For example, a metric may be well calibrated on correct answers but still be overconfident on incorrect ones, meaning the confidence metric would evaluate as worse on a less accurate prediction set. Therefore, for comparison purposes we use the same set of default LLM predictions (setting Temp=0) for GPT-3.5 and GPT-4 results.

In order to break possible ties in confidence when evaluating AURC and AUROC methods, we follow the approach of [] and add a small amount of gaussian noise ($\sigma=1e-6$) to each confidence score. We repeat this process for $r=10$ times and take the average AURC and AUROC scores. We provided We also follow common practice in previous works by using $M=10$ as the number of bins when calculating ECE [].

We use OpenAI’s gpt-3.5-turbo-1106 snapshot for GPT-3.5 experiments and gpt-4-1106-preview snapshot for GPT-4. Generating and evaluating confidence scores for each method on one dataset takes on the order of an hour for GPT-3.5-Turbo, and two hours for GPT-4-Turbo using OpenAI’s API.

Figure: Figure 9: Risk coverage (left) and receiver-operator (right) curves for confidence metrics generated on the MedQA questions using GPT-4. Our stability method outperforms others on this dataset as evidenced by larger area under the curves. We can also observe that questions with confidences in the top 50% were all correct.
Refer to caption: figures/example_curves.png

### B.4 Alternate Explanation Plausibility Measures

Inspired by [], which looks at the true/false token probability an LLM assigns to a given answer being true, we explore evaluating the probability that an explanation is ‘true’. To do this, we provide the model with both question and explanation and ask: ‘Is this the most likely explanation? (T/F)’. We also try asking the question ‘Does the explanation completely describe the question? (T/F)’. We then repeat the experiment in Section˜4.2, examining distributions of explanations measured via these probabilities. We find in figure˜10 that these measures fail to properly distinguish between different explanations.

Figure: Figure 10: Empirical distribution of MMLU explanations when measured via GPT-3.5 probability of being ‘most-likely explanation’ (left) and probability of ‘completely describing’ the question (right). One can see that true (blue) and false (red) answer-conditioned explanations are difficult to distinguish.
Refer to caption: figures/mmlu_law_explanation_p_likely.png

### B.5 Sensistivity to Explanation Prompting

Our stable explanation method reweights explanations based on entailment probability, but if the quality of sampled explanations is poor to begin with our resulting distribution will still be inaccurate. Here we will discuss the effect of instructing the LLM to generate explanations before or after an answer (i.e. the order of ‘explanation’ and ‘answer’ in the stability explanation prompt in Figure˜3). We observe in Table˜4 that generating explanations before the answer clearly results in higher quality explanations, as evidenced by improved performance on selective uncertainty and calibration tasks.

**Table 4: Comparing stability confidence performance using explanations generated before and after an answer for GPT-3.5. One can clearly observe that explanations generated before the answer (i.e. in chain-of-thought fashion) outperform those generated afterwards across all performance metrics.**
|  | Pre-Answer Stability (Default) |  | Post-Answer Stability |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ |
| CSQA | 0.901 | 0.779 | 0.123 | 0.866 | 0.731 | 0.201 |
| TruthQA | 0.801 | 0.853 | 0.21 | 0.792 | 0.839 | 0.254 |
| MedQA | 0.784 | 0.798 | 0.169 | 0.743 | 0.743 | 0.251 |
| MMLU Law | 0.642 | 0.736 | 0.259 | 0.629 | 0.706 | 0.289 |
| MMLU Physics | 0.792 | 0.834 | 0.186 | 0.779 | 0.811 | 0.252 |

### B.6 Varying Sample Size

In this section we briefly analyze the effect that the number of sampled explanation has on our confidence metric. In Figure˜11 we observe that selective uncertainty performance (AURC and AUROC) saturates quickly for simpler questions answering tasks such as commonsenseqa. On the other hand MedQA and MMLU Law datasets both demonstrate steady performance gains up to $M=5$ samples per question. Calibration error gradually decreases for all datasets examined.

Figure: (a) AURC vs. Number of Explanations for the stable explanations confidence metric.
Refer to caption: figures/aurc_vs_explanations.png

### B.7 Comparison to TTA

Contemporaneously to this manuscript’s submission, another method related to our approach was proposed []. The Think-Twice before assure (TTA) method asks for explanations conditioned on different answers, then does a top-k confidence elicitation using these explanations in the prompt. Although similar in the sense that confidence metrics are being generated by conditioning on explanations, their combination of explanations into a single prompt does not match the ensemble of test-time classifiers view that our method takes.
The authors have not yet released code or dataset splits, but we have implemented their method by following the written procedure and using the same prompts (see Figure˜12). We found during our implementation on the shared CSQA dataset, the evaluation results for selective uncertainty tasks are slightly below that what the authors report (AURC, AUROC), most likely due to the difference in specific questions used during testing. Nonetheless we report the full results of our implementation in table˜5, and note that this metric does appear to have lower ECE in many cases.

**Table 5: Evaluation for the TTA Confidence metric (Our implementation) on GPT-3.5. Results that outperform our stable explanations metric are marked with an asterisk.**
|  | TTA (Our Implementation) |  |  |  |
| --- | --- | --- | --- | --- |
|  | AURC $\uparrow$ | AUROC $\uparrow$ | ECE $\downarrow$ | Acc. $\uparrow$ |
| CSQA | 0.885 | 0.688 | 0.104* | 0.736 |
| TruthQA | 0.698 | 0.706 | 0.093* | 0.672* |
| MedQA | 0.641 | 0.581 | 0.207 | 0.505 |
| MMLU Law | 0.574 | 0.657 | 0.148* | 0.456 |
| MMLU Physics | 0.717 | 0.697 | 0.1* | 0.557 |

Figure: Figure 12: TTA Confidence Prompt
Refer to caption: figures/tta_prompt.png

## Appendix C Alternative Perspectives of Stable Explanations

### C.1 Confidence through the Viewpoint of Transductive Inference

Transductive learning selects a classifier at inference-time based on a combination of training and test data [, , ]. Typically transductive learning involves fine-tuning some classifier parameter $w$ based on an explicit loss objective. However, we claim that using an LLM to generate a sequence of text before an answer (i.e. an explanation) is an alternate way of doing transductive reasoning. First, note that answering a question after an explanation, such as in chain-of-thought prompting [], effectively changes the decision boundary of the LLM classifier at inference time. Second, consider that when an LLM generates an explanation, it produces concepts related to those in the question. These additional concepts can be thought of as forcing the LLM at inference time to pay more attention to the decision boundary in the area around the test datum.
In-context learning literature, which examines LLM performance after *manually* inserting demonstrations similar to the test question, has already shown a direct connection between transformer context adjustment and classical fine-tuning behavior [].

To formalize this perspective, let $D^{t}=\{(x_{1},y_{1}),\ldots,(x_{t},y_{t})\}$ be a dataset of sequential data up to time $t$, with $x_{i}\in X\subset{\mathbb{R}}^{M}$ and labels $y_{i}\in Y\subset\{1,\dots,K\}$. We denote with $D^{t}_{-}=\{(x_{1},y_{1}),\dots,(x_{t-1},y_{t-1}),x_{t}\}$ the dataset without the last label $y_{t}$. We can write our transductive prediction for $x_{t}$ given data $D^{t}_{-}$ including $x_{t}$ as:

$$ $\displaystyle\hat{y}_{t}=~\underset{w,y}{\rm argmin}~{}\underbrace{\frac{1}{t}\ell(f_{w}(x_{t}),y)+\frac{1}{t}\sum_{i=1}^{t-1}\ell(f_{w}(x_{i}),y_{i})}_{\doteq L(w;(D^{t}_{-},y))}.$ (17) $$

If $\ell$ is interpreted as a log likelihood, then $L$ can be interpreted as the negative log posterior probability over hypotheses.
If we think of optimizing instead over explanations where $f_{e}(x_{t})=\phi(x_{t},e)$, then the problem reduces to finding an explanation that strongly supports a single answer without biasing predictions on original test data. The second term in equation˜17 is expensive to compute at inference time, but if some approximation of this training loss existed it would make optimization tractable. We hypothesize that if the explanation under consideration is plausible and faithful to the question (as determined using the same LLM), it should not reduce the accuracy of previous decisions too much. Therefore we can avoid having to optimize over all previous questions and instead optimize over whatever faithfulness measure $g_{\phi}(e)$ we define:

$$ $\displaystyle\hat{y}_{t}=~\underset{e,y}{\rm argmin}~{}\ell(\phi(x_{t},e),y)+\lambda g_{\phi}(e)$ (18) $$

This looks exactly like the typical transductive setting but with a more easily computable ‘transductive prior’.

### C.2 Confidence throught the Viewpoint of Solomonoff Induction

While transductive inference typically finds single test-time classifier, our method looks for a *distribution* of likely classifiers. In this sense, our method can be seen as a special case of Solomonoff induction []. Solomonoff induction considers how well data-generating programs,$H$, (i.e. a binary string run on a Turing machine) explain the test data, $D$

$$ $\displaystyle P(H|D)=\frac{P(D|H)P(H)}{P(D|H)P(H)+\sum_{A\neq H}P(D|A)P(A)},$ (19) $$

where $A$ are alternative programs. Solomonoff induction formalizes the principle of Occam’s razor by choosing a universal prior $P(H)$ that gives a higher probability to shorter-length programs. Then to predict new data $D^{\prime}$ given previous observations, one simply computes

$$ $\displaystyle P(D^{\prime}|D)=\mathbb{E}_{H}[P(D^{\prime}|H,D)]=\sum_{H}P(D^{\prime}|H,D)P(H|D).$ (20) $$

While these Bayesian equations seem simple, Solomonoff’s induction is provably uncomputable. However, our method can be interpreted as restricting our hypothesis class from the set of all computable programs $H$ to the set of all *LLM-interpretable* programs $e$. Instead of a prior on program length, we can use the LLM’s prior likelihood of valid sequences in the language $p_{\phi}(e)$. This restriction makes our calculations more tractable, as we can easily approximate expectations over our hypothesis class by sampling explanations from the LLM.

### C.3 Confidence through the Viewpoint of Stability

Another recent line of work has been analyzing LLMs through the lens of stochastic dynamical models []. Through the perspective of stability analysis one could interpret our method’s preference for explanations convening to a single answer as searching for *fixed points* of a specific LLM system. This LLM dynamical system consists of two alternating steps, first generating an explanation conditioned on one of the answers ($e\leftarrow\phi(q,a)$) then generating a new answer based on this explanation ($a^{\prime}\leftarrow\phi(q,e)$). Intuitively this system mirrors how a human expert may think about a question by considering alternative conclusions one could draw given beliefs about the world. An answer with only a single plausible explanation that strongly supports that same answer (i.e. decision distribution collapses to a singleton) forms a stable cycle in this system.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: All my questions are in the weaknesses section.
soundness: 2
strengths: 1. **Interesting and novel approach to LLM uncertainty estimation.** I think the idea of accounting for the accuracy/plausibility of the explanation when computing uncertainty estimates is interesting and intuitive. However, I think in the current version of the paper, this key idea gets lost in a lot of jargon/details (e.g., the Bayesian framing). I think the paper would be improved if this idea was centered more.
2. **Strong results on the task of selective uncertainty.** The proposed method clearly outperforms baselines on this important task.
3. **Figure 1 is a nice, intuitive illustration of the benefits of the method.** I thought this was a great way to make the point that accounting explanation plausibility provides useful information about the probability that the resulting answer is correct.
summary: This paper proposes a new method for measuring the confidence of LLM answers. The key idea of the method is to estimate the distribution of LLM answers by marginalizing over the distribution of explanations given for each answer. And instead of using the empirical explanation distribution, the authors reweight sampled explanations by the probability that they are entailed by the question. The authors evaluate their method on five question answering datasets and two LLMs (both GPT models) and find that it outperforms baselines in terms of selective uncertainty, but not calibration.
weaknesses: 1. **Bayesian framing of your method is confusing/misleading.** You frame the method as the computation of a posterior distribution, as is done in Bayesian statistics. While your proposed method certainly resembles this, I don’t think this description is fully accurate. In Bayesian analysis, the computation of a posterior distribution involves updating your prior beliefs about a distribution (i.e., the prior) based on new evidence (i.e., the likelihood). To do this, you apply Baye’s rule, multiplying the prior by the likelihood and dividing by the evidence. However, your analysis does not involve the inclusion of a prior. Because your approach is not exactly Bayesian, I found this framing (as well as your use of the term “posterior”) confusing. I think it is a bit misleading and makes it more difficult to understand your method. While I still think it could be useful to describe the connection between your method and Bayesian analysis (e.g., you can say that your approach is similar to computing a posterior predictive distribution), I don’t think this should be used as the main way of describing your method. Instead, your computation of the answer distribution by marginalizing over the explanation distribution can be seen as an application of the law of total probability.
2. **Point about generalization to out-of-distribution test data is not fully supported.** You claim that one of the main weaknesses of existing uncertainty quantification methods is that they assume that the training and test data come from the same distribution. And you suggest that your method can overcome this weakness through its consideration of the explanation distribution. This is an interesting point that I believe is worth investigating, but it seems mainly speculative – the paper doesn’t provide any clear evidence to support this. In particular, it’s not fully clear to me that just because including an explanation as an intermediate step results in a new answer distribution, this new answer distribution will be better calibrated for the test data. And you don’t provide any sort of theoretical analysis to back this point. Further, your experiments are all on common benchmarks that could have been included in the GPT training data. Finally, it appears to me that the existing methods for uncertainty quantification based on CoT-consistency share the same benefit as your method with respect to this issue – they also consider explanations and therefore, “adapt” to the test data. However, you make it sound all existing methods make the ID assumption and therefore are harmed by differences between training and test data. Can you clarify this?
3. **Calibration performance is limited, and it’s not fully clear why we should discount this task compared to selective uncertainty.** The proposed method slightly underperforms some of the other methods on the ECE metric. I think this okay, given the improved performance in the selective uncertainty metrics. However, I am not fully convinced by the authors’ argument that this metric should be discounted. In the discussion section, the authors say that the ECE metric relies on the “hypothesis that the training, test, and calibration data are all drawn from the same distribution.” It is not clear to me why this is the case. I get that calibration methods rely on this assumption, but I don’t get why the *evaluation* of ECE relies on this assumption. Further, to account for the limitations around the effect of binning size, it seems like the authors could present results for multiple bin sizes or also present results for an alternative calibration metrics such as proposed in [1].
4. **Use of the term ‘stable’ is not clear.** You say that prompting a model to produce explanations, such as with CoT, “serves to ‘stabilize’ the sampled answers around an isolated minimum.” It is not totally clear to me what you mean by this, and at least in my reading of Wei et al., 2022, I don’t see this claim made. What exactly do you mean by this? Do you mean that prompting via CoT results in a lower entropy answer distribution? And do you have evidence to support this? I am mainly picking on this point because you term your method “stable” explanations, so I think it’s important that a reader can understand what you mean by “stable.”
5. **Writing lacks clarity.** This is a little bit redundant with the points made above, but in many places, I found that this paper was not fully clear, e.g.,:
* To describe your method in the intro, you say that it can be thought of as “computing the posterior predictive distribution by transductive marginalization over test-time classifiers.” This uses a lot of jargon, most of which you have not yet defined at this point (and some of which you never define, such as “transductive”). Also, as I said earlier, while what you do resembles computing a posterior predictive distribution, it isn’t the same in a strict sense. I think there are much simpler ways of describing your method, which would make the paper clearer.
* As described above, it is not clear what makes an explanation “stable.”
* In the methods section, you describe a step of your method that involves grouping explanations into equivalence classes by clustering them according to their resulting next token distributions. This step is unclear to me. Exactly how do you form these clusters? And then how do you use the equivalence classes in your computation of uncertainty?
* In your description of your method, I think it would be helpful to distinguish between the probability that an LLM outputs an answer (or explanation) from the probability that an answer (or explanation) is correct. Your current writing seems to conflate these two, which I think makes the text a bit harder to parse.
6. **Analysis is limited to GPT models.** It would strengthen the paper to see how the method works on other LLMs that are not GPT/OpenAI models (e.g., Llama, Claude, etc.).

[1] https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Why did the authors choose to compute entailment probability and marginal probability for obtaining the final results, rather than performing semantic clustering on the explanations? The introduction in line 217 is too brief and somewhat difficult to understand. I feel like it would also be beneficial to try the semantic clustering approach during the experiment section or include a case study.
2. Why does utilizing explanations in this manner help mitigate overconfidence compared to directly using answers? The motivation for using explanation seems unclear to me, and I am a little bit struggled to understand why explanation can bring much more benefit than using multiple output answers directly. 
3. In Figure 1a, why is there a need to condition on correct and incorrect answers instead of simply generating explanations based on the question?
4. The hyperlink in line 319 is invalid.
5. Why is the term "stable explanation" used?
soundness: 3
strengths: Originality: The paper proposes using explanations as latent variables to observe the model's uncertainty for each question and introduces a new ensemble method. This method evaluates the correctness of each generated explanation and then estimates answer confidence based on them. Compared to other methods such as consistency, it computes the final posterior probability of each answer based on all possible explanations. While the idea of using explanations may not be very new, the specific implementation and application of explanations in LLM's calibration setting has not been explored before.

Clarity: The paper is generally well-written and easy to follow. 

Significance/Quality: Overall, the paper introduces a new ensemble mechanism and demonstrates, through experiments, that it shows marginal improvements over baseline methods. However, its impact is limited by the focus on the multi-choice setting and a small set of benchmarks, making the broader applicability and superiority of the method less evident. Future work could extend this approach to more diverse problem settings to further establish its value.
summary: This paper addresses the task of confidence calibration in multi-choice QA setting, i.e., exploring how to quantify the likelihood that the answers provided by LLMs are correct. 

The authors propose an ensemble-based method involving the following steps:
1. Explanation Sampling: Multiple explanations are sampled from the LLM using prompts (set with a temperature of 0.7) to generate diverse logical justifications for these different answers.
2. Explanation Probability Calculation: The probability associated with each explanation is calculated using the entailment prompt, i.e., prompting LLM to self-evaluate whether the explanation is correct given the question. 
3. Marginal Probability Calculation: The overall answer confidence is calculated by computing the marginal probability of the answer token logits conditioned on each possible explanation. 

The main contribution of this work is introducing a new ensemble-based mechanism that leverages explanations generated by the LLM itself. Experimental results show that this method achieves slight improvements over baseline approaches, particularly in complex reasoning tasks. However, its scope is primarily limited to multi-choice problems and includes only a few benchmark datasets​.
weaknesses: Clarity: One area for improvement is the articulation of the motivation behind using explanations for the ensemble method. Providing more depth on why explanations are suited for this purpose would enhance the reader’s understanding, especially why using explanation in this manner can mitigate overconfidence.

Experiment setting: I view this method as an ensemble-based calibration method, so I feel like it should be compared with more ensemble-based methods besides CoT-consistency.

Expense: The existing method's compuational complexity is O(n^2), while the CoT-consistency only requires O(n). I feel like the time complexity cost should be considered.

Overall significance: In general, this paper's impact is limited by the focus on the multi-choice setting and a small set of benchmarks, making the broader applicability and superiority of the method less evident. Future work could extend this approach to more diverse problem settings.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * Could this method be used for open-ended generation tasks or is it only suitable for multiple choice questions?  
* The same model seems to be used to both generating the explanations and evaluating their entailment probability. Doesn’t this introduce circularity? A model that has learned an incorrect fact could use that fact in an invalid explanation and then give a high entailment probability for that explanation.
soundness: 3
strengths: * Strong empirical results with thorough experimental validation.
* Well-grounded theoretical framework.
* Practical implementation requiring only black-box access to LLMs
* Comprehensive ablation studies that validate the contribution of each component
summary: * The paper proposes a novel framework for measuring LLM uncertainty by analyzing the distribution of generated explanations. The key insight is combining explanation sampling with a reweighting mechanism based on logical consistency, measured through entailment probability.
* The method is evaluated comprehensively across 5 datasets using both GPT-4 and GPT-3.5. Performance is primarily assessed using AURC and AUROC metrics, with several competitive baselines for comparison.
weaknesses: * The paper could benefit from more figures, particularly a good figure 1 to help explain the method. Possibly some ROC figures in the main paper since AUROC is the main metric used.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Do you generate the explanation before or after generating the answer? From the text, I understood that you use "Question, Explanation, Answer", but in Figures 1 (and 8 in the appendix) it looks the other way around. Is this only for the experiments in Section 4.2? 
* Why did you set the temperature to 0.7 as opposed to 1? Do you have data supporting this choice? (E.g., higher accuracy)
* Can you report the missing baseline approaches, other models, variance estimates, and potentially higher number of questions per dataset, as outlined above? 
* Can you add the updated section 4.2 experiment, as highlighted above?
soundness: 2
strengths: * The authors use five datasets that are challenging to current LLMs, as opposed to simple datasets like TriviaQA.
* Chain of thoughts / explanation prompting is a topic with increased current interest. 
* The limitation section is well-written, and I would love to see more experimental analyses mentioned there (such as the difference in performance on simple / hard questions)
summary: The authors propose to estimate the uncertainty of LLM answers in single-token QA tasks by not directly recording an answer (probability) after giving a question, but asking for an explanation in-between. This is resampled multiple times per question and reweighted by how much the explanation fits to the question, judged by an NLI algorithm. First results show that this might improve the correctness AUROC on some datasets and models.
weaknesses: EDIT: Post-rebuttal, I've increased my score from 3 to 5 due to the new experiments and variances reported. Below are my weaknesses for the original score. 

* Many common baselines are missing, even if they are cited and discussed in the paper. I would have a higher trust in the results (and increase my soundness score) if the paper compared to P(True), Semantic Entropy, the entropy of the output distribution (eq. to Naive Entropy, because we only have one token), the number of different answers when prompted multiple times, and the Eccentricity of the NLI entailment graph between answers. To get you started quickly, I recommend the codebase of https://arxiv.org/abs/2311.07383 (I'm not author or affiliated with any authors of that paper, it's what I genuinely believe will help you the best and the fastest)
* The TTA approach has a higher ECE than the proposed method on 4 of the 5 datasets, but is hidden into Appendix B.7 because it is contemporary work. I agree that the underperformance should not be a reason for rejection because of the comtemporary nature of the work (and also because I don't believe that papers should be published if and only if SOTA) and it does not influence any of my scores (which I made before discovering it in B.7), but I strongly recommend bringing the results to the main paper for scientific integrity. 
* The authors only use 100-250 questions per dataset (Appendix B.2). This is likely to have large variance, but the variance is not reported. It could be that it is in the region of the recorded AUROC differences (~0.01 to 0.05). This lowers my soundness score. I believe a higher number of questions would be an easy way to improve the robustness of the experiments, since the experiments per Appendix B.3 currently take only 15 GPU hours (or more precisely, OpenAI-API hours). 
* Section 4.2 asks an interesting side-question, but makes an experimental choice which I believe confounds the results. That is, the (sequence) probability of explanations of wrong and correct answers is compared _after_ conditioning the generation of the explanation on the answer. Due to the autoregressive nature of Transformers, this will naturally lead to similar probabilities. Do you have data for explanations generated before giving the answer, i.e., the sequence probability of the "middle part" of a "question + explanation + answer" sequence (which is more aligned with what your method does)? This would be a highly interesting experiment, and allow to increase my soundness score. 
* The approach is limited to QA tasks with a single output token. This is a very synthetic setting compared to free-form answers, which limits the contribution score. 
* Experiments are limited to GPT-3.5 and GPT-4. These are very similar LLMs. It would be interesting to see whether the results transfer to, e.g., Llama-3.1-8b-chat (or Mistral/Falcon/Yi).
* Code and data are not released, hindering the replication of results.
* The notation is not clearly defined, hindering exact replication at the current state of the paper and being the prime reason for my presentation score (because the writing itself is good!). I.e., the (set?) $E$ is never defined. $\mathcal{A}^\tau$ is never defined. $a$ and $i$ can probably be used interchangably, since $a$ is always only a single token, bijectively identifiable by its index $i$. It's not defined whether the softmax is over the alphabet $\mathcal{A}$ or the set of possible answers $S$, and how that set $S$ is defined in the first place.
* The paper repeatedly makes the argument that the underperformance of token-likelihood based uncertainty estimates is due to a train-test distribution shift, particularly in the introduction and related works sections. However, there is no experimental data to back this claim and no data to show that the proposed method is better in this regard, such as transfer experiments or in-distribution vs out-of-train-distribution experiments. 
* The approach requires to sample multiple explanations, i.e., multiple sequences of additional tokens. This makes it expensive. 

Minor comments that do not influence my rating and do not need to be addressed in the rebuttal (only in the final camera ready):
* I found Appendix C (mathematical perspectives on your work and other works) much more interesting than Section 2.1 (general intro into uncertainty in ML). I think swapping the two would make the paper more interesting to expert readers. 
*  "is called well-calibrated if on average predictions with confidence r = 0.XX are correct close to XX% of the time." could be rewritten to something like "... $r \in [0,1]$ are correct $100r \\%$ of the time."
* typo in line 198: sef-consistency
* I think Figure 2 should be a Table
* Personal take, but I would tone done the transductive learning narrative, which only really gets explained in the conclusion. I believe this paper falls more into the category of CoT approaches, which is a great topic in and by itself.

Meta-review: The paper measures LLM confidence through stable explanations, leveraging the distribution of generated explanations and their entailment probabilities to estimate uncertainty. The reviewers acknowledged the novelty of using explanation-based reweighting but raised significant concerns regarding clarity, theoretical grounding, and experimental results. While additional experiments during the rebuttal improved clarity and introduced new baselines, the reviewers did not reach a consensus. I recommend rejection.

Decision: Reject