Large Language Model Confidence Estimation via Black-Box Access

## Abstract

Abstract Estimating uncertainty or confidence in the responses of a model can be significant in evaluating trust not only in the responses, but also in the model as a whole. In this paper, we explore the problem of estimating confidence for responses of large language models (LLMs) with simply black-box or query access to them. We propose a simple and extensible framework where, we engineer novel features and train a (interpretable) model (viz. logistic regression) on these features to estimate the confidence. We empirically demonstrate that our simple framework is effective in estimating confidence of Flan-ul2, Llama-13b and Mistral-7b on four benchmark Q&A tasks as well as of Pegasus-large and BART-large on two benchmark summarization tasks with it surpassing baselines by even over 10 % percent 10 10\% 10 % (on AUROC) in some cases. Additionally, our interpretable approach provides insight into features that are predictive of confidence, leading to the interesting and useful discovery that our confidence models built for one LLM generalize zero-shot across others on a given dataset.

## 1 Introduction

Given the proliferation of deep learning over the last decade or so [], uncertainty or confidence estimation of these models has been an active research area []. Predicting accurate confidences in the generations produced by a large language model (LLM) are crucial for eliciting trust in the model and is also helpful for benchmarking and ranking competing models []. Moreover, LLM hallucination detection and mitigation, which is one of the most pressing problems in artificial intelligence research today [], can also benefit significantly from accurate confidence estimation as it would serve as a strong indicator of the faithfulness of a LLM response. This applies to even settings where strategies such as retrieval augmented generation (RAG) are used [] to mitigate hallucinations. Methods for confidence estimation in LLMs assuming just black-box or query access have been explored only recently [, ] and this area of research is still largely in its infancy. However, effective solutions here could have significant impact given their low requirement (i.e. just query access) and consequently wide applicability.

There exists a slight difference in what is considered as uncertainty versus confidence in literature [] and so to be clear we now formally state the exact problem we are solving. Let $(x,y)$ denote an input-output pair, where $x$ is the input prompt and $y$ the expected ground truth response. Let $f(.)$ denote an LLM that takes the input $x$ and produces a response $f(x)$. Let $\lambda(.,.)$ denote a similarity metric (viz. rouge, bertscore, etc.) that can compare two pieces of text and output a value in $[0,1]$, where 0 0 implies the texts are very different while $1$ implies they are exactly the same. Then given some threshold $\theta\in[0,1]$, we want to estimate the following probability for an input text $x$:

$$ $\displaystyle\text{Probability of correct }=P\left(\lambda(y,f(x))\geq\theta|x\right)$ (1) $$

In other words, we want to estimate the probability that the response outputted by the LLM for some input is correct. Unlike for classification or regression where the responses can be compared exactly, text allows for variation in response where even if they do not match exactly they might be semantically the same. Hence, we introduce the threshold $\theta$ which will typically be tuned based on the metric, the dataset and the LLM.

Having black-box access to an LLM limits the strategies one could leverage to ascertain confidence, but if the proposed strategies are effective they could be widely applied. Previous approaches [, , ] predominantly exploit the variability in the outputs for a given input prompt or based on an ensemble of prompts computing different estimators. Our approach enhances this idea where we design different ways of manipulating the input prompt and based on the variability of the answers produce values for each such manipulation. *We aver to these values as a features.* Based on these features computed for different inputs we train a model to predict if the response was correct or incorrect. The probability of each such prediction is then the confidence that we output. Since, the models we use to produce such predictions are simple (viz. logistic regression) the confidence estimates are typically well calibrated []. Moreover, being interpretable we can also see which features were more crucial in the estimation. This general framework and the features we engineer are shown in Figure 1. The framework is extensible, since more features or prompt perturbations can be easily added to this framework.

We observe in the experiments that we outperform state-of-art baselines for black-box LLM confidence estimation on standard metrics such as Area Under the Receiver Operator Characteristic (AUROC) and Area Under Accuracy-Rejection Curve (AUARC), where improvements in AUROC are over $10\%$ in some cases. The confidence model being interpretable we also analyze which features are important for different LLM and dataset combinations. We interestingly find that for a given dataset the important features are shared across LLMs. Intrigued by this finding we apply confidence models built for one LLM to the responses of another and further find that they generalize well across LLMs. This opens up the possibility of simply building a single (universal) confidence model for some chosen LLM and zero shot applying it to other LLMs on a dataset.

Figure: Figure 1: Above we see our (extensible) framework to estimate confidence of LLM responses. We propose six prompt perturbations which then can be converted to features based on semantic diversity in the responses and lexical similarity. The input (tokenized) prompt can optionally be also passed as a feature. The output labels for each (input) prompt are created by checking if the LLM output is correct or not. A (interpretable) logistic regression model is then trained on these features and outputs so that for any new input prompt and LLM response we can estimate the confidence of it being correct based on our model. Moreover, we can also ascertain the features important in estimating these confidences.
Refer to caption: extracted/5895816/figures/confest.png

## 2 Related Work

The literature studying approaches for estimating the uncertainty in a machine learning model’s prediction is large. One organization of this body of work involves dichotomizing it into *post-hoc* and *ab initio* approaches. Post-hoc methods attempt to calibrate outputs of a pre-trained model such that the estimate uncertainties correlate well with the accuracy of the model. These include histogram binning [, ], isotonic regression [], and parametric mapping approaches, including matrix, vector, and temperature scaling [, , ]. While variants of these approaches [, ] have been adopted for LLMs they assume a white-box setting where access to the LLM’s representations are available. In contrast, our approach quantifies a LLM’s uncertainties without requiring access to the internals of the LLM. Ab initio approaches, including, training with mix-up augmentations [], confidence penalties [], focal loss [], label-smoothing [], (approximate) Bayesian procedures [], or those that involve ensembling over multiple models arrived at by retraining from different random initializations [] require substantial changes to the training process or severely increase computational burden, making them difficult to use with LLMs.

For LLMs in particular, recent works [, , ] have empirically found evidence of miscalibration and had varying degrees of success in better calibrating smaller LLMs using mixup [], temperature scaling and label smoothing []. Others [] have employed supervised fine-tuning to produce verbalized uncertainties to be better calibrated on certain tasks. However, this additionally requires the ability to compute gradients of the LLM’s parameters. Our black-box approach has no such requirement.
Another body of work [, , ], learns an auxiliary model for predicting whether a LLM’s generation is incorrect. We also employ an auxiliary model, but rely on only the prompts to the LLM and the generations produced by the LLM to train it.

Similar to us, other recent works have also explored black-box approaches. For instance, in [], multiple completions from an LLM are generated, grouped based on semantic content, and uncertainty is quantified across these semantic groups. [] exploit insights from spectral clustering to further finesse this process. In [, ] the authors use carefully crafted prompts for certain more capable LLMs to express better-calibrated uncertainties. However, this approach is less effective for smaller and open-sourced LLMs []. Others [] have relied on ensembles of prompts created using templates or reordering of examples in few shot settings to quantify confidences. We on the other hand propose dynamic variations of the prompt applicable (even) in the zero-shot setting, where for certain of our features we only analyze the response without any variation in the prompt.

**Table 1: Below we see examples of different prompt perturbations for a prompt from the SQuAD dataset. The color blue and strike outs indicate changes to the input prompt. i) SD does not change the prompt (hence empty cell), but using a stochastic decoding scheme samples multiple responses (four example samplings shown). PP paraphrases the prompt. SP randomly reorders some of the sentences. EFA repeats certain sentences with entities in them. SR removes stopwords. SRC checks for consistency in reasonable size random splits of the LLM response (again prompt is not perturbed). The splitting of the two sentences indicates inconsistency as depicted in red. Thus, the perturbations test an LLM in complimentary ways.**
| Input Prompt |
| --- |
| context: The Normans (Norman : Nourmands ; French : Normands ; Latin : Normanni) are the people who, in the 10th and 11th centuries, gave their name to Normandy, a region of France. They descended from the Normands ("Norman" comes from "Norseman") of the raiders and pirates of Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear allegiance to King Charles III of France of the West. During generations of assimilation and mixing with the native French and Roman-Gaulese populations, their descendants would gradually merge with the Carolingian cultures of West France. The distinct cultural and ethnic identity of the Normans originally emerged in the first half of the 10th century, and it continued to evolve over the centuries that followed. |
| question: In what country is Normandy located? |

## 3 Methodology

We now describe our methodology to estimate confidences for individual LLM outputs.

### 3.1 Prompt Perturbations

We first propose six black-box strategies that can elicit variable behavior in an LLM indicative of how trustworthy its output is likely to be. Based on this variability we construct features for our confidence model in the next subsection. Note that all strategies may not be relevant in all cases. For instance, some of the strategies require a context in the prompt, while others such as SRC require longer responses (two or more sentences).
For all the perturbations but for Stochastic Decoding and Split Response Consistency, the perturbations are applied to the context if available or to the question of the input.

Stochastic Decoding (SD): This is the simplest strategy which is also done in previous works. Here the prompt is not varied, but rather using various decoding strategies comprising of greedy , beam search and nucleus sampling [] multiple outputs are sampled. As seen in Table 1 first row after sampling one could have four different outputs, which could be indicative of the LLM not being confident in its response. Specifically in the experiments, we obtain one generation using greedy and beam search decoding technique and 3 generations using nucleus sampling.

Paraphrasing (PP): In this strategy we paraphrase the context in the prompt and observe how that changes the output.
An example of this is shown in Table 1.
For paraphrasing, we use back translation, where we convert the original prompt into another language and translate it back into English. We use machine translation models from Helsinki-NLP on huggingface and translate the text from English to French and then back to English.
This new prompt then can be used to query the LLM. Changes to the output could indicate brittleness in the LLMs original response.
One could also prompt an LLM to paraphrase the responses, however, in our initial experiments, we observed that when context is involved, the model does not paraphrase the entire context and parts of it are omitted.

Sentence Permutation (SP): If the input has several named entities, we noticed that when the order of the named entities is changed without changing the meaning of the sentence, the output of the LLM also varied.
We first use named entity detector to identify the named entities and then randomly reorder certain number of these sentences. An example of this is seen in Table 1 third row, where the last sentence in the prompt is now the second sentence. As such, if the number of sentences with named entities is less than five, we reorder all of them. If it is greater than five, then we randomly select five and reorder them. Most such reorderings do not affect the LLM output if it is confident.

Entity Frequency Amplification (EFA): Similar to above, repeating sentences with named entities could also throw off the model’s outputs.
We sample a sentence from all the sentences with named entities and repeat it three times. Again, here too the output of the LLM should be maintained if the LLM is confident. An example of this is seen in Table 1 fourth row, where the first sentence is repeated twice.

Stopword Removal (SR): We remove stopwords from the context as specified by the NLTK library. Stopwords are commonly occurring words (viz. "the", "are", "to", etc.) that are assumed to have limited context specific information. Removal of such words should ideally not alter the response of an LLM if the LLM is certain of the answer. An example of this is seen in Table 1 fifth row, where the stopwords are striked out. We ensured that the negative words were not removed as they would change the meaning of the sentence.

Split Response Consistency (SRC): In this case like in the SD case the prompt is not perturbed. Rather the output is analyzed where it is randomly split such that each part is at least a single sentence. Semantic inconsistency between the two parts is measured using an NLI models contradiction probability, where one part is taken as the premise and the other the hypothesis. An example of this is seen in Table 1 last row, where the two sentences are clearly at odds with each other. This strategy though requires that the response is at least a couple of sentences long.

*As seen in Table 7, the four perturbations above (PP, SP, EFA and SR) that alter the original prompt still maintain the semantics as intended in almost all cases.*

### 3.2 Featurization

Now based on the above strategies we can construct features to train our confidence model. For each of the first five strategies above we create two types of features: i) based on semantics of the outputs and ii) based on lexical overlap. For the SRC these are not relevant so we create a different feature as seen below.

Semantic Set: Based on the responses of the first strategies (run multiple times) we create semantically equivalent sets for each. A semantically equivalent set consists of outputs that are semantically the same. If an response entails another response and vice-versa, then they both are grouped under the same semantic set. The number of such sets is a feature for our model. As such, more the number of sets lower the confidence estimate.

Lexical Similarity: We compute the average lexical similarity for outputs of each of the first five strategies (run multiple times). The similarity can be measured using standard NLP metrics such as rouge, blue score etc. The higher the lexical similarity higher the estimated confidence. We use rouge score to quantify the lexical similarity.

SRC Minimum Value: As mentioned above, semantic inconsistency between the two parts is measured using an NLI models contradiction probability, where one part is taken as the premise and the other the hypothesis. The highest contradiction probability amongst multiple such samplings is the feature value for this strategy.

Note that optionally one can also pass the entire prompt as a feature in addition to the above. In the experiments, we saw minimal improvement with such an addition.
Semantic set and lexical similarity were first used by [] where they applied it only for SD perturbation discussed in the previous section.

### 3.3 Label Creation and Confidence Estimation

Once we have the input features to our confidence model we now need to determine labels for these inputs. For training the model we compute labels by matching the LLM output to the ground truth response in the dataset, where a match corresponds to the label $1$, while a mismatch corresponds to a label 0 0. In particular, we use the rouge score to compute the similarity between the output and the ground truth and if the score is greater than a threshold of 0.3, it corresponds to label $1$, otherwise it is deemed incorrect and is labeled 0 0 similar to previous works [].
With the described features and their labels we train a logistic regression model and use it for predicting confidence scores for out-of-sample outputs.

Given that logistic regression is also an interpretable model we can also study which of our features turn out to be most beneficial and if our model trained on one LLM is transferable to other LLMs for the same dataset. Transfer across datasets can be more challenging as some datasets have contexts (viz. SQuAD), while others do not (viz. NQ) amongst other factors such as difference in domains.

**Table 2: AUROCs on four Q&A and two summarization datasets (CNN, XSUM) using a total of five LLMs (Llama, Flan-ul2, Mistral, Pegasus, BART). Higher values are better. Best results bolded.**
| Dataset(LLM) | # of Semantic | Lexical | EigenValue | Eccentricity | Degree | Ours |
| --- | --- | --- | --- | --- | --- | --- |
| Sets | Similarity |  |  |  |  |  |
| TriviaQA(Llama) | 0.73 | 0.76 | 0.77 | 0.76 | 0.77 | 0.88 |
| TriviaQA(Flan-ul2) | 0.83 | 0.8 | 0.86 | 0.86 | 0.87 | 0.95 |
| TriviaQA(Mistral) | 0.65 | 0.72 | 0.76 | 0.75 | 0.75 | 0.81 |
| SQuAD(Llama) | 0.65 | 0.72 | 0.74 | 0.58 | 0.72 | 0.83 |
| SQuAD(Flan-ul2) | 0.6 | 0.7 | 0.67 | 0.65 | 0.67 | 0.8 |
| SQuAD(Mistral) | 0.59 | 0.7 | 0.67 | 0.65 | 0.67 | 0.84 |
| CoQA(Llama) | 0.61 | 0.74 | 0.76 | 0.76 | 0.77 | 0.92 |
| CoQA(Flan-ul2) | 0.61 | 0.76 | 0.78 | 0.78 | 0.79 | 0.87 |
| CoQA(Mistral) | 0.56 | 0.74 | 0.79 | 0.77 | 0.79 | 0.81 |
| NQ(Llama) | 0.65 | 0.75 | 0.75 | 0.73 | 0.74 | 0.85 |
| NQ(Flan-ul2) | 0.76 | 0.76 | 0.86 | 0.86 | 0.86 | 0.93 |
| NQ(Mistral) | 0.66 | 0.73 | 0.77 | 0.77 | 0.78 | 0.83 |
| CNN (Pegasus) | 0.51 | 0.67 | 0.73 | 0.72 | 0.72 | 0.77 |
| CNN (BART) | 0.51 | 0.60 | 0.52 | 0.48 | 0.54 | 0.57 |
| XSUM (Pegasus) | 0.51 | 0.58 | 0.69 | 0.70 | 0.71 | 0.73 |
| XSUM (BART) | 0.51 | 0.59 | 0.53 | 0.51 | 0.52 | 0.57 |

**Table 3: AUARCs on four Q&A and two summarization datasets (CNN, XSUM) using a total of five LLMs (Llama, Flan-ul2, Mistral, Pegasus, BART). Higher values are better. Best results bolded.**
| Dataset(LLM) | # of Semantic | Lexical | EigenValue | Eccentricity | Degree | Ours |
| --- | --- | --- | --- | --- | --- | --- |
| Sets | Similarity |  |  |  |  |  |
| TriviaQA(Llama) | 0.77 | 0.8 | 0.8 | 0.8 | 0.8 | 0.83 |
| TriviaQA(Flan-ul2) | 0.69 | 0.72 | 0.73 | 0.73 | 0.73 | 0.74 |
| TriviaQA(Mistral) | 0.55 | 0.63 | 0.64 | 0.64 | 0.64 | 0.64 |
| SQuAD(Llama) | 0.3 | 0.36 | 0.37 | 0.28 | 0.36 | 0.68 |
| SQuAD(Flan-ul2) | 0.73 | 0.95 | 0.83 | 0.82 | 0.83 | 0.96 |
| SQuAD(Mistral) | 0.72 | 0.93 | 0.82 | 0.82 | 0.82 | 0.96 |
| CoQA(Llama) | 0.56 | 0.67 | 0.67 | 0.67 | 0.67 | 0.71 |
| CoQA(Flan-ul2) | 0.7 | 0.79 | 0.8 | 0.79 | 0.79 | 0.8 |
| CoQA(Mistral) | 0.46 | 0.62 | 0.64 | 0.63 | 0.64 | 0.61 |
| NQ(Llama) | 0.37 | 0.41 | 0.42 | 0.41 | 0.41 | 0.45 |
| NQ(Flan-ul2) | 0.41 | 0.44 | 0.47 | 0.46 | 0.45 | 0.47 |
| NQ(Mistral) | 0.32 | 0.38 | 0.40 | 0.40 | 0.39 | 0.42 |
| CNN (Pegasus) | 0.45 | 0.51 | 0.53 | 0.43 | 0.52 | 0.74 |
| CNN (BART) | 0.21 | 0.22 | 0.21 | 0.21 | 0.21 | 0.34 |
| XSUM (Pegasus) | 0.16 | 0.17 | 0.19 | 0.17 | 0.17 | 0.27 |
| XSUM (BART) | 0.21 | 0.22 | 0.20 | 0.21 | 0.22 | 0.35 |

## 4 Experiments

We demonstrate the efficacy of our method on question answering and summarization tasks. For summarization, we used BART-large [] and Pegasus-large [] and for question answering, we used Mistral-7B-Instruct-v0.2 [], Llama-2-13b chat version [], and Flan-ul2 models [].
For question answering we elicited responses from these models on four datasets, namely, CoQA [], SQuAD [], TriviaQA [] and Natural Questions (NQ) []. CoQA and SQuAD provide the context and expect the model to respond to the question based on the context, while TriviaQA and NQ do not have a context and require the model to tap into its learnt knowledge. For our experiments, we use the validation splits for all the datasets as done previously []. CoQA has 7983 datapoints, TriviaQA has 9960 datapoints, SQuAD has 10,600 datapoints and NQ has 7830 datapoints. For summarization, we used CNN  Daily Mail [] and [] and XSUM [] datasets. We use a subset of the validation splits of both the datasets comprising of 4000 datapoints.
For detecting entailment, we use deberta-large-nli model which is specialized for NLI tasks [].

**Table 4: Up to four important features (absolute coefficient value $>1e^{-4}$) ranked based on our logistic regression model for the different dataset and LLM combinations. Rank 1 indicates the most important feature, while Rank 4 is the least important amongst the four.**
| Dataset(LLM) | Rank 1 | Rank 2 | Rank 3 | Rank 4 |
| --- | --- | --- | --- | --- |
| TriviaQA(Llama) | SD lexical | SD semantic | SR lexical | PP lexical |
|  | similarity | set | similarity | similarity |
| TriviaQA(Flan-ul2) | SD lexical | SD semantic | PP semantic | PP lexical |
|  | similarity | set | set | similarity |
| TriviaQA(Mistral) | SD lexical | PP lexical | SP semantic | SD semantic |
|  | similarity | similarity | set | set |
| SQuAD(Llama) | SP lexical | EFA semantic | - | - |
|  | similarity | set |  |  |
| SQuAD(Flan-ul2) | SP lexical | - | - | - |
|  | similarity |  |  |  |
| SQuAD(Mistral) | SP lexical | EFA semantic | - | - |
|  | similarity | set |  |  |
| CoQA(Llama) | SD lexical | EFA semantic | SD semantic | SR lexical |
|  | similarity | set | set | similarity |
| CoQA(Flan-ul2) | SD lexical | EFA semantic | SD semantic | SP lexical |
|  | similarity | set | set | similarity |
| CoQA(Mistral) | SD lexical | SD semantic | EFA semantic | EFA lexical |
|  | similarity | set | set | similarity |
| NQ(Llama) | PP lexical | SD semantic | SD lexical | SP lexical |
|  | similarity | set | similarity | similarity |
| NQ(Flan-ul2) | SR semantic | SD lexical | SP lexical | PP lexical |
|  | set | similarity | similarity | similarity |
| NQ(Mistral) | PP lexical | SD semantic | SD lexical | SP lexical |
|  | similarity | set | similarity | similarity |
| CNN(Pegasus) | SD lexical | EFA lexical | SR lexical | SP lexical |
|  | similarity | similarity | similarity | similarity |
| CNN(BART) | SR lexical | SP lexical | EFA lexical | SP semantic |
|  | similarity | similarity | similarity | set |
| XSUM(Pegasus) | SD lexical | EFA semantic | PP lexical | SD semantic |
|  | similarity | set | similarity | set |
| XSUM(BART) | SR lexical | SP lexical | EFA lexical | SP semantic |
|  | similarity | similarity | similarity | set |

We follow previous works [], which used $1000$ datapoints for hyperparameter tuning, to train our Logistic Regression Classifier and the rest of them were used for evaluation. As such, in Table 8 in the appendix, we show that our method is quite performant even with fewer training datapoints. For each of the prompt perturbations specified above, we use five generations for each perturbation for more robust evaluation. All results are averaged over three runs. We use zero-shot prompting for the datasets with context. For TriviaQA, Flan-ul2 and Mistral-7B-Instruct-v0.2 also worked well with zero shot prompting while Llama-2-13b chat was performant
with a two-shot prompt. For NQ, we used a five shot prompt. The details about the prompts used are provided in the Appendix A.
We used internally hosted models to generate the responses. Thus, we used V100s GPUs for the feature extraction step once the responses were generated. The logistic regression model was trained on an intel core CPU.

We consider methods proposed in recent works [] and [] which are state-of-the-art as the baselines. [] proposed computing the number of semantic sets and lexical similarity metrics from the generated outputs. [] use eigen value, eccentricity and degree metrics inspired from spectral clustering to estimate the uncertainty of the model. We use the open source code provided by the authors of [] for comparing with the baselines (^1^11https://github.com/zlin7/UQ-NLG/ The results are different in some cases from those reported in their paper possibly because of different random splits and different LLMs used, since we did run the provided code.).

### 4.1 Confidence Estimation

We use two metrics to evaluate effectiveness of the models: Area under the receiver operating characteristic (AUROC) curve which computes the model’s discrimination ability for various thresholds. The curve is plotted by varying the thresholds of the prediction probabilities of the model and the false positive rate and the true positive rate form the X and the Y axes. The area under this curve is called the AUROC.
An accuracy rejection curve can also be plotted by increasing the rejection threshold gradually and plotting the model’s average accuracy at that threshold. The area under this curve is called AUARC [].

In Table 2, we see that our method quite consistently outperforms all baselines on AUROC. For estimating the confidence of Llama’s responses on TriviaQA, our model is better than the best baseline by $11$ percentage points. We are also able to estimate the confidence on the SQuAD dataset using Mistral by $14$ percentage points better than the closest competitor. Qualitatively similar results are seen for the SQuAD dataset using Flan-ul2 (better by $10$ percentage points) and for the CoQA and NQ datasets using Llama (better by $15$ and $10$ percentage points respectively). Our results on the summarization datasets using LLMs that excel at summarization (viz. Pegasus and BART) we see again that we are either better or at least competitive.

Our performance is also superior to the baselines in most cases on the AUARC metric in Table 3. Our performance on Llama’s generations based on the SQuAD dataset exceeds the best baseline’s performance by $31$ percentage points. In the case of Mistral’s performance on TriviaQA and Flan-ul2’s generations on CoQA, we are as good as the baseline. We are worse than the baseline on Mistral’s generations of CoQA, where our AUROC was also minimally better than the best baselines. In all other instances, our performance is better than others by $1$ to $4$ percentage points.

We believe these improvements can be attributed to our constructed features and our framework in general. Hence, in the next section we try to ascertain which features for which datasets and LLMs played an important role in predicting the confidences accurately. Note that such an analysis with high confidence is possible because our trained model is interpretable. We also tried to pass the tokenized input prompt as additional features (maximum length 256) to our logistic model, however, the improvements were minimal at best and in some cases the performance even dropped possibly because of the model overfitting given that there were now 100s of features. Hence, we do not report these results, although passing the input prompt is still a possibility in general.

**Table 5: AUROC of the logistic confidence model for one LLM applied to another on a given dataset. As can be seen our confidence models transfer quite well based on AUROC.**
| Dataset | Source LLM | AUROC Self | Target LLM 1 AUROC | Target LLM 2 AUROC |
| --- | --- | --- | --- | --- |
| TriviaQA | Llama | 0.88 | 0.94 (Flan-ul2) | 0.80 (Mistral) |
| Flan-ul2 | 0.94 | 0.87 (Llama) | 0.80 (Mistral) |  |
| Mistral | 0.81 | 0.84 (Llama) | 0.91 (Flan-ul2) |  |
| SQuAD | Llama | 0.83 | 0.81 (Flan-ul2) | 0.80 (Mistral) |
| Flan-ul2 | 0.8 | 0.79 (Llama) | 0.78 (Mistral) |  |
| Mistral | 0.84 | 0.82 (Llama) | 0.83 (Flan-ul2) |  |
| CoQA | Llama | 0.92 | 0.79 (Flan-ul2) | 0.78 (Mistral) |
| Flan-ul2 | 0.87 | 0.87 (Llama) | 0.81 (Mistral) |  |
| Mistral | 0.81 | 0.88 (Llama) | 0.86 (Flan-ul2) |  |
| NQ | Llama | 0.85 | 0.91 (Flan-ul2) | 0.83 (Mistral) |
| Flan-ul2 | 0.93 | 0.83 (Llama) | 0.82 (Mistral) |  |
| Mistral | 0.83 | 0.85 (Llama) | 0.90 (Flan-ul2) |  |
| CNN | Pegasus | 0.77 | 0.57 (BART) | - |
| BART | 0.57 | 0.77 (Pegasus) | - |  |
| XSUM | Pegasus | 0.73 | 0.58 (BART) | - |
| BART | 0.57 | 0.71 (Pegasus) | - |  |

**Table 6: AUARC of the logistic confidence model for one LLM applied to another on a given dataset. As can be seen our confidence models transfer quite well based on AUARC as well.**
| Dataset | Source LLM | AUARC Self | Target LLM 1 AUARC | Target LLM 2 AUARC |
| --- | --- | --- | --- | --- |
| TriviaQA | Llama | 0.83 | 0.74 (Flan-ul2) | 0.64 (Mistral) |
| Flan-ul2 | 0.74 | 0.83 (Llama) | 0.64 (Mistral) |  |
| Mistral | 0.64 | 0.83 (Llama) | 0.73 (Flan-ul2) |  |
| SQuAD | Llama | 0.68 | 0.62 (Flan-ul2) | 0.63 (Mistral) |
| Flan-ul2 | 0.96 | 0.89 (Llama) | 0.91 (Mistral) |  |
| Mistral | 0.96 | 0.90 (Llama) | 0.91 (Flan-ul2) |  |
| CoQA | Llama | 0.71 | 0.79 (Flan-ul2) | 0.61 (Mistral) |
| Flan-ul2 | 0.80 | 0.70 (Llama) | 0.61 (Mistral) |  |
| Mistral | 0.61 | 0.69 (Llama) | 0.79 (Flan-ul2) |  |
| NQ | Llama | 0.45 | 0.46 (Flan-ul2) | 0.42 (Mistral) |
| Flan-ul2 | 0.47 | 0.45 (Llama) | 0.42 (Mistral) |  |
| Mistral | 0.42 | 0.45 (Llama) | 0.46 (Flan-ul2) |  |
| CNN | Pegasus | 0.74 | 0.34 (BART) | - |
| BART | 0.34 | 0.74 (Pegasus) | - |  |
| XSUM | Pegasus | 0.27 | 0.34 (BART) | - |
| BART | 0.35 | 0.25 (Pegasus) | - |  |

### 4.2 Confidence Model Interpretability and Transferability

Interpretability: We now study which features in our logistic model were instrumental for accurate confidence estimation. In Table 4, we see the top four features for each dataset-LLM combination. Blanks indicate that there were no features at that rank or lower where their logistic coefficient was greater than $1e^{-4}$. As can be seen the simplest feature SD plays a role in many cases. This indicates that variability of output for the same input prompt is a strong indicator of response correctness. Moreover, other features such as SP and EFA are also crucial in ascertaining confidence as seen in particular for the SQuAD dataset as well as the summarization datasets. This points to order bias when looking at contexts and brittleness to redundant information being also strong indicators of response accuracy. PP and SR also play a role in some cases, where they are more crucial for datasets with no contexts such as TriviaQA and NQ. This makes sense as the specific question is more important here in the absence of context and hence the absence of also other features such as SP and EFA. Both the lexical similarity and semantic set featurizations seem to be important in estimating confidence.

Looking across the datasets and LLMs we see an interesting trend. It seems that for a given dataset different LLMs have similar features that appear to be important. For instance, SP lexical similarity is the top feature for all three LLMs on SQuAD, while EFA based feature also appears for Llama and Mistral. For TriviaQA, SD and PP appear for all three models. For CoQA, SD and EFA appear. While for NQ, PP and SD appear as important for all the models. This trend points towards an interesting prospect of applying a confidence estimator of one LLM to other LLMs on a given dataset. As such, we could have a universal confidence estimator just built for one of the LLMs that we could apply across others with reasonable assurance. We explore this exciting possibility in the next part.

Transferability: Given the commonality between the important features across LLMs for a dataset we now try to test how well does our logistic confidence model for one LLM perform in estimating confidences of another LLM. As seen in Tables 5 and 6 our confidence models are actually quite transferable as they perform comparably or even sometimes better on the other LLMs than the LLM they were built for. This is particularly true for Mistral where, its confidence model performs better for the other two LLMs than itself even coming close in performance to their own confidence models in many cases.

This suggests that we could apply our approach to one LLM and then use the same confidence model to evaluate responses of other LLMs without having to build individual models for them. It would be interesting to further stress test this hypothesis in the future with more LLMs and datasets. Nonetheless, even in the current setup – of five LLMs and six datasets – this observation is interesting and useful.

## 5 Discussion

In summary, we have provided an extensible framework for black-box confidence estimation of LLM responses by proposing novel features that are indicative of response correctness. By building an interpretable logistic regression model based on these features we were able to obtain state-of-the-art performance in estimating confidence on six benchmark datasets (CoQA, SQuAD, NQ, TriviaQA, CNN Daily and XSUM) and using five powerful open source LLMs (Llama-2-13b-chat, Mistral-7B-Instruct-v0.2, Flan-ul2, Pegasus-large and BART-large). The interpretability of our confidence model aided in identifying features (viz. SD, SP, EFA,PP) that were instrumental in driving its performance for different LLM-dataset combinations. This led to the interesting realization that many of the features crucial for performance were shared across the confidence models of different LLMs for a dataset. We thus tested if the confidence models generalized across LLMs for a dataset and found that it indeed was the case leading to the interesting possibility of having an *universal* confidence model trained on just a single LLMs responses, but applied across many others.

Owing to the supervised nature of training the confidence model, one limitation of our approach is that at least some of the model’s generations must be close to the ground truth for us to obtain a reasonable confidence estimator. Another limitation is that the results and insights were obtained based on datasets in English, but these insights might vary when looking at datasets in other languages. More varied tasks and models could be tested upon in the future. We used rouge to test accuracy of generations consistent with previous works, however, rouge, like also other NLP metrics, can be error prone. In terms of broader impact, our approach can be widely applied as it is simple and works with just black-box access to the LLM. Access to logits or internals of the model are not required. However, our estimates although accurate can be imperfect and this should be taken into account when using our approach in high stakes applications involving LLMs. One should also be cognizant of adversaries aware of our features trying to induce misplaced trust in LLMs they create or prefer.

Given the extensibility of our framework, in the future, it would be interesting to add more features as LLMs evolve. One class of such features might be those where the correctness of a response is checked through creating questions that are (causally) related to the original question and context, and seeing how the response varies by asking this question by itself as opposed to in conjunction with the original question and response. Such and other strategies may help in generalizing these confidence estimators also across datasets something that has been seen when we have additional access to logits of LLMs.

## Appendix A Prompt Design

Prompts for TriviaQA:

- •
Flan-ul2 model:
Answer the following question in less than 5 words
Q: {question}
A:
- •
Llama-2-13b-chat model
Answer these following question as succinctly as possible in less than 5 words
Q: In Scotland a bothy/bothie is a?
A: House
Q: Who is Posh Spice in the spice girls pop band?
A: Victoria Beckham
Q: {question}
A:
- •
Mistral-7B-Instruct-v0.2 model


 Answer the following question as succinctly as possible in plain text and in less than 5 words. question 


Prompts for CoQA

- •
Flan-ul2 model and Llama-2-13b-chat model:
Provide an answer in less than 5 words for the following question based on the context below:
context: {context}
Question: {question}
Answer:
- •
Mistral-7B-Instruct-v0.2 model


 Provide an answer in less than 5 words for the following question based on the context below:
context: {context}
Question: {question}
Answer: 


Prompts for SQuAD

- •
Flan-ul2 model and Llama-2-13b-chat model:
Provide an answer for the following question based on the context below, in less than 5 words:
- •
Mistral-7B-Instruct-v0.2 model


 Provide an answer for the following question based on the context below, in less than 5 words:
context: {context}
Question: {question}
Answer: 


Prompts for NQ: For all the 3 models we used the following prompt:
Here are 5 Example Question Answer pairs:
Question: who makes up the state council in russia
Answer: governors and presidents
Question: when does real time with bill maher come back
Answer: November 9, 2018
Question: where did the phrase american dream come from
Answer: the mystique regarding frontier life
Question: what do you call a group of eels
Answer: bed
Question: who wrote the score for mission impossible fallout
Answer: Lorne Balfe
Now answer the following Question succinctly, similar to the above examples:
Question: {question}
Answer:

**Table 7: Percentage of prompt perturbations entailed by the original prompt for the SQuAD dataset. This dataset also has context unlike some of the other Q&A datasets and hence, is a more challenging case of our features to maintain semantics. As can be seen our perturbations produce the intended effect of maintaining the semantics of the original prompt in most cases.**
| Paraphrasing | Sentence Permutation | Entity Frequency Amplification | Stopword Removal |
| --- | --- | --- | --- |
| $99.81$% | $99.23$% | $99.66$% | $99.12$% |

**Table 8: Below we see how the AUROC, AUARC values vary with different number of samples used to train our logistic regression model for some of our datasets. As can be seen our uncertainty estimation procedure is performant even with fewer samples for training.**
| Dataset | LLM | $250$ samples | $500$ samples | $1000$ samples (results in main paper) |
| --- | --- | --- | --- | --- |
| TriviaQA | Llama | 0.83, 0.80 | 0.86, 0.81 | 0.88, 0.83 |
| Flan-ul2 | 0.95, 0.73 | 0.95, 0.74 | 0.95, 0.74 |  |
| Mistral | 0.80, 0.63 | 0.80, 0.63 | 0.81, 0.64 |  |
| SQuAD | Llama | 0.8, 0.65 | 0.81, 0.66 | 0.83, 0.68 |
| Flan-ul2 | 0.76, 0.91 | 0.78, 0.94 | 0.8, 0.96 |  |
| Mistral | 0.79, 0.90 | 0.81, 0.93 | 0.84, 0.96 |  |
| CoQA | Llama | 0.91, 0.70 | 0.92, 0.71 | 0.92, 0.71 |
| Flan-ul2 | 0.86, 0.79 | 0.87, 0.80 | 0.87, 0.80 |  |
| Mistral | 0.80, 0.60 | 0.81, 0.61 | 0.81, 0.61 |  |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: * Do you expect the gains over the baselines to remain as large using a different evaluation criteria, such as using e.g., Llama 3 70B as a judge of correctness? 
* Do you expect the results to extend to frontier systems (where we only have black-box access)?
soundness: 3
strengths: * This paper studies an important problem; assessing uncertainty in the language model’s output space. And they do so for open-ended generation tasks like summarization where uncertainty quantification is much harder (since you cannot look at the entropy at a specific token)
*The authors have a wide-range of experimental analysis in different settings
* Some of the findings I thought would be useful to the community — for example, I thought it was interesting that repeated sampling from the original prompt yields more useful features than perturbing the prompt (and I would be interested to see how this can be extended in a maximally sample-efficient way).
summary: This paper studies the problem of uncertainty quantification in language model responses using only black-box access to the language model. In order to quantify uncertainty, the authors study many different ways of varying outputs: they sample from the same prompt using different strategies, paraphrase the prompt via backtranslation, reorder sentences, repeat sentences, remove stopwords, and measure whether parts of the outputs entail other parts using an NLI model. They then convert the outputs of these perturbations into features (using different methods for different sources of variation), and finally train a logistic regression model on these features to predict “confidence”. The paper then runs experiments on Flan-ul2, Mistral 7B instruct, and Llama 2 13B. They find that their method outperforms baselines (such as studying semantic sets and lexical similarity, measured by AUROC), and that the feature that matters most in the regression model is typically the number of “semantically” equivalent sets of outputs sampled from the prompt.
weaknesses: * The primary weakness is the use of rouge score (a metric on top of n-gram overlap with ground truth summaries) for evaluation. Many of the perturbations, such as mixing up or repeating sentences with entities, help capture how entities are processed into the output (which rouge disproportionately captures), rather than actual correctness. 
* The procedure the authors propose is very inefficient; it requires many repeated samples from the language model. It also is not clear if this extra compute is deployed optimally; for example, it could be better to deploy more compute into stochastic decoding. 
* There are a number of ways the presentation could be improved. For example, the paper introduces the notion of confidence in the intro using lots of notation, but uses no notation to explain the (somewhat complicated) methods of extracting features for the logistic regression classifier. 
* I’d also encourage the authors to draw from some related work in estimating the confidence / probability of correct (e.g., [1] and [2] have studied this formally). 

[1] Bartlett and Wegkamp, 2008. Classification with a Reject Option using a hinge loss
[2] Geifman and El-Yaniv, 2017. Selective Classification for Deep Neural Networks

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Why not include scores like EigenValue, Eccentricity, or Degree in the logistic regression inputs?
2. Same-dataset, cross-model performance seems generally good. E.g. the same logistic regression weights can be reused for the same dataset with different models. Notably missing seems to be same-model, cross-dataset results. How important is it really to train this logistic regression model as compared to using, say, SP lexical similarity for everything?--or one fixed set of weights for everything?
soundness: 2
strengths: 1. Simple, extensible framework for wrapping black-box confidence scoring techniques into a meta-scorer, presented with several existing and novel confidence scoring techniques.
2. Useful interpretability results.
3. Good communication of approach, drawbacks, and strengths.
summary: edit Dec. 2: Score change from 5 to 6.

The authors introduce a framework to obtain confidence or P(correctness) values to score sampled responses from a black-box model (no access to weights, logits, or log-probs). The method is essentially using several different strategies, including non-deterministic sampling, prompt modification, and response splitting, to produce a small set of responses, and determining consistency among them via semantic and lexical metrics. The strategy-metric pairs each receive their own score, and these scores are used with a logistic regression model to obtain the final confidence (calibrated to a dataset-model pair). This method extends and generalizes work such as "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." (Kuhn et al., 2023).

The authors benchmark on a mid-sized set of LLMs and datasets, obtaining good performance compared to other black-box baselines, and demonstrate that the logistic regression model is able to show the relative importance of different metrics in determining confidence. Furthermore, they show that the trained logistic regression models may generalize on the same dataset across the results from different LLMs.
weaknesses: 1. In 083, it is claimed that because the models used to produce predictions are simple, the confidence estimates will be well-calibrated. I think this claim needs support; in particular, the paper might make use of ECE (expected calibration error) and/or other calibration metrics to assess the calibration of different approaches including the newly proposed approach.
2. Results are hard to contextualize without error bars.
3. Baselines are mostly focused on previous black-box sampling-based baselines. While these are useful, many useful baselines are in fact missing. For example, on black-box models, elicited verbalized confidence (Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback, Tian et al. 2023) could be used. Also, why not try a frequency-estimated semantic entropy of the equivalence sets (Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation, Kuhn et al. 2023), instead of just the number of sets? Furthermore, open-source/whitebox approaches such as P(true) or finetuned P(IK) (both from Language Models (Mostly) Know What They Know, Kadavath et al., 2022) could easily be used on an auxiliary open-weight model (e.g. assessing the accuracy of a black-box response with a different open-source model). All of these metrics seem that they could be added as baselines and as inputs in the logistic regression finetune.
4. Scope of model evaluation is limited. This limits the impact of the conclusions, although it may not be addressable due to cost limitations.

Small nitpicks:
- Spelling and grammar: "aver to these values as a features (80-81), "complimentary ways" --> "complementary ways" (169), "greedy ," (149), "Since, " (083), etc.
- Communication: I think 3.1 should not be entitled "prompt perturbations" as not all of the methods for obtaining multiple different answers/answer portions to compare are actually prompt perturbations. Prompt perturbations, Stochastic Decoding, and split response consistency should maybe be referred to collectively as something like "Elicitation of multiple black-box responses" or similar.
- Communication: (383) "we use two metrics to evaluate effectiveness of the models: {description of metric 1}. {Description of metric 2}" should likely be a list with (a), (b), instead of two distinct sentences.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Stochastic decoding and split response consistency really aren't prompt perturbations since they change the output, not the input. It seems like these should be pulled out into a separate section.
 - For paraphrasing, it was noted that Helsinki-NLP to do En->Fr->En did not preserve semantics. Why was Helsinki-NLP used?  It seems like using an LLM to do the translation or paraphrase could have worked - was this considered?
 - Sentence permutation could make the text invalid due to unresolved anaphora - can you comment on whether this was a problem?
 - Entity frequency amplification: why is there such an emphasis on named entities? I could imagine EFA and SP be generally applicable to any sentence. Is there some sort of empirical evidence that entities need to be treated specially?
 - For SRC, how was the NLI between the output computed?  This only works for long responses since the output needs to be split up into multiple parts? Why is NLI not computed on different samples instead? How does this relate to maieutic prompting (https://arxiv.org/pdf/2205.11822)?
 - The section on featurization (section 3.2) is a bit hard to follow; it would be nice to have an example.
 - Why do you think that the same features transfer across LLMs? Is it because there is a model-agnostic notion of example difficulty that is being captured here, or that models tend to make the same type of errors?
 - How important is the linear model assumption? I understand the desire for interpretability, but if you just cared about increasing AUROC, how far could you get by using non-linear models (for exapmle, https://arxiv.org/pdf/2006.09462 used random forests).
 - It would be interesting to compare your method to methods that use log probabilities (of other models):
   * https://arxiv.org/abs/2006.09462
 - While other papers that aggregate (e.g., https://arxiv.org/abs/2306.13063) are discussed in the related work, it would have been nice to see a more quantitative comparison.
 - Minor: use \citep (e.g., Goodfellow et al. (2016) in the first sentence) when the citation is used as a parenthetical.
 - 169: complimentary => complementary
soundness: 3
strengths: - The problem of producing confidence estimates for language models is an important problem.
 - The idea of aggregating multiple different sources of information across prompts is a natural direction which seems fruitful.
 - Studying the transferrability across language models is important for having general-purpose calibrators.
summary: - This paper seeks to provide uncertainty measures for language models given on prompting access to them.
 - The basic method is to perturb a prompt using various heuristics (e.g., paraphrasing, duplicating sentences) and generating a set of features that feeds into a logistic regression.
 - Evaluation is done on 4 QA tasks on 3 models (Flan-ul2, Llama-13b, Mistral-7b), and there are gains over baselines.
weaknesses: - The regime in which this paper operates could be more realistic. First, the models considered are relatively small models. Furthermore for these models, we actually have full white-box access, so evaluating methods that are meant for black-box access doesn't seem as well motivated. It would have been nice to see evaluations on closed API models (e.g., GPT-4, Claude, etc.). The contribution of this paper is fundamentally empirical, so the regime matters here.
 - The particular types of perturbations (e.g., paraphrasing) seem ad-hoc: why were these chosen and not others?  For each particular choice (e.g., paraphrasing), I also wonder about the execution - it seemed like relatively weak models were used and the paraphrasing wasn't effective. Also, regardless of what perturbations are chosen, there should be more analysis done about the effectiveness of each one (both in terms of downstream performance but in terms of whether the implementation of the pertubation actually produced the desired effects with valid outputs - e.g., paraphrasing where we don't just drop valuable content). I understand that the pitch is that this is a framework, but the framework is not novel; the contribution should be the choices of perturbation that lead to good results.
 - The cost of the method is not discussed. Compared to having to decode from the language model once, it seems like the method here is quite expensive and complicated; to know whether it is worth it, it would be good to document the costs, and in general, explore the cost-AUROC tradeoff.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Perhaps the author could comment on the robustness of the featurization steps. How automatized is the process? How well does it work without intervention? Would it be possible to provide more evidence that the method is robust?
- The few instances where the presented approach is not better than the others are not the same for the AUROC and AUARC metrics. Why do you think this is the case? Combined with the fact that most often the improvement is only a few percentages, it makes me wonder whether averaging the results over three runs is enough. Would it be possible to use more runs and add standard deviations to make results more significant?
soundness: 3
strengths: - The techniques proposed in this paper are black-box, which make the work appealing.
- Results appear better than competitors across the board.
summary: This work proposes a framework for black-box detection of correct vs incorrect LLM responses. The authors propose a set of input manipulation and decoding techniques to form a feature space. Using binary correctness labels, they train a logistic regression to derive confidence estimates that an answer is correct. Experiments shows good performance compared to baselines with respect to two metrics, for several QA datasets and LLMs. The simplicity of the logistic regression also consents to rank the importance of each technique on each dataset.

The paper is experimental in nature and is not backed by theoretical results.
weaknesses: - No theory to support the methodology. The proposed techniques feel reasonable, but there is no principle reason why they would strongly correlate with the correctness of the output in general. Clearly, using multiple techniques at the same time as authors do tend to mitigate this problem.
- Not fully clear to me how robustness some of the featurization steps are, and how much manual intervention is needed to make things work for all dataset.

Meta-review: The authors propose a method for calibrating LLM outputs given black-box query access. Their method involves 1) obtaining a set of generations by introducting heuristics perturbations in the input prompt (e.g., through paraphrasing) as well as directly in the output, 2) extracting some hand-designed features such as measures of diversity in these generations, and 3) learning a logistic regression model on top of these features to predict whether the LLM output is likely to be correct. 

Reviewers thought that this was a natural method for an important problem. However, as the paper is largely making an empirical contribution, the reviewers also found that several important aspects to be missing or not fully addressed. For example: the regime of models studied (which the authors partially addressed during the rebuttal); a deeper analysis of which heuristics worked well, both in terms of whether they achieved what they set out to do (e.g., whether paraphrasing retained the relevant information in the prompt) and whether they contributed to accurate calibration; a discussion about computational cost relative to other methods, given that the proposed method is expensive; and comparisons to other baselines, both white-box and black-box, in terms of cost and effectiveness. 

As such, I am unable to recommend acceptance. I encourage the authors to incorporate the reviewer feedback into their next submission.

Decision: Reject