Large Language Models Must Be Taught to Know What They Don’t Know
Authors: Sanyam Kapoor
Nate Gruver
Manley Roberts
Katherine M. Collins
Arka Pal
Umang Bhatt
Adrian Weller
Samuel Dooley
Micah Goldblum
Andrew Gordon Wilson
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
calibration
uncertainty estimation
prompting
Primary area: natural_language_processing

Abstract: When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is sufficient to produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient to achieve good calibration and then show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We show that a thousand graded examples are sufficient to outperform baseline methods and that training through the features of a model is necessary for good performance and tractable for large open-source models when using LoRA. We also investigate the mechanisms that enable reliable LLM uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators, applicable not just to their own uncertainties but also the uncertainty of other models. Lastly, we show that uncertainty estimates inform human use of LLMs in human-AI collaborative settings through a user study.

Large Language Models Must Be Taught to Know What They Don't Know
Sanyam Kapoor* New York University
Nate Gruver" New York University
Manley Roberts Abacus AI
Katherine Collins Cambridge University
Arka Pal
Umang Bhatt New York University
Abacus Al
Adrian Weller Cambridge University
Samuel Dooley
Micah Goldblum Columbia University
Andrew Gordon Wilson New York University
Abacus AI
Abstract
When using large language models (LLMs) in high-stakes applications, we need know when we can trust their predictions. Some works argue that prompting high- performance LLMs is sufficient produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient achieve good calibration and then show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We show that thousand graded examples are sufficient to outperform baseline methods and that training through the features of model is necessary for good performance and tractable for large open-source models when using LoRA. We also investigate the mechanisms that enable reliable LLM uncertainty estimation, finding that many models can be used ceneral-purposs uncertainty estimators, applicable not just to their own uncertainties but also the uncertainty of other models. Lastly, show that uncertainty estimates inform human use of LLMs in human-Al collaborative settings through a user study.
Introduction
"I have high cortisol but low ACTH on a dexamet.hasone suppression test. What should I do?" If the answer to such question is given without associated confidence, is not actionable, and if the answer is presented with erroneously high confidence, then acting the answer is dangerous. One of the biggest open questions about whether large language models (LLMs) can benefit society and reliably be used for decision making hinges on whether or not they accurately represent uncertainty
the correctness of their output.
There is anything but consensus on whether LLMs accurately represent uncertainty, or even how we should approach uncertainty representation with language models. Claims regarding language models' ability to estimate uncertainty vary widely, with some works suggesting that language models are increasingly capable of estimating their uncertainty directly through prompting, without any fine-tuning or changes to the training data [25, 51], and others suggesting that LLMs remain far too overconfident in their predictions [60, 61]. The task of uncertainty estimation in LLMs is further exacerbated by linguistic variances freeform generation, all which cannot be exhaustively
"Equal contribution. Order decided by coin flip. Correspondence to: sanyam@ nyu.edu & nvg7279@nyu.cc
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

What's the key to delicious pizza sauce?
Graded Dataset
Fine-Tuning
Zeno Shot
Classifler
Add non-toxic glue for
AM
Qreitin
Queetie
t
Verbelized
lackiness
Quastion
Sampling
What's your confidencee?
Naseett
Fine- fuueed
Yee
la the comred!
100%
20%
50% 70%
ECE
AUROC T
Figure 1: Large language models struggle to assign reliable confidence estimates to their generations. We study the properties of uncertainty calibration in language models, and propose fine-tuning for better uncertainty estimates using graded dataset of generations from the model. We evaluate our methods on a new open-ended variant of MMLU [18]. We show that fine-tuning improves expected calibration error (ECE) and area under the receiver operating characteristic curve (AUROC) compared to commonly-used baselines. Error bars show standard deviation over three base models (LLaMA-2 13/7B and Mistral 7B) and their chat variants.
accounted for during training. LLM practitioners are therefore faced with the challenge of deciding which estimation method to use.
One particular dichotomy in uncertainty estimation methods for language models centers around whether the estimates are black- or white-box. Black-box estimates do not require training and can be used with closed-source models like GPT-4 [1] or Gemini [48], while white-box methods require training parameters on calibration dataset. Although black-box estimates have become popular with the rise of restricted models, the increased availability of strong open-source models, such as LLaMA [53] or Mistral [24], has made more effective white-box methods more accessible.
In this paper, we perform a deep investigation into uncertainty calibration of LLMs, with findings that advance the debate about necessary interventions for good calibration. In particular, we consider whether it's s possible to have good uncertainties over correctness (rather than tokens) without inter- vention, how we can best use labeled correctness examples, how well uncertainty generalizes across distribution shifts, and how we can use LLM uncertainty to assist human decision making.
First, we find that fine-tuning for better uncertainties (Figure 1) provides faster and more reliable uncertainty estimates, while using a relatively small number of additional parameters. The resulting uncertainties also generalize to new question types and tasks, beyond what is present in the fine-tuning dataset. We further provide a guide to teaching language models to know what they don't know using a calibration dataset. Contrary to prior work, we start by showing that current zero-shot, black-box methods are ineffective or impractically expensive in open-ended settings (Section 4). We then show how to fine-tune language model for calibration, exploring the most effective parameterization (c.g. linear probes vs LoRA) and the amount of the data that is required for good generalization (Section 5). To test generalization, calibration data as well as questions that test robustness to significant distribution shifts. Lastly, we consider the underlying mechanisms that enable fine-tuning LLMS to estimate their uncertainties, showing ultimately that models can be used not just to estimate their own uncertainties but also the uncertainties of other models (Section 6). Beyond offline evaluation, if language models are have broad societal impact, it will be through assisting with human decision making. We conduct a user study demonstrating ways LLM uncertainty can affect Al-human collaboration (Section 7).2
evaluate uncertainty estimates on questions with similar formatting the
2 Related Work
As generative models, LLMs naturally express a distribution over possible outcomes and should capture variance in the underlying data. On multiple-choice tests, where the answer a single token, an LLM's predicted token probabilities can lead to a calibrated distribution over the answer choices [43]. When answers consist of entire sentences, however, language model likelihoods become a less reliable indicator of uncertainty because probabilities must be spread over many phrasings of the same concept. Kuhn et [30] attempt to mitigate this issue by clustering semantically equivalent answers. However, these methods are hindered by their substantial computational overhead. Accounting for equivalent phrasings of the same semantic content requires enumerating a large space of sentences and clustering for semantic similarity with auxiliary model.
Phtpps: / / thub.

---

Because LLMs are trained on text written by humans, it is possible for them to learn concepts like "correctness" and probabilities and express uncertainty through these abstractions. Leveraging this observation, Kadavath et al. [25] and Tian et al. [51] show that careful prompting can produce uncertainty estimates in text that grow more calibrated as model capabilities increases. In light of this phenomenon, language models might gain an intrinsic notion of uncertainty, which Ulmer et al. [54] use to generate per-task synthetic training data for an auxiliary confidence model. In the same vein, Burns et al. [9] and Azaria and Mitchell [4] find that pre-trained models have hidden representations which are predictive of truthfulness and use linear probes to classify a model's correctness.
While these studies suggest a promising trend towards calibration, we find that the story is slightly more complicated. Black- methods often fail to generate useful uncertainties for popular open- source models, and a careful fine-tuning intervention is necessary. In this way, our findings are closer to those of Xiong et al. [60], who show that zero-shot uncertainty estimates have limited ability to discriminate between correct and incorrect answers, even when used with the best available models (e.g., GPT-4). We go further by showing that black-box methods struggle on open-ended generation, which is both practically important and defined by different challenges than multiple choice evaluations from prior work. Moreover, while others have focused on improving black-box methods [30, 51, 60], we embrace open-source models and their opportunities for fine-tuning, showing that we can maintain the speed of prompting methods while dramatically boosting performance.
Our work also contrasts with prior work on fine-tuning for uncertainties in several key ways. While we build on prior work from Lin et al. [33] and Zhang et al. [63] that poses uncertainty estimation as text completion on a graded dataset, we introduce several changes to the fine-tuning procedure, such as regularization to maintain similar predictions to the base model, and provide extensive ablations that yield actionable insights. For example, we show that, contrary to prior work [4], frozen features are typically insufficient for uncertainty estimates that generalize effectively, and that fine-tuning on as few as 1000 graded examples with LoRA is sufficient to generalize across practical distribution shifts. Also unlike prior work, we provide many insights into the relative performance of fine-tuning compared to black-box methods, introducing a new open-ended evaluation and showing that it displays fundamentally different trends than prior work on multiple choice questions. Although Kadavath et al. [25] also considers calibration for multiple choice questions, many of our conclusions differ. For example, while Kadavath et al. [25] suggest that language models are strongest when evaluating their own generations and subsequently posit that uncertainty estimation is linked to self-knowledge, we find that capable models can readily learn good uncertainties for predictions of other models without any knowledge of their internals. Lastly, while many works motivate their approach with applications to human-Al Al collaboration, none of them test their uncertainty estimates on actual users, we do here.
3 Preliminaries
Question answering evaluations. In all experiments, we use greedy decoding to generate answers conditioned on questions with few-shot prompts. We then label the generated answers as correct or incorrect and independently generate P(correct) using one of the uncertainty estimators. For evaluation, we primarily use the popular MMLU dataset [18], which covers 57 subjects including STEM, humanities, and social sciences. Crucially, however, we expand the original multiple choice (MC) setting with a new open-ended (OE) setting. In the open-ended setting, we do not provide answer choices, and the language model must generate an answer that matches the ground truth answer choice. We determine a correct match by grading with a strong auxiliary language model (Appendix A.2). We verify that grading via language models provides a cheap and effective proxy for the gold standard human grading (Appendix A.3), consistent with related findings [10].
Metries. A model that assigns percentage p an answer is well-calibrated if its answer is correct P percent of the time it assigns that confidence. Calibration is typically measured using expected calibration error (ECE) [37], which compares empirical frequences with estimated probabilities through bimning (Appendix A.4). A lower ECE is better, and an ECE of 0 corresponds to perfectly calibrated model. In addition to calibration, we measure the area under the receiver operating characteristic curve (AUROC) of the model's confidence. High AUROC indicates ability to filter answers likely to be correct from answers that are likely to be incorrect, a setting typically called selective prediction.
3

---

Temperature scaling, Temperature scaling [42, 17] improves the calibration of a classifier by scaling its logits by T (where T is the temperature) before applying the softmax function. high temperature scales the softmax probabilities towards uniform distribution, while a low temperature collapses the distribution around the most probable output. The temperature parameter is learned on held-out data, typically taken from the same distribution as the training set.
Do We Get Good Uncertainties Out-of-the-Box?
In this section, we focus on black-box' methods for estimating a language model's uncertainty. Due to computational cost, we focus methods that require single sample forward pass and only consider sampling-base based methods in the next section.
For multiple choice tasks, language model's distribution over answers is categorical distribution as each answer choice is a single token. Early work on LLMs, such as GPT-3, showed that this distribution is often poorly calibrated [18]. Fundamentally, however, maximum likelihood training should encourage calibration over individual tokens [15], and the calibration recent LLMs appears to improve in proportion with their accuracy [43].
In open-ended generation, on the other hand, answers not limited individual tokens nor prescribed set of possibilities, which introduces multiple sources of uncertainty. The probability assigned to an answer can be low not just because it's unlikely to correspond the correct answer conceptually but because there are multiple possible phrasings that must receive probability mass (and normalization is intractable), or because the answer represents an unusual phrasing of the correct information, and the uncertainty is over the probability of a sequence of tokens and not correctness. For example, imagine a multiple-choiee test in which we add an additional answer choice that is synonym of another. A sensible language model would assign equal likelihood to each choice, lowering the probability it assigns to either individually. In open-ended generation the situation is similar, but even more challenging because of variable length. Adding extra tokens can artificially lower the likelihood of an answer even when expresses the same concept, as the sequence of tokens becomes less likely with increasing length.
We demonstrate the difference between multiple-choice question answering and open-ended gener- ation in Figure 2 (left), where we compare the AUROC of a likelihood-baal method for stan- dard MMLU and open-ended MMLU (ours). For open-ended generations, we use perplexity, (sl exp N
E log p((; s<ci) where is the tokenized sequence, because it is length- normalized metric and commonly used when token-level probabilities are exposed by the model [19]. From AUROCs,
N observe that while token-level uncertainties often improve in multiple choice as models improve, perplexity is generally not predictive of language model's correctness in open-ended settings and does not exhibit the same favorable scaling with the language model's underlying ability.
Because sequence likelihood (or perplexity) is limited as confidence measure, prompting methods have becoming an increasingly popular alternative. Lin et al. [33] introduced the following formats that lay the foundation for recent work [51, 63]:
Name
Format
Confidence
Zero-Shot Classifier
P(# PruTr""")) / (P("Truee") + P(" False"))
"Qestioo. Answer. True/False: True"
Verbalized
Question. Answer. Confidence: 90%"
(1oat("900%
In the first approach, the language model's logits are used to create a binary classifier by scoring two possible strings denoting true and false. Similarly, Kadavath et al. [25], the classifier takes in slightly modified prompt, "Is the answer correct? (a) Yes (b) No and confidence is then computed P("(a)")/ (P("(a)") + P("(b)")). the second approach (also used in [51, 60),, uncertainty estimates sampled as text and then converted into numbers. We provide the extended details in Appendix B.2.
'Here we consider access to model's samples and token level likelihoods as black- box. Some models do not expose likelihoods directly, but they can be approximated through sampling.

---

Zero-Shot Classifier
Verbal
Fine-tune
Softmax Prob
Neg. Perplexity
AUROC
70%
AUROC
80%
AUROC
65%
40%
60%
60%
20%
50%
60%
0%
45%
60%
75%
40%
50
60%
35% 40% 45%
50%
35% 40% 45% 50%
Accuracy
Accuracy
Accuracy
Accuracy
Figure 2: (Left) We compare common uncertainty estimates for multiple-choice questions (max softmax probability) and open-ended generation (perplexity). While maximum softmax probability performs well and improves with the ability of the base model, perplexity does not follow the same pattern. The plotted results are for all LLaMA-2 and LLaMA-3 models as well Mistral 7B (base and instruct). (Right) Prompting methods for eliciting uncertainty from language models perform poorly when compared our worst fine-tuned model (LLaMA-2 7B), shown with a dotted line. ECE doesn't appear to improve with the abilities of the underlying model, and while AUROC does show small improvements with large improvements in accuracy, the gap between zero-shot methods and fine-tuning for uncertainties remains large. Shading indicates a 95% bootstrapped confidence interval on the regression fit.
The prospects of calibration by learning to model human language. If we view language modeling as behavior cloning [46] on human writing, the optimal outcome is a language model that recapitulates the full distribution of human writers present in the training data. Unfortunately, most humans exhibit poor calibration on tasks they are unfamiliar with [28, 29, 32], and not all pre-training data is generated by experts. Therefore it might be unreasonably optimistic expect black-box methods to yield calibrated uncertainties without a sigmificant intervention. Aligmment procedures (e.g. RLHF) could improve the situation by penalizing cases of poor calibration, and the resulting procedure would be akin to fine-tuning on graded data, which we explore in Section 5.
Experiments with open-source models. We examine the quality of black-hox uncertainty estimates produced by open source models plotted against accuracy in Figure 2 (right). We use LLaMA-2 [52, 53], Mistral [24], and LLaMA-3 models, and we evaluate open-ended MMLU highlight how the methods might perform in "chat-bot" setting. Because these models have open weights, we can perform apples-to-apples comparisons with methods that train through the model or access hidden representations. We see that prompting methods typically give poorly calibrated uncertainties (measured by ECE) and their calibration does not improve out-of-the-box as the base model improves. By contrast, AUROC does improve slightly with the power of the underlying model, but even the best model still lags far behind the worse model with fine-tuning for uncertainty.
Black-box methods such as perplexity or engineered prompts have limited predictive power and scale slowly, not at all, with the power of the base model.
5 How Should We Use Labeled Examples?
Our goal construct an estimate for P(correct), the probability that the model's answer correct. Learning to predict a model's 'ss correctness is simple binary classification problem, which we learn on small labeled dataset of correct and incorrect answers. There are many possible ways to parameterize P P(correct), and we study three that vary in their number of trainable parameters and their use of prompting:
Following Azaria and Mitchell [4], we train small feed-forward neural network on the last layer features of a LLM that was given the prompt, question, and proposed answer as input. The model outputs P(correct) while keeping the base LLM frozen.
Probe:
LoRA: This parameterization is the same as Probe but with low-rank adapters (LoRA) added to the base model. As a result, the intermediate language features of the base model can changed improve the correctness prediction.
5

---

LoRA + Prompt:
Following Kadavath et al. [25]. we pose classifying correctness as a multiple choice response with two values, the target tokens "i" and "ii" representing 'no" and 'yes' respectively. We perform LoRA fine-tuning on strings with this formatting.
With these different parameterizations, we can study how much information about uncertainty is already contained in pre-trained model's features. Probe relies on frozen features, while LoRA and LoRA Promot can adjust the model's 's features for the purpose of uncertainty quantification. Comparing LoRA with LORA Pr Promtt also allows us study how much a language framing of the classification problem aids performance.
Datasets. For training, build diverse set of samples from collection of benchmark datasets, similar to instruction-tuning [57]. From the list of 16 benchmark datasets in Appendix C.2, we use a sampled subset of size approximately 20,000. We hold out 2000 data-points to temperature scaling calibration set [17].
Training and regularization. We consider three base models- LLaMA-2 7b, LLaMA-2 13b, Mistral 7B-and their instruction-tunee variants. For tinettuning, we use 8-bit quantization and Low-Rank Adapters (LoRA) [20]. For LoRA, we keep the default hyperparame- ters: rank r 8, a 32, and dropout probability 0.1. Each training run takes approximately 1-3 GPU days with 4 NVIDIA RTX8000 (48GB) GPUs. To keep LORA and LORA + Prompt the neigh- borhood of the initial model, we introduce a regularization term to encourage low divergence between the prediction of the fine-tuned model and the base model (ablation in Table 1).
Method w/o KL.
ECE
AUROC
29.9%
70.2%
w/ KL
10.8%
71.6%
Table I:
Regularization im- proves calibration. Numbers show the mean over six base models models. See Ap- pendix C.1 for discussion.
Sampling baseline. We estimate the uncertainty by clustering generations by semantic similarity [30]. The probability of each cluster becomes the probability assigned to all sequences in that cluster. To assign an uncertainty to a prediction, we find the cluster closest to the prediction and use the probability of the cluster as our uncertainty estimate (full details in Appendix B.1). The clear drawback of this approach to uncertainty estimation its poor scaling. We draw К samples from the model (K=10 in our case), and then these samples must be clustered using O(K2) comparisons with an auxiliary model of semantic similarity. Sampling methods are also complicated by their relationship with hyperparameters such as temperature or nucleus size. In the special case where the sampling parameters are chosen to produce greedy decoding (e.g. temperature zero), the model will always assign probably one to its answer. While this behavior does align with the probability of generating the answer, it is not useful measure confidence.
Fine-tuning results. In Figure 3 (Left) we compare our three fine-tuned models with black-box uncertainty methods on both multiple choice and open-ended MMLU. For multiple choice MMLU, we also include the language model's 's max softmax probability as baseline. Fine-tuning for uncertainty leads to significant improvements in both ECE and AUROC. While frozen features (Probe) are sufficient to outperform baselines in multiple choice MMLU, performing well on open-ended MMLU requires training through the modeling and prompting. Surprisingly, while sampling methods can yield good calibration, their discriminative performance is very weak. By contrast, verbal elicitation relatively strong in discriminative performance, being on par with weaker fine-tuning methods, but general has poor calibration, even after temperature scaling.
How much data do we need? In practice. labels can be expensive to generate, especially on problems where domain expertise is rare. Therefore, it would be advantageous if fine-tuning with even a small number of examples sufficient for building a good uncertainty estimate. In Figure (right), we show how calibration tuning is affected by decreasing the size of the fine-tuning dataset. We find that having around 1000 labeled examples is enough to improve performance over simpler baselines, but that increasing the size of the fine-tuning dataset yields consistent improvements in both calibration and selective prediction, although the marginal benefit additional data points decreases after around 5000 examples.
Supervised learning approaches, in which we learn to predict model's correctness, can dramatically outperform baselines with as few as 1000 graded examples. Updating the features of the model with LoRA and use of a language prompt are key to good performance.

---

Logits
Verbal
Classifier
Sampling
LLaMA-2
13B
7B Instruct
Probe
LoRA
LoRA Prompt
Zero-Shot Classineer
Sampling
30%
40%
AUROC ECE
20%
AUROC ECE
30%
10%
0%
10%
0%
ECE
0.2
AUROC
70%
70%
60%
60%
50%
50%
10a
10a
10"
102
10a
10a
MMLU (MC)
MMLU (OE)
Samples
Samples
Figure 3: (Left) ECE and AUROC both multiple choice (MC) and open-ended (OE) MMLU. ECE is shown after temperature scaling on a small hold-out set. Supervised training (Probe, LORA, LoRA Prompt) tends improve calibration and selective prediction. Probing its own (Probe) performs worse than training through the features with a language prompt (LoRA + Prompt), especially in an open-ended setting. Error bars show two standard deviations over six base models. Extended results in Appendix D. (Right) Effect of varying number of labeled datapoints on OE MMLU. In the most extreme case, we train on only 200 examples. Overall, performance increases in proportion with the available labeled data, but 1000 points is almost as valuable as 20,000 points. Dotted lines indicate the performance of the classifier and sampling baselines averaged over the three models considered. Shaded regions show one standard deviation over subsets of MMLU,
When and Why Do These Estimates Generalize?
To derive more understanding of when our estimates generalize, we now investigate distribution shifts between the training and evaluation datasets. To have practically useful tool, we might desire robustness to the following shifts, among others:
Subject matter. Ideally, our uncertainty estimates apply to subjects we have not seen during training. In Figure 4 (left), we show breakdown of our fine-tuning dataset using the supercategories from MMLU (Appendix A.5). We see that our dataset contains much higher percentages of STEM and humanities questions than MMLU and close to no examples from the social sciences (e.g. government, economics, sociology). Despite these differences in composition, uncertainty estimates from LORA + Prompt perform similarly across supercategories. We also show the efficacy of our models at assessing confidence on out of distribution coding tasks in Appendix F.
Format. Like change in subject matter, the way a question is posed should not break the uncertainty estimate. To test the effect of the question format independent its subject matter, we apply models fine-tuned on OE MMLU MC MMLU and vice versa. In Figure 4 (center), we see that fine-tuned models often perform better than a zero-shot baseline even when they are being applied across a distribution shift, though transfer from MC OE is more challenging than OE to MC. Probe is insufficient to generalize effectively from MC to OE, but training through the features of the model (LORA Prompt.) does generalize effectively, even out-performing probe trained on OE data.
Solvability. Even though we focus on questions with a single known answer, we might hope that estimates can be used even when question is ill- posep or does not have known solution, ideally returning high uncertainty. We generate answers, labels, and uncertainty estimates for the answerable and unanswerable questions in the SelfAware dataset [61] using the same procedure OE MMLU. In Figure 4 (right), we plot P(correct) from Zero-Shot Classi fier and LORA + Prompt predicted for each answerable and unanswerable question. Notably, calibration-tunet models have calibrated probabilities for the answerable questions and assign lower confidence to unanswerable questions than black-box methods.
6.1 What are uncertainty estimates learning?
Language models can generate useful uncertainty estimates after training on a relatively small number labeled examples. How this possible? We hypothesize two, potentially complementary mechanisms: (a) LLMS assess the correctness of an answer given question, or (b) LLMS recognize that certain topics often have incorrect answers. To understand the difference, let's explore a useful metaphor. Imagine speak only English, while my friend, Alice, is a linguaphile and dabbles in many
7

---

STEM
Social Sciences
Zero-Shor Classifier
Zere-Shot
Trained
Humanities
Other
Probe
LoRA Prompt
Answerable
(Transfer)
(Transfer)
Density w
MMMUU % Train
20%
40%
15%
10%
5%
50%
30%
MC
OE
50% 70% 90% 30%
0%
0%
10%
Unanswerable
40%
1
80%
Density - w us
AUROC
20%
60%
6.0%
40%
70%
0%
40%
P(correct)
Figure 4: (Left) We compare the composition of the fine-tuning dataset with MMLU, Notably, although the training dataset contains close zero examples from social sciences, uncertainty estimates from the model perform similarly across categories. (Center) Testing the generalization of supervised methods by taking models trained on one setting (MCQA or OE) and evaluating them on the other setting. The MCQA or OE labels denote the evaluation setting, with the method labels indicate whether the model was trained on the same or different setting. Fine- Fine-tuning through the model's 's features (LORA + Prompt) performs almost as well in transfer as on in-distribution data. Zero-S SSot Cl Classin inssifierr involves no supervised learning except a temperature-Soe useful reference point. Error bars show two standard deviations over six fine-tuned models. (Right) Fine-tuning leads to lower confidence on unanswerable questions, taken from the SelfAware dataset [61]. Assigning low confidence to unanswerable questions allows the model opt out of responding.
step and
languages. I have spreadsheet of how often Alice makes mistakes each language. Now, when I hear Alice attempting to converse in language A, can guess how likely she is to err by recognizing the language from its sound and consulting the spreadsheet. I can do this without understanding the language at all. Alternatively, I can learn each language, which would be more complex but would strengthen my predictions.
To disentangle these two possibilities in our setting, we perform an additional experiment, in which replace the language model's answers in the fine-tuning dataset with incorrect answer options. If a language model is simply learning patterns in the errors present in the training data, then we would expect this ablation to perform on par with the original method because it suffices to learn patterns in the content of the question and answer without needing the true causal relationship between question, answer, and correctness label. The results are shown in Figure 5 (left). We see the model trained on incorrect answers performs surprisingly well, on par with a Probe model, but significantly worse than model trained on the original sampled answers. Correlating question content with error rates while moderately successful cannot be full description of the LoRA Prompt estimates.
Self-knowledge. Lastly, we examine whether language model be used model not just its own uncertainties but the uncertainties of other models. Several prior works argue that models identify correct questions by way of internal representations of truth, which might be unique model evaluating its own generations [4, 9]. In Figure 5 (right), we show that, by contrast, Mistral 7B actual has better AUROC values when applied LLaMA-2 7B than LLaMA-2 7B applied itself. In Figure 5 (left), we show that SBERT [44] and OpenAl sentence embeddings are competitive with Probe both LLaMA-2 7B and Mistral. Together, these results suggest that LLM uncertainties likely not model-specific. The practical upside of this insight is that one strong base model can be used estimate the uncertainties of many other models, even closed-source models behind APIs. when a small labeled dataset is available or can be generated.
Learned uncertainty estimates generalize to new formatting, subject matter, and even the generations of other models. This generalization appears stem not simply from judg- ing a question's 's difficulty based on its subject matter (a short-cut) but also learning the correspondence between questions and correct answers.

---

Incarrect
Sampied
Probe
LoRA. Prompt
Probe LoRA. Prompt
SBERT
Probe
2 Mitool
OMIEmb
20%
Model
0.7
20%
10%
0%
0.6
0.70
0%
70%
0.5
Mistal LLaMA-2 0.65
80%
50%
Mitrral LLaMA-2
60%
30%
Trained
Trained
40%
Figure 5: (Left) We ablate the correspondence between questions and answers by training LoRA Prompt on dataset with correctness labels from the model's generations but with the actual generations swapped with incorrect answers. In this case, the only relationships that can be extracted by the model are between the correctness labels and the questions. The model trained on incorrect answers generalizes surprisingly well but is much worse than a model trained on the original answers. models. (Center) We test how
Error bars show two standard deviations over three instruction-tune well models can learn to predict the correctness of a different model (in terms of AUROC), and find that mistral models are often better at estimating the correctness of LLaMA models than LLaMA can on their own generations. (Right) We show that generic sentence embeddings can also perform on par with frozen language model representations (MMLU-OE), but training through model much better. SBERT and OAI Emb refer to training classifier top of sBERT [44] OpenAI sentence embeddings. Error bars show two standard deviations over tasks in MMLU.
7 Does Calibrated Confidence Improve Collaboration with AI Assistants?
One key motivation for estimating LLM uncertainty is to signal the model's reliability during collaborative decision making. To examine how uncertainty estimates can be used in this capacity, we perform a preliminary user study (with = 181 participants) in which participants complete a multiple choice exam in collaboration with LLM (Mistral 7B Instruct). For each question, the participant is provided both the LLM's prediction and an uncertainty estimate, which can be from a uncalibrated method. We hope show that users
calibrated method calibrated uncertainty scores as part of their decision process. A more detailed description of the setup our study available in Appendix G.
likely to adopt
People are sensitive to informed confidence scores. Figure shows density plots of the model's reported confidence and whether the user chose agree with the model's prediction. We find that participants are sensitive to the confidence scores and tend to use scores when deciding to agree or disagree with the model's prediction if the uncertainties are reliable. On the other hand, participants generally do not modulate their decision to rely on the output of random confidence baseline (Figure 6(c)), in which the display uncertainty estimate is generated uniformly random. We see the strongest discrepancy in reliance choices when LoRA + Probe confidence scores are presented, highlighting that calibrated confidence does influence user behavior.
We include additional details and results in Appendix G. We find that confidence scores have the biggest effect on improving the lowest performing users, rather than average accuracy. However, this is a preliminary result in the nascent field of studying LLM uncertainties in practical collaborative decision making with users. We are only still scratching the surface of this question. For more fine-grained conclusions, study should be devoted to this subject. We outline several limitations and future directions in Appendix G.
Users are sensitive to confidence scores and use their relative magnitude to modulate their decision to LLM. Lower performing users most improved by access confidence scores. However, future work is needed to disentangle the effects of calibration from how humans choose to leverage uncertainties.
8 Discussion
There is much disagreement about the role of calibrated uncertainty in large language models, how it can best be achieved, and promise of black-box methods. We hope to have shed light on these

---

0.06
Disagros 0.15 Agr
0.08
0.45
Proportion
0.08
Proportonn
0.04
0.30
0.04
0.03
0.02
0.82
0.81
0.00
0.00
0.00
Confidence
Model Confidence
(a) Zero-Shor Sero-Shot Prompt
(b) LoRA + Prompt
(c) Random (Control)
Figure 6: We compare the distribution of LLM confidence (for Mistral 7B Instruct) on its answers, and whether the users (N 20 per variant) agree with the answer generated by the model or not. (a) For the zero-shot prompt, we find that the model provides little signal since most mass is similarly clustered. However, (b) improving the calibration of the model reveals increased reliance on the LLM for more confident answers, and decreased reliance for less confident answers. Evidently, the users are sensitive calibrated confidence scores. (c) For reference,
verify that uniformly confidence scores do not provide meaningful signal, rendering users unable to modulate their decision rely the LLM. All variants are compared approximately the same average participant accuracy.
questions throughout this paper. In contrast to prior results, find that out-of-the-box uncertainties from LLMs are unreliable for open-ended generation and introduce a suite of fine-tuning procedures that produce calibrated uncertainties with practical generalization properties. In the process, discovered that fine-tuning is surprisingly sample efficient and does not seem to rely on representations of correctness specific model evaluating its own generations, allowing estimators to be applied from one model to another. Moreover, we found it is possible, at least in the cases we considered, for calibrated uncertainties be robust distribution shifts.
There are many exciting questions for future work. Currently fine-tuning relies on two separate models for question answering and uncertainty estimation. Ideally, we want single model that can generate questions and uncertainty without switching between model weights. We anticipate that an uncertainty-awes pre-training or alignment phase might become essential but implementing such procedure while maintaining base language modeling abilities will introduce challenging online learning problem where the correctness labels evolve during training.
Beyond improving the safety and usefulness of language models, high quality uncertainties can also be used in active learning procedures, c.g. for sample-efficient tie-ttning [39], where data points are selected based on the predicted utility and the model's uncertainty, in order to balance the explore-exploit trade-off. Uncertainty estimates also be used improve factuality of language models by increasing the likelihood of generations that the model confident about (judges likely to be correct), for example by using alignment procedure (e.g. RLHF, DPO) with reward function that encourages confident generations [50].
We also showed how uncertainty information could be used to influence human decision making. In the end, LLMs will impact society through decision making, and to make reasonable decisions we need uncertainty information particularly protect against rare but costly mistakes.
Acknowledgements
This work is supported by NSF CAREER IIS-2145492, NSF CDS&E-MSS 2134216, NSF HDR- 2118310, BigHat Biosciences, Capital One, and an Amazon Research Award.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087 000.00343..877 2023. 2
[2] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedzior-ke Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the
10

---

Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357-2367. Association for Computational Linguistics, jun 2019. doi:
10.18653/11/199113 1245. 19
[3] Lora Aroyo and Chris Welty. Truth is lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15-24 2015. 23
[4] Amos Azaria and Tom M. Mitchell. The internal state of an Ilm knows when its lying. ArXiv, 1bs/2304.13734, 2023. 3,8
[5] Umang Bhatt, Valerie Chen, Katherine M Collins, Parameswaran Kamalaruban, Emma Kallina, Adrian Weller, and Ameet Talwalkar. Learning personalized decision support policies. arXiv preprini arXiv:2304.06701, 2023. 21, 22, 23
[6] Christopher M Bishop. Pattern recognition and machine learning. Springer google schola, 1 1122-1128, 2006. 19
[7] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. ArXiv, abs/1911. 1tts/1911.11144. 1661, 2019. 19
[8] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning, A large annotated corpus for learning natural language inference. Conference on Empirical Methods in Natural Language Processing, 2015. 19
[9] Collin Burns, Hao- Tong Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. ArXiv, abs/2212.03827, 2022. 3,
[10] Cheng-Han Chiang and Hung yi Lee. Can large language models be alternative human evaluations? In Anmual Meeting of the Association for Computational Linguistics, 2023. 3
[11] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. ArXív, abs/1905. 10044, 2019. 19
[12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/la03..mm 003.05457, 2018. 19
[13] Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. In Proceedings of the 2023 AAAT/ACM Conference on AI, Ethics, and Society, pages 469.889, 2023. 22
[14] Marie-Cathering De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124, 2019. 19
[15] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-37, 2007. 4
[16) Andrew S. Gordon, Zomitsa Kozareva, and Melissa Roemmele. Semeval-2012 2 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In International Workshop on Semantic Evaluation, 2011. 19
[17] Chuan Guo, Geoff Pleiss, Yu Sun. and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. 4,
[18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, his/2009.033000 2020. 2, 4, 17
[19] James Hills and Shyamal Anadkat. Using logprobs, Dec 2023. URL. htt.ttttt:///okeeee openai. com/ examples wuwingiinnnggguuu//.uuu/..iiu..u hha.ggooooobbb..oooomo 4

---

[20] J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. ArXiv, abs/2106.09685. 2021. 6
[21] Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qu: Machine reading comprehension with contextual commonsense reasoning. In Conference on Empirical Methods Natural Language Processing, 2019. 19
[22] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. 21
[23] KJM Janssen, KGM Moons, CJ Kalkman, DE Grobbee, and Y Vergouwe. Updating methods improved the performance of a clinical prediction model in new patients. Journal of clinical epidemiology, 61():766-8, 2008. 23
[24] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, ArXiv, abs/2310.06825 2023. 5,5
[25] Saurav Kadavath, Tom Conerly, Amanda Askell, T. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish. Christopher Olah, and Jared Kaplan. Language Models (Mostly) Know What They Know. ArXiv,
2022.
3, 3,4,6
[26] Gideon Keren, Calibration and probability judgements: Conceptual and methodological issues. Acta psychologica, 77(3):217-273 1991. 22
[27] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In North American Chapter of the Association for Computational Linguistics, 2018. 19
[28] Justin Kruger and David Dunning. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated seff-assessments. Journal of personality and social psychology, 77(6): 121, 1999.
[29] Justin Kruger and David Dunning. Unskilled and unaware-but why? reply krueger and mueller (2002). American Psychological Association, 2002.
[30] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. ArXiv, abs/2302.09664, 2023. 2, 3, 6, 18
[31] Xin Li and Dan Roth. Learning question classifiers. In International Conference on Computa- tional Linguistics, 2002. 19
[32] Sarah Lichtenstein, Baruch Fischhoff, and Lawrence Phillips. Calibration of probabilities: The state of the art. In Decision Making and Change in Human Affairs: Proceedings of the Fifth Research Conference on Subjective Probability, Utility, and Decision Making, Darmstadt, 1-4 September, 1975, pages 275-324. Springer, 1977. 22
[33] Stephanic C. Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022, 2022. 3.
[34] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ArXiv, abs/1711.05101,, 2017. 17, 19
12

---

[35] David John Cameron MacKay. Information theory, inference, and learning algorithms. lEEE Transactions on Information Theory. 01:2544-55455 2004. 19
[36] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. Conference on Empirical Methods in Natural Language Processing, 2018. 8. 19
[37] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2901-2907, 2015.
[38] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. ArXiv, abs/1910.14599, 2019. 19
[39] Ian Osband, Seyed Mohammad Asghari, Benjamin Van Roy, Nat McAleese, John Aslanides, and Geoffrey Irving. Fine-tuning language models via epistemic neural networks, arXiv preprint arXiv:2211.01566 2022. 10
[40] Stefan Palan and Christian Schitter. Prolific. ac- subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17:22-27, 2018, 21
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, gegh-performanc deep learning library. In Neural Information Processing Systems, 2019. 19
[42] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances large margin classifiers, 10(3):61-74, 1999. 4
[43] Benjamin Plaut, Khanh Nguyen, and Tu Trinh. Softmax probabilities (mostly) predict large language model correctness on multiple-choice q&a. arXiv preprint arXiv:2402.13213. 2024. 2,
[44] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprini urXiv:I908$ 10084, 2019. 8,
[45] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. ArXiv, abs/1907.1064) 2019. 19
[46] Stefan Schaal. Learning from demonstration. Advances in neural information processing systems, 9. 1996. 5
[47] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant, Commonsenseqa: A question answering challenge targeting commonsense knowledge. ArXiv, abs/1811 2019.
19
[48]
Gemini Team. Gemini: A family of highly capable multimodal models, 2024. 2
[49] Thomas C Terwilliger, Dorothee Liebschner, Tristan I Croll, Christopher J Williams, Airlie McCoy, Billy K Poon, Pavel V Afonine, Robert D Oeffner, Jane S Richardson, Randy J Read, et al. Alphafold predictions are valuable hypotheses and accelerate but do not replace experimental structure determination. Nature Methods, pages 1-7, 2023. 23
[50] Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine- tuning language models for factuality. arXiv preprini 1XXi::33110080001 2023, 10
[51] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint
2023, 1, 18
13

---

[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971, 2023.
[53]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, htt/2337.09288. 2023. 2,5
[54] Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. Calibrating large language models using their generations only. In Anmual Meeting of the Association for Computational Linguistics, 2024.
[55] Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. Learning from disagreement: A survey, Journal of Artificial Intelligence Research, 72: 1385-1470, 2021. 23
[56] Kailas Vodrahalli, Tobias Gerstenberg, and James Y Zou. Uncalibrated models can improve human-ai collaboration. Advances in Neural Information Processing Systems, 35:4004-4001 4016, 2022. 22
[57] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021. 6
[58] Johannes WelbI, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. ArXiv,
2017. 19
[59] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. URL https: / /www/www.....oo......o.. aclweb. org/ hrgganthooogy// / 202000000.200 emnlp- demos. 6. 19
Empirical
[60] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi, Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in 1lms. ArXiv, abs/2306.13063, 2023. 1, 1,3,4 4, 18
[61] Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653-8665, Toronto, Canada, 2023. Association for Computational Linguistics. 7,
[62] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019. 19
[63] Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09667, 2023. 3,4
14

---

Appendix for Large Language Models Must Be Taught to Know What They Don't Know
Table of Contents
A Evaluation Methods
15
A.1 Evaluating Correctness
15
A.2 Grading
16
A.3 Comparison Grading Techniques
A.4 Metrics
A.5 MMLU Supercategory Classifier
B Baseline Methods
17
B.1 Sampling Methods
17
B.2 Verbal Elicitation
18
Fine-tuning Method
C.I Regularization Term
C.2 Training Data
19
C.3 Training Hyperparameters
19
D Extended MMLU Results
19
E Confidence as a Function of Target Length
20
F Generalization Coding Tasks
21
G User Studies
21
G.1 Additional Details Setup
21
G.2 Important considerations
22
G.3 Extended Results
22
G.4 Interface and Instructions
23
H Broader Impact and Implications
23
NeurIPS Paper Checklist
35
A Evaluation Methods
A.1 Evaluating Correctness
For given question with known and generated answers (Q, A, Å) the correctness True if the generated answer matches the ground truth answer A. For multiple-choice question-answerrng the matching process only involves checking the first token generated via greedy decoding.
For open-ended evaluations, determining if the answer given correct is more complex. One simple approach to check if the ground truth answer appears as a substring of answer Å. However, this does not capture rephrasings that may be essentially equivalent such as "NYC" for "New York
15

---

City," or "Daoism" and "Taoism." Conversely, it also has the potential to be over werr-geneoous if the model is particularly verbose and emits many incorrect answers along with the correct string. Given the difficulty involved in writing rule-based method for evaluating open-ended answer correctness, we use instead a strong auxiliary language model evaluate correctness. The auxiliary language model shown the query Q, the ground truth answer A, and the model's output Å, and is prompted to grade the answer whilst tolerating nuance. For full details of the prompt used see (Figure 7). In this human
paper we utilise GPT 3.5 Turbo as the auxiliary grading model. We conduct a comparison grading, substring grading, and GPT 3.5 Turbo grading select subsets of MMLU in Appendix A.3. find that humans and GPT 3.5 Turbo have much greater agreement than humans and the substring method.
A.2 Grading
Dataset Construction. To perform calibration-tumm (CT), we need tuples (Q, A, A, C) answers from a language model that have been graded for correctness. When calibration tuning on multiple choice questions,
use exact string match to generate C. To grade open-ended answers, use a strong language model and grading prompt G instead (Figure 7):
G: prompt used for grading answers Á with A.
Compared to alternatives like exact match, language model grading is insensitive to re-phrasings that are equivalent meaning such "NYC" and "New York City," "Daoism" and "Taoism" LLM grading can also penalize answers that are overly verbose use different meaning of the same word, potentially containing incorrect answers along with the correct string. For example, if the question is "What's 's it called when you move quickly by foot and both feet aren't 't always touching the ground?" and the LLM response is bank run", the grader should be able distinguish that this semantically dissimilar to the true answer "run".
In this paper, utilize GPT 3.5 Turbo the auxiliary grading model. When comparing many possible grading methods on subsets of MMLU, we find that GPT 3.5 Turbo has high agreement with humans while being cost efficient (Appendix A.3).
Grading prompt (G)
The problem is: Q The correct answer is: A student submitted: Á
The student's answer must be correct and specific but overcomplete (for example, if they provide two different answers, they did not get the question right). However, small differences in formatting should not be penalized (for example, 'New York City' is equivalent to 'NYC'), Did the student provide an equivalent answer to the ground truth? Please answer yes without any explanation: </a>
Figure For open-ended generation, we calculate the ground-trui correctness C using LLM and grading prompt (G). The token 
 is an end-of-sentence token. Blue text is included in the loss function when calibration-tuning.
A.3 Comparison of Grading Techniques
We conducted an analysis of the methods outlined in Appendix A.1 for open- open-ended evaluation. First, base LLaMA-2 13b-chat model was prompted with questions from the following test subsets of MMLU: World Religions, Philosophy, Anatomy, High School Chemistry and Elementary School Math. The questions were stripped their multiple-choice options before being supplied to the model.
16

---

A response was generated by the model via greedy decoding and this response was compared to the ground truth answer. The grading methods tested were Human, Substring Match, GPT 3.5 Turbo, and GPT 4.
The humans (a subset of our authors) were tasked to judge if the model response was essentially equivalent to the ground truth. For substring match, equivalence was determined by simply checking whether the ground truth answer existed as substring within the model response. For GPT 3.5 Turbo and GPT 4, the models were supplied with the question, the ground truth, and the base model response, as well as prompt indicating they should determine essential equivalence see Figure 7.
MMLU SUBSET
SUBSTRING MATCH
GPT3.5
GPT4
WORLD RELIGIONS
21.6%
6.4%
1.8%
PHILOSOPHY
22.8%
2.3%
14.5%
ANATOMY
13.3%
14.8%
1.5%
CHEMISTRY
13.8%
5.4%
1.0%
MATH
12.4%
14.8%
3.7%
AVERAGE
16.8 %
8.7 %
4.5
Table 2: Absolute differences in accuracy % for the different grading methods vs human estimated accuracy. lower value corresponds to an accuracy estimate closer to the human estimate.
We recorded the binary decision on correctness for each query and response by each of the grading methods above. Taking the human scores the gold standard of correctness, we computed the model accuracy for each subset, and then derived the absolute error in estimate of model accuracy by each of the other grading methods. These are displayed in Table 2. We see that GPT4 is a better estimator of human-judger correctness than GPT 3.5 Turbo, which in turn is substantially better than substring per-subset basis. For expediency of processing time and
match; although there is some variance cost, we chose to use GPT 3.5 Turbo in this paper.
A.4 Metries
Given N samples and B equally-spaced bins bj. examples are assigned to bins based on the confidence of the model, and ECE estimated as ECE Σラ1 lbyl [con[(b;) - acc(b;) where N
ECE
cbn is the average confidence of samples in bin by. acc(b,) ) is the accuracy within the bin, and 1byl is the number of samples assigned to bin j. In experiments conf is equivalent to P(correct).
A.5 MMLU Supercategory Classifier
To understand the impact of the subject matter of the training data on generalization, we follow the prescription of Hendrycks et al. [18] and categorize each of the 57 tasks into one of four supercategories- Humanities, STEM, Social Sciences, and Other. Since we do not have such categorization for the training set, we must estimate their proportions.
First, we use the OpenAl embeddings (dimension 1536) of the MMLU samples with their ground truth supercategories to train a linear 4-way classifier with 10 samples from each of the 57 tasks. We use AdamW [34] with learning rate le-3 and weight decay 1e-2. This classifier is then used to estimate the categories of each sample in the training set used for fine-tuning. Subsequently, the breakdown of results Figure (Left) follows.
B Baseline Methods
B.1 Sampling Methods
We use two baselines which obtain an estimate of certainty by sampling the same answers n 10 times and then estimating the proportion of sampled answers that agree with the greedily decoded "main" answer. There are several critical downsides to these approaches: (i) the uncertainty here depends on the sampling parameters-for example, in the limit where the sampling converges to mere greedy decoding, the I.I will produce n identical samples, and therefore the certainty will always be (ūi these approaches require O(n) answer generations to provide certainty estimate for
17

---

single generation. The intense computational restriction prevents us from easily searching the space of sampling parameters for the optimal set, so we choose parameters arbitrarily; here we sample with top_r- = 0.95.
Counting In this baseline, each sampled answer is compared to the greedy answer by prompting an expert LLM with both answers and asking it to judge their equivalence. The proportion of samples that are equivalent to the greedy answer is the certainty estimate. This baseline is similar to Label prob [51]; our method differs by not choosing the argmax semantic group as the final prediction, but instead using the greedy decode for the final prediction, so as to maintain the same accuracy performance as our uncertainty query method. This met
Likelihood accumulation In this baseline, we add up likelihoods sampled answers to estimate the mass associated with the predicted answer. We begin by prompting expert LLM in order to find which sampled answers are equivalent to the greedy answer like in the counting baseline. In this method, the certainty estimate is produced by adding the length- normalized likelihoods of those sampled answers equívalent to the greedy answer, and dividing this quantity by the sum of all sampled answers' length-norman noggmlierml likelihoods. This procedure of adding likelihoods of samples in order to estimate the likelihood of an equivalence class is similar to that used by [30], although they do not it for certainty estimates but instead to produce entropy scores. In practice, the scores produced by these two methods are actually very similar-so we report only likelihood accumulation numbers in the main text.
B.2 Verbal Elicitation
Although Tian al. [51] introduce several strategies for prompting. involving multiple guesses or multiple stages of interleaving prompting and generation, we did not find that any strategy consistently outperformed any other. This finding was consistent with the results of Xiong al. [60]. Ultimately, for convenience, we adopted a two stage strategy with a single guess because it can be used in tandem with logged datasets of generated answers per model.
The exact prompt we used is essentially the same at in [51], but with small modifications that improved the rate of correctly formatted responses:
"Provide the probability that your answer is correct. Give ONLY the probability, other words explanation.
For example:
Probability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever, just the probabilityl
Include probability for the answer below: Probability:"
Verbal elicitation methods typically output complex strings containing both answers and associated probabilities. This means that if any element of parsing fails, it can be challenging to construct partial results. This effect tends to diminish when using large models, which are more responsive to zero-shot prompting.
Parsing Details The original verbal elicitation prompts are given in the appendix of [51]. However, it is not clear how the original authors decide to parse answers from the generations and how failure to parse is handled. When we fail to parse the guess from the generation we return an empty string and associated probability 0.5. When we fail to parse a probability, we also return probability 0.5. For versions with multiple guesses, if any part of the parsing processes fails in an ambiguous way, we default back to an empty string for the answer and 0.5 for the probability. The only unambiguous cases are those which explicit succeed in the generating valid guess and probability in the first case but not subsequent cases. this scenario, we default to using the successfully parse first guess and associated probability.
18

---

C Fine-tuning Method
C.1 Regularization Term
To keep the valibration-tuner parameters 0 within the neighborhood of the initial parameters, oo. we use a regularization term that penalizes the divergence between the original sampling distribution and the calibration- tuned model on the target sequence A, yielding regularization R(; 0o), which we use with weighting parameter
Specifically, let Pe, be the language modeling distribution of the language model we wish calibration-tune, and qo the corresponding language modeling distribution We then use the Jensen-Shannon Divergence JSD(pe, (o) [35] between the two language modeling distributions as the regularizer, where JSD(p q) A (p m) + [L(g m)), where m - 1/2(p q) the mixture distribution. ISD regularization is applied only to the logits corresponding to the target sequence A.
as a consequence of calibration-tuning.
We note that using either direction of KL-divergence, i.e. the forward-KL. KL 9,) reverse- KL KL(g, Pe,) was insufficient for optimal performance with calibration tuning. The forward Kl-divergence encourages zero-avoiding behavior such that the mass of go spread across multiple modes of pe, to minimize the -l.-divergence to avoid assigning no mass to regions of the probability space. To the contrary, the reverse KL-divergence encourages zero-foreing behavior such the qe only needs to cover any one mode of pe, [6]. It is not necessarily obvious which one of these behaviors one should prefer for the specific case of large language models. Therefore, practical choice, we pick the one that provides us the most performant calibration-tume model.
C.2 Training Data
We reserve the following datasets for training.
AI2 Reasoning Challenge (ARC) [12],
Boolean Questions (BoolQ) [11],
Q [47].
CosmosQA [21].
HellaSwag [62],
MathQA [2].
Recognizing Textual Entailment (RTE/SNLI) [8].
Adversarial NLII [38].
OpenBookQA [36],
PIQA [7].
SciQ [58].
The CommitmentBank (CB) [14),
Multi-Sentence Reading Comprehension (MultiRC) [27],
Choice of Plausible Alternatives (CoPA) [16],
TREC [31],
Adversarial Winograd (Winogrande) [45].
C.3 Training Hyperparameters
We use HuggingFace Transformers [59] and PyTorch [41] for the implementation of these models. For all our experiments, we use the AdamW W optimizer [34] with a learning rate of 10-4 cosine decay schedule, and effective batch size M = 32. The training runs for G 10000 with initial linear warmup schedule for 1000 steps.
D Extended MMLU Results
We report the breakdown of uncertainty query accuracy and ECE on all MMLU tasks in Figures
to I1.
19

---

LLaM4-2 76
LLaM
LLaM6-2
LLaNsA-2
Mistral 718
Mistral 7B Instincct
78 Chat
138
158 Char
buires chis
ciriaal ercslerge
loge Lickor
age, computer scince
colegg medcne
callgge
compute cemultecsecuuty
dectiicl
derrerrryy nathemsties
farmal
fict
high sighushoo
Hight school,
gigh school ou suropaan Hutoryy
uphy
righ, school gevemmentt and polties
right schoo, mathematis
Hih scscol physics
BCE
AURIC
ECE:
AUROC
AURIC
AUROE
AURIC
AURDC
Figure 8: (Part 1) ECE and AUROC values for Query, CT CT-Probe, CT and CT-Query for each subset of MMLU in multiple-choice question answering (MCQA) setting.
E Confidence as Function of Target Length
As we noted when motivating calibration tuning, one limitation sequence-level probabilities their intrinsic connection to sequence length. The probability of sequence decreases with increasing length, regardless of the correctness of the response. By contrast, we wouldn't expect concept-level probabilities to have any discernible relationship with sequence length. In Appendix E, we show
20

---

Model
Method
Acc
ECE
AUROC
Zero-Shot Classifier
3.2%
41.0%
56.9%
LLaMa-2-7B
Lora + Prompt
3.2%
46.4%
80.0%
Zero-Shot Classifier
27.4%
70.2%
66.2%
Mistral-7B
Lora + Prompt
27.4%
21.4%
85.1%
Zero-Shot Classifier
21.0%
52.7%
47.1%
Mistral-7B-Instruct
Lora + Prompt
21.0%
56.1%
70.2%
Table 3: ECE and AUROC on livecodebench. generation lite (LeetCode easy subset). ECE is shown the original dataset mixture Appendix C.2. Acc is task accuracy (proportion of coding solutions that are correct). Supervised training (LoRA Prompt) seems always improve selective prediction, although supervised training only heavily improves calibration for Mistral-7B and in fact slightly degrades calibration for the two other models.
after temperature scaling on small hold-out
there is no consistent relationship between the confidence estimated by the calibration-tuned model and target sequence length on MMLU tasks.
key limitation of using token likelihoods is that they necessarily decay with the length of the generation. In Figures 12 to 14, confirm over all subsets of MMLU that the length of the target does not strongly correlate with the confidence associated with the targets. This behavior is an essential ingredient towards effective confidence estimation in practice, such that longer sequences are not penalized confidence despite being correct.
F
Generalization to Coding Tasks
Because there are no coding tasks in our training dataset, we can use coding competition task introduced in LiveCodeBench [22] to assess how well finetuned uncertainty estimation methods perform on completely out of distribution tasks.
To conduct the analysis Table 3, we evaluate several base models on the 62 LeetCode easy questions from the and grade the solution using test cases (marking it correct iff passes all test cases). We then apply Lora + P Prompt and Zero-Shot flassifier uncertainty estimation methods these methods only using training/eempertuure scaling data from our main dataset mixture which notably does not include any coding tasks Appendix C.2. Accuracy is shown to contextualize the model's 'ss overall level of performance on the task. On Mistral-7B, the best performing model on the coding task, the supervised Lora + Prompt approach dramatically improves calibration and selective prediction as compared to Zero-Shot Classi fier; on the worse-per porse-performing Mistral-7B Instrrcct and LLaMa-2-7B selective prediction improves but calibration slightly degrades.
srsrebennc__eneratiinn lite task. We asking for the model to write a Python solution
G
User Studies
G.1
Additional Details on Setup
Stimuli and Participant Selection We closely followed the setup of [5]. We used the same 180 MMLU questions from which were pre-batched into three sets of 60 MMLU questions. Within each variant, randomly assigned participants one of the three batches. In total, we recruit 181 participants (20 per variant*). All participants were recruited through the crowdsourcing platform Prolific [40]; restrict participant pool to those based in the United States who speak English as a first language.
Compensation Participants told that the study would take approximately 30 minutes and were paid at a base rate of $9/hr and informed that they would receive an optional bonus up to $10 for answering questions correctly. We applied the bonus to all participants.
'With the exception of extra participant due to random batching allocation effects.
21

---

LLM Answers and Uncertainty Elicitation Bhatt et al. originally used GPT-3.5 as their LLM. While first, we explored user performance when provided with confidence scores modulated over the original GPT-3.5 responses that the authors had collected, the authors had filtered L.M performance to ensure the L.M achieved high performance on biology, computer science, and foreign policy and poor performance on mathematics. As such, we noticed that participants overwhelmingly uptook the LLM's answer (which was rational behaviour, given the model's high performance). To explore more nuanced performance profile, we regenerated LLM answers using Mistral 7B Instruct via greedy decoding. We then generated confidence scores on top of the LLM responses. For our random baseline, sample confidence score uniformly between 0 and 100% for each question.
G.2 Important considerations
There many reasons to heed caution in interpreting our results as definitive indications of the utility of displaying confidence to users in LLM assistive settings. In particular: (i) users are presented with feedback after each trial as in [5] such, they can determine (potentially rapidly) whether or not model is reliable, even without confidence scores. However, in practical settings users may not know whether not the model truly correct and therefore confidence scores could have an larger impact; (ii) MMLU questions can be challenging for non-experts we see the biggest differences in performance for the no-LLM vs. any-LLMM --asssttasss condition. We may see wider range of reliance behaviors in settings wherein people have more confidence in their own abilities; (iii) we present users with numeric confidence; however, humans not always able reliably process confidence estimates nor appropriately calibrate uncertainty estimates themselves [26, 56, 13, 32]. It may be that alternate modes of communicating confidence improve users' ability to appropriately leverage the confidence scores in their decision making process. We see targeted exploration of each component through interdisciplinary collaboration across AI, behavioral science, and human coman-c interaction as ripe for future work.
G.3 Extended Results
Task Accuracy and Reliance Sensibility We depict average user task accuracy and reliance sensibility across variants in Figure 15. We follow Bhatt et in computing reliance sensibility as the proportion of times the user appropriately sided with the model prediction when the model was correct and did not respond with the model's prediction when the model was incorrect.
We depict per-topic accuracy, with the LLM's average performance in Figure 16.
GPT-3.5 Confidence Generalization As noted, we ran variants using the same GPT-3.5 generations as [5]. We show aggregate and per-topic accuracy in Figure 17, well reliance sensibility in Figure 18.
Freeform User Responses We permitted users to provide freeform responses at the end of the study. Some users were sensitive to confidence scores being reported and came up with their heuristics for whether to rely on the model's output. We include a sampling of comments across confidence variants:
"if it had confidence of less than 50% it made me very skeptical."
"The modelá confidence indeed helped me choose and select my answer as I trusted in them most of the time."
"I didní really rely on the confidence level. If had 0 confidence in the answer myself I relied on the Al regardless."
"if the models confidence fell below 45 decided to investigate it myself by remembering pieces of information. and also reasoning the question. If was above 45 would automati- cally agree its prediction but there some few cases challenged it even though was above 45"
"At first was hesistant to trust the model much because of the lower confidence levels but still trusted enough on topics struggled with. As it went on, was comfortable with confidence levels above 40."
22

---

"If the models confidence was low and I thought knew the answer (and it was different) I chose my answer"
G.4 Interface and Instructions
We show a sample interface of our extension of Modiste with user confidence in Figure 19, and present the the full of instructions provided and no-LLM conditions, we followed the instruction text from [5] directly, i.e., participants who saw only the LLM did see the instruction page about model confidence, and participants in the "No-LLM" variant were not instructed about any model variant and were just instructed to answer the questions as best they could by themselves. Participants also responded post survey questionarre after completing the user study, which we depict in Figure 22.
in Figures 20 and 21. Note, for the LLM-only
H Broader Impact and Implications
The goal of this work is to make LLM outputs have better confidence values associated with them. With successful, calibrated confidence values, the machine systems ultimately become more interpretable and trustworthy by a user [23]. When applied correctly, our advancements will help users be able to make decisions based off of LLM outputs in more informed way. Similar examples in other domains, like AlphaFold [49]. have shown how well-calibrated confidence scores can be useful in complex decision-main making domains. Our hope is to replicate those broad findings in LLMs.
We acknowledge the ongoing debate over the appropriateness, limitations, and harms of LLMs. We do highlight that the development of more confident, interpretable, and trustworthy L.Ms can lead to continued techno-solutionism in unintended applications. Specifically, we highlight that our work is limited to usc-cases with fact-based questions. Many applications of text-based LLMs generative, meaning that there is no way for our paradigm to be applied appropriately, and the use of confidences from calibration-tur tuttiinn-tteddddde models could be misleading damaging without checks and guardrails. Additionally, even within the fact-based paradigm, what is true can subjective, with ground truth in machine learning being contested topie [3, 55].
The philosophical debate on these topics beyond the expertise the authors: nonetheless, we believe that the ongoing debate over the appropriateness of LLMs should be considered in context with the benefits of our approach in making L.LMs more interpretable and useful.
23

---

MA-2 76
LLaMA-2
LLaMA-2 138
LLaMA-2
Mistral 78
Mistral 78 Instruct
78 Chat
13B Chat
highsshho, peycholooyy
Nigh Mig__schol_statee
high schoo httor
human, agin
Interno nnteraaional, ow
jutieeeadence
machine Jeam
management
mediallomeete
miscotanous
moral deoeress
scenaros
preh story
poresstonol.ccom Lacoortt
professiona_
profece smal, medic
professional cecrsroon___eeccoooeeeo Leeychology
pubbic
sec stuckes.
sodalogy
to form
virology
geligionn
20% 5995 60%5 90%
20% 50% 60% 90%
20% 59% 60% 90%
20% 50% 69% 90% ECE
20% 59% 60% 90% ECE
20% 50% 60% 90% ECE
ECE
AUROC
ECE
AUROC
ECE
AUROC
AUROC
AUROC
AUROC
Prompt
Figure 9: (Part 2) ECE and AUROC values for Query, CT-Probe, CT- LORA, and CT-Query for each subset of MMLU multiple-choiee question answering (MCQA) setting.
24

---

LLaM4-2 76
LLaKA-2
LLaMA-2 136
LLaMA-2
Mistral 78
Mistral 7B Instruct
70 Chat
150 Chat
clrkal menllegs
loge Billogy
oo lelesee chemisty
oge compotor scincs
of natiematiss
colage modens
collgge 2139
corcostual
alecttos
elemeerary, nathematies
formal
glotal phrts
school
sigtt chemistr
Nihh senool
gight wot.schoolsssporee sumppean, History
high hotpsroooggggeemeree gonvenment and politie
light school cotiu__nattinnaa natiommatiss
201650%
42%
90%
ECE
ALROC
ECE
AUROC
ECE
AUROC
BCE
MROC
ECE
AUROC
BCE
AUROC
Classifier
Figure 10: (Part ECE and AUROC values for Query, CT-Probe, CT-LoRA, and CT CT--ueryy for each subset of MMLU in open-ended (OE) setting.
25

---

MA-2 76
LLaMA-2
LLaMA-2 138
LLaMA-2 13B Chat
Mistral 7B
Mistral 78 Instruct
78 Chat
highsshhol poycholooyy
Nigh hop_schol_sttteeec
schoo wotorhhstoryy
human, agrn
Inter nnteraatonal, ow
jutiepeddence
oglccU/a fadisss
machin_ Jeam mi
management
pedioll_peneter
miscotanouus
moral deseeess
morel scenabbos
preh story
cofesstonaaaccoonnt
professiona_
profece onal, mmeici
professional cacoooool__occcooeeeee Leeychology
pubbic mevelonn
sec stuckes
sodalogy
fore
virology
209550% ECE
60%6 90%
209650% ECE
60% 90% AUROC
20%.90% ECE
60% 90%
209500% ECE
69% 90% AUROC
209650% ECE
60% 90% AUROC
20%500 ECE
60% 90% AUROC
AUROC
AUROC
Prompt
Figure 11: (Part 2) ECE and AUROC values for Query, CT-Probe, CT CT-LORA, and CT-Query for each subset of MMLU in open-ended (OE) setting.
26

---

abstract_po abstract_aleebra
anatomy
astronomy
ciinical knowledge
06
Comfitance
Corfidance
0,50
0.2
25
50
50 Target Length
100
202
100
Target Length
Target Length
Target Length
college_biologg
college, chemistry
college_computer computerppeeepoooomm science
college_mathereeee mathernatics
Comfoencee
0.75
0.8
Condideene
0.50
0.2
0.4
200
200
120 Target Length
50
100
100
Target Langth
Length
Target Length
college_mee medicine
computer_security
acanometrics
electrical eengineeing
Contdercs
0.75
Confoencee
Confarece
0.6
0.6
0.25
0.4
0.
100
Targel
Length
Target
elementary mathematics
formal formal_logic
global_ racts
high schoscooll, gy___ooo__bbloogy
3.775
Conteence
Confideece
0:75
Cnfioence
0.59
0.25
Target Length
Target Length
Targel Length
Target Langth
high high_school, schpso__hhhhistrryy
high, sor__schol__geeee peoggapphyyyyyoooo
hath_schoo__oommo wwwpppttrepccceeeeeee science
high, school, puuropaan history
Cantoencs
0.75
Confeeenee
Confaance
0.50
100
Target Length Length
Target Target Length
high_schad. school mathomatics
high school macroeconomics
sach__choo__ggo miccrocconomics
0.6 Confioence
Ouni dance
Bac
0.2
0.25
2.
100
Lengthh
Target Length
25
50
Target
Figure 12: Confidence versus Target Length for various MMLU subsets. A horizontal regression
line indicates weak correlation of confidence with the target length. See Figures and 14 for other subsets.
27

---

high schhol_physicee
high high_school_acc osycholoyyy
Nighsccool_ statistics
high_school us, history
1.0
Confeence
0.75
Corfidence
0.73
Confidence
025
Length
Length
Target
Longth
high_schou school world world_history
human,
sexuality
international law
Confelnce
Confeence
0.75
Confeence
0.50
0.4
0.5
0.00
0.0
Target Length
Length
Target Length
Length
jurisprudence
looical fallaci_s fallacies
machine
management
Corfidence
0.6
Confogene
Conforece
Corfidence
0.50
0.00
100
200
100
200
Target Length
Target Length
Torget Length
Target Length
marketing
medical, medical_genetics genetics
miscellaneous
moral morai_disputes
1.0
Corficence
66
Comfioecce
0.75
8.0 Confeeence
0.50
0.2
0:25
200
200
100
100 Target
200
180 Tbrget Length
Target Length
'Brget Length
nutrition
philosophy
prohistoryy
0.75
Cont Conedennc
0.50
025
0.25
100
200
100 Target Length
200
Target Langth
Tarpet Langth
professional accounting
professional psschologyy
public relations
security studies
D.6.
Conferencca
Confidence
0:75
Confidence
0.59
0.05
Length
Target
Thrget
Target
Figure 13: Continuing from Figure 12. See also Figure 14.
28

---

sociology
us Joreian_pelicy
virology
world religiors
0:75
Centerent senc
Confee Conferincc
0.50
0.50
50
0.25
0.25
0.23
100
50
100
50
103
50
Sarget Length
Sargel Langth
largel Length
Target Length
Figure 14: Continuing from Figures 12 and 13.
Retten By
No LLM LAV
LEE OuT
JIP Coef
17and
LLS Car
LUP CHF
Figure 15: (Left) User accuracy 60 MMLU questions per variant (N 20 users per variant); violin plots show quartiles as dashed lines (Right) Average reliance sensibility (proportion of instances where the user sided with the model when the model was correct, and overrode the model's prediction when the model was incorrect); higher indicates better reliance calibration.
High Billogy
High
N0 LLM
LJ DNY (Faii
LEX
JUN CoF
RV LMM
LIM
LR OFF
UN Gef
LUM Coel
12uer))
3011
(Tane)
10/49)
(CT)
Figure 16: User accuracies per topic for the Mistral variants. Red line indicates the model's average accuracy.
29

---

School Billey
High School CS
(fadd]
3011
US Foreign Polcy
Bemontary Noth
Figure User accuracies per topic for the GPT-3.5 variants (with generalization confidence computed for the CT and Query cases). Red line indicates the model's average accuracy.
Sensibilis
633
LUM
LLM Conf (Rand)
LLM Conf (Query)
LLM Conf (CT)
Figure 18: Reliance sensibility for the variants based on GPT-3.5
30

---

contie credecten
The meder's confidence coeldreee to araver la 805S.
the
homnolggy?
SURMIT
Figure 19: Example interface from Modi ste. Participants are informed of the question (and topic), as well as the LLM prediction and confidence. Participants are informed of their running score throughout the experiment.
31

---

Thes ope mese erccc ainate.
untie sie chene pundic vinu suc Ioca Nod youryyy rached whi
Your Belp anever questoo silec cargory dlodig or auton
Duingge sans, you prodic medesdd
You will also see the model's confidence prediction (wrich will be shown blue) for each question
Previous
You will receive bonus of up rate h10/hour (-$0.50) based on how ma vy questions you correctly answer:
You intormei whether not you correct each trall
Figure 20: Experiment instructions for the confidence variants.
32

---

You will see a total of 60 questions.
<
Previcus
Next
you reddy, please complete quick comprehension
moving the experiment.
Please make sure winh dow size full soreen, or substantisly large enouahhy to property view the ouestions.
Prevdoux
Check your knowlodge bnfore youbogin, you don't fnow the arowors, don't woryy we will soW you their spucdions aggins
What you be to detern in tuske*
anconer to mut cheice cumetion.
most lloey
madio
Selecting fiom drspdownmeer..
Figure 21: Experiment instructions for the confidence variants (continued).
33

---

Thank you for participating in our study!
Click "Finish" to complete the experiment and receive compersation. If you have any comments about the experiinent, please us know the form below.
challanging did you tho quest ons? (On a scale 1-10, with 10 being very challenging)
Did the model's conficince impact your
please ba specific possible sentences)
Were there any question tapics you struggled with?
Were there question topics always confident in?
you have any comments share
Finigh
Figure 22: Sample pot-survey questionnaire for users who were allocated variant wherein they saw model confidence.
34

---

I
NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We describe and link all claims in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a discussion on the limitations in Section 8.
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
35

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the complete code, and the complete list of datasets used for all experiments in Section reproduce all our experiments with instructions. All hyperpa- rameters are described in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification: We provide the complete code, and the complete list of datasets used for all experiments in Appendix C.2 reproduce all our experiments with instructions. All hyperparameters are described in Section 5.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: /ttp://nippp./. cc/
public/ guides/ Code
MioooloPlly)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc public/ gu ded Code Submi wwwssssssiiiooooooooooooisoissii icy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the complete code, and the complete list of datasets used for all experiments in Appendix C.2 to reproduce all our experiments with instructions. All hyperparameters are described in Section 5.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
All figures are appropriately labeled with the error bars.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide an estimate of the compute resources required in Section 5,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ /neurips....p// wttpscccppppbottee wwtpw/cwcppppc..c........ httppp//ooo/ttccccllleeeeee.ee
Answer: [Yes]
Justification:
We have read the ethics guidelines
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide a broader impact statement in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We train open- models with open-source datasets. We do not change their generation behavior, and all existing safeguards (if any) remain.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly cite all models in Section 5. All datasets used listed and cited in Appendix C.2.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, papers rswi thoode. com / da sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
39

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release our trained models for easy use via Hugging Face.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We provide screenshots our instructions, as well as details of compensation in Appendix G.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: We received prior approval from our respective institutional ethics review body for our user study. All users provided consent before partaking in the study.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
40

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "LMIE5Ov9s1", "reviewer": "Reviewer_oPif", "summary": "This paper investigates fine-tuning on a small dataset of correct and incorrect answers for an uncertainty estimator. Through various experiments, the authors show the trained estimator not only surpasses the previous prompting approach, also has generalizability to other models and subjects. Lastly, the authors conducted a user study to verify that calibrated confidence could improve human-machine collaboration by helping users modulate their decision to use the model.", "questions": "- In Figure 1 (right), there are 6 value points of “verbal” and 11 of “zero-shot classifier”. How are they selected? Furthermore, it seems to have insufficient values to draw the trend line.\n- In Figure 1 (left), isn’t AUROC sensitive to the scale of perplexity range?", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Uncertainty and calibration to reduce hallucination are currently crucial issues.\n- This paper provides various experiments and analyses. Moreover, the experiments are soundly conducted and most of the details are presented for reproducing.\n- The user study is fascinating where users solve MMLU tasks accompanied by a confidence score and predicted answers from an LLM. However, there remains future work to in-depth analyze the results.\n- This paper is well-written, neatly organized, and well-presented.", "weaknesses": "- The evaluation task only considers MMLU, which is highly likely to be contaminated.\n- In the section “When and Why Do These Estimates Generalize?”\n    - In both Figure 4 left and right, some results (Probe, Incorrect, sBERT) seem to have near AUROC of 0.5, which is random. Therefore, it’s hard to agree with the result drawn in this paper.\n    \n    > Learned uncertainty estimates generalize to new formatting, subject matter, and even the generations of other models. This generalization appears to stem not simply from judging a question’s difficulty based on its subject matter (a short-cut) but also learning the correspondence between questions and correct answers.\n    > \n- The following paper is related in terms of a fine-tuned confidence estimator.\n    - Ulmer, Dennis, et al. \"Calibrating Large Language Models Using Their Generations Only.\" *arXiv preprint arXiv:2403.05973* (2024).\n- In the section, “How Should We Use Labeled Examples?”\n    - The architectural details of the “small network” used in probing and others are absent. And, what is the regularization hyperparameter for KLD? What prompt is exactly used for the “LoRA+Prompt” setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful and supportive feedback!\n\n## Additional Benchmarks\n\nInspired by your comments, we provide results on the MMLU-Pro benchmark, which is a more challenging multi-task language understanding benchmark. This dataset is much newer than any of the models we test and bears no risk of contamination. Overall, we see that supervised methods provide much stronger performance in terms of ECE and AUROC over the zero-shot classifier.\n\n### LLaMA-2 7B-Chat\n\n| Method  | ECE   | AUROC   |\n|:---|---|---|\n| Zero-Shot Classifier  | 68.66  | 53.29  |\n| Probe   | 6.34 | 59.79  |\n| LoRA + Prompt   | 26.34  | 65.55  |\n\n### LLaMA-2 13B-Chat\n\n| Method  | ECE   | AUROC   |\n|:---|---|---|\n| Zero-Shot Classifier  | 32.11  | 54.01  |\n| Probe   | 7.58  | 63.10  |\n| LoRA + Prompt   | 9.38  | 68.33  |\n\n### Mistral 7B-Instruct\n\n| Method  | ECE   | AUROC   |\n|:---|---|---|\n| Zero-Shot Classifier  | 19.50  | 60.25  |\n| Probe   | 6.37  | 60.95  |\n| LoRA + Prompt   | 7.70  | 69.89  |\n\n## Training Details\n\nFor all probing experiments, we use small 3-layer MLP with hidden sizes of [256, 128, 64] and ReLU non-linearities, same as used by Azaria and Mitchell (2023) (reference [3] in the submission). \n\nWe find regularization parameter 1.0 to be sufficient for generalization across all models.\n\nFor LoRA + Prompt, we use the following prompt:\n```\nIs the proposed answer correct?\nChoices:\n  (i) no\n  (ii) yes\nAnswer:\n```\n\nWe will make these details explicit in our camera-ready version of Section 5.\n\n## AUROCs Near 0.5\n\nYou are correct that some methods, such as Probe and sBERT, have close to random performance. We don’t believe these results contradict our paper’s main claims. Firstly, we argue throughout the paper that fine-tuning through the model with LoRA leads to much better results than methods that use frozen features. In many cases frozen features get near 0.5 AUROC, while LoRA + Prompt gets near 0.7, which is significantly better than random. Likewise, the “Incorrect” label refers to a baseline and its relatively poor performance demonstrates that our model performs well because it learns a meaningful connection between questions and correct answers. Secondly, while several baseline methods have AUROCs near 0.5 in aggregate, and this demonstrates their inferior performance compared to other methods (such as LoRA + Prompt), there are many subsets of MMLU for which these models perform significantly better than random (though still worse than better methods like LoRA + Prompt). We provide a detailed breakdown over all MMLU tasks in Appendix D.\n\n## Related Work\n\nThank you for bringing Dennis et. al. to our attention. We see their work as complementary to ours. Both our paper and their paper show that LLMs can be fine-tuned to improve uncertainty estimation but the origin of the labels is different. In their paper, a sampling method is distilled into a single uncertainty estimator, while in our paper, we distill an expert grader into an uncertainty estimator. In practice we find that sampling methods had poor discrimination (low AUROC), so it’s unlikely that training on the outputs of the sampling procedure would yield a useful estimator in our case. In cases where sampling performs on par with an external grader, however, distilling from a sampling method might be preferable because of the cost of expert labels. We will add this discussion to our related work section in the paper.\n\n## Clarifications on Figure 1\n\nThere are 12 points in the plots in Figure 1 (left). These correspond to the following 12 models: LLaMA-2 7B, LLaMA-2 7B Chat,  LLaMA-2 13B, LLaMA-2 13B Chat,  LLaMA-2 70B, LLaMA-2 70B Chat, LLaMA-3 8B, LLaMA-3 8B Instruct,  LLaMA-3 70B, LLaMA-3 70B Instruct, Mistral 7B, Mistral 7B Instruct. The plots in Figure 1 (right) should contain the exact same models. In the case of “Zero-Shot Classifier”, the LLaMA-2 70B numbers were dropped accidentally. We include the AUROC and ECE numbers below:\n\n| Model    | Method      | Accuracy     |  ECE      |  AUROC     |\n|:---------|:------------|:------------|:------------|:------------|\n| LLaMA-2 70B  | Verbal | 47% | 32% | 56% |  \n\nThe 6 points displayed for “Verbal” correspond to only the Chat/Instruct model versions. Only the Chat/Instruct version were used because base model versions were all incapable of generating verbalized probabilities with prompt engineering on its own. We agree that including more models would be ideal. We are limited by our access to compute and make our best effort to run many of the state-of-art models available at the time of submission. We also included a 95% bootstrapped confidence interval for our linear fit to give a sense of the statistical power of the trend line. We don’t think there’s any reason to believe that there will be a sudden inflection point in the curves for “Zero-Shot Classifier” or “Verbal”, and other papers covering uncertainty estimates for models as powerful as GPT-4 have also found lack of calibration/discrimination that persists for even the most capable models [1]. \n\nYou also asked whether the scale of perplexity numbers affect the final AUROCs. AUROC is scale invariant because the true positive rate (TPR) and false positive rate (FPR) for a predictor $f$ and cutoff $k$ are equivalent to the TPR and FPR of a predictor $g = c \\cdot f$ with cutoff $c \\cdot k$. AUROC is an integral of TPR over the FPR in the interval [0,1], and we could rewrite AUROC($g$) as AUROC($f$) using a change of variables. AUROC therefore only captures how well each model ranks the incorrect and correct answers. ECE by contrast, is not scale invariant and is only appropriate for normalized values, which is why we use AUROC in this comparison. \n\n## References\n\n[1] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ArXiv, abs/2306.13063, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MMnhQjz2kq", "reviewer": "Reviewer_h75Z", "summary": "### Summary\n\nThis paper presents a method to quantify the confidence of predictions from an LLM. Unlike some previous work, the authors propose a training-based approach to teach a model to generate reliable confidence scores. The experiments report Expected Calibration Error (ECE) and AUROC, commonly used in calibration literature, to ensure that the model's predicted confidence aligns with actual likelihood. Typically, papers in this area either tune the model or do some post-processing on the model to obtain calibrated confidence scores.\n\nHowever, this paper is slightly different in its setup, and I found the title somewhat misleading. The main difference here is that there is an auxiliary model to emit the confidence scores, entirely independent of the primary model, as stated in Line 327 of the paper, \"Currently, finetuning relies on two separate models for question answering and uncertainty estimation.\" In this sense, this auxiliary model serves better as a potential guardrail to filter out less confident predictions, and framing the paper in this light would make it easier to convey its message.\n\nI also dislike using the terms \"uncertainty estimation\" and \"confidence estimation\" interchangeably. Please refer to Section 3.2 of the paper \"Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models,\" which does a good job of eliciting the difference between uncertainty and confidence.\n\n\n#### Main Contribution of the Paper:\n\nThe central contributions of the paper are\n- Assembling a finetuning dataset to calibrate the model on.\n- LoRA finetuning of the auxiliary model for reliable uncertainty estimate.\n- Testing the robustness of this model to distribution shifts.\n\nThe paper also contradicts findings from previous papers, such as\n- Frozen features are insufficient for uncertainty estimates (The internal state of an LLM knows when it's lying)\n- Transferability of uncertainty estimates across models, which is different from the findings in Kadavnath et al. (Language Models (Mostly) Know what they know)\n\nWhile I haven't read the previous works in detail, my understanding is that they attempt to predict the uncertainty from the model that generates it. There is no auxiliary model involved.\n\n\n#### Experiments and Results\n\n- Figure 1 shows that perplexity is not indicative of model uncertainty.\n\t- However, the open-ended generation setting seems slightly contrived and is not truly open-ended. **What is the average length of the answer in the MMLU dataset?** Since a Multiple Choice Dataset was converted to open-ended generation, the answers might comprise only a few tokens.\n- The paper proposes three approaches to emit the confidence score.\n\t- \"Probe,\" which trains a small feedforward neural network on the last layer\n\t- LoRA (same as Probe with LoRA weights)\n\t- LoRA + Prompt (same as LoRA with prompt posing it as a multiple choice question (i) and (ii)\nThe dataset used for finetuning was assembled from a diverse collection of benchmark datasets, such as AI2 Reasoning Challenge, Boolean Questions, CosmosQA, etc. I looked at the MMLU paper and found no evidence of any data contamination, i.e., test data accidentally leaking into the training data. I suggest calling this out explicitly in the paper.\n- Table 1 shows regularization improves calibration.\n- Figure 2 shows that the three proposed approaches improve calibration and selective prediction. However, since the open-ended generation setting is not truly open-ended (see my comment above), I suggest conducting additional experiments to predict the uncertainties of longer sequences. This could be one reason why the findings are contradictory to previous papers.\n- Regarding my comment about training an auxiliary model, the confidence generation model is entirely detached from the primary model, which generates the answer. Based on the result of the transferability of these scores, you can arbitrarily mix and match any open-weight model with any other model to get reliable uncertainty estimates. This suggests that this auxiliary model has learned a notion of \"correctness\" independent of the original model from the training data. It's unlikely that this technique would work on niche domains like finance and medicine since the training data is unlikely to contain such examples.\n\t- Line 253: Humanities and social sciences overlap, so this claim is not compelling.\n\t- Question Figure 3(b): I see that a tuned model leads to a more uniform distribution of probability scores in the first figure compared to the zero-shot which has a mode around 70%. Why is this calibrated? (Line 279) Shouldn't you use a calibration curve to claim if scores are calibrated?\n\n- The experiment presented in section 6.1 is convincing in showing that learned uncertainty estimates depend both on the intrinsic difficulty of the question and the learned correspondence between the question and the correct answer. This also corroborates my earlier skepticism about using this technique to niche unseen domains.\n\n\nI like Section 7 a lot. Thank you for this study on how uncertainty estimates correlate with user decision-making.\n\n\n### Nits and suggested improvements:\n- Calling out contributions of the paper more explicitly towards the beginning of the paper.\n- Line 39-40: \"In particular, we consider whether it's possible to have good uncertainties over correctness (rather than tokens) out of the box.\" - consider rephrasing this; this sentence is unclear.\n- Line 218: \"The probability of each cluster becomes the probability assigned to each sequence in that,\" this sentence is unclear.\n- Line 959 in appendix is incomplete sentence", "questions": "See comments above", "limitations": "See comments above", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "### Strengths\n\nThe paper is concise yet comprehensive, with clear and accessible writing. The appendix offers thorough explanations of hyperparameter selections, auxiliary techniques, and evaluation metrics. While the results align with expectations, they don't break significant new ground. The literature review provides a solid contextual foundation for the work.", "weaknesses": "See comments above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your comments below.\n\n## Connection between Base Model and Uncertainty Estimate\n\nAs you note, the model that generates uncertainties is not exactly identical to the model that generates answers. However, the assertion that these models are entirely independent or that our method is a departure from prior work is fundamentally incorrect.\n\nThe setup we employ is the standard in every supervised learning method in our related work section. Lin et. al. and Kadavath et. al., for example, are both foundational papers in LLM uncertainty estimation that employ a separate uncertainty model in their finetuning experiments.. Azaria and Mitchell also use a probe, which results in two separate models, one for generating uncertainties and one for generating answers. In fact, only the zero-shot classifier or sampling methods don’t adopt this two-model approach, and as we show in the experiments they tend to have weaker performance.\n\nEven if our setup was not standard in prior work, it would still be inaccurate to characterize the two models as independent. To make this clear, we can review the details of the three methods we explore: \n- Probe: The uncertainty model and base model share features, and the MLP has approximately 50K parameters, effectively sharing 99.999% of their parameters. \n- LoRA: For approximately 3.5M LoRA weights in a 7B parameter model, and 50K parameters in the MLP classifier, the models share 99.95% of their parameters. \n- LoRA + Prompt: This approach only uses LoRA and no new head. Therefore the models share 99.95% of their parameters. \nFor LoRA + Prompt, we also regularize the KL divergence between the base model and uncertainty estimate (Section 5, Table 1), and this regularization makes not just the weights but the outputs of the two models tied together.  \n\nMaking the two models exactly identical is an exciting avenue for future work. There is no conceptual reason that our method shouldn’t extend to that setting, but in practice it’s challenging to fine-tune state-of-the-art LLMs while avoiding catastrophic forgetting in answer generation will most likely require mixing large batches of graded data with standard pre-training data. Most academic groups are simply ill-equipped to run this experiment, and it doesn’t seem fair to penalize us for having limited compute, especially when our setting aligns with prior work.\n\n## Length of Open-Ended Answers\n\nTo determine whether open-ended MMLU is sufficiently different from multiple-choice MMLU, we can look at some basic statistics of the generations sampled from each model. \n\n| Model    | Avg Answer Length         |\n|:---------|:------------|\n| LLaMA-2 7B  | 9.7 | \n| LLaMA-2 7B Chat   | 11.5 |\n| LLaMA-2 13B    | 9.7 |\n| LLaMA-2 13B Chat  | 10.3 | \n| LLaMA-2 70B    | 9.5 |\n| LLaMA-2 70B Chat  | 56.9 | \n| Mistral 7B  | 10.3 |\n| Mistral 7B Instruct | 25.2 |\n| LLaMA-3 8B | 9.0 |\n| LLaMA-3 8B Instruct | 13.4 |\n| LLaMA-3 70B | 8.9 |\n| LLaMA-3 70B Instruct | 10.7 |\n\nWhile the generations do not consist of whole paragraphs, as we describe in Section 4 (“Do We Get Good Uncertainties Out of the Box?”), the behavior of probabilities over 10 tokens is fundamentally different from the behavior of probabilities over single tokens because of multiple possible phrasings. We make this distinction clear in Figure 1 left by showing that probabilities over single tokens are excellent discriminators while perplexities over many tokens are not. \n\nAdditionally, we run two experiments that should help clarify that our results do in fact extend to long sequences. First, Figures 11-13 in Appendix E show that the uncertainties predicted by our models do not have a consistent relationship with the length of the sequence. Many of the sequences displayed are between 100 and 200 tokens long, which is comparable to a long response in a dialogue with a chat model. \n\nIn addition to these results, we also ran an additional experiment to augment the perplexity results in Figure 1. We explore how perplexity relates to quality scores on a dataset with very long graded generations. MT-Bench contains approximately 100 long-form questions with answers in dialogue form. We use precomputed answers graded with GPT-4 for LLaMA-2 7B Chat, LLaMA-2 13B Chat, and LLaMA-2 70B Chat and compute the perplexity associated with each answer.\n\n| Model    | Spearman’s Rho        |\n|:---------|:------------|\n| LLaMA-2 7B Chat  | 0.46 | \n| LLaMA-2 13B Chat  | 0.27 |\n| LLaMA-2 70B Chat  | 0.28 |\n\nFrom the low Spearman’s rhos between the perplexity assigned by the model and the score given by GPT-4, we can see that perplexity is not a reliable predictor of quality, and furthermore, perplexity does not become a more reliable predictor of quality as model sizes and capabilities increase, unlike previous work.\n\n## Humanities vs. Social Sciences\n\nWhile “humanities” and “social sciences” are often synonymous, they do have distinction definitions as super-categories of MMLU. While literature is in the super category “humanities” in MMLU, legal briefs and economics are defined as “social sciences.” It’s not obvious that reading literature should confer an understanding of economics, or vice versa. \n\n## Calibration of Unanswerable Questions\n\nOur purpose in displaying the confidence score is to show that the confidence for unanswerable questions is significantly lower than the confidence scores for answerable questions. The ECE of the answerable questions is 0.185 and the AUROC is 0.78. These represent 36% and 15% improvements over the baseline respectively.\n\n## Closing Remark\n\nWe’ve done our best to address your comments in detail and provide additional experiments where possible. We hope that you will consider raising your score in light of these clarifications and new evidence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h4jliyEyUI", "reviewer": "Reviewer_BBBX", "summary": "This paper studies calibration of large language models. Firstly, the paper points out the zero-shot outputs of LLMs are poorly calibrated, especially in an open-ended generation setting. Secondly, the paper explores fine-tuning a separate LLM for calibration, and demonstrates its good properties such as OOD generalization, data efficiency, etc. Lastly, the paper presents a preliminary study of presenting model uncertainties to human users and investigating how this affects human decision making process.", "questions": "Clarification questions:\n\n* Line 162: Do your results contradict with those in Kadavath et al. 2024? If yes what are differences in your experiment settings that may lead to this?\n* Line 177: Does each dot in Figure 1 (right) represent one task in MMLU, or representing one LLM? Were different sizes of Llama2 and Llama3 used here?\n* Figure 1: Do you observe any patterns for base and instruct models? As discussed in https://arxiv.org/pdf/2303.08774 Figure 8, post-training may hurt calibration. \n* Section 5: I think more details are needed for me to understand how the fine-tuned calibration model is trained.\n  * How do you get P(correct) from the model. Did you train a classification head? Or is it obtained in a way similar to the zero-shot classifier in Line 159?\n  * How do you formulate your training set? It was mentioned in the abstract that a mixture of correct and incorrect answers are needed. However this was not explained clearly in Sec 5.\n  * How is the trained calibration model different from a trained “verifier”, e.g., as done in https://arxiv.org/abs/2110.14168\n* Line 287: “model trained on incorrect answers perform surprisingly well”, does this mean the two hypotheses are both true and account for part of the success of the the trained calibration model?\n* Figure 5: I’m not sure how to interpret the figure. How do you get “an increased reliance on the LLM for more confident answers, and decreased reliance for less confident answers” from subfigure (b)? Is this done by comparing subfigure (a) and (b)?\n\nDiscussion questions:\n* Line 132: “Early work on LLMs such as GPT-3 showed … poorly calibrated”. GPT-3 actually showed reasonable calibration on MMLU in a few-shot in-context learning setting (Figure 11 in https://arxiv.org/abs/2009.03300). I wonder what’s the authors comments on this. It seems that the paper mainly focuses on zero-shot settings. Is there any reason this is not included in the paper? Would few-shot ICL also be a viable method for calibration?\n* I like the idea of infusing calibration into the pre-training phase, discussed in Sec 8. Can you elaborate on the challenges of online learning here? From what I understand, the calibration training data used in Sec 5 is stationery.\n\nMissing Reference:\nYou may be interested in \n* https://arxiv.org/abs/2104.08315 (Corresponds to the phenomenon that you mentioned in Line 137)\n* https://arxiv.org/abs/2302.13439; https://arxiv.org/abs/2401.06730 (Studies verbalized uncertainty in LLMs)\n* https://arxiv.org/abs/2305.14802 (Studies task-level calibration by training a meta-model, also studies open-ended generation settings)\n\nOverall I think this is a solid and insightful submission. I'm open to adjust my ratings if some of these confusions are resolved.", "limitations": "Limitations are discussed in Section 8.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is very comprehensive and presents an in-depth analysis on the calibration of large language models. \n* The takeaway messages are summarized clearly after each section. The insights are useful to users and developers of large language models. Plus, the proposed future directions are interesting.", "weaknesses": "* The paper mainly focuses on calibration of zero-shot prompting LLMs, while prompting the model with few-shot examples may also be a viable calibration method. I think this should be added into the discussion and maybe added to the experiments to further strengthen the work.\n* Due to the page limit, many important details are left in the appendix and the paper may be improved with a better organization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re glad that you find the paper informative and appreciate your feedback. Thank you for sharing the additional reference, which we will incorporate into the related work section. We respond to your questions below, and make several clarifications. We also provide additional results in the general post which you might find helpful. We value your support and would appreciate it if you would consider increasing your score in light of our response and the timeliness and comprehensiveness of this work.\n\n\n## Differences from Kadavath et. al.\n\nKadavath et al. is closely related to our work, but a few key details of our evaluations and conclusions differ. From the top, our key emphasis is on answers with more than one token (open-ended versus multiple-choice). We explore how open-ended answers can have fundamentally different behavior from multiple choice answers because of ambiguity in phrasing. Unlike Kadavath et al., we show that prompting strategies (e.g. zero-shot classifier, verbal elicitation) are often relatively weak when compared to fine-tuning methods. These methods often improve very slowly, if at all, with the strength of the underlying model. By contrast, fine-tuning methods are effective even for relatively small models and their performance improves with the power of the underlying model.\n\nKadavath et al. also includes experiments with fine-tuning a model on graded examples, but they are different from our own in several ways. Kadavath et al. train a correctness classifier which takes only questions as input, without answers, estimating whether a question might be answerable but not whether a given answer is correct. As we explore in Section 6.1, a model conditioned on only questions is sensitive to learning shortcut features of the question, effectively clustering question topics. By contrast, we show that our model is learning the underlying relationship between the question and answer. Beyond differences in the setup of our fine-tuning experiments Kadavath et al. suggests that language models are strongest when evaluating their own generations and subsequently posit that uncertainty estimation is linked to self-knowledge. We find that capable models can readily learn good uncertainties for predictions of other models without any knowledge of their internals (see Section 6.1). \n\n## Clarifications to Figure 1\n\nEach dot represents a single model, the models included are LLaMA-2 (7B, 7B-Chat, 13B, 13B-Chat, 70B, 70B-Chat), LLaMA-3 (7B, 7B-Instruct, 70B, 70B-Instruct), and Mistral (7B, 7B-Instruct). For “Verbal” only the Chat/Instruct models are included because base models generated nonsensical probabilities from when using only prompting. The coordinates of each dot show the test metric averaged over all subsets of MMLU. \n\n## Base / Instruct Models\n\nIn terms of ECE and AUROC, Figure 7 of the appendix provides a detailed breakdown for all base and instruct models. Unlike in the paper you referenced, base models typically have worse accuracy, ECE, and AUROC when using prompting methods (e.g. zero-shot classifier). One potential reason for the difference is the mismatch between multiple-choice and open-ended settings. Post-training can impair the per-token calibration of a model without hurting its calibration at the level of correctness/incorrectness (captured by prompting methods or trained estimators). In our experiments, post-training can actually be helpful because it improves the model’s ability to adhere to formatting or instruction constraints. Instruction-tuned models tend to have better open-ended accuracies overall (because their answers are direct and concise) and perform better with zero-shot prompting methods where the output format can be complicated or abstract (as in the case of outputting probabilities as numbers). \n\nNotably, however, our fine-tuning procedures create useful estimates for both base and post-trained model variants, with improved ECE and AUROC. Models with calibration deteriorated by RLHF can be re-calibrated with post-training on graded examples.\n\n## Details on estimating P(correct)\n\n“Probe” is a classification head on top of the penultimate layer (before token prediction) of the LLM, where the input is the representation of the last token in the context. This head, a small 3-layer neural network, similar to the prescription of Azaria and Mitchell (reference [3] from the submission), serves as the model for binary classification (correct v/s incorrect).\n\nOur “LoRA + Prompt” method is essentially a trained version of the zero-shot classifier in Line 159, where instead of attaching a head, where we use the same prompt as in the zero-shot classifier but train through the features of the model using low-rank adapters (LoRA). This approach allows the correctness predictor to extract slightly different information from the intermediate layers of the LLM. The loss is computed over the two tokens corresponding to the options (i) and (ii).\n\n## Training Set Details\n\nOur training set is a uniform mixture of various open datasets listed in Appendix C.2. For each model, we generate outputs for the samples in the mixture labeled for correctness (as graded by GPT-3.5). Because we use generations from the model and the models are not perfect, we get both correct and incorrect labels in our training set. We will make these details explicit in Section 5.\n\n## Uncertainty Model vs Verifier\n\nIn principle, a trained calibration model is very similar to a trained verifier. However, a trained verifier in the linked paper is being used to select a single answer from a set of generated samples by estimating which is most likely to be the correct answer. By contrast, our goal is to estimate uncertainty for the answer that has been chosen. We therefore care about properties like calibration (ECE) which indicate that the model’s confidence is not just useful for filtering but also interpretable as a probability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ovcwZVMaWB", "reviewer": "Reviewer_XHRg", "summary": "The paper investigates uncertainty calibration via fine-tuning models with correct and incorrect answers. The paper concludes that training through the features of a model on a thousand graded examples using LoRA outperforms prompting-based method and linear probes. The paper also finds that models can be used not just to estimate their own uncertainties but also the uncertainties of other models. Human study shows LLM uncertainty can affect human decision making.", "questions": "Upon the statement of self-knowledge, “We show that, by contrast, Mistral 7B actually has better AUROC values when applied to LLaMA-2 7B than LLaMA-2 7B applied to itself” (line 295), how is Mistral 7B applied to LLaMA-2 7B? \nSpecially, 1, Which model is used for training? 2, Which models generate the answers for the training data and test data, respectively?", "limitations": "good discussion", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, The paper emphasizes the importance of training-based uncertainty estimation methods.\n2, The experiments cover various aspects, including data amount, distribution shifting, cross-model generalization, and a human study.", "weaknesses": "1. Lack of technical novelty. The training methods regarding Probe, LoRA, and LoRA+Prompt are highly similar to previous works. No new technical method is proposed.\n2. Limited experimental scope: The experiments are mainly based on the MMLU dataset. This raises concerns about the generalizability of the findings to more tasks and datasets. Previous research has shown that task/dataset complexity can significantly impact uncertainty calibration performance.\n3. Inconsistent results across MMLU subsets: As shown in Figures 8, 9, and 10, sometimes LoRA + Prompt performs better, sometimes LoRA performs better, and sometimes Probe performs better. Additionally, there is no consensus between AUROC and ECE on MMLU subsets. Methods that perform best on AUROC sometimes perform poorly on ECE. The inconsistency is a common observation rather than an exception. Based on that, the average performance among subsets (Figure 2, left) is not convincing when performance varies significantly across different subsets.\n4. Lack of confidence distribution for correct and incorrect predictions on MMLU: The paper does not show confidence distribution for correct/incorrect predictions on the MMLU dataset. Although Figure 3 (right) shows confidence distribution on answerable/unanswerable sets on the SelfAware dataset, a calibrated uncertainty method should also distinguish between correct and incorrect predictions on the MMLU dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback. We respond to your comments below, providing numerous clarifications and new results inspired by your comments. \n\n## Technical Novelty \n\nWe would like to clarify that our goal in the paper was to show that more complex methods, including those with technical complexity, are in fact not necessary when a small number of labeled examples are available. Given the rising popularity of approaches that use zero-shot prompting or sampling, we think this point is clearly important and runs contrary to prior work, making it sufficiently novel. Rather than introduce yet another method with technical complexity, we wished to step back and scientifically understand key facets of calibration for language models. And indeed our experiments provide many novel insights that should not be dismissed. We found that regularization and use of LoRA were critical for good performance. These observations go beyond prior work. We ultimately see use of existing popular tools like LoRA as a strength, facilitating broad adoption. \n\n## Extended Experimental Results\n\nInspired by your comments, we provide results on the MMLU-Pro benchmark, which is a more challenging multi-task language understanding benchmark. This evaluation was created after the release of all models we consider therefore there is no risk of contamination. Tables containing the results are available in our general rebuttal comment. Overall, the new results mirror our original results on MMLU and provide evidence that our estimates do generalize. MMLU is also considered significantly harder than MMLU and frontier models typically have lower accuracy on MMLU-Pro by 20% or more. \n\n## Trends in MMLU Performance\n\nWhile there is variance in each metric, there are clear trends in the relative performance. To make this clear we can look at the win rate of LoRA + Prompt over Zero-Shot Query as an illustrative example. The table below shows these win rates for AUROC and ECE. In AUROC, there is a clear advantage to using LoRA + Prompt for all models. ECE is less consistent, but the majority of variance is the result of the model, not the subset of MMLU: \n\n| Model  | AUROC Win Rate   | ECE Win Rate   |\n|:---|---|---|\n| LLaMA-2 7B   | 98%  | 20%  |\n| LLaMA-2 7B Chat  | 100%  |  93%  |\n| LLaMA-2 13B  | 96%  | 100%  |\n| LLaMA-2 13B Chat   | 98%  |  71%  |\n| Mistral 7B   | 98%  | 35%  |\n| Mistral 7B Instruct  | 96%  | 82%  |\n\nWe found that Mistral 7B in particular was highly anomalous among the models we considered. When we consider llama-3 models, which are already included in Figure 1, most models do not display good zero-shot calibration:\n\n| Model  | ECE   |\n|:---|---|\n| LLaMA-2 7B   | 0.19  |\n| LLaMA-2 7B Chat  | 0.56  |\n| LLaMA-2 13B  | 0.40  |\n| LLaMA-2 13B Chat   | 0.23  |\n| LLaMA-2 70B  |  0.32  |\n| LLaMA-2 70B Chat   | 0.18  |\n| Mistral 7B   | 0.12  |\n| Mistral 7B Instruct  | 0.24  |\n| LLaMA-3 8B   | 0.58  |\n| LLaMA-3 8B Instruct  | 0.61  |\n| LLaMA-3 70B  | 0.47 |\n| LLaMA-3 70B Instruct   | 0.49  |\n\nWe also include results for Qwen-2 7B-Instruct, a strong model released by Alibaba, to make it even more clear that Mistral 7B is anomalous in its ECE results: \n\n| Method  | ECE   | AUROC   |\n|:---|---|---|\n| Zero-Shot Classifier  | 47.57  | 68.06  |\n| Probe   | 20.72  | 61.16  |\n| LoRA + Prompt   | **14.90**  | **74.94**  |\n\nThese results have the corresponding win rates:\n\n| Model  | ECE Win Rate   | AUROC Win Rate |\n|:---|---|---|\n| Qwen-2 7B-Instruct   | 99%  |  85%  |\n\nWhile some models are surprisingly calibrated zero-shot, these models often have bad AUROCs. This is possible because probabilities can be calibrated without being useful for making predictions. For example, a model can completely ignore the input and still have perfect calibration if the statistics of its predictions match the statistics of the data, even if the predictions are completely random. Ideally we want uncertainty estimates that are both good at discrimination and are calibrated, because such a model can reliably filter incorrect answers from correct answers while also having probabilities that are easy to interpret (because they correspond to real rates of correctness or incorrectness).\n\n### Confidence Distribution on MMLU \n\nThere are two points worth clarifying. First, our main aim in showing the histogram of confidence scores was to show that unanswerable questions obtain lower confidence under our model. Lower confidence doesn’t necessarily imply better calibration, as correctness cannot be defined for unanswerable questions. Second, MMLU is different from SelfAware because correctness is well-defined for all questions, and therefore we can use AUROC and ECE to evaluate the quality of our uncertainty. Although we do not show the confidence distribution in a plot analogous to Figure 3 (right), the confidence distribution of correct and incorrect answers is already captured in the ECE and AUROC metrics. \n\n### Summary of Contributions\n\nGiven the widespread adoption of LLMs the significance of this paper’s subject is self-evident. We provide several useful observations that are directly applicable to improving the usability of LLMs for question-answering. These observations are deliberately simple but are novel and contradict statements made in prior work. For instance, we show that fine-tuning with even a small number of labeled examples can dramatically outperform prompting and sampling methods in the majority of cases. We also include an extensive analysis of our method’s robustness to distribution shifts and even study how our uncertainties affect user behavior in a study of human-AI collaboration. \n\nWe hope that you will consider raising your score in light of our updated experiments and clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bmaLKHX9vO", "reviewer": "Reviewer_AVYo", "summary": "This work investigates the calibration of LLMs, i.e. how to estimate reliable confidence in their responses that correlates well with the actual likelihood of being correct. Through experiments, the authors reveal that existing methods fall short of achieving accurate confidence estimates, and their calibration performance does not improve with the increased size of LLMs. The authors argue that fine-tuning is necessary to obtain reliable confidence estimates. They explore three fine-tuning approaches: Probe, LoRA, and LoRA + Prompt, and their experiments confirm the effectiveness of these paradigms. Additionally, the authors examine the generalizability of these methods and their benefits for users in making decisions.", "questions": "1. I don't quite understand Figure 3 (Left). The figure with the y-axis labeled \"# train\" shows the distribution of training samples. What does the figure with \"# MMLU\" represent?\n\n2. From my experience, in the sampling baseline, the size of the largest cluster can be used to represent uncertainty (size of the largest cluster/# of clusters), and sometimes it performs better than probability. Have you tried this approach?\n\n3. Do you use the correct and incorrect answers provided in the datasets, or do you sample LLM generations to construct incorrect answers? If it is the former, would the latter approach help to better evaluate the uncertainty of the LLM's own generation?\n\n4. The following related studies are missing:\n\n[1] LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses\n\n[2] Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.", "limitations": "The authors acknowledge the limitation of needing to train separate models. Personally, I believe that at least the Probe method can be directly applied to the LLM without altering the LLM itself. Exploring how to better evaluate confidence directly from the LLM could be a promising direction.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: This paper proposes three fine-tuning methods for estimating confidence in LLM responses. While previous works have used similar techniques like Probe to gauge uncertainty, the combination of LoRA and prompts for calibration is novel.\n\nQuality: The ideas presented in this paper are well-motivated, and the experiments effectively verify the authors' hypotheses.\n\nClarity: The paper is well-written and easy to follow.\n\nSignificance: The paper's conclusion, that LLMs need to be fine-tuned to predict accurate confidence, is valuable for the community. Additionally, the investigation into the benefits for users in making decisions is both novel and significant.", "weaknesses": "1. One clear drawback of the proposed methods is that they require access to the parameters of the LLM, making them inapplicable to black-box LLMs. Figure 4 (right) indicates that a fine-tuned smaller model can estimate the confidence of larger ones. However, it would be preferable to see a comparison between these fine-tuning-based methods and other techniques that do not require access to the LLM, such as sampling.\n\n2. The authors claim that \"(LoRA + Prompt) performs almost as well in transfer as on in-distribution data.\" However, in open-ended generation, the gap between transfer and in-distribution is almost 10 points for both ECE and Auroc, which is a significant margin. For instance, ECE increases by nearly 80%.\n\n3. Section 6.1 is somewhat confusing. While the two possibilities presented seem reasonable, the experimental investigation appears problematic (correct me if I'm wrong). The topic information is not only available in model generation but could also be reflected in the questions. The experiment that replaces the model generation with a random sentence does not adequately address this situation. On the contrary, the results seem to support that the model can predict reliable confidence based solely on the question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and supportive review! \n\nHere is our response to the weaknesses you listed pointwise: \n\n### Response 1 \n\nYes, our method requires access to the model parameters. Luckily for our method, there are incredibly strong open-source LLMs available. The most recent and relevant case is Meta releasing LLaMA 3.1 405B, which has performance on par with the best models from OpenAI and Anthropic. We’re confident that as the abilities of closed-source LLMs progress there will also be parallel progress in open-source LLMs, and therefore any method that requires fine-tuning the model should be possible to run in a fair comparison with state-of-the-art models. \n\nYou proposed a comparison with sampling when estimating uncertainties for a different model. We assume that you mean that you want to see a comparison between, for example, the uncertainty estimates of LLaMA-2 applied to Mistral generations and uncertainty estimates from sampling Mistral by itself. We can provide those numbers below:\n\n| Answer Model  | Method   | AUROC   |\n|:---|---|---|\n| Mistral 7B  | Sampling | 0.53 |\n| Mistral 7B  | LoRA + Prompt (LLaMA-2 7B) | 0.68 |\n| Mistral 7B  | LoRA + Prompt (Mistral 7B)  | 0.69  |\n\nAll models are trained/evaluated on answers generated from Mistral 7B and we show the model used to construct the uncertainty estimate in parentheses (where applicable). If you were instead proposing a different comparison, please let us know. From the table, it’s easy to see that even constructing an uncertainty estimate from a different language model (e.g. LLaMA-2 7B applied to Mistral 7B) significantly outperforms a sampling method applied to the same model.\n\n### Response 2\n\nWe included bars for Probe and Zero-Shot Classifier in the transfer plots so that they could be used as reference points when evaluating the transfer ECEs and AUROCs. While you’re correct that there are notable differences between the in-distribution and transfer performance of the runed models, these large percent differences are not so significant in the scheme of the broader evaluation. While the ECE numbers increase in transfer and AUROC numbers decrease, the final transfer values are significantly better than the baseline methods applied \\it{in-distribution}. Therefore in many cases it might be as practical to apply LoRA fine-tuned models in transfer as applying a worse method to data from the actual downstream task. \n\n### Response 3\n\nWe’re happy to provide some clarifications. As you point out, one possibility is that the uncertainty estimates are using information present in the question to predict how hard a question will be, independent of the actual answer that was generated. The baseline with ground truth questions and incorrect answers provides a sense of how good such a heuristic could be in practice. As we can see, it does actually perform reasonably well. But it performs significantly worse than when the model is provided with both the ground truth question and a real generated answer which demonstrates that the relationship between the answer and question matters. If the uncertainty estimates were purely using the topic of the question or answer, we would expect both methods to perform roughly the same, because the relationship between the question and answer wouldn’t affect the topic they are associated with. However, we observed that the performance is different across all models. \n\n### Question Answers\n\nHere are answers for your list of questions:\n- “MMLU (MC)” and “MMLU (OE)” in the x-axis label on the Figure 3 (left) represent the multiple-choice (MC) and open-ended (OE) settings of the massively multitask language understanding (MMLU) dataset. The position of the “MMLU” x-axis labels is suboptimal and we apologize for any confusion. The labels would be better placed above the plots as titles instead of x-axis labels.\n- Conveniently we already ran this comparison for the initial submission, and a more detailed explanation can be found in Appendix B.1. While we tried both counting and likelihood accumulation for the sampling methods, we did not find a significant difference in the performance of these variants on our evaluation, though there may well be a difference on other tasks. \n- We sample the model to construct correct and incorrect answers. As a result, the learned estimator is specific to the model used to generate the answers and not to the dataset of question-answer pairs. While the question-answer pairs are fixed for a given dataset, different models can have very different rates of correct and incorrect answers, which necessitates model-specific training.\n- Thank you for sharing these additional related works. [2] is already referenced in our submission as [59], with citations in both Section 2 and Appendix B.2. We will also add LitCab to our overview and cite it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for your feedback. We wish to emphasize that our paper is closely engaging with an extraordinarily significant and timely topic: how do we calibrate foundation models to select for factual accuracy? We consider many facets of this question, including even a human+LLM collaboration setting. We also provide insights into why uncertainty estimation becomes challenging in open-ended settings and how we can construct estimators when we have access to a small dataset of graded answers. 

Our work contains many novel findings, even contradicting prior work, such as pointing out the relatively weak performance of prominent prompting/sampling methods or demonstrating the role of learned uncertainty estimates in a user study. By focusing on an array of key practical questions, our work acts as an important and timely foundation for future research in LLM safety.

Several reviewers highlighted that our work is particularly comprehensive and provides many insights. We appreciate this recognition, as this work was a major undertaking. We have also put a significant effort into answering reviewer questions and providing clarifications, including several new results inspired by reviewer comments. We hope the timeliness of the work and our response can be carefully considered in the final assessment.

We briefly summarize some of the new results in this general post. Then, we provide an itemized list of contributions. Finally, we have separate posts for replying to reviewers individually.

## MMLU-Pro Results

To demonstrate that our results are not specific to MMLU, we’ve run an additional experiment on MMLU-Pro, a new benchmark that is completely separate from and more challenging than the original MMLU but built with the same goal of testing a broad range of knowledge and reasoning tasks. As in our original MMLU results, we construct an open-ended version of MMLU-Pro and use GPT-4 to match samples from the model with official answers. We show the results for 4 models below:

### LLaMA-2 7B-Chat

| Method  | ECE   | AUROC   |
|:---|---|---|
| Zero-Shot Classifier  | 68.66  | 53.29  |
| Probe   | **6.34** | 59.79  |
| LoRA + Prompt   | 26.34  | **65.55**  |

### LLaMA-2 13B-Chat

| Method  | ECE   | AUROC   |
|:---|---|---|
| Zero-Shot Classifier  | 32.11  | 54.01  |
| Probe   | **7.58**  | 63.10  |
| LoRA + Prompt   | 9.38  | **68.33**  |

### Mistral 7B-Instruct

| Method  | ECE   | AUROC   |
|:---|---|---|
| Zero-Shot Classifier  | 19.50  | 60.25  |
| Probe   | **6.37**  | 60.95  |
| LoRA + Prompt   | 7.70  | **69.89**  |

## Qwen Results

To show that our results are not limited to our original set of models (LLaMA-2, LLaMA-3, Mistra), we also include results for another strong 7B parameter model–Qwen-2 7B-Instruct–below:

| Method  | ECE   | AUROC   |
|:---|---|---|
| Zero-Shot Classifier  | 47.57  | 68.06  |
| Probe   | 20.72  | 61.16  |
| LoRA + Prompt   | **14.90**  | **74.94**  |

## Central Contributions

Our central contributions go beyond a practical recipe for training an uncertainty estimator. We believe that all the points below break significant ground by highlighting key limitations of current methods or proposing new solutions:
- We show that out-of-the-box estimators are unreliable. For example, perplexity fails to provide a consistently good discriminator between correct and incorrect answers. By contrast, supervised methods provide consistently superior uncertainty estimates.
- We highlight how defining uncertainty for variable length sequences is challenging, particularly due to the many ways a sentence can be phrased. We show that a trained predictor of correctness is independent of the input sequence length and has strong empirical performance.
- Our method benefits from scaling dataset size, but does not require a large dataset to work. In Figure 2, we see that even with a number as low as 1000 labeled samples, we are able to significantly improve upon the performance of out-of-the-box estimators, while showing continued improvement in both ECE and AUROC on further scaling.
- We provide a detailed qualitative analysis assessing the nature of generalization of our uncertainty estimators, showing that the estimator must be performing more than simple pattern matching on the topic or difficulty of the question (Section 6).
- Unlike prior work, we show that humans are sensitive to calibrated uncertainties and superior methods are indeed desirable in the first place (Section 7).

Meta-review: This paper looks at getting calibrated uncertainty scores from a model, i.e. estimating the probability that its answer is correct. The authors argue that this is difficult to achieve from prompting alone, but propose fine-tuning a model on a small dataset of correct and incorrect answers and find that this achieves good calibration performance. Moreover, they find that these estimates generalize to some degree across models.

Reviewers agree that this paper investigates an important topic, with implications for reducing hallucination or other factuality errors from LLMs. They find that the experiments were comprehensive and well-executed, with a clear take-away that an approach like LoRA + prompt that includes fine-tuning is necessary to get good calibration. Reviewers h75Z and BBBX also appreciated 

There was some disagreement about novelty, however: reviewers oPif and BBBX had initial concerns but were satisfied with the author response, while reviewer XHRg still felt that the proposed methods were too similar to previous work.

Finally, there were also some concerns, from reviewer XHRg in particular, about consistency of the ECE and AUROC metrics and how to interpret ambiguous performance between LoRA, LoRA + prompt, and probe methods. However, as other reviewers pointed out in discussion, these are all supervised methods and so still support the paper’s main conclusion that weight-based learning of some form is necessary for good calibration.
Decision: Accept (poster)