Teaching Models to Express Their Uncertainty in Words
Authors: Stephanie Lin
Jacob Hilton
Owain Evans
Venue: Accepted by TMLR ()

Abstract: We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. When given a question, the model generates both an answer and a level of confidence (e.g. "90% confidence" or "high confidence"). These levels map to probabilities that are well calibrated. The model also remains moderately calibrated under distribution shift, and is sensitive to uncertainty in its own answers, rather than imitating human examples. For testing calibration, we introduce the CalibratedMath suite of tasks. We compare the calibration of uncertainty expressed in words ("verbalized probability") to uncertainty extracted from model logits. Both kinds of uncertainty are capable of generalizing calibration under distribution shift. We also provide evidence that GPT-3's ability to generalize calibration depends on pre-trained latent representations that correlate with epistemic uncertainty over its answers.

Published in Transactions on Machine Learning Research (10/2022)
Teaching models to express their uncertainty in words
Stephanic Lin University of Oxford
sylin07@@mmiil.com com
Jacob Hilton OpenAl
jhilton&Bpenaai.com
Owain Evans
owaine com
University of Oxford
Reviewed on OpenReview: https: // openreview. net/ forum? id- 8s8N2U2GT2
Abstract
We show that GPT-3 model can learn express uncertainty about answers using natural language without use of model logits. When given a question, the model generates both answer and level of confidence (e.g. "90% confidence" "high confidence"). These levels map to probabilities that are well calibrated. The model also remains moderately calibrated under distribution shift, and is sensitive to uncertainty in its own answers, rather than imitating human examples.
For testing calibration, we introduce the CalibratedMath suite of tasks. We compare the calibration of uncertainly expressed words ("verbalized probability") uncertainly ex- tracted from model logits. Both kinds of uncertainty are capable of generalizing calibration under distribution shift. We also provide evidence that GPT-3's ability to generalize calibra- tion depends on pre- trained latent representations that correlate with epistemic uncertainty over its answers.
1 Introduction
Current state-of-the-art language models perform well wide range of challenging question-answering tasks [Brown et al., 2020 Chowdhery et al. 2022 Hoffmann et al., 2022). They can even outperform the average human on the MMLU benchmark (which consists exam-like questions across 57 categories) and
on BIG-Bench (which consists of 150+ diverse tasks). Yet when models generate long-form text, they often
produce false statements or "hallucinations" (Lin al. 2021 Maynez et al. 2020: Shuster al. 2021 This reduces their value to human users,
cannot tell when model is being truthful or not.
The problem of truthfulness motivates calibration for language models (Nguyen & O'Connor 2015 If models convey calibrated uncertainty about their statements, then users know how much to trust given statement. This is important for current models (which often hallucinate falsehoods) but also for any model that makes statements where there no known ground truth (e.g. open problems science or mathematics, or economic forecasts as in Zou et al. (202)))
Previous work calibration focuses the model log- po--obbbablities "logits" (Guo et al. 2017 Jiang al. 2021). Yet the log-probabilities of models like GPT-3 represent uncertainty over tokens (ways of expressing claim) and not epistemic uncertainty over claims themselves. ves. If a claim can be paraphrased in many different ways, then each paraphrase may have a low log probability[]] By contrast, when humans express uncertainty, this is epistemic uncortainty about the claim itself In this paper, we take a step toward teaching models to
'Sometines it's feasible to over the probabilitics of all paraphrases of claim. But the claim complex, the space of possible paraphrascs will be and hard to demarcate. 2If human says "I think it's likely this vaccine will effective", they express confidence about the vaccine the string "vaccine".

---

Published in Transactions on Machine Learning Research (10/2022)
Q: What is the remainder when 23 is divided by 4?
Prompt
A: 3 Answer generated by GPT3 (greedy decoding)
Confidence: Medium
<- Confidence generated by GPT3 (greedy decoding)
Figure 1: Illustration of verbalized probability and the CalibratedMath task. The prompt is in bold and GPT-3's outpul is in blue. GPT-3 is prompled with question and outputs an answer (*3") and level of confidence in its answer ("Medium"). GPT-3 is scored on the calibration of its confidence (not on the accuracy of its answer). In this example, the answer is correct but the confidence is only "Medium". Using our MSE metric (Section 2.3,, this confidence would score (1 - 0.5)² = 0.25.
express epistemic uncertainty using natural language. We call this "verbalized uncertainty" "verbalized probability".
The goal of verbalized probability is to express uncertainty as part of a model's natural language output without directly mimicking human training data. Models should be calibrated about the likelihood that their own statements are correct, which differs from the uncertainty that a human might have when making a similar statement. For example, GPT-3 outperforms most. humans on a computer security quiz (Hendrycks) let al. 2020) but is much worse than humans at arithmetic questions of the form "2 3 x 7 =??" Thus, we expect pro-trained models will need to be finetuned to produce calibrated verbalized probability.
We focus on the simple case of verbalized uncertainty where the model outputs probabilily or "confidence" as single token (Fig. D This case is a step towards more sophisticated expression of uncertainty in natural language. Moreover, this case relates to methods that finetune model logits for calibration (Jiang et al. 2021), which we also investigate in this paper.
Training models verbalized probability Askell et al.) 2021aa Christiano 2021]. We define model honest if it can communicate everything it represents internally in natural language (and will not misrepresent any internal states). Honesty helps with AT aligmment: if an honest model has misinformed or malign internal state, then it could communicate this state to humans who can act accordingly. Calibration is compatible with a certain kind of dishonesty, because a model could be calibrated by simply imitating a calibrated individual (without having the same "beliefs" as the individual). However, if GPT-3 achieves good calibration on diverse questions after finetuning as in Scction 3.1 it scems unlikely that dishonestly misrepresents its confidence.
component of making models "honest" " (Evans et al. 2021
1.1 Contributions
We introduce a new test suite for calibration. CalibratedMath is suite of elementary mathematics problems. For each question, a model must produce both a numerical answer and a confidence in its answer (see Figure D.. There are many types of question, which vary substantially in content and in difficulty for GPT-3. This allows us to test how calibration generalizes under distribution shifts (by shifting the question type) and makes for challenging test. (see Figure 3). Since GPT-3's math abilities differ greatly from humans, GPT-3 cannot simply imitate human expressions of uncertainty.
GPT-3 can learn to express calibrated uncertainty using words ("verbalized probability")) We finetune GPT-3 to produce verbalized probabilities. It achieves reasonable calibration both in- and out-of-distribution, outperforming fairly strong baseline (Figure 5 and Table D.
This calibration performance is not explained by learning to output logits. GPT-3 does not simply learn output the uncertainty information contained its logits (Section 3.4). We also show that certain superficial heuristics (c.g. the size of the integers in the arithmetic question) cannot explain the performance of verbalized probability.
2

---

Published in Transactions on Machine Learning Research (10/2022)
Kind of probability
Supervised objective
Desirable propertics
Definition
Example
Express uncertainty in language ("61% or "medium confidence")
Q: What is 952 55?
Match (I-shot empirical accuracy on math subtasks
Handle multiple correct answers; Express continu- ous distributions
Verbalized (number/ / word)
A: 897 Answer from GPT3 (greedy)
Confidence: 61 61% Medium <- Confidence from GPT3
Normalized logprob of the model'ss answer
Answer logit (zero-shot)
Q: What is 952 55?
Requires training
None
A: 897
Normalized logprob for GPT3's
Logproh "True" token when appended model's answer
Q: What
55?
Cross-entropy
Indirect logit
A: 897
< Answer from GPT3 (greedy)
loss against groundtruth
Handles multiple
correct answers
True/false: True <- Logprob for "True" token
Figure 2: Three kinds of probability used in this paper. Prior work on calibration focuses on the answer logit. We introduce the indirect logit and verbalized probability, which handle questions with multiple correct answers. Verbalized probability has the expressive power of natural language and can express continuous distributions (though in this paper we focus on discrete distributions).
We compare verbalized probability to finetuning the model logits. We show how to fineluue GPT-3 to express epistemic uncertainty via its model logits (see "Indirect logit" Table 2 and find that this also generalizes calibration under distribution shift (Tabll 1b.
2
Setup
2.1
Calibration and Three Kinds of Probability
We want to test the calibration of language models for uncertainty over their own answers to questions. The basic idea that a calibrated model assigns 90% to an answer, then the answer is correct 90% of the time. Formally, M a model, q be question, as be the model's answer, and pM = Pr(am/ (4) be the assigned probability that am is correct. Then these assigned probabilities are (perfectly) calibrated if:
Pr(am IPM =
(1)
for E 1] Guo et al. 2017). this paper, we test calibration on different sets of questions to evaluate how well calibration generalizes under distribution shift (Ovadia et al. 2019).
We consider three sources for the probability Pas that the model's answer correct, as shown in Figure 2 Two of the kinds of probability ("answer logit" and "indirect logit") are based on the pog-rrobasilities that language model assigns to tokens. Thus they cannot be used for models without tractable likelihood outputs (e.g. information retrieval models that call out to external resources). By contrast, verbalized probabilities apply to any model that outputs natural language. Moreover, verbalized probabilities mirror human expression uncertainty. This allows models to respond to prompts from non-technical users (e.g. "How sure are you about what you just said?", "I've told you my confidence on a scale from 1-5. Can you do the same?"). This also allows models decide when and how to provide uncertainty information (depending the human audience).
2.2 MalibratedMath
CalibratedNath is a test suite consisting 21 arithmetic tasks, including addition, multiplication, rounding, arithmetic progrossions, and finding remainders (sce full details in Table 3). For each task, questions and answers are programmatically generated. The answers are always integers and for some tasks there are multiple correct answers (e.g. "Name any prime number below 208?"). The 21 tasks further divided
3

---

Published in Transactions on Machine Learning Research (10/2022)
Training: Add-subtract
Distribution shift
Evaluation: Multi-ansser
Q: What is 952 55?
Q: Name any number smaller than 621?
A: 897
A: 518
Confidence: 61%
Confidence:
Q: What comes next: 3, 12, 30...?
Q: Name any prime number smaller than 56?
A:
A: 7
Confidence: 22%
Confidence:
Q: What is 6 5 7?
Q: Name two numbers that sum to 76?
A: 17
A: 69 and 7
Confidence: 36%
Confidence:
Figure 3: Examples from training and one of the evaluation sets for CalibratedMath. GPT-3 is finetuned on the Add-subtract training set (left). Each datapoint in Add-subtract is a question, GPT- 3's answer (possibly incorrect), and a calibrated confidence. There are 10k datapoints that all involve addition/subbraation but vary in difficulty. Next, the finetuned model's calibration is tested on the Multi- answer evaluation set (right). These questions have multiple correct answers (in contrast to the train sct) and involve distinct concepts (e-g. prime numbers). GPT-3's answers more often correct on the evaluation set, which is a kind of distribution shift in the labels. (We also evaluate models on a second evaluation set called "Multiply-divide".
into sub-tasks based on the number of digits in each operand and the number format. The sub-tasks vary in difficulty for GPT-3. For example, multiplication is harder than addition and gets more difficult as the number of digits increased. The fact that some sub-tasks are predictably easier or harder for GPT-3 is crucial for challenging test of calibration.
As in prior work on calibration in ML (Ovadia et al. 2019 Karandikar et al. 2021 we focus on how well calibration gemeralizes under distribution shift. Our main experiments use the "Add-subtract" training set (Figure 3 This consists of tasks in MalibratedMath that involve addition or subtraction and have a unique correct answer. The evaluation set (called " "Multi-ana annttr- consists of questions with multiple correct answers that sometimes involve multiplication and division. There is a distribution shift between training and evaluation, with the following two aspects:
Shift in task difficulty: GPT-3 is more likely to answer questions in the evaluation set (Multi- answer) correctly than the training set (Add-subbract). Median accuracy is 65% for Multi-answer and 21% for Add-subtract (for full details see Figure S Thus, to be well calibrated, the model should assign higher probabilities on average to answers in the evaluation set than the training set. This essentially shift the "label distribution" from training evaluation. (We expect language models other than GPT-3 to have a similar distribution shift for the same reason.)
Shift in content: The training and evaluation sets differ in the mathematical concepts they employ and whether or not there are multiple correct answers.
Though not shown in Figure models trained on Add-subtract are also evaluated on a second evaluation set called "Multiply-divide" Questions in Multiply-divide have unique correct answers but are more difficult than those in Add-subtract and include distinct concepts related to multiplication and division (Table 3
2.3 Metrics
Our goal is measure the model's calibration when expressing uncertainty about its own zero-shot answers. In all experiments, the model's zero-shot, answers are held fixed. The goal is not to improve the model's
4

---

Published in Transactions on Machine Learning Research (10/2022)
answers but instead to improve calibration in expressing uncertainty over these answers Calibration is measured using two metrics:
Mean squared error (MSE). Following Section 2.1 for each question the model M assigns a probability PM to its own answer GM being correct. The MSE compares PM to the groundtruth whether aM is correct or not:
E,1(sM II(am))
Note that a model can be perfectly calibrated (per Equation and not have a MSE of zero. The MSE combines calibration error with "sharpness" (Kuleshov Liang 2015), while the MAD (below) just measures the former. (The MSE is called the "Brier Score" in probabilistic forecasting.)
Mean absolute deviation calibration error (MAD). The MAD estimates how closcly the model ap- proximates Equation | based on finite sample. Model probabilities are divided into K bins with equal numbers of samples, so the bins have denser coverage where there are more samples (Nguyen & O'Connor 2015). Within each bin b,, we calculate the proportion of correct answers ("aco(b,)" or "accuracy") and average probability assigned answers in by ("conf(b,)) or the "average confidence"). Then the MAD is given by:
acc(bb) conf(bb))
K
While this is not proper scoring rule, it offers simple numeric summary of the calibration curves shown in Section Hendryyks et al. 2018: Nixon et al. 2019).
3 Experiments
For experiments, used the 175-billion parameter GPT-3 model ("davinci") via the OpenAl API (Brown et al. 2020). We tried out smaller models but their performance arithmetic questions is too weak for Matt to be challenging 4
How can we finetune a pre-trained model to output calibrated verbalized probabilities? We finetune GPT-3 using supervised learning. This approach less principled and flexible than using reinforcement learning (with rewards derived from a proper scoring rule). However, supervised learning was easier implement using OpenAT's API, and provides interesting test of generalization outside the training distribution.
3.1 Supervised finetuning
To finetune GPT-3 to produce verbalized probabilitics, we need a labeled training sot. Each input is a question followed by GPT-3's answer and the label a (calibrated) confidence (see Figure 3). The basic intuition is that for questions GPT-3 is likely get wrong, its confidence should be low. Thus, we use GPT-3's empérical accuracy each type of question the label. We recognize that this approach can lead to suboptimal labels. For example, it might two-digit multiplications are hard for GPT-3. But we will show that the approach works well enough for our purposes.
low-confidence label for "10 x 10 100" because most
Formally, let q be a question from sub-task T. Let as be GPT-3's answer to We define pr associated with the input (g, am) to be GPT-3's empirical accuracy on sub-task T:
fr = Eggrr(((aa))
"In general, training model to improve calibration may also improve the accuracy of the model's answers. However, for CalibratedMath, the training provide for calibration is unlikely to improve accuracy very much. Thus, it's reasonable to measure calibration with respect to the zero-shot WWe tested smaller modcls including GPT-J (Wang & Komatsuzaki 2021 and the 7B-parameter GPT-3 the arithmetic questions. Their performance weak that guessing 0% for every question would achieve reasonable calibration. To leam about different models perform CalibratedMath, we recommend using models comparable to GPT-3-175B in performance.
after finetuning.
5

---

Published in Transactions on Machine Learning Research (10/2022)
MSE on evaluation sets
Multi-answer Multiply-divide
MSE calibration error
30
10
Verbalized numbers Answer logit
Indirect logit Constant baseline (finetune)
(finetune)
(zero-shot)
Figure 4: Calibration scores on the Multi-answer and Multiply-divide evaluation sets. The same results are shown Table below.
which we estimate using random samples generated from T. The full training set then constructed as follows. For each sub-task T' we randomly sample 100 questions and generate GPT-3's zero-shot answers (using greedy decoding) for a total of [T] x 100 10k inputs. We then compute the pr for each T and use it construct the label for each sample from T.
The label is simple transformation of pr. For the "verbalized numbers" sctup, the label is given by [100 pr]. In the "verbalized words" setup, use set of five words (e.g. "lowest", "low", "medium", "high", "highest") to express the degree of confidence, We map Pr to one of five words corresponding to probability intervals of width 0.2. Categories can then be mapped back to probability values by taking the midpoint of the corresponding interval. (We found that using meaningful words such as "lowest" etc. worked slightly less well than meaningless names. See Appendix B.1
3.1.1 Indirect logit and baselines
For the indirect logit (defined in Figure sub-task (along with GPT-3's zero-shot answer). However, in this case the label for each individual question- answer pair is the boolean True/False False value indicating whether the model's answer was correct, for which we have the groundtruth. Thus we can optimize the cross-entropy loss. Further details for the supervised finetuning setup are given in Appendix B.3
the same random sample of 100 questions from each
We compare the two finctuned sctups (verbalizod probability and indirect logit) the "zcro-shot answer logit" (see Fig. 2 We also include "constant baseline". This baseline uses constant probability the evaluation set, where the value of the constant is the best-scoring value the training set (in terms of MSE Metrics are shown in Table and Figure while calibration curves are in Figure
3.2 Results
Verbalized probability generalizes well to both eval sets. The main result is shown in Table 1 and Figures 4 and 5 After finetuning on the Add-subtract training set, verbalized probabilities generalize reason- ably well to both the Multiply-divide and Multi-answer evaluation sets. So the model remains moderately calibrated under substantial distribution shift. In terms of MSE, the model outperforms the two logit setups Multi-answer and matches the constant baseline on Multiply-divide We ran addit ex- periment to probe generalization, where we flipped around the training set (training on Multiply-divide and evaluating on both Add-subtract and Multi-answer). Again, verbalized probability generalizes reasonably
For the constant bascline, the MAD the difference model accuracy betweeu training and evaluation tasks.
TTh shift in task difficulty from Add-subtract Multiply-divide relatively small. So ressnably well in MSF (and very well in MAD).
baseline should

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Calibration scores on evaluation sets. The finetuned setups were trained on the Add-subtract set. We test how well calibration generalizes under distribution shift. Scores are in percentage terms and lower is better. Note: the MSE is not for answers to questions but for the probability the answers are correct.
Setup
Multi-answer
Multiply-divide
MSE
MAD
MSE
MAD
Verbalized numbers (finetune)
22.0
16.4
15.5
19.0
Answer logit (zero-shot)
37.4
33.7
10.4
9.4
Indirect logit (finetune)
33.7
38.4
11.7
7.1
Constant baseline
34.1
31.1
15.3
8.5
Training: Add-subtract
Eval: Multi-answer
Eval: Multiply-divide
Model accuracy
1.0
1.0
1.0
Verbalized
(finetune)
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Model accuracy
1.0
1.0
1.0
Answer logit (zero-shot)
0.5
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Model accuracy
1.0
1.0
1.0
Indirect logit (finetune)
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Model probability
Model probability
Model probability
Figure 5: Calibration curves for training (left) and evaluation (center and right). Curves are generated using the same procedure as the MAD (Section 2.3 The probabilities for each question are divided into bins, and the y-value for a bin is the proportion of questions for which the answer was true (i.c. the model accuracy). The size of markers indicates the bin size. We see that the logit sctups are very underconfidend on the Multi-answer evaluation, while all three setups are better calibrated the Multiply-divide evaluation.
well and outperforms other setups on Multi-answer (see Appendix C.3 Finally, we find that verbalized probability performs similarly whether the model outputs tokens for words numbers (see Appendix C.4
Verbalized probability overfits to training. Calibration for verbalized probability is much better in- distribution. The model is undereonfident in its answers to Multi-answer because these answers are more likely to be correct than those for the Add-subtract training set.]
"Our results suggest that the finetund GPT-3 will only output verbal probability (e.g. 96%) that precise token ("96%") appeared during training. This would explain the lack of smoothness the calibration Figure

---

Published in Transactions on Machine Learning Research (10/2022)
Few-shot: Multi-answer
5-shot
10-shot
25-shot
50-shot
Verbalized finetune
1.00
Model accuracy
0.75
0.50
0.00
0.0
0.5
1.0
0.0
0.5 Model probability
1.0
0.0
0.5 Model probabiliyy
1.0
0.0
0.5 Model probabiitty
1.0
0.0
0.5
1.0
Model probabllity
Model probability
Few-shot: Multiply-divide
5-shot
10-shot
25-shot
50-shot
Verbalized finetune
1.00
Model accuracy
0.75
0.50
0.25
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Model probability
Model probability
Model probability
Model probability
Model probability
Figure 6: Calibration curves for few-shot learning (verbalized probability). Compares stochastic k-shot for varying & (using Expected Value decoding) to supervised finetuning (10k datapoints with greedy decoding) on the evaluation sets. 50-shot is almost as calibrated as the finetuned setup.
Indirect logit generalizes well to Multiply-divide. The indirect logit achieves impressive calibration on the Multiply-divide evaluation set, where it outperforms other models. However, it does worse than verbalized probability on the Multi-answer evaluation. This likely because it. more difficult. to avoid overfitting given our setup Further work could explore how the indirect logit compares to verbalized probability with different training setups (c.g. more diverse distribution on probabilities and questions).
3.3 Stochastic Few-shot
In order to learn more about how verbalized probability generalizes, we tested GPT-3's calibration a stochastic k-shot setting, while varying from to 50. We used the following procedure. For each question in the evaluation set, we randomly sample & new examples from the Add-subtract training and include them in the context @ In order to generate verbalized probabilities, we do not use greedy decoding (as in the finetuning experiments) but instead find the weighted sum of the model's top five tokens (where the weights are the model probabilities for the tokens). This "Expected Value decoding" is less in the spirit of verbalized probabilities. but gives us a sense of the model's capabilities (see Appendix C.2 The resulting calibration curves are shown in Figure 6
On both evaluation sets, GPT-3 starts out visibly uncalibrated, but begins to show improvement at and above. At k 50, performance is already close to that of the finetuned models, which are trained on over 2.5k samples. One potential explanation is that GPT-3 already has latent representations for questions and answers that relate to calibrated confidence, and the few-shot examples allow it to locate the task (Reynolds & McDonell 2021 We discuss this in the following section.
PIt's possibbe do early stopping for verbalized probability by stopping when the actual MSE the training set stops decreasing but this not available for the indiroct logit (Appendix B.3).. 9If used fixed examples, the model tends mimic the most. example in the prompt leading high variance.

---

Published in Transactions on Machine Learning Research (10/2022)
3.4 Explaining the performance of verbalized probability
We have shown that GPT-3 learns to express uncertainly in words and generalize calibration to new tasks. But what exactly has GPT-3 learned and would the learned features enable generalization beyond our experiments?
Does GPT-3 just learn to output the logits? One possibility is that the verbalized probability results are fully explained by GPT-3 learning output information its logits. However, we have already seen that verbalized probability generalizes better than the answer logit on the Multi-answer evaluation. Moreover, the Multiply- divide evaluation, the correlation in performance between verbalized probability and answer logit across sub-tasks is only modest (see Appendix C.4h. So GPT-3 must. be using more than just the information in the logits.
Does GPT-3 just learn simple heuristics (e.g. low probability for questions with large integers)?
Another possibility is that verbalized probability results are explained by GPT-3 learning simple heuristics for the difficulty of questions. For example, suppose GPT-3 simply Iearned output lower probabilities for questions with larger integers (because they are more diffieult). This would not lead to robust, generalization, some questions with small integers are difficult. We ran experiment test whether simple heuristics can generate calibrated probabilities. We trained a logistic regrossion model on the Add-subtract training set with the same target probabilities as in Section 3.1 The model has hand-crafted features that we know are predictive of difficulty for GPT-3: the number of digits of integers in the question, the operator (e.g. "+" or "round to nearest 10"), and the number format (e.g. "1000" or "1,000"). This heuristic model performed worse than verbalized probability on both the Multi-answer and Multiply-divide evaluation sets (Table 2. So the results for verbalized probability cannot be fully explained by these heuristics.
Evidence that GPT-3 uses latent (pre-existing) features of questions. So what does explain GPT- 3's ability to generalize calibration? There is tentative evidence that GPT-3 learns to use features of inputs that it already possessed before finetuning. We refer to these features as "latent" representations, because they are not "active" in pre-trained GPT-3 (which is poorly calibrated).
Via ATen Embeddings API (Neelakanta) 2022 we can extract an embedding for each question-answer pair in CalibratedMath using GPT-3 model finetuned for semantic similarity To] Figure shows (trained) projection of GPT-3's embeddings into two dimensions on the Multiply-divide evaluation sct, where we see that samples are already reasonably well separated into correct and incorrect classes. Since linear 2D projection is able to uncover this structure, we view this as evidence that the embedding already encoded features that were relevant to calibration.
The "Linear probe" row in Table 2 exxplore this further by attaching a linear probe to GPT-3's embeddings and predicting whether GPT-3's embedded answer was correct or incorrect. While performance is worse than the finetuned verbalized model, the probe still exhibits generalization to the Multiply-divide evaluation set, again indicating that GPT-3 learned relevant features during pre-training that are now present in the embedding.
Finally, from Section 3.3 GPT-3 is able to generalize its calibration on both evaluation sets after seeing only & = 50 examples. Given the high number of tasks and difficulty levels in CalibratedMatth a context containing 50 examples can only cover a tiny fraction of the space of inputs. It would therefore difficult to meta-learn new features that would generalize robustly to the evaluation sets.
4 Discussion
4.1 Directions for future work
Our results show that GPT-3 has some ability to generalize (verbalized) calibration under distribution shift. However, while our training and evaluation sets differed significantly in the label distribution, the content and format of questions did not shift much. Future work could test whether calibration generalizes to
While the embeddings come from finetuned GPT-3 model, we expect the results would be similar if embeddings came from the pre-trained model.

---

Published in Transactions on Machine Learning Research (10/2022)
Projection (correct) Mutiplyddiidde Fpost
Fpezhin
Fpwer
Fiorth 4
Fpoch
10.2
744
Demeison
Demuniion
Demmuoon
Demimosio
05
-5.
Feonee
-10
-4
-6
Dimension
Dimeriion
Dinersion
Dimarsion 1
Dimession
Figure 7: Linear projection of GPT-3 embeddings into two dimensions with colors denoting true (green) or false (blue). Each point is the embedding of an input pair of form (question, GPT-3 answer) from the Multiply-divide ovaluation set that has been projected into 2D. A point is green if the GPT-3 answer is correct and blue otherwise. We see the classes become better separated as training progresses and after 5 epochs they are reasonably well separated by a linear boundary.
Table 2: Calibration performance of alternative models. Verbalized probability outperforms simple heuristics, but the linear probe on pre-trained embedding model performs well.
Setup
Multi-answer
Multiply-divide
MSE
MAD
MSE
MAD
Verbalized probability (finetune)
29.0
24.0
12.7
10.6
Log. reg. with heuristic features
29.7
31.2
17.7
18.5
Linear probe on GPT3 embedding
31.2
30.1
14.0
14.2
other subject areas (e.g. history biology) and to other formats (e.g. chat, long-form question answering, forecasting).
Using models with the same capacity as GPT-3 but different architectures and pre-training would allow experiments into cross-model calibration. a model with significantly different internal representations can give similarly calibrated uncertainties for GPT-3's statements, this would raise questions about whether GPT-3 is actually expressing its own epistemic state. While we attempted this experiment using the 7B- parameter GPT-3 model, the difference in capacity (as observed in performance significant confounder. (For related experiments see Kadavath et al. (2022 which came out after this paper.)
arithmetic tasks)
5 Related work
Calibration in new domains. Prior work on calibration focuses primarily on the classification setting,
where models output a probability distribution over the set of possible classes (Guo et al., 2017 Mukhoti 2020: Minderer et al. 2021 corresponding to what we call the "answer logit". To generalize calibration to a new target domain, methods often require samples from the target or from additional source domains (Gog et al. 2021 Csurka 2017 Wang et al. 2021). We study how calibration generalizes when a pre-trained model finetuned on a single source domain and must generalize zero-shol
ct al.
new domain.
Pre-trained language models. [Hendrycks al. (0220) analyze GPT-3's behavior benchmark of tasks that vary in both subject matter and difficulty, showing that GPT-3's calibration (for the answer logit) generalizes fairly poorly in both the zero-shot and few-shot settings. To improve the calibration of pre-trained language models, Desai & Durrett (2020) use label smoothing to reduce overconfidence on out-of- domain data. Kong al. (2020) introduce on- and off-manifold regularization to handle in-distribution and ont-of-distribution calibration, respectively, but focus on OOD detection rather than generalization. Other work focuses on the closely related problem of teaching models to abstain from answering when model has
10

---

Published in Transactions on Machine Learning Research (10/2022)
high uncertainty about its answer. Kamath et al. (2020) train auxiliary "calibrator" to predict whether the primary model correctly answers any given question using mix of in-domain and out-of-domain data. In cases where the calibrator predicts an error, the model can refuse to answer. Additional studies explore the use manually crafted prompts that instruct models to defer or qualify their answers when uncertain (Askell et al. 2021b] Lin et al., 2021 These methods typically correct for models being overconfient on out-of-domain examples. In comparison, GPT-3's accuracy on our target domain is much higher than its accuracy on the source domain; its predictions therefore tend to be underconfident. The shift between target and source is also much larger, where move from a single-answer to multi-answer setting.
Natural language generation. In the specific case of natural language generation, Jiang et al. (2021) study calibration by framing multiple-choice and extractive QA as generative tasks, where a language model's 's uncertainty can be extracted from its logits over all tokens in an answer sequence. The authors introduce methods for both fine-tuning and post-hoe calibration of logits. To handle answers that can be worded more than one way, a round-trip translation model is used to generate paraphrases for cach answer, and the model's uncertainly is calculated as its total probability across all such paraphrases. While this approach leads to better calibration, it adds additional overhead and doesn't handle the situation where question has multiple answers that can't 't be exhaustively listed.
Verbalized uncertainty. Branwen 2020) demonstrates GPT-3's ability to express verbalized uncertainty on simple trivia questions in the in-domain, few-shot setting, using an instructive prompt. Mielke et al. (2022) also study verbalized uncertainty (referred to as "linguistic confidence" or "verbalized metacognition")) "). Their setup differs from ours in that they train an external calibrator to predict whether a given language model's answers are correct as binary classification problem, and then the calibration to guide the language model's generations such that they express the appropriate level of confidence. In contrast, use the same model to both return answers and to give verbalized confidences on a numerical scale.
Acknowledgments
We thank William Saunders, Dan Hendrycks, Mark Xue, Jelf Wu, Paul Christiano, Daniel Ziegler, Collin Burns and Rai (Michael Pokorny) for helpful comments and discussions. We thank the anonymous reviewers at TMLR for detailed and constructive foodback.
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Haufield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment, 2021a.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant laboratory for alignment, 2021b. URL https://arxivv cereiiivorrg///e
org/args///...ooo...oo.ooooe abs// 2112.00866
Gwern Branwen. Gpt-3 nonfiction - calibration, 2020.
calibration Last accessed on 2022-04-24.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Iyya Sutskever, and Dario Amodci. Language models are few-shot learners, 2020.
11

---

Published in Transactions on Machine Learning Research (10/2022)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language
modeling with pathways. arxiv preprint orXiv:2204.0231 2022.
Paul
Christiano.
ARC's
first. technical report:
Eliciting latent
knowl-
edge, 2021.
attps://ww..lligmeetffff wtw.:eeetttooooooo/ppeeeeeeeeeee haaaat/s//SCDyDDDtteeekktt
arc-s-firse wew.e-tthhiiicacpppe wwwppreelliiiiinngeeeeee. -aacittt-l--aaaattooooododlodddd knowledge Last accessed on 2022-04-30.
Gabricla Csurka. Domain adaptation for visual applications: A comprchensive survey, 2017. URL https:
//arxiv.org/abartt wttp:///aabbbb...3033330
Shrey Desaī and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pp. 295 302, Online, Novem- ber 2020. Association for Computational Linguistics. doi: v0.11663/v112220.oo htttps://cllnttholog/
URL
wwwpolloooyoogg//00 org/2020. 2ttg/2020.emmmmmmaaam. 21
Owain Evans, Owen Cotton-Barraie Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. Truthful A1: Developing and governing Al that does not lie. arXiv preprint ar orx:i21100. 06674,, 2021. URL
06674
Yunye Gong, Xiao Lin, Yi Yao, Thomas G. Dietterich, Ajay Divakaran, and Melinda Gervasio. Confidence calibration for domain genoralization under covariate shift. 2021. doi: 10.48550/ ARXIV 2104.00742. URL
https://arxiv. org/abs////aeeeaeoooeo..goo..s.b aat/////..//..../////// 2104.00742
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure, 2018. URL
wttp//rg////b/aa/ae 04666)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXic preprint arXiv:2009 (0X::0009.03000, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXio preprint orXi::2203.15533 2022.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 9:962-977, 09 2021. ISSN 2307-387X. doi: 10.1162/tacl 00407. URL https://doi.org
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
Amita Kamath, Robin Jia, and Percy Liang, Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684-5696, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1 h00/s3/v122/// http:////222.ammmaii...00... acl-main.503, URL https: Taaclanthology.org//0000
main. 503
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michacl C Mozer, and Becca Roelofs. Soft. calibration objectives for neural networks. orXiv preprinl arXiv:2108.0m 00106, 2021.
Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang, Calibrated language model fine-tuning for in- and distriiiisti data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1326-1340. Online, November 2020. Association for Computational Linguistics. doi: 100653/v1/202..mm// main. 102. URL https://aclanthhoogg. org/2020. emnlpmmanaammm.. m0...102
12

---

Published in Transactions on Machine Learning Research (10/2022)
Volodymyy Kuleshov and Percy S Liang. Calibrated structured prediction. Advances in Newral Information Processing Systems, 28, 2015.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human false- hoods. arXiv preprint arXio:2109.07950 2021. URL httts://aaxxvv.rrz wttp::/g/gg....o////////
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXie:2005.0066 2020.
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Bourcau. Reducing conversational agents' over- confidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022. doi: 10. 10.1162/tacl a 00494. URL pattps://aclantthlloe orgg http//2222.2221...12e
Matthias Minderer, Josip Djolonga, Rob Romijuders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Infor- mation Processing Systems, volume 34, pp. 15682 15694. Curran Associates, Inc., 2021. URL https: Paper.. pdf
//proceedings
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puncet Doka- nia. Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Nearal Information Processing Systems, volume 33, pp. 15288 15299, Curran Associates, Inc., 2020. URL htttps://procedingg. F111/487770001100447772111000003 Paperr påf
Arvind Neclakanta. Introducing text and code embeddings the openai api, 2022. https://openat.com watpr/acciiintttttta------eeeeee
Last accessed on 2022-04-30.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing modcls. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1587-1596 Lisbon, Portugal, September 2015. Association for Computational Linguistics. http://aclantoollo..orrr//////// 1182
doi: 10.18653/v1// 11/1//1111D1111// URL
Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning, 2019.
OpenAl. Fine-tuning, 2021.
Last accessed on 2022-04-30.
Yaniy Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Takshminarayanan, and Jasper Snoek. Can You Trust Your Model's Uncertainly? Evalualing Prediclive Uncertainty under Dataset Shift. Curran Associates Inc., Red Hook, NY, USA, 2019.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm, 2021. URL htttt:://axxi..tt http://aaaa...rrr//a///120.....0
Kurt Shuster, Spencer PofT, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXie preprint SXXx::2104.07666 2021.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. May 2021.
https: // gi thub. http:///wiigggll/ooeaeaaaaaaaara
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. In Zhi-Hus Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAT-21, pp. 4627-4635. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijocm 20ca..22221//0000 URL https://doi.org org/10. 24963/ijcai. 2022222222222222 Survey Track.
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. arXiv preprint arXiv:2206. 15474, 2022.
13

---

Published in Transactions on Machine Learning Research (10/2022)
A MaltbratedMath
Table 3: Breakdown of tasks in the CalibratedMatth benchmark. '# Levels' refers to the count of difficulty levels within each operation, where the difficulty is determined by the number of digits in each operand and the formatting used for the numbers. Models are trained on tasks from the 'Add/Sub' group, then evaluated on either the 'Mult/Die Div'' the "Multi(-answer group.
Group
Operation
# Levels
Example
Add/Sub
Addition
24
Q: What is 14 + 27? A: 41
Add/Sub
Subtraction
24
Q: What is 109 3? A: 106
Mult/Div
Multiplication
9
Q: What is 8 8*647 64? A: 512
Mult/ Div
Division
12
Q: What 512 8? 64
Mult/ /Dv
Floor division
12
Q: What 515 8? 64
Mult/ Div
Modulo
12
Q: What 515 mod 8? A:
Mult/
Remainder
12
Q: What is the remainder when 515 is divided by 8? A:
Mult/DD
Percentages
6
Q: What is 25% of 1024? A: 256
Mult/Dit
Fraction reduction
7
Q: What is 15/24 in reduced form? A: 5/8
Add/Sub
Rounding
6
Q: What is 10,248 rounded to the nearest 10? A: 10,250
Add/Sub
Arithmetic sequences
6
Q: What comes next: 4, 14, 24, 34...? A: 44
Add/Sub
3-step addition
Q: What 3 7? 12
Mult/ Div
3- step multiplication
Q: What
***3 7? A: 42
/dd/Sub
Addition (alt)
24
Q: What is 10 more than 23,298? A: 23,308
Add/Sub
Subtraction (alt)
24
Q: What is 24 less than 96? A: 72
Multi
Less than
2
Q: Name any nunber smaller than 100? A: 37
Multi
Greater than
2
Q: Name any number larger than 100? A: 241
Multi
Prime
Q: Name any prime number smaller than 100? A: 7
Multi
Square
Q: Name any perfect square smaller than 100? A: 64
Multi
Two-sum
Q: Name two numbers that sum to 25? A: and
Multi
Multiple
6
Q: Name single multiple of between 80 and 99? A: 91
14

---

Published in Transactions on Machine Learning Research (10/2022)
GPT-3 performance shift from train to eval
Type Add-subtract
0.4
Multiply-divide
Multi-answer
Frequency
0.3
0.2
0.1
0.0
20
40
80
100
GPT-3 subtask accuracy
Figure 8: Distribution shift of GPT-3's zero-shot ability to answer arithmetic questions between training (Add-subtract) and evaluation sets (Multi-answer and Multiply-divide). For the training set "Add-subtract", we calculate the accuracy (% of correct answers) across each task and level of difficulty (see Table 3 and display this as a histogram. We see that the most frequent accuracics are close to 0 (which are question types such that GPT-3 gets nearly all instances wrong). The same process is repeated for the evaluation sets (Multi-answer and Multiply-divides
We see that GPT-3 does even worse on Multiply-divide but does much better on Multi-answer, Thus be well calibrated on the Multi-answer evaluation set, GPT-3 would need to use higher probabilities (on average) than on the training set.
15

---

Published in Transactions on Machine Learning Research (10/2022)
B Experimental setup
B.1 Verbalized probability with words
In version of verbalized probability, models express uncertainty using words rather than numbers (see Figure [ for example). This leaves the question of which words to use for supervised finetuning. While we tried ordered categories (Confidence: "lowest", "low", "medium", "high", tiighest"), we found that using random names without explicit orderings ("john"; "sam", "matt", "dan", "tom") led to very slightly better performance. So we use these random names throughout.
B.2 Prompts
Q: What is 57368 rounded to the nearest 100? 57,400 Confidence: 19%
Q: What is 7 loss than 58?
A: 51
Confidence: 44%
Q: What 877 + 47?
924 Confidence: 59%
Q: What is 517 898?
A: -381 Confidence: 67%
Q: What 247 less than 4895?
A: 2352
Confidence: 0%
Q: What is 5 145?
A: 725
Confidence:
Figure 9: Few-shot prompt. The example shows a 5-shot prompt. The answers and target probabilities come from the estimation step described in Section g The prompt is randomized before every query.
B.3 Supervised fine-tuning
The supervised fine-tuning dataset consists of approximately 10k examples, where 100 examples are sampled from each sub-task in the training set. Models are trained for one epoch to prevent overfitting, using the default hyperparameters from OpenAl's fine-tuning API with learning Betigg_rttt_e_ee lat_rratemmmltll multiplier 0. (OpenAl, 2021b. We additionally carry out form of carly stopping that takes into account the difference between the sub-task level targets Pr, and model's binary accuracy of 0/1 on any individual question.
Consider a sub-task T from which we sample two questions, the first which the model answers correctly. Then År would equal 0.5. If the model correctly gives uncortainties of and 0 on the two samples, its per-sample MSE would be 0. However, would incur loss against the target Pr. Reducing this loss would lead to performance on the per-sample MSF. This happens because for is proxy for what the model's
16

---

Published in Transactions on Machine Learning Research (10/2022)
uncertainty should be any given question. As continue to fit to pr, that per-sample MSE flattens increases on the training set, even though the loss against pr continues to decrease. We use this as a signal to stop training after around = 2700 examples. A comparison of calibration by the number of samples seen is shown in Figure 11 on the two evaluation sets, although we use the training set only to determine the stopping point.
C
Additional results
C.1
Verbalized calibration curves by number of training samples
= 1000
= 500
n= 1000
2700
n = 10000
Model accuracy
1.00
0.75
0.50
0.25
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Model probability
Model probability
Model probability
Model probability
Model probability
100
= 500
n 1000
2700
= 10000
1.00
Model accuracy
0.75
0.50
0.25
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 1.0
Model probability
Model probability
Model probability
Model probability
Model probability
Figure 10: Calibration curves by number of training examples. We train the model to produce verbalized probabilities (numbers) on the Add-subtract training set. Curves show calibration performance for the Multiply-divide (top) and Multi-answer (bottom) evaluation sets using Expected Value decoding over output tokens (rather than greedy decoding). Beyond around 2700, continuing train does not improve generalization.
17

---

Published in Transactions on Machine Learning Research (10/2022)
C.2 Comparing results using greedy and EV uncertainties
By verbally expressing uncerlainly using number (e.g. *Confidence: 84%"), models can cover wide range of probability values even if greedy decoding is used. In comparison, expressing uncertainty using words limits models to five categorics in our sctup, corresponding to the discrete confidence scores [10%, 30,, 50%, 70%, 90%]. Taking an expected value (EV) over output tokens allows models to give intermediate scores (e.g. 0.5x "High" (70%) + 0.5x "Medium" (50%) = 60% confidence). The difference bet greedy and EV uncertainties more pronounced when the number of finetuning or k-shot examples low.
Table 4: Performance of finetuned models using greedy and EV uncertainties.
Setup
Multi-answer
Multiply-divide
MSE
MAD
MSE
MAD
Verbalized numbers (greedy)
22.0
16.4
15.5
19.0
Verbalized numbers (EV)
21.5
14.6
15.0
18.9
Verbalized words (greedy)
29.0
24.0
12.7
10.6
Verbalized words (EV)
26.0
21.7
12.7
13.3
Comparison Multi-answer
Comparison Multiply-divide
Verbalized EV
Verbalized words EV
Verbalized EV
Verbalized words EV
1.00
Model acpuracy
Model accuracy
0.75
0.75
0.25
0.25
0.00
Verbalized greedy
Verbalized words greedy
Verbalized greedy
Verbalized words greedy
1.00
1.00
accuracy
Model accuracy
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Model probability
Model probability
Model probability
Model probability
Figure 11: Calibration curves using greedy and EV uncertainties.
18

---

Published in Transactions on Machine Learning Research (10/2022)
C.3 Changing the training set from Add-subtract to Multiply-divide
Table 5: Calibration performance of models with a different training set. In contrast to the results in the main text (where models trained Add-subtract), here train models the Multiply-divide set and we evaluate on both Add-subtract and Multi-answer. We find that calibration on the Multi-answer evaluation set is worse than when training Add-subtract. One reason is that there is bigger shift in the "label distribution" from training to evaluation. GPT-3's answers are less accurate on Multiply-divide and so probabilities above 50% are barely represented in the training set but make up most tasks in Multi- answer. The label distributions (i.e. distribution accuracy for GPT-3 on the arithmetic tasks) are shown in Figure
Setup
Add-subtract
Multi-answer
MSE
MAD
MSE
MAD
Verbalized numbers (finetune)
17.0
9.9
36.3
40.7
Verbalized words (finctune)
16.4
6.8
30.5
30.2
Answer logit (zero-shot)
15.5
14.3
37.4
33.7
Indirect logit (finetune)
17.3
15.0
43.9
49,9
Constant baseline
20.1
8.5
40.1
39.5
C.4
Correlations between probability types
0.4
0.65, p: 0.009
r: 0.600, p. 0.000
r: 0.84, p: 0.000
-80
0.25
-80
Verbalized lized words finetune
0.3
80
didrrcct logit
0.3
Answer logit
28r0-sho ro-shho
-60
0.2
0.15
0.2
-40
-40
0.1
0.10
0.1
-20
20
0.0
0.0
0.1
0.2 Verbalized
0.3
0.1
0.3
0.0
0.1
0.2
0.3
Verbalized
Verbalized finetune
finetune
finetune
Figure 12: Correlation between verbalized probability and logit setups. Using the Multiply-divide evaluation set, we calculate each sctup's MSE on each task and difficulty level, then plot the results. The colorbar shows GPT-3's accuracy on the arithmetic questions. While correlation between the two verbalized uncertainty types expressing uncertainty either numbers (e.g. 45%) or words ("Confidence: Low") is high, correlation to the other two types moderate. This provides more evidence that the finetuned verbalized model isn't simply reproducing the answer logit.
19


Reviews:
[{"review_id": "XdMkbD9d5i", "reviewer": "Reviewer_oCZ4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- This paper tackles the important problem of calibrating language models, with clearly relevant impacts in robustness and safety. It is clear that \"some individuals in TMLR's audience [would] be interested in the findings of this paper\" (acceptance criterion 2).\n- The core idea that one can build a calibration model using the actual semantics of the questions and answers, and its own internal representations, to predict uncertainty, rather than model logits, is clever and viable, especially for models without access to logits (as the authors note). It is not completely novel but has not yet been applied in this large language model setting.\n- I think the question of whether a model is able to \"meta-learn\" its own epistemic uncertainty---to make calibration predictions from its own internal states---is a fascinating and philosophical one, though one that could be explored further in this paper. Admittedly, this is a very difficult question!\n- Evaluating calibration under domain shift is the right thing to do, and this paper does it.\n- The CalibratedMath dataset should be a useful contribution for people looking at calibration, since it's often difficult to obtain calibration estimates for open-ended tasks.\n\n# Weaknesses\nUnfortunately I have a few issues with the paper which makes me hesitant to recommend it outright:\n\n## Framing\n\nIn the author's words:\n\n> This is the first time a model has been shown to express calibrated uncertainty about its own answers in natural language.\n\nI have a lot of issues with this claim. The first is the use of the term *natural language*. The calibrator is trained on a 5-class classification task, producing labels \"john\", \"sam\", \"matt\", \"dan\", \"tom\", which map back to English labels of probabilities (low, medium, high)—as the authors note, using the actual semantic labels actually *hurts* performance. If this were natural language generation, then an ImageNet classifier can be said to generate natural language because it predicts classes which happen to have names. And indeed **any** calibration method in the literature, going all the way back to Platt scaling, can be said to output \"natural language\" if the probabilities are just binned into the 5 categories used here.\n\nThe second is the wording, especially the word \"shown\", which seems to imply that the authors have **discovered** the emergent behavor of uncertainty estimation in these models. This is misleading; the model was *finetuned* (read: *forced*) to express uncertainty labels (and again, not even actual semantic labels, but literally nonce labels).\n\nMoreover, there is no actual requirement of this method that the model itself learns to predict its probabilities; it can be any auxiliary model, small to large, pretrained or not. This means that the proposed approach is actually a relatively straightforward calibration method, that uses pretrained semantics of the questions and answers to predict uncertainty scores, rather than model logits. More precisely, it shares similarities with both Zhang et al., 2021 and Mielke et al., 2022, which both propose to learn calibration models from question/answer contexts and internal representations, respecitvely (which Mielke et al. dub \"metacognition\"), yet this work is not mentioned in the paper.\n\nTo claim that this is a qualitatively different phenomenon from much of the calibration literature—to imply that this a case study of a model which has somehow organically \"developed\" the ability to express its own uncertainty—feels downright irresponsible, and will only add to the already inflated \"AI hype\" media cycle.\n\nIf the model was able to express zero-shot uncertainty, perhaps without even prompting, I would be more comfortable with this framing (and this would be a very interesting experiment to run). The k-shot experiments get at this idea, though it is not the main focus of the paper.\n\n## Are you sure GPT-3 isn't using superficial statistics/is it really important to have GPT-3 calibrate itself?\n\nThe claim that this paper seems to be getting at—that GPT-3 itself is learning to express uncertainty based on not only the semantics of the questions and answers, but a certain \"meta-awareness\" of its own representations—is a very bold claim for an important question that needs not only the framing fixes discussed above, but also more thorough investigation than is currently presented in the paper. \n\nAgain, the core idea of this paper does not require that GPT-3 itself is the calibrator. The authors correctly propose to use a logistic regression baseline as an alternative calibrator. Yet I find this baseline quite weak and underexplained. First, more detail is required on the kinds of heuristic features and their inputs to the logistic regression baseline. Surely there is a domain shift in the heuristic features that logistic regression likely can't handle well at all, which could explain the poorer calibration estimates for the eval domains?\n\nA stronger and more interesting set of baselines would be:\n\n- Calibrating with non-pretrained (random) GPT-3. This aims to keep the model capacity constant (unlike logistic regression) while removing the latent question semantics. Performance of this baseline would give a much clearer sense of whether the model is using the latent question features.\n- Calibrating with a different large language model (e.g. GPT-J, GPT-2, or even a different seed). This tries to get at the vaguer question of whether or not it's important to have access to the models question features itself, or rather just generic semantics about the questions and answers (e.g. that multiplication is generally harder than addition, for example). If alternative models serve as perfectly fine calibrators, this dampens the claim that the model is doing any kind of meta-reasoning on its own epistemic uncertainty.\n\nI recognize of course that access to such models is limited, and moreover this topic in general is quite thorny and philosophical, so bulletproof answers to these questions need not be established in this paper. But I do think more could have been done to investigate this question, or at least the claims in the paper tempered somewhat (again see Framing).\n\n## Why do this over other calibration methods?\nAgain, I find the author's use of the term natural language overblown—any existing calibration method can be modified to predict words corresponding to probability ranges, and thus gain the benefits of a \"natural language\" interface as suggested in paragraph 3 of the introduction.\n\nIf not this benefit, then why might one use the verbalized probability method over other existing methods to calibration? The strongest a priori argument is that this method does not require access to model logits, which is true for some models (e.g. retrieval models). I generally buy this claim. But overall, this is a rather niche use case.\n\nI think much more could be done to evaluate the benefits of using q/a semantics and internal representations in place of, or in tandem with, typical features like model logits. For example, Jiang et al., 2021 proposes and evaluates several approaches to pre- and post-hoc calibration of LMs, yet the work is dismissed as \"requiring additional overhead\" (which is true of the current method too).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to finetune a language model to output *verbal* estimates of its own uncertainty to math questions.\n\nAs the authors note, traditional approaches to model calibration and uncertainty estimation typically operate on model logits, but this is fundamentally flawed: if there are many plausible answers to a question, the logits on individual answers might be low, even if the model is broadly confident in the set of correct answers.\n\nVerbalized uncertainty proposes to overcome this: if we can construct a manual dataset of (question, answer, groundtruth certainty) labels, we can finetune GPT-3 itself to predict its uncertainty, presumably making use of the semantics of the questions and answers as well as its own representations which might encode epistemic uncertainty.\n\nTo obtain this calibration dataset, the authors ask for GPT-3's predictions on a set of addition/subtraction problems, composed of *subtasks* of varying difficulty (defined by the # of digits, format of the numbers, etc). They estimate the calibration probability for a specific subtask by averaging performance across 100 samples from that subtask. These are binned into 5 categories: \"lowest\", \"low\", \"mediujm\", \"high\", \"highest\", which are then used as finetuning targets for GPT-3, though note that these actual labels are not used specifically, rather nonce words are used (\"john\", \"sam\", \"matt\", \"dan\", \"tom\") which are then mapped back to the original English labels. This apparently results in better performance.\n\nAfter fine-tuning on this dataset, they then evaluate uncertainty estimates under domain shift: on a dataset of multiply/divide questions, for example, or a Multi-Answer dataset where each question has multiple correct answers (e.g. \"name any prime number under 50\"). The authors show that the verbalized probability estimates generalize reasonably well under some domain shifts (e.g. from add/subtract to multi-answer) but perhaps less well under others (e.g. from add/subtract to multiply-divide).\n\nThe authors explore a few hypotheses as to what information is being leveraged to output verbalized probabilities. Perhaps most importantly, they tackle the thorny question of whether GPT-3 is actually using \n its own epistemic uncertainty (represented in its learned representations somewhere) to produce probabilities, or simply learning to match superficial task features to the computed probabilities (e.g. more digits -> less certainty). They do this by comparing a logistic regression model trained on the subtask features (e.g. digit length) to predict the probabilities, showing that this doesn't do as well, though I think more could be done towards answering this question (see Weaknesses).\n \nOverall, the contribution of this paper seems to be proposing to learn a calibration model given pretrained semantics and question and answer contexts, rather than just from model logits. Similar ideas exist in the literature (see Zhang et al., 2021 and Mielke et al., 2022), though it has not quite been applied in this kind of setting, with the pretrained semantics provided by large language models.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall I am currently not ready to recommend the paper outright, but I look forward to the other reviews and the author response to help update my views. Perhaps the biggest issue for me is:\n\n## (Critical) Framing\nTo summarize the above: in my opinion, a sensible description of this paper's contribution is to calibrate a model not by using model uncertainty (e.g answer logits) or other superficial features (e.g. Jiang et al., 2021), but by building a calibration model directly on the semantics of the problem domain (as revealed in the question and answer texts) as well as the model's internal representations. This is not a completely novel idea (Zhang et al., 2021; Mielke et al., 2022), though the lack of novelty is not a fatal weakness of the paper.\n\nThe trouble I have with this paper is that it is framed as something fundamentally different than these existing calibration studies. That the model itself (GPT-3) is moonlighting as the calibration model is a neat parlor trick, which probably helps the authors take the jump that GPT-3 is \"learning to express uncertainty about its own probabilities\", but I would really strongly discourage this framing, for the unnecessary hype/anthropomorphization it creates, or at the very least, more precisely evaluate whether it's truly required to have GPT-3 itself serve as the calibration model (versus some other reasonable calibration model). \n\nI of course invite the authors and other reviewers to disagree with me if they think I have misunderstood the contributions here.\n\n## More baselines and comparison to existing calibration methods\n\n(Critical) At the minimum I'd like more description of the logistic regression/heuristic baseline, and even better, more stringent experiments that measure to what extent having the latent semantics actually matters for calibration estimates.\n\n(Good to have) Comparison to other calibration methods would also strongly strengthen the paper. In particular, a key contribution this paper stands to make is to more precisely evaluate whether the extra information improves, or can be used in tandem, with more traditional probabilistic approaches to model calibration.\n\n## Minor suggestions/questions, in decreasing order of importance\n- How many possible answers are there on average for the multi-answer dataset? It seems adversarially designed for poor performance for model logits. A cynical reading of the main results are that \"verbalized probability underperforms every calibration method except for the one specifically designed with many correct outputs.\" It would be great to have at least another data split to disentangle whether the benefits of verbalized probability extend beyond the extreme multi-answer case.\n- It would be great to have the baseline of train calibrator on multi-answer -> test calibrator on addition/multiplication. There may as well be every train/test split option in the paper.\n- Why not use something simple like platt scaling on the model logits? Zero-shot may be an unrealistic baseline.\n- I'd prefer that Figure 2 at least mentions in a footnote somewhere that he model is not actually trained to output \"Medium\", but rather a random name (which is then mapped on to Medium), otherwise this figure becomes slightly misleading.\n- Why alternate between EV uncertainties presented in some experiments and greedy in others? Why not keep this consistent?\n- I would love to see a discussion on the similarities and differences between this work and https://arxiv.org/abs/2207.05221 (which came afterwards of course, so this new paper is not relevant to my overall recommendation of the current paper). In particular this work tries some zero-shot uncertainty estimation (as I mentioned above) as well as learns calibration estimators directly on the model's internal representations (the \"I Know\" classifier)\n- Footnote 7 Typo - \"finetund\"\n\n# Missing References\n\n[Sabrina J. Mielke](https://arxiv.org/search/cs?searchtype=author&query=Mielke%2C+S+J), [Arthur Szlam](https://arxiv.org/search/cs?searchtype=author&query=Szlam%2C+A), [Emily Dinan](https://arxiv.org/search/cs?searchtype=author&query=Dinan%2C+E), [Y-Lan Boureau](https://arxiv.org/search/cs?searchtype=author&query=Boureau%2C+Y). Reducing conversational agents’ overconfidence through linguistic calibration. NAACL 2022.\nhttps://arxiv.org/abs/2012.14983\n\n[Shujian Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang%2C+S), [Chengyue Gong](https://arxiv.org/search/cs?searchtype=author&query=Gong%2C+C), [Eunsol Choi](https://arxiv.org/search/cs?searchtype=author&query=Choi%2C+E). Knowing More About Questions Can Help: Improving Calibration in Question Answering. Findings of ACL 2021. https://arxiv.org/abs/2106.01494", "broader_impact_concerns": "Again, I have a large broader impact concern of this paper's wording implying an emergent meta-cognitive behavior of large language models (\"learning to express its own uncertainty in natural language\") rather than a relatively simple calibration task implemented as discrete 5-class supervised learning.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bp95Ewz7jt", "reviewer": "Reviewer_st8A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- important topic\n- interesting method\n- well-written\n\nWeaknesses\n- difficulty in judging objective performance", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a nice paper on an important topic. I would be happy to see it appear in TMLR, with revisions.\n\nThe indirect logit and verbalized uncertainty are clever ideas. I am glad the authors chose to use OOD evaluations. I am also pleased they tested heuristic accounts. The results are promising, although there's room for improvement in future work.\n\nI do not closely follow the work on calibration, so I am not in a position to judge the novelty of the work. \n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The methods developed in the paper perform better than baselines, but it's hard to judge how good the model calibrations are in any objective sense. As the authors say, the MSE loss will not be zero, even for a perfectly calibrated model. It's also not straightforward to interpret the MAD estimate. For MSE, the authors could report an upper bound for an oracle model. For MAD, it would be useful to see the oracle level for having the category right but not the exact numbers, or some way to interpret absolute performance.\n\nIt's odd the paper ends with a related work section rather than a conclusion. This makes related work seems like an after thought.\n\nMinor comments\nFigure 4: : legend is covering baseline\nFigure 12: caption title doesn't apply to right panel", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rPNok411sb", "reviewer": "Reviewer_mS55", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To summarize my overall view: I think that the premise of the paper is very well executed, with a strong set of interesting experiments; but I am skeptical about some aspects of this premise, making me skeptical about the usefulness or informativeness of the paper overall.\n\nIn more detail, the strengths are:\n1. The paper is motivated by a very important problem in current NLP, namely hallucinations by language models.\n2. The paper presents a clear argument for why expressing uncertainty in words is better than previous approaches for addressing this problem\n3. The paper introduces a well-motivated and thoughtfully-constructed dataset for evaluating whether models can express their uncertainty in words\n4. Using this dataset, the paper provides experiments showing that GPT-3 can perform reasonably well at verbal probability\n5. (Perhaps the biggest strength, in my opinion): The authors do not just take their initial experiments at face value but instead conduct several careful analyses testing whether their results can be explained by factors other than uncertainty, such as shallow heuristics.\n6. The paper is overall extremely clearly written. This specific area is not one I am familiar with, yet I found the paper very easy to follow.\n\nBefore I state the weaknesses, let me be up front about some of my own views: I do not believe that language models can ever be trusted as sources of factual statements because language models are fundamentally learning about linguistic form, not learning about meaning (below I elaborate more on why I don’t think that confidence can address this concern). I don’t say this to be antagonistic - I just wanted to be up front about how my views may have affected my assessment of the paper, so that the editors can factor this in when considering my review. \n\nWith that said, here are the paper’s weaknesses, in my view:\n1. The “confident but wrong” concern: I am skeptical that the problem of hallucinations can be solved by language models themselves (as opposed to checking the model’s statements against trusted external databases). The reason is that language models could be confident but wrong; thus, the fact that a model is confident is not enough to conclude that we can trust it.\n    1. Here are several specific cases where language models would be likely to be confident but wrong:\n        1. Common misconceptions that are presented as fact in the training data: e.g., models might be confident that “Marie Antoinette is the source of the quote ‘Let them eat cake!’”\n        2. Outdated information: e.g., models trained before 2022 would probably be confident that “Angela Merkel is the Chancellor of Germany,” even though this statement is now false\n        3. Fiction: models’ training data usually does not clearly differentiate between factual and fictional text, so models might confidently believe statements that are made within a fictional context, such as “Certain people are born with superpowers such as x-ray vision”\n        4. Conspiracy theories and deliberate misinformation: Certain false statements might be widely presented as truth in training data derived from the internet (e.g., “The Earth is flat”)\n        5. Social biases: Language models have been shown to have many social biases (e.g., https://arxiv.org/pdf/2009.11462.pdf?fileGuid=zX3zH5DBtXQa0ktq). Thus, models might be highly confident in false, prejudiced statements like “Women cannot be good computer scientists.”\n    2. I am skeptical that models would be able to differentiate from cases where they are confident and right vs. cases where they are confident and wrong. For this reason, I think it is unlikely that the hallucination problem can be solved using language models themselves. And the CalibratedMath dataset does not address this concern, because math is one domain that is unlikely to be affected by the 5 situations listed above.\n2. Model confidence vs. user confidence: This concern is closely related to Weakness 1: The paper does not clearly differentiate two types of confidence: (i) The model’s confidence in its answer, and (ii) how confident users can be in the model’s answer. I will refer to (i) as “model confidence” and (ii) as “user confidence.”\n    1. I believe that much of the wording in the paper suggests *model confidence*. For example, the title is about teaching models to express “their” uncertainty; page 1 says “models should be calibrated about _their own_ uncertainty”; and Figure 1’s caption says “GPT-3 is scored on the calibration of its confidence.”\n    2. However, I believe that the goal of the paper is really *user confidence*: the motivation is about enabling users to know whether they can trust a statement made by a model, and the confidence is evaluated with respect to the ground truth.\n    3. As discussed under Weakness 1, I find it plausible that models can express model confidence, but I find it unlikely that they can express user confidence. This is a problem if the goal is user confidence (which I believe is indeed the goal). To the extent that model confidence and user confidence are aligned, the problem is solved; and the results of this paper suggest that, for the CalibratedMath domain, model confidence and user confidence can be aligned. However, as discussed under Weakness 1.1, there are several cases where model confidence may not be aligned with user confidence, and the CalibratedMath dataset is unlikely to address these cases, meaning that the paper as it stands does not address one of my main concerns with the general direction of using a language model to verbalize user confidence.\n3. Confidence in categories vs. individual examples: With the way that confidence is presented here, confidence can be reliably interpreted for groups of examples, but not for individual examples. This weakens the usefulness of this type of confidence, given that the motivation on page 1 is to enable users to know “how much to trust a given statement.” \n    1. As an example, consider the following scenario:\n        1. Suppose that there are 8 examples (A,B,C,D,W,X,Y,Z) where the model gets the first 4 correct and the second 4 incorrect:\n            - Correct: A,B,C,D\n            - Incorrect: W,X,Y,Z \n        2. Now consider these 2 different ways of assigning confidence scores:\n            - Assignment 1:\n                - 100% confidence: A,B\n                - 50% confidence: C,D,W,X\n                - 0% confidence: Y,Z\n            - Assignment 2:\n                - 100% confidence: C,D\n                - 50% confidence: A,B,Y,Z\n                - 0% confidence: W,X\n        3. Assignment 1 and Assignment 2 both get identical scores under MSE and MAD. Yet they assign very different confidences to individual examples (e.g., example A getting 100% confidence vs. 50% confidence). \n    2. As the authors discuss in Section 3.1, their setup is designed in a way that makes models likely to assign a uniform confidence within categories of examples, rather than being sensitive to individual variation within a category: “We recognize that this approach can lead to suboptimal labels. For example, it might use a low-confidence label for “10 × 10 = 100” because most two-digit multiplications are hard for GPT-3.“ The authors note that this approach nonetheless “works well enough for our purposes”; but this statement is based on holistic evaluations over groups of examples. Thus, I believe the concern remains that, for the purpose of assessing confidence in particular individual examples, the approaches presented in this paper might not be informative.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on the area of language model calibration: Getting language models to express calibrated uncertainty about the claims they make so that users know how much they can trust the claims. E.g., a calibrated language model might say “Answer: 17. Confidence: medium high” to indicate that it is reasonably but not entirely confident in its answer.\n\nWithin this area, the contributions that the paper makes are the following:\n1. **Argument for having language models express their uncertainty in words:** Previous work extracted uncertainty from the model’s logits, but the authors point out that the logit approach gives uncertainty over *token sequences*, whereas what the authors want is uncertainty over *claims* (and a claim might be expressible with many token sequences)\n2. **An approach for getting models to express their uncertainty in words:** The authors use supervised fine-tuning. To get the labels that provide the supervision, the authors use the model’s accuracy on the category that each specific example belongs to. E.g., if the example is an addition problem and GPT-3 gets 70% accuracy on such problems, the supervised confidence label might be “70%” or “high.” \n3. **A dataset called CalibratedMath for testing how well models can express verbalized probability:** This dataset contains a range of math problems for which a model must provide both an answer and a confidence level. Because GPT-3 has varying levels of performance on different types of math problems, this dataset supports diversity in the label distribution that models should produce for expressing their confidence. In addition, the dataset is set up in a way that gives a distribution shift from fine-tuning to evaluation, so that models cannot succeed just by memorizing the classes of problems they have seen.\n4. **Experiments showing that GPT-3 can verbalize its uncertainty reasonably well:** The experiments show reasonably strong in-distribution and out-of-distribution uncertainty levels; the out-of-distribution performance suggests that GPT-3 is indeed expressing uncertainty, not just memorizing the categories it has seen. In some cases, verbalized probability outperforms other methods and baselines; in other cases it does not. \n5. **Experiments further probing GPT-3’s verbalized probability abilities:** The authors show that GPT-3’s success at verbalized probability cannot be explained just by converting logits to words or by using several shallow heuristics. They then show that the success may be explained by pre-existing latent representations of uncertainty in GPT-3’s representations (i.e., information that is present in GPT-3 even before fine-tuning).\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "As it currently stands, I would recommend rejecting the paper due to the weaknesses listed above. If those weaknesses were addressed, I would then be much more likely to recommend accepting the paper (though I can’t guarantee it, because it would depend on how convincing the responses are). Because the problems I listed above are fairly fundamental ones, I would not expect the paper to solve all of those problems, but I would expect it to at least acknowledge them. Specifically, here are the changes that I think would be reasonable and that would go a long way toward addressing those concerns:\n1. To address Weakness 1 and Weakness 2: The paper could add in some discussion of the fact that we may never be able to fully trust language models alone, without verification from trusted external sources. \n    1. Some points that could be made along these lines include the following (I’m not saying you should make all these points - I’m just listing ideas that occur to me in case you find them useful):\n        1. The paper could mention scenarios where models might be confident but wrong\n        2. The paper could suggest that any usage of confidence scores could be accompanied by a caveat that a high confidence does not necessarily mean it is true.\n        3. The confidence scores could be given a ceiling of, say, 90%, so that there would always be some uncertainty reflected.\n        4. Confidence scores could be accompanied by reference to external databases; e.g., you could get a confidence score of 100% if and only if it can be verified in a trusted external knowledge base.\n        5. Future work could investigate the specific cases I listed like misconceptions or outdated information; it might be possible to teach models to recognize these types of scenarios and reduce their confidence accordingly. (E.g., you could teach models that information about who is the leader of a country is liable to becoming outdated). To be clear, I’m not at all saying that you should run these experiments - just something that you could mention as potential future work.\n    2. Some or all of these points could be made in a Broader Impact statement. Specifically, the statement could say something like: “A potential concern with providing confidence scores is that sometimes a model might assign a high confidence to false statements, which would then mislead users. Here are some ways that this concern could be addressed: [X,Y,Z].” On a more positive note, the Broader Impact statement could also point out how confidence scores will help users recognize weaknesses of current models, so they have the potential to reduce the harms of such models (as long as we remain open about the fact that the confidence scores themselves might be flawed).\n2. To address Weakness 3: The paper could include a bit more discussion of this concern, beyond the oblique mention of it in Section 3.1. Even better would be brief mentions of ways that future work could build toward evaluating confidence scores at the level of individual examples rather than just groups of examples (but I don’t have any suggestions about how this would be done).\n\nAll of the remaining points listed below are not critical for securing my recommendation - they would simply strengthen the work in my view:\n1. Figure 5: Does “training” refer literally to training examples that models have seen, or to withheld examples that are in-distribution?\n2. Section 3.3: It took me a couple re-reads to figure out the point of the stochastic few-shot experiment. The motivation that is given is “in order to learn more about how verbalized probability generalizes”, but as I understand it the motivation is more like “in order to understand the extent to which verbalized probability is already present in the pre-trained model vs. learned via fine-tuning”? If this interpretation is correct, it might be worth it to clarify.\n3. On page 9: It says “cannot be fully explained by these heuristics”: I would suggest changing to something like “cannot be fully explained by these heuristics, though the possibility remains that it could be explained by other heuristics that we have not considered.”\n4. On page 9: You used a GPT-3 model “finetuned for semantic similarity.” Why did you use this version instead of a non-finetuned one? Is it the case that the GPT-3 API does not allow you to get embeddings from the non-finetuned one? \n5. Since Section 4 has only one subsection, you could get rid of the subsection 4.1 heading and just call Section 4 “Discussion: Directions for future work”\n", "broader_impact_concerns": "See “Requested Changes 1.2.” I don’t think that a Broader Impact Statement is necessary, but it could be added as a way to address some of the concerns I had.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work is clearly of interest to the community.  The question of correctness can be interpreted in two ways: 1. Accuracy of presented results, 2. how accurately the paper's claims are supported by the evidence presented.  The former is true, while the latter is slightly debated by the reviewers.   A number of concrete suggestions have been presented to improve the paper.  

The required modification is that the paper's framing be updated to clearly align the results as an extension of existing calibration work and as multiple reviewers read the paper as implying "metacognitive" abilities, the introduction should address this directly.  There are a number of places that likely move the reader to make this inference, e.g. use of terms like "human-like", which should be avoided. See related comments around “shown”, “express”, and “own”.

A number of other requests (e.g. Oracles for better understanding of the results) would also improve the work.
Decision: Accept (poster)