Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
Authors: Katherine Tian
Eric Mitchell
Allan Zhou
Archit Sharma
Rafael Rafailov
Huaxiu Yao
Chelsea Finn
Christopher D Manning
Venue: EMNLP 2023 Main (2023)
Keywords: calibration
RLHF
language model
verbalized probability
TL;DR: We find that using verbalized probabilities, rather than the model's conditional probabilities, to measure model confidence reduces the expected calibration error of RLHF-LMs.

Abstract: A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
Katherine Tian,* *t Eric Mitchell," # Allan Zhou, Archit Sharma, Rafael Rafailoy Huaxiu Yao,² Chelsea Finn, Christopher D. Manning
Harvard University Stanford University ktian@college harvard. edu gric.mitchell@es stanford edu
Abstract
Leg (Sanpled!)
10
Linguistic Expo
Calibration Error:
Calibration Error:
68-
trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to expert in cases of low-confidence predictions. Re- cent studies have shown that unsupervised pre- training produces large language models (LMs) whose conditional probabilitics are remarkably well-calibrated. However, the most widely- used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMS. and some studies have suggested that RI.EE- LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGITT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typi- cally better-calibrated than the model's condi- tional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the ex- pected calibration error by relative 50%.
666
accurscy
02
0.5
0.4
0.6
0.0
3.0
00
0.3
0.4
0.5
0.5
1.0
1.0
Calibration
Calibration Error:
0.234
0.085
scorso
0.0
0.4
06
0.00
3.0
0.0
0.2
0.4
0.5
2.0
Figure 1: Verbalized confidence scores (bluc) are better-calibrated than log probabilities (orange) for gpt-3. 5-turbo. Raw model probubilitics (top-left) are con- sistently over-confident. Verhalized numerical probabilities (bottom) are better-calibrated Considering more answer choices (bottom-right) further improves verbalized calibra- tion (as in "Considering the Opposite' in psychology: Lord et al. (1985).. Verbalized expressions of likelikood (top-right) also provide improved calibration. Bar height is average accu- racy of predictions in bin. Darker bars mean more predictions fall in that confidence range. Results computed on SciQ.
attention (Brown et al., 2020; Roberts et al., 2020; Bubeck et al., 2023), relatively little attention has been given to their well-callbraatedness (Kadavath et al., 2022). Further, most existing analyses of the calibratedness of LLMs focus on models trained with maximum likelihood, while in practice, the most widely-used LLMs (such as ChatGPT) are fine-tuned using methods such reinforcement learning from human feedback (Christiano et al., 2017). Some findings suggest that RLHF-LMs may sacrifice well-calibrated predictions for the sake of closer adherence to user instructions in dialogue (Kadavath et al., 2022; OpenAI, 2023), as the rein- forcement learning objective encourages the model to allocate probability mass to the most preferred answer(s), rather than matching the relative fre- quency of possible answers.
1 Introduction
Real-world prediction systems invariably make er- rors. However, some mitigation of these errors is possible if the system produces well-calibrated' confidence estimates. In this case, the system's least confident predictions correspond to those that are most likely be incorrect, potentially allowing these predictions to be skipped or overridden by a human. In the context of language models, one consequence of poor calibration may be hallucina- tion, where a language model confidently asserts incorrect facts or reasoning. While the ability of very large LMs to absorb and synthesize knowl- edge about the outside world has gained significant
"Equal contribution.
li..., the confidence in prediction accurately reflects the probability that the prediction is correct (Guo et al., 2017).
This paper evaluates several methods for ex- tracting confidences about model predictions from

---

ECE Comparison
AUC Comparioon (1)
calibrated predictions (Kadavath al., 2022; Xiao et al., 2022: Kuhn al., 2023). Other work focuses on the tendency of language and dialogue models to use linguistic expressions of uncertainty in a well-calibrated manner (Zhou et al., 2023; Mielke et al., 2022). However, existing studies focus on LMs trained purely with unsupervised learning (although Kadavath et al. (2022) briefly examine RLHF-LMs), while widely used models in prac- tice are fine-tuned with instruction-tuning or RLHF (Christiano et al., 2017). RLHF has been shown to effectively leverage annotations of human pref- erences to control sentiment (Ziegler et al., 2020), improve summarization or instruction-following quality (Stiennon et al., 2022; Ouyang et al., 2022), and inject behavioral priors of harmlessness (Bai et al., 2022b,a). However, recent work has raised the question of whether or not RLHF harms cali- bration (OpenAI, 2023). Our work is the first show that verbalized probabilities are often better- calibrated than the model's ' conditional probabili- ties for RLHF-LMs such as ChatGPT, GPT-4, and Claude, and Llama-2-70B-Chh
Figure 2: RLHF generally worsens the calibration of Llama-70B's log probabilities, as measured by ECE (lower is better) or AUC (higher is better). However, this paper (Ta- bles 1-5) will show that for several strong RI.F--MMs, the model's verbalized confidence is often calibr-calirrated than its log probabilities, reversing some of this degradation. This reversal is strongest for TruthfulQA, adversarial dataset testing common misconeeptions and other difficult queries.
RLHF-LMs. Due to concerns that RLHF may cause systematic overconfidence in the model's probabilities (Figure 2), as well the general un- availability of per-token peeprobabillities widely used RLHF-LMs, we pay particular attention to prompts that elicit verbalized probabilities, i.e., the model expresses its confidence in token-space, as either numerical probabilities or another linguistic expression of uncertainty. We find that, surpris- ingly, popular RLHF-LMs are able directly ver- balize confidence scores that better-calibrated than the model's conditional probabilities (esti- mated via sampling), without any fine-tuning to learn verbalization. To further improve calibration, we take inspiration from research in human psy- chology showing that overconfidence can be mit- igated by considering alternative answers before responding (Lord et al., 1985; Mussweiler al., 2000). We show that prompting model to produce several answer choices before giving its confidence scores significantly improves calibration of ver- balized probabilities. Combined with temperature scaling (Guo et al., 2017), this approach generally provides better calibration than model probabilities for ChatGPT GPT-4", and Claude 2 across three datasets, often reducing expected calibration error (ECH) by over 50%.
2 Evaluating Calibration in RLHF-LMs
To study the calibration of RLHF-LMs, we con- duct experiments with g 3-tturbb (ChatGPT), gpt-4 (GPT-4), claude-1 (Claude 1), claude-2 (Claude 2), and 2-ma--2-7bb-har (Llama-2- 70B-Chat).
Metrics. We measure calibration with multiple metrics. To measure ECE (expected calibration er- ror; Guo et al. (2017)), we bin model predictions by their confidence and measure the average accuracy of predictions in each confidence bin. The ECE defined as the average (squared) error between the average accuracy and confidence within each bin, where each error is weighted by the fraction of samples falling within the bin. We report raw ECE as well as ECE with temperature scaling (ECE-t), Temperature scaling fits single temperature value 3 to the model's confidences minimize negative log likelihood (NLL) the data, giving scaled probability D& of class P5 x p; See Figure for a depiction of ECE binning. Although ECE a standard and interpretable measure of calibration error, it completely fails to capture the confidences' discriminative power. We therefore also report
Related Work. Several studies have examined the calibration of large LMs (Lin et al., 2022a; Park and Caragea, 2022; Kadavath et al., 2022: Xiao et al., 2022; Kuhn et al., 2023), finding that combining large pre-trained LMs with tempera- ture scaling (Guo et al., 2017) produces very well-
"gpt-3.5-turbo accessed in June 2023.
3 For binary classification, system that guesses randomly and outputs 50% confidence each time has perfect ECE.
Card-Claude 2ff

---

TriviaQA
SeiQ
TruthfulQA
Method
ECE
ECE4
BS-t
AUC-
BCE
ECE-t:
BS-t
AUC
ECE
ECE-1
BS-t
1
Label prob.
0.140
0.097
0.142
0.869
0.256
0.180
0.223
0.752
0.451
0.317
0.345
0,418
'Is True' prob.
0.164
0.159
0.165
0.826
0.312
0.309
0.309
0.677
0.470
0.471
0.476
0.384
Entropy
0.547
0.483
0.236
Verb. IS up-1
0.068
0.076
0.138
0.879
0.234
0.084
0.214
0.744
0.389
0.256
0.322
0.545
Verb. IS 2op-2
0.050
0.053
0.139
0.894
0.132
0.050
0.201
0.766
0.361
0.115
0.252
0,485
Verb. IS top-4
0.054
0.057
0.144
0.896
0.065
0.051
0.209
0.763
0.203
0.189
0.284
0,455
Verh. 28 CaT
0.110
0.123
0.168
0.830
0.323
0.246
0.296
0.683
0.419
0.259
0.292
0.551
Verh. 28 top-1
0.131
0.099
0.148
0.855
0.340
0.2013
0.268
0.677
0.431
0.245
0.282
0.483
Verb. 28 up-2
0.047
0.045
0.147
0.887
0.169
0.040
0.201
0.768
0.395
0.101
0.224
0.517
Verb. top-4
0.050
0.051
0.156
0.861
0.130
0.046
0.211
0.729
0.270
0.156
0.246
0.463
Ling. 1S human
0.062
0.069
0.137
0.884
0.166
0,087
0.223
0.703
0,306
0.296
0.333
0.503
Ling. 1S-opt.
0.058
0.066
0.135
0.878
0.064
0.068
0.220
0.674
0.125
0.165
0.270
0.492
Table I: Mcasuring calibration of various methods for extracting confidences from 3.t-3.5-turee (ChaGGPT). The model's conditional probabilities are relatively poorly calibrated, whether using the model's conditional probability of the label given the query (Label prob.) or the probability assigned to 'True' given the query, proposed answer, and a prompt asking if the answer is correct ('Is True' prob.). Surprisingly, directly verbalizing probability (Verb. 1S and Verb. 2S) such as 'highly likely' (Ling. 18) yields significantly betler-calibrated confidence estimates. IS refers one-stage prediction. where the model provides an answer and confidence probabiltyylexpression together. 2S refers to two-stage prediction. where the model first gives only answer, and then in second stage confidence. To color the table cells, for each column, we demean and scale by constant to ohtain shade in [-1,1], where eyan indicates better and orange worse performance.
expression of confidence
TrivinQA
SciQ
TruthfulQA
Method
ECE i
ECE-t
BS-t
AUC
ECE
ECE-t
BS-t
AUC
ECE
ECE-t
BS-t
AUC
Label prob.
0.078
0.067
0.077
0.950
0.219
0.165
0.186
0.820
0.445
0.334
0.362
0.462
Verb. top-1
0.024
0,084
0,937
0.201
0.084
0.165
0.843
0.350
0.156
0.622
Verh. 18 u
0.025
0.034
0.084
0.949
0.140
0.048
0.185
0.813
0.315
0.112
0.228
0.623
Verb. 18 top-4
0.041
0.039
0.081
0.959
0.056
0.059
0.185
0.815
0.198
0.144
0.245
0.619
Ling. IS-human
0.051
0.041
0.086
0.931
0.148
0.024
0.170
0.835
0,241
0.151
0.228
0.651
Ling. 1S-opt.
0.056
0.051
0.088
0.927
0.028
0.052
0.172
0.828
0.082
0,105
0,212
0.632
Table 2: gpt-4's verbalized probabilities are substantially catter-calibrated than model probabilities themselves, even after temperature scaling. similarly to gpt-3. b-tur Table 1.
Brier Score (BS; Brier (1950)) temperature- scaled confidences (BS-t), proper scoring rule (Ovadia al., 2019) that the mean squared error between the confidences and the correctness labels. Finally, we assess calibration using a metric from the selective classification literature (Geifman and EI- Vaniv, 2017), specifically, the area under the curve of selective accuracy and coverage (AUC).
Evaluation protocol. For each dataset, gener- a response and corresponding confidence from each method on each of the evaluation questions. Because calibration essentially quantifies the re- lationship between model confidence and correct- ness, computing correctness is crucial to accurate measurements of calibration. However, we find doing so to be challenge, especially in datasets where only single ground-truth answer (but not aliases semantically equivalent rephrases) is pro- vided. To avoid excessive false negatives in our correctness computation as a result of exact-match evaluation, we use either GPT-4 or GPT-3.5 to eval- uate whether a response is essentially equivalent to the ground truth answer; see Appendix C for the complete equivalence- procedure.
Datasets. Our experiments use three question- answering datasets assessing factual knowledge. TriviaQA (Joshi et al., 2017) contains 650k question-answer pairs gathered by trivia enthusi- asts; SeiQ (Welbl et al., 2017) contains approxi- mately 14k crowdsourced science exam question- answer pairs; TruthfulQA (Lin et al., 2022b) con- tains 817 questions designed to test language mod- els' tendency to mimic human falsehoods. We sample 1000 questions from the validation split of TriviaQA (rc.wet web. nocontext) and SciQ and all 817 questions from the validation split of Truth- fulQA (generation) for our experiments.
Methods.
We compare a wide variety of methods for extracting confidence estimates from LLMs. For a comprehensive list of the prompts used for each method, see Appendix Table 6.
First, we consider two methods that leverage the true conditional distribution of the model to gener-

---

TriviaQA
SeiQ
TruthfulQA
Method
ECE
ECE4:
BS-t
AUC-
ECE:
ECE-t:
BS-t
AUC
ECE
ECE-1
BS-t
AUC
Label prob.
0.074
0.079
0.117
0.915
0.216
0.149
0.195
0.786
0.432
0.304
0.335
Verb. IS top-1
0.049
0.059
0.160
0.839
0.265
0.103
0.247
0.663
0.440
0.134
0.204
0.411
Verb. 2
0.046
0.047
0.158
0.875
0.207
0.040
0.225
0.693
0.450
0.085
0.197
Verh. IS top-4
0.075
0.079
0.176
0.814
0.151
0.057
0.226
0.667
0.372
0.105
0.183
0.377
Ling. 1S human
0.053
0,050
0.151
0.867
0.253
0.118
0.245
0.664
0.443
0.358
0.340
0.384
Ling. 1S-ope.
0.074
0.060
0.149
0.863
0.089
0,082
0.238
0,623
0.139
0.148
0.228
0,350
Table 3: Claude-I I produces similar- or better-calibrated log probabilities to gpt-3.5 barbb, but is less able to verhalize well-calibrated confidences, compared to models in the GPT family of RLHF-LMs. Claude-1 has since been deprecated.
TriviaQA
SciQ
TruthfulQA
Method
ECE
ECE-t:
BS-t.
AUC-
ECE:
ECE-t:
BS-t:
AUC
ECE
ECE-t
BS-t
AUC
Label prob.
0.089
0,089
0.137
0.882
0.181
0.176
0.237
0.762
0.409
0.368
0.405
0.319
Verb. IS top-1
0.072
0.071
0.141
0.903
0,204
0.054
0.201
0.776
0.345
0.115
0.215
0,573
Verb.
top-2
0.049
0.054
0.133
0.918
0.134
0.041
0.211
0.754
0.359
0.085
0.223
0.491
Verb. 1S top-4
0.072
0.063
0.158
0.890
0.048
0.052
0.216
0.711
0.274
0.075
0.208
0.473
Ling. IS human
0.085
0.061
0.151
0.878
0.238
0.026
0.209
0.756
0.381
0.242
0.305
0.530
Ling. IS-ope.
0.060
0.070
0.151
0.874
0,049
0.056
0.214
0.738
0.099
0.130
0.266
0.446
Table 4: Claude-2 has weaker conditional probabilities than Claude-1 and GPT+ but its verbalized calibration provides consistent improvement over conditional probabilities at a level comparable to GPT-3.5 and surpassing GPT-* on TruthfulQA.
ate confidence scores. The simplest is Label prob., which uses the conditional probability distribution p(y(z)) of the model given a question x, which we estimate using = 10 samples, since many RLHF- LMs are flosed-source and do not offer per-token probabilities. 67 We return the most common an- swer, using the LLM- equivalence function determine when two lexically different answers are semantically equivalent. In a variation of the method described by Kadavath al. (2022) (again, we use samples since model probabilities are not available), 'Is True' prob. samples a single answer y from the model given question x, and the prob- ability it is true is estimated by the probability the model assigns to "True' when asked if the given answer is true (where once again the probabilities are estimated via samples), i.c., p(True|z: z. 9).
sociated probability as the model's output and con- fidence. Verb. 2S top-k similarly uses numeri- cal probabilities, except the model is first asked to provide only its answers, and afterwards, in a second round of dialogue, asked to assign prob- abilities of correctness to each answer (i.e., '2 stages"). Verb. 2S CoT uses chain-of-thougee prompt before giving a single answer, and in a second round of dialogue, the model is prompted to assign a probability that answer (with the chain of thought present in the model's context). Ling, 1S-human uses linguistic likelihood expres- sions, rather than numerical probabilities, to ex- press uncertainty. The model is prompted to assign confidences to its guesses by choosing from set of linguistic expressions of uncertainty: (Almost certain, Likely, .... Almost no chance). Each linguistic likelihood expression is mapped to a probability using responses from a human sur- vey on social media with 123 respondents (Fagen- Ulmschneider, 2023). Ling, 1S-opt. uses a held out set of calibration questions and answers to com- pute the average accuracy for each likelihood ex- pression, using these 'optimized' values instead. Expressions that are not used for at least N of questions, where N is the number of calibration questions, simply use the human probability.
Next, we consider methods that extract con- fidence scores through verbalization (Lin et al., 2022a), i.e., where the model expresses its confi- dence in token space, either with numerical prob- abilities or linguistic expressions of likelihood." 8 First, Verb. IS top- prompts the model to pro- duce l guesses and a probability that each is cor- rect all in a single response (i.e.. stage"). We take the
prediction and its as-
"We evaluated gpt-3.5-te 3-5tub on all three datasets using n 20 samples, but the calibration did not meaningfully improve, we always
10 to reduce API costs. "For each closed LM, we use its default sampling param- eters (top-p 1.0 for GPT. and top-p 0.7 for Claude), For Llama-2, we use temperature
Results
Tables 1-5 show the results of evaluating various methods for extracting confidence from RLHF- gpt-4, claude-1,
and lop-p 1.0.
"However, note that none of the methods descrihed fine- tune the model to perform better on verbalization.
LMs on

---

TriviaQA
SeiQ
TruthfulQA
Method
ECE
ECE4:
BS-t
AUC-
BCE:
ECE-t:
BS-t
AUC
ECE
ECE-1
BS-t
AUC
Label prob.
0.151
0.124
0.156
0.865
0.266
0.189
0.243
0.707
0.405
0.361
0.396
0.407
Verb. IS top-1
0.071
0.067
0.186
0.793
0.196
0.053
0.239
0.648
0.386
0.172
0.266
0.502
Verb. 2
0.060
0.073
0.194
0.815
0.153
0.032
0.230
0.667
0.340
0.037
0.227
0.440
Verh. IS iop-4
0.069
0.079
0.182
0.816
0.105
0.043
0.229
0.648
0.231
0.102
0.237
0.465
Ling. 1S human
0.179
0.115
0.195
0.749
0.071
0.101
0.252
0.603
0.376
0.366
0.383
0,407
Ling. 1S-ope.
0.077
0.068
0.186
0.779
0.019
0,042
0,236
0.590
0.047
0.051
0.239
0.435
Table With Llama2-708- 70002--78--ha,,,, Chat, verhalized calibration provides improvement over conditional probabilities across some metrics, but the improvement is much less consistent compared to GPT- and Claude-*
claude-2, and Llama-2-70b-char 2-78b-chat, respectively. We distill several key conclusions from these exper- iments. 1. Large RLHF-LMs can often directly verbalize better-calibrated confidences (either a numerical confidence probability or an expres- sion such 'highly likely") than the models' conditional probabilities. 2. Among the methods for verbalizing probabilities directly, we observe that generating and evaluating multiple hypothe- ses improves calibration (see Figure 1), similarly to humans (Lord et al., 1985), and corroborating a similar finding in LMs (Kadavath et al., 2022). 3. Language models can express their uncertainty with numerical probabilities as well or better than with words, which is surprising in light of long- standing difficulties in representing numbers in lan- guage models (Thawani al., 2021). 4. Chain- of of-thought prompting does not improve verbalized calibration (see Appendix Figure for additional CoT results). 5. The calibration of both Claude models' conditional probabilities roughly falls be- 5 3.-...5.- turbo and gpt-4; however, while Claude 1 is much weaker at verbalizing its con- fidence, Claude 2 is generally bit stronger than gpt-3.5-tuu t.otubbb at verbalizing. The verbal calibra- tion of the open source model -lama-2-70b-chh is generally weaker than that of closed source mod- els but still demonstrates improvement over its con- ditional probabilities by some metrics, and docs so most clearly on TruthfulQA.
elicit calibrated confidences from RLHF-LMs by prompting the model to verbalize its confidence in token space. We find verbalized probabilities are better-calibrated than conditional probabilities across several closed models, with mixed results for Llama-2-70B-Cher
Our results raise several questions for future work. Most notably, the difference between GPT-*, Claude-#, and Llama-2's ability to verbalize confi- dence is significant. What factors are important for learning this skill? Additionally, the 1-stage and 2-stage verbalized numerical confidence prompts sometimes differ drastically in the calibration of their confidences. How can we reduce sensitivity of a model's calibration to the prompt? Going beyond guestion-answering can we leverage good calibra- tion in short-answer settings to improve the reliabil- ity of long-form generations, perhaps by breaking down long-form generation into a sequence of short questions? Finally, what extent does language model's calibration depend on the domain; do our conclusions in the context of factual recall hold in the context of reasoning or arithmetic? Answering these questions provides one path toward building more trustworthy and useful language systems.
tween gpt-3.
While our work demonstrates a promising new approach to generating calibrated confidences through verbalization, there are lim- itations that could be addressed in future work. First, our experiments are focused on factual recall- oriented problems, and the extent to which our ob- servations would hold for reasoning-heavy settings is an interesting open question. Additionally, the lack of technical details available for many state-of- the-art closed RLHF-LMs may limit our ability to understand what factors enable model to verbalize well-calibrated confidences and differences in this ability across different models. Finally, our study limited to short-form question-answeringg future work should extend this analysis to longer-form generation settings.
Limitations.
4 Discussion
In summary, we study the calibration of widely used RLHF-LMs. We first replicate the finding for GPT-4 (OpenAI, 2023) that RLHF can worsen the calibration of a model's conditional probabilities using the open-source Llama-2-70B base and chat models (Figuree 2). To mitigate this regression and ease extraction of calibrated confidence scores for models for which log probabilities are not avail- able, we propose and study new methods that can

---

Acknowledgements. CF and CDM are CIFAR Fel- lows. EM gratefully acknowledges funding from a Knight-Hennessy Graduate Fellowship. AZ is supported by the NSF graduate research fellowship program. This research was supported in part by Juniper Networks, Apple, and ONR grant N00014- 20-1-2675. The authors thank Yoonho Lee and Noah Goodman for helpful feedback on calibration metrics and experiment design.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with GPT-4. ArXiv preprint arXiv:2303.122 12712.
Paul F Christiamo, Jan Leike, Tom Brown, Miljan Mar- tic. Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
References
Wade Fagen- Seeen-Ummschnidder 2023, Perception of proba- bility words. Ms., UIUC, 05-24-2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dode Dodds, Danny Hemandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedhack.
Yonatan Geifman and Ran EI- Yaniv. 2017. Selective classification for deep neural networks. In Proceed- ings of the 31st International Conference on Neu- ral Information Processing Systems, NIPS' 17, page 4885-4894, Red Hook, NY, USA. Curran Associates Inc.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Pro- ceedings of Machine Learning Research, pages 1321- 1330. PMLR.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto. Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott John- ston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly. Tom Henighan, Tristan Hume, Samuel R. Bow- man, Zac Hatfield-Dor -odds.. Ben Mann. Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 20226. Constitutional AI: Harmless- ness from ai feedback.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver. Canada. Association for Computational Linguisties.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer. Zac Hatfeeld-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. Arxiv arxiv:2207.0522
Glenn W. Brier, 1950. Verification of Forecasts Ex- pressed in Terms of Probability. Monthly Weather Review, 7-.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voose Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever. and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. In The Eleventh International Conference on Learn- ing Representations.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022a. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022b. TruthfulQA: Measuring how models mimic human

---

falsehoods. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252 Dublin, Ireland. Association for Computational Linguistics.
Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021. Representing numbers in NLP: a survey and vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644 656, Online. As- sociation for Computational Linguisties.
Charles Lord, Mark Lepper, and Elizabeth Preston. 1985. Considering the opposite: corrective strat- egy for social judgment. Journal of personality and social psychology, 47:1231-43.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. ArXiv, abs/ 707.06609,
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y- Lan Boureau. 2022. Reducing conversational agents' overconfidence through linguistic calibration. Trans- actions of the Association for Computational Linguis- tics, 10:857-872
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis- Philippe Morency. 2022. Uncertainty quantification with pre-trained language models: A large-scale em- pirical analysis. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7273-7284 Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Thomas Mussweiler, Fritz Strack, and Tim Pfeiffer. 2000. Overcoming the inevitable anchoring effect: Considering the opposite compensates for selective uccessibility, Personality and Social Psychology Bul- letin, 26(9):1142-1150
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: Expressions of over- confidence and uncertainty in language models.
OpenAl. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman. Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2020. Fine-tuning lan- guage models from human preferences.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Pro- ceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc.
Seo Yeon Park and Cornelaa Caragea. 2022. On the cal- ibration pre-trained language models using mixup guided by area under the margin and saliency. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume Long Papers), pages 5364-5374, Dublin, Ireland. As- sociation for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of language model? In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLPP,, pages 5418855426 Online. Association for Computational Linguistics.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano. 2022. Learning to summarize from human feedback.

---

Usage likelihood expressions by 3.5-turbo
SCA
0.40
0.25
0.30
0.25
0.20
0.35
33
23
62
24
14
0.10
0.00
Chanco siere
Litl Chacrea
Pribabbl
Probobby
ATHoz Cortain
Sceet Chance
23
Likelihood Expression
0.340
Figure 3: 3.t-3.5- t-turbo usage rate of each likelihood ex- pression; the model displays much lower verbalized confi- dence on TruthfulQA than on standard factual recall problems.
10
27
62
30
Usage of likelihood expressions by GPT-4
0.210
Fractoo Responses
Theaos
22-
32
9.1
14
0.314
6:328
20
Ounoo en S Sigt
Bran
Dare
02
52
65
82
34
Likellhood Expression
Figure 5: Expected calibration error is not consistently im- proved for any CoT prompt variant on gpt-3. 3.5-brbbo.
Figure gpt-4 usage rate each likelihood expression; the model displays markedly lower verhalized confidence on TruthfulQA than on standard factual recall problems.
each fold, we use it once to fit a temperature and evaluate metrics on the remaining folds. We find that fitting the temperature on 20% of the data yields relatively stable temperatures across folds. We report the average temperature-scaled ECE and BS as ECE-t and BS-t.
A Additional Results
Here, we include the likelihood expression usage distribution for gpt-3. and gpt-4 in Figures and 4, respectively. gpt-3.5 is systematically less confident for TruthfulQA. The contrast between model confidence for TriviaQA and SciQ compared with TruthfulQA is even more stark for gpt-4.
To compute ECE and AUC for Ling. 1S-opt., we similarly split our total data into folds, using 4 folds to fit the probabilities behind each linguistic expression confidence, then evaluating on the remaining fold. To compute ECE-t and BS-t for Ling. 1S-opt, we hold out one of the 5 folds to fit temperature. We use 3 folds to fit probabilities for linguistic expressions, compute the temperature based these probabilities set, and evaluate metrics on the last fold. We then average metrics across all 20 rotations folds.
We also provide additional calibration results for chain-of-thought methods. We compare one- stage verbalized CoT prompt (Verb. IS CoT), two-stage verbalized CoT prompt (Verb. 2S CoT), and a two-stage verbalized method that uses CoT just before eliciting the numerical confidence (Verb, 2S Cot Prob) instead of before the guess, as shown for gpt-3.5 3.5 on Trivia QA, SciQ, and Truthful QA in Figure 5. We find that CoT does not noticeably improve calibration across any setting dataset.
the temperature
C Prompt Templates
The prompt template for each sampling method is provided in Table The question is substi- tuted for the variable SCTHE_ in each prompt. To evaluate answer correctness, we use 5-t-b.5-turb for SciQ and TruthfulQA and gpt-4 for TriviaQA due to 5-turboo'ss
B Fitting Procedure for Temperature and Probabilities for Linguistic Expressions
To fit the temperature that is used to compute ECE- and BS-t we split our total data into folds. For
5-turbo

---

Method
Template
Label prob.
Provide your best guess for the following question. Give ONLY the guess, no other words or explanation. \n\nnor example: Vn'nGuess: <most likely guess, as short as possible; not complete sentence, just the guess!>\nknThe question 1s:$<THE__UEESTOO)
'Is True" proh.
Question: $(QUESTION)\/Propseed Answer: S[ANSWER)\nIs the proposed onswer:\n\t(< True or\n)t(B) False?\n The proposed answer is:
Verh. top-1
Provide your best guess and probability that it correct (0.0 1.0) for the following question. Give ONLY the guess and probability, other words or explanation.
example: Vn\nGuess: <most likely guess, as short as possible; not complete sentence, just the guess!>An Probability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!/AnnnTii
question is: S(THE_QUESTION)
Verb. IS top-k
Provide your $(k) best guesses and the probability that each is correct (0.0 to 1.0) for the following question. Give ONLY the guesses and probabilities, no other words explanation. For example:Annnn AnknG1: <first most likely guess, short as possible; not complete sentence, just the guess!>\n\nP1: <the probability between 0.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!> ... GS(k): <$[k]-th most likely guess, short possible; not a complete sentence, just the guess!>\n\nPs(K): <the probability between 0.e e and 1.0 that G$(K) is correct, without any extra commentary whatsoever; just the probability!> \n'nThe question is: SCTHE_QUESTION)
Verh. 2S CoT
Provide your best guess for the following question. Before giving your answer, provide step-by-step explanation of your thought process. Then on new line give the guess with other words explanation. AnAnFor haample:\nnnExxplaattoo <one sentence step-by-step explanation of your thought process>\n\nnuess: <eost likely guess, short possible; question is: @(THE_QUESTION)
complete sentence, just the guess!>An\nThe
Provide the probability that your guess is correct. Give ONLY the probability, other words explanation. AnAnFor Shample:\n\/Probabill 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!>1n
<the probability between
Verb. 2S top-1
Provide your best guess for the following question. Give ONLY the guess, no other words or explanation. AnknFor example: Vn'nGuess: <most likely guess, as short as possible; not complete sentence, just the guess!>\nknThe question is: {
Provide the probability that your guess is correct. Give ONLY the probability, no other words explanation. AnAnFor example:Nn hitrroballtl <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!>>n
Verb. 2S top-k
Provide your $(k) best guesses for the following question. Give ONLY the guesses, no other words or explanation. For example: An\nG1: <first most likely guess, as short as possible; not a complete sentence, just the guess!>An\nP1: <the probability between 0.0 and 1.0 that GT is correct, without any extra commentary whatsoever; just the probability!> not complete sentence, just the guess!>in\nThe question Is::(THE_QUESTIO Provide the probability that each of your guesses is correct. Give ONLY the probabilities, probability between 8.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!>'n... PS(k): <the probability between 0.0 and 1.0 that GS(k) is correct, without any extra commentary whatsoever; just the probability!>
CS(K): <$[k)-th most likely guess, as short as possible;
other words or explanation. \nAnFor example:
<the
Ling. IS
Provide your best guess for the following question, and describe how likely it is that your guess is correct as one of the following expressions: Give ONLY the guess and your confidence, no other words or explanation. For example: An'nGuess: <most likely guess, as short as possible; not complete sentence, just the guess/>AnConfidence: <description of confidence, without any extra commentary whatsoever; just short phrase!>AnAnThe question is: $(THE_QUESTION)
CXXPRESSION_LIST)
Table 6: Prompt templates for each method evaluated. Methods above the double line use multiple samples in order to estimate confidence scores; methods below the double line use the verbalized confidences directly, requiring only a single sample.

---

high disagreement with human evaluator on TriviaQA. Using the ground truth answer as (GGLLD_ANSWER) and the model-generated answer as S(PRED_ANSWER), we use the following prompt template:
Are the following two answers to my question semantically equivalent?\n'n0C : (CGOLD_ANSWER)\NAA answer with single word, either "Yes." or "No.", and explain your reasoning.


Reviews:
[{"review_id": "jjCBs67owp", "reviewer": "Reviewer_qYcs", "summary": "", "questions": "- How much do you think that potential contamination could affect the models overconfidence?\n- Do you plan to compare these numbers with the non RLHF fine tuned models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our work, and your appreciation of its relevance in light of the widespread use of RLHF-LMs!\n\nRegarding data leakage, while it's possible that data leakage might contribute to the overconfidence of RLHF-LMs, we suggest that data leakage does not completely explain the poor performance of the model logits. Past work has shown that overconfidence due to overfitting to the training distribution may be largely mitigated by simple temperature scaling in both image classifiers [1] and large language models [2]. However, our results show that even with temperature scaling, the logits of RLHF-LMs show relatively poor calibration, suggesting that something other than data leakage is involved (likely the RLHF objective, to some extent). \n\nMoreover, in light of the Llama-2 models released since the submission deadline, we have been able to study larger-sized open RLHF-LMs as well. We confirm that the post-RLHF Llama-2 logprobs also have much worse calibration (ECE-t 0.134, 0.207, 0.382) than the pre-RLHF Llama-2 base model (ECE-t 0.032, 0.061, 0.228), following the same pattern as GPT4 [3]. \n\nRegarding calibration of non-RLHF models, the emphasis of our study is RLHF-LMs because they represent the wide majority of generative language models used in practice, and previous work has studied calibration of non-RLHF-LMs through fine-tuning [2] or prompting alone [4]. Our study is the first to show that verbalization without any fine-tuning is an effective strategy for improving calibration in RLHF-LMs, adding a significant caveat to the conclusion from [3] (the \"GPT-4 paper\") that RLHF largely destroys calibration. \n\nNonetheless, we conduct an additional study of some of the calibration methods for non-RLHF models including the pre-RLHF InstructGPT model and the base Llama-2 70B model. The results are below:\n\n* InstructGPT's model probabilities are well-calibrated across TriviaQA, SciQ, and TruthfulQA (ECE-t 0.040, 0.075, 0.216) in comparison to post-RLHF GPT-3.5/ChatGPT (ECE-t 0.097, 0.180, 0.317). However, InstructGPT demonstrates poor verbalized calibration with k=1 (ECE-t 0.221, 0.264, 0.371) in comparison to GPT-3.5 (ECE-t 0.076, 0.084, 0.256) and struggles to express k=4 verbalized calibration at all, while GPT-3.5 benefits from k=4 verbalization (ECE-t 0.057, 0.051, 0.189), outperforming the average ECE of InstructGPT's logprobs.\n\n* To consider the pre-RLHF Llama-2 base model, we use a few shot prompt that provides example answers and their corresponding model conditional probabilities. Similarly to InstructGPT, Llama-2-base has well-calibrated model conditional probabilities across three datasets (ECE-t 0.032, 0.061, 0.228) but has uninformative k=1 verbalized calibration (ECE-t 0.129, 0.245, 0.320).\n\n\nAlthough future investigation of more complex methods may be able to elicit calibrated verbalized confidences from pre-RLHF models, this ability is not easily found in pre-RLHF models. \n\n[1] On Calibration of Modern Neural Networks. Chuan Guo et al., 2017. ICML.\n\n[2] Teaching models to express their uncertainty in words. Stephanie Lin et al., 2022. TMLR.\n\n[3] GPT-4 Technical Report. OpenAI, 2023. arXiv.\n\n[4] Language Models (Mostly) Know What They Know. Saurav Kadavath et al., 2022. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and it’s easy to follow.\n- The topic of the paper is very relevant due to the broad deployment of RLHF models.", "reasons_to_reject": "- All the LMs analysed are closed so it’s difficult to know how potential data leakage affects the conclusions. \n- They don’t compare the calibration results of the non RLHF models.", "questions_to_authors": "- How much do you think that potential contamination could affect the models overconfidence?\n- Do you plan to compare these numbers with the non RLHF fine tuned models?", "paper_topic_and_main_contributions": "In this paper the authors present an evaluation of the calibration of RLHF fine-tuned LLMs. In order to do this, they extract different sources of confidence scores from the models and later apply a suite of metrics to measure how the confidence scores relate to the likelihood that the answer is correct. Their main conclusions are that verbalized confidences are better calibrated than using conditional distributions as confidence scores, as this ones tend to be overconfident. Verbalized confidences token level outputs by the language models. It is also interesting to see how generating multiple hypotheses improves calibration.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MXpfWas6EH", "reviewer": "Reviewer_M6yR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback - we appreciate that you saw our work as an extensive study providing novel perspectives!\n\nRegarding simplicity of the method:\n* Since fine-tuning widely-used closed models is expensive or impossible and these models are expensive to sample from, simple prompting-based approaches to improving calibration are more likely to see greater adoption and improve real-world model trustworthiness than more complex or expensive algorithms. Additionally, our method does not require having access to model weights and can be used on closed RLHF models. \n* Many users of RLHF-LMs are not AI experts, and so simple methods like our k-shot verbalized probability prompts are more amenable to being used by practitioners.\n\nRegarding novelty:\n* Although our work does not present a complex new methodology, our work is the first to show that LLMs can produce well-calibrated verbalized confidences without any fine-tuning or training specifically for this ability, which we consider to be a non-obvious result that is very relevant to current LLMs. Future research directions could investigate how or why this ability emerges, especially post-RLHF.\n* Beyond showing that top-k verbalized probabilities are quite well-calibrated, a primary scientific contribution of our work is showing that RLHF-LMs can produce better-calibrated confidences than implied by widely-cited recent work on RLHF (the \"GPT-4 paper\") [1].\n* To our knowledge, our work is the first to show a phenomenon like \"considering the opposite\" from human psychology (which is able to reduce human overconfidence) can improve calibration in LLMs (via top-k verbalized methods).\n\nWe hope these points address your concerns regarding the novelty of our contributions.\n\n[1] GPT-4 Technical Report. OpenAI, 2023. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces novel perspectives on calibration and presents some promising experimental results.", "reasons_to_reject": "The paper delineates a calibration approach and presents some favorable conclusions; however, the method is relatively simple and in some extent, lacks novelty.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper contributes by addressing the imperative requirement for a trustworthy real-world prediction system with well-calibrated confidence scores. It acknowledges the existing success of unsupervised pre-training in generating highly calibrated conditional probabilities within large language models (LMs). However, recognizing the potential calibration issues within reinforcement learning fine-tuned LMs (RLHF-LMs), the study undertakes an extensive assessment of techniques to derive confidence scores from such models. Notably, for RLHF-LMs like ChatGPT, GPT-4, and Claude, the research demonstrates that the utilization of verbalized confidences as output tokens yields improved calibration compared to the model's native conditional probabilities. These findings are validated across benchmark datasets including TriviaQA, SciQ, and TruthfulQA, resulting in a substantial reduction of the anticipated calibration error by approximately 50%.\n\n\n\n\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gUson6xHN2", "reviewer": "Reviewer_P3om", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review - we're glad you appreciated the simplicity of our methods and the completeness of our experiments! We've now performed initial experiments on Llama-2 (which, of course, came out after the submission date), including both the base model and the RLHF-tuned version. The results are below:\n\nFirst, we verify that RLHF worsens the calibration of model logprobs in LLama-2 (RLHF ECE-t 0.134, 0.207, 0.382 vs. Pretrained ECE-t 0.032, 0.061, 0.228), which follows the same pattern in GPT-4 [1]. \n\nNext, comparing Llama-2-RLHF to GPT-3.5 and Claude, we observe that GPT-3.5 has the best k=1 verbalized calibration overall. Llama-2-RLHF has the weakest calibration from model probabilities, and its verbalized calibration is overall comparable to Claude, with worse calibration on TriviaQA and better on TruthfulQA (see table of full results below). We hope that as more large open-source models become available, future work can study the differences between these models and how they lead to the ability to verbalize calibrated confidences. \n\n\nTable. ECE-t, BS-t, AUC on TriviaQA; ECE-t, BS-t, AUC on SciQ; ECE-t, BS-t, AUC on TruthfulQA: \n\nGPT-3.5 verbalized k=1:           0.076, 0.138, 0.879; 0.084, 0.214, 0.744; 0.256, 0.322, 0.545\n\nClaude verbalized k=1:            0.059, 0.160, 0.839; 0.103, 0.247, 0.663; 0.134, 0.204, 0.411\n\nLlama2 verbalized k=1: 0.045, 0.187, 0.779; 0.027, 0.235, 0.663; 0.097, 0.241, 0.475 \n\nGPT-3.5 sampling:                   0.097, 0.142, 0.869; 0.180, 0.223, 0.752; 0.317, 0.345, 0.418\n\nClaude sampling:                     0.079, 0.117, 0.915; 0.149, 0.195, 0.786, 0.304, 0.335, 0.418\n\nLlama2 sampling:          0.130, 0.155, 0.847; 0.234, 0.268, 0.690; 0.382, 0.412, 0.398\n\n[1] GPT-4 Technical Report. OpenAI, 2023. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper tackles an important and practical problem. Extracting confidence from a black-box LLM is an important direction that deserves more research. The solutions are simple and effective. The ablation experiment is complete.", "reasons_to_reject": "The trends of ChatGPT and Claude are slightly different. In this case, the paper would be stronger if more language models are analyzed, for example, LLaMA, in which logits can be extracted from the LM.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper tackles the question of extracting well-calibrated confidence scores from state-of-the-art language models. The main discoveries are\n\n1. LM can directly verbalize confidence scores well\n2. Eliciting multiple guesses can further improve the confidence scores\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eFhBXduBpB", "reviewer": "Reviewer_B7ZR", "summary": "", "questions": "see the reason to reject", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review of our work and belief that it has interesting findings!\n\nWe note that in addition to reporting evaluations of multiple widely-used RLHF-LMs, we make the following concrete conclusions of scientific and practical importance:\n* RLHF-LMs can often produce better-calibrated confidences than suggested by widely-cited recent work on RLHF (the \"GPT-4 paper\") [1] through verbalization\n* Practitioners using these models would benefit from using verbalized probabilities to assess model confidence, which was not previously known. This point is particularly important because using model logits to extract confidence is not an option for the most popular RLHF-LM APIs, so verbalized probabilities are the only choice.\n* The differences in verbalization ability between GPT* models and Claude may be impacted by the decision to perform RLHF (GPT*) or RLAIF (Claude), though it is difficult to verify this hypothesis without closer access to the models\n* To our knowledge, our work is the first to show a phenomenon like \"considering the opposite\" from human psychology (which is able to reduce human overconfidence) can improve calibration in LLMs (via top-k verbalized methods).\n\nBecause most of the training details of ChatGPT, GPT-4, and Claude are unknown, it is difficult to draw additional specific conclusions about the impact of particular training-time design decisions. However, we believe the discovery of the utility of verbalized probabilities without fine-tuning is of significant scientific importance (where does this ability come from?) and practical relevance (practitioners can benefit from using this technique).\n\nRegarding the applicability of points 2 and 3 to non-RLHF-LMs, we note that prior work [1] has shown that non-RLHF-LMs can be fine-tuned to verbalize probabilities. However, no work to our knowledge has shown that LMs can verbalize confidence effectively without any fine-tuning, which we hypothesize is due to the difficulty of constructing a prompt that effectively specifies the intended confidence-verbalization behavior. This situation is complicated by the finding that large pre-trained LMs may almost entirely ignore the labels in their few-shot examples [2], so the numerical confidences in the prompt may be ignored to a non-trivial extent.\n\nNonetheless, we perform several experiments with the pre-RLHF InstructGPT and pre-RLHF Llama-2 model, and discuss the results below.\n\n* InstructGPT's model probabilities are well-calibrated across TriviaQA, SciQ, and TruthfulQA (ECE-t 0.040, 0.075, 0.216) in comparison to post-RLHF GPT-3.5/ChatGPT (ECE-t 0.097, 0.180, 0.317). However, InstructGPT demonstrates poor verbalized calibration with k=1 (ECE-t 0.221, 0.264, 0.371) in comparison to GPT-3.5 (ECE-t 0.076, 0.084, 0.256) and struggles to express k=4 verbalized calibration at all, while GPT-3.5 benefits from k=4 verbalization (ECE-t 0.057, 0.051, 0.189), outperforming the average ECE of InstructGPT's logprobs.\n\n* To consider the pre-RLHF Llama-2 base model, we use a few shot prompt that provides example answers and their corresponding model conditional probabilities. Similarly to InstructGPT, Llama-2-base has well-calibrated model conditional probabilities across three datasets (ECE-t 0.032, 0.061, 0.228) but has uninformative k=1 verbalized calibration (ECE-t 0.129, 0.245, 0.320). \n\n\nAlthough future investigation of more complex methods may be able to elicit calibrated verbalized confidences from pre-RLHF models, this ability is not easily found in pre-RLHF models.\n\n\n[1] Teaching models to express their uncertainty in words. Stephanie Lin et al., 2022. TMLR.\n\n[2] Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Sewon Min et al., 2022. EMNLP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good writing and easy to follow.\n\nAn interesting finding of extracting confidence via two-stage prediction can improve the calibration of RLHF-LMs.\n", "reasons_to_reject": "The evaluation of three RLHF-LMs appears to lack meaningful conclusions beyond the presentation of two additional tables (i.e., Tables 2 and 3).\n\nI am interested in understanding the reason behind the statement, \"The calibration of Claude's conditional probabilities falls between GPT-3.5-turbo and GPT-4, but its ability to verbalize calibrated confidences is weaker than both GPT models.\" What are the specific differences between the two models, and are there any more generalized and concrete observations?\n\nAre conclusions 2 and 3 of Section 3 applicable to all LMs, or are these phenomena specific only to RLHF-LMs?", "questions_to_authors": "see the reason to reject", "paper_topic_and_main_contributions": "This paper measures the calibration of various methods for extracting confidences from RLHF-LMs (e.g., ChatGPT, GPT4, and Claude).\nThey demonstrate that Large RLHF-LMs, particularly GPT-family models, can often directly verbalize better-calibrated confidences.\nBesides, generating and evaluating multiple hypotheses improves calibration.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies model calibration of LLMs trained with RLHF based on existing observation that RLHF worsens the calibration of model, i.e. the confidence of model predictions less correlates with accuracy. Due to lack of log probabilities of closed models, authors propose to use verbalized probabilities. They also show that prompting a model to produce several answer choices before giving its confidence scores significantly improves calibration of verbalized probabilities. Per reviewers' requests, authors added experiments with llama2 and llama2-RLHF models. I think this paper studies a very important issue -- model calibration in the LLM regime, and provides timely and interesting studies regarding how to measure model calibration for closed models. I encourage authors to add more results of open-sourced models in the camera-ready and include more evidence on "RLHF worsens the calibration".
Decision: Accept-Main