Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Authors: Miao Xiong
Zhiyuan Hu
Xinyang Lu
YIFEI LI
Jie Fu
Junxian He
Bryan Hooi
Venue: ICLR 2024 poster (2024)
Keywords: uncertainty quantification
uncertainty estimation
calibration
failure prediction
large language models
black-box language models
LLM evaluation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a framework for eliciting confidence in black-box LLMs, revealing that while its calibration improves with model capacity, failure prediction remains a challenge.

Abstract: Empowering large language models (LLMs) to accurately express confidence in their answers is essential for reliable and trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on *white-box access* to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of *black-box* approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: *prompting* strategies for eliciting verbalized confidence, *sampling* methods for generating multiple responses, and *aggregation* techniques for computing consistency. We then benchmark these methods on two key tasks—confidence calibration and failure prediction—across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be *overconfident*, potentially imitating human patterns of expressing confidence. 2) As model capability scales up, both calibration and failure prediction performance improve, yet still far from ideal performance. 
3) Employing our proposed strategies, such as human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 
4) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as a strong baseline and provide insights for eliciting confidence in black-box LLMs. The code is publicly available at https://github.com/MiaoXiong2320/llm-uncertainty.

Published as a conference paper at ICLR 2024
CAN LLMs EXPRESS THEIR UNCERTAINTY? AN EMPIRICAL EVALUATION OF CONFIDENCE ELICI- TATION IN LLMS
Miao Xiong'; Zhiyuan Hu', Xinyang Lu', Yifei ,i", Jie Fu², Junxian He"! Bryan Hooi
1 National University of Singapore The Hong Kong University of Science and Technology
3 École Polytechnique Fédérale de Lausanne
ABSTRACT
Empowering large language models (LLMs) to accurately express confidence their answers is essential for reliable and trustworthy decision-making Previ- ous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMss, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling methods for generating multiple responses, and aggregation techniques for comput- ing consistency. We then benchmark these methods on two key tasks confidence calibration and failure prediction-acron five types of datasets (e.g.. commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence. 2) As model capability scales up. both calibration and failure prediction performance improve, yet still far from ideal performance. 3) Employing our proposed strategies, such human- human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 4) Comparisons with white-box meth- ods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC, Despite these udvancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as strong baseline and provide insights for eliciting confidence in black-box LLMs. The code publicly available at https////gtabbbcomm//ooooooooo22 ott.:
INTRODUCTION
A key aspect of human intelligence lies in our capability to meaningfully express and communicate our uncertainty a variety of ways (Cosmides & Tooby, 1996). Reliable uncertainty estimates are crucial for et al., 2017; Tomani & Buettner, 2021). Specifically, accurate confidence estimates of model can provide valuable insights into the reliability of its responses, facilitating risk assessment and error mitigation (Kuleshov et al., 2018; Kuleshov & Deshpande, 2022), selective generation (Ren et al., 2022), and reducing hallucinations in natural language generation tasks (Xiao Wang, 2021).
collaboration, enabling more rational and informed decision-making (Guo
In the existing literature, eliciting confidence from machine learning models has predominantly relied on white-box access internal model information, such as token-likelihoods (Malinin & Gales, 2020; Kadavath et al., 2022) and associated calibration techniques (Jiang et al., 2021), as well as model fine-tuning (Lin et al., 2022). However, with the prevalence of large language models, these
"Coresponning to: Miao Xiong uniao.xiong @ ed.))..edu). I Equal advising: bhooi @ comp wwwppppnuseuuuu.. edu.sg, junxianh@esec cse.ant.ssse...kkk

---

Published as a conference paper at ICLR 2024
methods are becoming less suitable for several reasons: 1) The rise of closed-source LLMs with commercialized APIs, such as GPT-3.5 (OpenAI, 2021) and GPT-4 (OpenAI, 2023), which only allow textual inputs and outputs, lacking access to primarily captures the model's 's uncertainty about the next token (Kuhn et al., 2023), rather than the semantic probability inherent in textual meanings. For example, in the phrase "Chocolate milk comes from brown cows", every word fits naturally based on its surrounding words, but high individual token likelihoods do not capture the falsity of the overall statement, which requires examining the statement semantically, in terms of its claims; 3) Model fine-tuning demands substantial computational resources, which may be prohibitive for researchers with lower computational resources. Given these constraints, there is growing need to explore black-box approaches for eliciting the confidence of LLMs their answers, a task we refer to as confidence elicitation.
or embeddings: 2) Token-likelihhnn
Recognizing this research gap, our study aims to contribute to the existing knowledge from two perspectives: 1) explore black-box methods for confidence elicitation, and 2) conduct a comparative analysis to shed light on methods and directions for eliciting more accurate confidence. To achieve this, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling strategies for generating multiple responses, and aggregation strategies for computing the consistency. For each component, we devise a suite of methods. By integrating these components, we formulate set of algorithms tailored for confidence elicitation. A comprehensive overview of the framework is depicted in Figure 1. We then benchmark these methods on two key cosks-confidence calibration and failure accrsssion-aross five types of tasks (Commonsense, Arithmetic, Symbolic, Ethics and Professional Knowledge) and five widely-used LLMs, i.c., GPT-3 (Brown et al., 2020), GPT-3.5 (OpenAI, 2021), GPT-4, Vieuna (Chiang et al., 2023) and LLaMA 2 (Touvron et al., 2023b).
Our investigation yields several observations: 1) LLMs tend to be highly overconfident when verbaliz- ing their confidence, posing potential risks for the safe deployment of LLMs ($5.1). Intriguingly, the verbalized confidence values predominantly fall within the 80% to 100% range and are typically in multiples of 5, similar to how humans talk about confidence. In addition, while scaling model capacity leads to performance improvement, the results remain suboptimal. 2) Prompting strategies, inspired by patterns observed in human dialogues, can mitigate this overconfidence, but the improvement also diminishes as the model capacity scales up ($5.2). Furthermore, while the calibration error (e.g. ECE) can be significantly reduced using suitable prompting strategies, failure prediction still remains a challenge. 3) Our study on sampling and aggregation strategies indicates their effectiveness in improving failure prediction performance ($5.3). 4) A detailed examination of aggregation strategies reveals that they cater to specific performance metrics, i.e., calibration and failure prediction, and can be selected based on desired outcomes ($5.4). 5) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC ($B. I.) Despite these insights, is worth noting that the methods introduced herein still face challenges in failure prediction, especially with tasks demanding specialized knowledge ($6). This emphasizes the ongoing need for further research and development in confidence elicitation for LLMs.
2 RELATED WORKS
Confidence Elicitation in LLMs. Confidence elicitation is the process of estimating LLM's s confi- dence in their responses without model fine-tuning or accessing internal information. Within this scope, Lin et al. (2022) introduced the concept of verbalized confidence that prompts LLMs to express confidence directly. However, they mainly focus on fine-tuning on specific datasets where the confidence is provided, and its zero-shot verbalized confidence is unexplored. Other approaches, like the external calibrator from Mielke et al. (2022), depend on internal model representations, which are often inaccessible. While Zhou et al. (2023) examines the impact of confidence, it does not provide direct confidence scores to users. Our work aligns most closely with the concurrent study by Tian et al. (2023), which mainly focuses on the use of prompting strategies. Our approach diverges by aiming to explore a broader method space, and propose a comprehensive framework for systematically evaluating various strategies and their integration. We also consider a wider range of models beyond those RLHF-LMS examined in concurrent research, thus broadening the scope of confidence elicitation. Our results reveal persistent challenges across more complex tasks and contribute to a holistic understanding of confidence elicitation. For more comprehensive discussion of the related works, kindly refer to Appendix C.
2

---

Published as a conference paper at ICLR 2024
Sampling Strategy
Prongt Sintegy
Response
Question
Vania
Mtdtisngp
Blackk-hon
Sumple Reqqurizc
Aggregator
Aiswer: Confidenor:
Seff Proding
Tep-k.
Response K
Prompt Vanilla
M--3
Aguraystor - AngeCoon
Self Random
Ansseer IsB Couildenee 100%
peine many are u the list 12....,000 1007
Qr Provide the answer and confidence
Black
Sumple
Resporses
Uiettdenzz
Ang Aggrogation
Confilence:
Ansreer:
Figure 1: An Overview and example Confidence Elicitation framework, which consists of three components: prompt, sampling and aggregator. By integrating distinct strategies from each compo- nent, we can devise different algorithms, e.g., Top-K (Tian et al., 2023) is formulated using Top-K prompt, self-random sampling with M = 1, and Avg-Conf aggregation. Given an input question, we first choose a suitable prompt strategy, c.g., the vanilla prompt used here. Next, we determine the number of samples to generate (M 3 here) and sampling strategy, and then choose aggregator based on our preference (e.g. focus more on improving calibration or failure prediction) to compute confidences in its potential answers. The highest confident answer is selected as the final output.
EXPLORING BLACK-BOX FRAMEWORK FOR CONFIDENCE ELICITATION
In our pursuit to explore black-box approaches for eliciting confidence, we investigated a range of methods and discovered that they can be encapsulated within unified framework. This framework. with its three pivotal components, offers a variety of algorithmic choices that combine to create diverse algorithms with different benefits for confidence elicitation. In our later experimental section ($5), we will analyze our proposed strategies within each component, aiming to shed light on the best practices for eliciting confidence black-box LLMs.
3.1 MOTIVATION OF THE FRAMEWORK
Prompting strategy. The key question aim answer here is: in black-box setting, what form of model inputs and outputs lead to the most accurate confidence estimates? This parallels the rich study in eliciting confidences from human experts: for example, patients often inquire of doctors about their confidence in the potential success of a surgery. We refer to this goal as verbalized confidence, and inspired by strategies for human elicitation, design series of human-inspired prompting strategies to elicit the model's verbalized confidence. We then unify these prompting strategies as building block of our framework ($3.2). In addition, beyond its simplicity. this approach also offers an extra benefit over model's 's token-likes likelihood: the verbalized confidence is intrinsically tied to the semantic meaning of the answer instead of its syntactic lexical form (Kuhn al., 2023).
Sampling and Aggregation. In addition to the direct insights from model outputs, the variance observed among multiple responses for given question offers another valuable perspective on model confidence. This line of thought aligns with the principle extensively explored in prior white-bos box access uncertainty estimation methodologies for classification (Gawlikowske et al., 2021), such as MCDropout (Gal & Ghahramani, 2016) and Deep Ensemble (Lakshminarayanan et al., 2017). The challenges adapting ensemble- based methods lie in two critical components: 1) the sampling strategy, i.e., how to sample multiple responses from the model's answer distribution, and 2) the aggregation stralegy, i.e., how to aggregate these responses to yield the final answer and its associated confidence. To optimally harness both textual output and response variance, have integrated them within a unified framework.
3.2 PROMPTING STRATEGY
Drawing inspiration from patterns observed in human dialogues, we design a series of human human-inspired prompting strategies tackle challenges, e.g., overconfidence. that are inherent in the vanilla version of verbalized confidence. See Table for an overview these prompting strategies and Appendix for complete prompts.

---

I C, where n represents the total number of Published as a conference paper at ICLR 2024
Table Illustration of the prompting strategy (the complete prompt in Appendix F), To help models understand the concept of confidence, we also append the explanation "Note: The confidence indicates how likely you think your answer true." " to every prompt.
Method
Prompt
Vanilla
Read the question. provide your answer, and your confidence in this answer.
CoT
Read the question, analyze step by step. provide your answer and your confidence in this answer.
Self- Probing
Question: [...] Possible Answer: [...] Q: How likely is the above answer to be correct? Analyze the possible answer, provide your reasoning concisely, and give your confidence in this answer.
Multi-Step
Read the question, break down the problem into K steps, think step by step, give your confidence in each step, and then derive your final answer and your confidence in this answer.
Top-K
Provide your K best guesses and the probability that each is correct (0% to 100%) for the following question.
COT. Considering that a better comprehension of a problem can lead to more accurate understanding of one's 's certainty, we adopt a reasoning-augee augmented prompting strategy. In this paper, we use zero- shot Chain-of oh-Tuhhgh,, CoT (Kojima et al., 2022) for its proven efficacy in inducing reasoning processes and improving model accuracy across diverse datasets. Alternative strategies such as plan-and-solve (Wang et al., 2023) can also be used.
Self-Probing. A common observation of humans is that they often find it easier to identify errors in others' answers than in their own, they can become fixated on a particular line of thinking, potentially overlooking mistakes. Building on this assumption, we investigate if a model's 's uncertainty estimation improves when given a question and its answer, then asked, "How likely the above answer to be correct"? The procedure involves generating the answer in one chat session and obtaining its verbalized confidence in another independent chat session.
Multi-Step. Our preliminary study shows that LLMs tend to be overconfident when verbalizing their confidence (see Figure 2). To address this, we explore whether dissecting the reasoning process into steps and extracting the confidence of each step can alleviate the overconfidence. The rationale is that understanding each reasoning step's 's confidence could help the model identify potential inaccuracies and quantify their confidence more accurately. Specifically, for given question, we prompt models to delineate their reasoning process into individual steps St and evaluate their confidence in the correctness of this particular step, denoted as C.. The overall verbalized confidence is then derived by aggregating the confidence of all steps: Cimultistep step reasoning steps.
Top-K. Another way to alleviate overconfidence is to realize the existence of multiple possible solutions answers, which acts normalization for the confidence distribution. Motivated by this, Top-K (Tian et al., 2023) prompts LLMs to generate the top K guesses and their corresponding confidence for given question.
3.3 SAMPLING STRATEGY
Several methods can be employed to elicit multiple responses of the same question from the model: 1) Self-random, leveraging the model's inherent randomness by inputting the same prompt multi- ple times. The temperature, an adjustable parameter, can be used to calibrate the predicted token distribution, i.e., adjust the diversity of the sampled answers. An alternative choice is to introduce perturbations in the questions: 2) Prompting, by paraphrasing the questions in different ways to generate multiple responses. 3) Misleading, feeding misleading cues to the model, e.g.. 'I think the answer might be ...". This method draws inspiration from human behaviors: when confident, individ- uals tend to stick to their initial answers despite contrary suggestions; conversely, when uncertain, they are more likely waver adjust their responses based misleading hints. Building on this observation, we evaluate the model's 's response to misleading information to gauge its uncertainty. See Table 11 for the complete prompts.
3.4 AGGREGATION STRATEGY
Consistency. A natural idea of aggregating different answers is to measure the degree of agreement among the candidate outputs and integrate the inherent uncertainty in the model's output.
4

---

Published as a conference paper at ICLR 2024
For any given question and associated answer Y. sample set of candidate answers Y,, where i E {i,... M. The agreement between these candidate responses and the original answer then serves measure of confidence, computed follows:
Connsistency M
E =Y-
(I)
Avg-Conf. The previous aggregation method does not utilize the available information of verbalized confidence. It is worth exploring the potential synergy between these uncertainty indicators, i.e., whether the verbalized confidence and the consistency between answers can complement one another. For any question and associated answer Y, sample candidate set {YI,...Yuu Yur} with their corresponding verbalized confidence {Crr...Crr, and compute the confidence as follows:
Coout E I(, = Y) C
Σ C2
This aggregation strategy is tailored for responses generated using the Top-K prompt, as it mainly utilizes the ranking information of the model's Top-K guesses. The underlying assumption is that the model's 's ranking between two options may be more accurate than the verbalized confidence it provides, especially given observation that the latter tends to exhibit overconfidence.
Pair-Rank,
Given a question with N candidate responses, the i-th response consists of K sequentially ordered answers, denoted as K (S(") .s(") ..... s(i)), Let A represent the set of unique answers across all N responses, where M is the total number of distinct answers. The event where the model ranks
(i)
answer Sg above S, (i.e., S,, appears before S,)) in its i-th generation is represented (S,, > Sy). In contexts where the generation is implicit, this is simply denoted as (S, > 5.). Let Elit) be the event where least of Sy and S, appears in the i-th generation. Then the probability of (S, > S,), conditional on E(i) and a categorical distribution P. expressed as IP(S.. > S, Eli)
We then utilize (conditional) maximum likelihood estimation (MLE) inspired approach to derive the categorical distribution P that most accurately reflects these ranking events of all the M responses:
N
min ΣΣ Σ 1=1 S.EASEEA
Su
So
log P (Sw > S. P.,
subject to Σ ((S)) =1
S.EA
Proposition 3.1. Suppose the Top-K answers are drawn from a categorical distribution P without replacement. Define the event (S., > Sw) to indicate that the realization S, is observed before S, in the i-th draw without replacement. Under this setting, the conditional probability is given by:
(S.. S. 1,,
P(S.)
P(S.) P(S.)
The optimization objective to minimize the expected loss is then:
N
P(Sw)
S. Se log P(S.)+ min - -E
S.F. Σ P (4)
1=1 S.S.AAS.AAA
S.EA
To address this constrained optimization problem, we first introduce change of variables by applying the softmax function the unbounded domain. This transformation inherently satisfies the simplex constraints, converting our problem into an unconstrained optimization setting. Subsequently, optimization techniques such as gradient descent can be used to obtain the categorical distribution.
EXPERIMENT SETUP
Datasets. We evaluate the quality of confidence estimates across five types reasoning tasks: 1) Commonsense Reasoning two benchmarks, Sports Understanding (SportUND) (Kim, 2021) and StrategyQA (Geva et al., 2021) from BigBench (Ghazal et al., 2013); 2) Arithmetic Reasoning on two math problems, GSM8K (Cobbe et al., 2021) and SVAMP (Patel et al., 2021); Symbolic Reasoning
5

---

Published as a conference paper at ICLR 2024
KGC 9:15 BUROU 9M
ALC 9-2E DUROE BEC
ALC SAY MUNDE 933
ALC SAZ BUROE B.k-
ansiwer
correct arover
unower
correct Laronee
Corfidiance
60
90
100
50
70
90
100
78
60
90
100
50
63
74
90
300
Confidence
Condidence
Confidence
Confienne
Figure 2: Empirical distribution (First row) and reliability diagram (Second row) of vanilla verbalized confidence across four models on GSM8K. The prompt used is in Table 14. From this figure, we observe that 1) the confidence levels primarily range between 80% and 100%, often in multiples of 5; 2) the accuracy within each bin is much lower than its corresponding confidence, indicating significant overconfidence.
on henchmarks, Date Understanding (DateUnd) (Wu & Wang, 2021) and Object Counting (ObjectCou) (Wang et al., 2019) from BigBench; 4) tasks requiring Professional Knowledge, such as Professional Law (Prf-Law) from MMLU (Hendryeks et al., 2021); 5) tasks that require Ethical Knowledge, c.g., Business Ethics (Biz-Ethics) from MMLU (Hendrycks et al., 2021).
Models We incorporate range of widely used LLMs of different scales, including Vicuna 13B (Chiang et al.. 2023). GPT-3 175B (Brown et al., 2020), GPT-3.5-turbo (OpenAI, 2021), GPT-4 (OpenAI, 2023) and LLaMA 2 70B (Touvron et al., 2023b).
Evaluation Metrics. To evaluate the quality of confidence outputs, two orthogonal tasks are typically employed: calibration and failure prediction (Naeini et al., 2015; Yuan al., 2021; Xiong al., 2022). Calibration evaluates how well a model's 's expressed confidence aligns with its actual accuracy: ideally, samples with an 80% confidence should have an accuracy 80%. Such well-calibrated scores crucial for applications including risk assessment. On the other hand, failure prediction gauges the model's capacity to assign higher confidence to correct predictions and lower to incorrect ones, aiming to determine if confidence scores effectively distinguish between correct and incorrect predictions. our study, we employ Expected Calibration Error (ECE) for calibration evaluation and Area Under the Receiver Operating Characteristic Curve (AUROC) for gauging failure prediction. Given the potential imbalance from varying accuracy levels, we also introduce AUPRC-Posiee (PR-P) and AUPRC-Negative (PR-N) metries emphasize whether the model can identify incorrect and correct samples, respectively.
Further details datasets, models, metrics, and implementation can be found in Appendix E.
EVALUATION AND ANALYSIS
To provide insights the best practice for eliciting confidence, we systematically examine each component (see Figure 1) of the confidence elicitation framework ($3). We test the performance on eight datasets of five different reasoning types and five commonly used models (see $4), and yield the following key findings.
5.1 LLMS TEND TO BE OVERCONFIDENT WHEN VERBALIZING THEIR CONFIDENCE
The distribution of verbalized confidences mimics how humans talk about confidence. To examine model's capacity to express verbalized confidence, we first visualize the distribution of confidence in Figure 2. Detailed results other datasets and models are provided in Appendix Figure 5. Notably, the models tend to have high confidence for all samples, appearing as multiples of and with most values ranging between the 80% 100% range, which is similar to the patterns identified in the training corpus for GPT-like models as discussed by Zhou et al. (2023). Such behavior suggests that models might be imitating human expressions when verbalizing confidence.

---

Published as a conference paper at ICLR 2024
Table 2: Vanilla Verbalized Confidence of 4 models and 8 datasets (metrics are given by x Abbreviations used: Date (Date Understanding), Count (Object Counting). Sport (Sport Under- standing), Law (Professional Law), Ethics (Business Ethics). ECE 0.25, AUROC, AUPRC-Positive AUPRC-Negative 0.6 denote significant deviation from ideal performance. Significant deviations in averages are highlighted in red. The prompt used is in Table 14.
Metric
Model
GSM8K
SVAMP
Date
Count
Strategy
Sport
Law
Ethics
Avg
GPT-3
82.7
35.0
82.1
52.0
41.8
42.0
47.8
32.3
52.0
ECE 4
Vicuna
76.0
70.7
17.0
45.3
42.5
37.5
45.2
34.6
46.1
ILaMA
71.8
36.4
38.5
58.0
26.2
38.8
42.2
36.5
43.6
GPT-3.5
66.0
22.4
47.0
47.1
26.0
25.1
44.3
23.4
37.7
GPT-4
31.0
10.7
18.0
26.8
16.1
15.4
17.3
8.5
18.0
GPT3
51.2
51.7
50.2
50.0
49.3
55.3
46.5
56.1
51.3
ROC 1
Vicuna
52.1
46.3
53.7
53.1
50.9
53.6
52.6
57.5
52.5
LLaMA
58.8
52.1
71.4
51.3
56.0
48.5
50.5
62.4
56.4
GPT-3.5
65.0
63.2
57.0
54.1
52.8
43.2
50.5
55.2
55.1
GPT4
81.0
56.7
68.0
52.0
55.3
60.0
60.9
68.0
62.7
GPT-3
85.0
37.3
82.2
52.0
42.0
46.4
51.2
41.2
54.7
Vicuna
96.4
87.9
34.9
65.4
53.8
51.5
75.3
70.9
67.0
PR-N 1
LLaMA
92.6
57.4
88.3
59.6
38.2
40.6
61.0
58.3
62.0
GPT-3.5
79,0
33,9
64.0
51.2
35.7
30.5
54.8
35.5
48.1
GPT-4
65.0
15.8
26.0
28.9
26.6
31.5
40.0
39.5
34.2
GPT-3
15.5
65.5
17.9
48.0
57.6
59.0
45.4
66.1
46.9
PR-P
Vicuna
4.10
11.0
69.1
39.1
47.5
52,0
27.2
38.8
36.1
ILaMA
11.9
46.3
46.6
41.4
68.6
58.3
39.2
65.0
47.2
GPT-3.5
38.0
81.3
57.0
54.4
67.2
67.5
45.8
70.5
60.2
GPT-4
57.0
90.1
88.0
73.8
78.6
79.3
73.4
87.2
78.4
Calibration and failure prediction performance improve as model capacity scales. The compari- son of the performance of various models (Table 2) reveals trend: as we move from GPT-3, Vicuna, GPT-3.5 GPT-4, with the increase of model accuracy, there also noticeable decrease in ECE and increase in AUROC, e.g., approximate 22.2% improvement in AUROC from GPT-3 to GPT-4.
Vanilla verbalized confidence exhibits significant overconfidence and poor failure prediction, casting doubts on its reliability. Table 2 presents the performance of vanilla verbalized confidence across five models and eight tasks. According to the criteria given Srivastava al. (2023), GPT-3, GPT-3.5, and Vicuna exhibit notably high ECE values, e.g., the average ECE exceeding 0.377, suggesting that the verbalized confidence these LLMs are poorly calibrated. While GPT-4 displays lower ECE, its AUROC and AUPRC-Negattee scores remain suboptimal, with average AUROC of merely 7067cooo to the 50% random guess threshold hiid--hhihhihiinng challenges in distinguishing correct from incorrect predictions.
5.2 HUMAN-INSPIRED PROMPTING STRATEGIES PARTIALLY REDUCE OVERCONFIDENCE
Hlmman-inspired prompting strategies improve model accuracy and calibration, albeit with diminishing returns in advanced models like GPT-4. As illustrated in Figure 3, we compare the performance of five prompting strategies across five datasets on GPT-3.5 and GPT-4. Analyzing average BCE, AUROC, and their respective performances within each dataset, human-inspired strategies offer consistent improvements in accuracy and calibration over the vanilla baseline, with modest advancements in failure prediction.
No single prompting strategy consistently outperforms the others. Figure 3 suggests that there is no single strategy that can consistently outperform the others across all the datasets and models. By evaluating the average rank and performance enhancement for each method over five task types, find that Probing maintains the most consistent advantage over the baseline on GPT-4, while Top-K K emerges the top performer on GPT-3.5.
While ECE can be effectively reduced using suitable prompting strategies, failure prediction still remains challenge. Comparing the average calibration performance across datasets ('mean
7

---

Published as a conference paper at ICLR 2024
OPT:3.5: ECE
OPT-4: ECE
5.5 AUROC
GPT-4: AUROC
Figure 3: Comparative analysis of 5 prompting strategies over 5 datasets for 2 models (GPT-3.5 and GPT-4). The "average' bar represents the mean ECE for a given prompting strategy across datasets. The 'mean ECE' line is the average across all strategies and datasets. AUROC is calculated in similar manner. The accuracy comparison is shown in Appendix B.4.
ece' lines) and the average failure prediction performance ('mean auroc'), we find that while we can reduce ECE with the right prompting strategy, the model's failure prediction capability is still limited, i.e., close to the performance of random guess (AUROC=0.5). A closer look at individual dataset performances reveals that the proposed prompt strategies such as CoT have significantly increased the accuracy (see Table 8), while the confidence output distribution still remains at the range of 80% 100%, suggesting that reduction in overconfidence due to the diminished gap between average confidence and accuracy. not necessarily indicating substantial increase in the model's ability to judge the correctness of responses. For example, with the CoT prompting on the GSM8K dataset, GPTT-4 with 93.6% accuracy achieves near-optimal ECE 0.064 by assigning 100% confidence to all samples. However, since all samples receive the same confidence, it challenging to distinguish between correct and incorrect samples based on the verbalized confidence.
5.3 VARIANCE AMONG MULTIPLE RESPONSES IMPROVES FAILURE PREDICTION
Table 3: Comparison of sampling strategies with the number of responses M = 5 on GPT-3.5. The prompt and aggregation strategies are fixed as CoT and Consistency when M > 1. To compare the effect of M, we also provide the baseline with M from Figure 3. Metries are given by x x10
GSMBK
Prf-Law
DateUnd AUROC
StrategyQA AUROC
Biz-Erhics
Average AUROC
Method
ECE
AUROC
ECB
AUROC
ECE
ECE
ECE
AUROC
BCE
Misleading (M=5)
8.03
88.6
59.3
20.5
67.3
21.8
17.8
71.3
17.3
69.6
Self-Randon (M=5)
6.28
92.7
26.0
65.6
17.0
66.8
23.3
60.8
20.7
79.0
18.7
73.0
Prompt (M=5)
35.2
74.4
31.5
60.8
23.9
69.8
16.1
61.3
15.0
79.5
24.3
69.2
CoT (M=1)
10.1
54.8
39.7
52.2
23.4
57.4
22.0
59.8
30.0
56.0
25.0
56.4
Top-K (M=l)
19.6
58.5
16.7
58.9
26.1
74.2
14.0
61.3
12.4
73.3
17.8
65.2
Consistency among multiple responses is more effective in improving failure prediction and cali- bration compared verbalized confidence (M - 1), with particularly notable improvements on the arithmetic task. Table 3 demonstrates that the sampling strategy with 5 sampled responses paired with consistency aggregation consistently outperform verbalized confidence in calibration and failure prediction, particularly on arithmetic tasks, e.g., GSM8K showcases a remarkable improvement in AUROC from 54.8% (akin to random guessing) to 92.7%, effectively distinguishing between incorrect and correct answers. The average performance in the last two columns also indicates improved ECE and AUROC scores, suggesting that obtaining the variance among multiple responses can be good indicator of uncertainty.
As the number of sampled responses increases, model performance improves significantly and then converges. Figure 7 exhibits the performance of various number of sampled responses M from M = 1 to M = 13. The result suggests that the ECE and AUROC could be improved by sampling more responses, but the improvement becomes marginal the number gets larger. Additionally, as the computational time and resources required for M responses go linearly with the baseline (M=l), M thus presents a trade-off between efficiency and effectiveness. Detailed experiments investigating the impact of the number of responses can be found in Appendix B.6 and B.7.
5.4 INTRODUCING VERBALIZED CONFIDENCE INTO THE AGGREGATION OUTPERFORMS CONSISTENCY-ONIYY AGGREGATION
Pair-Rank achieves better performance in calibration while Avg-Conf boosts more in failure prediction. On the average scale, we find that Pair- emerges as the superior choice for calibration that can reduce ECE to as low as 0.028, while Avg-Conf stands out for its efficacy in failure prediction.
8

---

Published as a conference paper at ICLR 2024
Table 4: Performance comparison of aggregation strategies on GPT-4 using Top-K Prompt and Self-Random sampling. Pair-Rank aggregation achieves the lowest ECE in half of the datasets and maintains the lowest average ECE in calibration; Avg-Conf surpasses other methods in terms of AUROC in five out of the six datasets in failure prediction. Metrics are given by x10²
Metric
Aggregator
GSM8K
Law
Date
Sport
Strategy
Ethics
Mean & Var
Consistency
4.80
21.1
6.00
13.4
13.5
13.2
12.0 +.
ECE +
Avg-Conf
10.0
14.4
7.70
10.6
5.90
20.2
14.8 #07
Pair-Rant
7.40
15.3
8.50
2.80
3.50
3.80
6.90 +02
Consistency
84.4
66.2
68.9
60.3
65.4
56.3
66.9 1008
AUROC
Avg-Conf 41.0
68.0
72.7
64.8
70.5
84.4
66.9 111.
Pair-Rank 80.3
66.5
67.4
61.9
62.1
67.6
67.6 110.4
This observation agrees with the underlying principle that Pair-Rank learns the categorical distribution of potential answers through our K observations, which aligns well with the notion of calibration and is therefore more likely to lead a lower ECE. In contrast, Avg-Conf leverages the consistency, using verbalized confidence as a weighting factor for each answer. This approach is grounded in the observation that accurate samples often produce consistent outcomes, while incorrect ones yield various responses, leading to a low consistency. This assumption matches well with failure prediction, and is confirmed by the results in Table 4. addition, our comparative analysis of various aggregation strategies reveals that introducing verbalized confidence into the aggregation (e.g.. Pair-Rank and Avg-Confi is more effective compared to consistency-ons when LLM queries are costly, and we are limited in sampling frequency (set to M 5 queries in our experiment). Verbalized confidence, albeit imprecise, reflects the model's uncertainty tendency and can enhance results when combined with ensemble methods.
aggregation (e.g., Consistency), especially
DISCUSSIONS
In this study, we focus on confidence elicitation. i.e., empowering Large Language Models (LLMs) to accurately express the confidence in their responses. Recognizing the scarcity of existing literature on this topic, we define systematic framework with three components: prompting, sampling and aggregation to explore confidence elicitation algorithms and then benchmark these algorithms on two tasks across eight datasets and five models. Our findings reveal that LLMs tend to exhibit overconfidence when verbalizing their confidence. This overconfidence can be mitigated to some extent by using proposed prompting strategies such as CoT and Self- Probing. Furthermore, sampling strategies paired with specific aggregators datasets. We hope this work could serve as a foundation for future research in these directions.
improve failure prediction, especially in arithmetic
Comparative analysis of white-hox and black-hox methods. While our method is centered on black-box settings, comparing it with white-box methods helps us understand the progress in the field. We conducted comparisons on five datasets with three white-box methods (see $B.1) and observed that although white-box methods indeed perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. This finding underscores that the field remains challenging and unresolved.
Are current algorithms satisfactory? Not quite. Our findings (Table 4) reveals that while the best-performing algorithms can reduce ECE quite low value like 0.028, they still face challenges in predicting incorrect predictions, especially in those tasks requiring professional knowledge, such as professional law. This underscores the need for ongoing research confidence elicitation.
What is the recommendation for practitioners? Balancing between efficiency, simplicity, and effectiveness, and based on our empirical results, we recommend a stable-performing method for practitioners: Top-K prompt + Reff-Random sampling + Avg-Cont or Pair- aggregation. Please refer to Appendix for the reasoning and detailed discussions, including the considerations when using black-box confidence elicitation algorithms and why these methods fail in certain cases.
Limitations and Future Work: 1) Scope of Datasets. We mainly focuses on fixed-form and free- form question-answering QA tasks where the ground truth answer is unique, while leaving tasks such as summarization and open-ended QA to the future work. 2) Black-box Setting. Our findings indicate black-box approaches remain suboptimal, while the white-box setting, with its richer information access, may be a more promising avenue. Integrating black-box methods with limited white-box access data, such model logits provided by GPT-3, could be promising direction.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023).
REFERENCES
Kendrick Boyd, Kevin H. Eng. and C. David Page. Area under the precision-recall curve: Point estimates and confidence intervals. Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Zelezný (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 451-464 Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 77833-642-4099433
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 221220055.116.... 14165, 2020.
Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. A close look into the calibration of pre-trained language models. arXiv preprint trXiv:2211.0015A1 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez. lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
///msys.
2aelllgg/22220022 2023-03- 30 1-0---3vveeaall vicuuaa.....
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021.
Leda Cosmides and John Tooby. Are humans good intuitive statisticians after all? rethinking some conclusions from the literature judgment under uncertainty. cognition, 58(1):1-73, 1996.
Ailin Deng, Miao Xiong, and Bryan Hooi. Great models think alike: Improving model reliability via inter-model latent agreement. arXiv preprint Xiv:22305.0148) 2023.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation:
Representing model
uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Paul H Garthwaite, Joseph B Kadane, and Anthony O'Hagan. Statistical methods for eliciting probability distributions. Journal of the American statistical Association, 100(470)6800-01,, 2005a.
Paul H Garthwaite, Joseph B Kadane, and Anthony O'Hagan. Statistical methods for eliciting probability distributions. Journal of the American statistical Association, 100(470)6600-01, 2005b.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, al. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.0333 03342,, 2021.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies, 2021.
Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 ACM SIGMOD international conference on Management of data, pp. 1197-1208, 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
10

---

Published as a conference paper at ICLR 2024
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How know when language models know? on the calibration of language models for question answering. Transactions of the 2021.
Association for Computational Linguistics,
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran roan-Jonsoo,, et al. Language models (mostly) know what they know, arXiv preprint ar arXiv:2207.00 005221, 2022.
Ethan Kim. Sports understanding in bigbench, 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205. 11916, 2022. URL https: //aps:
org wwg/CorpusIDD : 249017743.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 2023.
Volodymyr Kuleshov and Shachi Deshpande. Calibrated and sharp uncertainties deep learning via density estimation. In International Conference on Machine Learning, pp. 11683-11693, PMLR, 2022.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference machine learning, pp. 2796-2806 PMLR, 2018.
Balaji Lakshminarayanann, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. /4334, 2022.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07655 2020.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, (0:857-872) 2022.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems, volume 34, pp. 15682-15694, 2021.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
OpenAI. ChatGPT. https:t///// //tp:://wwww.. wwwnww.penii..... httpeeaaa.com/gttttoaae 2022. Accessed: April 21, 2023.
OpenAl. Gpt-4 technical report, 2023.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2090-2094, Online, June 2021, Association for Computational Linguistics. doi: 10.18653/v1/2021 168. URL https: www.acooonotoooo.ooooooooooooooe org /0021 nwacl -main. 168.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanal and Peter Liu. cut-of-distribution detection and selective generation for conditional language models. arXiv preprint 2009 /5558, 2022.
11

---

Published as a conference paper at ICLR 2024
Quintin P. Solano, Laura Hayward. Zoey Chopra, Kathryn Quanstrom, Daniel Kendrick, Ken- neth L. Abbott, Marcus Kunzmann, Samantha Ahle, Mary Schuller, Erkin Otley, and Brian C. George. Natural language processing and assessment of resident feedback quality. Journal of Surgical Education, 78(6)=e72-777, 2021. ISSN 1931-7204. doi: hups://doi.org/10/ 1016/ j.jsurg.2021 2021.05.12..... URL https: www. sC1 enced rect. com http://eeenneeeeceeeeeeee p11/$19317200210011
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023, ISSN 2835-8856. URL https: /twp///eeee...ettt...eeee forum? dduuyTLBBoosss
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305. /4975, 2023.
Christian Tomani and Florian Buettner. Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9896,9996,70 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXih:2307.09288 2023b.
Jianfeng Wang, Rong Xiao, Yandong Guo, and Lei Zhang. Learning to count objects with few exemplar annotations. arXiv preprint / 1905.07899 2019.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy KaiWei Lee, and Ec-Peng Lim. Plan-and-solve prompting: Improving zero-shot thain-of-though reasoning by large language models. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https: www.seeentiiee sesmanticscholaa org/ CorpusID 258558102.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 1111
Xinyi Wu and Zijian Wang. Data understanding in bigbench, 2021.
Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in conditional language generation. arXiv preprint arXiv:2103. 2021.
Miao Xiong. Shen Li, Wenjie Feng, Ailin Deng, Jihai Zhang. and Bryan Hooi. Birds of feather trust together: Knowing when to trust a classifier via adaptive neighborhood aggregation. arXiv preprint ar 2rx: 1.2211114000 16466 2022.
Miao Xiong, Ailin Deng, Pang Wei Koh, Jiaying Wu, Shen Li, Jianqing Xu, and Bryan Hooi. Proximity-infon informedd calibration for deep neural networks. arXiv preprint arXiv:2306.04590 2023.
Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 1040-3049, 2021.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pp. 609-616, 2001.
Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning, pp. 11117-11129 128. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2024
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. arXiv preprint arXiv:2302.13339 2023.
13

---

Published as a conference paper at ICLR 2024
A PROOF OF PROPOSITION 3.1
Notation.
Given a question with N candidate responses, the i-th response consists of K sequentially ordered answers, denoted as K ( = (s S.ai ...... ......k)) Let A = S2. Srr represent the
set of unique answers across all N responses, where M the total number of distinct answers. The (8) event where the model ranks answer S,, above S, in its i-th generation is represented as (Sa > S.). In contexts where the generation implicit, this is simply denoted as (S > S.). Let F.l be the event where at least one of S. and Sy appears in the i-th generation. The probability of (Sw > S.), given E and a categorical distribution P. is expressed as P(S,, > SelP, Eli)).
Proposition A.1. Suppose the Top-K answers are drawn from categorical distribution P without replacement. Define the event (S,, > S.) to indicate that the realization S,, is observed before S, in the i-th draw without replacement. Under this setting, the conditional probability is given by:
P (S... 1
P(S.)
P(S.) P(S.)
The optimization objective minimize the expected loss is then:
min -E E= SAE.S.E
Su Sp log P(S.) P(S.)+P(SS)
P(S.)
S.1. Σ ((S)) =1 (5)
SEEA
Proof. Let begin by examining the position j in the response sequence S where either S. or S. first sampled, and the other has not yet been sampled. We denote this event as F))) (Su. So), and for simplicity, refer to it as Fj:
Fj 5 (S., S.) the earliest position in
((i) SK where either S. or S. appears is 1
(6)
Vm, E (,,2,.... N) s = Sa, 5(1) S min(m. m)
Given this event, the probability that Su is sampled before S, across all possible positions j is:
N
IP(S > S. P, F()
Σ Ø=1
P(B P, E(i) x IP(S. > S, P. P. FJ)
(a)
To further elucidate (1), which is conditioned on Fj. we note that the first sampled answer between Su and S, appears position j. We then consider all potential answers sampled prior to j. For this, we introduce a permutation set Hi-1 to encapsulate all feasible combinations of answers for the initial j samplings. representative sampling sequence is given by: S3-1 (S > S(2) > S VI E (1.2...., 1}, S(1) E AA) S.)).
Consequently, (a) can be articulated as:
P(S. > S. P.
E
No F2)
Σ
P(S P. E FJ) x F(S > S. P, P(D E S Fj)
1
(b)
(8)
Consider the term (b). which signifies the probability that, given the first j samplings and the restriction that the j-th sampling can only be S, or Sy, S, is sampled prior to S,. This probability is articulated as:
F(S) ( = S. P. (0) Sz1)
P(S, S. P. F() Sy1)
P(S Sn P. F(i) F5, + P(S = S. E(i) F5,S)-1)
P(S.)
T-Esme Emmm6,-1 P(S...
P(S+)
P(Su)
P(S..) 11 Sm P(S...
5mE55-1
P(S.)
P(S.) + P(S.)
14

---

Published as a conference paper at ICLR 2024
Integrating equation (9) into equation (8), obtain:
P(S.)
P(S. > S. I. F3)=
Σ
P(S 1., x
P(S.) + P(S.)
-
P(S.)
P(S.) + P(S.) x
Σ
P(S) P. P,
(10)
5 IEH,
P(S.)
P(S.) + P(S.)
Subsequently, incorporating equation (10) into equation (7), we deduce:
P(S.)
P(S. > Sy (.E
Σ P(F, P. E x P(S.) + P(Se)
P(S.)
K
P(F, P. )
(11)
x
P(S.) + P(S.)
j=1
P(S.)
P(S.) + P(S.)
The derivations in (c) and (d) employ the Law of Total Probability.
Incorporating Equation 11 into Equation 3, the minimization objective is formulated as:
N
P(S.) + P(S.)+((SS)
min
Σ
T(Su S.} x log
s.t. Σ
P(S.) =1 (12)
1=1 S.E.SA.AA
S.EA
15

---

Published as a conference paper at ICLR 2024
B DETAILED EXPERIMENT RESULTS
B.1 WHITE-BOX METHODS OUTPERFORM BLACK- BOX METHODS, BUT THE GAP IS NARROW.
Comparative Analysis of White-Box and Black-Box Methods: Which performs better - white- box or black-box methods? Do white-box methods, with their access to more internal information, outperform their black-box counterparts? If so, how large is the performance gap? To address these questions, we conduct a comparative analysis of white-box methods based on token probability against black-box models utiliizig verbalized confidence.
Implementation details: We utilize the probabilities of each output token to develop three token- probability- based white-box methods: 1) Sequence Probability (seq-prob), which aggregates the probabilities of all tokens; 2) Length-Normalized Sequence Probability (len-norm-prob), which normalizes the sequence probability based on sequence length, i.e., seq-prob I/lnnghh 3) Key Token Probability (token-prob), designed to focus on the result-specific tokens, e.g., "35" in the output sequence "Explanation: ....; Answer: 35; ...", thereby minimizing the influence of irrelevant output tokens. For our implementation, we use the Chain-of-TT Chain-of-Thought and Top-K Verbalized Confidence prompt to acquire verbalized confidence and select GPT3 as the backbone model.
Findings: Our comparative analysis, detailed in Table 5 and Table 6, yields several key insights: I) Generally, white-box methods exhibit better performance, with length-normalize sequence prob- ability and key token probability emerging the most effective methods across five datasets and four evaluation metrics. 2) The gap between white-box and black-box methods is relatively modest. Moreover, the best- test-eerfrrmng white-box methods fall short of achieving satisfactory results. This is particularly apparent in the AUROC metric, where the performance of nearly all methods across various datasets ranges between 0.5-0.6, signifying a limited capability in distinguishing between correct and incorrect responses. 3) These experimental results suggest that uncertainty estimation in LLMS remains challenging and unresolved issue. As mentioned in our introduc- tion, the logit-based methods, which predominantly capture the model's 's uncertainty regarding the next token, are less effective capturing the semantic uncertainty inherent in their textual meanings. Although several alternative approaches like semantic uncertainty (Kuhn et al., 2023) have been proposed, they come with significant computational demands. This scenario underscores the need for future research on both white-box and black-box methods to discover more efficient and effective methods for uncertainty estimation in LLMs.
B.2 How MUCH DOES THE ROLE-PLAY PROMPT AFFECT THE PERFORMANCE?
To explore how the verbalized confidence elicitation performance varies when LLMs are asked to play different personalities such "confident" and "cautious", we conduct the experiment in Figure and in Table 7. The results are derived when adding "You are confident GPT" (Left) and "You are cautious GPT" (Right) to the beginning of the Chain of Thought (CoT) prompt (Table 15). The experimental results show that the difference between their confidence distribution seems minimal, suggesting that assuming different personalities does not significantly affect performance metrics such as accuracy, ECF, and AUROC.
HOW IS THE DISTRIBUTION OF VANILLA VERBALIZED CONFIDENCE ACROSS MODELS B.3
AND DATASETS?
Figure presents the empirical distribution of vanilla verbalized confidence across models and datasets. Notably, all the models output confidence as the multiples of with most values ranging between the 80% to 100% range. This behavior resembles the patterns identified in the training corpus for GPT-like models as discussed by Zhou et al. (2023). Such behavior suggests that models might be imitating human expressions when verbalizing confidence.
B.4 DETAILED PERFORMANCE DIFFERENT PROMPTING STRATEGIES
Multi-step and Top-K prompting strategies demonstrate promising results in reducing ECE and improving AUROC, with Top-K being relatively more effective. Figure 6 presents a comparison of various prompting strategies (CoT, Multi-Step, Top-K) against vanilla verbalized confidence.
16

---

Published as a conference paper at ICLR 2024
Table 5: Performance comparison (metrics are given by x10²) of token- siieen-robabblity-beee white- box methods including the baseline sequence probability ("seq-prob"), length-normalizer sequence
probability ("len-norm-probb) and key token probability ("token-prob"), and black-box verbalized
confidence ("Verbalized"" GPT-3 using Top-K Prompt.
Dataset
Acc
Method
ECE
AUROC
AUPRC-P
AUPRC-N
StrategyQA
59.90
Verbalized
39.04
50.34
60.06
40.27
seq-prob
7.14
55.50
62.99
45.22
len-norm-prob
37.65
55.50
62.99
45.22
token-prob
32.43
60.61
69.90
47.10
Biz-Ethics
61.00
Verbalized
18.20
66.27
71.95
50.59
seq-prob
48.49
62.30
71.07
52.23
len-norm-prob
33.70
62.30
71.07
52.23
token-prob
27.65
67.00
74.89
55.01
GSM8K
11.52
Verbalized
77.40
54.05
12.70
89.01
seq-prob
7.73
69.80
20.40
94.71
len-norm-prob
72.41
70.61
21.23
94.75
token-prob
35.60
69.29
20.63
94.27
DateUND
15.72
Verbalized
83.47
50.80
15.93
84.54
seq seq-prob
16.10
62.93
22.39
90.61
len-norm-prob
81.27
62.93
22.39
90.61
token-prob
74.19
54.25
19.28
83.85
Prf-Law
44.92
Verbalized
41.55
49.54
44.43
55.78
seq-prob
32.31
51.07
45.75
56.70
len-norm-prob
49.66
51.06
45.75
56.79
token-prob
43.26
61.24
53.84
64.69
ACC 0.71/ / AUROC 0.57 ECE 0.27
ACC 0.70 AUROC 0.59
ECE 0.28
wrong answer
wrong answer
correct answer
correct answer
Count
Count
Confidence (%)
Confidence (%)
(a) "You confident GPT"
(b) "You are cautious GPT"
Figure 4: Distribution of the verbalized confidence with different specified role descriptions in prompts. The results are derived when adding "You are a confident GPT" (Left) and "You are a cautious GPT" (Right) to the beginning of the Chain of Thought (CoT) prompt (Table 15). All other aspects of the prompts remain identical to the standard CoT format.
The detailed performance of CoT, Multi-Step, and Top-K prompt can be found in Table 8, Table and Table 10, respectively. Judging from the 'average" bar, which computes the mean value across five datasets, both Multi-step and Top-K prompting strategies effectively reduce ECE and enhance AUROC. Moreover, Top-K shows relatively better performance improvements. The intuition behind this improvement is that this prompting strategy, requesting the model to generate multiple guesses along with their corresponding confidences, naturally nudges the model to be aware of the existence of
17

---

Published as a conference paper at ICLR 2024
Table 6: Performance comparison (metrics are given by x10²) of token- siieen-robabblity-Beee white- box methods including the baseline sequence probability ("seq-prob"), length-normalizer sequence probability ("len-norm-probb) and key token probability ("token-prob"), and black-box verbalized confidence ("Verbalized"" GPT-3 using CoT Prompt.
Dataset
Acc
Method
ECE
AUROC
AUPRC-P
AUPRC-N
DateUND
62.33
Verbalized
37.40
50.36
62.50
38.12
seq-prob
62.30
56.37
65.14
43.21
len-norm-prob
15.78
58.70
66.57
47.24
token-prob
27.32
40.27
55.20
35.69
StrategyQu QA
67.57
Verbalized
29.74
51.37
68.16
34.54
seq-prob
67.56
52.04
69.58
33.48
len-norm-prob
6.79
52.11
70.41
33.43
token-prob
30.59
53.00
68.80
36.89
Biz-Ethies
59.00
Verbalized
40.90
49.15
58.59
41.00
seq-prob
26.50
58.99
64.30
47.45
len-norm-prob
39.43
58.99
64.30
47.45
token-prob
36.31
67.38
75.33
54.89
GSM8K
52.31
Verbalized
47.49
50.32
52.47
48.02
seq-prob
52.30
57.47
56.75
54.39
len-norm-prob
29.80
57.92
58.84
55.23
token-prob
44.94
58.44
57.54
60.43
Prf-Law
44.85
Verbalized
53.43
50.13
44.90
55.91
seq-prob
44.85
51.88
46.62
56.09
len-norm-prob
31.00
50.10
45.34
55.32
token-prob
51.75
57.83
50.53
62.52
Role
Model
ACC
ECE
AUROC
AUPRC-P
AUPRC-N
Confident
chatgpt-0613
0.7103
0.2741
0.5679
0.7398
0.3635
Cautious
chatgpt-0613
0.6983
0.2812
0.5946
0.7415
0.4009
Table 7: Performance Comparison of Verbalized Confidence Elicitation with two types of prompt: "You are a confident GPT" and "You are a cautious GPT". The difference between these two prompts seems minimal, suggesting that asking LLMs to take on different personae does not significantly affect the performance.
various possible answers, preventing overconfidence in single response and promoting re-evaluation of given answers.
B.5 TOP-K VERBALIZED CONFIDENCE PERFORMANCE
The detailed experiments performance of Top-K verbalized confidence can be found in Table 10.
B.6 IMPACT OF MISLEADING PROMPTS IN MISLEADING SAMPLING STRATEGY
To study the effect of different misleading prompts used in misleading sampling strategy, we categorize the prompts into three types: Weak Claim, Strong Claim, and External Source. The specific prompts are shown in Table 11. The Weak Claim category represents prompts that exhibit uncertainty, such as "I vaguely remember the answer is" or "I think the answer should be". These prompts suggest that the user providing misleading information to the Large Language Model (LLM) but lacks confidence in the information provided. The Strong Claim category includes prompts like am pretty sure that this is" "I am very confident that". which express high degree of certainty. The External Source
18

---

Published as a conference paper at ICLR 2024
GPT3
GPT3.5
GPT4
Vicuna
MIXEO MOUNE
AIRIO KROP
ACC 0.20 ALBOC
ACC ABROC 940 0-18
ACC 0:2 NUBOC /OC.C////CCCC/C E.B
ACC E41 ASROC 5:41 ECT
DateUND
conrect
StrategyCJA
60
106
90
70
90
186
60
Carkieres
83
180
50
70
160
wran Mone
ACE OUB AUROE 6:01
ACC O:77 ABROC 8:3 /.00/EEE 0.22
ACC 0:1 AUBOC
ACC 6:34 ABROC D 0J9
Biz-Ethics
Figure 5: Empirical distribution of vanilla verbalized confidence across 4 models and 5 datasets. The prompt used is in Table 14. From this figure, we can observe that 1) the confidence levels primarily range between 80% and 100%, often in multiples of 5; 2) a large portion of incorrect predictions (red) has been observed even in the 100% confidence bar, indicating significant overconfidence.
ECE
AUROC
-10
AUROC DIff
ECE Diff
-20
-40
-5
-50
-10-
CoT
Multi-Step
Top-K
CoT
Multi-Step
Top-K
Figure 6: Performance Comparison of four verhalized confidence methods: vanilla, CoT, Multi-Step. Top-K in terms of ECE and AUROC for five types of datasets on GPT-3.5. Refer to Table 10 for detailed results.
category represents prompts that cite external sources as their evidence, such "Wikipedia says" or "the latest research shows that".
Our experimental results (Table 11) indicate that the Weak Claim category performs better. A possible explanation is that on one hand even providing weak misleading information, the model will analyze and reassess their answers. On the other hand, since the misleading answers generated randomly, confidently providing this information can sometimes lead to negative effects. For example, the model provides correct answer with moderate confidence. However, if misleading hint is provided
19

---

Published as a conference paper at ICLR 2024
Table 8: Improvement verbalized confidence with Shain-of-Thouge Prompts
GPT3.5
Dataset
CoT
ACC(%)
ECE
AUROC
28
66
65
GSM8K
80.3
10
55
47
48
65
DateUnd
73.2
23
57
65.8
26
53
StrategyQA
67.9
22
60
45.5
44
50
Prf-Law
51.7
37
49
67
23
55
Biz-Ethics
61
30
56
Table 9: Evaluation multistep verbalized confidence for GPT-3.5
Models
GPT3.5
Dataset
SA
ACC(%)
ECE
AUROC
80.3
10
GSM8K
V
76.2
22
60
73.2
23
57
DateUnd
V
63.6
26
72
67.9
22
60
StrategyQA
V
68.7
17
59
X
51.7
37
49
Prf-Law
/
49.6
27
49
X
61
30
56
Biz-Ethics
61.6
with high confidence is supported by external source, the model may be inclined believe the prompt and alter its predictions.
B.7 IMPACT OF THE NUMBER OF CANDIDATE ANSWERS
We investigate the impact of the number of candidate answers, denoted as K, utilized in the sampling strategy. Specifically, K represents the number of queries used to construct the set of candidate answers for consistency caleulation. We illustrate its calibration performance (ECE) and failure prediction performance (AUROC) in relation to varying numbers of K (ranging from K = 1 to K 13) in Figure
The results indicate that, in terms of AUROC, higher candidate set size K contributes to superior performance and reduced variance. However, the optimal candidate size for ECE varies across different datasets. For instance, the StrategyQA dataset exhibits improved performance with larger K, whereas the Business Ethics dataset generally performs better with a moderate number of candidate answers (e.g., K 4). This observation can be attributed to the limited variability of misleading information (restricted to types) used in our experiments for the Business Ethics dataset, implying that the introduction of large number of more queries does not significantly enhance the information pool. Therefore, to strike a balance between computational efficiency and performance, set the candidate set to be in study.
20

---

Published as a conference paper at ICLR 2024
Table 10: Evaluation of Top-K verbalized confidence on GPT-3.5.
GPT3.5
Dataset
ACC(%)
ECE
AUROC
GSM8K
22.8
19.6
58.5
DateUnd
33.3
26.1
74.2
StrategyQA 61.3
14
61.3
Prf-Law 42.2
16.7
58.9
Biz-Ethics
67.0
12.4
73.3
Table 11: Different Prompts used for misleading sampling strategy.
Group
Consistency Confidence Mislcading Prompts
Weak Claim
Hint: I think the answer should be
Weak Claim
Hint: I read online the answer is
Weak Claim
Hint: I vaguely remember the answer
Externel Source
Hint: My teacher tells me the answer is
Externel Source
Hint: The textbook shows that answer is
Externel Source
Hint: An expert says that the answer
Externel Source
Hint: Latest research shows that the answer
Externel Source
Hint: Wikipedia claims that the answer is
Strong Claim
Hint: I am 80% sure the answer is
Strong Claim
Hint: I am pretty sure the answer is
Strong Claim
Hint: Undoubtedly the answer
Strong Claim
Hint: With my 100% confidence the answer is
B.8 PERFORMANCE OF DIFFERENT CONFIDENCE ELICITATION METHODS
C RELATED WORKS
Confidence Elicitation in LLMs. Confidence elicitation refers the process of estimating LLM's confidence in their responses, without relying on model fine-tuning or accessing the proprietary information of LLMs. Within this scope, Lin al. (2022) proposes the concept of verbalized confidence that elicits the model to output confidence directly. However, the evaluation is tailored
Misleading: ECE
Minlesding: AUROC
Misleading Varbalized: ECE
Misleading Verbelized: AUROC
8.750
g
0.23
0.175
0.725
8,700
0.20
0.75
0.79
8,450
2.425
0.15
0.18
0,000
0.575
0.10
0,69
10
28
12
6
10
12
Figure 7: Impact of the number responses responses on GPT-3.5. The sampling strategy is fixed as misleading. For every given number of misleading hints, randomly sample the specified number of queries for 5 times and calculate the mean ECE and AUROC, and compute its variance(plotted as error bar). Note that the number of hints plus 1 is the number of responses sampled during experiment.
21

---

Published as a conference paper at ICLR 2024
Table 12:
The performance of varying prompt groups in StrategyQA on GPT-3.5. The group exhibiting the optimal performance is emphasized in bold. The experimental results indicate that the Weak Claim category performs better.
GPT-3.5
Method
Hint Group
ECE
AUROC
Induced Consistency Confidence
Weak Claim
19.7
62.0
Strong Claim
19.5
61.4
External Source
18.2
60.8
cerbsllieed-oonsisteneyy confidence
Weak Claim
19.8
65.4
Strong Claim
19.5
64.6
External Source
18.2
63.4
for pretrained language models that are fine-tuned on specific datasets, and its zero-shot verbalized confidence remains unexplored. Mielke al. (2022) proposes train an external calibrator while relies on model representations that are not readily accessible. Zhou et al. (2023) examine the impact of confidence in prompts but does not directly provide confidence to users. Our work aligns most closely with the concurrent study by Tian et al. (2023), which also focuses on the use of prompting strategies. However, our approach diverges by aiming to explore a broader method space, introducing unified framework consisting of three components and conducting a systematic evaluation of strategies within each. The Top-K method, as proposed (Tian al., 2023), serves as an instance within our framework, and its performance can be augmented when integrated with other strategies from our framework. Furthermore, our investigation extends beyond the RLHF-LMs primarily analyzed in the concurrent study, and encompasses a broader spectrum of models. This allows us to probe the implications of different model sizes and structures. Our findings also underscore that all existing methods still face challenges with more complex tasks, contributing to more holistic understanding of confidence elicitation in the field.
Calibration. Modern neural networks shown be poorly calibrated, often manifesting overconfi- dence (Guo et al., 2017; Minderer et al., 2021; Xiong et al., 2023). Calibration seeks to address the issue by aligning the model's confidence with the accuracy of samples within the same confidence level (Guo et al., 2017; Minderer et al., 2021). To achieve this, a variety of methods have been proposed, which can be broadly divided into scaling-base methods (Guo al., 2017; Deng et al., 2023; Zhang et al., 2020) and binning- based methods (Zadrozny & Elkan, 2001; Zhang et al., 2020). Within the scope of LLMs, Jiang et al. (2021) investigates the calibration of generative language models (T5, BART, and GPT-2) and discovers that these models' probabilities on question-answering tasks are poorly calibrated. Similarly, Chen et al. (2022) finds that PLMs are not well calibrated and pretraining improves model calibration. On the other hand, Kadavath et al. (2022) studies the calibration of LLMs (parameter size ranging 800M to 50B). finding that larger models appear to be well-calibrated on multiple choice and true/false questions when provided in the right format. How- ever, these evaluations mainly focus on the probabilities derived from logits, which are unavailable for closed-source LLMs like GPT-4. This also motivates to study confidence elicitation methods that do not require model fine-tuning or access to model logits or embeddings.
D BEST PRACTICE AND RECOMMENDATIONS FOR PRACTITIONERS
D.1 WHAT IS THE RECOMMENDATION FOR PRACTITIONERS?
Balancing between efficiency, simplicity, and effectiveness, we recommend a stable-performing method from our empirical results as advice for practitioners: Top-K prompt + Self-Random sampling + Avg-Conf or Pair-Rank aggregation. The recommendation is based on: 1) Top-K outperforms all other methods on GPT-3.5 and is comparable to the oor-erfoninng method Self- Probing on GPT4 Compared to Self- Seff-Probing which requires two inference phases, the Top- K prompt is chosen for the balance between effectiveness and efficiency. 1) As shown in Sec 5.3, ensemble
22

---

Published as a conference paper at ICLR 2024
Table 13: Performance of different confidence elicitation methods: verbalize-basee (Top-K and CoT Verbalized Confidence), consistency-basee based hybrid combinations. The best-performing method for each dataset highlighted in bold.
and Induced consistency), and their
Metric
Method
GSM8K
DateUND
StrategyQA
Prf-Law
Biz-Ethics
Avg
Top-K (M=l)
39.8
40.1
14.0
16.7
12.4
24.6
COT' (M=1)
10.1
23.4
22.0
39.7
30.0
25.0
ECE
RanRoonnnCnnnsseeeeyy (M=5)
6.28
17.0
23.3
26.0
20.7
18.7
Misleading Cons (M=5)
8.03
20.5
21.8
18.3
17.8
17.3
Self-Random Avg-Conf (M=5)
9.28
14.6
15.9
18.3
15.8
14.8
Mislcading Avg-Conf (M=5)
7.40
17.6
15.0
12.8
18.2
14.2
Top-K (M=l) Col' (M=1)
76.3
61.3
58.9
73.3
65.9
54.8
57.4
59.8
52.2
56.0
56.4
ROC1
Sell-Random-Consistenn (M=5)
66.8
60.8
79.0
Misleading Cons (M=5)
88.6
67.3
61.5
59.3
71.3
69.6
ScIf-Random Avg-Conf (M=5)
92.5
68.8
66.2
79.5
Mislcading Avg-Conf (M=5)
88.8
63.8
65.6
60.4
72.4
70.2
Top-K (M=l) CoT (M=l)
27.7
62.8
68.4
49.3
82.2
58.1
76.6
72.8
49.2
64.3
68.9
PR-P 1
Self-Kandom-Conisteee (M=5)
96.9
81.0
73.7
59.4
82.3
78.7
Mislcading Cons (M=5)
95.1
81.0
74.1
54.7
77.6
76.5
Self-Random Avg-Conf (M=5)
97.0
84.4
78.3
60.3
83.1
80.6
Misleading + Avg-Conf (M=5)
95.3
79.0
79.1
56.4
80.9
78.1
Top-K (M=1)
80.2
79.8
45.7
56.0
50.7
62.5
CoT' (M=1)
23.1
30.7
40.5
53.9
43.7
38.4
PR-N1
Self-Random-Consitteee (M=5)
79.7
44.6
39.5
63.8
63.4
58,2
Mislcading + Cons (M=5)
71.2
44.2
41.3
58.7
55.1
54.1
Self Self-Random Avg-Conf (M=5)
81.5
51.8
45.8
65.3
64.9
61.9
Misleading (M=S)
(M=5)
73.5
42.4
45.4
60.9
57.1
55.9
methods (e.g., M 5) are consistently more effective than verbalized confidence (M 1) in eliciting a model's confidence. Regarding the sampling strategies, Self-Random selected for being more straightforward and commonly used, since the performance difference of different sampling strategies is minimal.. 3) For aggregation, strategies based on both answers and verbalized confidences (e.g.. Avg-Conf and Pair-Rank) outperform #aggregation based on answers only (e.g., consistency) Then we recommend Pair-Rant and Avg-Conf for different downstream tasks according to their relatively good performance on different metrics. For example, for tasks that prioritize the exact confidence values, like calculating expected risk, Pair-Rank is recommended, while Avg-Conf is better suited for tasks related to failure prediction, e.g., factual error detection. Additionally, noteworthy that using Top-K alone does not improve accuracy as much as Chain of Thought (CoT), but the use of ensemble methods compensates for this.
D.2 WHAT ARE CONSIDERATIONS WHEN USING BLACK-BOX CONFIDENCE ELICITATION ALGORITHMS?
Careful consideration is necessary due significant limitations: 1) The reliability of the given confidence must be assessed by considering multiple metrics, such as both ECE and AUROC. As discussed in section 5.2, a high ECE does not imply that the model's outputs accurately represent model correctness. Metrics including AUROC and detailed information such the confidence distribution plot should also be considered
a comprehensive evaluation and better understanding.
2) LLMs are not explicitly modeled to express uncertainty in textual outputs, and descriptions of uncertainty in the training corpus are mostly human expressions, which are often considered inaccurate (Garthwaite al., 2005b). Dependence such confidence for real-world applications requires careful checking, especially given the consistently high confidence levels shown in Figure 2, matter whether the question is correctly answered not.
23

---

Published as a conference paper at ICLR 2024
D.3 DISCUSSIONS ON WHY SOME STRATEGIES WORK, AND WHY SOME DO NOT WORK
In this section, we discuss the effective strategies and analyze the rationale behind these mechanisms.
Sampling Consistency among multiple responses is more effective compared verbalized con- fidence (M = 1), with particularly notable improvements on the arithmetic task. This is be- cause sampling more queries allows us directly approximate the model's internal distribution, moode (xe (x:tt1),, which is trained to mirror the ground truth data distribution. Issues making this method ineffective can be: 1) the model's poor calibration (Kuhn et al., 2023), i.c., Pmodet X1:t-1) does not align well with Paata ( (x|(X1:t-1): or 2) the computational constraints limiting the number of sampled queries, leading to inaccurate estimates.
Aggregation Aggregation based on answers and verbalized confidences (c.g., g., Avg-Conf and Pair- Rank) outperforms aggregation based answers only (e.g., consistency), especially when LLM queries are costly and the number of queries we can sample is constrained. This is due to the coarse granularity of the consistency-baset aggregation's output-limited to 6 possible values (0, 0.2, 0.4, 0.6, 0.8, 1) when M=5. This can lead to poor calibration performance. The verbalized confidence, despite being less precise. still captures the model's uncertainty tendency and allows for finer-grained output values, and hence can be combined to enhance calibration performance.
Verbalized Confidence For verbalized confidence, note that humans are able to verbalize their uncertainty, c.g., giving insight as to whether our answers and reasonings are correct or not. So is reasonable to expect LLMs to have also learned this ability, to learn it at some point in the future. The current suboptimal performance of verbalized confidence points to an important research gap, and this might be explained by the inherent inaccuracy of the training data, particularly human expressions of uncertainty. For example, as studied by Garthwaite et al. (2005a), humans sometimes tend to exaggerate their priori probability for event that has occurred.
Prompting Strategy In addition, compared to Vanilla prompt, Top-K, CoT, and Multi-Step can significantly reduce ECE in ChatGPT. We argue that the improvement is largely due to these prompt strategies enhancing the model's ' accuracy, which narrows the gap between average confidence and actual accuracy, rather than a significant boost in their ability to differentiate between correct and incorrect samples. This is also supported by the modest gains in AUROC and AUPRC, compared to the significant improvement in BCE.
E EXPERIMENT SETUP
E. DATASETS
To evaluate the quality of confidence estimates in varied tasks, we select the tasks of commonsense reasoning, arithmetic calculation, symbolic reasoning, professional knowledge, and ethical knowledge as evaluation benchmarks. In detail, the datasets for each task are listed below:
Commonsense Reasoning: Sports Understanding (SportUND) dataset (Kim, 2021) and Strate- gyQA dataset (Geva et al., 2021) from BigBench (Ghazal et al., 2013). We select StrategyQA as the more representative dataset since it contains more data.
Arithmetic Reasoning: Graduate School Math (GSM8K) dataset (Cobbe et al., 2021) and Simple Variations on Arithmetic Math word Problems (SVAMP) dataset (Patel et al., 2021). We select GSM9K as the more representative dataset because it has a wider usage.
Symbolic Reasoning: Date Understanding UnateUnd) dataset (Wu & Wang, 2021) and Object Counting (ObjectCou) dataset (Wang et al., 2019) in BigBench. We select Date Understanding as the more representative dataset since it is more difficult than Object Counting.
Professional Knowledge: Professional Law (Prf-Law) dataset from MMLU (Massive Multitask
Language Understanding) (Hendrycks et al., 2021)
Ethical Knowledge: business ethics (Biz-Ethics) dataset from MMLU (Hendrycks et al., 2021).
24

---

Published as a conference paper at ICLR 2024
E.2 EVALUATION METRICS
In line with previous evaluation setting in (Naeini et al., 2015; Yuan et al., 2021; Xiong et al., 2022), use confidence calibration and failure prediction metrics measure estimated confidence:
Expected Calibration Error (ECE): It measures the calibration of a classifier by quantifying the discrepancy between predicted probabilities and observed accuracy.
Area Under the Receiver Operating Characteristic curve (AUROC): It assesses the discriminative ability of a classifier across different classification thresholds (Boyd et al., 2013).
Area under the Precision- Recall Curve (AUPRC): measures the trade-off between precision and recall at different classification thresholds. Specifically, AUPRC-Positin measures the AUPRC for positive instances and AUPRC-Negatton
for negative samples.
Specifically, calibration metrics (ECE) measure the alignment of confidence scores with the ground truth uncertainty. enabling their utilization in tasks such as risk assessment; while failure detection (AUROC and AUPOR) metrics measure whether the confidence score can appropriately differentiate correct answers and incorrect answers. These metrics also play crucial role in accurately assessing calibration measurements works such as Mielke al. (2022) and Solano et al. (2021).
E,3 MODELS
In experiments, incorporate range of representative LLMs of different scales, including Vicuna (Chiang et al., 2023), GPT3 (Brown et al., 2020), GPT3.5 (GPT3.5) (OpenAI, 2021), and GPT4 (OpenAI, 2023). The number of parameters in each model is 13 billion for Vieuna, 175 billion for GPT3, and larger for GPT3.5 and GPT4. While GPT3.5 and GPT4 have been widely acknowledged due their outstanding performances, GPT3 is selected as former version of them. Vicuna is a smaller model fine-tuned from LLaMA (Touvron et al., 2023a).
Example: Open-Number Question
Read the question, provide your and your confidence in this answer. Note: The confidence indicates how likely you think your answer true.
Use the following format to answer:
Answer and Confidence (0-100): [ONLY the number; not complete sentence), (Your confidence level, please only include the numerical number the range of 0-100]]%*"
Only the answer and confidence, don't give me the explanation
Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
Now, please answer this question and provide your confidence level.
Answer and Confidence: 3, 85%
Figure 8: Example of complete prompt and the model's output. The vanilla prompt used.
E.4 IMPLEMENTATION DETAILS
For the use of sampling strategy, sample M responses. For the of Self Seld-Random, set the temperature
as 0.7 to gather a more diverse answer set, as suggested in Wang al. (2022). The p
25

---

Published as a conference paper at ICLR 2024
F PROMPTS
The prompts used in our work consist three components: the description, the question, and the misleading hints (used for misleading sampling strategy). The description part outlines the definition of the task presented to the LLMs, requesting them to provide an answer together with the confidence level for the answer. See Figure 8 for a complete example of full prompt and the model's output. The detailed prompt is provided below:
1. Vanilla: Table
2. Chain-of-Thought-bassttt Table
3. Self-Probing: Table 16
4. Multi-Step: Table 17
5. Top-K: Table 18
26

---

Published as a conference paper at ICLR 2024
Table 14: The designed vanilla prompt for two different tasks.
Task
Vanilla Prompt
Read the question, provide your answer and your confidence in this answer. Note: The confidence indicates how likely you think your answer is true.
Use the following format to answer:
"AAnswer and Confidence (0-100): [ONLY the option letter; not a complete sentence], [Your confidence level, please only include the numerical number in the range of 0-100]%"" Only the answer and confidence, don't 't give me the explanation.
Multi-choice questions
(Seetion:Sppecficc Question Here]
Now, please answer this question and provide your confidence level.
Read the question, provide your answer and your confidence in this answer. Note: The confidence indicates how likely you think your answer is true.
Use the following format to answer:
"'Anwwe and Confidence (0-100): [ONLY the number; not complete sentence], [Your confidence level, please only include the numerical number in the range of 0- 0-100]%
Open-number questions
Only the answer and confidence, don't give me the explana- tion.
Question:(Specific Question Here]
Now, please answer this question and provide your confidence level.
27

---

Published as a conference paper at ICLR 2024
Table 15: The prompt designed for Chain-of-Thought prompting strategy.
Tasks
Definitions of Tasks in Prompts in Chain-of Chain-of-Thought Confidence
Read the question, analyze step by step, provide your answer and your confidence this answer. Note: The confidence indicates how likely you think your answer is true.
Use the following format to answer:
"Exxlanation: [insert step-by-step analysis here]
Answer and Confidence (0-100): [ONLY the option letter; not a complete sentence), [Your confidence level, please only include the numerical number in the range of (0 0-100]%%"
Multi-choice questions
Only give me the reply according to this format, don't give me any other words.
Question:[Specifie Question Here]
Now, please answer this question and provide your confidence level. Let's s think it step by step.
Read the question, analyze step by step, provide your answer and your confidence this answer. Note: The confidence indicates how likely you think your answer is true.
Use the following format to answer:
"'Explnattoon: [insert step-by-step analysis here]
Answer and Confidence (0-100): [ONLY the number; not complete sentence], [Your confidence level, please only include the numerical number in the range of 0-100]%
Open-number questions
Only give the reply according this format, don't give me any other words.
Question:(Specific Question Here]
Now, please answer this question and provide your confidence level. Let's think it step by step.
Table 16: The prompt designed for self-probing prompting strategy.
The prompt designed for self-probing prompting strategy
Question: [The specific question]
Possible Answer: [Thh answer candidates]
Q: How likely is the above answer be correct? Please first show your reason- ing concisely and then answer with the following format:
""Confidence: [the probability of answer (answer) to be correct, not the one you think correct, please only include the numerical number]"
28

---

Published as a conference paper at ICLR 2024
Table 17:
The designed prompt for multi-step prompting strategy.
The designed prompt for multi-step prompting strategy
Read the question, break down the problem into K steps, think step by step, give your confidence in each step, and then derive your final answer and your confidence in this answer. Note: The confidence indicates how likely you think your answer true.
Use the following format to answer: "Step 1: |Your reasoning], Confidence: [ONLY the confidence value that this step correct]%
Question
Step K:
[Your reasoning], Confidence: [ONLY the confidence value that this step correct]% Final Answer and Overall Confidence (0-100): [ONLY the answer type; not complete sentence], [Your confidence value]%""
Table 18:
Prompts used elicit Top-K Verbalized Confidence.
The designed prompt for Top-K prompting strategy.
Provide your best guesses and the probability that each is correct (0% 100%) for the following question. Give ONLY the task output description of your guesses and probabilities, no other words expla- nation. For example:
G1: <ONLY the task output description of first most likely guess; not a complete sentence, just the guess!> Pl: <ONLY the probability that GI is correct, without any extra commentary whatsoever; just the probability!>
Question
Gk: <ONLY the task output description of k-th most likely guess> Pk: <ONLY the probability that Gk correct, without any extra commen- tary whatsoever, just the probability!>
29


Reviews:
[{"review_id": "aduFO3tpk2", "reviewer": "Reviewer_GBf5", "summary": "This paper studies approaches to eliciting reliable confidence estimates from large language models about their statements.  The paper studies closed-box methods for confidence elicitation exploring (1) prompting strategies for verbalized confidence; (2) sampling approaches to measure variance across multiple responses; and (3) aggregation methods to compute consistency measures.", "questions": "Is there a reason to believe that the model's certainty _should_ be correlated with model correctness?  Or to understand a priori (e.g., based on training data) when it might be more or less likely to be so?\n\nMinor variations in prompt wording can have significant effects on performance.  How was the specific prompt text chosen for each of the prompt strategies?  was it optimized or experimented with?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper studies an important problem.  When an LLM's reliability is critical to an application or higher-level task, knowing the likelihood that the LLMs responses is correct or not is critical.\n\nThe overall approach, combining prompting, sampling, and aggregation, is an elegant approach for closed-box confidence solicitation, if it can be calibrated or its reliability as a measure otherwise completely characterized.\n\nThe experiments span many task-domains and multiple models.", "weaknesses": "The experimental results show that the method is useful in identifying uncertainty, but performance varies significantly across benchmark tasks and no single method is clearly better than others.  Given a new task, it is not clear which method will give the best performance, or even if the best performing method on a hold out set will generalize to real problems in a domain.  I appreciate that the authors call this out in the discussion, saying that none of the current algorithms are satisfactory.\n\nThe high level observations enumerated in the introduction are interesting but also seem very preliminary, and difficult to operationalize or build upon.\n\n\n\nminor:\n- the acronym ECE should be defined at first use.  \n- typo page 6: \"aevaluation\" -> \"evaluation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J14t1LReeT", "reviewer": "Reviewer_Bvbz", "summary": "A comprehensive study on self-estimation of uncertainty by LLMs. The authors introduce multiple design choices that can affect uncertainty elicitation and then evaluate them on existing LLMs.", "questions": "see the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Uncertainty estimation is an important question for LLMs. The findings of this study can be impactful in practice\n- The 3-part framework of prompting strategy, sampling and aggregation provide clarity.\n-The evaluation is comprehensive across many datasets.", "weaknesses": "The strategies presented are reasonable (but not super novel, as far as I understand). So the contribution should be evaluated on the empirical evaluation. Here, it is unclear how to interpret the results. No strategy seems to be work well always. \n\nI understand it is hard to analyze black-box LLMs, but still some discussion on why some strategy works, and why it does not, will be useful. \n\nWhat is the best combination of prompting, sampling and aggregation? can the authors propose the best way for practitioners while also noting the limitations? Right now, the message seems to be that nothing works. But it may be useful to present a version of the best performing method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKqaBzIyL5", "reviewer": "Reviewer_uQRx", "summary": "The paper proposes a unified framework under black box setting to evaluate how calibrated large language models are for their predictions. Specifically, no logits or internal states of LLM are assumed given but the models can be asked to provide explicit confidence. The authors evaluate 5 different prompting strategy and various aggregation strategy to elicit better calibrated results from the models.", "questions": "Please see the weakness section above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is very well-written and tackles the black box confidence calibration in a timely manner. The approach is very targeted to LLMs which provide many insights that are not available from calibrating image classification models.\n\n2. The evaluation is very thorough and the two calibration tasks defined are reasonable.", "weaknesses": "1. It seems black-box based confidence elicitation approaches are generally unsatisfactory in calibration performance. Can the authors directly compare them with white-box based calibration approaches such as token probability based calibration approach on open-source models and observe how large the gap can be on ECE and failure prediction?\n\n2. Equation 3. Shouldn't we maximize with respect to P since we are doing an MLE estimation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iUEiScoJTN", "reviewer": "Reviewer_bmdf", "summary": "The paper evaluateds an LLM's capability to express uncertainty. To this end they define a systematic evaluation framework.\n\n+ LLMs are a highly relevant topic\n+ Elicitating confidence is also an important topic in deep learning in general\n+ The study performs extensive evaluation on multiple datasets and LLMs\n- The evaluation and outcomes yield little novel insights (see also detailed comments)\n- The methods are standard, a methodological contribution would be highly appreciated. The idea to use LLMs for self-assessment is all-to-common. That said, it is also interest to assess it in this context, but on its own a marginal contribution. The prompting/aggregation techniques at best partially mitigate this, but they are also well-known. In short, more would be expected for a strong contribution.\n\n\nDetailed comments?\n> 1) LLMs, when verbalizing their confidence, tend to be overconfident...\nLLMs are known to be \"Fluent but non-factual\" or \"Convincing though wrong\". Given that this study focuses only on evaluation I would have expected more novel findings as key fining.\n> 2) As model capability scales up, both calibration and failure prediction performance improve, yet still far from ideal performance.\nThat larger LLMs are better (on all benchmarks) than smaller ones is also general knowledge - check the LLama2 paper, for example, and compare benchmarks for 7B,13B...\n> It would be interesting (and maybe even necessary) to study how much the uncertainty estimates depends on the prompt, i.e., if you state in the prompt \"Appear confident\" are estimates more overconfident?", "questions": "see above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "see above", "weaknesses": "see above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)