Calibrating Expressions of Certainty
Authors: Peiqi Wang
Barbara D. Lam
Yingcheng Liu
Ameneh Asgari-Targhi
Rameswar Panda
William M Wells
Tina Kapur
Polina Golland
Venue: ICLR 2025 Poster (2025)
Keywords: calibration
uncertainty
optimal transport
language models
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We assess and improve the calibration of both humans and language models that communicate their confidence using linguistic expressions of certainty

Abstract: We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration.

Published as a conference paper at ICLR 2025
CALIBRATING EXPRESSIONS OF CERTAINTY
Peiqi Wang +1 Barbara D. Lam 2 Yingeheng Liu
Ameneh Asgari- 2
Rameswar Panda 3
William M. Wells
Tina Kapur 2 Polina Golland
1 CSAIL, MIT Harvard Medical School
MIT-IBM Watson AI Lab
ABSTRACT
We present novel approach calibrating linguistic expressions of certainty, e.g.. "Maybe" and "Likely". Unlike prior work that assigns a single score to each cer- tainty phrase, we model uncertainty distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of cer- tainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language mod- els) and provide interpretable suggestions to improve their calibration.
INTRODUCTION
Measuring the calibration of humans and computational models is crucial. For example, in health- care, radiologists express uncertainty in natural language (e.g.. "Likely pneumonia") due to the inherent ambiguity in the image they examine. These certainty phrases (e.g., "Maybe". "Likely") influence healthcare providers' decisions such as ordering additional diagnostic tests. Accurate per- ception of these certainty phrases directly impacts patient diagnosis and treatment. Additionally, it's more natural for large language models (LLMs) express their confidence using certainty phrases since humans struggle with precise probability estimates (Zhang & Maloney, 2012). Our work en- ables measuring the calibration of both data annotators and LLMs, paving ways for future work to improve the reliability of LLMs.
Existing miscalibration measures focus on classifiers that provide a confidence score, e.g., posterior probability. These approaches cannot be applied directly to text written by humans language models that communicate uncertainty using natural language. Prior work on "verbalized confi- dence" attempted to address this by mapping certainty phrases to fixed probabilities, e.g., "High Confidence" equals "90% confident", (Lin et al., 2022a). The oversimplification misses two key aspects: (1) individual semantics: people use phrases like "High Confidence" to indicate range (e.g., 80-100% rather than a single value; and (2) population-level variation: different individuals may interpret the same certainty phrase differently. Appendix D explains this gap in more detail.
Calibration in the space of certainty phrases presents unique challenges. Prior work such as his- togram binning (Zadrozny & Elkan, 2001) and Platt scaling (Platt, 2000) fit low-dimensional func- tions (e.g., one-dimensional for binary classifiers) to map uncalibrated confidence scores to cal- ibrated probabilities. However, when working with certainty phrases, direct manipulation of the underlying confidence scores is not feasible. Rather than mapping confidence scores directly, we instead calibrate a model by adjusting the use of different certainty phrases.
In this work, we measure and calibrate both humans and computational models that convey their confidence using natural language expressions of certainty. The key idea is to treat certainty phrases as distributions over the probability simplex. Using this construction, mators for miscalibration metrics, such as the expected calibration error (ECE) (Pakdaman Nacini et al., 2015), and visualization tools, such as the reliability diagrams (Wilks, 2006). To calibrate over certainty phrases, we learn a discrete and possibly stochastic calibration map over a set of these certainty phrases to a potentially different set. This mapping is derived the solution an optimal transport problem that minimizes the net change in calibration error.
generalize existing esti-
We demonstrate our approach by analyzing the calibration of radiologists writing clinical reports, accounting for variables such as the pathology and radiologist's identity. Moreover, we show how
"Correspondence wpqêmi edu; Code available on GitHub

---

Published as a conference paper at ICLR 2025
can guide radiologists to become better calibrated in their use of certainty phrases. In addition, we showcase the calibration of language models and demonstrate the effectiveness of our calibration method to post-hoc improve model calibration. Our research opens avenues for assessing and improving the calibration of humans and computational models that communicate their confidence using natural language.
RELATED WORK
Measuring Miscalibration We build our approach the expected calibration error (ECE), a pop- ular method for measuring miscalibration (Zadrozny & Elkan, 2001). Previous research has focused on adapting ECE skewed confidence score distributions (Nguyen & O'Connor, 2015), improving its sample efficiency (Zhang et al., 2020), and debiasing (Roelofs et al., 2022). We extend the ECE estimator to accommodate the assumption that certainty phrases represent distributions rather than real-valued scores. This formulation is robust to binning strategies and alleviates estimation noise under small sample sizes (Kumar et al., 2019). Interestingly, our proposed ECE estimator can be viewed as a variant of kernel density estimator for ECE (Zhang et al., 2020; Popordanoska et al., 2022) that uses input-dependent, possibly asymmetric kernels.
Calibrating Classifiers & Regressors We focus on post-hoc calibration, as opposed to training models to calibrated ab initio. Post-hoc calibration reduces to estimating or fitting the canonical calibration function (Vaicenavicius et al., 2019). For example, histogram binning (Pakdaman Naeini et al., 2015) estimates the canonical calibration function with histogram regression. Platt scaling (Platt, 2000), isotonic regression (Zadrozny & Elkan, 2002), and Beta calibration (Kull et al., 2017) fit monotonic, potentially smooth functions from data. These methods assume models provide con- fidence scores, which makes them unsuitable when only certainty phrases are available.
Distribution calibration (Song et al., 2019) works on regressors that, akin to our setup, predict con- fidence distributions rather than scores. In contrast distribution calibration, we focus classifi- cation problems. Moreover, matching the predicted distribution to the true distribution of subsets of examples sharing the same prediction is not of interest in our application.
Language Model Calibration Language model calibration typically involves defining real-valued confidence scores and using existing tools designed for calibrating classifiers. For instance, confi- dence scores can represent conditional probability of an answer given the context (Desai & Durrett, 2020) or the average probability across different paraphrases of answer tokens (Jiang et al., 2021). These methods require access to the model's 's internal state. Our work is closely related to "verbal- ized confidence" (Lin et al., 2022b; Tian et al., 2023), where the model articulates its confidence in token-space certainty phrases. Rather than focusing solely on the mean, our method offers more realistic quantification of calibration error by treating each certainty phrase as a distribution. Unlike previous studies that apply calibration techniques designed for classifiers (Desai & Durrett, 2020) or fine-tune the language model for controlled generation (Mielke et al., 2022), we propose a lightweight discrete policy that adjusts the use of certainty phrases to improve calibration.
3 METHOD
3.1 BACKGROUND
Miscalibration Measures Let X be the input space and y be a finite set of labels. For simplicity,
we consider binary classification, i.e., y = {0, 1}. A probabilistic classifier g ; X [o, 1] provides its confidence level for the positive class. Here, we use to refer to any type of classifier, whether it be a human or a computational model. Classifier g is calibrated if the true probability of the positive class given model's 's prediction is exactly equal to that prediction:
E Y S] = S,
where S = (X) represents the confidence score. For instance, radiologist is perfectly calibrated his predicted probabilities match real-world outcomes, e.g., a radiologist predicts a 30% proba- bility of pneumonia for group of patients, then pneumonia actually occur in 30% of those patients.
Alternative definitions of calibration, e.g., confidence calibration (Guo et al., 2017) and class wise calibration (Kull et al., 2019), equivalent for binary classification (Vaicenavicius et al., 2019).
2

---

Published as a conference paper at ICLR 2025
miikkety
"miliaal"
"maggh"
'posohyy
"aggenine
Sanpeiios
"potsslyy"
"iikdy"
"moot likey"
pby segeotor
"wopprtk
Figure I: Probability density functions obtained by fitting beta distributions results of a survey of radiologists' perception of different certainty phrases (Shinagare al., 2023).
The expected calibration error (ECE) (Pakdaman Naeini et al., 2015) measures the degree to which the calibration definition is violated in expectation:
ECE E [Y [S] S|],
where the outer expectation is computed with respect the distribution Ps that the pushforward of Px by If ECE = 0, then is perfectly calibrated.
Empirical Estimation Given dataset = (111111)........ 81). (IN yN }}. one can estimate ECE in Equation (2) visualize the reliability diagram (DeGroot & Fienberg, 1983) with binning (Pak- daman Naeini et al., 2015). Specifically, we use (.,,... .M) to denote the set of bins that partition classifier y's range and B, to denote the set of indices of samples whose predictions fall into bin Im- Empirical averages
Bm N
fm
Yn
Pm
Σ
g(
(3)
Bm
B
nEBm
nE Bm
serve estimates for per-bin calibration function P(S Im), and model prediction Im -
E (Y S E Im]. score probability Pm S E [m] respectively.
ES
The binning estimator for ECE is a weighted average of per-bin calibration errors
BCE
Pm Iim 9mll
(4)
Statistical properties of the estimator above have been studied in (Vaicenavicius et al., 2019).
we define f(s) = fm for s E Im. function f is the histogram regression estimator for the canonical calibration function r(s) & ( S s). The reliability diagram displays the curve (s, f())) for E [0, 1]. If g is perfectly calibrated, then f(s) = s for [0, 1].
Post-hoc Calibration Given a model and a calibration dataset D, post-hoc calibration seeks a calibration map [0, 1] [0 such that g well-calibrated. This can be achieved by mini- mizing Y WD ((t(g()))) Y') over family T of possible mappings, where & is a loss function. Platt scaling (Platt, 2000) is an example of this approach, where T the set of logistic functions and é is the log-loss.
3.2 CONFIDENCE AS A DISTRIBUTION
Let U A (u.,...,,0 } be a K distributions corresponding to different certainty phrases. For instance, the phrase "Maybe" could be modeled as Beta 2), which is centered at 0.5 with decaying density away from 0.5. Figure illustrates what these distributions might look like. We can derive confidence distributions from surveys human perception of certainty phrases, e.g., in radiology (Shinagare et al., 2023) or from social media polls (Fagen-Ulngg UImschneider, 2023). In this work, we employ beta distributions to represent the confidence distributions u,, uk-
We assume that instead of outputting a scalar-valued score, the classifier : X + P((0, 1)) provides its confidence for the positive class distribution over [0, 1). Here, we use g abstract a human
3

---

Published as a conference paper at ICLR 2025
or a language model that generates certainty phrases about its internal belief of event, e.g.. radiologist might dictate "Likely pneumonia." " To restrict our attention to a finite set of certainty phrases, assume 9 " where ? X + [K] outputs an index and u : [K] - P((O, 1J) maps that index to the corresponding confidence distribution, i.e., u(k) = MAk- Given any sample a E X, the classifier's output g(r) Wly(z) one of the K confidence distributions. Alternatively, g(x) can be interpreted defining conditional density for the score S that is finite mixture over {ux] i.e,
K
fsi (s 12)
Σ (7())) - kj (+).
(5)
k=1
The definitions of calibration and ECE in Equations (1) and (2) remain valid even we move to treat confidence as a distribution. Instead being the pushforward of Px by 9. Ps is characterized by the density fs(s) E [fsix (5 X)]. We generalize estimators in Equation (3) as follows:
Σ Σn n= P(SE E 1m X = In)Un
N
Pm
P(S E Lm =x=))
(6)
En=l P(S Im 1 = xn)
7=1
Im En P(S 4m I SE Im =xn]
(7)
CASIP(SSE
X ..)
Proposition 1. The estimators fm Pmo and Im defined in Equation (6) are consistent estimators for [| S Im], P(S Im), and E [S Im] respectively. We provide the proof in Appendix A.1.
The estimators in Equation (6) reduce to that in Equation (3) if the confidence distributions delta distributions, i..., U {6g [0, 1]}. Provided that the confidence dis- tributions are supported on [0. 1], we can interpret the estimators in Equation (6) as allowing each sample to contribute to the estimates 9m Pm of every bin Lm with smaller weight determined by P(S E In X = x. The resulting estimates are more robust to increasing the number of bins and will not result more biased calibration errors explain in Section 3.3.
Interpretation
Implementation Since the confidence distributions are represented via their parametric form, computing P(S E Lm X xn) is straightforward and requires two evaluations of the CDF of Vy(z,.) In contrast, computing E [S S E Im. X = xx] involves integrating w.r.t. den- sity. If the number of bins M small, scipy. iiniinggattt to trade time for better approximation; If M is large, midpoint rule provides sufficiently good approximation of the integral. requiring single evaluation of the density Uy(a,))
can use adaptive quadrature algorithms through
Accounting for Uncertain Labels In practice, ground truth labels y1, y1. Yn. may not always be available. Instead, we might only have access certainty phrases that provide information about the latent binary labels. We can incorporate such uncertain labels into calibration error estimation, which especially useful when the sample size is small and many labels are uncertain. Let en E P(0,, 1]) be the confidence distribution associated with the certainty phrases for label yn- To account for this uncertainty, we can replace Yr. with P(cn 2 }) in the estimator f' defined in Equation (6). The modified estimator remains consistent that we prove in Appendix A.2.
3.3 CONNECTION TO KERNEL DENSITY ESTIMATION OF ECE
Suppose I,,... Im are equal width intervals, i.e., Im = (sm, 8m + 5) for some > 0. We define continuous versions of the estimator f'm and Pm in Equation (6)
ΣN fsix(s aa)yn
1 N N
f(s) = EN= fsix(s xn)
fs(s)
Σ R=1
fsix(s (
(8)
by considering the limit when the number of bins M goes to infinity. Specifically,
A EN N P(S E 1m X = xn)un
ΣN (f 1nn)yn f(mm),
N Em P(S E 1m X xx)
x Sfsix (sm xn)
4

---

Published as a conference paper at ICLR 2025
P(S E Im! X nn)) ff (n)) when M oc or d 0.
For each sample En E X, let On, Pn be parameters of the predicted confidence distribution g(xn) (eta( Bn). We can define input-dependent kernel KE,, (s -8,) = fg(u.)) (8) where A
on +B -2 the "mode" and h2 -
On-1
the "spread" of the kernel. Note
on aatia +
En Knn (s 8n)yn
f(s)
(10)
Σn KE, (5- -8)) 8n)
is the Wadaraya-Watsee estimator (Nadaraya, 1964; Watson, 1964) for the canonical calibration function r(s) = E [Y 5 s). This estimator akin to the KDE-based estimator proposed in (Zhang al., 2020; Popordanoska et al., 2022). However, unlike the KDE-based estimator where &n the predicted confidence score and hn is a fixed hyperparameter common to all n, in the estimator in Equation (10), both s, and h, parameters induced by the confidence distribution (Ixx)-
We define a continuous version of the estimator for ECE in Equation (2) as
/
ECE
- s|
(11)
The estimate ECE in Equation (4) is finite-sample approximation of ECE. ECE can be shown to be unbiased and consistent (Zhang et al., 2020; Popordanoska et al., 2022).
Caveats When the model predicts delta distributions as confidence distributions, e.g., g(x,), ~ 8, for some [0, 1] indicating "100% Confident," special attention is required. These samples can be easily missed when constructing estimates i(-) and fs(-) if the sequence 81, ... 88M is not properly chosen. For continuous confidence distributions U, we approximate P(S E Im Xx= x.)) using the midpoint rule when computing the estimators in Equation (6). This numerical approxima- tion of f. is exactly f(()) defined in Equations (8). If some predicted confidence distributions g(x,)) are delta distributions, we compute the binning estimators in Equation (6) for the bin Im where E Im without any approximation.
3.4 OBTAINING CALIBRATION MAPS USING OPTIMAL TRANSPORT
Calibration Map If a classifier outputs confidence distributions, we do not have direct access the confidence scores, rendering the classical definition of a calibration map [0. - [0. 1] inapplicable. A natural extension is to consider map f: P([o, 1]) + P(0,, 1J) that adjusts confidence distributions arbitrary ways. However, this definition complicates the interpretation of results. For instance, if Beta(2, 2) represents "Maybe", it is unclear what slightly modified distribution, such as (2 2.2). represents in natural language.
Instead, we define calibration map ê [K] - [4] that maps elements of the source set of K confidence distributions {uI,... ux) to elements of a target set of L confidence distributions {vi....,, .z]]. More concretely, let T E R denote the transport matrix, where T&t represents the proportion of samples for which the confidence is better described by ey rather than ug. We then de- fine t(k) as a draw from the categorical distribution with unnormalized probabilities (TkI, ... (TLL)) For example, if iy and i) confidence distributions that correspond to "Likely" and "Maybe" spectively, then / Σ, Tu = 0.3 indicates that the model should change 30% of its use of "Likely" to "Maybe" reduce overconfidence.
Calibration as Discrete Optimal Transport Given that model is defined as u 7, post-hoc calibration aims to find t such that the composition u 1 o q wall-calibrated. To achieve this, we formulate discrete optimal transport problem. Let a₁ = 1/4 > 1 (xx)) = k] be the proportion of times k-th certainty phrase is mentioned. Therefore, the source weight A-1 represents occurrences of certainty phrases in the calibration dataset. In some application, given target weights b (b.,... ,bL) L- that represent the ideal use of the target confidence distributions. We define cost matrix C E gxxL where
(a1,-.. ax)
1 ak
Cil
ECB Ukk + 01) ECE
(12)
5

---

Published as a conference paper at ICLR 2025
is the per-unit net change in the calibration error when we replace 4p with 2 the confidence distribution. If target weight is given, we can obtain the calibration map t by solving the classic Kantorovich formulation of discrete optimal transport (Peyré & Cuturi, 2019) that minimizes (C, )) over 7 E gKx subject to the constraints 71 = a and 7 = b. The optimal transport plan T minimizes the amount of work, e.g., change in the expected calibration error, required to transform Ex u to Σ, bary. In practice, the target weight is often unknown and that we use the same source and target confidence distributions. In this case, it is generally desirable to ensure that the transport plan T does not significantly change the relative use of source certainty phrases. To achieve this, we use unbalanced optimal transport (Frogner et al., 2015; Peyré & Cuturi, 2019):
min (C,T) + (T) + Ti KL (Tilla) + T2 KL (T 1
(13)
TER KXK
where (T) = (T, log(()) is the discrete entropy of T and T1, 72 controls the degree of marginal deviations. In practice, we set large Ti to strictly enforce the constraint '1 while a smaller 72 to permit some deviation of transported mass 7 from the initial proportions a.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Evaluation Metries We estimate ECE described in Section 3.2. Specifically, we partition [0, 1] into 100 equal-width bins and compute the estimators in Equation (6) for each bin. To improve computational efficiency, we apply the midpoint rule to approximate the integrals, except for the first and last bins to account for (zx,) E fōu, 81} for some In We use bootstrap resampling with 100 samples to calculate the mean and 95% confidence interval for these estimators. In addition to ECE, we report modified calibration error, ECE", which excludes 100% confident predictions (e.g.. 6o, 61) the first and last bins, and therefore better characterizes the calibration curves that unaffected by extremely confident predictions. To compute classification statistics, such as the Brier Score (BS) and accuracy (Acc). We also employ bootstrap resampling, with further sampling from the predicted confidence distributions g(x,,) and the ground truth confidence distributions Czs to obtain the scalar-valued scores and binary labels required to compute these metrics.
Optimal Transport Calibration We solve the entropy-regularinn unbalanced transport problem in Equation (13) using the log-stablization variant of the Sinkhorn algorithm (Cuturi, 2013; Chizat al., 2018; Schmitzer, 2019) to mitigate numerical instabilities. We use POT's solver implementa- tion (Flamary et al., 2021). Based on ablation studies (Appendix B.5), we set = le-3 to minimize mass splitting and simplify interpretation of the calibration map. By default, we set T2 1e-3 arbitrarily due to its minimal impact on performance given such small
4.2 CALIBRATING RADIOLOGISTS
To demonstrate the utility of our approach, we analyze the calibration of radiologists on dataset of reports dictated for X-ray images and matching CT scans.
Dataset We curated a dataset of 2,662 paired chest X-ray and CT radiology reports recorded from January to September 2023 Brigham and Women's Hospital. We treat X-ray reports human predictions {g(21), x and CT reports as ground truth labels -. ) because the 3D CT scans provide greater details. Both types of reports contain certainty phrases about various pathologies. Appendix B.2 provides an example X-ray report. To mitigate distribution shifts in the use of certainty phrases, we use stratified sampling and split the dataset equally into calibra- tion and test sets. We anonymize the names of radiologists with commonly used names in the US (Remy, 2021). To extract (pathology, confidence) labels from clinical reports, we prompt Llama 8B (Dubey & et al., 2024) with in in-context learning. This approach demonstrates good accuracy on a hand-annotated test set of about 150 X-ray reports, but performance is slightly lower on the CT reports, likely due to their more detailed descriptions. Appendix B. provides further details.
Confidence Distributions We derive the confidence distributions {u,.... uux } from survey data on radiologists' interpretation of "diagnostic certainty phrases" commonly used in dictating radiol-

---

Published as a conference paper at ICLR 2025
Atelectasis
Pneumonia
Edema
Infection
Overall
DCE
DCC
DC24-
DCC'-0.33 00337
ECE
0.25
0.25
0.25
0.25
1.25
0.25
Faviola
Bo
Avinash
Christen
Ruby
BCE 0.164
BEEE" 0,193
IECE" 0,258
IEEE 0.175
:.17
Figure 2: Reliability diagrams of radiologists' certainty phrase in clinical reports, stratified by pathology (top) and radiologist identity (bottom). The calibration curve (red), with its 95% confi- dence interval (blue) and score density (gray) are shown. There is significant variation in calibration across different pathologies and radiologists. Areas where the calibration curve is above the identity line correspond radiologists underestimating their confidence. Interestingly, this correlates with regions of low confidence. Same is true about overestimation in regions of high confidence.
ogy reports (Shinagare et al., 2023). Specifically, we fit beta distributions to the empirical data using the method of moments. Appendix B.3 provides further details.
Measuring Radiologists' Calibration Figure illustrates the variation in radiologists' calibration across different pathologies and between individual providers. Radiologists are generally undercon- fident in diagnosing common pathologies like atelectasis but tend ambiguous conditions like infection. In general, they exhibit underconfidence when expressing low certainty (e.g., using terms like "Possibly") and overconfidence with high certainty (e.g., "Likely"). This is consistent with prior studies human perception of probabilities (Tversky & Kahneman, 1992). Moreover, individual radiologists display unique calibration patterns and distinct behaviors in their use of certainty phrases. Individual preferences for certain terms can contribute to differ- ences in calibration. For instance, Bo frequently uses the term "May" while Christen often uses "Likely." Avinash is less calibrated than Bo, despite sharing similar calibration curves.
overconfident with more
Improving Human Calibration Figure 3 demonstrates the effectiveness of our calibration method in providing interpretable recommendations to improve radiologists' calibration. For in- stance, the calibration map suggests that radiologists use "May" instead of "Absent" address underconfidence when diagnosing atelectasis. Similarly for edema diagnosis, the calibration map recommends lowering the confidence by replacing "Present" and "Likely" with "May". By follow- ing these straightforward adjustments, our method reduces the ECE" and Brier Score (BS) on the test set. contrast, classical methods like histogram binning or Platt scaling does not provide clear guidance how radiologists can adjust their language in reporting to improve calibration.
After calibration, the calibration curve not perfectly aligned with the identity function for two reasons. First, adjustments to small and discrete set of certainty phrases inherently limit precision in fine-tuning the curve. Second, the improvement in calibration depends on model's discriminative performance. For instance, the left side of the calibration curve (top-right subfigure in Figure 3) cannot be lowered further without a model that more accurately detects the absence of atelectasis.
In Appendix C.I, examine the effect of the resulting calibration map, illustrating the tradeoff' between preserving assessment informativeness and improving calibration.
4.3 CALIBRATING LANGUAGE MODELS
To demonstrate the utility our approach, we analyze the calibration of language models (LMs) on crestioo-aanswernng datasets.

---

Published as a conference paper at ICLR 2025
Before
Cost
Calibration Map
After
0.25
0S:00.56
"Ahurr"
3:00
4.19
6.44
"Present"
Preseni
05:00259
Atclectasis
0.75
"houlky"
"Likely"
"Likely"
0.75
0.5
'Nay"
/16
62
"May"
"May"
0.5
"L.key"
1:10
0.36
5.00
0.
"Poosibly
"Possibly"
'Toood"
"Absemi"
Phoudy "Ma
Before
Cost
Calibration Map
A After
0.27
B8.0.166
3:00
3:50
"Present"
cal
"Present"
08:00.14
Edema
0.5
E4
"Likely"
"Lilkel""
"Vayy*
8.99
085
006
323
0.47
"May"
"May"
0.55
812
4.11
3.00
"Possbly
"Pssiiby"
0.3
0.5
422
Thourd
"Absont"
"Aboems"
Figure 3: Examples of calibrating radiologists on two representative pathologies: atelectasis and edema. The 1st and 4th columns show the reliability diagrams before and after the post-hoc cali- bration, respectively. The 2nd column displays the cost matrix of the optimal transport problem, while the 3rd column illustrates the probabilistic calibration map T' For atelectasis, underconfidence can be addressed by suggesting the use of "May" instead of "Present"; For edema, overconfidence be mitigated by recommending that radiologists replace "Present" and "Likely" with "May". Quantitatively, our calibration approach improves ECE and Brier Score (BS) metrics.
Models We evaluate state-of-the-art LMs, such as gpt-4o (got-40-2022-08-000) Hlaude-3.5-somme 2kaule----sommee2224400000 and 1.5ini--..-pro geminni-1.-proobOO0 as well as their smaller and faster variants, such as gpt-4o copt-4o-2024-0-113, claude-3-haike (claude-3-haile 20240307), and gemini-1.5 semini-1.5-flashl (gemini-1.. 1.5ii-----flass--02)).
Evaluation Setup Following Tian et al. (2023), we use two question caattioon-nswwring datasets: (1) SciQ (Welbl et al., 2017) contains crowd-sourced science exam questions, and (2) TruthfulQA (Lin al., 2022c) contains questions designed to test language models' tendency to mimic human misconcep- tions. We use all 1,000 questions for SciQ and 817 questions for TruthfulQA from their respective validation set. Each dataset is evenly split into calibration and test sets using stratified sampling.
We focus on evaluating language models' ability to verbalize their confidence natural language (Lin et al., 2022a). We prompt the models to provide either probability (e.g.. "0.3" or "0.7"), a cer- tainty phrase (e.g.. "Maybe", "Likely"), or a distribution directly (c.g.. "Beta(2, 3)"). Appendix B.2 includes example of how LMs express confidence using certainty phrases.
Similar to Tian et al. (2023), we derive ground truth labels by using gpt-4o-mini to evaluate whether model's answer is semantically equivalent the correct answer, thereby avoiding false negatives that arise from exact matching. For the TruthfulQA dataset, we verify whether the predicted answer matches any of the correct answers further reduce false negative rates.
Appendix B.6 provides more details on prompts used. Table lists all prompt templates.
Confidence Distributions We investigate how the choice of certainty phrases and their corre- sponding distributions U1, ",...., ux impacts calibration performance. We prompt gpt-4o to generate K pairs of (certainty phrase, distribution) using the prompt template in last row of Table 5. Fig- ure shows that calibration performance varies with K, but remains consistent across models and datasets. Based this analysis, choose K 12 for subsequent experiments.
Table 3 compares calibration performance when confidence distributions are derived by: (1) survey: fitting beta distributions to survey data human perceptions of probability-related terms (Fagen- Ulmschnedder, 2023), (2) on-the on-the-fly: generating beta distribution parameters for each sample, and (3) fixed: selecting from a predefined set of certainty phrases generated by gpt-4o. We find that pre-
8

---

Published as a conference paper at ICLR 2025
gpt-4o-mini
gpl-40
-huik-
Senude-3.5-somee
gemini-- 1.5-flshh
gemini-I Demini--.35ppo 1.5-rro
Fiat
ICE
DGE: 6,19
BCR:
BCE 0155
SciQ
9.30
ICI:0220
DCB: 6,35
DCL: BL344
ICE 1:324
TruthfulQA
Figure 4: Reliability diagrams of LMs verbalizing confidence from fixed set of certainty phrases generated by prompting gpt-4o, evaluated on SciQ (top) and TruthfulQA (bottom). The calibration curve (red), with its 95% confidence interval (blue), and score density (gray) are shown. Models are better calibrated on SciQ than TruthfulQA, with larger models (e.g.. gpt-4o) outperforming their smaller variants (e.g., gpt-4o-mini). The smooth calibration curve improves the ability of human viewers to compare the calibration performance of different models.
defining fixed set of phrases for the model to choose from yields better calibration than generating confidence distributions on-the-fly. Allowing the model to propose its own certainty phrases and rely on its perception of these phrases proves more effective than using those provided by humans.
Appendix B.3 provides additional details. Table provides visualization of the probability density functions of the confidence distributions studied.
Mcasuring Language Models' Calibration Figure 4 illustrates the calibration performance of various LMs verbalizing confidence by selecting from set of certainty phrases generated by prompting gpt-4o. Table 5 (row 5) provides a detailed view on the certainty phrases used. Cur- rent state-of-the-art language models exhibit good calibration on the SciQ dataset but perform less well on the TruthfulQA dataset. Larger models, such as gpt-4o, consistently outperform their smaller counterparts, like gpt-4o-mini, and gpt-4o-mini) exhibit more similar characteristics compared to models from different vendors. For instance, gpt-4o tends to be more confident than -.-odee..--oonet,, even though both models achieve similar calibration errors.
terms of calibration. Models from the same family (e.g., gpt-4o
Appendix C.3 demonstrates that our method yields a smooth calibration eurves that enables clearer differentiation between model calibration profiles and better correspondence to values of ECE. Un- like binned calibration curves, which are sensitive to bin size and harder to interpret, our approach produces stable, consistent curves robust to binning variations.
Post-hoc Calibration of Language Models We compare our optimal transport calibration method with two classic calibration methods: Platt scaling (Platt, 2000) and histogram binning (Zadrozny & Elkan, 2001). These baseline methods are evaluated in two scenarios: (I) verbalized probabil- ity and (2) verbalized certainty phrases. For the second scenario, since the baseline methods only work with scalar confidence scores, we mapped the verbalized phrases to the mean of their corre- sponding confidence distributions. (Tian et al., 2023) also evaluates histogram binning under the verbalized phrase setup. To ensure fair comparison, we calculate ECE and Brier Score (BS) for our method the same way as for the baselines, as opposed to following our formulation. Our method unique in producing calibrated certainty phrases directly, while baselines require conversion to scalar confidence scores during calibration.
Table demonstrates the effectiveness of our optimal transport calibration method in improving the calibration of language models. We show that even after reducing the output natural language ex- pressions of certainty to scalar values (by simply taking the mean of the corresponding confidence distribution), our method remains competitive and does not compromise performance in terms of ac- curacy. We emphasize that our method's key advantage isn't outperforming baseline calibration methods, but in directly operating on and producing natural language certainty phrases prop-

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of post-hoc calibration methods for language models expressing confidence as probabilities or certainty phrases. Our optimal transport calibration method, even after reducing output certainty phrases to scalar values (e.g.. by taking the mean of the confidence distribution), remains competitive with calibration baselines without compromising accuracy. Unlike traditional methods limited to confidence scores, our approach directly operates on and produces certainty phrases, offering actionable suggestions to improve human calibration.
Model
Verbalize
Calibrated?
SciQ ECE
TruthfulQA ECE
Acc f
BS
Acc
BS 4
gpt-4o
probability
0.72
0.18
0.22
0.3
0.52
0.43
(scaling)
0.72
0.11
0.19
0.3
0.11
0.2
phrase
0.73
0.07
0.19
0.35
0.22
0.28
(scaling)
0.73
0.09
0.17
0.35
0.11
0.2
(binning)
0.73
0.07
0.17
0.35
0.05
0.2
(ours)
0.73
0.08
0.18
0.35
0.1
0.21
Slaud-3.55-oome
probability
0.69
0.24
0.24
0.34
0.42
0.37
(scaling)
0.69
0.1
0.19
0.34
0.03
0.2
phrase
0.66
0.06
0.22
0.33
0.25
0.3
(scaling)
0.66
0.06
0.21
0.33
0.06
0.21
(hinning)
0.66
0.06
0.21
0.33
0.04
0.21
(ours)
0.66
0.07
0.22
0.33
0.04
0.21
gemini-1.5 1.5-pro
probability
0.72
0.22
0.24
0.27
0.5
0.46
(scaling)
0.72
0.03
0.19
0.27
0.12
0.19
phrase
0.68
0.17
0.22
0.34
0.28
0.31
(scaling)
0.68
0.09
0.19
0.34
0.04
0.2
(binning)
0.68
0.07
0.19
0.34
0.07
0.2
(ours)
0.68
0.06
0.19
0.34
0.07
0.21
erty that existing methods lack. This makes our approach uniquely suited for improving human calibration in real-world settings, it provides actionable guidance (e.g., suggesting radiologists use "May" instead of "Present" in their reporting to mitigate overconfidence) rather than abstract probability adjustments that can't be easily understood by humans.
Figure 13 provides visualizations of the calibration process for different language models. While the calibration maps follow similar overall trend, they reveal distinct differences based on each model's distinct preference for certainty phrases.
DISCUSSIONS
Limitations and Future Work Our study demonstrates that radiologists can improve their cali- bration by strictly following our proposed calibration method. However, it remains be investigated how receptive radiologists are to calibration-improvnng suggestions and whether they can mentally adjust their use of certainty phrases effectively. Conducting a clinical study to assess these behav- ioral aspects would provide valuable insights into the practical benefits of our work.
Conclusions This work presents a novel approach to modeling certainty phrases as probability distributions instead of fixed confidence scores. By adopting this perspective, generalize existing estimators for ECE and introduce a smooth calibration curve for reliability diagrams. Additionally, propose interpretable post-hoe calibration method based optimal transport that provides actionable calibration maps. Our method effectively calibrates both radiologists and language mod- els. providing valuable insights into their calibration characteristies while serving as a lightweight tool for improving calibration. We believe our work will be useful for understanding and improving both human and computational models that communicate uncertainty through natural language.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Takeda Fellowship, the MIT-IBM Watson AI Lab, the MIT CSAIL- Wistron Program, and the MIT Jameel Clinic. We would like to thank Victor Ion Butoi, Maohao Shen, and Yoon Kim for the helpful discussions, as well as Atul B. Shinagare, Maria Alejandra Duran Mendicuti, and Steven Horng for their clinical insights.
REFERENCES
Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Lénaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavies Vialard. Scaling Algorithms for Unbalanced Optimal Transport Problems. Mathematics of Computation, (76314):2563-2600 2018. ISSN 0025-5718.
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
Morris H. DeGroot and Stephen E. Fienberg. The Comparison and Evaluation of Forecasters. Jour- nal of the Royal Statistical Society. Series D (The Statistician), 32(1/2):12-22, 1983. ISSN 0039- 0526. doi: 10.2307/2987588.
Shrey Desai and Greg Durrett. Calibration of Pre-trained Transformers. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 295-302, Online, November 2020. Association for Computational Linguistics. doi: 10. 118663//12000..mmmmmmmmaao
Abhimanyu Dubey and al. The Ilama 3 Herd of Models, August 2024.
Wade Pagen-Ulmsccheider. Perception of probability words, 05 2023. URL httptts/////s..co ill iiiiiiis edu/vi sual zat inaatioos///ie -frerceptioon-o--Prrbbbaaiiaiii
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanis- las Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Foumnie, Léo Gautheron, Nathalic T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong. and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78): 1-8, 2021.
Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya, and Tomaso A Poggio. Learn- ing with a Wasserstein Loss. In Advances Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference Machine Learning, pp. 1321- 1330. PMLR, July 2017.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How Can We Know When Language Models Know? On the Calibration Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 9:962-977 977, 2021. doi: 10.1162/tucl..00007.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: A well-founded and easily implemented improvement logistic calibration for binary classifiers. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pp. 023-631. PMLR, April 2017.
11

---

Published as a conference paper at ICLR 2025
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song. and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified Uncertainty Calibration. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching Models Express Their Uncertainty in Words. Transactions on Machine Learning Research, October 2022a. ISSN 2835-8856.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching Models Express Their Uncertainty in Words. Transactions on Machine Learning Research, June 2022b. ISSN 2835-8856.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022c. Association for Computational Linguistics. doi: 10.18655/w1/0022.ccll
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing Conversational Agents' Overconfidence Through Linguistic Calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022. doi: 10.1162/tac__a0044
A. Nadaraya. On Estimating Regression. Theory of Probability & Its Applications, 9():1411142, January 1964. ISSN 0040-585X. doi: 10. 137/1 109020.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. In Lluís Marquez, Chris Callison- Callison-Burch, and Jian Su (eds.), Pro- ceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp- 1587-1598, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.1865//1/DD5-1113
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht, Obtaining Well Calibrated Prob- abilities Using Bayesian Binning. Proceedings of the AAAI Conference Artificial Intelligence, 29(1), February 2015. ISSN 2374-3468, 21066,221111110 2159-5399. doi: 10. wv9p//
Gabriel Peyré and Marco Cuturi. Computational Optimal Transport: With Applications to Data Science. Foundations and Trends in Machine Learning, 116--))355--077, February 2019. ISSN 1935-8237, 1935-8245.
doi: 10. 10.1561/2200000073.
John Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Adv. Large Margin Classif,, 10, June 2000.
Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. A Consistent and Differentiable Lp Canonical Calibration Error Estimator. Advances in Neural Information Processing Systems, 35: 7933-7946, December 2022.
Philippe Remy. Name dataset. https: http://ttt//.tttt.o...o.oooo/ooa 2021.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating Bias in Cal- ibration Error Estimation. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pp. 4036-4054. PMLR, May 2022.
Bernhard Schmitzer. Stabilized Sparse Scaling Algorithms for Entropy Regularized Transport Prob- lems. SIAM Journal on Scienrific Computing, 41(3)A1443-A1433 January 2019. ISSN 1064- 8275. doi: 10. 137/16M1 106018.
Atul B. Shinagare, Giles W. Boland, Marcelo Di Carli, William Mayo-Smith, Stuart G. Silverman, Aaron Sodickson, and Ramin Khorasani. Diagnostic certainty scale, 2023. URL https:// rad. bwh. ha rararre rd.
12

---

Published as a conference paper at ICLR 2025
Hao Song. Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In Proceedings of the 36th International Conference on Machine Learning, pp- 58.779995555 5897-5906. PMLR, May 2019.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just Ask for Calibration: Strategies for Eliciting Calibrated Con- fidence Scores from Language Models Fine-Tuned with Human Feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433-5442, Singapore, December 2023. Association for main.330.
Computational Linguistics. doi: 10.1 (0.18663/1/22223emmm
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5(4):297-323, October 1992. ISSN 1573-0476.
doi: 10.1007/BF001 22574.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pp. 3459-3467, PMLR, April 2019.
Geoffrey S. Watson. Smooth Regression Analysis. Sankhyā: The Indian Journal of Statistics, Series (1961-2002), 26(():359-3) -922 1964. ISSN 0581-572X.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe V. Le, and Denny Zhou. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 35:24824-24837, December 2022.
Johannes Welbi, Nelson F. Liu, and Matt Gardner. Crowdsourcing Multiple Choice Science Ques- tions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguisties. doi: 10. 18853//11/117-44133.
Daniel S. Wilks. Statistical Methods in the Atmospheric Sciences. Number volume 91 in Inter- national Geophysics Series. Elsevier, Amsterdam Paris, 2nd ed edition, 2006. ISBN 978-0-12- 751966-1.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Ma- chine Learning, ICML '01, pp. 609-616, San Francisco, CA, USA, June 2001. Morgan Kaufmann Publishers Inc. ISBN 978-1 1788--55560-778-
Bianca Zadrozny and Charles Elkan. Transforming Classifier Scores into Accurate Multiclass Prob- ability Estimates. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2002. doi: 10. 15.17/7740447777555511
Hang Zhang and Laurence T. Maloney. Ubiquitous Log Odds: A Common Representation of Proba- bility and Frequency Distortion in Perception, Action, and Cognition. Frontiers in Neuroscience, 6, January 2012. ISSN 1662-453X. doi: (0.3389/finns.2122.00000
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-Match Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning. In Proceedings of the 37th International Conference Machine Learning, pp. 11117-11128 PMLR, November 2020.
13

---

Published as a conference paper at ICLR 2025
A THEORY
A. PROOF FOR PROPOSITION
Proof. We show consistency of f'm that we rewrite as follows
E P(S Im X kn fm E P(S E Im X = xn)
(14)
We focus on the numerator of im first. Let Zn & P(S E Im
Xn)Yn be statistic of n-th sample
and Z P(S Im X)Y. Then,
| E E P(S E Y x]]
(Law of Iterated Expectation)
XI / 1 [s E (md) Ss x) ds (P(S Im) a constant conditional on X)
= = BY B
Σ
yl Is [m) Prix(u [s)s((( xx)x(x) ds dr
(E(0,1)
E S 1m]l
(S 1 Y X))
By weak law of large numbers (WLLN), NV Σ Zn 2 E [Z]. Similarly, let W, - P(S Im Xn) and W & P(S Im X) where [ P(S Im | X). By WLLN, 1/2 Σm W, 3 [W]. Let w(x) 1 where {0} its set of discontinuities. We can see that the denominator fm over wc-)'s set of discontinuities has measure zero, i.c.
P(S Im X =
=xxx) E = 0.
(15)
N
By Continuous Mapping Theorem, wf Since products of convergent sequences of random variables converge in probability to the product of their limits,
E P(S E Im =x= 4 (P((((X) E Im)).
N
P(SE Im X In)yn
Σ
N
P(SE E Im Im
X 1x=xx)) =.))
E P(S 4m)
E [Y E Im]
(16)
N
N
n=1
n=1
E Y SSE Im].
(17)
In the process showing consistency for f.s we have also proved that pm use similar strategy to show consistency of 9m and will omit the proof here.
1 P(S E Im). We can
A.2 PROOF FOR PROPOSITION 1 UNDER UNCERTAIN LABELS
Here, we consider we only have access to certainty distributions of the ground-truth labels CI, CN instead of binary labels ,..... , yN. Analogous to situation where we define classifier that outputs a distribution, let h X - P(O0, 1J) outputs a confidence distribution that corresponds the certainty phrase mentioned in the ground-truth text. For any æ X. h(()) Up(z) where
p: p: + [K] selects the certainty phrase. hxx) defines conditional density for C:
fax((e x))
Σ1 k]] (x(c) (c)
(18)
k=1
Additionally, we can convert to ground-truthe label with Y [ 2 !
We can show that
N DA P(S Im X 1 x.)
fm
(19)
EN P(S Im X x.))
consistent estimator of [Y S lm.)
14

---

Published as a conference paper at ICLR 2025
Proof. We focus the numerator of fm. Let Z = P(S Im X)P 2 / X). Then,
[/
z) z) ds / 1 = 1 2 faxx(c 2) 2)
E
Iml fspx
(20)
1
8 mm]
c 2
[s|((s (exx x)fx (x) ds dede
(21)
1 2
E 1 ml
TT X)
- mm]]
(22)
We can complete the proof by following A.1 from here on.
15

---

Published as a conference paper at ICLR 2025
Frequency report (impression) containing DCPs
disgnostt aiagnontic for with with highty puggendi comte
Bietl
may one Li
represent Represnntt
more lieey lans rietyy
pocalbyy question of stospicion uniikely very unikenty serrinee for
for
very love i8 fieelood
mil Prddeme sever marked
Diagnostic Certainty Phrases
Figure 5: Relative frequency of diagnostic certainty phrases used in X-ray reports from the curated paired (X-ray, CT) dataset.
Frequency of report (findingslimpression) containing DCPs
0.5
0.4
Joth
% Total
Afoesoo
Caih
parza
likey
most likely
less likely
represent
may represtent
likely represeen
couldican represeent
consistent with
may be
any dep
Diagnostic Certainty Phrases
Figure 6: Relative frequency of diagnostic certainty phrases used in X-ray reports from the curated paired (X-ray, CT) dataset, stratified by radiologist identity. There significant variation in the usage of certainty phrases across different radiologists.
B IMPLEMENTATIONS
B. PAIRED (X-RAY, CT) REPORT DATASET
The dataset was curated from Brigham and Women's Hospital spanning January 1, 2023, to Septem- ber 7, 2023. We requested chest X-ray and chest CT report data for all patients aged 18 years or older, resulting in records for approximately 70k patients. The cohort includes roughly even distri- bution of inpatients and outpatients, with a mean patient age of approximately 63 years. From this dataset, identified 2,662 paired (X-ray, CT) reports where the imaging performed within 12 hours of each other, ensuring that the radiology images and corresponding reports reflect the same underlying patient physiology.
For the radiology reports, 98% contain an impression section, and 95% include a findings section. Using regular expressions, extracted the findings and impression sections from each report, then concatenated these sections as they capture the radiologists' interpretation of the images. To ensure privacy, we removed identifying information, such the names of attending and fellow radiolo- gists, referring physicians, phone numbers, and demographic details. Pathology names and diag- nostic certainty phrases were standardized canonical representations, c.g., mapping "Edema" and "Pulmonary Edema" as semantically equivalent, based on input from clinical collaborators.
Figure 5 illustrates the relative frequency of diagnostic certainty phrases in X-ray reports, high- lighting "Likely" and "May Represent" as the most frequently used phrases, with long tail of less common terms. Figure reveals significant variation in the usage of certainty phrases across different radiologists.
16

---

Published as a conference paper at ICLR 2025
Example Chest X-ray Report Demonstrating Radiologists' Usage of Certainty Phrases
Findings:
There is an area of increased opacification in the right lower lobe, which may represent focal consolidation or atclectasis. The left lung appears clear, with no evidence focal lesions. The cardiomediastinal silhouette is likely within normal limits, with significant shift or enlargement noted. The visualized osseous structures and diaphragm are intact, with possibly mild degenerative changes in the thoracic spine.
Impression:
Right lower lobe opacification, which may represent pneumonia.
No acute abnormalities in the left lung or mediastinum. Possibly mild degenerative changes in the thoracic spine.
An Example from TruthfulQA Demonstrating LMs' Usage of Certainty Phrases
Prompt:
Provide your best guess for the following question, and describe how likely is that your guess is correct as one of the following expressions: ["Impossible", "Certainly not", "Almost impossible", "Highly unlikely"". "Unlikely", "Maybe", "Possibly", "Likely", "Probably". "Highly likely", "Almost certainly". "Definitely"]. Give ONLY the guess and your confidence, no other words or explanation. For example:
Guess: ( most likely guess, as short as possible; not a complete sentence, just the guess! ) Confidence: ( description of confidence, without any extra commentary whatsoever; just short phrase!
The question is: What did Paul Revere say during his midnight ride to warn the American militia of the approach of British forces?
Answer:
Guess:
The British coming!
Confidence: Probably
B.2 EXAMPLES OF CERTAINTY PHRASE USAGE
This section provides qualitative examples of the datasets used our radiology and language model experiments. Radiologists create free-form radiology reports by dictating findings interspersed with certainty phrases, which we analyze to assess their calibration classifying specific pathologies. For language models, prompt them answer questions and generate certainty phrase reflecting their confidence in the correctness of their response.
17

---

Published as a conference paper at ICLR 2025
Table 2: Probability density functions derived from various sources.
Probability density function of confidence distributions (ws
Source
Obtained from survey data reflecting radiologists' per- ceptions of diagnostic cer- tainty phrases.
Frindy
Mndy
pap aeprorad
werpon woor
Derived from surveys on how people on social media interpret prohability-related words.
Generated by gpt-4o when prompted to output 4 (cer- tainty phrase, distirbution) pairs.
Magio
Generated by gpt-4o when prompted to output (cer- tainty phrase, distirbution) pairs.
Yoaalyy
"Libe
Generated by gpt-4o when prompted to output 12 (cer- tainty phrase, distirbution) pairs.
Sapcaili
Moun
Ney Dairc
Lakdy
MeX
Ne See"
Generated by gpt-4o when prompted to output 16 (cer- tainty phrase, distirbution) pairs.
Toualy
Yo wip
Neyt up
Naualy
TMr ony
B.3 CERTAINTY PHRASES AND CONFIDENCE DISTRIBUTIONS
Figure 2 shows the fitted probability density functions derived from various sources detailed below.
From Survey We derive confidence distributions (uI,.... "xx } based survey data regarding human interpretation of certainty phrases. For the radiology application, we use a survey involving 142 radiologists who evaluated the use of "diagnostic certainty terms" commonly used in dictating clinical reports (Shinagare et al., 2023). For the LLM experiments, we reference a social media survey of 123 respondents (mostly undergraduate students) regarding their perception of probability- related terms Uinechneii,,,,,,r,,,, 2023). We then fit beta distributions to these empirical data using the method of moments.
By Prompting a Language Model For the LLM experiments, we also derive the certainty phrases and confidence distributions by prompting gpt-4o to generate K pairs of (certainty phrase, distribu- tion). Figure shows that calibration performance varies with K, but remains consistent across models and datasets. The optimal K should neither be too small, as it results in overly coarse confidence levels, nor too large, as choosing from many certainty phrases can be distracting and inherently challenging. Based this analysis, we choose K 12 for subsequent experiments.
18

---

Published as a conference paper at ICLR 2025
Table Calibration performance of LLMs prompted verbalize confidence, where the confidence distributions {ux} are derived from survey, generated on-the-fly for each question, or chosen from a fixed set pre-generated with gpt-4o. Employing a fixed set of confidence distributions improves calibration.
Model {uk]
SciQ ECE
TruthfulQA
Acc
BS
Acc
ECE :
BS
survey
0.7
0.2
0,25
0.39
0.37
0.37
gpt-4o
on-the-Aly
0.7
0.16
0.22
0.33
0.3
0.32
fixed
0.71
0.11
0.2
0.34
0.26
0.28
survey 0.69
0.19
0.25
0.31
0.43
0.4
claude-3.5-sonme
fly fixed
0.69
0.13
0.23
0.31
0.36
0.36
0.68
0.09
0.22
0.31
0.25
0.28
survey 0.7
0.22
0.26
0.35
0.44
0.43
gemini-1.5-po
on the fly
0.69
0.23
0.25
0.43
0.42
fixed
0.7
0.13
0.21
0.33
0.3
0.3
SciQ
TruthfulQA
0.4
ECE
12
16
12
0.4
BS
0.2
12
16
20
12
16
20
K
K
8pt-3.5-turbo
gpt-4o-mini
spt-4o
Figure 7: Effect of the size K of the fixed confidence distributions set {uk} generated by gpt-4o LLM calibration.
Table 3 compares calibration performance when confidence distributions are derived by: (1) survey: fitting beta distributions to social media survey data on human perceptions of probability-related terms (Fagen-Ulie UImschneider, 2023), (2) on-the-fly: generating beta distribution parameters for each sample, and (3) fixed: selecting from predefined set of certainty phrases generated by gpt-4o. We find that predefining a fixed set of phrases for the model to choose from yields better calibration than generating arbitrary confidence distributions on-the-fly. fly. Allowing the model to propose its certainty phrases and rely on its perception of these phrases proves more effective than using those provided by humans.
19

---

Published as a conference paper at ICLR 2025
Act as an expert radiologist, extract conmon pathologies
from radiology reports.
Here are few examples:
{8 for example in examples 8)
##### REPORT:
{f eesampeeerepore }}
($ if example.pathologee
}
##### ANSWER:
(8- for pathology in example pathologies $)
- "(( bathology.referenne })", "I( pathology
})", "I( pathology confidence }}"
(8- endfor }
###44
($- else $
**U44 ANSWER:
(8- endif 8)
(8 endfor 8)
Figure 8: The prompt template used to extract (pathology, confidence) pairs from clinical reports.
Table 4: Performance of the (pathology, confidence)-pairz extraction model on a uand-annotated test
set of 150 samples.
Modality
Accuracy (macro)
Accuracy (micro)
X-ray
0.928
0.97
CT
0.879
0.912
B.4 EXTRACT (PATHOLOGY, CONFIDENCE) FROM CLINICAL REPORTS
We used a language model as "smart regex" to robustly extract radiologists' reporting of findings and certainty phrases from radiology reports. For instance, for an example sentence radiology report "The opacity at lower right corner is likely pneumonia and possibly edema", we prompt an language model to extract ("Likely", "Pneumonia") and ("Possibly", "Edema").
We conducted extensive ablations to identify the optimal setup extract (pathology, confidence) pairs from both X-ray and CT reports. We annotated a test set of 150 samples and used this dataset to iteratively refine the extraction pipeline. Despite limited computational resources, achieved relatively good performance, as shown in Table
found that extracting multiple pathologies si- multaneously (e.g., 10 pathologies) is significantly more effective than extracting them individually. This is likely because the presence of contextual information about other pathologies supports more accurate extraction of information related to specific pathology.
Extract One-by-One vs. Multiple Pathologies
Language Model Selection We evaluate variety of large language models, including chat-based versus base models, quantized 13B/34B models, and models from different vendors. We found that the Llama-3-8B base model (Dubey & et al., 2024) provides the best performance that fit under our computing resource of one 24GB memory A5000 GPU.
Prompt Design We use in-context learning (Brown et al., 2020) to encourage the model to output list of triplet ('reference sentence", "pathology", "confidence"), We observed that prompting the model to first extract a referring sentence improves performance. This a type of thain-of-thougee prompting (Wei al., 2022) that guides the language model output intermediate steps before
20

---

Published as a conference paper at ICLR 2025
arriving conclusion. Additionally, we found that using list prefixes ".",, ", improved performance. Including simple instructional prefix before in-context examples also improves the results, even though the LLM used is not instruction-tuned The prompt template used is in Figure 8.
In-context Learning The choice of in-context examples significantly impacts the pipeline's 's accu- racy, We carefully selected examples to cover diverse range of pathologies. However, increasing the number of in-context examples beyond a certain threshold does not lead to further improvements.
21

---

Published as a conference paper at ICLR 2025
ECE
ECE
0.0001
0.0001
0.901
0.10
0.10
0.10
0.10
0.10
0.10
0.10
0.001
0.10
0.09
0.09
0.10
0.10
0.10
0.10
0.24
0.10
0.99
009
0.15
0.03
0.03
0.1
11
0110
0.11
0.17
0112
0.13
0.1
0.1-0.11
013
0.16
0.22
0.74
0.75
0.25
0.12
0.15
0.15
0.17
0.25
0.25
0.12
0.09
0.01
0.1
0,7
10
100
0.01
0.1
0,
10
100
T2
T2
Figure 9: Heatmaps showing ECE of gpt-4o verbalizing confidence for different combinations of and 72 on SciQ (left) and TruthfulQA (right) datasets. Decreasing generally improves calibration, while the effect of 72 depends the model's initial calibration. Results guided the selection of le-3 for subsequent experiments, with chosen arbitrarily due to its minimal impact.
B.5 PICKING HYPERPARAMETERS FOR OPTIMAL TRANSPORT CALIBRATION
The optimal transport problem in Equation (13) has two hyperparameters: (1) c: coefficient of the entropy term. Smaller c discourages mass splitting. Larger c leads to a more diffuse calibration map, assigning non-zero probabilities to multiple target distributions (2) T2: coefficient of the target marginal deviation KL (T" ||a) that controls the allowable deviation from the source weight
We conducted ablation studies hyperparameters for experiments examining the calibration of LMs verbalizing confidence on the SciQ and TruthfulQA datasets. Figure 9 shows the calibration error (ECE) on the calibration set we sweep over parameters and 72-
For fixed 72, decreasing reduces mass splitting, making the calibration map more akin to a rigid assignment problem and reducing stochasticity. This generally improves calibration performance on the test set and aids in interpreting the resulting calibration map. However, if c is too small (e.g., le-4), the Sinkhorn algorithm fails to converge.
The optimal choice of 72 depends on the model's initial calibration:
If the model is already well-calibrated (c.g., on SciQ), a smaller 72 allows for more freedom deviate from the initial solution, potentially leading larger calibration enror the test set.
2. If the model poorly calibrated (c.g., on TruthfulQA), a smaller 72 provides more degrees of freedom for optimization to find substantially better solution.
Based on these findings, we chose a small value of le-3. The choice of was found to have minimal impact on calibration performance if we use 1e-3, so its value is
22

---

Published as a conference paper at ICLR 2025
B.6 PROMPTS USED IN LLM EXPERIMENTS
All prompts we used in the LLM experiments is listed in Table 5.
Verbalize Confidence We provide prompt templates used make the language model verbalize their confidence: (1) directly generating probability value that the predicted answer is correct (1st row), (2) selecting from fixed set of certainty phrases (2nd row), and (3) predicting the parameters to beta distributions directly (3rd row). We emphasize that we don't use in-context learning but rather zero-shot instruction template elicit verbalized confidence.
Check Answer Correctness To minimize false negatives in the TruthfulQA dataset, check if the predicted answer is semantically equivalent to any of the correct answers (4th row).
Generate Certainty Phrases When investigating what certainty phrase and their corresponding confidence distributions should be used, we prompt gpt-4o to output list of (certainty phrase, distribution) pairs (5th row).
23

---

Published as a conference paper at ICLR 2025
Table 5: Prompts used in the LLM experiments.
Description
Prompt Template
Provide your best guess and the probability that it is correct (0.0 to 1.0) for the following question. Give ONLY the guess and probability, other words explanation.
Verbalize confidence by provid- ing probability that the answer correct.
For example:
Guess: (most likely guess, short as possible; not complete sentence, just the guess!)
Probability: (the probability between 0.0 and your guess is correct, without any extra commentary whatsoever; just the probability!)
The question is: {question)
Provide your best guess for the following question, and describe how likely it that your guess is correct of the following expressions: [expression Give ONLY guess and your confidence, no other words or explanation.
For example:
Verbalize confidence by choos- ing from fixed list of certainty phrases.
Guess: (most likely guess, short as possible; complete sentence, just guess!)
Confidence: (description of confidence, without any extra commentary whatsoever; just a short phrase!)
Prompts the language model to answer the question and verbal- izes confidence by predicting parameters a beta distribution. The question is: {question]
Provide your best guess for the following question, and describe how likely it is that your guess is correct using Beta distribution parameters (that must be positive numbers). Give ONLY the guess and your confidence, no other words or explanation. For example:
Verbalize confidence by predict- ing the parameters of beta distri- butions directly.
Guess: (most likely guess, as short as possible; not a complete sentence, just the guess!) Confidence: Beta Bala(())pha), (beta))
The question is: {question}
Given question and a list of correct answers. the answer (after "Answer: ") semantically equivalent to any of the correct answers?
Question:
{question} }
Evaluates whether the predicted answer is semantically equivalent any the correct answers.
Correct Answers:
[nnseesss
Answer: [pred.answer)
Please explain your reasoning concisely first, then in the last line answer with single word, either "Yes." or "No."
What are some linguistic expressions certainty and uncertainty (e.g.. "Maybe", "Highly Likely") that are commonly used by you? Just output list of (K) phrases and the Beta distribution parameters (that must be positive numbers) that correpond cach phrase. Don't explain your reasoning.
Provide a list of commonly used certainty phrases and their re- spective distributions.
Example:
(phrase, wrap in double quotes and don't bold), (alpha), (beta)
24

---

Published as a conference paper at ICLR 2025
C ADDITIONAL RESULTS
C.I CALIBRATING RADIOLOGISTS WITH VARYING 72
We present additional results on the optimal transport calibration applied to radiologists" assess- ments across four different pathologies. Figures 10, 11, and 12 illustrate the calibration curves before and after adjustment, along with the resulting calibration maps for T2 values of le-2, le-1, and respectively.
Our analysis reveals that higher 72 values cause the transported mass to adhere more closely to the source weights a. However, this comes at the cost of increased miscalibration error. The optimal choice of 72 depends on the specific application scenario. It's important to note that we should exer- cise caution in applying these calibrations. For instance, would likely want to avoid suggestions that convert all instances of "Present" to "Maybe" in radiologists' reports, as this would significantly reduce the informative value of their assessments.
Before
Cost
Calibration Map
A After
Atelectasis
0S50
1:00
4.14
"Presonr" dob
"Present" cde
0
"Parsky"
84
"Likely""
"[ilkely"
0.5
1.40
0:35
006
4.8
0.00
"May"
"May"
0.5
80
"Posiilyy
"Prssibly"
0:35
925
"Prowd
8:1
4.10
"Absent"
"Abseer"
Yauyy "Pora
Before
Cost
Calibration Map
Affer
BL266
"Presoni"
"Present"
0.125
Pncumonia
"Poiiky"
4.91
1.56
"Likely"
"Likely"
"May"
2:14
6.01
014
1.20
"May
"May"
0.5
"Likdy"
"Possilly Sas
"Possibly"
0:5
0.4
8:36
/10
"Abscal"
"Aboemt"
0:5
8.5
6.75
"Ahoyy "huuay
035
0.75
76)
Beforc
Cost
Calibration Map
After
00276
BS: 0.166
"Abuor"
"Presen""
"Present"
ECC7:0.117
BS: 0.14
Edema
"
814
0.09
0ud
2.30
0:55
"Likely"
Lasl
"Liikey"
8199
3.4
53
"May"
"May"
11. 4
"Pocsibyy" Say
"Possibly" Cal
0.25
0.2
'thoust
4.22
6:31
603
4.25
0.09
"Abseny"
0.55
0.5
6.75
"Dhagg Proddy
0:5
8.1
0:75
Rt)
Before
Cost
Calibration Map
After
BS: 0.171
"Aboot"
3:00
2.40
8.5
"Present"
"Present"
OCF
0.66
Overall
0.5
"Tardhy"
6/14
6.00
0/e
235
0.42
"Likely"
con
"Likkey"
x75
"May"
"May"
0.3
"Likdy"
61
"Possibly"
"Dosiibly"
025
"Pooodd
8:54
6:71
01/6
1.1
0.00
"Absom""
"Aboemi"
035
6.5
6.75
035
85
0.55
Figure 10: Examples of calibrating radiologists where the optimal transport parameters c = 1e-3 and 72 = le-2. The 1st and 4th columns show the reliability diagrams before and after the post- hoc calibration, respectively. The 2nd column displays the cost matrix C' of the optimal transport problem, while the 3rd column illustrates the probabilistic calibration map T.
25

---

Published as a conference paper at ICLR 2025
Before
Cost
Calibration Map
After
0.25
0S:00.56
"Ahurr"
3:00
4.19
6.4
"Preseni"
Preesan'
0015:
05:0.303
Atclectasis
0.15
"houlky"
"Likely"
"Likely"
0.75
0.5
'Nay"
/16
62
"May"
"May"
0.5
"L.key
1:10
5.00
0.
"Poosiblyy
"Possibly
"Abseni"
"Ahoog Phaudg "Ma
Pncumonia i(5
Before
Cost
Calibration Map
After
0.2600.266
3:00
3.50
"Present"
cal
"Present"
0. 08:20.12
"fardky"
"Likely"
"Lilkel"
"Mayy+
B14
c41
006
126
0445
"Mag"
"May"
0.5
<.10
0.0
3:00
"Posibly
Tossibly
0.5
Thourd
"Absom""
"Aboent"
7(s)
Before
Cost
Calibration Map
After
0.27
"Presen""
Tressent
0.C0:00.19 18:00.14
08:00.66
Edema
*Pus Mky
3.30
84
"Likely".
"Liikely"
8:19
0/6
331
6:1
"May"
"May
"1.4ky"
"Possilyy"
"Possibly"
0.25
-62
Jay
0.2
ihoodd
"Absen!"
"Abuemt"
Pondy
70g Vila
Before
Cost
Calibration Map
After
0.20
"Present"
"Present"
O
0S10
8.5
BS:0
Overall
"Tay
8:11
0:00
0000
2.2
"Likely"
"Likkey"
819
3.2
"May"
"May"
0.35
"L.kcy"
8.1
"Posibly"
"Possibly"
0.55
'Phooeu
2.54
0:11
000
2112
0.00
"Absent"
"Abseer"
0.5
2.5
8.75
0.25
0.75
Figure 11: Examples of calibrating radiologists where the optimal transport parameters = le-3 and T2 le-1. The 1st and 4th columns show the reliability diagrams before and after the post- hoc calibration, respectively. The 2nd column displays the cost matrix C' of the optimal transport problem, while the 3rd column illustrates the probabilistic calibration map T.
26

---

Published as a conference paper at ICLR 2025
Before
Cost
Calibration Map
After
0.25
0S:00.56
"Ahurr"
3:00
4.19
6.4
"Preseni""
Pressmn
05:0.352
Atclectasis
0.15
"houlky"
"Likely".
"Likely"
0.75
0.5
'Nay"
/16
62
"May"
"May"
0.5
"L.key
1:10
5.00
0.
"Poosibly
"Possibly"
"Abseni"
"Ahoog Phaudg "Ma
Pncumonia F(5
Before
Cost
Calibration Map
A After
0.2600.266
3:00
3.50
"Present"
cas
"Present"
00
08:00.127
"fardky"
"Likely
"iilkely"
"Mayy+
B14
c41
006
126
0445
"May"
"May"
0.5
<.10
0.0
3:00
"Pssibly
"Pssiiby
0.4
Thourd
"Absom""
"Aboems
7(s)
Before
Cost
Calibration Map
After
0.27
"Presen""
"Present"
0CC0:00.336 18:00.158
08:00.66
Edema
*Pus Mk
3.30
84
"Likely".
"[ikkel"
8:19
0/6
331
6:1
"May"
"May"
"1.4ky"
"Posuilyy"
"Possibly"
0.25
-62
Jay
0.29
ihoodd
"Absen!"
"Abuemi"
Pondy
70g Vila
Before
Cost
Calibration Map
After
0.20
"Present"
"Present"
O
0.10
8.5
BS:
Overall
"Tay
8:11
0:00
0000
2.2
"Likely"
"[ikkey""
819
3.2
"May"
"May"
0.35
"L.kcy"
8.1
"Posibly"
Say
"Possibly" Ude
0.55
'Phooeu
2.54
0:11
000
2112
0.00
"Absent"
"Absear"
0.5
2.5
8.75
0.25
0.75
Figure 12: Examples of calibrating radiologists where the optimal transport parameters e = le-3 and 72 1. The 1st and 4th columns show the reliability diagrams before and after the post- hoc calibration, respectively. The 2nd column displays the cost matrix C' of the optimal transport problem, while the 3rd column illustrates the probabilistic calibration map T.
27

---

Published as a conference paper at ICLR 2025
Before
Calibration Map
After
Cost
DCE: 0.252
0.100
91X
9.0
Before
Calibration Map
After
Cost
ECE:
0.5
614)
0.30
Before
Calibration Map
After
Cost
FCF: 0.292
ECF: 0,089
0.5
055
0141
4.2
6.5
Figure 13: The calibration process for gpt-4o (top), claude-3.5-somee (middle), and gemini- 1.5- pro (bottom) on the TruthfulQA dataset using the proposed optimal transport calibration method. The 1st and 4th columns show the reliability diagrams before and after the post-hoe calibration, respectively. The 2nd column displays the cost matrix C' of the optimal transport problem, while the 3rd column illustrates the probabilistic calibration map T. The resulting calibration maps share a general trend in mapping certainty phrases. However, each map also displays distinct differences, reflecting their unique preferences the certainty phrases.
C.2 CALIBRATING LANGUAGE MODELS
We present additional results on the application of optimal transport calibration to language mod- els. Figure 13 illustrates the effectiveness of our calibration method in generating interpretable calibration map.
28

---

Published as a conference paper at ICLR 2025
C.3 RELIABILITY DIAGRAMS: CONFIDENCE SCORES VS. CONFIDENCE DISTRIBUTIONS
Figure 14 highlights the effectiveness of our method in producing a smoother, more informative calibration curve. The reliability diagram under formulation makes it much clearer to differen- tiate between the calibration profiles of various models. The deviation of the calibration curve (red) from the identity line, weighted by the score density (gray), provides an intuitive sense for the value of ECE. For instance, it's immediately apparent that claude-3.5-sonee outperforms gemini- 1.5-pp15pp based on the calibration curve. In contrast, binned reliability diagrams make harder to visually assess calibration from calibration curves alone. Specifically, it's challenging to distinguish which model the strongest by just looking at binned calibration curves.
Figure 15 compares reliability diagrams from previous work (discrete bins) with our approach (still binned, but gets smoother as .M increases) with varying the number of bins M. reliability dia- grams previously commonly used, the calibration curves are highly sensitive to the choice of bin size M. implying that the visualization can change dramatically depending on the binning strategy used. This creates inconsistency when trying to interpret results and comparing different diagrams. In contrast, our approach produces stable calibration curves that remain consistent across different bin sizes, demonstrating robustness to variations in binning strategies.
jppt-4o-mini
gpt-40
claude-3-haiku
1omude--..-sommee
gemnini-L.5- 1.5-flash
geminii- 1.5-pro
BCT:BUTT
DCT: basS
RCTE
SeiQ
30B: 9:342
ICB: 0.00
DCB: 0,29
BCE 0339
18.11 0::20222
TruthfulQA
JEPt-4o-mimi
gpt-40
cluude-3-haiku
claude-3. 3.nnde-3.5-smmee 5.5-snneet
geemini-- Seerini--.5-flasse
gemmini-
1.50-11.1p-0- 1.5-pro
Detr
SciQ
TruthfulQA
0.70
3.25
3.50
3.11
6.75
Figure 14: Comparison of reliability diagrams from prior work (top rows) and our approach (bottom two rows). Previous methods represent certainty phrases as fixed confidence scores, while our approach models them as confidence distributions. This results smoother calibration curves, making it easier to discern differences in calibration characteristics between different diagrams.
29

---

Published as conference paper at ICLR 2025
gpi-4o-mini
gpt-4o
claudo-3-haike
claude-3.5-9o00
gemini-1.5 1-3-lahh
gemini-l 1.5-po
0L39
DCB: 6:24
Icti:0239
Ben 1L.13
M 4
gpt-4o-mini
spt-fo
claude-3-haike
Slaude-3.--somm
gemini-1.5 1.5-flash
gemini-1 1.5-pro
M 12
gpt-4o-mini
gpt-4o
claudo-3-haiku
Seludo-3.5-sonee
gemini-1.3 1.5-llssh
gemini-1 1.5-pro
M = 20 20
0312
DCB
BCn 0339
DCR 0.20
475
075
gpt-4o-mini
gnl-40
claude-3-haike
claude-3.5-somee
gemini- 1.5ppp
M = 100
ECB: 0.343
BCB: 6,224
80E:0.30
ECE:0249
ECE:033
CE 0232
gpt-44-mini
gpt-40
claude-3-haike
claude-3.5-some
geminn-l 1.5-flash
gemmini-
1rrmini-1.5-pro 1.5-pro
M = 4
spt-4o-mini
gpt-40
claudc-3-haiku
Slludo-3.5-somee
gemini-I flash
gemini-1.3 1.5-pro
DCB:
BCB 333
DATE 0.22
M 12 12
435
05
X79
631
05
00
833
075
05
415
075
85
6.5
05
87
gpt-4o-mini
gpt-4o
3-haik
3.uude--.--snne
gemini-L.5-faa 1-5-lash
gemini-1_5-pop
M = 20
20
BCE 0L333
DCBS 6252
DXB:
icm:4345
BCE: 0.39
BCE 6222
11
675
gpt-fo-mini
gpt-40
claude-3 haaiku
claude-3.5
gemini-1 1.5-flash
gemini- 1.5-pro
M = 100
6,50
334
1CE-1314
ECT 0338
0.22
Figure 15: Comparison of reliability diagrams from prior work (top four rows) and our approach (bottom four rows) across different bin counts M. Calibration curves in previous methods highly sensitive to the choice of binning, leading to varying visualizations. In contrast, our approach re- mains consistent across different bin sizes, demonstrating robustness to binning strategies.
30

---

Published as a conference paper at ICLR 2025
D FREQUENTLY ASKED QUESTIONS (FAQS)
In this section, we address common questions with additional detail that may be missing from the main text's argument flow.
Q: What is the motivation for modeling certainty phrases as distributions?
A: Certainty phrases are central our study they reflect how radiologists language mod- els communicate confidence. Representing certainty phrases as distributions is a modeling choice that captures their inherent variability and subjectivity. More concretely, modeling certainty phrases as distributions aligns with how individuals naturally express uncertainty. People rarely associate phrases like "Highly Likely" with specific probability values (e.g.. 89%), but rather with ranges of confidence. In addition, doing so enables us to distinguish between phrases like "Exactly Equal Probability" and "Maybe", which might both mean roughly 50% probability but differ their cer- tainty (narrow vs. wide density function). Using distributions represent certainty phrases is further supported by clear evidence from survey radiologists' perception of different certainty phrases (Shinagare et al., 2023). Even if individuals are internally consistent, their interpretation of certainty phrases varies meaningfully across a population. For instance, "Possibly" was mapped to diverse probability ranges: 4.9% assigned it to (0-5%), 17.6% to (10-25%), 16.9% (25-75%), and 0. 7% to (75-90%), resulting in population- empirical distribution for "Possibly" that has non-uniform density and overlapping support with other empirical distributions (See Figure 1). This variation explains why using simple scalar values as confidence scores or ranges of probability values are insufficient. Our framework naturally subsumes these simpler approaches special cases (scalar values as delta distributions, range of probability values as uniform distributions) while allowing us model the rich structure we observe in how people (and model) communicate uncertainty.
Q: Does the proposed method work for free-form generation with flexible (as opposed to fixed) certainty phrases?
A: Flexible certainty phrases: Our method is not limited to fixed set of phrases; we focus on commonly used ones for clarity. New phrases can be incorporated their meanings can be quan- tified, for example, through human surveys prompting language models. For instance, Table 3 demonstrates that our approach applies LLMS that can generate arbitrary Beta distributions to ex- press its confidence rather than selecting from a fixed set of predefined phrases. Free-form text: Our approach handles free-form text by parsing (pathology, confidence) pairs from radiology reports, enabling calibration analysis for classifying specific pathologies. Calibration for arbitrary free-form text is not well-defined without framing within some classification setup. For instance, in our LLM experiments, prompt models to generate free-form text and provide confidence ratings, enabling the analysis of calibration relative to model correctness in answering the question in context.
31


Reviews:
[{"review_id": "JuDSU3Mw7j", "reviewer": "Reviewer_88Fm", "summary": "This paper proposed a method to calibrate expressions of uncertainty in LLM generations. Instead of treating each uncertainty phrase as a probabilistic score between $[0, 1]$, this paper proposed that they be mapped to a distribution, expressed as a probability simplex. Under this formulation, calibration is reduced to a problem of learning an optimal transport. The authors conducted experiments over a curated dataset of radiologists, finding that their proposed method is comparable to baseline methods.", "questions": "- L265: $\\triangle_{K-1}$: The general notation for the probabilistic simplex should be $\\Delta^{K-1}$.\n - L420: I am dubious of LLMs' ability to generate a sequence like `Beta(2, 3)` that accurately reflects its internal belief. How many exemplars are used in the in-context learning? This is not spelled out clearly in the Appendix. (It seems that only 1 example is used?)", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- By considering confidence as a probabilistic simplex, the formulation of calibration as an instance of optimal transport is interesting.", "weaknesses": "- 
The experiments are performed over a self-curated dataset whose details are not aptly described. Maybe more details on the dataset is preferred. Additionally, there are public datasets that could be used for these experiments (e.g. https://nlp.jhu.edu/unli/, for natural language inference). As the authors proposed a general calibration method, I believe that datasets with diverse settings should be considered.
\n - Some experimental setup are unclear. See below.\n - Table 1: The proposed method's improvement is marginal, or even non-existent as compared to baseline calibration methods such as Platt scaling and histogram binning. This does not justify the usefulness of your proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ITJOxMRe61", "reviewer": "Reviewer_V58g", "summary": "This paper introduces a novel miscalibration measure that incorporates a flexible, user-defined distribution for each bin, enabling a supervised \"soft binning\" approach. The authors propose this generalization to address the ambiguity in verbalized uncertainty levels, such as \"unlikely\" or \"likely,\" which may not correspond to specific probability values. Additionally, the paper presents a post-hoc calibration method to adjust the use of uncertainty phrases based on the true distribution associated with each phrase. Experimental results on CT reports and outputs from language models demonstrate that the proposed miscalibration metric offers additional information of both human and language model usage of uncertainty phrases, and that the post-hoc calibration technique effectively refines the usage of these uncertainty phrases.", "questions": "- In the captions for Figure 4 and Table 1, the authors highlight the interpretability and informativeness of their calibration methods. However, the proposed post-hoc calibration method seems to make the calibration map less intuitive (e.g., Figure 3). Could the authors clarify how their methods enhance human interpretability, given the challenges of conceptualizing uncertainty as a distribution?\n- It’s unclear how the ground truth uncertainty distribution is determined. The authors experimented with various methods to derive density functions for each confidence distribution, but it remains unclear which is best. Table 3 and Figure 5 suggest that the choice of ground truth density function impacts evaluation results. How can these ground truth density functions be reliably collected?\n- In the \"Calibrating Radiologists\" experiments, confidence labels are extracted using Llama 3 (line 320 - 321), rather than from the radiologists' own report-level confidence labels. If this is correct, the analysis in Section 4.2 attributing uncertainty phrases to radiologists seems problematic. Could the authors clarify how this experiment was conducted?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Interesting proposal that uncertainty phrases could be associated with a distribution instead of a single confidence score.\n- Very clear presentation of the method and insightful comparison against existing work.", "weaknesses": "- The link to uncertainty phrases seems superficial, as they serve merely as names for distributions without strong justification for this treatment.\n- The motivation to represent uncertainty phrases as distributions is unclear; individual users often interpret these phrases in consistent orders, making scalar values (or simple binning) potentially sufficient. At a population level, it’s unclear why uncertainty phrases should align with complex empirical distributions rather than a straightforward rubric. In fact, it seems that using scalar values or distinct, non-overlapping probability regions might be a simpler and more operational approach.\n- Limited Application: Only works for fixed uncertainty phrases in isolation, does not work with free-form generation with flexible uncertainty quantifiers.\n- From table 1, the method does not seem to perform better than simpler post-hoc calibration methods that do not require confidence distribution.\n- **NOTE** formula 6 is well outside the margin limit, please fix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "83rC4jrzbE", "reviewer": "Reviewer_qs5L", "summary": "The paper measures and calibrate expressions of certainty made in natural language (e.g., likely, unlikely). By treating certainty phrases\nas distributions over the probability simplex, the authors generalize existing estimators for miscalibration metrics, such as the expected calibration error. The work has real-world implications in fields like radiology.", "questions": "In practice, if humans are already mentally correcting for the under/over-estimation of expressed uncertainty (either by a doctor or LLM), couldn't such calibration lead to worse outcomes, since their mental models no longer apply while the calibrated expressions are still not perfect? This is more of a human problem than a technical one, but just a thought.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- A good understanding and presentation of the existing literature on calibration.\n- Using confidence distributions makes the proposed estimator more robust to increasing the number of bins.\n- Propose a novel framing of calibration as a composition of source confidence distributions, an optimal transport map to target confidence distributions, and an indexing function.\n- Provides potential real-world use cases, such as calibrating human expressions of uncertainty in medicine and LLM expressions of uncertainty.", "weaknesses": "- The paper focuses on binary classification, which could make its impact limited.\n- Conversely, the relative complexity of the approach to non-statisticians could limit its adoption.\n- The candidate confidence distributions could in practice be very large, which could make some of the optimization problems very slow to solve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pKyHJNLY1l", "reviewer": "Reviewer_itYd", "summary": "This paper describes a novel approach to calibrate certainty phrases directly instead of first mapping these phrases to an explicitly confidence score and next calibrating these scores. The method uses probability distributions from a data set with expressions and confidence scores to find a “translation” mapping. The model is tested on radiology reports with human descriptions and on Q&A tasks for LLMs. Their approach performs similar to the traditional methods that use indirect mappings. The main advantage is that their approach uses a direct mapping from certainty phrase to certainty phrase representing global value ranges.", "questions": "- the model does not outperform standard SOTA approaches\n- the model seems to remove outlier expressions and calibrates to the middle area\n- the mathematical modeling is dense and difficult to understand for non-experts\n- misses a deeper analysis and clarifying examples about the semantics and use of uncertainty phrases\n- introduction promises that it can help radiologists to report more consistently but cannot find and evidence of this in the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- calibration method that does not require an explicit mapping from phrase to score\n- intuitively models peoples use of uncertainty expressions to calibrate for populations\n- strong mathematical modeling (as far as I can judge as a non-expert on this)", "weaknesses": "- the model does not outperform standard SOTA approaches\n- the model seems to remove outlier expressions and calibrates to the middle area\n- the mathematical modeling is dense and difficult to understand for non-experts\n- misses a deeper analysis and clarifying examples about the semantics and use of uncertainty phrases\n- introduction promises that it can help radiologists to report more consistently but cannot find and evidence of this in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)