QA-Calibration of Language Model Confidence Scores
Authors: Putra Manggala
Atalanti A. Mastakouri
Elke Kirschbaum
Shiva Kasiviswanathan
Aaditya Ramdas
Venue: ICLR 2025 Poster (2025)
Keywords: Calibration
Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Presents a concept of grouped calibration along with corresponding algorithms designed for language models.

Abstract: To use generative question-and-answering (QA) systems for decision-making and in any critical application, these systems need to provide well-calibrated confidence scores that reflect the correctness of their answers. Existing calibration methods aim to ensure that the confidence score is, *on average*, indicative of the likelihood that the answer is correct.  We argue, however, that this standard (average-case) notion of calibration is difficult to interpret for decision-making in generative QA. To address this, we generalize the standard notion of average calibration and introduce QA-calibration, which ensures calibration holds across different question-and-answer groups. We then propose discretized posthoc calibration schemes for achieving QA-calibration. We establish distribution-free guarantees on the performance of this method and validate our method on confidence scores returned by elicitation prompts across multiple QA benchmarks and large language models (LLMs).

Published as a conference paper at ICLR 2025
QA-CALIBRATION OF LANGUAGE MODEL CONFI- DENCE SCORES
Putra Manggala University of Amsterdam putra. Intrramaangaaalr 1agggalll@mmmlllm com
Atalanti Mastakouri, Elke Kirschbaum Shiva Prasad Kasiviswanathan Amazon
Aaditya Ramdas
Amazon and Carnegie Mellon University
ABSTRACT
To use generative question-and-answering (QA) systems for decision-making and in any critical application, these systems need to provide well-calibrated confidence scores that reflect the correctness of their answers. Existing calibration methods aim to ensure that the confidence score is on average indicative of the likelihood that the answer is correct. We argue, however, that this standard (average-case notion of calibration is difficult to interpret for decision-making in generative QA. To address this, we generalize the standard notion of average calibration and introduce QA-calibration, which ensures calibration holds across different question-ant cesstion-and-nnsee and-answer groups. We then propose discretized posthoc calibration schemes for achieving QA-calibration. We establish distribution-fres guarantees on the performance of this method and validate our method on confidence scores returned by elicitation prompts across multiple QA benchmarks and large language models (LLMs).
INTRODUCTION
Language models (LMs) built on transformer-based architectures are capable of producing texts that both coherent and contextually relevant for large range of applications (Brown et al. 2020 Chowdhery et al. 2023 Achiam et al., 2023). In question-ande crrsstion-and-answeein (QA) systems, these models generally perform well, but occasionally produce inaccurate answers phenomenon generally referred as hallucination (Huang et al., (023). Confidence estimates that are paired with the answers can be used as an interpretable indicator of the LM's accuracy (Steyvers et al. (024)). But for this, these confidence scores have to be well-calibrated, i.c., match the actual accuracy of the model.
To evaluate whether the obtained confidence scores are actually well-calibrated, common criterion is the expected (average-case calibration error (Tin et al.) 2023) Xiong et al.) 2024). Suppose a model claims that its answer has confidence of p. Based on only this answer, it possible know whether this confidence was well-calibrated or not. But when considering multiple question-answer pairs-say No pairs each with a claimed confidence p. we can verify how many answers are actually correct and measure the error between the claimed confidence and the model's accuracy. Averaging over these errors quantifies of the model's confidence scores
While this average-case calibration measure makes sense for models trained and evaluated specific tasks, its applicability to generative QA is questionable because it averages over all QA pairs. This is because generative QA systems can be applied across various domains and topics, such as geography. politics, medicine. Consider, for instance, the QA pairs shown in Figure [ On average, this model has a calibration error of 0.5. From User I's perspective, the calibration is much worse, with an error of 0.8. Meanwhile, User 2 has a completely different experience, finding the model to be
Work partially done when author was an intern at Amazon. Note that the accuracy of the LM itself unaffected by calibration, as calibration does not change the weights of the LM model.

---

Published as a conference paper at ICLR 2025
User (interested in countries)
User fintresteed politics)
Question: What timezone is Toronto, Canada?
Question: Does Singapore have prime minister?
Answer: CET
Answer: Yes
Confidence: 0.8
Confidence: 0.8
Label: Incorrect
Label: Correct
Question: Which currency used Hungary?
Question: How many years has Justin Trudeau been prime minister?
Answer: Lira
Answer:
Confidence: 0.8
Confidence 2: 0.8 Label: Correct
Label: Incorrect
Calibration error for User - 0.8
Calibration error for User 2 0.2
Figure 1: Two users interact separately with an LM by inputting questions and obtaining answers and confidence scores from the LM. The LM could be calibrated on average across user types, but each individual user may not have calibrated confidence scores. See Example 2. [2. more details.
much better calibrated than indicated by the average calibration error. This motivates our notion of QA-calibration, where the calibration target is conditional the group of the question-and-answerr pair.
Previous works have explored group- calibration for classifiers, using pre-specified groupings of their covariates (e-g., race or gender) (Kleinberg et al. 2017 Pleiss al. 2017). However, is clear how this idea can be transplanted to the generative QA setting.
A common approach to obtain calibrated confidence scores in LMs is to use confidence elicitation via prompting (Tian et al. 2023; Xiong et al.) 2024). The advantage of this approach that it can be executed with only black-box access to LMs However, the issue with pure elicitation via prompting approach is that the performance is sensitive to choice of prompts and model (Sclar et al., 2024). and it does not have any rigorous calibration guarantees, These issues can be mitigated by performing posthoc calibration of the elicited confidence scores. Our proposed approach is such posthoc calibration method.
A limitation of relying on LM-elicited confidence directly, or posthoc calibrated using temperature scaling TTian et al., 2023), is that the output probability is not discretized, making performance difficult to assess (Kumar et al. 2019). We overcome this problem by developing posthoc QA- calibration schemes on elicited confidence scores that use ideas of (histogram) binning (Zadrozny & Elkan 2001) and scaling-binnic (Kumar al. 2019) that are by construction discretized.
Our Contributions: make the following contributions in this paper:
1. We introduce QA-calibration, a principled and interpretable notion calibration in genera- tive QA. Crucially, it defines fixed mapping 3 from all possible question answer pairs finite set. Our QA-calibration notion generalizes the standard average-case calibration notion by requiring calibration conditional on B.. Due to this conditioning on 8, the guaran- tee of QA-calibration is stronger than the standard calibration guarantee. By instantiating this framework with different B's, we have the flexibility of defining question-ant andoon--nd-nnweeeee groups across which we would like calibration guarantees to hold. In this paper, we present instantiation of / kd-tree, an adaptive multivariate histogram method that provides cohesive groupings of QA pairs.
2.
We propose two posthoc calibration techniques for QA-calibration: (a) QA binning and (b) scaling QA binning. The latter uses the former as a subroutine. This is particularly useful in the practical setting where some QA induced groups may lack data. We show that both methods satisfy distribution-fiee approximate QA-calibration guarantee. In both cases, the approximation level is used to decide on the number of points per bin, a key hyperparameter.
3. Finally, we experiment on newly developed elicitation prompts across 5 commonly used QA benchmark datasets of various sizes. We find that QA binning and scaling QA binning achieve lower QA-calibration error than elicited confidence scores and other baselines. We further justify our framework by demonstrating its performance in a downstream selective QA task. Our posthoc calibration algorithms achieve around 10-40% increase in QA- calibration performance and up 30% increase in selective answering performance.
*Recent research indicates that, even in setting where has access to token-based likelihood, it does necessarily capture the overall semantic uncertainty (Kuhn al. 2023)
2

---

Published as a conference paper at ICLR 2025
Question q
Answer
Confidence h(q,
P(Y 1/g.a)
911
a1l
0.8
0.6
412
Q12
0.8
0.7
Q21
0:1
0.8
0.9
422
a22
0.8
1.0
Table 1: QA pairs with confidence and correctness for Example 2.1
DEFINING QA- QA-CALIBRATION
2.1 NOTATION
We first define the question-ande and-answeringg process with verbalized confidence elicitation. In the most minimal form interaction, a question q, embedded into a prompt using a prompt function m(g). The user obtains an answer c(m(q)) from an answering function c Q - A. Note that c could be randomized function, typical LMss are. Confidence elicitation (Xiong et al.) 2024 TTian ict al. 2023) enhances the prompt to let the user obtain confidence ((m)()), a) from the confidence function h Q x A -> [0. 1]. Note that both answering and confidence functions are implicit in the LM interaction. In the following, for simplicity, we omit the dependence on prompt function m, and use <(q) for ((m(g)) and (g.a). a) for h(m(4)) a). We assume that we have access to the binary ground truth y for each pair (q. a). which indicates whether the answer is correct question q. Our N-element dataset then D ((g,, ai, h;s ys) [E[N] where qi the ith question, a, c(q:). h; h(g,, a:) and y; the label which indicates whether the answer a, correct for the question q:- For more details, refer to Table 3 We assume that each instance (g. a, h, y) of D is an i..d. realization of r.v. (Q, A c(Q), H A), drawn from fixed distribution P over the
1) for the
Q x [0, x [0, 1] where
P: Pq x PAte x PHiga x Pylga- Definition 2.1 (Calibration). We say that is calibrated for distribution Pf:
E Y A)
pl a.s. for all p [0.
(2)
In words, that the conditional distribution Y conditional the prediction that h(Q, A) p Bernoulli distribution with bias p. the following, we refer to Eq. (2) as average-case calibration to distinguish it from QA-calibration.
While perfect calibration is impossible in finite samples, standard measure for calibration error of h defined as follows.
Definition 2.2 (Expected (Average-case) Calibration Error). The expected (average-case) calibration enror of h is defined as:
CE(h) Eq.A [ A(Q, h(Q, A)ll
2.2 QA-CALIBRATION INSTEAD OF STANDARD CALIBRATION
In practice, user who obtains an answer to their question q with confidence p wants know the probability that y = among "similar" (q, a) pairs with confidence (e.g., within the same topic of interest). A perfectly calibrated h however, may not satisfy this requirement, as we illustrate in the following cxample
Example 2.1. Suppose the dataset consists of two sets of QA pairs, 31 = {(g ((g.i,nn). (912. 1 and B2 ((221, (22)), (122, (22)) with confidence function h and P(Y 1/4, a) shown in Table
can easily be seen that h is perfectly calibrated according to Eq. 2 when we assume that each QA pair have the same sampling probability. However, when we consider the sets B1 and B2 separately, each set well-calibrated.
1
1
E| (i(g, a) 0.8, 81]
(P = 1 911 and +P = 1 912, (12)) = (0.6 + 0.7) = 0.65,
2
2
1
1
E[Y 0.8, 8,] = (PlY =
221. a21 + P = 1 922. (22)) = (0.9 + 1.0) = 0.95.
2
2
As Y' is Bernoulii r.v., the LHS can also he written as Pr[Y = hxQ, A) pl. *We give the smallest example where each group has than pair, since we illustrating group calibration, but the argument holds when there is only one element per group
3

---

Published as a conference paper at ICLR 2025
Practically, when user is only concerned with a particular set 33 B2, the calibration claim does not align with reality. For Bi, h is overconfident, and underconfident for 32.
In the above, the expectation E|y h(g, a)] is not interpretable from a prcision-making perspective. The space of generative QA pairs 2 x A intuitively too large for user, particularly since they are typically only interested in a smaller subset of (g, a) pairs. We hence consider a generalization Eq.
(2) necessary to perform adequate calibration for generative QA.
Definition 2.3 (QA-calibration).
QA calibrated for the distribution in Eq. 1 if:
E Y h(Q, A) p. B(Q, A)] P.
a.s. for all p E [0, IJ. where 3 Q x A S is a fixed mapping to some embedding space S.
In this work, we focus on finite set and /3 is a (deterministic) discretization scheme that maps (g, a) to a finite set S, i.c., 3 induces a partitioning of the QA space according to the output of 3. Note that this QA-calibration reduces to calibration in Eq. (2) if B(q, a) is the same for all (g, a). Intuitively, 3 is chosen such that the pre-image a specific value of B represents a grouping that an end-user might be interested in.
question-and-ansee
Going back to Example 2 CA-calibration would tell us the following: "among question and answer pairs with confidence p that are mapped to the same 3 value (as the user's (g, a) pair), the fraction who are correct (y = 1) is also p". Further, we also extend Eq. (3) to propose an error metric for QA-calibration:
Definition 2.4 (QA-calibration error). The QA-calibration error of h with fixed mapping ,3 is defined as:
CE(h; = EE.AA ]I[[ ((Q, A)] h(Q. A)].
2.3 GENERALIZING (HERAGE-CASE) CALIBRATION VIA KD-TREE INSTANTIATION OF
The definition of QA-calibration (Definition 2.3) requires predefined | mapping. While our schemes developed in Section can work with any arbitrary / mapping, propose a general approach for 3 in the language model QA setting: the embed-then-bin method. This involves computing vector embeddings from question-and -add--nndd-nrrr pairs and then creating multidimensional histogram over these embeddings. The range of is then set to index over the histogram bins of the embeddings. We write B = Arin Bemb, where Bemb Q x A + RM computes an M -immenniional embedding and Boin R M + S computes the index of the embedding histogram bin containing the embedding.
Specifically, we use the [CLS] token embedding from the pre-trained DistilBERT model (Sanh
et al. 2019): Bemb Q x A + R 768 (M = 768). As for the histogram, we adapt kd-tree (Bentley 1975) to bin each vector to an integer representing the index of the partition containing the vector. Chin 768 + S, where S is the set of partition indices. See Appendix C for details on our kd <d-tree construction. An important hyperparameter is the maximum depth of the tree d, which determines the number of partitions.
While DistiBBERT is designed to be smaller and faster than alternatives, it still able to generate high-quality contextual embeddings that are rich in semantic information. Next, a kd-tree performs efficient and adaptive binning for the high-dimensional embedding space by successively splitting along different dimensions. Each partition of the tree contains semantically-sinne similar (q, a) pairs that are "near" each other in the embedding space.
The choice of kd-tree generalizes the standard calibration when d = CI(h;; (9) reduces CF(A). The hyperparameter d should be chosen based on the downstream metric that needs to be optimized. In the experiments section, we further describe how Aoin are constructed.
3 ACHIEVING POSTHOC QA-CALIBRATION
As discussed earlier, an elicited confidence score function (h) of LM-model is not guaranteed to be QA calibrated (or even calibrated under average-case). For any / Q x AA + S, the goal of posthoc QA-calibration is to design scheme that that can take any f and transform it to a QA calibrated confidence score. We wish to learn a posthoe calibrator g: [0, 1] - [0, 1] such that g o is (approximately) QA calibrated using calibration dataset D = {(q,, a,, h4, heeN. Additional missing details from this section are collected in Appendix A
4

---

Published as a conference paper at ICLR 2025
Algorithm I: QA binning: Train-time Subroutine
Input: Calibration data: D = {(g a, h4 1.) (ee Hyperparameters: Minimum number of points
per bin E N (say 50), tie- breekre parameter 5 > 0 (default: 10-10)
Output: Hash table G of fitted UMD calibrators gUMD'S ss keyed by partition index E & 1 First construct UMD calibrator for points outside of bounded kd-tree spaces and store in g with a default key, say root //
1: @D t- UMD(D, B [D]/b]] 6].
2: G[ooot] 8 9D
3:
for E S do
4:
D. ((h; B(qi az) = s}ie[n]
5:
/ Df is empty then continue to the next %/
6:
n2 < [Dal
7:
G[s) UMD(DD = D., B Lm,/bl, 8 6)
8: end for
9: return g
Our recalibration schemes utilize the existing building blocks in posthoc calibration literature, like histogram binning (Zadrozny & Elkan, 200T) and scaling (Platt) 1999). The novelty of our recalibration methods lies how we generalize and combine these building blocks achieving the best of both worlds: our binning component generalizes histogram binning to any partitioning and enables distribution-fres guarantees, and our hierarchical scaling component reduces overfitting. enabling learning to be performed across partitions which may have varied number of data points.
Our schemes take the 3 function as input, and all our schemes and theoretical results hold for any B. In our experiments, we will use the kd-trees based 3 instantiation from Section 2.3 We focus on methods that output discretized scores, since this has been shown to be casier to assess (Appendix A.T].. We achieve this by first discretizing the QA space using / and then ensuring that for each QA partition E S we output a calibrated confidence score. We start by describing an adaptation of the classical histogram binning (Zadrozny & Elkan, 200T) idea for achieving QA-calibration. In Section 3.2 we build upon this to provide a more robust algorithm that also uses scaling before binning.
3.1 APPROACH 1: QA BINNING
QA binning is a standalone posthoc QA-calibration algorithm. It uses as a subroutine, uniform- mass-double-dipping histogram binning (UMD) (Gupta & Ramdas 2021). which we describe in Algorithm Informally, the UMD procedure partitions the interval J0. into bins using the histogram of h values from the calibration dataset D, ensuring that each bin has approximately the same number of calibration points. It returns a calibrator function JUMD that takes input an uncalibrated confidence score, then allocates it to one of the B bins, and returns the probability of label being 1 (estimated as the average of the y values from the calibration dataset D that are mapped into that bin).
In Algorithm 1 we present a subroutine that uses the UMD procedure to construct a different calibrator for each QA QA-partition, which we will invoke with different inputs. Algorithm [ takes as input dataset D = {(; ai, his (5:444) (s)]ej where h, the (clicited) confidence score for the answer a, and & is the (potentially noisy) label of the correctness of answer a; for the question q.- Algorithm I partitions the input dataset D to (D. (DEEE where D.- = {gg; ai, h;; 44) B(q,, a;) s). For each s E S. UMD is fit using D, to construct JUMD calibrator that stored in hash table g, keyed by s. We next describe the training and test processes of QA binning.
Training Process. We invoke Algorithm [ on our calibration dataset D, i.e., D = D. Hence, 4 in Algorithm| iss set to the ground truth ys. "The output is a hash table of UMD calibrator functions indexed by the entries in the range S of i3.
Test Process. For test input (Teats Ggest: Mest) we invoke Algorithm 2 A fitted UMD (SUMD) for this (grest, ases) pair is retrieved from the hash table & using p(q. a) as key, which then invoked to obtain a QA calibrated confidence score 9UMD (bies). In our kd-tree instantation, if g does not have a calibrator for 3(4uat (res), we use UMD to calibrate that pair. This corresponds to points that lie outside of the bounded kd-tree spaces (refer to Appendix C Since we can generate large amount of question-and and---nd---aeeee pairs without requiring ground truth, the number of test points in our experiments that fall outside the bounded spaces is negligible.
5

---

Published as a conference paper at ICLR 2025
Algorithm 2: QA binning: Test-time Subroutine
Input: Test question, answer and confidence score: (Nest Aess Mest) Hash table G Output: (Approximately) QA calibrated confidence score
if [(usa: (rest) E G then
2: JUMD + (((esees (rest]
3: return gUMD (hast)
4: else
5: / In this case, the test input (ftest, ateet) does not lie in any of the bounded kd-tree spaces*/
6: 9D g[root]
7: return 9D(Mes))
8:
end if
Note that while assumed here access to the true labels y 's, also operate QA binning with proxy labels generated by say another LM on QA pairs, as discussed more in following subsections. In practice, users are likely to define large S (for example, large maximum depth of tree d) in order to obtain more cohesive groups of QA pairs. UMD, and therefore QA binning, may overfit if the number of data points within D3 is small - a highly likely scenario if 3 induces very fine partitions. We overcome these issues with our next approach that combines scaling with binning.
3.2 APPROACH 2: SCALING QA BINNING
Scaling QA binning is a standalone posthoc QA-calibration algorithm, that is based performing a scaling step prior to QA binning. In the original scaling-binning approach (Kumar et al., 2019), confidence scores are first scaled to their (maximum likelihood) fitted values using logistic regression, and the fitted values are then used as proxy ground truth for histogram binning. Adapting this paradigm, our scaling QA binning, we use a scaling subroutine (defined below) to produce fitted values, which are then used as proxy ground truth for QA binning (e.g., in Algorithm is set to these fitted values).
We adapt the scaling procedure from Kumar et al. (2099) with a crucial change. The confidence score distributions in different partitions 8 E S may have different miscalibration profiles, since an LM may be underconfident for partition but overconfident for another. While scaling is useful for average-case recalibration (Platt 1999). for QA-calibration, we propose to use a hierarchical logistic regression model with partial pooling (Goldstein) 2011). Hierarchical model distinguishes between fixed effects (consistent across different partitions) and random effects (allowing for variations at different partitions). The latter explains the variability between partitions that may not be captured by fixed effects alone. Partial pooling allows for information sharing across partitions, which improves estimates for partitions with few data points. In our kd-tree instantiation, as the maximum depth parameter d increases, some partitions may have only a few data points.
We next describe the training and test processes of scaling QA binning.
Training Process. With an abuse of notation, let s/i) denote the partition index containing (q: a.). We define the scaler Iscaler
[0, 1] + [0, 1] via the following likelihood of y::
Y1 Bernoulli (logit" (Bo + Uata + (Bi + Voig)):))
where Bo is a fixed intercept, Usii is the random intercept for partition s[i], B1 is the fixed slope for confidence score h;, and Vaii the random slope for partition s(i]. On top of random intercepts, which assumes a different additive baseline per partition, we use random slopes which allow the relationship between the accuracy and confidence score to differ for each partition.
We first split the calibration dataset D (by default equally) into sets, D and D2 and invoke Algorithm 3 on D2 and D2 Similar to the QA binning approach (A (Algorithm D. this produces a hash table y of gUMD'S The partition of each instance determines which random intercept U, and slope V. to use. In our kd-tree instantiation. a point lies outside bounded kd-tree spaces, we set the random effects to zero, thus assuming that this point behaves similarly to the overall population average.
Test Process. Similar to QA binning, for test input (yestt ares Iess), we invoke Algorithm 2
Due to the scaling step. the computational cost of scaling QA binning training process is more expensive than that of QA binning. Both schemes have the same test process.

---

Published as a conference paper at ICLR 2025
Algorithm 3: Scaling QA binning: Train-time Subroutine
Input: Calibration data for scaling: D = ((at + at. h4. (t) hef(.)) Calibration data for binning:
D2 = =( at. kR. f2) [r Hyperparameters: Minimum number of points per bin E N (say 50); tie-breaking parameter å (default: 10-19)
Output: Hash table G of fitted UMD calibrators gUMD'S 's keyed by partition index s E
Osealer + Fit Eq. (5) on (()
2: / Construct proxy ground truth y. See Definition 3.2
3: {5t hiiinal - {gsealer (ha) Jae[nal
4: 9 Execute Algorithm [ with parameters D {(g2 at 11, (E) b,
5: return G
3.3 DISTRIBUTION FREE ANALYSIS OF QA BINNING AND SCALING QA BINNING
In the next result, we prove a high probability bound on the QA-calibration error (Definition 2.4 of the above schemes. To formalize the guarantee, we adapt the conditional calibration notion from Gupta et al. 2020)); Gupta & Ramdas (2021) to the QA-calibration setting, defined below
Definition 3.1 (Conditional QA-calibration). Let e, a E (0,1) be some given levels of approximation and failure respectively. Confidence function
A + [0, 1] is (c, a conditionally QA calibrated for discretization scheme 3 Q x A + S if for every distribution P defined in Eq.
P (V's 8,p range(h), [E[Y (MQQ,A) A) p, (0, A) = s]
S a
This is a distribution-
(DF) guarantee since they are required to hold for all distributions P over Q x A x (0. 1] x 10 without restriction. In order to estimate for both our posthoc algorithms, we permit label misspecification defined follows:
Definition 3.2. (Misspecified proxy ground truth) Let the random variable ý [0, 1] with distribution ()) be a proxy for ground truth Y. We constrain the misspecification in the following way: assume that there is some (minimal)
[0, such that for all p [0, 1].
max(E[y IQQ,, A) = p] v, 0) < Fly IMQ,, A) p) min(Ely MQQ A)=
1),
In practice, ÿ can come from two sources: 1) an LM-constructed ground truth (see ground truth proxy in Table 3). and 2) the fitted values from our step scaling QA binning. Let D (gi ai, h;, ".)))e))e [E[] denote a (proxy) dataset with samples from P= Po x PAIe x PH q.a X Pr|a.- To prove calibration guarantees for our schemes, we rely on the following result.
Theorem 3.1 (Distribution- free QA-calibration Guarantee). Consider an input calibration dataset Í defined above with misspecification factor from Definition 3.2 Assume that the h,'s are distinct, number of points per bin b 2 2, and number of instances within each partition n, 2 5 for every E S. The calibrator JUMD retrieved in Line 2 of Algorithm 2 trained using Algorithm I with input
D = D, is (c, conditiooal QA calibrated for any E (0, 1), with 6
[og(2N/bb))
2ib
The proof is Appendix A.3 The dependence on 1/ 1bv factor comes because the Algorithm delegates at least b points to every bin. We now discuss how Theorem 3.1 is applicable for both QA binning and scaling QA binning with different v's.
Applying Theorem 3.1 to QA Binning & Scaling QA Binning. In our description of QA binning (Section 3. D. we assumed is set to the ground truth y (in Algorithm TT. hence, by definition v 0.
Theorem 3. can also be used to choose b, see the plots for 0 in Figure
If the true labels are not available, then we can still use QA binning say by using an LM to produce proxy ground truth. In this case, the misspecification constant v depends on the data generating process of misspecified labels. When an LM is used to produce proxy ground truth, if there is hold-out set containing the ground truth, then a bound on can be estimated empirically.
In scaling QA binning, where ÿ is set to be the fitted values of a hierarchical logistic regression model, the magnitude of misspecification factor depends the goodness-of-fi of-fit of the fitted values. In
'One could define (c, a) -arrinaal QA-calibration: IP ( h(Q, A), ((, Conditional calibration is a stronger definition than marginal, it requires the deviation hetween TE(Y h(Q, B(Q, A)) and h(Q, A) to be most for every (s, r), including rare ones, not just on average.
A)) <
7

---

Published as a conference paper at ICLR 2025
Calforation approximation rumber per
Figure 2:
The relationship between c, number of points per bin and nisspecification constant v in Theorem 3.1 Based on the plot, when v 0. practitioners should set 6 ~ 300 when N 1000, b = 400 when N 5000, b = 500 when N 20000 (attaining
0.35
1000, p-0
R- 10000 0-0.1
3.50
5000,
5000, u-
0.25
N- 20000 0
0.1). When a ground truth proxy is misspecified (Definition 3.2). e-g., 0.1, for certain levels of e, the same bound can be attained with a larger b. For example, for achieving the same 0.15, if v 0 then 6 needs to be only approximately 250, whereas v = 0.1 then has to be 1000.
0.20
0.15
0.10
500
1500
2000
3000
practice, we can estimate v empirically using hold-out dataset. This estimate can be used choose For some levels of e, the same e as in the case of y = can be attained by setting to be a higher number. In our kd-tree instantiation, this amounts to using smaller maximum depth hyperparameter.
4 RELATED WORK
Calibration for Language Models. Reinforcement learning from human feedback objective may prioritize adherence to user instructions in dialogue over producing well-calibrated predictions. (Ka davath et al. 2022). Lin et al. (2022) introduced the concept of verbalized confidence that prompts LMs to express conndence directly, focusing on ane-ttnnng, instead of zero-shot verbalized confi- dence. Mielke et al. (2022) uses an external calibrator for a white-box large language model. Other methods consistency measures improve LM calibration (Lyu al. 2024). Our experimental
setup closely relates to recent works LM confidence elicitation (Tian et al. 2023 Xiong et al.. 2024). These methods lack novel posthoc calibrators and do not offer the rigorous calibration guaran-
tees that ours provide. Calibration has been shown impact selective QA performance Kamath et al. (2020), but they focus on uncertainty quantification and assumes that the LM allows access to the model likelihood.
Group Notions of Calibration. Previous works highlight the limitations of average-case calibration. Group-wise calibration, which uses predefined groupings (Kleinberg et al. [017 Pleiss et al. (017). has been adapted for language models (LMs). Li et al. (2024) train model that approximates the prciision-trreshee curve for a given group by using few-shot samples to predict the LM's s empirical precision at various confidence thresholds, Ulmer et al. (2024) train an auxiliary model using accuracy per group target to predict an LM's confidence based on textual input and output. Detommaso et al.) (2024) achieves multicalibration the data. Our work complements multicalibration, and our methods could extend to Algorithm 3 in Detommaso et al. (2024). [uoo et al. (2022) measure calibration over set of similar predictions, quantified by a kernel function on feature space. Again, the notions of calibrations and their guarantees are incomparable.
simultaneous calibration across various intersecting groupings of
this by adapting
Other Metrics for Measuring Calibration Error. Brier score (Brier 1950) measures the accuracy of probabilistic predictions but while it can be decomposed into calibration and refinement (Blattenberger & Lad, 1985), it doesn't 't directly assess calibration. As a result, a model with lower squared error may still be less well-calibrated. Maximum Calibration Error (MCE) examines the maximum miscalibration across confidence bins (Guo et al. 2017). but in a QA setting, it faces the same issues as calibration error (CE(h)), as shown in Hxample 2.1 We generalize MCE to the QA-calibration setting in Appendix DD Through QA-calibration, we present principled and interpretable calibration target for QA settings.
EXPERIMENTS
Datasets, Models, and Prompts. We use 5 QA datasets: TriviaQA (Joshi et al. 2017) SciQ (Welbl al.] 2017). BigBench (Srivastava et al.) 2022), OpenBookQA (Mihaylov et al. 2018 ) and MMLUU (Hendrycks et al., 2021) (see Table for more details). We use two performant mod- els: Mistral (Jiang et al. 2023) and Gemma Team et al. (024). To elicit confidence scores,
two prompt techniques recently suggested in literature: VerbIS-Topl & LinglS-Topp from Tian et al.) (2023). See Table (Appendix B.1) for details about the prompts. Central to the implementation of posthoe calibration and evaluation of calibration is the availability of label for question-and-anee
8

---

pair kd-tree partition Published as a conference paper at ICLR 2025
pair - -specifically, whether the answer provided by the LM is accurate for the question posed by the user. In practice, a common idea to generate this label is to take an LM provided answer, and then use another LM to assess whether the proposed answer is semantically equivalent to the true (ground truth) answer (Tian et al. 2023). To construct the ground truth proxy for y, we use Llama 3.1 (Dubey et al. 2024). See Appendix for qualitative examples of guestion-and-anse assignments.
Our Methods. We compare the performance of our calibrators: QA binning (QAB) from Subsec- tion 3.1 and hicrarchical scaling QA binning (HS- QAB) from Subsection 3.2 We also include a fully pooled version of scaling QA binning (S-QAB), by setting a|3] to a constant (thus, one partition) in Eq. (5) To set the hyperparameter minimum number of points per bin b (Algorithm []. we set an e that is not too large as per Figure 2 and use root finding with the e expression in Theorem 3.] to choose b. We then search over range of b's by allowing for misspecification range between O and 0.05 and range of maximum kd-tree depths depending on the size of the dataset such that each partition admits a 3-10 bins. To set B in UMD, we follow the guidelines in Gupta & Ramdas (2021).
Note that their bound does not involve misspecification factor.
Baselines. We consider the following baselines: no recalibration (None) which returns state-of-the -art
elicited confidence scores (Tian et al. (2023). histogram binning (UMD) (Gupta & Ramdas] (002]) Platt scaling (S) (Platt 1999). and sealing-binning (S-B) (Kumar al. 2019). These baselines consist of the state-of-the -he-thrrrt art ideas in posthoe calibration. Note that the techniques UMD, S, and S-B, aim to
minimize the expected calibration eπror CE (Definition 2.2) and do not take the partitions induced by
into account.
Metries. In this section, we primarily use two metrics for comparison. First, is the QA-calibration error CE(h; B) (Definition 2.4. which is the metric that our methods (presented in Section 3) optimize for. A lower CE (3) indicates more effective scheme for achieving QA-calibration. As explained in Section 2 QA-calibration error generalizes the expected calibration error (CB(A)). Second, to measure the downstream impact of using QA-calibration as the confidence calibration notion in QA setting, we adapt selective question answering (Kamath et al. 2020) to our setting: given a threshold y E [0, 1]. the answer a is returned for a question q to the end user if h(g, a) 2 т, and abstains (no answer is returned) otherwise 6 We adapt the area under risk covercoe curve, which is a standard way to evaluate selective prediction methods Yaniv & Wiener, 2010) to our setting. Our posthoc schemes produce discretized confidence scores ht often with large number of ties, which lead to unreliable coskrcoverege calculations. Therefore, we measure the area under the curve (AUAC) by sampling grid points from 0 to 1, computing the accuracy (based on ground truth y) of points with confidence greater than each grid point. A higher AUAC indicates a more effective selective QA scheme.
We perform a 4- w-way (20:60: split of each dataset: the first is used to construct the kd-tree, second is used for posthoc calibration training, third is used for hyperparameter tuning and fourth is for testing. We find it crucial to optimize for AUAC during hyperparameter tuning as our schemes already aim to minimize CE(h; 3). obtaining the appropriate maximum kd-tree tree depth d and binning parameters b and B. Missing experimental details are presented in Appendix B..]
Training.
Results. Table 2 shows the performance of the posthoe calibrators on MMLU and BigBench datascts. More results are provided in Tables 6 and 7 Our methods (QAB, HS-QAB, and S-QAB) generally achieve the best QA-calibration error (E(,, and best area under the curve (AUAC). While the first result, in itself, may not be surprising as our proposed schemes aim to minimize CE(h; ()), the gap between our techniques and baselines is significantly huge. For example, notice the difference in the calibration error when just using the SOTA confidence elicitation prompts (None) from (Tia et al. 2023) vs. our schemes in Table 2. Among our schemes, HS-QAB generally performs best. This because a parametric model (especially a partially pooled model like hierarchical scaling) helps reduce the variance of the downstream binning averages.
For selective QA, we again notice that our proposed schemes consistently outperforms the baselines. In some cases, the underlying LLM using confidence elicitation prompts (None) is performant in selective answering and attains high AUAC (similar results using different LLMSs were noted by Tian et al. (2023)). but are not well-calibrated as demonstrated by their high QA-calibration score. Our schemes, S-QAB and HS-QAB are generally the top-two performing schemes for this task with comparable and in many cases better AUAC scores than the None scheme. Since the accuracy- confidence curve is generated by examining the accuracy of answers above confidence threshold, the results demonstrate the desirable quality that the confidence scores provided by our proposed
'For example, when 0, all answers are returned irrespective of the confidence score.

---

Published as a conference paper at ICLR 2025
Dataset
Prompt
LLM
Calibrator
CE(h; ()
AUAC
MMLU
Lingls-Topl
Mistral
QAB (ours)
0.171 + 0.005
0.20 +.202...2.
MMLU
Lingls-Topl
Mistral
HS-QAR (ours)
0.16=00000
0.269 0.022
MMLU
Lingls-Topl
Mistral
S-QAB (ours)
0.163 0.003
0.19
MMLU
Lingls-Top1
Mistral
S-B
0.393 0.004
0.141 0.003
MMLU
Lingls-Top1
Mistral
S
0.249 0.005
0.002
MMLU
Lingls-Top1
Mistral
0.392 0.003
0
MMLU
Lingls-Topl
Mistral
None
0.532 0.008
0.269 0.007
MMLU
Lingls-Top
Gemma
QAB (ours)
0.232 + 0.005
0.211 0.032
MMLU
Lings--Topl
Gemma
HS-QAB (ours)
0.182 + 0.005
0.26 = 0.01
MMLU
Lingls-Top1
Gemma
S-QAB (ours)
0.181 + 0.009
0.275 0.029
MMLU
Lipgls-Top1
Gemma
S-B
0.382 0.006
0.184 0.002
MMLU
Lingls-Topl
Gemma
S
0.201 0.009
0.19 0.003
MMLU
Ling Io-p
Gemma
0.385 0.003
0.183 1.000.000
MMLU
Ling TopT
Gemma
None
0.603 0.006
0.249 0.01
MMLU
Topp
Mistral
QAB (ours)
0.197 + 0.004
0.198 + 0.046
MMLU
Verb1s-Top1
Mistral
HS-QAB (ours)
0.149 0.004
0.306 0.035
MMLU
Verb1s-Top1
Mistral
S-QAB (ours)
0.16 0.004
0.217 0.057
MMLU
Verb1s-Top1
Mistral
S-B
0.362 0.005
0.139 + 0.004
MMLU
Verbls-Topl
Mistral
S
0.258 1 0.005
1116-00.00
MMLU
Verb Verbls-Top1
Mistral
0.352 0.004
MMLU
Verb Verbls-Top1
Mistral
None
0.639 0.006
0.255 0.008
MMLU
Verb1s-Top1
Gemma
QAB (ours)
0.197 + 0.006
0.345 0.035
MMLU
Verbls-Top1
Gemma
HS-QAB (ours)
0.3 0.035
MMLU
Verbls-Topl
Gemma
S-QAB (ours)
0.161:0.0 0.006
0.227 0.074
MMLU
Verbls-Topl
Gemma
S-B
0.361 0.007
0.133 0.005
MMLU
Verbls-Top1
Gemma
S
0.228 0.005
0.137 0.004
MMLU
Verb1s-Topl
Gemma
B
0,352 + 0.004
0.144 + 0.011
MMLU
Verbls-Top1
Gemma
None
0.636 + 0.007
0.258 + 0.006
Table 2: Performance of our QA-calibration methods, QA binning (QAB), Pooled scaling QA binning
(S-QAB) and Hierarchical scaling QA binning (HS- QAB)), compared to the baselines, UMD, (B, (Gupta& & Ramdas 202 ), Platt Scaling (S, (Platt) 1999)). and Scaling-
(S-B, (Kumar et al.) 2019)), and prompt- based approach None (Tian et al. 2023) The bold entry the columns for CE(h; 3) and AUAC represent the best performing schemes for that metric. However, note that in some case the confidence intervals overlap, especially between schemes.
schemes are better at ranking accurate answers higher than inaccurate ones. Crucially, the lower performance of other baselines (S-B, S, and B) demonstrates the advantage of QA-calibration. In particular, QA-calibration framework identifies the optimal kd-tree depth, which empirically is never equal zero (standard average-case calibration). We observe similar results in Appendix when 3 is varied. Lastly, we analyze the shift in confidence scores after posthoc calibration in Appendix Gad per- partiaatition reliability in Appendix
CONCLUSIONS
We proposed QA QA-calibration, new notion of calibration which conditions groups of QA pairs. We propose two new posthoc calibration schemes for LM-elicited confidence scores. Our algorithms are effective on various QA datasets. For future work, we plan to investigate alternative notions of calibration for groups, such as multi-calibration.
Limitations. The interpretability of the calibration guarantee for the user largely depends on the choice of B - users want the partitions to be very fine- or coarse-grained, then /3 must be built with the appropriate depth. Furthermore, our algorithms assume that the output space of 3 is fixed, which may a limited assumption, given that the "information" space of generative QA may increase indefinitely over time.
ACKNOWLEDGEMENTS
P.M was funded in part by the Hybrid Intelligence Center, a 10-year programme funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research with grant number 024.004.022.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmid, Sam Altman, and Shyamal Anadkat. GPT-4 technical report. 01877 2023.
Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communica- lions of the ACM, 18(9):509-517, 1975.
Gail Blattenberger and Frank Lad. Separating the brier score into calibration and refinement compo- nents: graphical exposition. The American Statistician, 39(1):26-33) -2, 1985.
Glenn W Brier. Verification of forecasts expressed terms of probability. Monthly weather review, 78(1):1-3, 1950.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, and Sebastian Gehrmann. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Gianluca Detommaso, Martin Bertran, Riccardo Fogliato, and Aaron Roth. Multicalibration for confidence scoring in LLMs. In International Conference Machine Leurning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama herd of models. arXiv preprint arXiv:240..21783, 2024.
Ran Ylniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
Wade Fagen-UImsccheiddrr Perception of probability words. 2015. URL https:///attttt wwa. cs cessed: 2024-05-21.
Ac-
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2): 243-268, 2007.
Harvey Goldstein. Multilevel statistical models. John Wiley & Sons, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Chirag Gupta and Aaditya Ramdas. Distribution-ffee calibration guarantees for histogram binning without sample splitting. In International Conference on Machine Learning, pp. 3942-3952. PMLR, 2021.
Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas. Distribution-free binary classification: prediction sets, confidence intervals and calibration. Advances Neural Information Processing Systems, 33:3711-377 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, and Bing Qin. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv:231 S.t.:2311.052322 1.5232, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06823 2023.

---

Published as a conference paper at ICLR 2025
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield- -adfield-Dodds, Nova DasSarma, and Eli Tran-Johnson. Language models (mostly) know what they know. arXivv2207.0522.. 2022.
Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs the fair de- termination of risk scores. Innovations in Theoretical Computer Science Conference (ITCS), 2017.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. International Conference on Learning Representations, 2023.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
Xiang Lisa Li, Urvashi Khandelwal, and Kelvin Guu. Few-shot recalibration of language models. arXiv:2403. 18286, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions of Machine Learning Research, 2022.
Rachel Luo, Aadyot Bhatnagar, Yu Bai, Shengjia Zhao, Huan Wang. Caiming Xiong, Silvio Savarese, Stefano Ermon, Edward Schmerling, and Marco Pavone, Local calibration: Metries and recalibra- tion. In Uncertainty in Artificial Intelligence, pp. 1286-1295. PMLR, 2022.
Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, and Chris Callison-Bure Burch.. Calibrating large language models with sample consistency. 2tt::22402.1944 2024.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau, Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the Conference Empirical Methods in Natural Language Processing. 2018.
John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likeli- hood methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinherger. On faimess and calibration. Advances in Neural Information Processing Systems, 30, 2017.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXi::1910.01108 2019.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How learned to start worrying about prompt formatting. In International Conference on Learning Representations, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alone Alonso. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas Mayer, and Padhraic Smyth. The calibration gap between model and human confidence in large language models. arXiv:2401. 13835, 2024.
12

---

Published as a conference paper at ICLR 2025
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprini wrXiv:2403.08295, 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2023.
Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. Calibrating large language models using their generations only. arXivv24433.5597, 2024.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy Eser-generated Text, 2017.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs ex- press their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In International Conference Learning Representations, 2024.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.
Xlnet: Generalized autoregressive pretraining for language understanding, 2019. URL http: wttp://g//aaa/////// /1906 08237 cite Noiiv:1906.08233Coomm code available at https://tthabb.ommiiiiaggaaaaai
//arxiv. org/ abs
Pretrained models and
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. International Conference on Machine Learning, volume 1, pp. 609-616, 2001.
13

---

Published as a conference paper at ICLR 2025
Name Question
Notation
Description
Example
Inputted question.
What happens to you if you eat watermelon seeds?
Ground truth
ag
Intended output answer. Note that omit this in mathematical description and work directly with the ground truth label In practice we use a to construct the ground truth proxy.
The watermelon sceds pass through your digestive system
Prompting function
m(
A function that converts the question into speciffic form by inserting the question q. The example prompting function is Verb. top-k in [Tian et al.) We denote the filled (by the question) prompt m(g). Note that multiple prompts may be generated (see the 2S prompts in Tian ct al. 2023
Provide your three best guesses and the probability that each is correct (0.0 10) for the following question. Give ONLY the guesses and probabilities, no other words or explamation. For example: Gl: <first most likely guess, short as possible; complete sentence, just the guess! >PI; <the probability between 0.0 and 1.0 that GI is correct, without any extra commentary whatsoever, just the prohability! G3: <third most likely guess, as short as possible; not complete sentence, just guess!>P3: <the probability between 0.0 and 1.0 that G3 is correct, without any extra commentary whatsoever, just the probability The question is: <q>
[2033]. with
Answering function
c(q)
answering function e(g) c(m(g)). that mod- els the pipeline, that takes as input m(g) and out- puts an answer a. This is invoked & times to obtain answers. The function subsumes the postpro- cessing performed on the LM response obtain answer
Implicitlydccfiee in the LM interaction
Answers
(a), )
Three sampled answers, obtained after text normal- ization of the raw output.
(nothing, grow watermelon, stomachache)
Confidence function
hig.od a)
A confidence function l(q.aa a) = h((((g), a), that models the pipeline, and outputs the confidence that the swer correct for question This is invoked k times to obtain & confidences. The function sub- sumes the postprocessing performed on the I.M response to obtuin float confidence value.
Implicitly- definned in the LM interaction
takes input mlg) and
Confidence values
(h',h,, 19,,,,9) (3)
Three confidence values associated with the three answers. Note that they may not he normalized.
(0.95, 0,05, 0.2)
Ground truth proxy (proxy of y)
g(g. a²))
The returned truth value is postprocessed Yes/No to 1/0. In practice, this is used as proxy ground truth. We also experimented with the query (checking for semantic equivalence) from Tian [et al. (2023), but observe high false nega- live rate.
map
We query an LLM using the following prompt: Do follow- ing two answers to my ques- tion Q agree with each other? <q>, Al: <a²>, A2: <a²>, Please answer with a single word, either "Yes." or "No.". and explain your reasoning.
Table Important variables. Note that our mathematical description, we treat the output answers and confidences as 3 binary instances. Given a prompt function m() and question g, at the end of an LM's interaction, obtain answers and confidences.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DETAILS FOR SECTION 3
A.1 LIMITATIONS OF NON-DISCRETIZED METHODS
Estimation of | when H = h(Q, A) is continuous is difficult task (umar et al. confidence must also be discretized for it to achieve guarantees of marginal calibration (Gupta & Ramdas 2021). Prompts used to elicit confidence (Tian et al. 2023 Xiong et al. 2024) are not guaranteed to induce discretized confidence score (both in the average-case. and QA-caliraatoon cases).
2019). The
In practice the estimation of CB(h;; 3) is done by partitioning D according to 3: D U D... where D2 {qg d; fuz)) (g; (4) s}. and taking the weighted average of CE(h)s (from Eq. ( (3)) from each D. with weight [D,]] For each D., H is typically binned into intervals, and the calibration error in each bin is estimated as the difference between the average of confidence values and labels in that bin.
Lastly, estimation of CE(h) for direriicretized methods may involve binning and can underestimate the eтror (Proposition 3.3 in (Kumar et al. 2019)). In our case, this is a possibility when comparing scaling against the other approaches.
A.2
DIPPING HISTOGRAM BINNING (UMD)
We adapt UMD (Algorithm from Gupta & Ramdas (202T) to our notation in Algorithm + UMD takes as input calibration data D [[,, he(N). where h; 's are (uncalibrated) confidence scores and 1,'s is the corresponding target labels. The function order-stats returns ordered confidence scores (h(1)..... (())) where ((1)) < h(2) <... <...< (m))
Algorithm 4: UMD
Input: Number of bins B, Calibration data {(,,, t,) hei) (dataset cardinality n depends on the inputted dataset, we refer to the dataset within the algorithm as D.)
Output: Calibrator function JUMD [0, 1] 1]
(haa)...,,hu (())) < order-stats(h
h,)
2: (la))..,,44 ((x) (4,,...,6,) (n) ordered as per the ordering of (h(2)....,
3: A + (n+1)
4: fl <- empty array of size B
5: A + O-indexed array([0, [ [2A]),.... n
6: for +- to B do
7: 1+ A-A-1
8: u utA, As
9: II, + mean(4 .... ((u-1))
10: end for
(h(o). h)(+1)) + (0.
12: Define the calibrator function, JUMD: 9UMD (Awea) Ebe 1(h(A6-11) S hust < Hibb
return JUMD
A.3 DISTRIBUTION-FREI GUARANTEES
In Gupta et al. (020). UMD procedure (Algorithm 4
is assumed to take ground truth as input 2) Since in our setting, it possible to pass a proxy ground truth, we now describe
generalization of conditional calibration guarantees of UMD procedure (Gupta & Ramdas 2021
Theorem 3).
Theorem A.1 (Conditional Calibration Guarantee of Algorithm under Label Misspecification). Consider an input calibration dataset Ò defined Section 3.3 with misspecification factor v from Definition 3.2 Assume that the hy 's are distinct and N > 2B. Then calibrator JUMD outputted by
Algorithm with input D D, is (e, --condiitinnaly calibrated for any (0, 1), with
1og(2B/o)
(6)
2(LN/B] 1)
15

---

Published as a conference paper at ICLR 2025
Proof. For b E (0,1..... B), define K2 [b(N + 1/B)]. Fix h() 0 and ((++)) 1 as the smallest and largest order statistics, respectively. As per Algorithm of the input data and gather the following points into a set: M =- [T(k))).......... 1))
4 we compute the order statisties
Let B [0, 1] + [B] be the binning function: B(H) = b + m(mm))) S gUMD(H)) < h(a)). Given
M, the function B is deterministic, i.e., for every b E []. R2 (Y) b] is deterministic.
Consider some E {O. BB and denote / = bo & Ramdas (02T), the unordered (denoted by h((,)) confidence scores [((41), [1142),..... [+++1 are i.i.d given M, with the same conditional distribution as that of H given B(H) = b. There- fore, 1(++1), ((cct)))............... are i.i given S, with the conditional distribution BernoulliEE{y B(H)= = bj.
kb. By Lemma from (Gupta
We now show that HI (defined in Line 9 and returned by gUMD in Line 12 in Algorithm 4 the average of the in a bin b values, concentrates around LE[ B(H)) = b]. For any E (0, 1), by Hoeffding's inequality, w.p. at least y.
2(u 1- - log(2/7)
log(2/7)
B| B(H) = b| ils -
(7)
2([N/B]-1
where the second incquality holds since for any b, u - -- [(b + I) + +1)/B]
16(N + 11/BB N
[N/B]. Using law of total probability (partitioning (15(h) = b} to (h: B(h) = b)), we obtain:
E B(H) b] < EE| B(H) = -
which implies that
log(2/7) 2([N/B B] -
E[Y B(H) =b] - S b]]
+ v.
(8)
Set 7 a/B'n in Eq. 8 and take union bound over all b B. With probability at least - a., for
every b E B, E (I) b] N1 e, where is the RHS of Eq. (8) This implies that:
| H] H
|ETTY B(H), H] | H] H
| B(H) H]] - H|
|E[E [B()) H H
JE[EY |B(H)] ( H
EE B(H) [ls(m) H
Where the first equality is due to the law of total expectation, the fourth equality is by the definition of the quantity returned by 9UMD, and the first inequality due Jensen's 's inequality. This completes the proof, showing that gUMD is (c, a-) conditioonlyy calibrated for any a E (0, 1).
Theorem 3.1 (Distribution--fe free QA-calibration Guarantee). Consider an input calibration dataset ò defined above with misspecification factor from Definition 3.2, Assume that the h₂'s are distinct, number of points per bin b 2 2, and number of instances within each partition n. 2 b for every E S. The calibrator gUMD retrieved in Line 2 of Algorithm 2 trained using Algorithm I with input
D D, is (e, connoiiiio QA calibrated for any (0, 1), with = V
Tog Na/ba)
216
V.
Proof. For s E S, let P. denote the distribution of (h, f) conditional on B(g. = s. The tuples in D. are i.i.d. samples from P. and g, G[s) the corresponding fitted UMD calibrator. The number of bins is B, = p.. Since, n, 2 ////b/ 2 2B.,
Let B- -1 B, and a = aB./ B. Note that B S E.Es ng/b b = N/b. We apply the bound in Theorem A.I obtain:
16

---

Published as a conference paper at ICLR 2025
log log(2Bs/a)
vp E range((s); E
813(g,0) - 8,0x(h) =P] $
+
B(q,a)= =
2( ma/ B₂ - 1)
Cgs.
Note that
log(2Ba/as)
log(2B/a) S
log(2N/bo)
1og(2N//br)
2([nz/ B, -1] S
2( [////B Bx -1
2([n,//.,
2(b
For every 8 E S:S:
P vp range(g.); B
( =) - =p] - Se
B(q.u) = 1221 21-0s
Taking union bound over gives:
Vs E L.S. rangelga), E Y (1g( (h)
3(q,a) = 2 Σ a2 s
1 a.
This completes the proof.
To remove the assumption that h; 's are distinct in Theorem 3.1 we could use the randomized version
of UMD (Algorithm 2 in Gupta Ramdas (202T). I.. This adds additive factor to in Theorem [3] can be made arbitrarily small. For simplicity, we have chosen to provide the guarantees for the non-randomized version of UMD.
B ADDITIONAL DETAILS FOR SECTION 5
B. MISSING EXPERIMENTAL DETAILS
Details on LM Querying. We set the LM cemperatura to close to 0 to minimize output stochastic- ity and set max tokens be able to process the prompt m(g). We include the prompts in Table The variable EXPRESSION LIST in LingIS-Topp prompt is taken from Fagen- Tragen-Ulmschniddee Ulmschneider (2015).
Compute Resources. The experiments were run using 3090Ti GPU and 64 GB of RAM.
Training Details.
perform random splitting times using different seeds and use the repeated measures to form our confidence intervals. We set the number of bins B in the UMD baseline and sealing-binningg by deriving hyperparameter search ranges from setting different in Theorem in (Gupta & Ramdas 2021). We describe how we set the number of points per bin in Algorithm| and Algorithm in Section 5] We have also attempted to ensure that for every B we can find in its hyperparameter tuning range, there in its hyperparameter tuning range, such that B _N/b]. This correspondence. coupled with the same target variable to optimize, helps isolate the impact on posthoe calibration.
17

---

Published as a conference paper at ICLR 2025
Dataset
Size
Type
SciQ
11609
QA
Bigbench
20320
QA
MMLU
13869
Multiple-choice QA Multiple-choice QA
OpenBookQA
4835
TriviaQA
11313
QA
Table 4: Dataset statistics. Multiple-choice QA datasets provide set of possible answers. We add additional prompt text for them, see Table
Prompt Name
Prompt Template
VerbIS-Topl
Provide your best guess and the probability that it correct (0.0 to 1.0) for the following question. Give ONLY the guess and probability, no other words or cxplanation. For example: Guess: <most likcly guess, as short as possible; not a complete sentence, just the guess! > Probability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability! The question is: <q>.
Ling1S-Topl
Provide your best guess for the following question. and describe how likely it that your guess is correct SION LIST. Give ONLY the guess and your confidence, no other words explanation. For example: Guess: <most likely guess, as short as possible; not a complete sentence, just guess!> Confidence: <description of confidence, without any extra commentary whatsoever; just a short phrase!> The question is: <q
of the following expressions: SEXPRES-
Additional prompt text for multiple- choice QA task with set of choices C'
The answer must chosen from the following list of size <|C >: <C Only the actual answer (not choice number or index) from the list should be used in the response.
ground truth proxy
See the example for ground truth Proxy in Table 3
Table Prompts used in the experiments.
18

---

Published as a conference paper at ICLR 2025
Dataset
Prompt
LLM
Calibrator
CE((;; 9)
AUAC
SciQ
MnglS-Topl
Mistral
QAB (ours)
0.165 + 0.003
0,482 - 0.024
SciQ
LingIS-Top!
Mistral
HS-QAB (ours)
0.175 0.004
0.437 1 0.028
SciQ
Ling1S-Topl
Mistral
S-QAB (ours)
0.171 +
0.426 + 0.034
SeiQ
LingIS-Topl
Mistral
S-B
0.466 0.003
0.21 0.007
SciQ
LingIS-Topl
Mistral
S
0.255 = 0.005
0.21 + 0.005
SciQ
Ling1S-Topl
Mistral
B
0.356 + 0.025
SciQ
LinglS-Top1
Mistral
None
01366 + 0.011
0.451 + 01.009
SciQ
Ling1S-Top1
Gemma
QAB (ours)
0.357 + 0.023
SciQ
Ling1S-Top1
Gemma
HS-QAB (ours)
0.148
0.363 0.032
SeiQ
Ling1S-Topl
Gemma
S-QAB (ours)
0.146
0.366 0.085
SeiQ
Ling1S-Topl
Gemma
S-B
=.000
0.144 0.191 + 0.008
SciQ
Ling
Gemma
S
0
SeiQ
T-Top
Gemma
B
0.195 + 0.011
SeiQ
Gemma
None
0.011
0.403 0.018
SeiQ
Verb1S-Top
Mistral
QAB (ours)
0.262 0 0.009
0.428 + 0.027
SeiQ
Verb1S-Top1
Mistral
HS-QAB (ours)
0.203 0003=00 0.006
0.474 1174+0000 0.048
SciQ
Verb1S-Top1
Mistral
S-QAB (ours)
0.182 0.006
0.474 0.095
SciQ
Verb1S-Top1
Mistral
S-B
0.472
0.237 0.006
SeiQ
Verb1S-Top1
Mistral
S
0l2288 0.005
0.27 + 0.02
SciQ
VerbIS-Topl
Mistral
0.458 0.002
0.377 0.012
SciQ
VerbIS-Topl
Mistral
None
0.451 0.013
0,451 0.015
SciQ
VerbIS-Topl
Gemma
QAB (ours)
0.268 0.005
0.278 1 0.021
SciQ
Verb1S-Top1
Gemma
HS-QAB (ours)
0.194 +: 0.006
0.386 0.034
SciQ
Verb1S-Top1
Gemma
S-QAB (ours)
+
0.279 0.023
SciQ
Verb1S-Top1
Gemma
S-B
0l.77 + 0.002
0.149
SciQ
VerbIS-Topl
Gemma S
0,243 1: 0.007
0.246
SciQ
Verb1S-Topl
Gemma
0.464 0.003
0.344
SciQ
VerbIS-Top1
Gemma
None
0.495 1 0.01
0.421 0.00
TriviaQA
Ling1S-Top
Mistral
QAB (ours)
0.26 2: 0.003
0.467 0.024
TriviaQA
Ling
Mistral
HS-QAB (ours)
0.195
0.499 + 0.014
TriviaQA
LingIS-Topl
Mistral
S-QAB (ours)
0.004
0.021
TriviaQA
Ling
Mistral
S-B
0.441 0.003
0,275 0.006
TriviaQA
Ling
Mistral
S
0.266 0.005
0.28 1 0.000
TriviaQA
Ling
Mistral
0.432 0.003
0.441 0.009
TriviaQA
Ling1S-Top
Mistral
None
0.326 - 0.008
0.537 0.01
TriviaQA
LingIS-Topl
Gemma
QAB (ours)
0.268 0.007
0.310 + 0.034
TriviaQA
Ling
Gemma
HS-QAB (ours)
0.188 + 0.02
0.359 0.035
TriviaQA
Gemma
S-QAB (ours)
0.18 0.005
0.351 0.051
TriviaQA
Ling1S-Topl
Gemma
S-B
0.235 0.015
0.310 0.007
TriviaQA
Ling1S-Top
Gemma
S
0.211 0.012
0.086 0.005
TriviaQA
LinglS-Top
Gemma
B
0.468 0.003
0.199 0.019
TriviaQA
Ling 1
Gemma
None
0.486 = 0.01
0.349 + 0.007
TriviaQA
Verb1S-Tow -opp
Mistral
QAB (ours)
0.242 0.006
0.618 0.016
T
Verb1S-Topl
Mistral
HS-QAB (ours)
0.191 0.01
1.73200022
T
Verb1S-Tow
Mistral
S-QAB (ours)
0.18
T
Verb1S-Ton
Mistral
S-B
0.436 +.000
0.324
TriviaQA
Verb1S-Top 1op1
Mistral
S
0.27 0.005
0.352 4.014
TriviaQA
Verb1S-Top1
Mistral
B
0.409 + 0.005
4.525+0.00
TriviaQA
Verb1S-Toph
Mistral
None
0.38 + 0.01
0.011
TriviaQA
Verb1S-Tow
Gemma
QAB (ours)
0.249 + 0.249 0.006
0.381 0.02
T
Verb1S-Tow
Gemma
HS-QAB (ours)
0.192 0.006
0.417 0.008
TriviaQA
Verb1S-Top
Gemma
S-QAB (ours)
0.337 0.038 0.038
TriviaQA
Verb1S-Top
Gemma
S-B
0.444 +.040000000
0,231 0.012
TriviaQA
Verb1S-Top1
Gemma
S
0l258 + 0.008
0.225 + 0..06
TriviaQA
Verb1S-Tom
Gemma B
0.429 + 0.004
0.37 + 0.015
TriviaQA
Verb1S-Top
Gemma None
0.527 + 0.008
0.417 0.008
Table 6: (Continuation of Table 2) Performance our QA-calibration methods, QA binning (QAB),
Pooled scaling QA binning (S- (S-QAB) and Hierarchical scaling QA binning HHSQQAB), compared to the baselines, UMD, (B, (Gupta & Ramdas 2021)), Platt Scaling (S, (Plati 1999)). and Scaling-
binning (S-B, (Kumar et al. 2019)).
19

---

Published as a conference paper at ICLR 2025
Dataset
Prompt
LLM
Calibrator
CE(h; 8)
AUAC
OpenBookQA
Ling I
Mistral
QAB (ours)
0.285 + 0.009
0.353 + 0.043
OpenBookQA
Topp
Mistral
HS-QA HI-QQBB (ours)
0.211 0.015
0.37 1 0.065
OpenBookQA
Ling
Mistral
S-QAB (ours)
0.193 0.012 0.012
0.34 0.062
OpenBookQA
Tipp
Mistral
S-B
0
0.135 0.004
OpenBookQA
Ling
Mistral
S
0.32 + 0.006
0.133 0.005
OpenBookQA
Linp
Mistral
B
0.471 0.00
0.255 0.023
OpenBookQA
LingIS-Topl
Mistral
None
0.441 0.013
0.38
Ling
Gemma
QAB (ours)
0.316
0.25 0.028
OpenBookQ4
Ling
Gemma
HS-QAB (ours)
0.221 0.016
+ 0.261 0.032
OpenBookQA
Tipp
Gemma
S-QAB (ours)
0.215
0.257 0.082
OpenBookQA
Linp
Gemma
S-B
0.005
=.19
OpenBookQA
Ling
Gemma
S
0.421
0.013
OpenBookQA
LingIS-Topl
Gemma
B
0.483 + 0.003
0.017
OpenBookQA
LingIS-Topl
Gemma
None
0.434
0.017
QA
VerbIS-Topl
Mistral
QAB (ours)
0.288 + 0.008
0.338 + 0.035 0.035
OpenBookQA
VerbIS-Topl
Mistral
HS-QAB (ours)
0.232 0.012
0.044
OpenBookQA
VerbIS-Topl
Mistral
S-QAB (ours)
0.225 0.011
0.316 1 0.027
OpenBookOA
VerbIS-Top1
Mistral
S-B
0.011
OpenBookQA
VerbIS-Top1
Mistral
S
0.282 +
0.3+0.000
OpenBookQA
VerbIS-Topl
Mistral
B
0.447 0.003
0.28 0.02
OpenBookQA
VerbIS-Topl
Mistral
None
0.561 0.018
0.311 1 0.021
OpenBookQA
VerbIS-Topl
Gemma
S-QAB (ours)
0.228 0.011
0.343 1 0.027 0.027
OpenBookQA
VerbIS-Topl
Gemma
S-B
0.011 0.006
OpenBookOA
VerbIS-Topl
Gemma
S
0.298 0.01
0
OpenBookQA
VerbIS-Top1
Gemma
B
0.447 + 0.003
0.293 0.04
OpenBookQA
Vorp
Gemma
None
0.561 0.018
0.315 0.036
OpenBookQA
Vorb
Gemma
QAB (ours)
0.261 0.008
0.332 1 0.048
OpenBookQA
Vorp
Gemma
HS-QAB (ours)
0.228 0.012
0.321 1 0.011
BigBench
LingIS-Topl
Mistral
QAB (ours)
0.195 + 0.002
0.622 + 0.009
BigBench
Linp
Mistral
HS-QAB (ours)
0.138 0.006
0.690 0.031
BigBench
LippIS-Topl
Mistral
S-QAB (ours)
0.157
0.641 0.02
BigBench
Linp
Mistral
S-B
0.387 0.006
0.523 = 0.006
BigBench
Topp
Mistral
0.224 0.003
0.52 1 0.004
BigBench
LingIS-Topl
Mistral
B
0.379 + 0.003
0.612 0.017
BigBench
LingIS-Topl
Mistral
None
0.245 + 0.004
0.684 0.007
BigBench
LingIS-Topl
Gemma
QAB (ours)
0.198 0.006
0.41 =
BigBench
LinpIS-Topl
Gemma
HS-QAB (ours)
0.142 0.009
0.421
BigBench
Lipp
Gemma
S-QAB (ours)
0.193 0.002
0.321 0.052
BigBench
Lopp
Gemma
S-B
0.326 0.002
0.302 0.005
BigBench
LingIS-Topl
Gemma
0.180 + 0.006
+ 0.005
Big BigBench
LingIS-Topl
Gemma
B
0.468 + 0.001
0 0.014
BigBench
Ling TopT
Gemma
None
0.427 + 0.006
0.01
BigBench
VerbIS-Topl
Mistral
QAB (ours)
0.161 0.003
0,665 + 0.031 0.031
BigBench
Vepp
Mistral
HSSQABB (ours)
0.132 1: 0.009
0.672 0.079
BigBench
Verb1S-Top1
Mistral
S-QAB (ours)
0.134
0.012
BigBench
Verb1S-Topl
Mistral
S-B
0.361 0.006
BigBench
Verb1S-Topl
Mistral
S
0.208 - 0.004
0.522 0.018
BigBench
Verb1S-Topl
Mistral
B
0.351 + 0.002
0.592 0.014
BigBench
Verb1S-Top1
Mistral
None
0.248 0.005
0.671 + 0.005
BigBench
Verb1S-Top
Gemma
QAB (ours)
0.171 + 0.003
0.503 0.019
BigBench
Verb1S-Top
Gemma
HS-QAB (ours)
0.159 0.006
0.502 0.011
BigBench
Verb1S-Top
Gemma
S-QAB (ours)
0.005
0.465 0.024
BigBench
Verb1S-Top T
Gemma
S-B
0.456 + 0.001
0.289 = 0.011
BigBench
Verb1$-Ton
Gemma S
0.259 + 0.004
0.292 + 0.007
BigBench
Verh1S-Top
Gemma B
0.431 + 0.002
0.441 + 0.014
BigBench
Verb1S-Topl
Gemma None
0.458 + 0.01
0.499 + 0.01
Table 7: (Continuation of Table 2} Performance of our QA-calibration methods, QA binning (QAB),
Pooled scaling QA binning (S-QAB) and Hierarchical scaling QA binning (HS-QAB),, compared to the baselines, UMD, (B, (Gupta & Ramdas 2021)), Platt Scaling (S, (Plati 1999)). ), and Scaling-
binning (S-B, (Kumar et al. 2019)).
20

---

Published as a conference paper at ICLR 2025
C KD-TREE CONSTRUCTION FOR QA-CALIBRATION
We adapt kd-tree (Bentley 1975) construct our partitions. We first recall the standard construction of a kd-tree. Let Z4 (Zi1...., Zim ZeM) ), = =1, 1,..., N be the M-dimensional dataset to bin and let H* with k = 0 represent (L,,..., ZN J. Let z denote the median of all the rath coordinates of the Zs H Let Z j)] denote the yth coordinate a Z vector. Set base case value of O.
The partitioning scheme be applied recursively is as follows: split H into halfspaces by piv- oting on = to obtain 1/2k+1 = 2 E 11 Z[pl I =m and 122k+2 : {z E 11 -|p] > =m The coordinate index m is set to llg (k + 1) + 1, i.e., as we split the nodes, we only change the coordinate index when switch another level. When reach m M, we reset m 0.
In our setting, we form bounded spaces so that our calibrators that contain a QA binning subroutine can generalize well during test time (outliers that are the closest to the boundary of a space will not be assigned to that space). We take the observations with the smallest and largest order statistic in each coordinate used for pivoting, and use them as bounding values for that coordinate.
At test-time, an M-vector instance is inserted into one of the leaves by following the same rule as in the partitioning scheme. If any coordinate value outside its bounding values, assign none of the leaves. Otherwise, cycle through the coordinates and assign the instance to the left tree if the coordinate is less than or equal to 2k or the right tree otherwise. Repeat until leaf reached.
D GENERALIZING MAXIMUM CALIBRATION ERROR (MCE) TO MAXIMUM CALIBRATION ERROR (QA MCE)
Guo et al. (2017) defined the maximum calibration error (MCE) as:
MCE 3)
sup rErange(h)
| h(Q,,) A) = r] -rl]
We generalize this definition by incorporating 3: Definition D.1 (QA Maximum Calibration Error). The QA maximum calibration error (QA MCE) of h is defined as: B-MCE(h;
max eErange( (8) rErango(h)
sup
| (QQ, = ((Q, - -(].
This metric assesses the maximum deviation between the confidences and true probabilities over all confidences and partitions. Intuitively, the confidence that achieves the supremum in MCE may differ from the one that maximizes over partitions in 3-MCE, as the latter is also conditioned on the specific partition.
E QUALITATIVE EXAMPLES OF PARTITION CONTENT
Examples of question-and-answer pairs assigned 3 partitions are presented in Table S
F EXTRA EXPERIMENTS WITH VARYING B CHOICES
In addition using kd-tree as Gain and DistilBERT embedding as Bamb, we also use k-means as B and XLNet (Yang et al. 2019) embeddings Semb We utilize the [CLS] token embedding from a pre-trained XLNet and conduct a hyperparameter search over the number of centroids, using AUAC as the validation metric. We present the results in Table Similar patterns to those in Table 2 are observed: our methods, S-QAB and HS-QAB, achieve the highest performance for CE(h; (3) and AUAC.
G SCATTERPI.OT OF RECALIBRATED CONFIDENCE SCORES OVER THE CONFIDENCE ELICITATION BASELINE
We examine how the recalibrated confidence scores differ for baseline confidence scores obtained using the (Figure 3. The confidence scores obtained by our method QA binning have
21

---

Published as a conference paper at ICLR 2025
Question
LLM Answer
If weather is stormy then there is a greater chance of Bill missed high tide. so had to wait until when to see again Great lakes may have come to be thanks to a floud occurring, there was most likely
everything getting wet
lomorrow
ice pillars
great droplets repeating
Some animals are aided in finding food sources by Flowers make themselves attractive hummingbirds with Hummingbirds gather nectar using their
aroma
optimal angle
bills
An example of an adult animal laying eggs all aside from
kittens
When human's organs stop working and he stops breathing. that person If a thing experiences burning combustion, then An exertion thing that is going against the thing's intended direction, when in motion will When we think of bees, we also think of pollen. This is because bees
perished
damaged
oppose
consume it
Table 8: Qualitative examples of three random partitions and four random question-amm iitiiioon-nd-aaae answer pairs from the OpenBookQA dataset, multiple-choice dataset. The prompt used is Ling 1 (Table and answers are generated using Mistral. The partitioning function / leverages DistilBERT embeddings and kd-tree with maximum depth of The underlying topic for the first two partitions seem clear: "water" and "animal", and the third partition is likely to have identified the form of the question.
Dataset
Prompt
LLM
Calibrator
Bonh
8un
CB(h;
AUAC
OpenBookQA
Verb18-TTvg
Mistral
QAB (ours)
DistiBERT
kd-tree
0.288 + 0.008
0.338 + 0.035
OpenBook QA
Verb1S-Topp
Mistral
HS-QAB (ours)
DistiBBRRT
kd-tree
0.232 : 0.012
0.344 0.018
OoenBooKQA
Verb1S-Topl
Mistral
S-QAB (ours)
DistiBERTT
kd-tree
0.225 0.011
0.316 1 0.027
Ooo
1erb1S-Topp
Mistral
S-B
DistiBERT
kd-tree
0.445 = 0.006
0.331 + 0.011
OpenBookQA
Verb1S-Toph
Mistral
DistiBERT
kd-tree
0.282 0.01
0.3 + 0.009
Open QA
Verb1S-Topl
Mistral
B
DistiBBRRT
kd-tree
0.447 + 0.003
0.28 + 0.02
OpenBookQA
Verb1S Topp
Mistral
None
DistilBERT
kd-tree
0.561 + 0.018
0.311 + 0.021
OpenBookQA
Verb1S-Topl
Mistral
QAB (ours)
DistiBBERT
k-means
0.310 = 0.009
0.334 + 0.012
OpenBook
Verp
Mistral
HS QAB (ours)
DistiBBERT
k-means
0.240 0.013
0.340 + 0.019
OpenBook A
Verb1S-Topp
Mistral
S-QAB (ours)
DistiBBRRT
k-means
0.255 + 0.012
0.321 + 0.019
Open
VerbIS-Topl
Mistral
S-B
DistilERRT
k-means
0.450 0.007
0,331 + 0.011
OpenBookQA
1-rp1S
Mistral
DistiBERT
k-means
0.285 = 0.011
0.3 - 0.009
OpenBookQA
Verb1S-
Mistral
DistilBERT
k-means
0.004
0.02
OpenBookKQA
Verb18-Tipp
Mistral
None
DistiBERT
k-means
0.565 + 0.018
0.311 + 0.021
Open
Verb18 Toppl
Mistral
QAB (ours)
XLNet
kd-tree
0.295 + 0.010
0.337 + 0.034
OponBook
Verb1S-Topp
Mistral
HS-QU HS-QAB (ours)
XLNet
kd-tree
0.238 0.012
0.341 + 0.021
Ooon
Verb1S-Topl
Mistral
S-QAB (ours)
XLNet
kd-tree
0.220 0.011
0.347 = 0.017
O
Verb1S-Topl
Mistral
S-B
XLNet
kd-tree
0.442 0.008
0.331 + 0.011
OpenBookQA
Verb15-
Mistral
XLNet
kd-tree
0.280 0.012
0.3 + 0.009
OooaBookQA
Verb ITop
Mistral
XLNet
kd-tree
0.450 = 0.004
0,28 + 0.02
OpenBookQA
Verb1S-T7 -
Mistral
None
XLNet
kd-tree
0.561 0.018
0.311 + 0.021
OpenBookQA
Verb1S-
Mistral
QAB
XLNet
k-means
0.312 0.011
0.335 + 0.033
OpenBookQA
Verb Topl
Mistral
HS QAB (ours)
XLNNt
k-means
0.246 = 0.013
0.342 + 0.019
OpenBook
Verb1S-Tiop
Mistral
S-QAB (ours)
XL.Ntt
k-means
0.235 0.010
0.350 0.018
Ope QA
Verb1S-Topl
Mistral
S-B
XLNet
k-means
0.448 = 0.007
0.331 + 0.011
OpenBookQA
Verb1S-Topl
Mistral
XLNet
k-means
0.283 = 0.011
0.3 + 0.009
OpenBook
Verb
Mistral
XLNet
k-means
0.455 0.005
0.28 + 0.02
OpenBookQQ
Verb1S-Topl
Mistral
None
XL.Net
k-means
0.565 + 0.018
0.311 + 0.021
Table 9: We vary Bemb (DistilBERT, XLNet) and Boin E (kd-tree, k-means). Note that AUAC metrics for S-B, S, and None remain constant as they do not utilize 3. In this experiment, we observe that across the four variations listed above, the optimal Poin results in comparable number of partitions.
higher sharpness (Brier 1950 Gneiting et al. 2007) since we are fitting a UMD fitter per partition. The baseline UMD (B), shown in blue, maps an elicited confidence score from Ling I STopp to either single value small set of nearby values, due the cattoorandooiznioonn in UMD. In contrast, our method, QA binning (QAB), shown in grey, applies a partition-speciffi UMD that adapts to the specific question-and and-on--nd-annwee pair. As a result, the scores calibrated by QA binning span a broader range. This is intuitively reasonable, as a prompt-elicited confidence score of 0.1 from topic like "geography" may require a different recalibration compared to the one from the "politics".
22

---

Published as a conference paper at ICLR 2025
Ling1STop1 posthoc-calibrated confidence scores
1.0
0.8
Posthoc-calibratee scores
0.4
0.2
UMD (B)
0.0
QA binning (QAB)
0.0
0.2
0.4
0.6
0.8
Confidence scores from elicitation using the Ling1STopl prompt
Figure 3: Scatter plot and regression lines for posthoc-calibrated scores vs confidence scores from Ling1STopl. Note that confidence scores from LingISTop (shown in the the x-axis) are discretized since confidence statement is drawn from an expression list (Table We use the OpenBookQA dataset and Mistral LLM.
H RELIABILITY ANALYSIS OF QA PARTITIONED DATA
In this section, we demonstrate reliability analysis using ECE and reliability plot through an example. We perform analysis for each of the four 3 partitions (derived using kd-tree with maximum depth of two) in Figure 4 We compare the baselines None (from 1S- prompt) and UMD (which is known to best optimize ECE) against our method QA binnning. Partition- wise analysis reveals that the ECE for each partition (representing question-and-answe
pairs that semantically similar) best achieved by our method, QA binning. However, performing this reliability analysis becomes increasingly challenging and unwieldy the cardinality of the range of 3 grows.
23

---

Published as a conference paper at ICLR 2025
Perfectly Callbrated
Perfectly Calirated
None: ECE = 0.37 UMD 18:: 60B - 0:15
Nome: CCE = 0.55
UND (Bl: ECE 0.21
Di-bkniing 10A3): 0.06
- 04 binring 1048t: ECE 0.32
Confiiace
conidernc
Relibiiity Diagram foe Partitio
Bellabilty Dlegram for Surtition
Some DCE 035
Soree DCC 0.30
QRlume (2M0) B0B 0.35
00-te miia 10481 w
Moen Confeareescce
Can Meneie
Figure 4: The reliability plot compares the baselines None UMD (B), and our method, QA binning (QAB), across each QA partition. Using the OpenBookQA dataset, LingIS-Top prompt, Mistral LLM, and kd-tree with maximum depth two. four QA partitions are generated, and conduct an analysis on each partition. UMD (B) and QA binning (QAB), provide confidence scores that are better calibrated at each of the partition compared to None. However, QA binning (QAB) yields better calibrated confidence scores than UMD (B).
24


Reviews:
[{"review_id": "9MwjgZ2cha", "reviewer": "Reviewer_snQU", "summary": "The paper introduces a novel concept called $\\beta$-calibration, which calculates calibration errors across various QA groups to enhance user decision-making. It then proposes post-hoc calibration techniques specifically tailored for $\\beta$-calibration, establishing guarantees without additional assumptions. Experimental results highlight the effectiveness of $\\beta$-calibration across a range of tasks and models, demonstrating improvements in calibration performance.", "questions": "1. In Equation (5), how are the fixed and random intercepts/slopes calculated separately? It appears possible to compute an intercept-slope pair for each partition, so how do you differentiate between fixed and random components? Additionally, how does this approach differ from simply calculating an intercept-slope pair for each partition?\n2. What value of $\\alpha$ was used when plotting Figure 2 and how is that chosen?\n3. When calculating calibration error, how are the bins constructed? Do you use the same binning strategy as the binning method in the paper, or is a fixed-width re-binning applied?\n4. How tight is the bound compared with the actual results? How does the bound informs the selection of hyper-parameter? It seems like the choice of B follows prior work rather than leveraging the newly calculated bound.\n\nOverall, I think this is a good paper. I’m open to raising the score if my concerns are addressed.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of calibration dependency on the dataset is highly significant. The QA pairs in different subsets can exhibit vastly different subset calibration errors, affecting the user's experience. To the best of my knowledge, this is the first paper to identify and directly address this issue. While the proposed methods are relatively straightforward adaptations of existing approaches, they represent an essential first step in tackling this problem. Although initial, this approach is an important contribution.\n2. The paper is well-structured and easy to follow. Notations are defined with clarity and rigor, enhancing overall comprehension.\n3. The experiments are thoughtfully designed, evaluating not only calibration error but also the utility of $\\beta$-calibration in selective answering, demonstrating that the model is both calibrated and practically useful. Experiments are conducted across diverse models and tasks, with results showing significant improvements over baseline methods. Additionally, the inclusion of rigorously proven distribution-free guarantees completes the study.", "weaknesses": "1. **Generalizability of $\\beta$:** The paper is motivated by the idea that users may have specific interests in different groups of QAs, necessitating calibration that is tailored to user needs. However, \n- The paper demonstrates the method’s effectiveness using only one type of $\\beta$ (partitioning based on DistillBERT embeddings), leaving it unclear how well these methods generalize to other partitioning strategies. Showing results with other partitioning methods or justifying this choice’s relevance to user interests would strengthen the paper. \n- The framework assumes each question belongs exclusively to one group. However, in real-world scenarios, a single question could naturally align with multiple groups, as users with differing interests may ask the same question. It is unclear how the framework would function under this condition. Discussing this limitation or proposing a solution for handling questions that span multiple user-interest groups would enhance the paper’s practical value.\n2. **Method Choice Motivation:** The framework, which partitions then calibrates each subset, could work with various post-hoc methods (e.g., isotonic regression, Platt scaling), yet only binning is tested. Including experiments to test the framework with different post-hoc strategies or discuss the reasons for this choice could strengthen the paper by demonstrating its generalizability. Additionally, it is unclear why isotonic regression—a common baseline—is excluded.\n3. **Limited Result Analysis:** While partition-specific calibration errors are a focus, results are only reported on average. Reporting variance across partitions, or providing results for each partition in an additional table, along with identifying and discussing those showing the most improvement, would offer valuable insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mo8bWF7B52", "reviewer": "Reviewer_cg5J", "summary": "This work presents a beta calibration error, a calibration metric which partitioning the test set and computing calibration error over each partition individually. Partitioning is done based on sentence representations from a pretrained LM. The authors then present methods for post-hoc calibrating existing confidence estimates from an LM. Their approach involves partitioning training data using the same partitioning function as is used during evaluation and fitting calibrators for each partition.", "questions": "As an additional note MCE (noted in related work) was also introduced by [1]. Frequently used metrics like AUROC and other metrics should be also included in related work and discussed, many are summarized by [2].\n\n[2] Plex: Towards Reliability using Pretrained Large Model Extensions\nDustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, Balaji Lakshminarayanan", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work defines a generalization of calibration error metrics.\n\nThe work introduces methods for optimizing for their proposed calibration metric and theoretical results backing up their methods.", "weaknesses": "W1. One concern with its work is a possible mischaracterization of how expected calibration error (ECE) is used to evaluate calibration. In the toy example in Table 1, the authors note that standard calibration error is 0; however, the often used ECE metric (from [1]) involves binning examples during test time by sorting and partitioning examples based on their predicted confidence. In the Toy example in Figure 1, ECE  (with 2 bins) and beta-calibration are equivalent. While Beta-Calibration can be framed as a generalization of the standard ECE binning method, it's unclear why it's better motivated or more useful. \n\n[1] Obtaining Well Calibrated Probabilities Using Bayesian Binning\nMahdi Pakdaman Naeini, Gregory F. Cooper, Milos Hauskrecht\n\nW2. Following up on point 1, while it can make sense to introduce alternative binning when computing ECE, by using different beta-calibration functions, it is not made clear why the proposed binning method is inherently better and why methods should be optimized for the specific binning function used during evaluation.\n\nW3. Another weakness is in using only one base calibration method. Furthermore, while the authors use verbalized confidence scores as their base confidence scores, citing them as a SOTA method, it's unclear whether this is true for the scale of model evaluated in this work (side note, work also lacks core modeling details like the size of pretrained LMs evaluated and whether they use instruction-tuned variants).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5TKbepfGgu", "reviewer": "Reviewer_YLZ3", "summary": "This paper deals with post-hoc calibration of “verbalized” confidences from LLMs, which are typically not well-calibrated out-of-the-box. The paper uses as a subroutine, uniform-mass double-dipping histogram binning (UMD) (Gupta & Ramdas, 2021) to perform this post-hoc calibration. A further scaling version is explored in S3.2 based on the scaling procedure from Kumar et al. (2019). The proposed approach is compared to existing (general purpose) post-hoc calibration methods on standard QA benchmarks.", "questions": "* Why not include additional metrics such as ECE and Brier score?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper tackles an important problem (LLM confidence estimation)\n* The approach is validated on 5 different QA datasets.\n* The “selective” QA metric provides some insight into the practical benefit of improved calibration.", "weaknesses": "* The approach assumes access to an “oracle” to measure semantic equivalence between a predicted and reference answer. I’m not sure if Llama-3.1 is such an oracle, unless perhaps it has memorized the QA datasets in question, which raises some other concerns.\nThe approach requires dataset-specific calibration sets. I’m not sure if it’s fair to compare to baselines such as out-of-the-box prompts that don’t use this information. It’s also a bit of a limitation since it presumably makes it difficult (impossible) to apply the proposed approach to novel problems.\n* Overall, the approach seems to largely be an application of existing methods to post-hoc calibration of a particular QA problem with LLMs. The adaptation to LLMs does not appear to involve any significant technical innovations.\n* The paper evaluates the results using the very calibration metric that is optimised by the proposed method. This is fraught and it’s unclear why further calibration metrics (such as one or more variations of ECE, Brier score, etc.) are not also reported. \n* The approach relies on a smaller/older model DistilBERT which seems like a key limitation.\n* While there is a short paragraph addressing the limitation of choosing \\Beta, I would have liked to see further more in-depth discussion of the limitations of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTE7JO1mRY", "reviewer": "Reviewer_iAXj", "summary": "The paper introduces a stronger form of calibration which requires that subgroups also be calibrated. It then presents a set of techniques for achieving this form of calibration, first by binning *within* the subgroups and then introducing two improvements: scaling by replacing the labels with predicted labels during the calibration step (s-beta-binning) and then replacing the predictor with a hierarchical classifier to get pooling across groups. \n\nThe main theoretical result is that these methods can achieve bounded calibration error with high probability, using a bound in which the error rate grows as $O(\\log b / (2b - 1))$ with $b$ equal to the number of bins. \n\nEmpirically, these techniques improve calibration error over reasonable baselines on a suite of five question answering tasks; they sometimes improve selection prediction as well.", "questions": "While the paper is focused on language model confidence and generative QA, is there anything in the method that is specific to this application? It seems like it could be applied to any prediction task over examples that can be partitioned by a KD tree in which recalibration might be required. \n\nSo I would love to know more about (a) which examples end up in which bins in these datasets; (b) the relationship between recalibrated confidence scores with and without beta-binning (e.g. something like a scatterplot of confidence scores with \"B\" vs \"BB\").", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a solid contribution that offers a novel and useful new perspective on post-hoc calibration. The methods are well motivated and extend prior work in sensible but interesting ways, which will likely be of interest to other researchers working on calibration. The exposition is unusually clear and theoretical results are helpful.", "weaknesses": "The empirical results may not be strong enough to motivate adoption by practitioners -- while the method outperforms baselines in table 2, over the full suite of tasks the selective prediction performance is often worse than or statistically indistinguishable from elicited confidence scores (tables 6 & 7). \n\nA related point from the practitioner's perspective is that the utility of beta-calibration seems to depend on the meaningfulness of the groups -- if the questions really are partitioned by users as suggested by figure 1, then these stronger guarantees seem crucial; but if the groups are not intuitively meaningful then it's not clear why a practitioner would be motivated to pursue these stronger guarantees.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)