Hidden Heterogeneity: When to Choose Similarity-Based Calibration
Authors: Kiri L. Wagstaff
Thomas G Dietterich
Venue: Accepted by TMLR ()

Abstract: Trustworthy classifiers are essential to the adoption of machine learning predictions in many real-world settings. The predicted probability of possible outcomes can inform high-stakes decision making, particularly when assessing the expected value of alternative decisions or the risk of bad outcomes. These decisions require well-calibrated probabilities, not just the correct prediction of the most likely class. Black-box classifier calibration methods can improve the reliability of a classifier’s output without requiring retraining. However, these methods are unable to detect subpopulations where calibration could also improve prediction accuracy. Such subpopulations are said to exhibit “hidden heterogeneity” (HH), because the original classifier did not detect them. The paper proposes a quantitative measure for HH. It also introduces two similarity-weighted calibration methods that can address HH by adapting locally to each test item: SWC weights the calibration set by similarity to the test item, and SWC-HH explicitly incorporates hidden heterogeneity to filter the calibration set. Experiments show that the improvements in calibration achieved by similarity-based calibration methods correlate with the amount of HH present and, given sufficient calibration data, generally exceed calibration achieved by global methods. HH can therefore serve as a useful diagnostic tool for identifying when local calibration methods would be beneficial.

Published in Transactions on Machine Learning Research (1/2023)
Hidden Heterogeneity: When to Choose Similarity-Based Calibration
Kiri L. Wagstaff
dir..ugstaff@orrgnnstttt.dddd
School of Electrical Engineering and Computer Science
Oregon State University
Thomas G. Diettcrich
tgd orst. edu
School of Electrical Engineering and Computer Science
Oregon State University
Reviewed on OpenReview: https: // opwnreviee......... net/ pert/formm? id= R3O
Abstract
Trustworthy classifiers are essential to the adoption of machine learning predictions in many real-world settings. The predicted probability of possible outcomes can inform high-stakes decision making, particularly when assessing the expected value of alternative decisions or the risk bad outcomes. These decisions require well-calibrated probabilities, not just the correct prediction of the most likely class. Black-box classificr calibration methods can improve the reliability of a classifier's output without requiring retraining. However, these methods unable to detect subpopulations where calibration could also improve prediction accuracy. Such subpopulations are said to exhibit "hidden heterogeneity" (HH), because the original classifier did not detect them. This paper proposes a quantitative measure for HH. It also introduces two similarity-weighted calibration methods that can address HH by adapting locally to each test item: SWC weights the calibration set by similarity to the test item, and SWC-HH explicitly incorporates hidden heterogeneity to filter the calibration set. Experiments show that the improvements in calibration achieved by similarity- based calibration methods correlate with the amount of HH present and, given sufficient calibration data, generally exceed calibration achieved by global methods. HH can therefore serve as useful diagnostic tool for identifying when local calibration methods would be beneficial.
1 Introduction
How do we know when to trust prediction? Let f(X) be classifier that outputs discrete probability dis- tribution P(Y|X) over the K possible class labels {1,.... K}. Tdeally, each prediction made by the classifier will be point-wist wise calibrated, that is, the true class distribution for each X matches P: P(Y P(Y (X). Many investigators have studied a weaker requirement that each distinct output value is calibrated, such that P(Ygg) = E[Y] over the set predictions for which P(Y = (Vaicenavicius et al. 2019 Widmann et al. 2019 This ensures aggregate calibration for the set,, but it allows individual predictions to have P(Y[X) XX)))). Others use the same approach but require calibration only for the most-likely class's predicted probability (e.g., Guo al. 2017 Patel et al. 2021 Luo et al. 2022) marginal class proba- bilities (e.g., Zadrozny & Elkan 2002 Kumar al.) 2019), both of which also do not enforce point-wise calibration. Point-wisc calibration of each individual prediction, for its full class distribution, is important for safely-critical applications. Well-calibrated predictions improve the trust wort of systems and support downstream cost-sensitive decisions (e.g., medical diagnosis, autonomous driving, financial decisions). Like- wisc, calibration is necessary when combining or comparing prodictions from different sources (Bella et al., 2013) or in classifier cascades that use low-cost but less accurate classifier's output to decide whether apply higher-cost. but. accurate secondary classifier (Enomoto & Eda 2021 Good calibration is

---

Published in Transactions on Machine Learning Research (1/2023)
beneficial in any decision making setting in which uncertainty matters (e.g., active learning or classification with a rejection or abstention option).
We focus on an increasingly common use case in which we would like to apply a pre-trained, possibly proprictary, model M to our own data set D with corresponding distribution PD(Y In this scenario, the original training data set is unavailable and P(Y[X), the distribution for which M was trained (and possibly calibrated), is unknown. Any domain shift between P and PD could prevent M from generating reliable predictions on D. Morcover, even in the absence of domain shift, M may perform poorly on D due to "hidden heterogeneity", which occurs when M assigns the same posterior probability to items with different true probabilities.
Concerns about poorly calibrated classifiers are not new (e.g., Zadrozny & Elkan 2001 Niculescu- Mizil & Caruana, 2005), and several bost-training calibration correction methods have been developed (e.g., Guo ct al., 2017 Kumar et al., 2019: Kull et al. 2019 Alexandari et al.) (020). In general, these methods devise calibration map & that transforms the original predicted probabilities into values that are better calibrated. We denote the output of classifier fa;) applied to item z; as the probability vector i. of length K (number of classes) that sums to (i.e., resides in the simplex A ) The calibration map 4 AK- + AK-1 is derived from an independent calibration set C to transform P. to a more reliable q: = ((p)).
A key limitation of these calibration maps is the implicit assumption that all items with the same pre- dicted probability vector á should be given the same correction. Such maps cannot accommodate hidden heterogeneity, which manifests as subpopulations with distinct P(Y values to which the classifier has erroneously assigned the same á value. For example, in predicting cancer risk, there could be many different reasons (age, lifestyle, family medical history, etc.) that given individual is predicted to have p 0.9. Even if the predictions satisfy aggregate calibration, this probability could be an over- for some individu- als, while for others it could be an under-estimate. No global calibration map can address this heterogeneity to achieve point-wise calibration, because they map all items with the same ò to the same q.
We propose a method to quantify hidden heterogeneily (HH) as a signal for when global calibration may be inadequate. Once HH is detected, we face a choice of either (a) training new classifier on the available calibration data or (b) improving the existing classifier using the calibration data. Because HH is local phenomenon, a natural way to improve the classifier is to apply a local, similarity- calibration technique. We introduce two local calibration methods that leverage the location of in feature space to yield 9: = ( (r.). These methods determine the calibrated probability q by taking a weighted vote of data points in the calibration set C. The first method, Similarity- Weighted Calibration (SWC), assigns weights to every point in the calibration set based on similarity to TG. The second method, SWC-HH, uses only items within a local neighborhood defined by the estimated HH. We refer to the weighted number of calibration data points as the "calibration support" for Hi, which indicates how much calibration data is available for estimating d:. This measure of the calibration quality of 4. for each F; is unique advantage of local calibration.
We note that any post-hoc calibration method can be viewed as form of model stacking (Wolpert 1992 in which the output of the original classifier is transformed via p, a model itself. Our SWC and SWC-HI methods are stacking methods that focus on improving local calibration. As a consequence, they also reduce or eliminate HH and can thereby improve classifier accuracy.
The major contributions of this paper are
1. The identification of hidden heterogeneity as a property of classifier predictions that thwarts global calibration methods (Section 3),
2. A method for quantifying hidden heterogeneity to indicate when local calibration is needed (Sec- lion
3. Two local calibration methods based on Similarity- Weightee Calibration (Section 4 and
4. Results of experiments that assess the relationship between hidden heterogeneity and calibration, yielding useful guidance for practitioners (Section 5
We provide context from previous work in Section
2
Key conclusions and limitations of local, similarity-
based calibration are discussed in Section 5
2

---

Published in Transactions on Machine Learning Research (1/2023)
2
Related Work
There are several methods for improving the reliability (calibration) of classifier predictions. Many recent advances were inspired by the recognition that deep neural networks in particular may sacrifice calibration to achieve higher generalization accuracy (Guo et al. 2017). Strategies include using ceabbratioonsnnsiiiivy training methods, if the original training set is available (e.g., via modifications to the loss function, as
proposed by Kumarr et al. 2018 Mukhoti et al.] 2020 Enomoto & Eda 2021 Tomani & Buettner 2021),
using domain-specift reprosentations that lead to improved calibration (Kalmady et al. 2021], or adopting network architectures that do not use convolutions (Minderer et al. 2021 In contrast, post-hoc calibration correction methods that directly modify the classifier's predictions on new observations, without re-training, can be employed even when the training data (or model) are proprietary or when the data distribution has changed and we wish to recalibrate an existing classifier to extend its applicability.
Global, parametric calibration methods re-map the predicted probabilities, f. output by the classifier by fitting a chosen functional form (e.g,, logistic curve) from the probabilities to the labels to compute q .()). For binary classifiers, Platt scaling (Platt] 1999) transforms P. into a value between 0 and using sigmoid function with two parameters, AA and B: q. = T+edp,Ter The parameters A and B are chosen to optimize the negative tog-likelihood of predictions made on the calibration set. Platt scaling was generalized to multi-class problems for neural nelworks (Guo et al. 2017) via a method called temperature scaling, which operates on the logits 2/ (not the probabilities) by optimizing temperature parameter T in u ug[k] = e=1B//T where 2 2,[k]] is the logit for item i and class k, and u[[k] is the corresponding unnormalized probability. Those values Temperature
are normalized as [k] = u.[k]
[A] The same T value is used for all classes.
Scaling (Alexandari et al. 2020 adds a bias term for each class.
There are also several approaches that construct probability bins and assign the average (or other aggregate) accuracy within bin B3 as its calibrated probability, == Ace,, Vi E Bg. Histogram binning (Zadrozny & Elkan 2001) assigns items to bins based on their uncalibrated predictions P., often using equally-spacee bin boundaries or divided so that each bin has the same number of items (*equal frequeney") from the calibration set. Kumar et al. (2019) found that the latter strategy, as well as using a larger number of bins, yields better results. Isotonic regression (Zadrozny & Elkan 2002) adds further flexibility by optimizing the bin boundaries to minimize the squared loss between & and ys. Recently, [Patel et al. (2021) proposed selecting the bin boundaries to maximize the mutual information between bin predictions Q: and yi-
To date, very few calibration methods have leveraged the location of items in feature space, X. Zhao et al. (2020) introduced "individual" (per-item) calibration for regression problems and confidence intervals. Partial specialization for classification problems can be achieved by estimating a different T per subpopulation (unlabeled cluster (Gong et al. 2021) or labeled "domain" " (Yu et al. 2022)). ) then employing lincar regression to estimale a new T" for each test iLem. Our approach operates at finer (per-item) granularity and is not restricted to probability rescaling. Like our method, the LoRe calibration method (Luo et al. 2022) considers the similarity of the calibration set items to the test item x. However, LoRe restricts the similarity calculation to calibration items that fall into a probability bin based on the probability max; P. [k] of the highest- probability class. This can produce high variance estimates when the bin contains few calibration items. Our method avoids this problem by considering the full predicted distribution f when computing similarity. LoRe also only calibrates the distribution over all K classes. Consequently, it does not support rownsstream tasks such as computing the expected costs of misclassification (in cost-sensitive problems) or re-estimating class probabilitics (Alexandari et al. 2020
prediction; it does not produce a calibrated probability
One calibration approach that employs similarity to compute the complete [k] vector is Similarity- -innnig Averaging or SBA-10 (Bella et al., 2009), which creates bins (neighborhoods) that contain an item's 10 nearest neighbors (in Euclidean distance) in an "augmented" feature space x+ = X' x AK- defined by the item's feature vector I, of dimension d concatenated with its probability vector pe E AK-1- SBA-10 computes the calibratod probability 4 [k] as the probability of class k (in the calibration set) within item i's assigned bin, with each item contributing equally (Bella et al.] 2209).. In contrast, our approach uses a mimilarity-weighhee contribution from every item in the calibration set, not just the 10 nearest. neighbors.
3

---

Published in Transactions on Machine Learning Research (1/2023)
Algorithm Hidden Hoterogeneity (HH)
Input: Test item 2t, calibration data C, prodicted probabilitios p, and radius r
Output:
Hidden helerogeneity in neighborhood around dit
1: Construct probability neighborhood around ze; Ut = (x; E () Di) <r} (using Equ. 2
2: Train dr using labeled data in 24,
3: Collect model predictions for the neighborhood: f(4)) = (Diles E Ut).
4: Collect It predictions for the neighborhood: gr (2(44) = (x; U...
5: Collect labels for the neighborhood: Yu,- = (yiler le E U.).
Calculate HHu, using f(44), ge (e()), and Yu, (Eqn. 3).
3 Hidden Heterogeneity
Global post-hoc calibration methods, such as Platt scaling and temperature scaling, perform very well for some data sets and algorithms and less well for others. Similarly, local methods like SBA-10 do not always improve upon these global methods. What causes the failure of global methods, and under what conditions can local methods do better? Our hypothesis is that global post-hoc calibration, which focuses on aggregate calibration, fails when the data exhibits hidden heterogeneity (HH) with respect to the predicted probabilities p. HH characterizes situations where there are subpopulations in the feature space X' to which the classifier assigns the same p but that require different calibration corrections.
3.1 Hidden Heterogeneity
Definition 1 A classifier f exhibits hidden heterogeneity with respect to feature space X' if there exists a subregion U C X' such that f(xx) s p for all x E and yet. U can be partitioned into M disjoint subregions u = Ui II- ... L[ such that the true class probabilities P(y|æ E Um) + P(ylæ E Um)) for all distinct pairs m, m' E (1.....M), m + m'.
An extreme example of HH occurs for classifier that ignores all features and predicts the majority class for all items. Imagine a data set composed of 60% cats and 40% birds, for which classifier predicts P(y "cat") = 0.6 for all items (i.e., U X). This classifier is perfectly calibrated in terms of aggregate calibration, but it is uninformative about any individual animal. If cats and birds are not separable in the feature space, this may be the best one can do. However, if the items have a feature such as "mumber of legs", then two subregions Ui for animals with two legs and U2 for animals with four legs can be defined with true conditional probabilities of (for "cats") and 0 (for "birds"). This heterogeneity is hidden in the classifier's predictions.
This extreme situation (complete HH) could happen for number of reasons (majority-class classifier, clas- sifier only trained on cats, etc.). More commonly, any classifier may have one more subregions V/ in its predicted probabilities that likewise obscure informative heterogeneity, due to model misspecification, an overly constrained hypothosis space, over-eegulaiization, or data shift. Detecting HH can alert the prac- titioner to limitations of the classifier. While global methods that map Å to q cannot address HH, local calibration could model the subregions separately and assign & differently for each U,
3.2 Detecting Hidden Heterogeneity
Algorithm [ provides a method to compute the delectable hidden heterogeneity for region U C C given a labeled calibration data set C sampled from the same distribution the test set. HH is calculated as the potential improvement (compared to the original classifier) achieved by training a specialized classifier on only the items in 24
In step 1, we define item z,'s probability neighborhood Uy to contain calibration items that are close to 2, the probability simplex AK-1- More precisely, Ur contains the items within radius r of item x, AK-1. There is no priori best choice for r, but obtain reliable HH estimates, one should choose r such that set Ue is excessively small. We employ the standard choice of Hellinger distance DH (Fquation) I to
4

---

Published in Transactions on Machine Learning Research (1/2023)
calculate the distance between probability vectors P. and py. Hellinger distance is the probabilistic equivalent of Euclidean distance, and it is more suitable here than KT, divergence, which is not symmetric.
K Σ
(p).
V.lM
V (p,kk)
(1)
v2
Conveniently, the Hellinger distance can be expressed as the Euclidean norm of the difference of the element- wise square root of each probability vector (Krstovski et al. 2013):
Dn((..Pj) V2 VV vDj
(2)
This in turn allows the use of efficient methods (e.g., k-d tree) for populating neighborhood Ui.
For each lest item Tt, a new (local) classifier Jr is trained using only the nearby calibration items in Ue (step 2). This classifier gr can be any classifier type. We employed an ensemble method that can perform internal gencralization estimates without an additional validation set. We trained a baggod ensemble of 50 dccision trees with no depth limit and no limit on the number of features searched for each split. We used out-of- bag error to determine how much pruning to employ to achieve good generalization and avoid overfitting to the calibration set. We searched over 7 values of the pruming complexity parameter, evenly spaced between 0.0 (no pruning) and 0.03, as input to the minimal cost-complexity pruning method (Breiman et al. 1984
Finally (step 6), we calculate HH for U, by comparing the Brier score Brier 1950) of the original predictions by model f on Ue (step 3) with those generated by the local model gr. (step 4) using true labels Yu, (step 5):
HHu. Bricr(f(U4), Yu.) Bricr(g. Yu.),
where the Brier score is the mean squared error between predictions 6 [k] [0, 1] and labels ye, for N items and K possible classes:
N
K
1 N
pc(k]] - 1 = x))
Brier(p, Y) = A=1
(4)
We enforce the condition that Se no worse than f by clipping HHu, to O. Regions with large HH values provide both a warning that global calibration methods may not perform well and an opportunity for local specialization by using item similarity during calibration.
4 Similarity-Weighted Calibration
We propose to improve point-wise calibration by leveraging information in feature space as well the uncalibrated probabilities p.. Given test item 2e, the goal is to estimate well-calibrated g.(k] = P(y k(zz)) for each class & E
Similarity- Weighted Calibration (SWC) is described in Algorithm Let
(1, () = sim([ fe), (:)) E [0, 1]
be the similarity between item Dn and item F, measured in the augmented space X+,, where [a, b] is the concatenation of vectors and b. similarity of 1 perfect identity. The investigator chooses how best to measure similarity in this space. Use of X enables calibration to benefit from information encoded by the classifier (6) as well as item position in feature space (z). Further, supervised similarity measure can learn the relative importance of each component for the problem at hand. We employ such measure; the random forest prozimitt function (RFprox). R.prrxx trains random forest on a labeled data set and defines the similarity between items I; and Iy as the fraction of times they are assigned to the same leaf in each tree of the ensemble (Breiman 2001 Cutler et al. 2012). Effectively, the random forest encodes a "kernel" defined by those weights (lcaf co-occurrences) (Hastic al., 2009). We employ the calibration data learn the relevant R.prox measure using a random forest with 100 trees, no depth limit, and considering random set of Vd. features for each split, given d total features. Note that defining sim() using standard
5

---

Published in Transactions on Machine Learning Research (1/2023)
Algorithm 2 Similarity- Weightted Calibration (SWC)
Input: Test item 2t, calibration data x; E C and labels yi
Output: Calibrated probabilities (e[ Vk
Pe[k] for & E 2: Collect model predictions for the calibration set: Pi(k] for E C, [l....,K}
Collect model predictions for item
3: Compute pairwise similarity as s(t, i) for I; E C.
4:
Compute a [k]
s(t,i) Σ, s(t, i)1 = k) for & E {I,..., K} (Equ. 5).
kernels, such as the Gaussian kernel, over X would impose a fixed weighting on the x and f components. potential direction for future research would be apply multiple kernel learning (Gönen & Alpaydn 2011 optimally combine separate kernels for æ and p.
SWC computes the similarity of x, to every item in the calibration set (step 3) and uses this information to replace á with similarity-weighted combination of labels from the calibration set (step 4).
1 s(, E (()) = =k). k).
@.[k] =
(5)
The similarity-based approach to calibration enables local spocialization within the data set, but does not directly make use of the calculated HH. We also developed the SWC-HH algorithm,, which filters the calibration set to restrict which items are used to generate q. HH, which is computed separately for each test item (Algorithm D,, is employed an additional filter for calibration. In step 4, SWC-HH only includes items with similarity to item 2; of at least 1 HHH, Note that the maximum value for HH is 2.0 since it is the difference in Brier scores, clipped to 0.0, and each Brier score ranges between 0.0 and 2.0. Dividing by 2.0 normalizes the threshold to the range 0.0 0.0-1.0, making it suitable as a similarity threshold.
5 Experimental Results
We conducted experiments with variety of classifiers and data sets to compare local and global calibration methods and to determine the role that hidden heterogeneity plays. Our hypotheses were that (1) local, similarity-based calibration is more effective at reducing Brier score than global calibration methods, (2) the amount of improvement correlates with the hidden heterogencity score, and (3) calibration support can serve an indicator of per-item calibration quality. Our implementations of SWC, SWC-HH, and other calibration methods, along with scripts to replicate the experiments, are available at https://giitub..co com/
wkiri/simcall
5.1 Methodology
We assessed calibration methods for six tabular data classifiers implemented in the scikit-learn Python library (Pedregosa et al. 2011). including a decision tree with min_ sat__smmles__eee leaf 10 (DT), random forest with 200 trees (RF), an ensemble of 200 boosted trees (GBT), a linear support vector machine (SVM), Gaussian kernel (7 x C' = 1.0) support vector machine (RBFSVM), and Naive Bayes classifier (NB). Any parameters not explicitly mentioned were set their default values. We also conducted experiments with pro-trained deep neural networks for classifying images (dctails in Scction 5.4
Data sets. We analyzed four tabular and two image data sets:
moons: synthetic 2D data set with two partially overlapping classes, chosen to enable visualization
of classifier outputs and hidden heterogeneity in fcature space. Obscrvations were generated using
the scikit-learn make_moons () function with noise set to 0.3 and random_staatee stanisstttteeeee set to 0.
letter (letter recognition): 26-class data set of capital letters from the English alphabet represented
by 16 statistical and geometrical features that describe the image of the letter (Fry & Slate 1991
available at https://archive ics. uci wtt.a///////aacttee//etettooooo/

---

Published in Transactions on Machine Learning Research (1/2023)
the MNIST handwritten digit data set (LeCun et al., 1998 composed of 28x28 pixel (d = 784) images containing digit from O to 9. We used the data set as provided by OpenMI. the original source is httt::/yyann... lecun. hom//xxb/mmiiss// In addition to the 10-class data set (mnist10), we created several binary subsets consisting only of two digits, such as "4" and "9" (mnist-4v9). This data set is "tabular" (1-d feature vector); no 2D image structure preserved.
mnist:
fashion-mnist: grayscale images of clothing and accessories (10 classes) that was designed to be more challenging than the MNIST data set yet have the same dimensionality (28x28, d 784) (Xiao al., wttp:/thw..oooaaaaanneeeeeeeeeee http///aaaannoeeercc/eeeeeeeeeee wwwpchh/ffshhhiommmmmiae
2017), available ad https:: http::://gthbbb.
CIFAR-10: 60,000 images (64 x 64 pixels) labeled into 10 distinct classes; the test set contains
10,000 images (Krizhevsky) 2009)
CTFAR-100: disjoint set of 60,000 images labeled into 100 different. classes (50k train, 10k test)
For tabular data sets, we randomly sampled 10,000 items and for each trial randomly split them into 500 train, 500 lest, and 9000 for a calibration pool. For the "mnist10" and "letter" data sets, we used 1000 itemss each for training and test, due to their large number of classes (10 and 26, respectively). We created a serics of nested calibration sets of size { 50, 100, 200, 500, 1000, 1500, 2000, 2500, 3000 } to assess the data efficiency of each calibration method. For image data sets, in each trial we generated a ratifidd random split of the standard test set into 5000 test items and reserved the remainder as the calibration set. We report average performance across 10 trials along with the standard error for the observed values.
Calibration methods. We compared three similarity-based calibration methods (SBA-10, SWC, and SWC-HH) to standard calibration methods including Platt scaling (Platt] 1999). histogram bin-
ning (Zadrozny & Elkan 2001]. and isotonic regression (Zadrozny & Elkan 2002). SBA-10 employed
Fuclidean distance to identify the nearest neighbors, while SWC and SWC-HH used the RFprox similarity
measure. When computing hidden heterogeneity, we used a probability radius of r = 0.1 in the probability simplex.
Platt scaling optimizes the negative log-likelihood of predictions against the target probabilities rather than discrete {0, 1} labels (Platt 1999 Niculescu-Mizil & Caruana, 2005). With n4 as the number of calibration items in the positive class and n. as the number of negative items, the target probabilities are V+ n++2
n++1 and
y- +2 For multi-class problems, we applicd temperature scaling (Guo et al. 2017). For classifiers that
1- To avoid dividing by zero (or taking its logarithm), we clipped pi]k] to the range [e, - c], where e = 1 x 10-12, We applied the histogram binning method implemented by Kumar et al. (2019 and followed their recommendation to use equal-mass bins (100 bins).
output probabilities instead of logits, we first transformed f.: into logils as z; [k] In p.
Metrics. We employ Brier score (Equation 4 to measure point-wise prediction quality, following earlier researchers (e.g.. Zadrozzy & Elkan 2001 2002 Nieulescu-Mizil & Caruanal 2005). Tt has several advantages
over a commonly used calibration metric called the Expected Calibration Error (ECE) (Naeini et al. 2015
which assigns predictions to bins to assess aggregate calibration. ECE can be trivially minimized to 0 by always predicting the empirical average probability of a given class, yielding perfectly calibrated but uninformative predictions (Kull et al.] 2017 Widmann al. 2019 Ovadia et al. (201)). The Bricr score incorporates not just calibration error (or "reliability") but also "resolution" or sharpness, which rewards predictions that do more than predict the average probability (Ferro & Fricker 2012). In addition, ECE is sensitive to the number and choice of bins (Vaicenavicius et al. 2019 Kumar et al. 2019 Patel et al., 2021), it exhibits undesirable edge effects (discontinuities at bin boundaries). and it only assesses calibration of the predicted class. Brier score characterizes prediction quality across all classes, and it avoids artificial discretization and edge effects, since it does not employ binning.
5.2 Similarity-based calibration for tabular data sets
To test our first hypothesis, we compared similarity- calibration to global methods. Figure [ shows Brier score as a function of available calibration data for the binary classification task of distinguishing two
1 Available https::/p/// h/tp:://ggtttubeee /t//ghhht..cm//p-mmmaaaa/rrrtte/ cabbraatiioiiiiii

---

Published in Transactions on Machine Learning Research (1/2023)
0.5- lucal
0.35
Ram
0.35
Plat
535-10
SMC
0.25
Brler 5c0re
0.25
SWOC MM
0.20
0.20
0.2
0.15
0.1
0:10
0.05
25000
2000
Number ralionation
Number calibration
Number calibration
(a) Naive Bayes
(b) Decision tree
(c) Random forest.
Figure 1: Calibration performance for three classifiers on the MNIST 4-vs-9 data set. Each plot shows Brier score (lower is better) with increasing calibralion set size. Error bars show one standard error over 10 trials.
handwritten MNIST digits ("4" vs. "9"). The uncalibrated predictions yielded different starting Brier scores for each classifier (red dashed lines). Platt scaling and isotonic regression improved the Brier score for the Naive Bayes (NB) and random forest (RF) classifiers but only marginally for the decision tree (DT). No further improvements were achieved beyond 500 calibration items. Similarity-based calibration (SBA-10, SWC, and SWC-HH) generally did not perform well with small calibration sets but achieved much larger benefits for NB and DT when at least 500 items were used for calibration, and Brier score continued to improve as more calibration data was provided. The random forest, which had the best initial Brier score and thorefore less room for improvement, showed an advantage for similarity-based calibration after at least 1500 items were used. SWC-HH yielded a clear advantage at all calibration set sizes for NB and DT, and it also provided small advantage over SWC for RF with calibration set sizes of least 1500 items.
SWC and SWC-HH consistently pet-performed SBA-10. Since RFprox uses labels to learn the similarity measure, it can elevate the importance of individual features in X+ (like p(k]) by placing them higher in individual decision trees within its ensemble. SWC and SWC-HH also include evidence from the entire calibration set rather than only the nearest neighbors. Importantly, SBA-10 showed almost no difference in Brier score for different classifiers, effectively ignoring their individual differences in p. MNIST is represented by 784 features, so the addition of two dimensions in x has little effect. In contrast, the fact that SWC obtained different absolute results for each classifier indicates that RFprox effectively leveraged the features. Experimental results for all classifiers and all tabular data sets, reporting Brier score and accuracy results,
are provided in Appendix A (Figures and
5.3 Similarity-basee calibration exploits hidden heterogeneity
Our second hypothesis was that HH helps explain why and when similarity- based calibration is effective. We examined Brier score improvement across large combination of data sets, classifiers, and random trials. We found that large HH can be present even in the absence of domain shift, which creates large opportunity for local calibration. Figure 2 shows the improvement (reduction) in Brier score after calibration as function of the average HH across the test set. The four binary data sets were MNIST "1" vs. "7" (relatively easy), "4" vs. "g" and "3" vs. "8" (intermediate), and "3" vs. "5" (difficult). The three multi-class data sets were "mnist "fashion-mnist", and "letter". We compared Platt (for binary) or temperature scaling (multi- class) calibration to SWC and SWC-HH for six classíficrs, using 10 trials per combination of data set and classifier. SWC and SWC-HH achieved Brier score improvements that correlate strongly with the amount of measured HH. The relationship cannot detect exploit HH. These results show that HH, which can be computed prior to calibration, is a useful diagnostic indicator that can guide the choice of calibration strategy. When HH is high, it is advisable to employ a similarity-based calibration method like SWC. When HH is low or there is not much calibration data, global methods such as Platt scaling are recommended.
far weaker for the global Platt/ hett/ttemeraturee scaling methods, which
To better understand how SWC and SWC-HH improve Brier score, we visualize the calibration process in Figure for the two-dimensional "moons" data set. Fach row corresponds results for different. classifier
8

---

Published in Transactions on Machine Learning Research (1/2023)
SWE Improvemenzin Brier score
0.8
0.7
0.-
0.7
Platers inerevement Beler
GAT
45
3.5
SVW
RBT3VM
05-
SYM
60F5/M
0.
5VM
inprovement
0.3
0.3
0.3
0.1-
01
0.0
0.0
0.2
0.3
0.6
0.7
0.0
0.1
0.2
0.4
0.5
0.6
0.2
0.3
0.6
0.7
Average per-item hiddeen heteregeneity
Average persitem hiodlen heterogeneity
Average per-item hioden heterogeneity
(a) Platt scaling (4 binary)
(b) SWC binary)
(c)
SWC-HH binary data sets)
Plaaeits improvenent Breer score
0.8
SWC Improvement score
0.8
SWC HH improvement Brier score
0.8
GBT
0.6
0.6-
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
9.0
0.2
0.4
0.6
0.8
3.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
3.0
Average per-item hidden heterogenellty
Average per-item hidden heterogeneity
Average per-item hidden heterogeneity
(d) Temperature scaling multi-class)
(e) SWC (3 multi-class)
(f) SWC-HH (3 multi-class data sets)
Figure 2: Brier score improvement versus average hidden heterogencity for four binary (top row) and three multi-class (bottom row) tabular data sels, for classífier types and 10 random trials.
Uncalibrated (SVM), Brier 0.218
Platt scaling, Brier 0, 223
Hidden heterogenelty
SWC, Brier 0.133
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.0
0.0
0.0
0.0
Uncalibrated (DT), Brier 0.156
Platt scaling, Brier 0.158
Hidden heterogeneity
SWC, Brier 0.129
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.2
0.2
-1
Uncalibrated (RF), 0.134
scaling, Brier
heterogeneity
1.0
1.0
0.6
0.6
0.6
0.6
0.2
0.2
0.2
02
0.0
0.0
Figure 3: Visualization of uncalibrated predictions (first column) for the "moons" data set, Platt scaling (second column), and SWC (fourth column). The third column shows hidden helerogeneity values that highlight areas of potential improvement, which align with SWC improvements.

---

Published in Transactions on Machine Learning Research (1/2023)
0.125
Brier score
0.100
Brier score
0.075
Uncal
0.3
15
0.050
150 9og
0.2
11st bin
0.025
5WC
5BA-30
0.1
SWCCH
0.000
resnet20 (0.02)
resnet56 (0.03)
repvgg a2 (0.02)
resnet20
resnet56
repvgg a2 (0.02)
(0.01)
(0.03)
Classifler
Classifier
(a) CIFAR-10
(b) CIFAR-100
Figure 4: Calibration performance on CIFAR-10 and CIFAR-100 using three pre-trained neural networks over 10 trials (error bars show standard error). HH is shown in parentheses.
(linear SVM, decision tree, and random forest). The third column shows the computed HH values. The classifiers were trained on 500 points, calibrated using 1000 points, and Brier score evaluated on 500 points. The linear SVM cannot model the nonlinear decision boundary very well. The diagonal region where the classes are mixed yet separable in the fcature space has a high value for HH. When SWC is applicd, it is the á values in this band that receive the biggest modifications. These changes reduce (improve) the Brier score from 0.218 to 0.133. Platt scaling increases (worsens) the Brier score slightly. The decision tree (second row of Figure 3 exhibits less hidden heterogencity on the same data set, because it is able to model the nonlincar decision boundary more effectively. SWC improves the Brier score from 0.156 to 0.129. Finally, the random forest. (bottom row of Figure 3) exhibits more areas with hidden heterogeneity, due to overly conservative predictions in the upper right and lower left areas. SWC creates smoother regions as the calibration data informs updates to the posterior probabilities and improves the Brier score from 0.134 to 0.122.
Importantly, the difference in results for the rightmost column in Figure 3 demonstrates that SWC adapts (calibrates) most where the classifier exhibits hidden heterogeneity, yielding a result that is customized to the original classifier and more flexible than global calibration.
5.4 Similarity- calibration for image classifiers
We also conducted calibration experiments with the CIFAR-10 and CIFAR-100 data sets (Krizhevsky 2009) using three pre-trained neural networks of increasing complexity E ResNet20 (He et al., 2016) has 20 layers and 0.27M parameters, ResNet56 (He et al.) 2016) has 56 layers and 0.85M parameters, and RepVGG A2 (Ding et al., 2021)) has 22 layers and 25.49M 49M parameters. For these data sets, the similarity measure sim used by SWC and SWC-HH operates in the latent space learned by each network. Specifically, we used the output activations of the avgpool (for ResNet models) and gap (for RepVGG A2) layers feature vector (dimensionality 64, 64, and 1408 respectively). We again used learned RF proximily function to compute similarity in this space. We found that probability radius of 0.05 yielded reasonably sized neighborhoods for computing HH.
Improvements for CIFAR-10 and CIFAR-100 are evident as more complex neural networks are trained; the Brier score consistently decreases from ResNet20 to ResNet56 to RepVGG _2 (see Figure 4 However, HH values were very low for all three networks (0.01 0.03), leaving little room for improvement with local calibration. Indeed, we found that global calibration (temperature scaling isotonic regression) yielded the best results for these data sets. Consistent with the results on tabular data shown in section 5.2 calculating HH in advance provides guidance as to which method will be most useful.
There is room for additional improvement. Recent studies of deep network latent spaces suggest that dis- tances computed neural network latent spaces often do not work well. For example, nearest neighbor classifiers using latent space distances perform substantially worse than the standard multinomial logistic
2Pre-trained models were obtained from http:////gggttobee///////atyyyte
10

---

Published in Transactions on Machine Learning Research (1/2023)
Support class P(7]: 0.960 0 653
055,520, p(7:: 1,000 0.951
184,990, P(7): 0.894 0.752
Support class P()): 0. 902 0.785
Support class p(7): 0.997 0.932
7
7
7
7
-
Figure 5: Test items from "mnist-Iv7" with the lowest calibration support for linear SVM classifier. For all five items, SWC reduced the confidence of the correct class, instead of increasing it.
regression (softmax) classifiers [Garrepalli] 2022). These latent spaces are also not able to represent di- mensions of variation that were poorly sampled the training data (Dietterich & Guyer 2022.. Tikewise, decision tree classifiers do not perform well on these Icarned representations |Garrepalli] 2022). Since HH is influenced by the data representation, hidden heterogeneity could be more detectable for these data sets using different representation. Likewise, even if HH is large, similarity-baan based local calibration may not always be able to improve the Brier score due to limitations of the representation. Employing t-SNE or PCA to reduce dimensionality for similarity caleulations (as done by Luo et al. (002)) could also be beneficial. Exploring the connection between choice of representation and calibration efficacy is an important future direction.
5.5 Calibration support highlights calibration data gaps and domain shift
Our third hypothesis that measuring calibration support, which is unique capability of similarity-based calibration methods, provide useful information about the relevance of the calibration set to each item being calibrated. We define the calibration support S for item t that informs (((tt) as the sum of similarity weights for items drawn from calibration set C:
Si. Σ st CC.
(6)
Identifying items with low values for Sr.c can draw allention to observations that are not well represented by the calibration set. These could be individual outliers or, if there is large number of such items, they could indicate distribution shift between the calibration and test sets. Low St,c values signal the need for more data (or more represent ative data) to be added to While previous studies focus solely on calibration performance as a function of the total calibration set size, similarity-base calibration can characterize the relevance of the calibration set individual test items.
Consider linear SVM trained on 500 MNIST "1" and "7" digits and calibrated using SWC with 3000 digits. For most items in the test set, calibration improves. However, examining items with low calibration support helps us understand failures. Figure 5 shows the five test items (out of 500 total) with the lowest calibration support. Calibration with SWC was detrimental (4iv) decreased) for all five. These items are not necessarily ambiguous in an objective sense, but the fact that they have low representation in the calibration set signals (correctly) that the calibrated output may be less reliable.
Likewise, low calibration support can provide warning when domain shift is present between the calibration and test sets (or any new prediction). We simulated domain shift (covariate shift) by rotating subset of the test items 90 degrees counter-clockwise Figure 6 ((a) shows the distribution of calibration support values without any rotation; the peak value is around 900, and items with low support are rare. With 10% of the test set rotated (iguuee6(bb), the overall histogram changes little, and the rotated items (orange bars) tend have low calibration support, signaling a need for inspection. With 50% (partial domain shift) rotated (Figure B(c),, distinct populations for the rotated and unrotated items are clear, and with 100% rotated (complete domain shift), the whole histogram shifts to lower values (peak around 300).
11

---

Published in Transactions on Machine Learning Research (1/2023)
100
100
100
Westetee Notriet
Iecoltted fotiet
Uerelated
80
80
80
80
Counts
Counts
Counts
Counts
40
20
200
400
1000
0
600
200
500
600
Calibration support
Calbbitton support
Calbratiio support
Calbbratinn support
(a) 0% rotated
(b) 10% rotated
(c) 50% rotated
(d) 100% rotated
Figure 6: Distribution of calibration support values for 500 test items ("mnist-lv7") classified by a linear SVM with progressively more of the test set items rotated. Bar plots are stacked.
This result suggests that in a deployment setting, it is useful to monitor the calibration support values that are reported by SWC. Knowledge about typical support values (or better, their distribution) could enable early detection of domain shift when new items originate from a changed distribution. That signal indicates that the calibration set, and likely the trained model as well, require revision. Platt, temperature scaling, histogram binning, and other methods provide a fixed mapping k(p) without regard to the item being calibrated; there is no signal to indicate whether is still relevant. SWC provides an intrinsic measure of calibration set relevance through the calibration support values obtained by each new item.
6 Conclusions, Limitations, and Future work
In this work, we explored the benefits of local, individual calibration for each test item, in contrast to widely used global classifier calibration methods. We identified hidden heterogeneity (HH) as strong indicator of the need for local calibration, when there are subpopulations within data set that have the same uncalibrated predicted probability á yet require different corrections to achieve well-calibrated probability g. We provided a method for calculating HH before calibration to inform selection of the calibration method. Experiments with tabular data sets and diverse machine learning classifiers indicate that local calibration improves Brier score in proportion to the average hidden heterogeneity (HH) value in the data set. We highlight this finding as an important step towards not only correcting miscalibration but also explaining and understanding it. When HH is very low (as we found with several deep neural networks), or little calibration data is available, global methods such as temperature scaling are sufficient, but otherwise, local calibration is preferred. On the other hand, because local calibration has far more degrees of freedom than parametric, global methods, it tends to require more calibration data. If calibration data is scarce, global methods may be preferred.
We proposed a similarity- approach to local calibration (SWC) that weights evidence from the calibra- tion set according to its similarity to the test item an "augmented" feature space that includes both the input features and the predicted class probabilities. This concept goes beyond prior work such as Similarity- Binning Averaging (SBA-10), which calibrates (without weighting) using the 10 nearest neighbors based on Faclidean distance in the augmented feature space (Bella et al. 2009). In most cases, we found that SWC out-performs 10. Additional benefits can be obtained by incorporating HH directly into the SWC algo- rithm (SW A final and unique benefit of similarity-based calibration is that the explicit measurement of calibration support can serve to warn when a given test item lacks good representation in the calibration set. This can also be an indicator when distribution shift or domain shift is present.
Runtime. The computational cost of local calibration methods tends to be higher than that of global hods since each item is independently modeled rather than constructing a single model to apply to all items. However, this also means that calibration can be conducted lazily, as needed, given a similarity measure. The computation of hidden heterogeneity requires (1) the identification of an item's nearest neighbors in the probability simplex AK-1, which can be costly with large number of classes, and (2) training a specialized classifier to estimate the potential Brier score improvement (Fquation 3
Limitations:
12

---

Published in Transactions on Machine Learning Research (1/2023)
Limitations: Preservation of accuracy. SWC and SWC-HH are not rank- lank-preserving calibration meth- ods. This means that in addition to modifying the calibration properties of the predictions, they can also change the predicted class and therefore the accuracy of predictions. Improvements in accuracy are re- flected in improved Bricr scores. Temperature scaling, in contrast, does preserve rank and accuracy because, without a bias term, it cannol move iterns to the other side of the decision threshold. Some researchers favor annkpreserving methods (Zhang et al. 2020 Patel et al. 2021). since they seek improve calibration without sacrificing accuracy. However, this constraint also prevents them from increasing aceuracy, which is an outcome available to Platt scaling (via its bias term), histogram binning, SWC, etc. On the whole, we agree with Bella et al. (2013) that there is no need to preserve item rankings given the opportunity to improve both accuracy and calibration. However, we acknowledge that in some applications, there could be a need to choose a rank-preserving calibration method to ensure accuracy is unchanged (up or down) for user acceptance (Srivastava et al. 2020
Future work. There are several important directions for future work. It is possible that within the same data set, some items are best calibrated with global methods while others (where HH is present) benefit from local calibration. A hybrid method that selectively applies global/local calibration, or some combination of the two, for each lest. item could polentially out-perform either one. For a given problem, alternative choices for data representation and similarity measures could yield additional improvements. In addition, SWC is well designed to naturally accommodate domain shift, if the calibration data set is drawn from the shifted distribution. Sampling bias in the training set, whether intentional or not, induces a particular kind of domain shift that is especially important to address to meet fairness goals when predictions are made in a deployment setting.
Author Contributions
KW: Problem refinement, algorithm development, implementation, experimentation, data analysis, and writing.
TD: Initial problem formulation, critical feedback, assisting with writing.
Acknowledgments
This matcrial is based upon work supported by the Defense Advanced Rescarch Projects Agency (DARPA) under Contract No. in this material are those of the authors and do not necessarily reflect the views of the DARPA. The authors thank the reviewers and the editor for valuable discussions that significantly improved the paper.
Any opinions, findings and conclusions or recommendations expressed
References
Amr M. Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-correctod calibration is hard-to-beat. at. label shift adaptation. In Proceedings of the 37th International Conference 2020.
on Machine Learning, pp.
Antonio Bella, Cèsar Ferri, José Hernández- Orallo, and María José Ramírez- Quintann
averaging: A generalisation of binning calibration. In Intelligent Data Engineering and Aulomated Learning IDEAL 2009, Lecture Notes in Computer Science, volume 5788, pp. 341-349 2009.
Antonio Bella, Cèsar Ferri, José Mernández-Orally and María José Ramírez- Quareez--uinaaa... On the effect of calibration in classifier combination. Applied Intelligence, 38:566-585, 585, 2013.
Leo Breiman. Random forests. Machine Learning, 45:5-32, 2001.
Leo Breiman, Jerome H. Friedman, R. A. Olshen, and Charles J. Stone. Classification and regression trees. Chapman and Hall, New York, NY, 1984.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):
1-3, 1950. doi:
13

---

Published in Transactions on Machine Learning Research (1/2023)
Adele Cutler, D. Richard Cutler, and John R. Stevens. Ensemble machine learning: Methods and applications, chapter Random forests. Springer, 2012.
Thomas G. Dietterich and Alex Guyer. The familiarity hypothesis: Explaining the behavior of deep open set methods. Pattern Recognition, 132:108931, 2022. doi: (..0016/j.patcog2000 202.tcoo..222.00000 108931.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. RepVGG: Making VGG-style ConvNets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 13733 13742, Jun 2021.
Shohci Enomoto and Takeharu Eda. Learning to cascade: Confidence calibration for improving the accuracy and computational cost of cascade inference systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35 pp. 7331-7339, 2021.
C. A. T. Ferro and T. E. Fricker. A bias-corrected decomposition of the Brier score. Quarterly Journal of the Royal Meteorological Society, 138(668):1954-195) 1960, 2012.
Peter W. Frey and David J. Slate. Letter recognition using Holland-style adaptive classifiers. Machine Learning, 6:161-182 182, 1991. doi: 10.1007/BF00114162.
Risheek Garrepalli. Oracle analysis of representations for deep open set detection. arXiv, version of November 4, 2022. URL https///arxx..orr/aaa//220 11350
Mehmet Gönen and Ethem Alpaydm. Multiple kernel learning algorithms. Journal of Machine Learning Research, 12:2211 2268, 2011. ISSN 15324435.
Yunye Gong, Xiao Lin, Yi Yao, Thomas G. Dietterich, Ajay Divakaran, and Melinda Gervasio. Confidence calibration for domain generalization under covariate shift. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 8958-8967, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1321-1330, 2017. doi: 0555555/305581333000
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Sunil Kalmady, Weijie Sun, Justin Ezekowitz, Nowell Fine, Jonathan Howlett, Anamaria Savu, Russ Greiner, and Padma Kaul. Improving the calibration of long term predictions of heart failure rehospitalizations using medical concept embedding. In Proceedings of AAAI Spring Symposium on Survival Prediction Algorithms,, Challenges, and Applications, volume 146 of PMLR, pp. 70 82, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kriste Krstovski, David A. Smith, Hanna M. Wallach, and Andrew McGregor. Efficient nearest-neighhor search in the probability simplex. In Proceedings of the 2013 Conference on the Theory of Information Retrieval, pp. 101-108. 2013. doi: (1445///99178.249999
Meelis Kull, Telmo M. Silva Filho, and Peter Flach. Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statistics, 11:5052 5080, 2017.
Meelis Kull, Miquel Percllo-Nicto, Markus Kängsopp, Telmo Silva Filho, Hao Song, and Peter Flach. Obtain- ing well-calibrated multiclass probabilities with Dirichlet calibration. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
14

---

Published in Transactions on Machine Learning Research (1/2023)
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Proceedings of the 33rd Conference on Newral Information Processing Systems, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. doi: 1t.1109/5.72677
Rachel Luo, Aadyot Bhatnagar, Yu Bai, Shengjia Zhao, Huan Wang, Caiming Xiong, Silvio Savarese, Stefano Ermon, Edward Schmerling, and Marco Pavone. Localized calibration: Metrics and recalibration. arXiv, version of August 18, 2022. URL https://arxiv org/
Matthias Minderer, Josip Djolonga, Rob Romijuders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H.S. Torr, and Puneet K. Dokanía. Calibrating deep neural networks using focal loss. In Proceedings of the 34th Conference on Newral Information Processing Systems (NewrIPS), 2020.
Mahdi Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2901 2907, 2015.
Alexandru Niculescu-Mizil and Rich Caruana, Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference Machine Learning, pp. 625-632, 2005.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncortainty under dataset shift. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
Kanil Patel, William Beluch, Bin Yang, Michael Pfeiffer, and Dan Zhang, Multi-class uncertainty calibration via mutual information maximization-ban based binning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61 140:661-14,111 1999.
Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. An empirical analysis of back- ward compatibility in machine learning systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3272-3280, 2020. doi: (..1145/3394486.344
Christian Tomani and Florian Bucttner. Towards trustworthy predictions from deep neural networks with fast adversarial calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35(11), pp. 9886-98888 2021.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B. Schön. Evaluating model calibration in classification. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS). 2019.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying franework. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS). 2019.
15

---

Published in Transactions on Machine Learning Research (1/2023)
David H. Wolpert. Stacked generalization. Neural networks, 5(2):241-259, 1992. doi: 80023-1.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv, version of August 28 2017. URL https://arxvv.org/abb.. 2102.
10809
Yaodong Yu, Stephen Bates, Yi Ma, and Michael Jordan. Robust calibration with multi-domain tem- perature scaling. In Proceedings of the ICML 2022 Workshop on Spurious Corvelations, Invariance, and Slabilily, 2022.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the International Conference on Machine Learning, pp. 609-60 616, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 694-699, 699, 2002.
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-match Ensemble and compositional meth- ods for uncertainty calibration in deep learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 11117 11128, 2020.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. Tn Proceedings of the 37th International Conference Machine Learning, pp. 11387 11397, 2020.
16

---

Published in Transactions on Machine Learning Research (1/2023)
A
Appendix
This appendix provides experimental results for seven data sets and six classifiers, comparing similarity-basod calibration to other methods. In these experiments, we randomly sampled 10,000 items from each data set and randomly split them into 500 train (for binary problems) or 1000 train (for multi-class problems), 500 test, and 9000 for a calibration pool.
A.1 Binary tabular data
Figure 7 presents results across all classifiers and calibration methods for the four binary MNIST data sets after 3000 calibration items were employed. Complete numeric results are shown in Tables to Classifiers appear in order of improving (decreasing) Brier score for the uncalibrated predictions.
Platt scaling improved performance for the Naive Bayes and random forest classifiers, but it yielded little benefit for the others. Isotonic regression sometimes provided additional improvements, primarily for Naive Bayes, and usually out-performed histogram binning. Tn contrast, similarity-based methods were highly effective for all classifiers in reducing Brier score.
SWC-HH consistently provided the best results. The SBA-10 approach described by Bella et al. (2009) weights all ten neighbors equally. We learned (Ferri, personal communication, Oct. 29, 2022) that the SBA authors have subsoquently employed weighting in the averaging process so that cach neighbor contributes inversely to its distance from the item to be calibrated. We included the weighted variation in our ex- periments, denoted "SBAW-10", Weighting provides slight advantages over SBA-10 in some cases, but SWC/SWC-HH yielded the best results. We can view SBAW as an intermediate choice between SBA and SWC, as it adopts the distance weighting of SWC but not the other aspects (supervised distance metric, HH filtering) that make SWC-HH the strongest method overall. As noted earlier, and most evident in Tables | to both SBA and SBAW generate nearly identical results for all classifiers, because they are dominated by feature space distance, and the classifier's initial predictions have little influence. In contrast, SWC and SWC-HH adapt to each classifier's individual limitations (hidden heterogeneity). SWC/SWC-HH improvements correlate with the average HH value, shown in parenthesis under the axis labels. In addi- tion, similarity-basee calibration also increased test accuracy (see right column of Figure 7 and subtables in Tables to SWC-HH consistently achieved the highest accuracy.
Table 1: Results for mnist-lv7 (Neat 3000, 10 trials). The best result(s) for each model (within 1 standard error, shown as subscript) are in bold.
Brier
Model DT
Uncal.
Platt
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
0.0678g.o oo8
0.067 lo.00s
0.0650p.006
0.0913o.co 009
0.0203g.om 003
0.0198g.0m
0.0226g.
0.0207g.00r
NB
0.0452g.om 0004
0.0442o.om 004
0.0423p.om 0033
0.0674o 022
0.0200o.co 003
0.01950.00 .003
0.0165g 002
0.0168..om
RBFSVM
0.0411g.ouz
0.0455g.u04
0.0365p.ouz
0.0358u.orz
0.0201..por
0.0197
p.003 0.003
0.0189
0.0172g.00z
GBT
0.0380g.u00
0.0390u.um .004
0.0282
0.0284u.oo
0.0203
0.0198p.ous
0.0205u. .002
0.0191g.um
RF
0. 0.0246g.ouz
0.01 0.0170p.om 70p.0u2
0.0173u. 7211777.000 0022
0.0182g.00z
0.02020.0or 0o0
0.0197g.003
0.0182u.oor
0.0171 ..0171.000
SVM
0.0155p.002
0.0157p.002
0.01488.co
0.0155g.om
0.0200p.003
0.0196g.oog
0.0149h.00
0.01544.cm
Accuracy
Model
Uncal.
Platt
Iso Reg
Hist. hin
SBA-10
SBAW-10
SWC
SWC-HH
DT
0.9586,.o0 .044
0.9592g.od .044
0.9614p.00d
0.9484o.co
0.9866..00
0.9880p.00
0.9852o.001
0.9874..om
RBESVM
0.97400 003
0.9738o.om 002
0.9718p.00r
0.9750p.oo
0.9868o.o0
0.9880b.00
0.9876g.002
0.9906..00
NB
0.9774g .022
0.9774g.om 002
0.9774p.00
0.9766o.oo
0.9868p.00
0.9880g.o0
0.9894..cm 001
0.9894..om
GBT
0.9788g.
0.9790g.0m
0.9808p.oom
0.9808u.co
0.98666.002
0.9880p.001
0.9864p.001
0.9878g.00
RF
0.9890g.00z
0.9898p.00r
0.9888p.001
0.98920.00
0.98660.001
0.9878g.001
0.9894p.oo1
0.9912g.o0
SVM
0.9898g.om
0.99044.com
0.9910n.con
0.9900n.001
0.98680.001
0. 0.98500.001
0.9902a.c0
0.99144.cm
17

---

Published in Transactions on Machine Learning Research (1/2023)
1.0
0.10
Uncal
Piat
Brier score
0.08
los Rag
Histiin
Accuracy
0.9
0.06
SBAM113
SixC SICC HH
SRA 30
Iso Rlg
Unco:
Paatt
0.04
0.8
Hist tin
5B4 10
SB4I 10
5MCC SMCG-H
0.00
DT
NB RBFSVM GBT RF
SVM
SVM (0.06) (0.04) (0.05) (0.03) (0.02) (0.01)
DT RBFSVM NB GBT
(0.06)
(0.05) (0.04) (0.03) (0.02)
(0.01)
Classifier
Classifier
(a) mnist- 1v7 Brier score
(b) mnist- 1v7 accuracy
1.0
0.5
Unca:
Part
Brier score
0.4
Hist
Iso Rog
Accuracy
5166-10
SRAWK
SWC
Rog
0.2
0.8
0.1
SBAWN 10
5
SWCC-HH
0.0
0.7
DT
RF
SVM RBFSVM GBT
DT SVM RBFSVM GBT
RF
(0.50)
(0.16)
(0.08)
(0.07) (0.07) (0.06)
(0.50)
(0.16)
(0.07) (0.07) (0.06) (0.08)
Classifier
Classifier
mnist-4v9 Bricr
(d) mnist-4v9 accuracy
1.0
Brier score
Hist bin
Beg
Accuracy
0.3
SB4W 10
3166. 10
0.9
Uhco
SWI
Pial
SWC *+
0.1
0.0
0.7
NB
DT
RF
SVM RBFSVM GBT
NB
DT RBFSVM SVM
RF
GBT
(0.40)
(0.15)
(0.07) (0.06) (0.06) (0.04)
(0.40)
(0.15) (0.06) (0.06)
(0.07) (0.04)
Classifier
Classifier
(e) mnist-3v8 Brier score
(f) mnist-3v8 accuracy
0.4
1.0
Uncal
Brier score
0.3
Accuracy
0.9
Uncs
0.2
SWC
Plat
SWS-Itt
Iso Rog
0.8
18sd bin
SB4 104-10
0.1
SBAWN-10
SWC
5WC-111
0.7
NB
DT
SVM
RF RBFSVM GBT
NB
DT
SVM RBFSVM GBT
RF
(0.35)
(0.22)
(0.08) (0.08) (0.06) (0.04)
(0.35)
(0.22)
(0.08) (0.06) (0.04) (0.08)
Classifier
Classifier
(g) mnist-3v5 Brier score
(h) mvist-3v5 accuracy
Figure 7: Calibration performance (left) and accuracy (right) for the binary MNIST data sets (10 trials; error bars indicate standard error). Classifiers sorted in order of improvement based on the uncalibrated classifier's 's score, and average HH values are below each classifier.
18

---

Published in Transactions on Machine Learning Research (1/2023)
Table 2: Results for mnist-4v9 (Neat = 3000, 10 trials). The best result(s) for each model (within 1 standard error, shown as a subscript) are in bold.
Brier score
Model
Uncal.
Platt
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.4932
0.3472a.007
0.2216
0.2724p.006
0.067
0.0653p.002
0.05666.000
0.03350.008
0.1657..007
0.1679a.or 0077
0.1558a.oor 0oG
0.1651p.006
0.0655o. 002
0.0638p.o02
0.0585g.u03
0.04366..00
RF
0. 0.1005u.bor
0.0609a.ous
0.0619u.oe
0.0627p.004
0. 0.0659u.ur
0.0642p.ou2
0.0528g.uur
0.0410p.008
SVM
0.0975p.000 004
0.0988g.008 .05
0.0984n.00 005
0.0993g.005
0.0655g.00 002
0.0638g.002
0.05470.004
0.04620.004
RBFSVM
0.08630.00 004
0.0912g.or 004
0.0827p.om 004
0.0861o.004
0.0652n.or 002
0.06366.002
0.0518p.om
0.0393
GBT
0.07410.000
0.0773a.007 000
0.07020.006
0.0717p.006
0.0647o.om 0000
0.0630p.002
0.0522g.003
0.04130.003
Accuracy
Model
Uncal.
Platt
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
0.7534g.dr .000
0. 0.7534, 008
0.8720g.00 007
0.8710g.00s
0.96344. 002
0.9624g.003
0.9602g.or 002
0.98160.00
DT
0.8846u. 001
0.8834a.00s
0.8928u.
0.8856p.uus
0.9656u. 002
0.9642o.ous
0.9600s.u03
0.97520.001
SVM
0.9382u.o0r o03
0. 0.9368g.003 .00a
0.9360u.od
0.9358p.004
0.9648u. 0012
0.9638p.ouz
0. 0.96466.uvz
0.97260.002
RBFSVM
0.9420g.co .033
0.9424g.cm
0.9406g.oo .033
0.94100..ons
0.9652 002
0.96422.o0s
0.96566.002
0.97860.001
GBT
0.95344.000 .055
0.9532g.oos
0.95388.oo 0044
0.9514g.o0s
0.96644.om 002
0.96466p.003
0.96566.002
0.97720.002
RF
0.95860.000 .033
0.95944.008
0.9584g.oo 003
0.9594o.003
0.9648o. 002
0.96322.008
0.9644g.008
0.97740.002
Table 3: Results for mnist-3v8 (Neat 3000, 10 trials). The best result(s) for each model (within 1 standard error, shown as a subscript) are in bold.
Brier score
Model
Uncal.
Platt
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.4212p.00 054
0.3029g.oo 02a
0.1749..0 004
0.2136o.013
0.0451g.or
0.0446o.00 .02
0.0613p.o0
0.04296.00
DT
0.14770.oo0
0.1465a.ud
0.1359g.mg
0.1494p.007
0.0448p.o0z
0.0443u.
0.0594u.004
0.0486u.bo
RF
0.0902u.o03
0.0590g.004
0.0595u.00
0.06140.004
0.0452p.002
0.04466.00r
0.0559p.004
0.0441u.006
SVM
0.0833a.om 005
0.08418..om
0.08447..o 005
0.0871p.00s
0.0450p.002
0.0444o.om
0.05390.005
0.04544.000
RBFSVM
0.0733n.004 .004
0.0764g.00s
.005
0.0682 0004
0.0707p.004
0.0448g.002
0.0443p.002
0.04944.005
0.0418p.oo
GBT
0.064460.00 .006
0.0657g.om.
0.0581o.oo .04
0.0593p.004
0.04466.002
0.0441o.00 002
0.0508$.00
0.0445
Accuracy
Model
Uncal.
Platt
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.7878o.oo 027
0. 0.7888p.oo 027
0.9132g.om 005
0.9104g.004
0.9744p.002
0.9750 002
0.9620p.00s
0.9760p.oo
DT
0.9030u.00a o008
0. 0.9024a.om o0e.
0.9116u.od .006
0.9054p.006
0.9744p.000
0.9754u.001
096122.002
0.9726g.002
RBFSVM
0.9472o.om .003
0.9480u.or 0044
0.95220. .003
0.9488o.oor
0.97444.000
0.9754p.00
0.9684p.or
0.97788.o00
SVM
0.9488o.000 .033
0.9486g 003
0.9480g.0o0 .004
0.9490o.003
0.97444.002
0.97540.oo 0002
0.9630o.00
0.97320.003
RF
0.9608a.oom 0003
0.9624g.om
0.9616n.co
0.9608o.oos
0.97.44..om
0.9756o.co
0.9644p.oos
0.9766g.nos
GBT
0.9616n.oo 0044
0. 0.9620g.om
0.9630p.om 003
0.9628p.003
0.97444.00r
0.9758p.oo
0.9686p.003
0.9758
A.2 Multi-class tabular data
Figure 8 and Tables 5 to present results for the multi-class data sets "mnist10", "fashion-mnist" and "letter", after 5000 calibration items were employed. Temperature scaling in general only improved Brier score for the tree-based methods (RF, GBT), and not consistently. Histogram binning was again beneficial for the Naive Bayes models, but it often made calibration worse for decision trees. Isotonic regression yiclded small additional improvements.
For "m and "fashion-mnist" similarity-based calibration provided the best results. SWC and SWC- HH out-performed SBA-10. SWC-HH usually improved over SWC, except for the more challenging "fashion- mnisu" data set. In this data set, the filtering employed by SWC-HH (lo ignore calibration items with insufficient similarity) often resulted in no calibration items remaining. We handle this case by using the single nearest neighbor, even if its similarity below the threshold. This leads to values for & that are based only on one calibration item. In many cases, the single nearest neighbor belongs to the correct class, yielding good accuracy, but when it is from an incorrect class, the Brier score penalty is large. However, the
19

---

Published in Transactions on Machine Learning Research (1/2023)
Table 4: Results for mnist-3v5 (Neat = 3000, 10 trials). The best result(s) for each model (within standard error, shown as a subscript) are in bold.
Brier score
Model
Uncal.
Platt
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.3670g.oz
0. 0.2906g..14
0.20844.oo
0.2350p.012
0.05980.om
0.0592g.003
0.0522g.00x
0.031 1.03150.008
0.2265a.po8 pos
0.2200g.oms oos
0.2081a.0x0 000
0.2141p.007
0.05950.0r 003
0.0589p.o0s
0.05446p.u04
0.03540.008
SVM
0.1034u.bor
0.1040a
0.1038u.0r
0.1070p.ous
0.0592u.um
0.0586p.ous
0.05499.u00
0.04288.000
RF
0.1021o.00
0.0524g. 0004
0.0537 0.3
0.0552g.o0s
0.0597 0008
0.0591g.00s
0.0455g.om
0.03440.00s
RBFSVM
0.0756g.00.
0.0742g.om 003
0.0689 003
0.07060.00s
0.0587p. 003
0.0582p.00s
0.0442o.000
0.0380o.00
GBT
0.06150.ce 55..500.0 .055
0.0649g.00s
0.0593g.00 005
0.05900.004
0.05870.00
0.0581p.000 p.003
0.04640.004
0.03930.004
Accuracy
Model
Uncal.
Platt
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
0.8158o.01 011
0.8160g 011
0.8092g.00 007
0.8654g.007
0.9632o 003
0.9662g.002
0.9662o.o0 0003
0.98160.00
DT
0.8500u. 009
0.85400..mr
0.8568u.od 000
0.8566u.uus
0.9634u.um oo3
0.9656p.ou2
0.96422..uz2
0.97840.002
SVM
0.9328u.wor 0004
0. 0.9340u.oog 0044
0.9330u.0d 0003
0.9328p.ous
0.9638u. ooa
0.9660p.ous
0. 0.9628p.u0a
0.9744p.002
RBFSVM
0.9454o.cm 0033
0.94544.oom
0.9544n.oom 003
0.9524p.002
0.96440p.om 003
0.9662o.002
0.96944.om
0.9788p.002
GBT
0.95940.oor .022
0.9600g.oo
0.9590p.oo 0003
0.9600p.o03
0.9636p.or 003
0.9660p.003
0.9682g.on
0.9780p.002
RF
0.9634o.002
0.9648g.002
0.9632o.00 .002
0.9634o.002
0.9632o.00 003
0.9658p.00s
0.9700g.002
0.98120.00
SWC-HH results were still comparable better than SBA-10 and the global calibration methods on this data set. In addition, SWC-HH yielded the best accuracy.
The "lctter" data set is umusual in that SBAW-10 achieved the best results (Figure 8 f) and Table 7 except for the random forest classifier, which is best calibrated using SWC or SWC-HH. SBAW-10 on this data set also outperforms the unweighted SBA-10 approach described by Bella et al. (2009) We interpret this mean that "letter" exhibits even stronger subpopulation locality than the others we have studied. These results reinforce the importance of employing some form of weighting when calibrating using similarity. However, the choice of 10 neighbors to use does not always work best, and the ideal constant would be difficult to estimate in advance. Therefore, we recommend the use of the entire data set (via SWC or SWC-HH) as a more robust solution.
Table 5: Results for mnist10 (neat 5000, 10 trials). The best result(s) for each model (within standard error, shown subscript) are in bold.
Brier
Model
Uncal.
TS
Iso Reg
Hist bin
SBA-10
10
SWC
SWC-HH
0.81930.015
0.8031 015
0.4802o.00 .005
0.5910p.o13
0. 0219 005
0.1702g.005
0. 0.1637p.004
0.1164g.001
DT
0.5119g.0r
0.5128g.um ooss
0.4832u.oo o08
0.5442o.o11
0.1722u. 001
0.1695p.ous
0.16788.004
0.1236p.004
RF
0.2977u.0o o003
0.
0.1505u.00 0004
0.1620p.aus
0.1726u.oe o0s
0. 0.1699p.ous
0.1323u.00 00s
0.131
GBT
0.2199g.oo 005
0. 0.2102g.cm .004
0.2061 004
0.2090
0.1713p. 005
0. 0.1686p.o0m
0.14469..om
0.1411g.om
SVM
0.1926p.oom 003
0. 0.1805g.od
0.1786p.co 003
0.1830p.003
0.1719p.0m 005
0.16920.005
0.1347 .04
0.13388.004
RBFSVM
0.1919
0.18544..om
0.1766n.oo
0.1813p.004
0.17170.o00
0.1690p.o0s
0.13120.004
0.1223g.004
Accuracy
Model
Uncal.
TS
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.5895p.cm 007
0.5895g.or .007
0.6806o.o0 004
0.5779p.02s
0.8841o 003
0.8909p.00s
0.8991o.00s
0.9415g.009
DT
0.6495g.000 .088
0.6495g.o00 .088
0.6551g.oom 007
0.6427p.007
0.8847o. 003
0.8910p.003
0.8947p.004
0.9368g.002
GBT
0.8589u.004
0.8580u.004
0.8618u.0az
0.8631p.004
0.88560.00z 003
0.8918p.ouz
0.9018p.00s
0.9091g.002
RBFSVM
0.8647o.oom
0.8647a.003
0.87770.co
0.87650.002
0.8852 003
0.8914p.003
0.91590.003
0.9230p.008
SVM
0.8785n.002
0.8786g.002 .002
0.8772n.002
0.8780p.oos
0.8853n.cm oo3
0.8912p.o0s
0.9112n.00 0033
0.9118g.ons
RF
0. 0.9006n.com
0.9006g.cog
0.90-43n.oom
0.9023p.004
0.88419p.oo 003
0. 0.8908p.00s
0.9123..oo
0.9134..ons
20

---

Published in Transactions on Machine Learning Research (1/2023)
0.8
T5
Isc Reg
0.9
Brier score
0.6
Accuracy
SBA-10
SBAW-10
0.8
Uncal
SWC-HH
0.7
Hist
Iso Rog
0.2
0.6
SBAW
SWC
0.0
0.5
NB
DT
GBT SVM RBFSVM (0.02) (0.06)
GBT RBFSVM SVM (0.78) (0.47) (0.06) (0.06) (0.02) (0.01)
(0.78)
(0.47)
(0.01) (0.06)
Classifier
Classifier
10 Brier
mnist.10 accuracy
1.0
Uncal.
1.0
15
Brier score
0.8
Hist bin
Isc Rea
Accuracy
SBAW-10
0.8
Uncal
SWC
Rog
0.7
Hist bin
$156-10
0.2
0.6
SWC
SWG-H
0.0
0.5
NB
DT
GBT
RF
SVM RBFSVM (0.18) (0.23)
NB
DT
GBT RBFSVM SVM (0.24) (0.23) (0.18) (0.13)
RF
(0.89)
(0.49)
(0.24) (0.13)
(0.89)
(0.49)
Classifier
Classifier
(c) fishion-mnise Brier score
(d) fashion-mnin mnist accuracy
1.0
Lncal.
15
Brier score
0.6
0.9
Hist bin
Accuracy
SBA-10
Uncal
0.4
SWC SWC-HH
TS
0.7
Hist
0.2
SBA 10
0.6
SWC
SWCHH
0.0
0.5
NB
DT
SVM
RF RBFSVM GBT
NB
DT
SVM RBFSVM GBT
RF
(0.11)
(0.47)
(0.00) (0.00) (0.01) (0.03)
(0.11)
(0.47)
(0.00) (0.01) (0.03) (0.00)
Classifier
Classifier
(e) letter Brier score
(f) letter accuracy
Figure 8: Calibration performance (left) and accuracy (right) for the multi-class "mnist10", "fashion- mnist and "letter" data sets (10 trials; error bars indicate one standard error). Classifiers are sorted in order of improvement based on the uncalibrated classifier's score, and average HH values are below each classifier.
21

---

Published in Transactions on Machine Learning Research (1/2023)
Table 6: Results for fashion-mnist (Noot 5000, 10 trials). standard error, shown as subscript) are in bold.
The best result(s) for each model (within 1
Brier score
Model
Uncal.
TS
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.95860.oaz 032
0.9415g.os2
0.59266.00 000
0.6247p.oos
0.2857u. 005
0.2833p.ous
0.2531o.008
0.2991a.om 0ess
DT
0.561
0.5579g.0m
0.50320.00
0.6216p.009
0.28550. 005
0.2832p.005
0.25366.00
0.29966.00
GBT
0.39766.0z 012
0.3493g.008
0.3370o 008
0.3427p.008
0.28450. 005
0.2821 0.005
0.2504g.00
0.2859o.0m 006
RF
0.3400p.oom
0.2997g.007
0.29130.com
0.3000p.006
0.2855n.cdd 005
0. 0.2832p.00s
0.24977.o00
0.26740.000
0.3333d.por 007
0.32266.oo .007
0.3101n.oom D00G
0.3176p.007
0.2856p. ooo
0.2832p.005
0.24288.oop
0.2682n 0.08
RBFSVM
0.3327a 000
0.3290o 007
0.3120g 008
0.3182o.cc
0.2855o 005
0.28320.005
0.2486o.cm .077
0.2807p 007
Accuracy
Model
Uncal.
TS
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.5202g.016
0.5202g.or
0.5950g.00m
0.5716p.010
0.7986n. 006
0.8022p.006
0.8206p.005
0.8498g.004
0.6402g.008 0008
0.6402g.om 00s
0.6510p 1o.000
0.5760o.016
0.7988o. 006
0.8014p.006
0.82000.005
0.8476g.001
GRT
0.7584o.dm .088
0.7584o.co o0s
0.7702o.om 007
0.76766.c0
0.79960 006
0.80280.000
0.82466.om
0.83520.00s
RBFSVM
0.76344.o00 .005
0. 0.7634d.o05
0.7710g.oom .077
0.7708p.oos
0.79980. 0oG
0. 0.8022p.006
0 0.8244p.004
0.8426g.004
SVM
0.7738u.oor D0G
0. 0.7738a.uo
0.78040. 004
0.77820.004
0.7996u.cm DoG
0. 0.8022o.ous
0.83120.000
0.84066.00x
RF
0.7872p.ck 005
0. 0.7872a.00s
0.7912g 008
0.7902o.006
0.7996o. ooe
0.8024g.006
0.82620.006
0.83366.006
Table 7: Results for letter (Neat = 5000, 10 trials). The best result(s) for each model (within standard
error, shown as a subscript) are in bold.
Brier score
Model
Uncal.
TS
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.5342g.00 007
0.5073a.0d .05
0.4753o. 005
0148999.00s
0.2284g. 004
0.1 0.1863g.00 .003
0.2603p. 005
0.2527o.000
DT
0.49755..00 .04
0.5237g.003
0.4839 004
0.7296p.006
0.2243p 004
0.1812a.00
0.2492g.006
0.2576p.oo
SVM
0.331 0..022
0.2629g.omg
0.2632g.oma 0003
0.2806p.ous
0.2141
0.1737g.002
0.2107
0.2102p.004
RF
0. 0.2633u.oo
0.1769u.ons
0.1830g.co
0.2002p.ous
0.2121u. 004
0.1679g.
0.1630p.ood
0.1630u.mm
RBFSVM
0.2495u.004
0.2186g.ues
0.2189g.vas
003
0.2308p.ous
0.2056u.uo4
0.1673g.u0z
0.1963g.uur
0.1957o.auc
GBT
0.2237a.cm
0.21470.005
0.2129h.om .04
1.2219..00
0.1835n.cr
0.14911.om
0.21 0.2114g.om
0.2115p.005
Accuracy
Model
Uncal.
TS
Iso Reg
Hist bin
SBA-10
SBAW-10
SWC
SWC-HH
NB
0.6291..0m 005
0.6291g.o0 005
0.6511o.c 003
0.64920.00r
0.8438g 006
0.8881g.0m
0.8160p.oo
0.8251o.004
0.6321 003
0.6321g.0 003
0.6375g.00 003
0.51199.006
0.8465p 006
0.8904g.0mg
0.82230.005
0.8654g.003
SVM
0.8202g.om 002
0.8202g.0m 0022
0.8217o 003
0.8168o.om
0.8584g 005
0.8982g.om
0.8581g.0m
0.85890.00s
RBFSVM
0.8461u.v00
0.8461g.0n4
0.8459u.000 .044
0.8434p.ous
0.861 5u o0s
0.8998g.002
0.8701g.uu3
0.871 18.711
GBT
0.84790.oom 0004
0.8479g.om.
0.8492u.co o03
0.8474p.003
0.8774u.cm 004
0.91033.om
0.8629g.om
0.8633u.00
RF
0.8722n.cm 002
0.8722g.n02
0.8746n 002
0.8710p.002
0.8565n.cg 006
0.9013g.00g
0.8904g.on2
0.8904p.002
22


Reviews:
[{"review_id": "24SWT1xyh9", "reviewer": "Reviewer_DmA5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The text is clear and understandable.\n* There are experiments about tabular data as well as images.\n\nWeaknesses:\n* Local calibration has been studied before by Luo et al 2021 whom the current paper cites briefly but not in sufficient detail, given the relevance to this work. Therefore, the concept of hidden heterogeneity is not fully new, although it is new as a term. In some sense, HH means just 'not locally calibrated'.\n\n* Following this, it would be essential that the paper would compare against the method proposed by Luo et al 2021. Furthermore, there are many more methods that should be considered to include in the comparisons. Currently, the paper only compares against the most basic calibration methods. For example, the paper states about Platt scaling that 'No further improvements were achieved beyond 500 calibration items'. Of course, this is very much expected because Platt scaling only needs to fit 2 parameters which is easy to do on a small dataset and more instances do not help to improve much anymore. Therefore, it would be important to include some other existing calibration methods that either have more parameters or that are non-parametric. For example, including isotonic calibration in the experiments would be absolutely essential.\n\n* There is enough HH that can be exploited and improved over using local modelling only when the original classifier was not doing it already. As the authors discovered, neural networks on images are already exploiting local information and hence, HH values were too low to be exploited further. This highlights the general limitation of the proposed methods - they are only useful if the original classifier was not very good.\n\n* The definition of calibration at the beginning of the introduction is wrong. It has been defined through P(Y|X), whereas actually the conditioning should not be on X but on the output of the classifier. Otherwise, only the Bayes-optimal model would be well-calibrated. However, according to the standard definition of calibration, for example even a constant classifier that predicts the true class distribution P(Y) is also calibrated.\n\n* 'Our implementation of Platt scaling ...' sounds like the authors would have done something non-standard. However, this is exactly as how J. Platt originally proposed it and even the Python scikit-learn implementation of Platt scaling is doing the same, i.e. the targets are not {0,1} but instead smoothed probabilities.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper defines the new notion of hidden heterogeneity (HH) about situations where a model is predicting the same class probability vector to instances for which the true posterior probabilities are not the same. It then defines local calibration methods SWC and SWC-HH to reduce HH and improve classifier performance as measured by Brier score, and by accuracy as well. Experiments are performed on multiple tabular and image datasets. Results show that the proposed methods outperform Platt scaling, histogram binning, and temperature scaling in many situations.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The experiments should include some state-of-the-art calibration methods and also the classical isotonic calibration. The paper by Luo et al 2021 should be discussed in more detail and the relationship of HH to Luo's work should be discussed. It would be discussed more when the proposed methods are useful and when they would not help because of low amount of HH. The definition of calibration should be fixed.", "broader_impact_concerns": "No further concerns beyond what was mentioned above.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "whm6Asm4Hr", "reviewer": "Reviewer_ijrk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A fresh take on a classic problem – calibration – and challenging the convention that samples with the same predicted probability should be given the same calibration correction.\n- Experiments on different types of datasets, not just data from one type of modality.\n\nWeaknesses:\n- Hidden heterogeneity, the problem that the paper defined and is trying to solve for, is defined by comparing predictions from the original model and a newly trained model (a random forest, according to page 3). Hidden heterogeneity is then used as a gold standard to compare calibration methods. Yet, the calibration methods introduced in this paper leverage random forest proximity. This feels a bit like “training for the test”. This would be more convincing if there were some synthetic data experiments where hidden heterogeneity is introduced using different types of data mechanisms and showing that the calibration methods can still perform well. Or using different types of distances in the calibration methods – essentially a sensitivity analysis of distance type and how that impacts results.  \n- Could go deeper to try to explain some of the observed phenomenon. For example, in the results in Section 5.3, the random forest classifier “somewhat to our surprise, exhibits more areas with hidden heterogeneity”. Do the authors have any ideas why? In Section 5.4, when the authors said “using random forests to measure hidden heterogeneity in neural network latent spaces may fail to detect HH even though a better representation (or a better classifier) could detect that it is present”, again, a few sentences to discuss why would improve the presentation of these ideas and help the reader gain intuition into failure modes of the method.\n- Even if hidden heterogeneity (subpopulations where better calibration could improve accuracy) does exist, it is not clear that improving the calibration for this subpopulation is the preferred method to correct for this, if a better classifier could be trained. A very personalized calibration function could be harder to deploy in practice than a global calibration function. If a better model could be trained such that these subpopulations have better accuracy, this local calibration idea might not be needed. I think adding some of these baselines could be interesting: 1/ retraining the model, upsampling subpopulations found to have hidden heterogeneity; 2/ using more powerful model classes.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper challenges the convention that samples with the same predicted probability should be given the same calibration correction, and proposes to detect subpopulations where better calibration could improve accuracy, which the paper calls \"hidden heterogeneity\". The paper introduces two \"local\" calibration methods, Similarity-Weighted Calibration (SWC) and SWC-HH that leverage similarity between samples in terms of features. SWC and SWC-HH differ slightly, in that SWC weighs all points in a calibration dataset based on similarity to the given point x that is being studied, while SWC-HH uses only a subset of these points. The paper tests the method on tabular and image datasets. \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- [Critical in my opinion] Experiments where model is retrained, or more powerful model classes used, and show that they don’t outperform local calibration methods. \n- [Critical in my opinion] Experiments where hidden heterogeneity is introduced using different types of data mechanisms and showing how the calibration methods perform in this case.\n- [Critical in my opinion] Since SBA-10 is a key baseline compared against in this paper, it would be good to confirm if SBA-10 uses Euclidean distance , cf. \"Our implementation of SBA-10 employs Euclidean distance to identify the nearest neighbors, which we believe to be the method employed by Bella et al.\". \n- [Strengthen the work] I found the results in Section 5.4 a bit confusing. It sounds like the similarity measure used in SWC and SWC-HH is a feature space learned by the neural net. But that is followed by this line “We again used a learned RF proximity function to compute similarity.”. Clarification would be great. \n- [Strengthen the work] If the local calibration methods are not tied to a specific similarity measure, like random forest proximity, this could be made clearer in Section 4 when introducing the methods.\n- [Strengthen the work] For SWC-HH, from this line in Section 4: “HH, which is computed separately for each test item t, is provided as an additional input for calibration”, on which dataset split is this HH computed on? ", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2z5F3MBmrc", "reviewer": "Reviewer_94q8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed algorithm is fairly simple to understand. Instead of calibrating the probabilities globally, the main idea is to calibrate the probabilities locally using the features x too when there are model mis-specification. The design and steps in the algorithm are very clearly written. \n\n- The authors have done a fairly extensive evaluations on a few standard ML datasets such as MNIST and letters, and also on larger image datasets such as CIFAR10 and CIFAR100. They show improved probability calibrations compared to global calibration methods and competing local calibration method such as SBA-10, especially for weaker classifiers. \n\n- In the paper there are fairly interesting empirical analysis on how the local calibration improves classification accuracy (section 5.3), and how the calibration support set helps discover anomalies in the prediction (section 5.5). \n\n\nWeaknesses: \n- The current paper does not discuss the relation between the proposed method to model mis-specification. When different subpopulations U_i, U_j having the predicted probabilities f(x) but different true conditional probabilities, isn't that a sign of model mis-specification? (consider the cats vs birds example in section 3). When such a situation arise, instead of calibrating the probabilities with local regression (a post-hoc fix), we could revise the model using local regression or mixture of experts type of model. The authors do not seem to consider such a possibility but go straight to correcting the model using local probability calibrations. \n\n- Certain choices in the scoring of hidden heterogeneity and the algorithm seem arbitrary and not well-motivated enough. How different would the results be if we use a KL-divergence or L1-distance measure on the probability vectors instead of Hellinger distance? The algorithm also makes use of random forests in its computation of similarities between items, a decision that the authors did not discuss their reasoning. What would happen if we replace the RFprox measure with some other measures? \n\n- In Algorithm 1, the local probability predictor g_t works with a potentially much smaller sample. Wouldn't this cause instability problems in the calculation of the HH measure? For example, in Figure 1c, when the base classifier is powerful enough (random forest), local calibration methods such as SBA-10 or the proposed method actually hurts probability calibration for small sample size compared to global method such as Platt scaling, when the number of calibration items are small. Also, how is the radius r for neighborhood chosen? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a notion called 'hidden heterogeneity', which describes the situation when subpopulations having the same predicted probability exhibit different true conditional probabilities. They propose to remedy the situation with a local regression correction that can improve both the probability prediction and classification accuracies. They perform fairly extensive empirical evaluations on several datasets, and their method has the best improvements when the model underfits the data.  \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Discuss the relation of the proposed method to model mis-specification. \n- Discuss the rationale behind using Hellinger distance and RFprox as similarity measures in greater details. \n- Some of the texts in Figures 1, 2, 6 are too small to be readable. Please fix. \n", "broader_impact_concerns": "- There are no strong ethical implication from this piece of work on probability calibrations. \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the "hidden heterogeneity" (HH) problem, wherein global calibration of classifier outputs may fail due to the presence of latent sub-populations with similar predictions, but different characteristics. The paper proposes a metric to quantify HH, and proposes local calibration methods to combat it.

### Reviewer recommendations

Classifier calibration is a fundamental problem, and progress on this topic is of broad interest. Reviewers found the paper's treatment of calibration to be interesting, with results presented on both tabular and image data.

From the reviewers' final recommendations, two were (leaning) positive, while another leaned negative. The main critiques raised by the reviewers were as follows.

(1) _Applicability to powerful models_. One concern was whether the approach is limited to cases where the base model performs poorly. The authors responded that they focus on settings where the base model is given, and there is no access to the training data. They also noted that strong models could suffer from HH under distribution shift.

(2) _Impact of model misspecification_. One question was whether HH was simply a symptom of misspecification of the base model, which might be fixed by constructing an MoE-style model. The response was similar to the above.

(3) _Impact of different mechanisms of HH_. One question was whether different causes for HH could result in varied performance. The author's response was that they evaluate the impact of the model and data distribution, with the impact of data representation left for future work.

(4) _Limitation to post-hoc calibration_. One critique was that the paper focusses on post-hoc calibration, which may be a niche setting.

(5) _Comparison to existing work_. The preprint of Luo et al., which also studies local calibration, was not discussed in detail or compared to in the original version. The author response included a more detailed discussion, including limitations of this work.

(6) _Risk of fitting on small sample_. The algorithm relies on local probability estimates fit on a potentially small sample, which could bias the HH score. The author responded that this might indicate scenarios where good similarity-based calibration is not possible.

(7) _Justification for design choices_. The paper makes some design decisions (e.g., the use of Hellinger distance, RFProx as a similarity measure) that are not ablated. The response noted that the Hellinger distance is equivalent to the L2 distance between probabilities, and that RFProx is a standard similarity measure.

Of these, I find the author response (or paper's treatment) to be adequate for (1)-(5), and the reviewers mostly concur. I find (6) and (7) to potentially require more discussion, even after the revision. The discussion of the choice of RFProx in particular could be treated in more detail, and contrast against more common measures (e.g., Gaussian kernel similarity). That said, the paper does demonstrate that the use of RFProx leads to reasonable results in many practical settings.


### AE recommendation

The TMLR guidelines for acceptance are based on the following two questions:

(a) _Are the claims made in the submission supported by accurate, convincing and clear evidence?_

(b) _Would some individuals in TMLR's audience be interested in the findings of this paper?_

For (a), most of the critiques above were adequately addressed in the response. There are some remaining points that would improve the clarity of the work. Nonetheless, in my estimation, the paper's claims are generally well-supported.

For (b), the general reviewer consensus is the paper provides an interesting perspective on calibration. This is supported by my reading. I thus believe some readers would find the results of interest.

Given these, we recommend acceptance with revisions as suggested below.


### Suggested revisions

- Amend the definition of calibration. It would be easiest to provide a precise mathematical definition. If a textual description is to be retained, then it should be reworded to be clearer.

- Section 3 could be split into two sub-sections, one for "What is HH?", and another for "How do we detect HH?"

- (optional) For Algorithm 1 and 2, perhaps x_t rather than t is clearer to refer to the test item.

- Equation 3, 4, 5: use \text{.} for "Brier", "sim"

- In Section 4, what does the notation \langle x_t, \hat{p}_t \rangle mean? Vector concatenation? If so, perhaps [ x_t, \hat{p}_t ] or similar would be clearer (as the former is sometimes used to denote inner product).

- Regarding the use of RFProx in Section 4:

	- It is stated that "If a domain-specific similarity measure is not available, one can use a supervised similarity measure that can learn to place high importance on the predicted probabilities. We employ such a measure: the random forest proximity function (RFprox)." It is not clear from the description why the RFProx necessarily learns to place high importance on the predicted probabilities. Is the claim that it _can_ learn that these probabilities are _typically_ very important? Clarification is suggested.

	- Further to the above, it would be good to make explicit what features are fed into the RFProx method. Is it the concatenation of both x_t and \hat{p}_t? If the stated goal is to place high importance on the latter, why not just use the latter? Further discussion is suggested.

	- It is mentioned that RFProx can be viewed as a kernel. The reader may naturally wonder whether one can use more standard kernels, such as the linear or Gaussian kernel, on top of the concatenation of x_t and \hat{p}_t. This would appear place some importance on the predicted probabilities to measure calibration. Are such choices admissible? What is the reason for choosing RFProx over these? My understanding from the response is that these kernels would be considered "unsupervised", and thus potentially not adequately balance similarities in feature and predicted probability space; this point is however not too clear from the current Section 4. Further to the above, it is not clear whether just constructing a kernel over the predicted probabilities would not then be admissible. More discussion is suggested.

	- If indeed other standard kernels are admissible, it would be ideal if some additional experiments could be presented (potentially on a few setups) with one of these choices. This could demonstrate how important it is to pick an supervised similarity measure.

- The author response to a few critiques relied on the focus being on post-hoc settings. This is reasonable. It might however be interesting to discuss what strategies might be appropriate in settings where one does have the ability to modify the original model. Is it the case here that simply increasing model capacity fixes HH? Would there still be value in similarity-based local calibration?

- The authors stated that "Relatedly, if the calibration set as a whole is too small then a similarity-based method may not have enough information to perform good calibration. We previously made this point in the Conclusions and have now also added a statement about it to section 5.2". From my reading, these points were not too clear. They could be made more explicit.

Decision: Accept (poster)