Local Calibration: Metrics and Recalibration 
Authors: Rachel Luo
Aadyot Bhatnagar
Yu Bai
Shengjia Zhao
Huan Wang
Caiming Xiong
Silvio Savarese
Stefano Ermon
Edward Schmerling
Marco Pavone
Venue: ICLR 2022 Submitted (2022)

Abstract: Probabilistic classifiers output confidence scores along with their predictions, and these confidence scores should be calibrated, i.e., they should reflect the reliability of the prediction. Confidence scores that minimize standard metrics such as the expected calibration error (ECE) accurately measure the reliability on average across the entire population. However, it is in general impossible to measure the reliability of an individual prediction. In this work, we propose the local calibration error (LCE) to span the gap between average and individual reliability. For each individual prediction, the LCE measures the average reliability of a set of similar predictions, where similarity is quantified by a kernel function on a pretrained feature space and by a binning scheme over predicted model confidences. We show theoretically that the LCE can be estimated sample-efficiently from data, and empirically find that it reveals miscalibration modes that are more fine-grained than the ECE can detect. Our key result is a novel local recalibration method LoRe, to improve confidence scores for individual predictions and decrease the LCE. Experimentally, we show that our recalibration method produces more accurate confidence scores, which improves decision making and fairness on classification tasks using both image and tabular data.

Under review as a conference paper at ICLR 2022
LOCAL CALIBRATION: METRICS AND RECALIBRATION
Anonymous authors Paper under double-blind review
ABSTRACT
Probabilistic classifiers output confidence scores along with their predictions, and these confidence scores should be calibrated, i.e., they should reflect the reliability of the prediction. Confidence scores that minimize standard metrics such as the expected calibration error (ECE) accurately measure the reliability on average across the entire population. However, it is in general impossible to measure the reliability of an individual prediction. In this work, we propose the local calibration error (LCE) to span the gap between average and individual reliability. For each individual prediction, the ICE measures the average reliability of set of similar predictions, where similarity is quantified by kernel function on pretrained feature space and by binning scheme over predicted model confidences. We show theoretically that the LCE can be estimated sample-efficiently from data, and empirically find that it reveals miscalibration modes that are more fine- fine-grained than the ECE can detect. Our key result is novel local recalibration method LoRe, to improve confidence scores for individual predictions and decrease the LCE. Experimentally, we show that recalibration method produces more accurate confidence scores, which improves downstream fairness and decision making on classification tasks with both image and tabular data.
INTRODUCTION
Uncertainty estimation is extremely important in high stakes decision-making tasks. For example, a patient wants to know the prohability that medical diagnosis correct; an autonomous driving system wants to know the probability that pedestrian is correctly identified. Uncertainty estimates are usually achieved by predicting probability along with each classification. Ideally, we want to achieve individual calibration, i.e., want to predict the probability that each sample is misclassified.
However, each sample is observed only once for most datasets (e.g., image classification datasets do not contain identical images), making it impossible to estimate, of incorrect classification for individual samples. Because of this, commonly used metrics such as the expected calibration error (ECE) measure the gap between classifier's confidence and accuracy averaged across the entire dataset. Consequently. ECE can be accurately estimated but does not measure the reliability of individual predictions.
define, the probability
In this work, we propose the local calibration error (LCE), a calibration metric that spans the gap between fully global (e.g.. ECE) and fully individual calibration. Motivated by the success of kernel-based locality in other fields such as fairness (where similar individuals should be treated similarly) (Dwork et al., 2012: Pleiss et al., 2017) and causal inference (where matching techniques are used to find similar neighboring samples) (Stuart, 2010), we approximate the probability of misclassification for an individual sample by computing the average classification error over similar samples, where similarity is measured by kernel function in pre-trained feature space and binning scheme over predicted confidences. Intuitively, two samples are similar if they are close in a pretrained feature space and have similar predicted confidence scores. By choosing the bandwidth of the kernel function, we can trade off estimation accuracy and individuality: when the bandwidth is very large, recover existing global calibration metrics; when the bandwidth is small, approximate individual calibration. We choose an intermediate bandwidth, so our metric can be accurately estimated, and provides some measurement on the reliability of individual predictions.

---

Under review as a conference paper at ICLR 2022
Theoretically, show that the LCE can be estimated with polynomially many samples if the kemel function is bounded. Empirically, we also show that for intermediate values of the bandwidth, the LCE can be accurately estimated and reveals modes of miscalibration that global metrics (such as ECH) fail to uncover.
In addition, we introduce non- non-parametric post-hoc localized recalibration method (LoRe), for lowering the LCE. Empirically, LoRe improves fairness by achieving low calibration error on all potentially sensitive subsets of the data, such as racial groups. Notably, it can do without any prior knowledge of those groups, and is more effective than global methods at this task. In addition, our recalibration method improves decision making when there a "safe" action that is selected whenever the predicted confidence is low. For example, an automated system which classifies tissue samples as cancerous should request a human expert opinion whenever it unsure about classification. In simulation on an image classification dataset, we show that recalibrated prediction models more accurately choose whether to use the "safe" action, which improves the overall utility.
In summary, the contributions of our paper are as follows. (1) We introduce local calibration metric, the LCE, that is both easy to compute and can estimate the reliability individual predictions. (2) We introduce a post-hoc localized recalibration method LoRe, that transforms a model's confidence predictions to improve the local calibration. (3) We empirically evaluate LoRe on several downstream tasks and observe that LoRe improves fairness and decision decision-making more than existing baselines.
2 BACKGROUND AND RELATED WORK
2.1 GLOBAL CALIBRATION METRICS
Consider a classification task that maps from some input domain (e.g., images) X' a finite set of labels y = {1. .. A classifier is pair (f.p) where f X' - y maps each input x E X to label E y and p : + [0, 1] maps each input x to a confidence value c. Let Pr be a joint distribution on X' x 3i (e.g., from which training or test data pairs (z, y) are drawn). The classifier (f,p) is perfectly calibrated (Guo et al., 2017) with respect to Pr for all c E [0.1]
Pr[f = Y p(X) = =c
(1)
To numerically measure how well a classifier is calibrated, the most commonly used metric is the expected calibration eтror (ECE) (Naeini et al., 2015; Guo et al., 2017), which measures the average absolute deviation from Eq. 1 over the domain. In practice, given finite dataset, the ECE is approximated by binning. The predicted confidences are partitioned into bins B1, ...... Bg. and then a weighted average is taken of the absolute difference between the average confidence conf (B.) and average accuracy acc(B) (B)) for each bin B₁:
Ba N
ECE(f.P) =
conf (B:) - acc (B,)l.
(2)
Similarly, the maximum calibration error (MCE) (Naeini et al., 2015; Guo et al., 2017) measures the average deviation from Eq. 1 in the bin with the highest calibration error, and is defined as
MCE(f.P) == max (conf(B.) acc(B,).
(3)
2.2 EXISTING GLOBAL RECALIBRATION METHODS
Many existing methods apply post-hoc adjustment that changes model's confidence predictions to improve global calibration. including Platt scaling (Platt, 1999), temperature scaling (Guo et al., 2017), isotonic regression (Zadrozny & Elkan, 2002), and histogram binning (Zadrozny & Elkan, 2001). These methods all learn simple transformation from the original confidence predictions to new confidence predictions, and aim to decrease the expected calibration error (ECE). Platt scaling fits a logistic regression model; temperature scaling learns single temperature parameter to rescale confidence scores for all samples simultaneously; isotonic regression learns a piece-wise constant monotonic function; histogram binning partitions confidence scores into bins {[0, <), [c, 2e), [1 e, 1]} and sorts each validation sample into a bin based on its confidence p(z);; it then resets the confidence level of all samples in the bin to match the classification accuracy of that bin.
2

---

Under review as a conference paper at ICLR 2022
2.3 LOCAL CALIBRATION
Two notions of calibration that address some of the deficits of global calibration are class-wise calibration and group-wise calibration. Class-wise calibration groups samples by their true class label (Kull al., 2019; Nixon al., 2019) and measures the average class ECE, while group-wise calibration uses pre-specified groupings (e.g., race or gender) (Kleinberg et al., 2016; Pleiss et al., 2017) and measures the average group-wise ECE or maximum group-wise MCE.
A few recalibration methods have been proposed for these notions of calibration as well. Dirichlet calibration (Kull et al., 2019) achieves calibration for groups defined by class labels, but does not generalize well to settings with many classes (Zhao et al., 2021). Multicalibration (Hébert-Johnse et al., 2017) achieves calibration for any group that can be represented by polynomial sized circuit, but lacks& a tractable algorithm. If the groups are known a priori, one can also apply global calibration methods within each group; however, this impractical in many situations where the groups are not known for new examples at inference time.
At an even more local level, Zhao et al. (2020) look at individual calibration in the regression setting and conclude that individual calibration is impossible to verify with a deterministic forecaster, and thus there is no general method to achieve individual calibration.
2.4 KERNEL-BASE BASSED CALIBRATION METRICS
Kumar et al. (2018) introduce the maximum mean calibration error (MMCE), a kernel-based quantity that replaces the hard binning of the standard ECE estimator with kernel similarity k(p(z), p(x")) between the confidence of two examples. They further propose to optimize the MMCE directly in order to achieve better model calibration globally. Widmann et al. (2019) extend their work and propose the more general kernel calibration error. Zhang al. (2020) and Gupta al. (2020) also consider kernel-based calibration. However, these methods only consider the similarity between model confidences p(x'), rather than the inputs x, x' themselves.
THE LOCAL CALIBRATION ERROR
Recall that commonly used metrics for calibration, such the ECE the MCE, are global in nature and thus only measure an aggregate reliability over the entire dataset, making them insufficient for many applications. An ideal calibration metric would instead measure calibration level; however, doing so is impossible without making assumptions about the ground truth distribu- tion (Zhao al., 2020). A localized calibration metric represents adjustable balance between these two extremes. Ideally, such a metric should measure calibration at a local level (where the extent of the local neighborhood can be chosen by the user) and group similar data points together.
individual
In this section, we introduce the local calibration error (LCE), a kernel-based metric that allows us measure the calibration locally around prediction. Our metric leverages learned features to automatically group similar samples into a soft neighborhood, and allows the neighborhood size to be set with a hyperparameter 7. We also consider only points with similar model confidence as the prediction, so that similarity is defined in terms of distance both in the feature space and in model confidence. Thus, the LCE effectively creates soft groupings that depend on the feature space; with a semantically meaningful feature space, these groupings correspond to useful subsets of the data. We then mention a few design choices and visualize LCE maps over 2D feature space to show that we use our metric to diagnose regions of local miscalibration.
3.1 LOCAL CALIBRATION ERROR METRIC
We propose a metric to measure calibration locally around given prediction. The calibration of similar samples should be similar, provides similarity scores, to define soft local neighborhoods. (, x²) has bandwidth y > O, which determines the extent of the local neighborhood similar (i.e., further away) samples x' have less influence on the local calibration metric at æ. Also, with the ECE and MCE (Eqs. 2 and 3), use binning and consider only the points in the same
use a kernel similarity function k., X x X' - R_, which
as y increases, the neighborhood grows. Less
3

---

Under review as a conference paper at ICLR 2022
confidence bin as Thus, the samples that influence the local calibration metric are similar in both features and model confidences.
More formally, let X' IR be feature map that transforms input feature vector, and let ky be parameterized as ky (I, x²) g((((z) ((<x)))>) for some Lipschitz function g R + Rt. Then given data point æ E X' and classifier (f, the local calibration error (LCE) of the model at æ is the expected difference between the model's confidence and accuracy on a randomly sampled data point x' Pr, weighted by the kernel similarity (r,
We say a probabilistic classifier (Ø, J) is perfectly locally calibrated with respect to k, if
(e''s (0(x') - 1[f(x') - (x,)), p(x') p(x)] = 0.
sup zEsupp(Pt)
=L.CE (x:f--))
Similar to perfect calibration, perfect local calibration is achieved by the Bayes- classifier. In general, perfect local calibration is a much stricter notion than perfect calibration due to localizing to each indiviudal data point x, and reduces perfect calibration if ky (x, x²) is trivial kernel.
To define LCE on a finite dataset, we perform an additional binning on the confidence to deal with the conditioning. Let D ((21, 1/)............... (IN, yN)) be a dataset, and 3(x) {i p(x,) E B($(z)) be the set of indices of the points in DD occupying the same confidence bin as æ. Then we can compute the LCE by
Eien(e) (f(z.) - -
(x;
(4)
EiEs(a) k(x,,,)
Note that the quantity (p(x:) (x;) = y:l) simply the difference between the confidence and the accuracy for sample 24. and the denominator is normalization term.
We then define the maximum local calibration error (MICE) as
MLCE, (S. max LCE,, (z;
(5)
Intuitively, the LCE considers a neighborhood about a sample æ (as defined by the kernel ky and the confidence bin B), and computes the kernel- weighted average of the difference between the confidence and accuracy for each sample in that neighborhood. Note that by changing the bandwidth we can interpolate the LCE between an individualized calibration metric (as 7 - 0) and global one (as 7 - oo). Lemma makes this more concrete under the assumption that lim (,, (proof in Appendix C). For example, the Laplacian and Gaussian kernels satisfy this condition. Lemma 1. As + co, the MLCE converges to the MCE.
Theorem shows that under certain regularity conditions, the finite-sample estimator LCE(2)) converges uniformly and sample-efficiently to its true expected value LCE"(z);
Theorem 1. (Informal) Let a < inf E [k, [x)) B(ğ(x))] be lower bound on the expectation of the kernel, and be the dimension of the kernel's feature space. If the sample size is at least Ödde odfa²²)) where > 0 is a larget accuracy level, then with probability at least 1 1 we have
sup [LCE, xex
(CE, f.p) <.
Here, hides log factors of the form (ommloog(////g)) (0g(1)))))). practice, a depends inversely on ?.
To summarize, the MLCE measures a worst-case individual calibration error as 7 - 0 (i.e., the effective neighborhood very small) and converges the global MCE metric 7 - oo (i.e., the effective neighborhood is very large). In practice, one must pick intermediate values of 7 to balance more local notion of calibration error with the sample efficiency of its estimation. A more formal statement and full proof of Theorem can be found in Appendix D.
3.2 CHOICE OF KERNEL. AND FEATURE MAP
In this work, we compute the LCE using 15 eiuth-width bins and use the Laplacian kernel
(,,,, exp [1o(x))) <(<x))l
dy
4

---

Under review as a conference paper at ICLR 2022
Because distances in nigh-dimensional imput space (e.g., image data) may not be meaningful on their own, we evaluate the kernel on a feature representation of x rather than on z itself. Features learned from neural networks have proven useful for a wide range of tasks, and they have been shown to capture useful semantic features of their inputs (Huh et al., 2016; Chen et al., 2020; Li et al., 2020). The kernel similarity term ki, (2r, x²) in the LCE thus leverages learned features to automatically capture rich subgroups of the data. For image data, we chose an Inception-v3 model as our feature map, since Inception features are widely accepted as useful and representative many areas (e.g., g., for generative models (Salimans et al., 2016)), though other neural features can also be used (Appendix B). For tabular data, we used the final hidden layer of the neural network trained for classification.
In general, we also use t-SNE or PCA to reduce the dimension the feature space. For example, the 2048-I) Inception-v3 em- bedding still very high-dimensional. We
pcaño
identity
report results using t-SNE to reduce the di-
02
mension to 2 or 3, as well as PCA reduce the dimension to reduce the dimension 50 for image data and 20 for tabular data. Thus the overall representation function is
MLCE
0.6
y 0.4
0.4
<(x) = ( (z:)), where Φ1 maps from the inputs to the neural features, and Φ2
0.2
reduces the feature space dimension.
10-3 10-2
10-1
10" 101
10²
Figure plots the MLCE function of the kernel bandwidth for an ImageNet clas- sification task. Note that when Y small, the MLCE is (a worst-case individual cal- ibration error), and when y large, the ML.E approaches the global MCF. To ob- tain a single summary statistic describing the local calibration error, plot and pick a value of y between the lim- iting behaviors. We find that 0.2 and y 0.4 good intermediate points for the 3-D t-SNE and 50-D PCA features, respectively (Figure 1).
Kernel Bandwidth y
Figure 1: MLCE of Resnet-50 classifier the ImageNet test split, as a function Laplacian kernel with feature map Φu o Φ1. where d1
kernel bandwidth 7. We use
X' + 12 is the Inception-v3 model's hidden layer, Blue: 22 + I is t-SNE; orange: 62 122048 R the identity.
PCA; green: (2)
view this
3.3 LOCAL CALIBRATION ERROR VISUALIZATIONS
CDF of LCE(X) In Bin 5/15 (Best Global Calibration)
LCE(X 5/15 (Best Collbration)
0.2
0.8
Prabaa 1iee
0.6
CDF LCE0I
50
Callbneiin Error
40
0.02
0.04
0.06 0.08
0.30 0.12
LGEIX
CDF of LCE|X] in Bin 12/15 (Worst Global Callbrotion)
Calibration)
012
LCE
00
0.4.
-g of 20f600
CDP of LCBis Bn Calbrotton Emor
-0 10
0.0
0.02
0.04
0.06
0.8
0.10 0.12
Libid
visualize L.F. f.p) for a ResNet-50 classifier (f. pre-trained on ImageNet, for every image x in the ImageNet validation set. We focus on the hins with the best and worst global calibration emrors.
Figure 2:
To provide more intuition for the LCE, we will now visualize some examples the LCE metric over 2-D feature embedding. We consider ResNet-50 model pre-trained on ImageNet classifier
5

---

Under review as a conference paper at ICLR 2022
(f, and pre-trained Inception-v3 features feature map Φ1 X - IR 2048. Φ2 R - R then reduces the 2048-D feature vectors with t-SNE to two dimensions for case of visualization the LCE landscapes, landscape of o. (x; f. as a function of ((x) for the entire ImageNet validation set, as well as the marginal CDF of 2 f.t). We show these visualizations for the two confidence bins with the best and worst global calibration.
overall representation function is ( (z)). Figure 2 visualizes the
In the bin with the best global calibration, Figure (top) shows that the landscape of the LCE is highly non-uniform, and the CDF of the LCE lies almost entirely to the right of the bin's average calibration error. Numerically, the bin's average calibration error is 0.0022, while its average LCE is 0.0259. This implies that the regions where the model is underconfident and overconfident are spatially clustered within the bin, Because global calibration metries solely consider the average accuracy and average confidence within bin, confidence predictions that are too high and too low are averaged out obtain a low overall error value; they fail to capture this localized miscalibration.
In the bin with the worst global calibration, Figure 2 (bottom) clearly shows that the LCE still has high variance, even though the average calibration error of the bin (0.0455) much closer to its average LCE (0.0515). The CDF plot provides more evidence that the landscape is not flat there no sharp rise at the bin calibration error. However, in this case the regions that are underconfident and overconfident are not clustered spatially.
4 LCE RECALIBRATION
In this section, introduce local recalibration (LoRe), non-parametric recalibration method that adjusts a model's s output confidences to achieve better local calibration. Our method improves the LCE more than existing recalibration methods, and using our method improves performance on both downstream fairness tasks and downstream decision-making tasks. Specifically, we can leverage the kernel similarity achieve strong calibration for all sensitive subgroups of a population, without knowing those groups a priori. As long as the feature space is semantically meaningful, LoRe provides utility for downstream tasks without needing subgroup labels for the samples, the subgroups are known, one can recover standard group-wisc recalibration methods (and metrics) by using the improper kernel (2,xx) 1 [zz x! in same group].
The idea behind our method is simple: we can compute the kernel-weighte accuracy for each point of all the points that are in the same confidence bin as z, and then reset the confidence of æ to this keighl-eeighed accuracy value. Note that using the kernel function to compute this value is intuitively like taking a weighted average the accuracy the points in the local neighborhood of x. Thus, LoRe can be considered a local analogue to histogram binning.
More formally, given a trained classifier (f.p). recalibration dataset D ((x1, vn).... (EN, UN )), and fixed point E X. let ((x) = {i p(x,) E B(b(x))} be the set of indices of the points in D occupying the same confidence bin as x. Then, we compute the recalibrated confidence as
# (x) = Eier(ay a, vil
(6)
EiEr(E) (x,
Equation 6 represents the kerghl-weighted average accuracy of all points in the same confidence bin as x. In the limit as the kernel bandwidth + co, Ø (x) + Eies(.) [[f(xx) = vr]/1((z) recovers histogram binning. As 7 (zz + 1[f(x.)) ys-], where i' arg minkee k, (z, 2z), thus recovering a nearest-neighhor method. For intermediate 7, our method interpolates between the two extremes. Throughout this work, used used 7 0.2 for LoRe with tSNE and y 0.4 for LoRe with PCA throughout this work, since these represent intermediate points between the limiting behaviors of the LCE (e.g., see Fig. 1).
EXPERIMENTS
In this section, show empirically that LoRe substantially improves LCE values, and that these lower LCE values lead to better performance on downstream fairness and decision-making tasks. In particular, evaluate the local calibration through the MI.E, because we are interested in

---

Under review as a conference paper at ICLR 2022
1500
Original
15
Improvement in Reward
1000
HB
500
MLCE
Lafte
Original
-500
HB
1000
MMCE
-1500
LoRc
3.0
16-1 Kernel Bandwidth Y
8
12
Reward Ratio
Figure 3: ML.EE vs. kernel bandwidth y for ImageNet. LoRe (with t-SNE and 0.2) achieves the lowest MLCE for wide range 7. This suggests that LoRe leads to lower LCE values across the whole dataset.
Figure 4: Reward attained vs. reward ratio for the ImageNet dataset (higher is better). LoRe achieves the highest rewards across wide range of reward ratios.
understanding a model's worst-case local miscalibration. On each task, we compare the performance of LoRe no recalibration (' ('Original'), temperature scaling ("TS") (Guo et al., 2017), histogram binning ('HB') (Zadrozny Elkan, 2001), isotonic regression ('TR') (Zadrozny & Elkan, 2002), and direct MMCE optimization ("MMCE') (Kumar et al., 2018), all strong global recalibration methods.
We first run extensive experiments on three datasets to demonstrate that LoRe outperforms all baselines and achieves the lowest MLCE over wide range of 7 values. We then evaluate the performance of our method on a fairness task, where it important that a model is well-calibrated for all sensitive subgroups of given population, and we demonstrate that it achieves the lowest group-wise MCE. Notably, we find that the MLCE is well-correlated with the group-wise MCE across all experimental settings, and thus achieving low MLCE is good indicator that model has good pooup-wise calibration. Finally, we compare the performance of our method against the baselines on cost-sensitive decision-makee task, where there low cost for a prediction of "unsure" but high cost for an incorrect prediction, and show that our method achieves the lowest cost.
5.1 DATASETS
ImageNet dataset (Deng et al., 2009): A large-scale dataset of natural scene images with 1000 classes; over 1.3 million images total. The training/validatioonness split is 1.3mil /ttp//255000020000
UCI Communities and Crime dataset (Dua & Graff, 2017): This tabular dataset contains number of attributes about American neighborhoods (e.g., race, age. employment, housing, etc.). The task is predict the neighborhood's violent crime rate. The traiiingvvlliaationtess split is 1494/500/500 We randomize the snining/validation/ien split over multiple trials.
CelebA dataset (Liu et al., 2015): A large-scale dataset of face images with 40 attribute anno- tations (e.g., glasses, hair color, etc.); 202,599 images total. The training/validatioonnest split is 162,770/ 19,867 19,962.
5.2 RECALIBRATION PERFORMANCE
LoRe substantially improves the LCE values. In Figure 3, we plot the MLCE as a function of We can see that our method outperforms all baselines (strong global calibration methods) across wide range of values on ImageNet. This is true despite the fact that we only implement LoRe for single y Appendix provides similar results on the Communities Crime, CelebA, CIFAR-10, and CIFAR-100 datasets. Note that LoRe works well regardless of the feature map and the dimensionality reduction method (see Section 5.3 for results with both t-SNE and PCA). Although the results shown in this section Inception-v3 features, (Krizhevsky, 2014), 12 (Huang et al., 2018), and ResNet101 (He et al., 2015) features.
show similar results in Appendix B with AlexNet
Recall that y gets large, the MLCE recovers the MCE; because LoRe does well even at large 7 our method also works well at minimizing global calibration errors. The fact that LoRe lowers the worst-case I.E suggests that it leads to lower LCE values across the entire dataset.
7

---

Under review as a conference paper at ICLR 2022
5.3 DOWNSTREAM FAIRNESS PERFORMANCE
Experimental Setup In many fairnes--related applications, it is important to show that model well-calibrated for all sensitive subgroups of given population. For example, when predicting the crime rate of a neighborhood, a model should not be considered well-calibrated if it consistently underestimates the crime rate for neighborhoods of one demographic, while overestimating the crime rate for neighborhoods of a different demographic. Therefore, in this section, we examine the worst-case group-wise miscalibration of a classifier, as measured by the maximum group-wise MCE when evaluated only on sensitive sub-groups. We consider the following experimental settings:
1. UCI Communities and Crime: Predict whether a neighborhood's ' crime rate is higher than the median; group neighborhoods by their plurality race (White, Black, Asian, Indian, Hispanic). 60 random seeds for model training.
2. CelebA: Predict a person's hair color (bald, black, blond, brown, gray, other); group people by hair type (bald, receding hairline, bangs, straight, wavy, other). 20 random seeds for model training.
3. CelebA: Predict person's 's hair type; group people by their hair color; inverse of Setting 2. 20 random seeds for model training.
For each task, we train a classifier (see Appendix A for full details) and recalibrate its output confidences using each of the recalibration methods.
Results Table 1 reports the maximum group-wise MCE for each of the recalibration methods on each of the three tasks. LoRe outperforms the other baselines, achieving an average 49% reduc- tion over no recalibration and an average 23% improvement over the next best global recalibration method. (Note in Figures 5, 6, and 7 in Appendix B that LoRe the most effective method of lowering the MLCE over wide range of 7). Notably, LoRe is robust to the feature map used (tSNE vs. PCA). even outper- forms global methods applied to each individual group, implying that correcting local calibration errors is robust way to improve group calibration that generalizes better than naive alternatives.
Recalibration method
Setting
Setting 2
Setting 3
No recalibration Temperature scaling
0.588 + 0.107
0.407 0.087
0.446 0.083
0.521 0.092
0.532 0.089
0.441 1 0.079
Histogram binning
0.515 a: 0.081
0.218 & 0.056
0.268 a: 0.067
Isotonic regression
0.596 + 0.063
0.615 # 0. 1000
0.716 & 0.082
MMCE optimization
0.526 + 0.172
0.429 0.079
0.475 + 0.079
Group temp. scaling Group hist. binning
0.423 # 0.066
0.673 0.075
0.329 + 0.108
0.542 0.083
0.260 0.053
0.352 +:2 0.068
LoRe ((SNE) (ours)
0.351 0.084
0.165 0.055
0.235 # 0.063
LoRe (PCA) (ours)
0.392 0.071
0.167 0.013
0.154 0.082
Table 1: Performance on downstream fairesss, as measured by maximum group-wise MCE (lower is better). Experimental settings as described in Section 5.3. Mcan and standard deviations are computed over 60 random seeds for setting 1, and 20 for settings and 3. Best results are bold.
shows that the maximum group-wise MCE is well-correlated with the MLCE, and it in fact much better correlated with MLCE than with global calibration metrics Taken together, our results indicate that lowering the LCE has positive implications in faimesss set- tings that cannot be achieved by simply lowering global metrics like the ECE. For reference, we also include the performance of all recalibration methods on various global calibration metrics in Table 3, which shows that LoRe is able to improve worst-case group-wise calibration with- out meaningfully sacrificing (and in some cases improving) average-case global calibration.
Moreover, Table 2
Setting
Setting
Setting 3
ECR
0.102
-0.061
-0.195
MCE
0,233
0.439
0.281
NLL
0.542
0.045
-0.287
Brier
0.101
0.144
-0,280
MLCE (ISNE)
0.642
0.801
0.591
MLCEa.4 (PCA)
0.639
0.659
0.778
Table 2: Pearson correlation between max group- wise MCE and other calibration metries (higher is better). Experimental settings as described in Section 5.3. Best results bold. MLCE is better-correlated with the max group-wise MCE than any of the global metrics.
5.4 DOWNSTREAM DECISION-MAKING
Machine learning predictions are often used to make decisions, and in many situations agent must select best action in expectation. As example, suppose there is low
Experimental Setup
8

---

Under review as a conference paper at ICLR 2022
Recalibration method
Setting NLL
Setting
Setting
ECE(S)
Brier
BCE(%)
NLL
Brier
ECE(%)
NLL
Brier
No recalibration
15.12.7
29
.17.02 .22
1.80.8
.617.004
.641.004
1.1o.3
.782.006
.571,006
Temperature scaling
4.91.7
.43o33
.14.01
2.0p.3
.69..04
.22. 003
1.0p.2
.781 006
.509.002
Histogram binning
3.31.1
.48.03
.150.1
2.50.2
.619.004
.614.00 .03
2.50.4
.788 006
.552.
Isotonic regression
30.62.3
.9..s
.30
2.60.2
.6180004
.65,,ox ous
2.40.2
.785
.553,
MMCE optimization
4.41.3
43,03
.14.01
3.8u.7
.646,114
.679,012
5.40.8
.808.09
.619,009
LoRe (tSNE) (ours)
3.51.1
42,02
.13,01
2.80.2
.623,004
.613,003
2.60.3
.792 006
.551,002
LoRe (PCA) (ours)
4.51.4
.44,02
.14.01
3.10.2
.628,001
.606 003
2.80.4
.792.007
.538 002
Table 3: Performance on global calibration metrics, formatted as mean Lower is better. Experimental settings as described in Section 5.3. Best results are bold. Across all settings, LoRe generally achieves global calibration error that is comparable to the baselines.
cost u associated with returning "unsure" and a high cost w associated with returning an incorrect classification (e.g., in situations such autonomous driving, being unsure incurs only the small cost of calling a human operator, but making an incorrect classification incurs a high cost). An agent with good uncertainty quantification classification or return "unsure"; for a calibrated model, it would be optimal for the agent to return "unsure" below the confidence threshold of u/ w. and return prediction above this threshold.
make more optimal decision about whether to return
Following this policy - i.e., returning "unsure" when the confidence is below this threshold and returning& prediction when the confidence above we used a ResNet-50 model to make predictions on ImageNet, and recalibrated the predictions with each of the recalibration methods. For each method, we then calculated the total reward attained under various reward ratios w/ w/u, as well as the prediction rejection area ratio (PRR) (Malinin et al., 2020), which measures the quality of the rejection curve when a model can choose to not provide any prediction. Note that a PRR of 1.0 indicates optimal rejection, and PRR of 0.0 indicates "random" rejection; higher PRR is better.
Results In Figure 4, we show the improvement in the total reward over the original classifier (i.e., no recali- bration) as function of the reward ratio w/u /u (the ratio of the cost of an incorrect classification to the cost of being unsure). Across a wide range of reward ratios, LoRe achieves the high- est reward. The MLCE curves for this task are shown in Figure 3; note that LoRe also achieves lower LCE values than the global recalibration methods. Table 4 also reports the PRR achieved with each method under cost function of u 1, w 100, well several global calibration metrics; LoRe generally outperforms the other baselines in terms of both PRR and global calibration metrics. These results indicate that our recalibration method most effectively lowers LCE values without sacrificing (and indeed often improving) average-car global calibration, and that these lower LCE values correspond better performance on this decision-making task.
Recalibration method
PRR
ECE
NLL
Brier
No recalibration
0.566
0.037
0.959
40.64
Temperature scaling
0.561
0.022
0.948
40.60
Histogram binning
0.461
0.012
0.952
40.59
Isotonic regression
0.570
0.011
0.945
40.59
MMCE optimization
0.571
0.061
0.965
40.67
LoRe (ours)
0.575
0.007
0.955
40.58
Table 4: Performance downstream decision-making
Ima-
geNet, as measured by the PRR (higher is better). We also report the global calibration metrics ECE, NLL, and Brier score (lower is better). Best results in bold.
6 CONCLUSION
In this paper, we introduce the local calibration error (LCE), metric that measures calibration in a localized neighborhood around a prediction. The LCE spans the gap between fully global and fully individualized calibration error, with an effective neighborhood size that can be set with bandwidth parameter y. We also introduce LoRe, a recalibration method that greatly improves the local calibration. Finally, demonstrate that achieving lower LCE values leads to better performance on downstream faimess and eecision-making tasks. In future work, we hope to explore better feature spaces to define similarity, since the quality of our metric depends the quality of the feature space.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL http: httt:://proceeeiiee mlr. wtt.ppssssv119/eeeeeee
html.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive.. ics.uci. www.uuu/i..uee..eom edu/ml.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS 12. pp. 214-226, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450311151. doi: 10.1 00.11452000233.22000355
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:// http:///coceeeennsseeee mlr.. /tsess//0//gnt 7a htt.//.lll..l........
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2015.
Úrsula Hobert-Johmson, Michael P. Kim, Omer Reingold, and Guy N. Rothblum. Calibration for the (computationallly -idettiiiiii)) masses, 2017. URL http: htt//aixvvvv//// wwwp:ii.ogg/////////aaee abs////ll11 0117111.111...
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. 2018.
Minyoung Huh, Pulkit Agrawal, and Alexci A. Efros. What makes imagenet good for transfer learning?, 2016. URL http:: org/ abs/1608 08614.
Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores, 2016. httpht/tp//aiiii org. abs///b////// 11696.66.00 05807.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. CoRR, abs/1404, hbss/4444.999, 2014. URL /tttt:://ariii wwtp/bs//////////// 5997.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. pp. 12316-12326. Cur- ran Associates, Inc., 2019. URL https: http:////oooeeeeeeeeeeeeoee neuri ps. cc waappr//0100 file/ aale//ccalea@00 01 011122244444444999999 pdf.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2805-2814 Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http:/ / http://pocceeiineeee mlrr press v80 kumar a. html.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119-9130. Association for Computational Linguistics, November 2020. doi: 10.
URL
https: // wwww///wwww/////////////// acl web. wew.boorg/aee wwtpooorg//ooolgggog.ooooo ogy 2200 emn p- i n. 733.
10

---

Under review as a conference paper at ICLR 2022
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
A. Malinin, Bruno Mlodozeniec, and M. Gales. Ensemble distribution distillation. ArXiv, abs/1905.000777 2020.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, pp. 2901-2907, 2015.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration deep learning. ArXiv, abs/1904.01185,, 2019.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pp. 61-74. MIT Press, 1999.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neurul Information Prucessing Systems, volume 30, pp. 5680- 5689. Curran Associates, Inc., 2017. URL httpthtttt:::ceeeeeeeeee http///oooeeedeeeeeeeoeeoo neurips.cccppperr. 2017/file
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.
Elizabeth A. Stuart. Matching methods for causal inference: A review and a look forward. Statistical science : a review journal of the Institute of Mathematical Statistics, 25(1):1-21, 2010. doi: N0.121400-STSS13
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran As- sociates, Inc., 2019. URL https: p/coceedigg// neurips wttppppee//1/////eeeee 2019//iiee///// file/ 1c33 6233668888888 223350000F2222222222996662222222
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In In Proceedings of the Eighteenth International Conference on Machine Learning, pp. 609-616. Morgan Kaufmann, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In SIGKDD, 2002.
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. 2020.
Shengjia Zhao, Tengyu Ma, and S. Ermon. Individual calibration with randomized forecasting. In ICML, 2020.
Shengjia Zhao, Michael P. Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration, 2021.

---

Under review as a conference paper at ICLR 2022
A MODEL ARCHITECTURE. TRAINING, AND OTHER HYPERPARAMETERS
For ImageNet and CclebA, we compute the ECE, MCE, and LCE using 15 equal-width confidence bins. For the UCT communities and crime dataset, use equal-width bins because the dataset is much smaller (500 datapoints for recalibration). These numbers of bins represent a good tradeoff between bias and variance in estimating the relevant calibration errors. We also ran some initial experiments with equal-mass binning, but found that the results were very similar to those obtained with equal- width binning.
A.1 IMAGENET
For all experiments with the ImageNet dataset, we used the pre-trained ResNet-50 model from the PyTorch torchvis sion package as our classifier. To calculate the LCE and apply LoRe, we used pre-trained Inception-v3 features, applying either t-SNE to reduce their dimension to 3 or PCA to reduce their dimension to 50, as a feature representation for the kernel.
A.2 UCI COMMUNITIES AND CRIME
For all experiments with the UCI communities and crime dataset, we used 3-hidden-layer dense neural network as our base classifier. Each hidden layer had a width of 100 and was followed by a Leaky ReLU activation. We applied dropout with probability 0.4 after the final hidden layer. We trained the model using the Adam optimizer with batch size of 64 and a learning rate of 3 x 10-4 until the validation accuracy stopped improving. All other hyperparameters were PyTorch defaults. Training was done locally on a laptop CPU. We trained 60 different models with different random seeds perform the experiments described in Section 5.3 and Figure 5. To calculate the LCE and apply LoRe, we used the final hidden layer representation learned by our model, applying t-SNE to reduce the dimension kernel.
PCA to reduce their dimension to 20, as a feature representation for the
A.3 CELEBA
For all experiments with the CelebA dataset, we trained a ResNet50 model and used it as our base classifier. We applied standard data augmentation to our training data (random crops & random horizontal flips), and trained all models for 10 epochs using the Adam optimizer with a learning rate of 1 10-3 and batch size of 256. All other hyperparameters were PyTorch defaults. Training was distributed over GPUs, and training a single model took about 30 minutes. For both Setting 2 and Setting 3 (described in Section 5.3), we trained 20 models with different random seeds to perform the experiments shown in Figures and 7. To calculate the LCE and apply LoRe, we used pre-trained Inception-v3 v3 features, applying t-SNE reduce their dimension to 2 or PCA to reduce their dimension to 50, as a feature representation for the kernel.
B ADDITIONAL EXPERIMENTAL RESULTS
In Figures 5, 6, and 7, we visualize the MLCE achieved by all recalibration methods for the three experimental settings evaluated in Section 5.3. Figure 3 in the main paper shows the same visualization for all methods on ImageNet. In Figure 8, we plot the MLCE achieved by all recalibration methods for CIFAR- 100, and in Figure method LoRe is the most effective at minimizing MLCE across a wide range of y, even accounting for variations between runs.
do the same for CIFAR-10. Across all settings and datasets,
12

---

Under review as a conference paper at ICLR 2022
Original
Original
15
HB
IR
MLCE
0.6
Lafe
MLCE
0.6
Laft
0.-
0.4
10-1
Kernel Bandwidth
Kernel Bandwidth Y
Figure 5: MICE vs. kernel handwidth y for all methods on task of Section 5.3, predicting whether a neighhorhood's crime rate is higher than the median. LoRe achieves the best (or competitive) MI.CE for most 7y Left: 2D t-SNE features. Right: 20D PCA features.
Original
Original
HB
MNCE
MLCE
0.6
LoRe
LoRe
0.4
0.2
0.0
0.0
10
10"
107
10
30"
10' Kernel Bandwidth
Kernel Bandwidth
Figure 6: MLCE vs. kernel bandwidth y for all methods on task 2 of Section 5.3, predicting hair color on CelebA. LoRe achieves the best MLCE for virtually all values of Left: 2D t-SNE features. Right: 50D PCA features.
Original
Original
T5
HB
IR
MUCE
Lakte
MLCE
Lofe
0.2
0.0
10-2
Kernel Bandwidth
Kernel Bandwidth
Figure 7: MLCE vs. kernel bandwidth for all methods on task of Section 5.3, predicting hair type on CelebA. LoRe achieves the best MLCE for all and is tied with histogram binning for y >> 1. Left: 2D t-SNE features. Right 50D PCA features.
In these figures, "Original" represents no recalibration, "TS" represents temperature scaling, "HB" represents histogram binning, "IR" represents isotonic regression, "MMCE" represents direct MMCE optimization, and "LoRe" our method.
Next, we examine the influence of the specific feature map used. In Figures 10, 11, 12, and 13, we plot the MLCE achieved by all recalibration methods for ImageNet using Inception-v3, AlexNet, DenseNet121, and ResNet101 features. In Figures 14 and 15, we plot the MLCE achieved by all recalibration methods for ImageNet when the features used to calculate the MLCE are different from the features used by LoRe. For completeness, in Figures 16, 17, 18, and 19, we also visualize the average LCE for all experimental settings. All plots show similar results: LoRe performs best over a wide range of 7
13

---

Under review as a conference paper at ICLR 2022
1.
10
Original
T5
0.8
NMCE
MLCE
LoRe
0.6
HB
IR
0.2
0.2
Loke
10
10:4
10e
10
10
10-1
10*
10
Kemel Bandwidth
Kerne Bandwidth
Figure 8: MLCE vs. kernel bandwidth for all recali- bration methods CIFAR-I00 (3D I-SNE features). LoRe achieves lower MLCE for most 7.
Figure 9: MLCE kernel bandwidth for all recali- bration methods CIFAR-I0 (3D t-SNE features). LoRe achieves lower MLCE for most
Original
Original
HB
HD
MLGE
LoRe
MLCE
LoRe
0.4
0.
0.0
Kernel Bandwidth
Kerel Bandwidth
Figure 10: MLCE vs. kernel bandwidth for all re-
Figure 11: MLCE vs. kernel bandwidth for all recal- ihration methods on ImageNet using AlexNet features. LoRe achieves the best MI.CE for most
calibration methods ImageNet using Inception-v3 features. LoRe achieves the hest ML.CE for most y.
Original
1.0
Original
15
TS
0.8
IR
MMCE
MMCE
MLCE
0.8
MLCE
0.6
0.4
0.2
0.0
10"
20"4
10"1
10
10
10"7
10"1
19"
101
107
Kemee Bandwidth Y
Kemel Bandwidth
Figure 12: MLCE vs. kerne bandwidth y for re- calibration methods on ImageNet using DenseNet121 features. LoRe achieves the best MLCE for most ?y.
Figure MLCE vs. kernel bandwidth 7 re- calibration methods on ImageNet using ResNet101 features. LoRe achieves the best MLCE for most
14

---

Under review as a conference paper at ICLR 2022
Original
1.0
Original
T5
TS
0.0
10
0.0
ID
MMCE
NMCE
MLCE
Loke
MLCE
LoRe
0.0
0.0
10
10
10c Kernel Bandwidth
10
10a
10
10
10-1
10t
10
Kernel Bandwidth
Figure 14: MLCE kernel bandwidth for all calibration methods ImageNet using Inception-v? features calculate the MI.CE AlexNet features for applying LoRe.
Figure 15: MLCE vs. kernel bandwidth for all re- calibration methods on ImageNet using DenseNet121 features to calculate MI.CE and AlexNet features for applying LoRe.
Original
Original
0.4
TS
020
ELCE
0.1
MMCE
LoRe
0.1
3.00
10:7
10
30
10
10
Kernel Bandwidth
Kernel Bandwidth
Figure 17: Average LCE vs. kernel bandwidth for all recalibration methods in task (crime data, 2D t-SNE features). LoRe gets lower average LCE for most 7.
Figure 16: Average LCE vs. kemel bandwidth y for all recalibration methods on ImageNet (3D t-SNE features). LoRe gets lower average LCE for most
0.35
Original
0.40
Original
15
0.35
IR
0.20
IR
MMCE
MMCE
ELCE
0.20
0.5
0.20
0.10
0.05
8.55
10
10-7
10
30
10
10
10-7
10
10'
167
Kernel Bandwidth y
Kernel Bandwidth y
Figure 18: Average LCE vs. kemel bandwidth ry all recalibration methods in task 2 (CelehA, 2D i-SNE features). LoRe gets lower average I.CE for most
Figure 19: Average LCE vs. kernel bandwidth y for all recalibration methods in task 3 (CelehA, 2D i-SNE features). LoRe gets lower average LCE for most 7
15

---

Under review as a conference paper at ICLR 2022
C PROOF OF LEMMA
We restate Lemma below, and provide the proof:
Lemma 2. Assume that lim Kn (2, x') 1 for all x, x' E N. Then, as y + sc, the MLCE converges to the MCE.
Proof. Since lim, ki (2, x') = identically,
lim max LCE, S.D) 1-00
m18x
13(x))
1
Σ ((x;) - 1 (f(x)) mil
2
iEA(()
1
Σ p(x;) - 1 [f())) val
max
&
B&
iEBn
max |coul (Bx) - aco B&)
&
MCE(2;
D FORMAL STATEMENT AND PROOF OF THEOREM 1
Let Bı.... By denote a set of bins that partition [o, 11, and (p)) denote the bin that a particular
p E [0, 1] belongs to. Let a y) = (xx y] indicate the accuracy the classifier (J, p) on an input x. We consider the signed local calibration error (SLCE):
SLCE, = [ a (X, )),,, .r) P(X) E B(p(z))
Ei (X, x)) ((X) E B((()))
[E(((X)= a (X, (X,, (Ø(X) B((()))I
Elki, (x) I (x() E B()(x))]
D.1 ASSUMPTIONS AND FORMAL STATEMENT OF THEOREM
We make the following assumptions:
Assumption A (Lipschitz kernel), The kernel ky takes the form
ky( = ( ((xx)) M(x)) -(4')
where Φ X + Red a representation function, and TRd + [0, 1] is T.-Lipschitz with respect to some norm ||-1]-
Note this definition may require an implicit rescaling (for example, we can take Φ(x) + featuuee (I) / for a d-dimensional feature map of feetuur and take g(2) = exp(-11=111), which corresponds to the Laplacian kernel we used in Section 3.2).
Assumption B (Binning-awer aware covering number). For any e > O, the range of the representation X33 has c-cover in the |1-I|-norm of size (C/c)a for some absolute constant C > 0: There exists a set N, E X with [W.] < (C/c)² such that for any x E X, there exists some d' N, such that []ő(x) d(x')ll || and B(5(x)) B(ğ(z')).
function of ((X)=
(Ø(x)=
Assumption C (Lower bound on expectation of kernel within bin). We have
inf xEX
(X, (XX) B(i(z))I
for some constant a E 10.
The constant characterizes the hardness of estimating the SLCE from samples. Intuitively, with smaller a, the denominator in SLCE gets smaller and we desire a higher accuracy in estimating both the numerator and the denominator. Also note that in practice the value of typically depends y-
16

---

Under review as a conference paper at ICLR 2022
We analyze the following estimator of the SLCE using samples: (T..F., 1 (==1((((;)) a (Tz, 2/ (x,, [p(x;) e B(((z)))
(7)
- ky (2i, x) p(x))) E B(p(z))
Theorem 2. Under Assumptions A, B, and C. Suppose the sample size n 2 Ō(d/ate) where e is target accuracy level, then with probability at least 1 we have
(1; f. SICE, S.P) S,
sup TEX
where O hides log factors of the form log L/ycda).
Theorem 2 shows that a samples is sufficient estimate the SLCE simultaneously for all E X. When a = Q(1), this sample complexity only depends polynomially in terms of representation dimension d and logarithmically in other constants (such as L, and the failure probability 8).
D.2 PROOF OF THEOREM 2
Step 1. We first study the estimation finitely many r's. Let C X be finite set of x's with W = N. Since ky E [0, 1] and (p(x) - ay(2 (2, y)1 I 1 are bounded variables, by the Hoeffding inequality and union bound, have
Σ P sup xEN
(p(()) e B(p(z))
[((( - (X ((X) B(((z))] > ac/11)
S <exp(-cna's log N).
Therefore, long 2 OllogNN/8)/e@uu samples, the above probability is bounded by 6. In other words, with probability at least 8, we have simultancously
N/8)/e2a²)
Σ (p(x;) a (1 (: 1 (p(x)) B(((a)))
72
2mm A(z)
-1 e B()(=)))
:=A(w)
S Sae/10.
for all x E N. Similarly, when n 2 O(log(N/6)/22 we also have (with probability at least 1- 8)
ky (I, x) 1 (px;)) e -B( (X, (xx) e B((())]]]
sate/10
B(x)
BB))
On these concentration events, we have for any æ E N that
A(x) A(x)
SLCE,, (4; (c.C..
B(x)
B(x)
A(x))
1
1
A(x)
A(x))
B(x))
B(x)
B(x)
a²e/10
1
aee10
a(a - a²e/10)
α
Sc.
17

---

Under review as a conference paper at ICLR 2022
We now extend the bound to all X' using the covering argument. By Assumption B, we can take an a² (10//1100 denote the covering set (in the X' space). This means that for any x E X, there exists x' e such that [|x(x) - <(<')l I (107/100)) amd B(î(x)) = B(p(z'), which implies that for any N E X have
Step 2.
of 6(2) with cardinality N S (10CL/afe)) Net C X
- - x(,,x')1 1 [()) 0(x)
( 6((x)
1
[1(x)) o(x'))l
7
e/1110
where we have used the aischitt asssastiomnsnnooos of g (Assumption A). This further implies
1 n
A(x) - (())-A(r))--
Σ -
[fx;)) e B(f(z)))
Σ (p(x)))
(ax,)) B(((2)))
(p(x:) a ( [dtr)
(pxx;) e B(((z))]
1p(z;) - af(2,. vi)l -
ky (2,, -,, (px;) E B(ş(z)))
11
$ <a²e/10.
Similarly, we have |A(x) - A(x') S a²e/10, B(x) B(x')l S a²e/10, and [B(x) B(x') s a²e/10. This means that the estimation error is close to that at x' N and consequently also bounded by e:
A(x)
A(x)
SLC f.D) SLCE, (1; f.0)
B(x)
B(x)
A(x)
A(x')
A(x')
A(x')
A(x')
A(x)
+
B(x)
B(x')
B(x²)
B(x²)
B(x')
B(x)
a²c/10
< 3
+
a²e/10
a(a- - a2e/10)
a(a
Se
Therefore, taking this A in step know that long the sample size
N O 1og((W1/0) =o
[[loo(10CLL//20 log(1 [1/8]] =((d/at))
Paa
ate²
we have with probability at least 1 5 that
sup
(x: - (T: Sc
xEX
This is the desired result.
18


Reviews:
[{"review_id": "fevOVRSNN6F", "reviewer": "Reviewer_76PS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper is generally well written and clear. It is easy to follow and the experiments support the argument made in this paper.\n- The notion of local calibration seems novel to me. Although I have not scoured papers in this area, conditioning calibration error not just on confidence, but also feature similarity in the classification setting seems new to me.\n\nWeaknesses:\n- While the experimental results look positive, I believe they are not surprising because, as far as I can tell, LoRe uses the same feature map and kernel as the core evaluation metrics (LCE, MLCE). What happens empirically when different feature maps and kernels are used for applying LoRe and evaluating with LCE, MLCE?\n- Further, it seems as though computing the proposed metric (LCE, MLCE) depends on 1) a specific feature map, 2) a specific method for further reducing dimensions, 3) a specific kernel function, and 4) specific settings of the hyperparameters for the kernel. It is hard to be convinced that this is a robust metric when calculation depends on so many choices, i.e. I can imagine different rankings for the methods if the evaluation metric was computed with different choices of the above. \n- Continuing from the above point, is there a reason why the authors chose a Laplacian kernel? Have the authors experimented with other kernels? How about different feature maps?\n- Choosing a suitable $\\gamma$ seems quite arbitrary. Last line of Section 3.2, the authors simply state, \"0.2 is a good intermediate point\" without any justification. In practice, how should $\\gamma$ be chosen?\n- How much does the LCE metric diverge from classwise-ECE? If I understand correctly, LCE is meant to measure ECE in localities over the input feature space. Classwise-ECE, in a certain way, provides a proxy of this, because ECE is measured for each true class label, and inputs features should be well clustered within each class label (assuming meaningful features). In fact, if the features were separable by class, I believe LCE should be identical to classwise-ECE with an appropriate kernel and hyperparameter.\n- Continuing from the point above, I believe it would have been more meaningful to 1) compare against methods that target classwise-ECE (i.e. perform recalibration by class), and 2) to additionally report classwise-ECE values in the experiments.\n- Is LoRe an efficient algorithm during test-time? It seems as though Equation 6 needs to be computed for each test point x, where x_i indexes all points in the recalibration dataset $\\mathcal{D}$. This would be in contrast with other post-hoc methods which, e.g. learn a lookup table or tune a scalar during the recalibration phase, and can predict during test-time with a single forward pass of the classifier.\n\nMinor comments:\n- What is $d$ in the denominator of the Laplacian kernel definition (Section 3.2)?\n- In Figure 2, is the vertical dashed line simply ECE? It is referred to as \"average calibration error\" in text in Section 3.3.\n- Are the experiments with ImageNet run with just 1 seed?", "summary_of_the_paper": "This work proposes a new metric for calibration in classification where calibration is measured over localities in the input space, and the localities are determined with a kernel over the feature space. A recalibration algorithm (LoRe) is additionally proposed, which aims to recalibrate the class predictions w.r.t. local calibration error (LCE). The experiments show that 1) LoRe achieves better local calibration (measured by MLCE, i.e. maximum LCE) than baseline recalibration methods on ImageNet, 2) LoRe achieves better group-wise calibration (measured by worst group-wise maximum calibration error) across 3 datasets with identifiable groups, and 3) achieves competitive performance in a simple decision making task where confident incorrect predictions incur high costs. ", "summary_of_the_review": "While the concepts introduced and methods proposed are interesting, I do not believe the proposed metric is necessarily robust, and am not convinced of its advantages over an established metric (classwise-ECE, which also has not been reported in this work).\nI believe elaborating (theoretically or experimentally) on these points could make a much stronger case for this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tud0wxVutgr", "reviewer": "Reviewer_hPxn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- local calibration is a sensible idea and it's good to see it worked out like this\n- local/individualized notions of calibration are an important problem\n- same thing for the proposed method: it's simple and seems like the thing you would try in this situation\n- the experiments show fairly consistently that this method provides gains in improving local calibration metrics\n\nWeaknesses:\n\n- it's not clear to me why we are filtering for prediction bin as well as weighting along input space - my intuition around what \"local\" calibration should mean would involve reliability among all similar inputs, rather than just those that also have similar outputs. The analogy to global calibration metrics is not immediately obvious to me and merits more discussion\n- the choices around representations which are operated on seem under-discussed. For instance, my guess from reading this paper is that t-SNE is quite important to the practical results of this method - would be good to hear if that's true or not, or how other approaches compare results-wise. If t-SNE is essential to the application of the method, it indicate the method may not be that robust\n- Additionally, I'm curious about the application of t-SNE here - how is a test point placed in a t-SNE representation space which was fit on a validation set? As far as I know, t-SNE representations are usually not amortized\n- 2-d pictures in Fig 2 are very nice but I find the right side confusing - I need some clarification on how to read these CDFs. Also, you state that the regions of over/under-confidence are clustered in input space, but it's not clear how we would tell if this is more than we'd expect given random variation or not\n- I'm skeptical that MLCE is the only metric worth looking at here - I'm also curious about something like average LCE\n- Table 2: how is this correlation calculated - over what sample of models?\n- Table 4: I'd like to see standard deviations/confidence intervals here. This type of metric is similar to an average LCE, which is good, but the wins seem uneven (which is fine). Also curious to here why some metrics show different results than others\n\n\nOther thoughts:\n\n- would be interesting to think about customizable kernels to inject notions of similarity we care about (e.g. the improper group-wise kernel mentioned in section 4)\n- Fig 3: tell me what these legend abbreviations mean\n- as bandwidth goes to \\infty, can you discuss what other methods it becomes similar to?\n- I would be interested to see exploration of which regions are uncalibrated for models\n", "summary_of_the_paper": "This paper introduces a new metric for measuring model reliability called local calibration, which allows for some insight into which regions of input space a model may be less well-calibrated in. They provide a post-hoc adjustment method for improving local calibration and empirically show that it is effective, without hurting overall measures too much.\n", "summary_of_the_review": "This is a sensible idea (metric and method) applied to an important problem. The exposition is fairly clear although some design choices are under-discussed. The experiments show that the method works as intended, although they could benefit from some deeper analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "P18G5ewrHqO", "reviewer": "Reviewer_Es9F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the idea of evaluating a model on both the feature space and output space is quite interesting. It provides a potential direction to extend the current probability calibration framework.\n\nOn the positive side, while the proposed LCE straightforwardly applies the kernel method, the authors put some reasonable effort into investigating the proposed measure's theoretical properties. While the proof of Theorem 1 relies on some existing work on kernel methods, it might provide some valuable references to other researchers interested in similar problems.\n\nOn the negative side, one main concern with this paper is the ambiguity of the proposed definition of Local Calibration. On the one hand, we have an existing definition of a perfectly calibrated model, which is solely defined in terms of the probabilistic output space. On the other hand, we also have the definition of a Bayes-optimal model, which is defined only on the feature space. For both cases, we can define a classifier that is optimal for either measure. Furthermore, we know that being a Bayes-optimal model is a sufficient but not necessary condition for being a perfectly calibrated model.\nHowever, while the proposed local calibration tries to incorporate both output space and feature space, it is unclear what is an optimal classifier under the definition due to the introduced kernel space. The author did not clearly show the requirements for a classifier to be perfectly local calibrated and its relationship to the Bayes-optimal model / perfectly calibrated models. It hence could be problematic when we want to treat local calibration as a measure to optimise.  \n\nAlso, while the authors used NLL and BS for their experiments, they did not discuss much the framework of Proper Scoring Rules. PSRs can be used to evaluate the overall quality of a probabilistic forecaster in terms of both refinement and calibration. A classifier with minimal refinement loss and calibration loss is Bayes-optimal and optimal both on the feature and probabilistic output spaces. It is therefore closely related to the local calibration concept proposed in this paper. I would therefore encourage the authors to consider potential links to PSRs further so that the local calibration concept can be clearly linked to calibrated / Bayes-optimal classifiers.\n\nA minor suggestion is also to consider links to existing kernel-based probabilistic classifiers (relevant vector machines / Gaussian Processes), where the probability output is directly optimised with a given kernel space.", "summary_of_the_paper": "This paper proposes a new measure of calibration called Local Calibration.\nWhile conventional calibration measures are only defined with probabilistic outputs, the proposed local calibration measure further incorporates the feature space by considering the neighbouring region with a kernel.\nThe authors also propose a calibration method according to the definition and experimentally demonstrate the advantages.", "summary_of_the_review": "This paper provides an interesting method to improve probabilistic outputs with the proposed definition of local calibration. However, the local calibration concept is not linked with the existing definition of calibrated / Bayes-optimal classifiers, so it might not stand firm as a measure to optimise. Some further links to exciting kernel-based classifiers will also bring better insights to the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9_4PstEB6KC", "reviewer": "Reviewer_fisU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n    1. The motivation of this work is very clear.\n    2. The article is very clear and easy to understand.\n    3. A large number of experiments were carried out to prove authors' claim.\nWeakness:\n    1. Why choose the binning scheme over predicted model confidences? How about taking all samples into account, or other schemes? \n    2. For a unique sample, the neighborhood the authors defined maybe change when the size of its bin is changed, does it make sense? I think you can refer to [1].\n    3. Is the promising experimental result due to the superiority of the method, or is it an accidental result from a specific kernel function and feature map?\n    4. Since performance is sensitive to many choices, as discussed in Section 3.2 ``the MLCE is very sensitive to \\gamma\", the authors set \\gamma as 0.2 and show the good performance, but how can I choose the best one in real-word applications? So clear and strong ablation study is necessary.\n   5.  The difference between the confidence and the accuracy for one sample is not reasonable when using it in Eq. 4. Since the accuracy is not a good metric for only one sample, although the authors propose using neighbors (what about the case the prediction of the current sample is wrong?)\n   6. Using kernel to introduce locality is not new.\n\nOther concerns:\n    1. The style of Table 3 is different from other tables, is there anything special the authors want to emphasize? Although the article is clear on the whole, there will still be misunderstandings on this point.\n    2. In Fig. 3 and Fig. 4, there are no error bars, while the other same type figures (Fig. 5, Fig. 6 and Fig. 7) have reported that. \n\n[1] Karandikar, A. , et al. \"Soft Calibration Objectives for Neural Networks.\" (2021).", "summary_of_the_paper": "This paper focuses on proposing the local calibration error (LCE) to span the gap between average and individual reliability, which measures the average reliability of a set of similar predictions, proposing a metric to measure calibration locally around a given prediction. Experiments are conducted and the results seem good, beating most baselines in the experiments chosen.", "summary_of_the_review": "The paper proposes a new metric for calibration, and propose a post-calibration algorithm for this metric. Overall, the novelty is a little weak and the improvements are not quite clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3F3RXalhb1", "reviewer": "Reviewer_784d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: the paper is well written and the proposed LoRE method seems simple yet effective. The proposed local calibration concept is novel. \n\nWeakness:\n\n1. Lack of motivation for the proposed local calibration concept. The authors indeed mentioned about fairness and causal inference as two potential interesting applications, and conducted fairness related tasks in the experimental section. But the connection is not clear to me. Furthermore, the authors should  provide some comments on the pros and cons of LCE over existing fariness related group-wise calibration measures (ref i).  \n\n2. The dimensional reduction procedure. There should be some detailed discussion on how to choose the kernel bandwidth (based on the empirical observation of the transition behavior?). \n\n3. Experiments. While the provided experiments are highly useful, I think both 5.2 and 5.3 need improvement. Specifically, in 5.2, include some popular baseline datasets such as CIFAR 10/100 ; in 5.3, the authors might want to compare against existing group-wise calibration measures & recalibration methods.\n\n4. Some discussions and/or comparisons about highly related and recent work are missing, such as [i] for fairness and calibration and [ii, iii] for recently proposed kernel-based calibration measures.\n\n[i].Pleiss, Geoff, et al. On Fairness and Calibration. NeurIPS, 2017.\n[ii]. Zhang et al. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. ICML, 2020. \n[iii] Gupta et al. Calibration of neural networks using splines. ICLR, 2021. ", "summary_of_the_paper": "The paper proposes a novel calibration metric, the local calibration error (LCE) that measures the average miscalibration degree of local predictions, which complements the existing global ECE metric. In addition, the authors also propose a novel local recalibration method called LoRe to minimize LCE.", "summary_of_the_review": "As discussed above, my primary concerns for this paper is on the motivation of local calibration, as well as some possible improvement over the experimental section (such as comparison with existing approaches). I hope that those concerns can be addressed in the rebuttal process.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all generally find the paper both well-motivated in addressing an important challenge as well as well-written. However, there's quite a bit of hesitation around whether the proposed metric is convincing enough as an approach to measure local calibration.

Reviewer 76PS and 784d's concerns around the choice of feature map and associated hyperparameters remain unaddressed, and I agree with their concern. There is no clear understanding of what constitutes a "good" feature map, which makes the metric quite difficult to use whether as a benchmark of ML methods or for general application.  I recommend the authors use the reviewers' feedback to enhance their preprint should they aim to submit to a later venue.
Decision: Reject