Measuring Calibration in Deep Learning
Authors: Jeremy Nixon
Mike Dusenberry
Ghassen Jerfel
Linchuan Zhang
Dustin Tran
Venue:  (2020)
Keywords: Deep Learning
Multiclass Classification
Classification
Uncertainty Estimation
Calibration

Abstract: Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. We propose two new measures for calibration, the Static Calibration Error (SCE) and Adaptive Calibration Error (ACE). These measures take into account every prediction made by a model, in contrast to the popular Expected Calibration Error.

Under review as a conference paper at ICLR 2020
MEASURING CALIBRATION IN DEEP LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. We propose new measures for calibration, the Static Calibration Error (SCE) and Adaptive Calibration Error (ACE). These measures taks into account every prediction made by model, in contrast to the popular Expected Calibration Error.
INTRODUCTION
The reliability of machine learning model's confidence in its predictions is critical for high risk applications, such as deciding whether to trust a medical diagnosis prediction (Crowson et al. 20016) Jiang et al. [201T: Raghu et al. 2018). One mathematical formulation of the reliability of confidence is calibration (Murphy & Epstein, 1967 Dawid 1982).
Intuitively, for class predictions, calibration means that if model assigns class with 90% probability, that class should appear 90% of the time.
Calibration not directly measured by proper scoring rules like negative log likelihood or the quadratic loss. The reason that we need to assess the quality of calibrated probabilities is that presently, we optimize against proper scoring rule and our models often yield uncalibrated probabilities.
Recent work proposed Expected Calibration Error (ECE; Nacini et al. 2015), a measure of calibration error which has lead (e.g., Guo et al.) 2017 Kuleshov et al., 2018). In this paper, we show that ECE has numerous pathologies, and that recent calibration methods, which have been shown to successfully recalibrate models according to ECE, cannot be properly evaluated via ECE.
surge of works developing methods for calibrated deep neural networks
Calibration (and uncertainty quantification generally) is critical in autonomous vehicles, the explo- ration phase of may algorithms, medical applications, and many more safety-critical applications of machine learning. suite of recent papers (c.g.. Lee et al. 2017, Vaicenavicius et al. 2019) Thulasidasan et al.) 2019 Kumar al. 20188] Guo et 2017 Seo et al. (2019) use ECE to validate their models' calibration. We identify concerns with that methodology. As this metric has become expect criticism to change the decisions of anyone attempting to train well-calibrated classification models. We recommend that rather than using Expected Calibration Error, practitioners use Static Calibratinon Error or Adaptive Calibration.
the default choice for measuring calibration in industry and for researchers,
BACKGROUND & RELATED WORK
2.1 MEASUREMENT CALIBRATION
Assume the dataset of features and outcomes (z, v)} are i.i.d. realizations of the random variables X, Y ~ P. We focus on class predictions. Suppose a model predicts a class with probability p. The model is calibrated if p is always the true probability. Formally.
P(Y = P p)
for all probability values E [0, 1] and class labels y E {O..... K KK-1,, 13. The left-hand-side denotes the true data distribution's 's probability of a label given that the model predicts p; the saght-hand- side denotes that value. Any difference between the left and right sides for a given p is known as calibration error.

---

Under review as a conference paper at ICLR 2020
Expected Calibration Error (ECE). To approximately measure the calibration error in expecta- tion, ECE discretizes the probability interval into a fixed number of bins, and assigns each predicted probability to the bin that encompasses it. The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (confidence). Intuitively, the accuracy estimates P(Y y D p), and the average confidence is setting of ECE computes weighted average of this eπror across bins:
B ECE = Σ
n N
acc(bb - con
where ny is the number of predictions bin b, N the total number of data points, and acc( and (b) are the accuracy and confidence of bin b, respectively. ECE as framed in Naeini et al. (2013) leaves ambiguity in both its binning implementation and how to compute calibration for multiple classes. In Guo al. (2017), they bin the probability interval [0, into equally spaced subintervals, and they take the maximum probability output for each datapoint (i.e., the predicted class's probability). We use this for our ECE implementation.
2.2 PROPER SCORING RULES & OTHER MEASURES OF CALIBRATION.
Many classic methods exist to measure the accuracy of predicted probabilities. For example, the Brier score (quadratic loss) measures the mean squared difference between the predicted probability and the actual outcome (Gneiting & Raftery 2007). This score can be shown to decompose into sum of metries, including calibration error. practice, training against this proper scoring rule, like training against log likelihood, does not guarantee well calibrated model. The floser-Lemeshow test is a popular hypothesis test for assessing whether a model's predictions significantly deviate from perfect calibration (Hosmer & Lemesbow 1980). The reliability diagram provides visualization of how well-calibraten a model is (DeGroot & Fienberg. 1983) Adaptivity was explored by (Nguyen & O'Connor 2015) and applied in (Hendrycks al. 2018). Kuleshov al. (2018) extends ECE to the regression setting. Unlike the Brier score and Hosmer-Lemeshen we'd like the metric to be scalar-valued in order easily benchmark methods, and to only measure calibration.
2.3
POST-PROCESSING METHODS
2.3.1
STANDARD APPROACHE FOR MULTICLASS CLASSIFICATION
One common approach to calibration is to apply post-processing methods to the output of any classifier without retraining. The two most popular cost-processing methods are the parametric approach of Platt scaling (Platt et al. 1999) and the non-parametric approach of isotonic regression (Zadrozny & Elkan 2002).
Platt scaling fits logistic regression model to the logits of a classifier, on the validation set, which can be used to compute calibrated predictions at test time. The original formulation of Platt scaling for neural networks (Niculescu-Mine Mizil & Caruana 2005) involves learning scalar parameters a, e R on held-out validation set and then computing calibrated probabilities p. given the uncalibrated logits vector 2, on the test set as pú (az; b). These parameters are typically estimated by minimizing the negative log likelihood.
Platt scaling can be extended the multiclass setting by considering higher-dimensional parameters. In matrix scaling is replaced with W e IR KK while we consider b E IR As for vector scaling; W E R Calibrated probabilities for either of these extensions can then be computed as
max o (W b).
An even simpler extension is temperature scaling Guo et al. 2017) which reduces the set of regression parameters the inverse of single scalar T such that
ps max (2/
On the other hand, isotonic regression is a common non- parammetttticc processing method that finds the stepwise-constant non-decrasinng (isotonic) function f that best fits the data according
2

---

Under review as a conference paper at ICLR 2020
Predictions ECE Omits
SCalibration Sçores Againşt Label Noise
% Omitted Predictions Threshold (0.1)
0.25
BCE
Calibration Error
0.10
ACE
0.20
0.00
0.06
0.04
0.10
00000
0.05
0.00
0.00
-0.01
0.00
0.01
0.02
0.03
0.04
0.05
0.66
Label Noise
Accuracy Against Label Nojse
Model Confidence ys. Label Noise
10
Mean Confidence Maximum Prediction
1.0
0.9
0.9
Validation Accuracy
0.8
0.8
0.7
0.7
0.5
0.4
03
0.2
-0.01
0.00
0.01
0.02
0.03
0.04
0.05
0.06
-0.01
0.00
0.01
0.02
0.03
0.04
0.05
0.06
Label Noise
Label
Figure 1: Top Left: Calibration measured through three metrics MNIST with randomly assigned labels, with the fraction of randomly assigned labels on the axis (all others labels are correct). Models (here, multinomial logistic regression) are re-trained each noise level. The base level of calibration error will be different, ACE and SCE account for all predictions, while ECE only accounts for the prediction with maximum confidence for each sample. These predictions have different average calibration errors. errors.Top Right: Often, important predictions (confidence 0.1) are not captured the calibration error by ECE. As models trained with more label noise exhibit more uncertainty, that number increases the amount of label noise increases. Bottom: As label noise increases, model output certainty and accuracy decrease.
mean-squared loss function (((())) y where pa are the uncalibrated probabilities and y; the labels.
The standard approach for extending isotonic regression to the multiclass setting is to break the problem into many binary classification problems (e.g. one-versus-all problems), to calibrate each problem separately, and then combine the calibrated probabilities (Zadrozny & Elkan 2002).
ISSUES WITH CALIBRATION METRICS
3.1 NoT COMPUTING CALIBRATION ACROSS ALL PREDICTIONS
Expected Calibration Error was crafted to mirror reliability diagrams, which are structured around binary classification such as rain vs not rain (DeGroot & Fienherg 1983) (Figure 1). A consequence that the eиror metric is reductive in multi-class setting. In particular, ECE is computed using only the predicted class's 's probability, which implics the metric does not assess how accurate a model is with respect to the K other probabilities.
We examine increasing the importance of omitted predictions via label noise. As label noise increases, trained models become less confident in their predictions. Secondary and tertiary predictions correct with higher frequency, and so for many samples the prediction with the correct label isn t included in ECE's measure of calibration error. We find that ECE becomes worse approximation of
3

---

Under review as a conference paper at ICLR 2020
Reilability Diagram
12.4
Paniclass Tast ACE
Prediction Accurcyy
0.6
608
-02 Prediction Confidence
0.8
Class
Figure 2: Left: Per-class Adaptive Calibration Error (ACE) for a trained 110-layer ResNet on the CIFAR-I00 test set. We observe that calibration error non-uniform across classes, which difficult to express when measuring error only on the maximum probabilities. Right: Reliability Diagrams are an interpretable visualization of calibration error. This is an example of the reliability of slightly overconfident CNN trained on
the calibration error as relevant predictions not included in the ECE metric become common (Figure 1).
In Figure 2] we observe that calibration error, when measured in per-class manner, is non-uniform across classes. Guaranteeing calibration for all class probabilities is an important property for many applications. For example, physician may deeply about secondary tertiary prediction of a disease. Decision costs along both ethical and monetary axes are non-uniform, and thus clinical decision policies necessitate comprehensive view of the likelihoods of potential outcomes (e.g., Tsoukalas et al., 2015). Alternatively, a true data distribution exhibits high data noise (also known as high aleatoric uncertainty, weak labels), then we'd like models to be calibrated across all predictions as no true class label is likely.
3.2 FIXED CALIBRATION RANGES
One major weakness of evenly spaced binning metrics caused by the dispersion data across ranges. In computing ECE, there often a large leftward skew in the output probabilities, with the left end of the region being sparsely populated and the rightward end being densely populated. Sharpness, which is the desire for models to always predict with high confidence, i.c.. predicted probabílities concentrate to 0 or 1, is a fundamental property (Gneiting et al. (007). (That is, network predictions are typically very confident.) This causes only a few bins contribute the most ECE typicalyy one or two as bin sizes are 10-20 in practice (Guo et al. 2017
3.3 BIAS- VARIANCE TRADEOFF
'Ground truth' is the calibration eтror we would measure if we could see an infinite number of predictions particular confidence level. That would give long-run error for every prediction, which we could aggregate into an overall calibration error. Because we don't 't have that infinite set of reference predictions for every real-valued prediction, we try approximate that set by using the values that are close together as reference predictions. As we increase the number of datapoints in bin, we get more accurate estimate of the calibration in a given range through the added (seen as reduction in variance). But as the datapoints in the bin start to differ from one another due to the extension of the range, they bias the estimate of the calibration within that bin (we refer to this as bias).
Selecting the number of bins has a sias-variance tradeoff as it determines how many data points fall into each bin and therefore the quality of the estimate of calibration from that bin't 's range. In particular, a larger number of bins causes more granular measures of calibration error (low bias) but also high variance of each bin's measurement as bins become sparsely populated. This tradeoff
4

---

Under review as a conference paper at ICLR 2020
compounds particularly with the problem of fixed calibration ranges, due sharpness certain bins have many more data points than others.
3.4 PATHOLOGIES IN STATIC BINNING SCHEMES
Metrics that depend on static binning schemes like ECE suffer from issues where you can get near calibration eтror due to overconfident and underconfident predictions overlapping in the same bin. These cancellation effects may not always be extreme enough push an incredibly uncalibrated model to O calibration error, but it can be difficult to judge when using static metric whether differences between models or techniques are due to a true improvement in the model's calibration or whether improved calibration is due to cancellation.
Code in the appendix (Figure 5) demonstrates this effect.
CHALLENGES IN CALIBRATION
Before describing new metrics for calibration, we first outline broad challenges with designing such metrics.
4.1 GROUND TRUTH & COMPARING CALIBRATION METRICS
There are many challenges in measuring a network's s calibration, beginning with the absence of ground truth. In principle, one can limit comparisons to controlled, simulated experiments where ground truth is available by drawing infinitely many samples from the true data distribution. However, even with ground truth, any estimation property such as bias remains difficult to compare across estimators, as "ground truth error" is multi-value and estimators may make measurements for different elements of these values. Specifically, calibration is a guarantee for all predicted probabilities E [0. 1 and class labels y (O....: K }} (see also the calibration function of Vaicenavicius al. (2019)) An estimator may have lower bias and/or variance in estimating the error for specific ranges of and but higher bias and/or variance in other ranges.
The other major challenge with respect to ground truth is that due to the differences in their assump- tions, different calibration metries approximate different overall calibration values. ECE doesn't 't look at all predictions, and is estimating a different value than SCE. The ACE vs SCE comparison possible with respect to the probabilities they look at, but as the static binning scheme imposes different bias than the adaptive scheme, it's a challenge to compare static metrics to adaptive metrics.
4.2 WEIGHTING
Because calibration error is multi-valued. and a desirable metric scalar-valued, the question of how to weight probability values (where one thresholding 0/1 weighting datapoints below above the threshold) creates a set of differences in calibration error metrics that choose to emphasize different aspects of calibration performance. In many contexts what matters most is the rare event-the network's 's classifications leading up to an accident, the presence of a planet, or the presence of rare disease. Knowing the difference between whether a class's 's true probability is ,01 and ,001 (the difference between 1 in 100 and 1 in 1000) is both extremely difficult to discern and may be much more relevant than difference between .3 and .301, which these calibration metrics would treat as equivalent. In these contexts, weighting the ends of the interval close to 0 and I would be ideal.
Additionally, in the context of out-of-distribution detection, we would prefer to be well-calibrated in the middle of the spectrum, whereby a prediction of (high uncertainty) more likely to happen.
NEW CALIBRATION METRICS
Each calibration error metric is evaluated on each class, independently. The results for each class are averaged.
5

---

Under review as a conference paper at ICLR 2020
5.1 MULTICLASS & STATIC CALIBRATION ERROR
We first introduce Static Calibration Error (SCE), which is a simple extension of Expected Calibration Error every probability in the multiclass setting. SCE bins predictions separately for each class probability, computes the calibration emror within the bin, and averages across bins:
K
Tbbk |aco(b, k) conf(b, (b,bbb))) k)]
SCE
Here, acc(b, k) and (b, k:) are the accuracy and confidence of bin for class label k, respectively; nbk is the number of predictions in bin b for class label k;; and N is the total number of data points. Unlike ECE, assuming infinite data and infinite bins, SCE is guaranteed to be zero if only if the model calibrated.
5.2 ADAPTIVITY & ADAPTIVE CALIBRATION ERROR
Adaptive calibration ranges are motivated by the bias-variance tradeoff in the choice of ranges, suggesting that order to get the best estimate of the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions). This leads to introduce Adaptive Calibration Error (ACE), uses an adaptive scheme which spaces the bin intervals so that each contains an equal number of predictions.
In detail, ACE takes as input the predictions (usually out of a softmax), correct labels, and number of ranges R.
R
ACE
Jaco(r, k) conf(r,
KR
Here, acc(r; k) and conf k) are the accuracy and confidence of adaptive calibration range r for class label k. respectively; and N the total number of data points. Calibration range defined by the [NV/ R/th index of the sorted and thresholded predictions.
5.2. SHARPNESS & ADAPTIVE CALIBRATION ERROR
major motivation for the Adaptive Calibration Eiror the realization that the majority of predictions made by deep learning models are close to or O. static binning scheme will group almost all predictions into two bins, one at each end, with very sparse predictions falling in the bins that are in between (Figure 3, Left). ACE adapts to focus on the areas where predictions are densest (Figure Right).
6 INTERPRETABILITY & ANALYSIS
6.1 INTERPRETATION OF CALIBRATION ERROR METRICS
The natural interpretation for both metrics that account for every prediction made by a classification model (SCE and ACE) that the calibration error is the average of all predictions made by the mode. The natural interpretation for ECF, in contrast, is that its calibration error is the average error of the highest confidence prediction made by the model.
6.2 CRITICISMS & LIMITATIONS OF ADAPTIVE CALIBRATION METRICS
These adaptive calibration metrics can create excessively large ranges there is very sparse output in region. In those cases, it may be better to have higher variance estimate of the calibration, but only compare datapoints that are closer together (dropping datapoints that are farther away). This level of granularity can be replicated with an adaptive scheme that has very large number of ranges, but that will come at the cost of easy interpretability and will increase the within-range variance across all ranges, not just the sparse regions.

---

Under review as a conference paper at ICLR 2020
CNN Prediction Sharpness
12000
ACE Calibration Ranges
Bin Lower Bound
1.0
0000
0.8
4000
0.2
0.0
Training Step
0.6 0.8
Figure 3: Left: Model sharpness, where the vast majority of confidence scores are very near 1 or very near 0, motivates ACE. This caasicoondiional histogram of confidence scores 12,000 samples from Bushion-MNIS"" demonstrates that sharpness. Right: Binning scheme chosen by ACE measured each step of training Fashion-MNIST. Each bin measuring the calibraion error of datapoints within a particular range has lower bound to the bin and an upper bound. As the model's predictions become sharper, the chosen bins adapt focus on regions with a high concentration the predictions. Each color represents the lower bound of the nth largest bin, for 100 bins.
This scheme does not accomplish transfer smoothly between related predictions, although en- sembling hierarchical calibration ranges could approximate that smoothness. This limitation does suggest an smoother metric, akin to our algorithm that leverages dynamic programming to look at very many range schemes simultaneously & efficiently.
With static bin calibration metric can straightforward to compare the outputs two very different models through reliability diagram. Due to the variability of their binning scheme based on the density of the model's predictions, adaptive calibration metrics make it more difficult to compare two models sidee in reliability diagram, as their points fall in different regions depending the bin locations that are selected.
6.3 HYPERPARAMETER ANALYSIS
Adaptive calibration metrics are more sensitive the binning hyperparameter. Empirically, there larger changes in calibration error due to bin count choice than static metrics. This variance decreases as the bin count rises, leveling off at 50 our experiments. We suggest using ACE with 50+ bins for models trained on datasets akin to CIFAR ImageNet.
EXPERIMENTS
7.1 RECALIBRATION EXPERIMENTS
We trained ResNet- 110 model (He et al. 2016) on CIFAR- 100 (Krizhevsky & Hinton 2009). We then estimated the regression parameters, for each postprocessing method, on the validation held-out set. Finally, we used parameter estimates to compute calibrated probabilities on the test set. The results are reported in Table 3] The same experiment with ResNet-50 model on ImageNet 2012 is reported in Table 4 and echoes the findings of Table
Table 3 shows that temperature scaling outperforms the other 3 methods ECE as reported in (Guo ct al. (017). However, temperature scaling falls short in comparison to vector scaling on the SCE and ACE metrics. In fact, the relative gap SCE is the largest, among all metrics and across methods. This is unsurprising given that SCE is a multiclass extension ECE which considers all probabilities, instead of the top one. Vector scaling typically does best.
For example, temperature scaling (Guo et al. 2017) has been shown to effectively minimize expected calibration error better than alternative techniques such isotonic regression and Platt scaling
7

---

Under review as a conference paper at ICLR 2020
Method
ECE
SCE
ACE
Uncalibrated
19.64%
0.41%
0.13%
Temp. Scaling
2.16%
0.06%
0.06%
Vector Scaling
2.27%
0.04%
0.01 %
Matrix Scaling
12.11%
0.26%
0.26%
Isotonic Regr. 17.85%
0.35%
0.12%
Table 1: ECE, TACE, SCE, and ACE (with 15 bins) ResNet- applied to CIFAR-10K 100 before calibration, and after the application of post-processing methods. The best recalibration method depends the metric, which motivates its study.
Method
ECE
SCE
ACE
Uncalibrated
6.63%
0.02%
0.015%
Temp. Scaling
5.42%
0.01%
0.001%
Vector Scaling
1.44%
0.002%
0.004%
Matrix Scaling
5.06%
0.01%
0.001% %
Isotonic Regr. 3.474%
0.01%
0.000%
Table 2: ECE, SCE, and ACE (with 15 bins) ResNet-50 applied to ImageNet before calibration, and after the application of various extensions to Platt scaling and Isotonic regression. These percentages across metrics are not directly comparable.
(Plat al. 1999). The question whether these scaling techniques effectively minimize more sophisticated error metrics is an important standard for their efficacy.
In fact, by design having only a single scalar parameter which uniformly scales predicted probabil- ities across all classes, temperature scaling is likely perform worse when accounting for, for example, calibration errors across all predictions. We hypothesize that this would be the case with respect to adaptive calibration metrics (e.g. ACE) as well as calibration metrics that account for the probabilities of all classes instead of that of only the top predicted class (e.g. SCE).
When designing post-processing methods, not accounting for the properties detailed in Section 3 during evaluation can lead to misleading conclusions about a post-processing method's success.
7.2
LABEL NOISE
We train softmax regression (multinomial logistic regression) models 40 levels of label noise from 0.0 to 0.05. The label noise value corresponds to the percentage of labels that are randomly assigned, where the correct label included one of the possible random assignments.
One way to measure the importance of sample is its confidence. We look at the fraction of important predictions ECE omits by setting a threshold confidence (.01 in our visualized example, see Figure Top Right) and observe that this fraction rises with the difficulty of the modeling task (measured by model accuracy and mean maximum confidence, see Figure I Bottom). This result holds for wide range threshold levels.
8 CONCLUSION
This paper studies metrics for evaluating calibration. We makes the following contributions:
1. We identify problems with the widely used ECE metric to assess calibration.
2. We propose new metrics (SCE and ACE) to address each of these problems.
3. We recommend best practices for how evaluate calibration using our metries.
We believe that the way neural network uncertainties evaluated in future must be aware of the challenges we raise, and in light of those challenges recommend the use of SCE or ACE over ECE for evaluating multiclass classifier's calibration error.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Cynthia S Crowson, Elizabeth Atkinson, and Terry M Therreau. Assessing calibration of prognostic risk scores. Statistical methods in medical research, 25(4):1692-1706, 2016.
A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association, 77 (379):605-619) 610, 1982.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12-22 1983.
Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-377 2007.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2): 243-268, 2007.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2016. http:://///ax: org/ abbs/1122 01335511
Dan Hendrycks, Mantas Mazeika, and Thomas G Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 1812. 1812.0466. 2018.
David W Hosmer and Stanley Lemesbow. Goodness of fit tests for the multiple logistic regression model. Communications statistics-Theory and Methods, 110):1043-1069, 1980.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno- Machado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19(2):263-274, 2011.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint arXiv:1807.00263. 2018.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference Machine Learning, pp. 2810-2819, 2018.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibratedd classifiers for detecting out-of-distribution samples. arXiv preprint arXiv:1711.09325, 2017.
Allan H Murphy and Edward S Epstein. Verification of probabilistic predictions: A brief review. Journal of Applied Meteorology, 655):748-755, 1967.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint
2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised Icarning. In Proceedings of the 22nd international conference Machine learning, pp. 2005.
ACM,
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.

---

Under review as a conference paper at ICLR 2020
Method
ECE
TACE
SCE
ACE
Uncalibrated
19.64%
10.03%
0.41%
0.13%
Temp. Scaling
2.16%
0.52% %
0.06%
0.06%
Vector Scaling
2.27%
0.61%
0.04%
0.01%
Matrix Scaling
12.11%
3.98%
0.26%
0.26%
Isotonic Regr. 17.85%
2.83%
0.35%
0.12%
Table 3: ECE, TACE, SCE, and ACE (with 15 bins) ResNet-110 applied to CIFAR-100 100 before calibration, and after the application of post-processing methods. The best recalibration method depends the metric, which motivates its study.
Method
ECE
TACE
SCE
ACE
Uncalibrated 6.63%
2.51%
0.02%
0.015%
Temp. Scaling
5.42%
2.64%
0.01%
0.001%
Vector Scaling
1.44%
1.25%
0.002 %
0.004%
Matrix Scaling 5.06%
1.98%
0.01% %
0.001 %
Isotonic Regr. 3.474%
1.862%
0.01%
0.000%
Table 4: ECE, TACE, SCE, and ACE (with 15 bins) on ResNet-50 applied to ImageNet before calibration, and after the application of various extensions Platt scaling and Isotonic regression. These percentages across metrics are not directly comparable.
Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Robert Kleinberg, Sendhil Mullainathan, and Jon Kleinberg, Direct uncertainty prediction for medical second opinions. 2018.
Seonguk Seo, Paul Hongsuck Seo, and Bohyung Han. Learning for single-shot confidence calibration in deep neural networks through stochastic inferences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9030-9038, 2019.
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. arXiv preprint arXiv: 1905. 1001, 2019.
Athanasios Tsoukalas, Timothy Albertson, and Ilias Tagkopoulos. From Data to Optimal Decision Making: A Deiven Probabilistic Machine Learning Approach to Decision Support for Patients With Sepsis. JMIR Medical Informatics, 3(1) February 2015. ISSN 2291-9694. doi: jm org 201 e11/
10.2196/medinform..445
|RL http:d//mmeddaee fow.://memoooooommmooomooooooooo
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schön. Evaluating model calibration in classification. arXiv preprint UrXiv:1902.06977 2019.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694 699. ACM, 2002.
A APPENDIX
A.1 THRESHOLDING & THRESHOLDED ADAPTIVE CALIBRATION ERROR
One initial challenge is that the vast majority of softmax predictions become infinitesimal (Figure 3, Top Left). These tiny predictions can wash out the calibration score, especially in the case where there many classes, where large proportion of them model's 's predictions correspond to incorrect class. One response is to only evaluate on values above a threshold c. Mathematically. TACE is identical to ACE, with the only difference being that TACE is only evaluated on values above These predictions overlap with the predictions evaluated by ECE (all maximum values per datapoint), leading them to have similar reactions recalibration methods.
10

---

Under review as a conference paper at ICLR 2020
ACE Calibration Ranges
TACE, MNIST Training Set
Bin Lower Bound
1.0
Bin Lower Bound
0.8
1.0
0.8
0.6
0.4
0.4
0.2
0.2
0.0
-0.2
-0.2
Training Step
Training Step
ACE Calibration Ranges
TACE Calibration Ranges
Bin Lower Bound
1.2
Bin Lower Bound
12
1.0
0.6
0.4
0.4
0.2
0.0
-0.2
-0.2
Training Step
Training Step
Figure 4: Top Left: Lower bounds of calibrations ranges over the course of training for adaptive calibration eтrоr on Fashion-MNISS focusing almost entirely on small ranges and motivating thresholding. Top Right: On the MNIST training with thresholding, few values are small that the bottom of the lowest range often spikes to .99 and higher due to every datapoint being fit. Bottom Left: ACE on Mashionn-MISS validation with 100 calibration ranges. Bottom Right: Thresholded adaptive calibration with 50 calibration ranges over the course of training on Fashion-MNISTTs validation set.
predictions []
correct values []
for in (ange(450))
value = .48
predictions. append [1-value, value])
correct values.appendi (1p
for in =ang)(550))
value = .42
predictions append [l-value, value])
correct salleaaalleeeeeeeeeeeeeeeeeeeeee (ppend
softmax = mp. (ppparrray(reedicriooon))
http///etto.ccaaaaaataooooooooee (softmax, correct values, b_bins=10))
Figure 5: This python code (with numpy imported as np, and with an expected calibration error function loaded the environment) implements a pathology in ECE which will return 0 calibration error by including overconfident and underconfident values in the same bin. The softmax here has 450 predictions .52 (ECE only looks the maximum value) and evaluates all of them to be incorrect, despite being predicted with high confidence. The softmax also has 550 predictions at .58, all evaluated be correct, though they't predicted with low confidence (relative for correct and 0 for incorrect). Despite this extreme miscalibration, the expected calibration error is exremely low (0.003).
11


Reviews:
[{"review_id": "rJe58YUoYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\tThis paper proposes two novel scalar-valued evaluation measures, namely Static Calibration Error (SCE) and Adaptive Calibration Error (ACE)  for evaluating the quality of calibration (the accuracy of estimating p(y|x) in classification) in deep learning. The authors suggested that the existing Expected Calibration Error (ECE), which is the current most popular scalar-valued metric for calibration, has several drawbacks, especially in the multi-class scenario. Intuitively, ECE only focuses on the predicted class probability (argmax) and ignores class probabilities of other K-1 classes, which implies that ECE may fail to capture the class probability estimates for all classes. They also illustrated the drawback of ECE under label noise.\n\n========================================================\nContribution:\n1. Pointing out that ECE has several drawbacks in multiclass scenario, e.g., does not take predictions of all classes but only the one with maximum confidence.\n2. Proposing two novel measures: SCE and ACE, where SCE is a natural extension to make sure the metric consider all class probability estimates, and ACE is adaptive in the sense that it is focus on the regions where many predictions are made.\n3. Conducting experiments to illustrate that although temperature scaling may work very well when we used ECE as a metric, vector scaling can be advantegous when we consider SCE or ACE.\n\nClarity:\nAlthough there are typos, it is not difficult to understand the motivation and what this paper is trying to propose. But it is suggested that the paper was done in a rush manner.\n\n1. Issue in Figure 1 and experiment 7.2:\n\nI found that Figure 1 is difficult to understand and I may misunderstand. Moreover, I couldn't get the main message of it. \t\t1.1 (Top-left), the message I got is that the error based on each metric is bigger as the label noise increases (but each metric is incomparable). \n1.2 (Top-right), I couldn't get what Predictions ECE omits over threshold 0.1 means. It would be better to clearly explain it, e.g., how to compute % ECE omits over a certain threshold.\n1.3 (Bottom), I learned that with label noise, both accuracy and model confidence (which I think is max p(y|x) decreases as the noise increases, which is common. Moreover, does Against and vs. mean the same thing in this context, then using the same word can make it more consistent. \n\n\tMoreover, I wonder why we have to focus on the label noise because even in the normal scenario, ECE should have drawbacks already too and it is more interesting for me to see the illustration in the normal scenario. On the other hand, if I don't misunderstand, ECE did not omit a lot of predictions according to the top-right figure if there is no label noise. In practice, we may use a more sophisticated method to handle label noise. Because under label noise, the class probability estimation is already incorrect theoretically, i.e., it may shift depending on the noise type (Menon+, ICML2015: Learning from Corrupted Binary Labels via Class-Probability Estimation). And I am not sure why we have \"(Figure 1)\" at the end of the first sentence of Sec. 3.1, is that sentence related to Figure 1? In my opinion, Figure 2 is much better to visualize that ACE may capture things that ECE fails to capture. Finally, Figure 1 is a part of experiment 7.2 and I think it is fine to move this to the experiment section as an additional experiment under the label noise scenario. Finally, how to train your model in Figure 1, is it uncalibrated version? i.e., without temperature scaling or other modifications.\n\n2. Figure 3 is difficult to understand. \n\t2.1 (Left) what is sharpness, how to calculate sharpness and what is the y-axis? And what is the x-axis here, is it a confidence score?.\n\t2.2 (Right) What is the training step? And there is no value specified in the x-axis. I couldn't understand how to plot this figure.\n3. Tables 1 and 2 are never discussed and in \"Table 1: ECE, TACE, SCE, and ACE\", there is no TACE and never mention in the main body of the paper. Instead, the main body discussed about Tables 3 and 4, which is not in the main body (or is it? since it is in between the reference).\n4. How many trials did you run the experiment and what criteria you use to give boldface to a method? Since this paper also highly relies on the experimental results, it would be great to clarify. \n\n========================================================\nComments:\nThe authors did a great job to point out the problem of ECE. Although SCE is a very simple and natural extension of ECE, its contribution is significant because it relevates the drawback of ECE as the authors suggested. I believe this work can make an impact to the field. For ACE, I have an impression that it is difficult to use. Also, it would be nice to see the performance with respect to Maximum Calibration Error (MCE), which is completely ignored in this paper. Because in MCE, we can see that temperature scaling did not almost always perform significantly better than other methods as it performed with respect to ECE (in Appendix of Guo et al., 2017), which is similar to what we observed in SCE and ACE.\n\nUnfortunately, although I like the idea of this paper, I found that the clarity of the paper is insufficient in its current state. It seems that the paper was really done in a rush and thus the writing can be highly improved. As a result, given the current manuscript, I vote a weak reject for this paper.\n\n========================================================\nAdditional questions:\n1. Why the name of SCE is static calibration error? If you mean it is not adaptive as ACE, then ECE is also static in this sense. Therefore, it may be a good idea to come up with a different name,e.g., classwise calibration error.\n2. May SCE and ACE suffer from class-imbalance scenario more than ECE?\n3. Are there any advantages of ECE over SCE and ACE?\n\n========================================================\nPotential typos I found:\n1. Abstract: the last sentence: taks -> takes\n2. Abstract: Overconf. and underconf. is -> are\n2. INTRO: the first sentence of the last paragraph may algorithms -> many algorithms\n3. INTRO: 4th paragraph: has lead -> has led\n4. INTRO: last sentence: Static Calibratinon -> Static Calibration\n5. INTRO: last sentence Adaptive Calibration -> Adaptive Calibration Error\n6. 2.1: {(x,y)} should be {(x,y)}_{i=1}^{N}?\n7. all predictions made by the mode -> all predictions made by the model\n8. in fig.3: calibraion -> calibration\n9. Table 1: remove TACE\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1escKFm5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary: This paper proposes two simple extensions to Expected Calibration Error (ECE):  1) SCE which accounts for multiclass classification settings by averaging over all the errors due to all classes (as opposed to error in the top one class only in ECE) and 2) ACE which attempts as distributing the predictions equally across the bins (as opposed to have too few highly populated bins in the interval). Authors evaluated their approach using ResNet-110 on CIFAR100 and ResNet-50 on ImageNet against most common post-training calibration methods. \n\nPros:\n(+): The paper is well-motivated.\n(+): The problem is important and has direct real world applications.\n(+): The idea is simple and viable to improve ECE.\n\nCons that significantly affected my score and resulted in rejecting the paper are as follows:\n\n1 - Experimental setting and evaluations:\nThe biggest drawback in this paper is the experimental setting which is not rigorous enough to show the effectiveness of the proposed metrics due to the following reasons:\n(a) The first proposed extension (SCE) is too incremental. However, while I am not against simple and effective methods (I have even listed this as a pro above), I think it should be backed up with more thorough experiments and discussions. SCE is supposed to be more effective in case of having more number of classes however authors do not shed light into this properly by taking advantage of comparing their CIFAR100 and ImageNet 1K experiments. They keep emphasizing on the fact that the numbers provided in Table 1 and 2 are not comparable so how is the reader supposed to understand their differences? They also show their metrics give far less error to models than ECE in Table 1 and 2 and yet it is not clear why that is. Not to mention that authors need to state these results are obtained on how many runs and to report stds because the numbers appear too close and are hence not conclusive. \n(b) Datasets and architectures: Authors have used MNIST and FASHION-MNIST when discussing the shortcomings of ECE. I was wondering what architecture they used for these experiments? The reason I am asking is that it is a known fact (see Guo et al., ICML17) that the simpler the architecture is, the more calibrated its predictions tend to be and vice versa. For instance LeNet 5 appears to be much more well-calibrated than more modern neural networks such as ResNet variants despite being less accurate (Guo et al, 2017). Therefore, it will probably be in their favor to show these issues on a less calibrated model.\n(c) Can authors please explain why they have used ResNet-110 on CIFAR100 and ResNet-50 on ImageNet? I assume it might be because they intended to compare to (Guo et al, 2017) but there are more experiments there for comparison. Have they also tried any other architecture? It has been shown before that there can be a noticeable difference across different architectures used on a fixed dataset. Authors may want to add the arch effect to their evaluation. \n\n2) The second metric (ACE) while it is well-motivated, it is defined such that it leaves the impression that applying it will involve lots of heuristics as there is no systematic procedure is given. In section 6.3 (where I think could be a section to address the implementation details for this) no effective information is given either expect that authors recommend using 50+ bins for CIFAR and ImageNet without any quantitative support. It is also not clear why authors used 15 bins in their experiments for these datasets in Table 1 and 2 (maybe comparing to the baseline? but they could show both specially if 50+ bins is better for ACE). Moreover, in section 5.2 where it says “the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions).“, I was wondering if authors have any suggestions on how to identify the predictions with low confidence as coming from in or out-distribution?\n\n2- Structure of the paper, writing, and visualizations:\n\n(a) Writing: The paper, in its current form, needs to be thoroughly proofread and reorganized. The text does not read well and is vague in most parts. Authors have spent too much time explaining the drawbacks of the prior work in 4 pages (entire sections 2,3, and 4) only to propose their ideas in section 5.1 and 5.2 in almost half a page. Unfortunately the analysis section (6) acts very poorly in providing a thorough exploration into their method.\n\n(b) Dividing each section to too many subsections has hurt the flow as each section is too short and does not provide deep evidence to support its title/subtitle. \n\n<<< Note that the followings are less major and are given only to help, and not necessarily part of my decision assessment >>>\n\n(c) Results shown in Table 3 and 4 are reporting “TACE” which is technically explained in the Appendix. Apart from the fact that authors need to make sure the Table comes before the references (not in the middle) but more importantly, as a paper submitted to ICLR I would expect it to be self-contained and be able to provide all the details needed. Authors should either move this Table to the appendix or move section (A.1) to the main text. \n\n(d) In section 3.4. Authors provide support for their claim on issues with binning scheme with a screenshot of their code in the Appendix. I think it is important to show this effect but in perhaps a table with quantitative results within the main text. \n\n(e) Title for Section 5 as the main contribution of this paper should be changed to reflect that.\n\n(f) Abstract in its current wording, does not provide absolute no detail into the proposed metrics. Whereas it can, because 1) it encourages the reader to keep reading 2) the method is simple enough to be summarized here.\n\n(g) The figures do not meet the conventional scientific standards and have to be significantly improved. Standard deviations are missing.\n\n(h) There are also grammar errors and typos, (for example on page 1 paragraph 5, the word \"may\" should be \"many\"), for which I have found passing my writing through the free version of Grammarly very helpful in getting rid of most such errors. \n\nAs a final note, I think this paper can be contributive to the field as it provides a novel and simple extension to a widely used calibration metric. However it needs to be written in a more effective way and be supported by a more rigorous experimental setting. Therefore, I will be willing to change my score if the presented issues will be addressed. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxmYff5Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After an interesting review of calibration methods, the paper describes two new methods for assessing calibration. The first method, SCE, is an extension of the usual ECE to the multi class setting. The second method, ACE, is a slight variation where bins are computed adaptively.\n\nThe paper is interesting and relatively well written. Although the contribution is rather simple (can be describe in less than a page), I can see myself using the SCE/ACE metrics in the near future.\n\nIn order to strengthen the submission, I still feel that the authors should try to describe why the newly introduced metrics can help in applied scenarios.\n(1) OK, the ECE omits a lot of important predictions (e.g. fig 1) --> give a real application where this matters\n(2) OK, adaptive binning seems a sensible approach. --> give a real application where a difference between 0.99 and 0.999 does make a difference\n\nThe newly introduced metrics *are* interesting, and the theoretical justifications *are* sensible. The paper would be even better if the applied motivations were better described. Since the proposed method does have the potential to be applied in industrial/applied scenarios, it is slightly disappointing that it is presented as another academic exercise.\n\nminor remark: it is not clear why the factor (1/K) is needed in the definition of SCE since the weights are already summing up to one -- this makes SCE comparisons between datasets with different number fo classes more difficult.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose two measures of calibration that don't simply rely on the top prediction. The reviewers gave a lot of useful feedback. Unfortunately, the authors didn't respond.
Decision: Reject