Title: MEASURING CALIBRATION IN DEEP LEARNING

Abstract: Overconfidence and underconfidence in machine learning classifiers is measured by calibration: the degree to which the probabilities predicted for each class match the accuracy of the classifier on that prediction. We propose new measures for calibration, the Static Calibration Error (SCE) and Adaptive Calibration Error (ACE). These measures taks into account every prediction made by a model, in contrast to the popular Expected Calibration Error.
The reliability of a machine learning model's confidence in its predictions is critical for high risk applications, such as deciding whether to trust a medical diagnosis prediction (Crowson et al., 2016; Jiang et al., 2011; Raghu et al., 2018). One mathematical formulation of the reliability of confidence is calibration (Murphy & Epstein, 1967; Dawid, 1982) .
Intuitively, for class predictions, calibration means that if a model assigns a class with 90% probability, that class should appear 90% of the time.
Calibration is not directly measured by proper scoring rules like negative log likelihood or the quadratic loss. The reason that we need to assess the quality of calibrated probabilities is that presently, we optimize against a proper scoring rule and our models often yield uncalibrated probabilities.
Recent work proposed Expected Calibration Error (ECE; Naeini et al., 2015) , a measure of calibration error which has lead to a surge of works developing methods for calibrated deep neural networks (e.g., Guo et al., 2017; Kuleshov et al., 2018) . In this paper, we show that ECE has numerous pathologies, and that recent calibration methods, which have been shown to successfully recalibrate models according to ECE, cannot be properly evaluated via ECE.
Calibration (and uncertainty quantification generally) is critical in autonomous vehicles, the exploration phase of may algorithms, medical applications, and many more safety-critical applications of machine learning.

Expected Calibration Error (ECE).:

To approximately measure the calibration error in expectation, ECE discretizes the probability interval into a fixed number of bins, and assigns each predicted probability to the bin that encompasses it. The calibration error is the difference between the fraction of predictions in the bin that are correct (accuracy) and the mean of the probabilities in the bin (confidence). Intuitively, the accuracy estimates P(Y = y | p = p), and the average confidence is a setting of p. ECE computes a weighted average of this error across bins:

ECE = B b=1 n b N |acc(b) -conf(b)| ,

where n b is the number of predictions in bin b, N is the total number of data points, and acc(b) and conf(b) are the accuracy and confidence of bin b, respectively. ECE as framed in Naeini et al. (2015) leaves ambiguity in both its binning implementation and how to compute calibration for multiple classes. In Guo et al. (2017) , they bin the probability interval [0, 1] into equally spaced subintervals, and they take the maximum probability output for each datapoint (i.e., the predicted class's probability). We use this for our ECE implementation.

2.2. PROPER SCORING RULES & OTHER MEASURES OF CALIBRATION.:

Many classic methods exist to measure the accuracy of predicted probabilities. For example, the Brier score (quadratic loss) measures the mean squared difference between the predicted probability and the actual outcome (Gneiting & Raftery, 2007) . This score can be shown to decompose into a sum of metrics, including calibration error. In practice, training against this proper scoring rule, like training against log likelihood, does not guarantee a well calibrated model. The Hosmer-Lemeshow test is a popular hypothesis test for assessing whether a model's predictions significantly deviate from perfect calibration (Hosmer & Lemesbow, 1980) . The reliability diagram provides a visualization of how well-calibrated a model is (DeGroot & Fienberg, 1983) . Adaptivity was explored by (Nguyen & O'Connor, 2015) and applied in (Hendrycks et al., 2018) . Kuleshov et al. (2018) extends ECE to the regression setting. Unlike the Brier score and Hosmer-Lemeshow, we'd like the metric to be scalar-valued in order to easily benchmark methods, and to only measure calibration.

2.3. POST-PROCESSING METHODS:



2.3.1. STANDARD APPROACHES FOR MULTICLASS CLASSIFICATION:

One common approach to calibration is to apply post-processing methods to the output of any classifier without retraining. The two most popular post-processing methods are the parametric approach of Platt scaling (Platt et al., 1999) and the non-parametric approach of isotonic regression (Zadrozny & Elkan, 2002) .

Platt scaling fits a logistic regression model to the logits of a classifier, on the validation set, which can be used to compute calibrated predictions at test time. The original formulation of Platt scaling for neural networks (Niculescu-Mizil & Caruana, 2005) involves learning scalar parameters a, b ∈ R on a held-out validation set and then computing calibrated probabilities pi given the uncalibrated logits vector z i on the test set as pi = σ(az i + b). These parameters are typically estimated by minimizing the negative log likelihood.

Platt scaling can be extended to the multiclass setting by considering higher-dimensional parameters.

In matrix scaling a is replaced with W ∈ R K * K while we consider b ∈ R K . As for vector scaling; W ∈ R K . Calibrated probabilities for either of these extensions can then be computed as

pi = max k σ(W z i + b).

An even simpler extension is temperature scaling (Guo et al., 2017) which reduces the set of regression parameters to the inverse of a single scalar T > 0 such that

pi = max k σ(z i /T ).

On the other hand, isotonic regression is a common non-parametric processing method that finds the stepwise-constant non-decreasing (isotonic) function f that best fits the data according to a mean-squared loss function i (f (p i ) -y i ) 2 where p i are the uncalibrated probabilities and y i the labels.

The standard approach for extending isotonic regression to the multiclass setting is to break the problem into many binary classification problems (e.g. one-versus-all problems), to calibrate each problem separately, and then to combine the calibrated probabilities (Zadrozny & Elkan, 2002) .

3. ISSUES WITH CALIBRATION METRICS:



3.1. NOT COMPUTING CALIBRATION ACROSS ALL PREDICTIONS:

Expected Calibration Error was crafted to mirror reliability diagrams, which are structured around binary classification such as rain vs not rain (DeGroot & Fienberg, 1983 ) (Figure 1 ). A consequence is that the error metric is reductive in a multi-class setting. In particular, ECE is computed using only the predicted class's probability, which implies the metric does not assess how accurate a model is with respect to the K -1 other probabilities.

We examine increasing the importance of omitted predictions via label noise. As label noise increases, trained models become less confident in their predictions. Secondary and tertiary predictions are correct with higher frequency, and so for many samples the prediction with the correct label isn't included in ECE's measure of calibration error. We find that ECE becomes a worse approximation of the calibration error as relevant predictions not included in the ECE metric become common (Figure 1 ).

In Figure 2 , we observe that calibration error, when measured in a per-class manner, is non-uniform across classes. Guaranteeing calibration for all class probabilities is an important property for many applications. For example, a physician may care deeply about a secondary or tertiary prediction of a disease. Decision costs along both ethical and monetary axes are non-uniform, and thus clinical decision policies necessitate a comprehensive view of the likelihoods of potential outcomes (e.g., Tsoukalas et al., 2015) . Alternatively, if a true data distribution exhibits high data noise (also known as high aleatoric uncertainty, or weak labels), then we'd like models to be calibrated across all predictions as no true class label is likely.

3.2. FIXED CALIBRATION RANGES:

One major weakness of evenly spaced binning metrics is caused by the dispersion of data across ranges. In computing ECE, there is often a large leftward skew in the output probabilities, with the left end of the region being sparsely populated and the rightward end being densely populated. Sharpness, which is the desire for models to always predict with high confidence, i.e., predicted probabilities concentrate to 0 or 1, is a fundamental property (Gneiting et al., 2007) . (That is, network predictions are typically very confident.) This causes only a few bins to contribute the most to ECE-typically one or two as bin sizes are 10-20 in practice (Guo et al., 2017) .

3.3. BIAS-VARIANCE TRADEOFF:

'Ground truth' is the calibration error we would measure if we could see an infinite number of predictions at a particular confidence level. That would give us a long-run error for every prediction, which we could aggregate into an overall calibration error. Because we don't have that infinite set of reference predictions for every real-valued prediction, we try to approximate that set by using the values that are close together as reference predictions. As we increase the number of datapoints in a bin, we get a more accurate estimate of the calibration in a given range through the added (seen as a reduction in variance). But as the datapoints in the bin start to differ from one another due to the extension of the range, they bias the estimate of the calibration within that bin (we refer to this as bias).

Selecting the number of bins has a bias-variance tradeoff as it determines how many data points fall into each bin and therefore the quality of the estimate of calibration from that bin's range. In particular, a larger number of bins causes more granular measures of calibration error (low bias) but also a high variance of each bin's measurement as bins become sparsely populated. This tradeoff compounds particularly with the problem of fixed calibration ranges, as due to sharpness certain bins have many more data points than others.

3.4. PATHOLOGIES IN STATIC BINNING SCHEMES:

Metrics that depend on static binning schemes like ECE suffer from issues where you can get near 0 calibration error due to overconfident and underconfident predictions overlapping in the same bin. These cancellation effects may not always be extreme enough to push an incredibly uncalibrated model to 0 calibration error, but it can be difficult to judge when using a static metric whether differences between models or techniques are due to a true improvement in the model's calibration or whether improved calibration is due to cancellation.

Code in the appendix (Figure 5 ) demonstrates this effect.

4. CHALLENGES IN CALIBRATION:

Before describing new metrics for calibration, we first outline broad challenges with designing such metrics.

4.1. GROUND TRUTH & COMPARING CALIBRATION METRICS:

There are many challenges in measuring a network's calibration, beginning with the absence of ground truth. In principle, one can limit comparisons to controlled, simulated experiments where ground truth is available by drawing infinitely many samples from the true data distribution. However, even with ground truth, any estimation property such as bias remains difficult to compare across estimators, as "ground truth error" is multi-valued and estimators may make measurements for different elements of these values. Specifically, calibration is a guarantee for all predicted probabilities p ∈ [0, 1] and class labels y ∈ {0, . . . , K -1} (see also the calibration function of Vaicenavicius et al. (2019) ). An estimator may have lower bias and/or variance in estimating the error for specific ranges of p and y but higher bias and/or variance in other ranges.

The other major challenge with respect to ground truth is that due to the differences in their assumptions, different calibration metrics approximate different overall calibration values. ECE doesn't look at all predictions, and is estimating a different value than SCE. The ACE vs SCE comparison is possible with respect to the probabilities they look at, but as the static binning scheme imposes a different bias than the adaptive scheme, it's a challenge to compare static metrics to adaptive metrics.

4.2. WEIGHTING:

Because calibration error is multi-valued, and a desirable metric is scalar-valued, the question of how to weight probability values (where one can see thresholding as a 0/1 weighting on datapoints below / above the threshold) creates a set of differences in calibration error metrics that choose to emphasize different aspects of calibration performance. In many contexts what matters most is the rare event-the network's classifications leading up to an accident, the presence of a planet, or the presence of a rare disease. Knowing the difference between whether a class's true probability is .01 and .001 (the difference between 1 in 100 and 1 in 1000) is both extremely difficult to discern and may be much more relevant than a difference between .3 and .301, which these calibration metrics would treat as equivalent. In these contexts, weighting the ends of the interval close to 0 and 1 would be ideal.

Additionally, in the context of out-of-distribution detection, we would prefer to be well-calibrated in the middle of the spectrum, whereby a prediction of .5 (high uncertainty) is more likely to happen.

5. NEW CALIBRATION METRICS:

Each calibration error metric is evaluated on each class, independently. The results for each class are averaged.

5.1. MULTICLASS & STATIC CALIBRATION ERROR:

We first introduce Static Calibration Error (SCE), which is a simple extension of Expected Calibration Error to every probability in the multiclass setting. SCE bins predictions separately for each class probability, computes the calibration error within the bin, and averages across bins:

SCE = 1 K K k=1 B b=1 n bk N |acc(b, k) -conf(b, k)| .

Here, acc(b, k) and conf(b, k) are the accuracy and confidence of bin b for class label k, respectively; n bk is the number of predictions in bin b for class label k; and N is the total number of data points.

Unlike ECE, assuming infinite data and infinite bins, SCE is guaranteed to be zero if only if the model is calibrated.

5.2. ADAPTIVITY & ADAPTIVE CALIBRATION ERROR:

Adaptive calibration ranges are motivated by the bias-variance tradeoff in the choice of ranges, suggesting that in order to get the best estimate of the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions). This leads us to introduce Adaptive Calibration Error (ACE). uses an adaptive scheme which spaces the bin intervals so that each contains an equal number of predictions.

In detail, ACE takes as input the predictions P (usually out of a softmax), correct labels, and a number of ranges R.

ACE = 1 KR K k=1 R r=1 |acc(r, k) -conf(r, k)| .

Here, acc(r, k) and conf(r, k) are the accuracy and confidence of adaptive calibration range r for class label k, respectively; and N is the total number of data points. Calibration range r defined by the N/R th index of the sorted and thresholded predictions.

5.2.1. SHARPNESS & ADAPTIVE CALIBRATION ERROR:

A major motivation for the Adaptive Calibration Error was the realization that the vast majority of predictions made by deep learning models are close to 1 or 0. A static binning scheme will group almost all predictions into two bins, one at each end, with very sparse predictions falling in the bins that are in between (Figure 3, Left) . ACE adapts to focus on the areas where predictions are densest (Figure 3 , Right).

6. INTERPRETABILITY & ANALYSIS 6.1 INTERPRETATION OF CALIBRATION ERROR METRICS:

The natural interpretation for both metrics that account for every prediction made by a classification model (SCE and ACE) is that the calibration error is the average error of all predictions made by the mode. The natural interpretation for ECE, in contrast, is that its calibration error is the average error of the highest confidence prediction made by the model.

6.2. CRITICISMS & LIMITATIONS OF ADAPTIVE CALIBRATION METRICS:

These adaptive calibration metrics can create excessively large ranges if there is very sparse output in a region. In those cases, it may be better to have a higher variance estimate of the calibration, but only compare datapoints that are closer together (dropping datapoints that are farther away). This level of granularity can be replicated with an adaptive scheme that has a very large number of ranges, but that will come at the cost of easy interpretability and will increase the within-range variance across all ranges, not just the sparse regions. This scheme does not accomplish transfer smoothly between all related predictions, although ensembling / hierarchical calibration ranges could approximate that smoothness. This limitation does suggest an smoother metric, akin to our algorithm that leverages dynamic programming to look at very many range schemes simultaneously & efficiently.

With a static bin calibration metric it can be straightforward to compare the outputs of two very different models through a reliability diagram. Due to the variability of their binning scheme based on the density of the model's predictions, adaptive calibration metrics make it more difficult to compare two models side-by-side in a reliability diagram, as their points fall in different regions depending on the bin locations that are selected.

6.3. HYPERPARAMETER ANALYSIS:

Adaptive calibration metrics are more sensitive to the binning hyperparameter. Empirically, there are larger changes in calibration error due to bin count choice than static metrics. This variance decreases as the bin count rises, leveling off at 50 in our experiments. We suggest using ACE with 50+ bins for models trained on datasets akin to CIFAR or ImageNet.

7. EXPERIMENTS:



7.1. RECALIBRATION EXPERIMENTS:

We trained a ResNet-110 model (He et al., 2016) on CIFAR-100 (Krizhevsky & Hinton, 2009) . We then estimated the regression parameters, for each postprocessing method, on the validation held-out set. Finally, we used the parameter estimates to compute calibrated probabilities on the test set. The results are reported in Table 3 . The same experiment with a ResNet-50 model on ImageNet 2012 is reported in Table 4 and echoes the findings of Table 3 .

Table 3 shows that temperature scaling outperforms the other 3 methods on ECE as reported in (Guo et al., 2017) . However, temperature scaling falls short in comparison to vector scaling on the SCE and ACE metrics. In fact, the relative gap on SCE is the largest, among all metrics and across methods. This is unsurprising given that SCE is a multiclass extension of ECE which considers all probabilities, instead of the top one. Vector scaling typically does best.

For example, temperature scaling (Guo et al., 2017) ( Platt et al., 1999) . The question of whether these scaling techniques effectively minimize more sophisticated error metrics is an important standard for their efficacy.

In fact, by design of having only a single scalar parameter which uniformly scales predicted probabilities across all classes, temperature scaling is likely perform worse when accounting for, for example, calibration errors across all predictions. We hypothesize that this would be the case with respect to adaptive calibration metrics (e.g. ACE) as well as calibration metrics that account for the probabilities of all classes instead of that of only the top predicted class (e.g. SCE).

When designing post-processing methods, not accounting for the properties detailed in Section 3 during evaluation can lead to misleading conclusions about a post-processing method's success.

7.2. LABEL NOISE:

We train softmax regression (multinomial logistic regression) models at 40 levels of label noise from 0.0 to 0.05. The label noise value corresponds to the percentage of labels that are randomly assigned, where the correct label is included as one of the possible random assignments.

One way to measure the importance of a sample is its confidence. We look at the fraction of important predictions ECE omits by setting a threshold confidence (.01 in our visualized example, see Figure 1 Top Right) and observe that this fraction rises with the difficulty of the modeling task (measured by model accuracy and mean maximum confidence, see Figure 1 Bottom). This result holds for a wide range threshold levels. 

Method:



A APPENDIX:

A.1 THRESHOLDING & THRESHOLDED ADAPTIVE CALIBRATION ERROR One initial challenge is that the vast majority of softmax predictions become infinitesimal (Figure 3 , Top Left). These tiny predictions can wash out the calibration score, especially in the case where there are many classes, where a large proportion of them model's predictions correspond to an incorrect class. One response is to only evaluate on values above a threshold . Mathematically, TACE is identical to ACE, with the only difference being that TACE is only evaluated on values above . These predictions overlap with the predictions evaluated by ECE (all maximum values per datapoint), leading them to have similar reactions to recalibration methods. The softmax here has 450 predictions at .52 (ECE only looks at the maximum value) and evaluates all of them to be incorrect, despite being predicted with high confidence. The softmax also has 550 predictions at .58, all evaluated to be correct, though they're predicted with low confidence (relative to 1 for correct and 0 for incorrect). Despite this extreme miscalibration, the expected calibration error is exremely low ( 0.003).

:



Figures and Tables:

Figure1: Top Left: Calibration measured through three metrics on MNIST with randomly assigned labels, with the fraction of randomly assigned labels on the x axis (all others labels are correct). Models (here, multinomial logistic regression) are re-trained at each noise level. The base level of calibration error will be different, as ACE and SCE account for all predictions, while ECE only accounts for the prediction with maximum confidence for each sample. These predictions have different average calibration errors.Top Right: Often, important predictions (confidence > 0.1) are not captured in the calibration error by ECE. As models trained with more label noise exhibit more uncertainty, that number increases as the amount of label noise increases. Bottom: As label noise increases, model output certainty and accuracy decrease.

Figure 2: Left: Per-class Adaptive Calibration Error (ACE) for a trained 110-layer ResNet on the CIFAR-100 test set. We observe that calibration error is non-uniform across classes, which is difficult to express when measuring error only on the maximum probabilities. Right: Reliability Diagrams are an interpretable visualization of calibration error. This is an example of the reliability of a slightly overconfident CNN trained on Fashion-MNIST.

Figure 3: Left: Model sharpness, where the vast majority of confidence scores are very near 1 or very near 0, motivates ACE. This class-conditional histogram of confidence scores on 12,000 samples from Fashion-MNIST demonstrates that sharpness. Right: Binning scheme chosen by ACE measured at each step of training on Fashion-MNIST. Each bin measuring the calibraion error of datapoints within a particular range has a lower bound to the bin and an upper bound. As the model's predictions become sharper, the chosen bins adapt to focus on regions with a high concentration the predictions. Each color represents the lower bound of the nth largest bin, for 100 bins.

Figure 4: Top Left: Lower bounds of calibrations ranges over the course of training for adaptive calibration error on Fashion-MNIST, focusing almost entirely on small ranges and motivating thresholding. Top Right: On the MNIST training set with thresholding, so few values are small that the bottom of the lowest range often spikes to .99 and higher due to every datapoint being fit. Bottom Left: ACE on Fashion-MNIST validation with 100 calibration ranges. Bottom Right: Thresholded adaptive calibration with 50 calibration ranges over the course of training on Fashion-MNIST's validation set.

Figure5: This python code (with numpy imported as np, and with an expected calibration error function loaded in the environment) implements a pathology in ECE which will return 0 calibration error by including overconfident and underconfident values in the same bin. The softmax here has 450 predictions at .52 (ECE only looks at the maximum value) and evaluates all of them to be incorrect, despite being predicted with high confidence. The softmax also has 550 predictions at .58, all evaluated to be correct, though they're predicted with low confidence (relative to 1 for correct and 0 for incorrect). Despite this extreme miscalibration, the expected calibration error is exremely low ( 0.003).

has been shown to effectively minimize expected calibration error better than alternative techniques such as isotonic regression and Platt scaling ECE, TACE, SCE, and ACE (with 15 bins) on a ResNet-110 applied to CIFAR-100 before calibration, and after the application of post-processing methods. The best recalibration method depends on the metric, which motivates its study.

Table content: <table><tr><td>Method</td><td>ECE</td><td>SCE</td><td>ACE</td></tr><tr><td>Uncalibrated</td><td colspan="3">19.64% 0.41% 0.13%</td></tr><tr><td colspan="2">Temp. Scaling 2.16%</td><td colspan="2">0.06% 0.06%</td></tr><tr><td colspan="2">Vector Scaling 2.27%</td><td colspan="2">0.04% 0.01%</td></tr><tr><td colspan="4">Matrix Scaling 12.11% 0.26% 0.26%</td></tr><tr><td>Isotonic Regr.</td><td colspan="3">17.85% 0.35% 0.12%</td></tr><tr><td>Method</td><td>ECE</td><td>SCE</td><td>ACE</td></tr><tr><td>Uncalibrated</td><td>6.63%</td><td>0.02%</td><td>0.015%</td></tr><tr><td colspan="2">Temp. Scaling 5.42%</td><td>0.01%</td><td>0.001%</td></tr><tr><td colspan="2">Vector Scaling 1.44%</td><td colspan="2">0.002% 0.004%</td></tr><tr><td colspan="2">Matrix Scaling 5.06%</td><td>0.01%</td><td>0.001%</td></tr><tr><td>Isotonic Regr.</td><td colspan="2">3.474% 0.01%</td><td>0.000%</td></tr></table>

ECE, SCE, and ACE (with 15 bins) on a ResNet-50 applied to ImageNet before calibration, and after the application of various extensions to Platt scaling and Isotonic regression. These percentages across metrics are not directly comparable.

Table content: <table/>

ECE, TACE, SCE, and ACE (with 15 bins) on a ResNet-110 applied to CIFAR-100 before calibration, and after the application of post-processing methods. The best recalibration method depends on the metric, which motivates its study.

Table content: <table><tr><td/><td>ECE</td><td>TACE</td><td>SCE</td><td>ACE</td></tr><tr><td>Uncalibrated</td><td colspan="4">19.64% 10.03% 0.41% 0.13%</td></tr><tr><td colspan="2">Temp. Scaling 2.16%</td><td>0.52%</td><td colspan="2">0.06% 0.06%</td></tr><tr><td colspan="2">Vector Scaling 2.27%</td><td>0.61%</td><td colspan="2">0.04% 0.01%</td></tr><tr><td colspan="3">Matrix Scaling 12.11% 3.98%</td><td colspan="2">0.26% 0.26%</td></tr><tr><td>Isotonic Regr.</td><td colspan="2">17.85% 2.83%</td><td colspan="2">0.35% 0.12%</td></tr><tr><td>Method</td><td>ECE</td><td>TACE</td><td>SCE</td><td>ACE</td></tr><tr><td>Uncalibrated</td><td>6.63%</td><td>2.51%</td><td>0.02%</td><td>0.015%</td></tr><tr><td colspan="2">Temp. Scaling 5.42%</td><td>2.64%</td><td>0.01%</td><td>0.001%</td></tr><tr><td colspan="2">Vector Scaling 1.44%</td><td>1.25%</td><td colspan="2">0.002% 0.004%</td></tr><tr><td colspan="2">Matrix Scaling 5.06%</td><td>1.98%</td><td>0.01%</td><td>0.001%</td></tr><tr><td>Isotonic Regr.</td><td colspan="3">3.474% 1.862% 0.01%</td><td>0.000%</td></tr></table>

ECE, TACE, SCE, and ACE (with 15 bins) on a ResNet-50 applied to ImageNet before calibration, and after the application of various extensions to Platt scaling and Isotonic regression. These percentages across metrics are not directly comparable. Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Robert Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. Direct uncertainty prediction for medical second opinions. 2018. Seonguk Seo, Paul Hongsuck Seo, and Bohyung Han. Learning for single-shot confidence calibration in deep neural networks through stochastic inferences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9030-9038, 2019. Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B Schön. Evaluating model calibration in classification. arXiv preprint arXiv:1902.06977, 2019. Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694-699. ACM, 2002.

Table content: <table><tr><td>Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.</td></tr><tr><td>On mixup training: Improved calibration and predictive uncertainty for deep neural networks.</td></tr><tr><td>arXiv preprint arXiv:1905.11001, 2019.</td></tr><tr><td>Athanasios Tsoukalas, Timothy Albertson, and Ilias Tagkopoulos. From Data to Optimal Decision</td></tr><tr><td>Making: A Data-Driven, Probabilistic Machine Learning Approach to Decision Support for</td></tr><tr><td>Patients With Sepsis. JMIR Medical Informatics, 3(1):e11, February 2015. ISSN 2291-9694. doi:</td></tr><tr><td>10.2196/medinform.3445. URL http://medinform.jmir.org/2015/1/e11/.</td></tr></table>

Peer review (iclr2020):
Paper summary: This paper proposes two simple extensions to Expected Calibration Error (ECE): 1) SCE which accounts for multiclass classification settings by averaging over all the errors due to all classes (as opposed to error in the top one class only in ECE) and 2) ACE which attempts as distributing the predictions equally across the bins (as opposed to have too few highly populated bins in the interval). Authors evaluated their approach using ResNet-110 on CIFAR100 and ResNet-50 on ImageNet against most common post-training calibration methods. Pros: (+): The paper is well-motivated. (+): The problem is important and has direct real world applications. (+): The idea is simple and viable to improve ECE. Cons that significantly affected my score and resulted in rejecting the paper are as follows: 1 - Experimental setting and evaluations: The biggest drawback in this paper is the experimental setting which is not rigorous enough to show the effectiveness of the proposed metrics due to the following reasons: (a) The first proposed extension (SCE) is too incremental. However, while I am not against simple and effective methods (I have even listed this as a pro above), I think it should be backed up with more thorough experiments and discussions. SCE is supposed to be more effective in case of having more number of classes however authors do not shed light into this properly by taking advantage of comparing their CIFAR100 and ImageNet 1K experiments. They keep emphasizing on the fact that the numbers provided in Table 1 and 2 are not comparable so how is the reader supposed to understand their differences? They also show their metrics give far less error to models than ECE in Table 1 and 2 and yet it is not clear why that is. Not to mention that authors need to state these results are obtained on how many runs and to report stds because the numbers appear too close and are hence not conclusive. (b) Datasets and architectures: Authors have used MNIST and FASHION-MNIST when discussing the shortcomings of ECE. I was wondering what architecture they used for these experiments? The reason I am asking is that it is a known fact (see Guo et al., ICML17) that the simpler the architecture is, the more calibrated its predictions tend to be and vice versa. For instance LeNet 5 appears to be much more well-calibrated than more modern neural networks such as ResNet variants despite being less accurate (Guo et al, 2017). Therefore, it will probably be in their favor to show these issues on a less calibrated model. (c) Can authors please explain why they have used ResNet-110 on CIFAR100 and ResNet-50 on ImageNet? I assume it might be because they intended to compare to (Guo et al, 2017) but there are more experiments there for comparison. Have they also tried any other architecture? It has been shown before that there can be a noticeable difference across different architectures used on a fixed dataset. Authors may want to add the arch effect to their evaluation. 2) The second metric (ACE) while it is well-motivated, it is defined such that it leaves the impression that applying it will involve lots of heuristics as there is no systematic procedure is given. In section 6.3 (where I think could be a section to address the implementation details for this) no effective information is given either expect that authors recommend using 50+ bins for CIFAR and ImageNet without any quantitative support. It is also not clear why authors used 15 bins in their experiments for these datasets in Table 1 and 2 (maybe comparing to the baseline? but they could show both specially if 50+ bins is better for ACE). Moreover, in section 5.2 where it says “the overall calibration error the metric should focus on the regions where the predictions are made (and focus less on regions with few predictions).“, I was wondering if authors have any suggestions on how to identify the predictions with low confidence as coming from in or out-distribution? 2- Structure of the paper, writing, and visualizations: (a) Writing: The paper, in its current form, needs to be thoroughly proofread and reorganized. The text does not read well and is vague in most parts. Authors have spent too much time explaining the drawbacks of the prior work in 4 pages (entire sections 2,3, and 4) only to propose their ideas in section 5.1 and 5.2 in almost half a page. Unfortunately the analysis section (6) acts very poorly in providing a thorough exploration into their method. (b) Dividing each section to too many subsections has hurt the flow as each section is too short and does not provide deep evidence to support its title/subtitle. <<< Note that the followings are less major and are given only to help, and not necessarily part of my decision assessment >>> (c) Results shown in Table 3 and 4 are reporting “TACE” which is technically explained in the Appendix. Apart from the fact that authors need to make sure the Table comes before the references (not in the middle) but more importantly, as a paper submitted to ICLR I would expect it to be self-contained and be able to provide all the details needed. Authors should either move this Table to the appendix or move section (A.1) to the main text. (d) In section 3.4. Authors provide support for their claim on issues with binning scheme with a screenshot of their code in the Appendix. I think it is important to show this effect but in perhaps a table with quantitative results within the main text. (e) Title for Section 5 as the main contribution of this paper should be changed to reflect that. (f) Abstract in its current wording, does not provide absolute no detail into the proposed metrics. Whereas it can, because 1) it encourages the reader to keep reading 2) the method is simple enough to be summarized here. (g) The figures do not meet the conventional scientific standards and have to be significantly improved. Standard deviations are missing. (h) There are also grammar errors and typos, (for example on page 1 paragraph 5, the word *may* should be *many*), for which I have found passing my writing through the free version of Grammarly very helpful in getting rid of most such errors. As a final note, I think this paper can be contributive to the field as it provides a novel and simple extension to a widely used calibration metric. However it needs to be written in a more effective way and be supported by a more rigorous experimental setting. Therefore, I will be willing to change my score if the presented issues will be addressed.