Revisiting the Calibration of Modern Neural Networks
Authors: Matthias Minderer
Josip Djolonga
Rob Romijnders
Frances Ann Hubis
Xiaohua Zhai
Neil Houlsby
Dustin Tran
Mario Lucic
Venue: NeurIPS 2021 Poster (2021)
Keywords: uncertainty
calibration
image classification

Abstract: Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.

Revisiting the Calibration of Modern Neural Networks
Matthias Minderer Xiaohua Zhai
Josip Djolonga Neil Houlsby
Rob Romijnders Dustin Tran
Frances Hubis Mario Lucic
Google Research, Brain Team
fmjlm, lucic)@google.com
Abstract
Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, generations, such as decay calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount pretraining do not fully explain these differences, suggesting that architecture is major determinant of calibration properties.
among the best calibrated. Trends observed in prior model
Introduction
Neural networks, especially vision models, are increasingly used in safety-critical applications such autonomous driving (Bojarski al., 2016), medical diagnosis (Esteva al., 2017: Jiang al., 2012), and meteorological forecasting (Sønderby et al., 2020). For such applications, is essential that model predictions are not just accurate, but also well calibrated. Model calibration refers the accuracy with which the scores provided by the model reflect its predictive uncertainty. For example, in medical application, we would like to defer images for which the model makes low-confidence predictions to physician for review (Kompa et al., 2021). Skipping human review due to confident, but incorrect, predictions, could have disastrous consequences.
While intense research and engincering effort has focused on improving the predictive accuracy of models, less attention has been given model calibration. In fact, over the last few years, there have been many reports that calibration of modern neural networks can be surprisingly poor, despite the advances in accuracy (e.g. Guo et al. 2017: Lakshminarayanan
al. 2017; Malinin & Gales 2018; Thulasidasan et al. 2019; Hendrycks et al. 2020b; Ovadia et al. 2019; Wenzel et al. 2020; Havasi et al. 2021; Rahaman & Thiery 2020; Leathart & Polaczuk 2020). Some works suggest trend for larger, more accurate models to be worse calibrated (Guo et al., 2017).
These concerns are more relevant than ever, since the architecture size, amount of training data, and computing power used by state-of-the-att models continue to increase. At the same time, rapid advances in model architecture (Tolstikhin et al., 2021; Dosovitskiy et al., 2021) and training approaches (Chen et al., 2020; Mahajan et al., 2018; Radford et al., 2021) raise the question whether past results calibration, largely obtained standard convolutional architectures, extend current state-of-the th-the-art models. Since model advances are quickly translated to real- world, safety-critical applications (e.g. Mustafa et al. 2021), there is urgent need to re-assess the calibration properties of current state-of-the-art models.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Contributions. To address this need, we provide a systematic comparison of recent image classifi- cation models, relating their accuracy, calibration, and design features. We find that:
1. The best current models, including the non-convolutional MIP-Mixer (Tolstikhin et al., 2021) and Vision Transformers (Dosovitskiy et al., 2021), well calibrated compared past models and their performance is more robust to distribution shift.
In-distribution calibration slightly deteriorates with increasing model size, but this is out- weighed by a simultaneous improvement in accuracy.
Under distribution shift, calibration improves with model size, reversing the trend seen in- distribution.
Accuracy and calibration are correlated under distribution shift, such that optimizing for accuracy may also benefit calibration.
Model size, pretraining duration, and pretraining dataset size cannot fully explain differences in calibration properties between model families.
Our results suggest that further improvements in model accuracy will continue to benefit calibration. They also hint at architecture as an important determinant of model calibration. We provide code and a large dataset of calibration measurements, comprising 180 distinct models from 16 families, each evaluated on 79 ImageNet-scale datasets and 28 metric variants."
2 Related Work
Measures of model calibration. The losses that are commonly used to train classification models, such as cross cross-entropy and squared error, are proper scoring rules (Gneiting et al., 2007) and are therefore guaranteed to yield perfectly calibrated models at their minimum-in the infinite-data limit. However, in practice, due to model mismatch and overfitting, even losses based on proper scoring rules may result in poor model calibration. Miscalibration is commonly quantified in terms of Expected Calibration Error (ECE; Naeini et al. 2015), which measures the absolute difference between predictive confidence and accuracy. We focus on ECE because it is a widely used and accepted calibration metric. Nevertheless, difficult because estimators can be strongly biased and many estimator variants exist (Nixon et al., 2019; Roelofs et al., 2020; Vaicenavicius al., 2019; Gupta et al., 2021). Section 5 discusses these issues and our approaches to mitigate them.
well understood that estimating ECE accurately is
Alternatives to ECE include likelihood measures, Brier score (Brier, 1950), Bayesian methods (Gel- man et al., 2013), and conformal prediction (Shafer & Vovk, 2008). Further, model calibration can be represented visually with reliability diagrams (DeGroot & Fienberg, 1983). Figure and Appendix F provide likelihoods, Brier scores, and reliability diagrams for our main analyses.
Empirical studies of model calibration. There have been many recent empirical studies on the robustness (accuracy under distribution shift) of image classifiers (Geirhos et al., 2019; Taori et al., 2020; Djolonga et al., 2020; Hendrycks et al., 2020a). Several works have also studied calibration. Most notable is Guo et al. (2017), who found that "modern neural networks, unlike those from decade ago, are poorly calibrated", that larger networks tend to be calibrated worse, and that "miscalibration worsen[s] even as classification error is reduced." " Other works have corroborated some of these findings (e.g., Thulasidasan et al. 2019; Wen et al. 2021). This line of work suggests a trend that larger models are worse calibrated, which would have major implications for research toward bigger models and datasets. We show that for more recent models, this trend is negligible in-distribution and in fact reverses under distribution shift.
Ovadia et al. (2019) empirically study calibration under distribution shift and provide a large compar- ison of methods for improving calibration. They report that both accuracy and calibration deteriorate with distribution shift. While observe the same trend, we find that the calibration of some recent model families decays so slowly under distribution shift that the decay in accuracy is likely more relevant in practice (Section 4.3).
Ovadia et al. also find that, across methods for improving calibration, improvements on in-distribution data do not necessarily translate to out-of -ott----istriutiion data. This finding may suggest that there is little correlation between in-distribution and outtof-distribution calibration in general. However, our
Available at http:p//ggthaacocooooegeeerrerrr rerare/oooererrcrrrroooottteeeee robustness_meett httpsttssssnnttteeee/pp////trrrs www.:iaiiii................
http://wwwoooooeeeeeeo/oooototte wttps/seeeeeereeeeeeeeeeeeeeeeee /

---

results show that, across model architectures, the models with the best in-distribution calibration are also the best-calibrated on a range of out-of-distrbutioo benchmarks. The important implication of this result is that designing models based on in-distribution performance likely also benefits their out-of-distribution performance.
Improving calibration. Many strategies have been proposed to improve model calibration such as post-hoc rescaling of predictions (Guo al., 2017), averaging multiple predictions (Lakshmi- narayanan et al., 2017; Wen et al., 2020), and data augmentation (Thulasidasan et al., 2019; Wen al., 2021). Here, we focus on the intrinsic calibration properties of state-of satee-of-teeeatee ff-the-ar model families, rather than methods to further improve calibration.
As baseline top of model's intrinsic calibration properties, study temperature scaling (Guo et al., 2017). It is effective in improving calibration and so simple that it can be applied in many cases minimal additional cost, in contrast many more sophisticated methods. Temperature scaling re-scales a model's logits by a single parameter, chosen to optimize the model's likelihood on a held-out portion of the training data. This temperature factor changes the model's confidence, i.e., whether the model predictions are on average too certain (overconfident), optimally confident, or too uncertain (underconfident). The classification accuracy of the model is not affected by temperature scaling. A large fraction of model miscalibration is typically due to average over- or underconfidence, e.g. due to suboptimal training duration (Guo et al.. 2017). By normalizing a model's 's confidence, temperature scaling not only improves calibration, but also removes a primary confounder that can hide trends in calibration between models (see Section 4.2 and Appendix D). Therefore, we study both unscaled and lemperature-scaled predictions in the paper.
3 Definitions and Notation
We consider the multi-class classification problem, as analyzed by Brücker (2009), where we observe variable X and predict categorical variable Y E (1,2....,kk, We model our predictor as function that maps every input instance X to a categorical distribution over k labels, represented using
vector f(X) belonging to the (k- 1)-dimensional simplex A {p E [0, 1}&h Ey= Py = =}}.
Intuitively, a model f is well-calibrated if its output truthfully quantifies the predictive uncertainty. For example, if take all data points for which the model predicts [f()]], 0.3, we expect 30% of them to indeed take on the label y. Formally, the model f is said to be calibrated if (Bröcker, 2009)
Vp E A: P(Y = y f(X) = p) = Py-
(I)
We will focus on a slightly weaker, but more practical condition, called top-label or argmax calibration (Kumar et al., 2019; Guo al., 2017). This requires that the above holds only for the most likely label, i.e., Vp" E [0,
P(Y arg maxp max f(X) = pp) =p"
where the max and arg max act coordinate wise..
The most common measure of the degree of miscalibration is the Expected Calibration Error (ECE), which computes the expected disagreement between the two sides of eq. (2)
E Ip - E|Y E arg max f(X) max f(X) = -p"1]-
Unfortunately, eq, (3) cannot be estimated without quantization as it conditions on a null event. Hence, one typically first buckets the predictions into m bins Bı,..., Bm based on their top predicted probability, and then takes the expectation over these buckets. Namely, if we are given a set of n ii.d. samples (I1, 91)) yi),..., (In, V/.) distributed P(X, Y), then assign each j E (1,.... n] to a bucket B, based on max f(zz). Then, we compute in each bucket B, the confidence(B.) [B. Ejen, max fxxz) and the accuracy (B,) = 1 ΣjEB, [yv E arg max / f(x;) ). where [|]] the Iverson bracket. Finally, we construct an estimator by taking the expectation over the bins
[B.
172 [.l
BCE
|accuracy(BB) cBnfidence(B:)).
(4)
In Section 5 we discuss the statistical properties of this estimator, possible pitfalls, and several mitigation strategies.
3

---

Sample frac.
MLP-
VIT-H/14
BiT-L (8152x4)
WSL 32x48d
SimCLR-4x
0.09
1.0
Ino recallbration)
0.08
0.5
0.07
BIT
0.06
CUP
0.0
1.0
0.05
Aleodded
Gua
0.8
0.04
0.6
0.2
0.0
0.1
0.2
0.3
0.4
0.5
1.0
1.0
1.0 (unscaled)
ImageNet error
Confidence (unscaled)
Confidence (unscaled]
Confidence (unscaled)
Confidence (unscaled)
Confidence
Figure 1: Some modern neural network families are both highly accurate and well-calibrated, Left: Expected calibration error (ECE) vs. classification error IMAGENET for state-of-the-arr image classification models. Marker size indicates relative model size within its family. Points labeled "Guo et al." the values reported for DenseNet-161 and ResNet-152 in Guo et al. (2017). Right: Confidence distribution (top row) and reliability diagrams (bottom row) for some of the models.
Empirical Evaluation
4.1 Experimental Setup
Model families. In this study, we consider range of recent and some historic ofate-of-the-at image classification models. Our selection of models covers convolutional and non-convolutional architectures, as well as supervised, weakly supervised, unsupervised and zero-shot training. We follow the original publications in naming the model variants within each family (e.g. different model sizes). See Appendix A. for detailed description of all used models.
1. MLP-Mixer (Tolstikhin et al., 2021) is based exclusively on multi-layer perceptrons (MLPs) and is pre-trained large supervised datasets.
2. ViT (Dosovitskiy et al., 2021) processes images with a transformer architecture originally designed for language (Vaswani al., 2017) and is also pre-trained on large supervised datasets.
3. BiT (Kolesnikov et al., 2020) is ResNet-based architecture (He et al., 2016). It is also pre-trained on large supervised datasets.
4. ResNext-WSS WSL (Mahajan et al., 2018) is based on the ResNeXt architecture and trained with weak supervision from billions of hashtags on social media images.
5. SimCLR (Chen et al., 2020) is ResNet, pretrained with an unsupervised contrastive loss.
CLIP (Radford et al., 2021) is pretrained on raw text and imagery using a contrastive loss.
7. AlexNet (Krizhevsky et al., 2012; Krizhevsky, 2014) was the first convolutional neural network to win the ImageNet challenge.
All models are either trained or fine-tuned on the IMAGENET training set, except for CLIP, which makes zero-shot predictions using IMAGENET class names queries.
Datasets. We evaluate accuracy and calibration on the IMAGENET validation set and the following out-of-distribution benchmarks using the Robustness Metrics library (Djolonga al., 2020):
IMAGENETV2 (Recht et al., 2019) is new IMAGENET test set collected by closely following the original IMAGENET labeling protocol.
IMAGENET-C (Hendrycks Dietterich, 2019) consists of the images from IMAGENET, modified with synthetic perturbations such as blur, pixelation, and compression artifacts at range of severities.
3. IMAGENET-R (Hendrycks al., 2020a) contains artificial renditions of IMAGENET classes such as art, cartoons, drawings, sculptures, and others.
4. IMAGENET-A (Hendrycks et al., 2021) contains images that are classified belonging to IMAGENET classes by humans, but adversarially selected to be hard to classify for ResNet50 trained on IMAGENET.
For the post-hoc recalibration of models, we reserve 20% of the IMAGENET validation set (randomly sampled) for fitting the temperature scaling parameter. All reported metrics are computed on the remaining 80% of the data. For evaluations on IMAGENET-C, we also exclude the 20% of images that are based the IMAGENET images used for temperature scaling.

---

Calibration metric. Throughout the paper. we estimate ECE using equal-mass binning and 100 bins. Appendix E shows that our results hold for other ECE variants and are consistent with the Brier score and model likelihood.
4.2 In-Distribution Calibration
We begin by considering ECE on clean IMAGENET images (referred to as in-distribution). Figure 1 shows in-distribution ECE and reliability diagrams before any recalibration of the predicted prob- abilities. We find that several recent model families (MLe-Mixer. ViT, and BiT) are both highly accurate and well-calibrated compared to prior models, such as AlexNet or the models studied by Guo et al. (2017). This suggests that there may be no continuing trend for highly accurate modern neural networks to be poorly calibrated, as suggested previously (Guo et al., 2017: Lakshminarayanan et al., 2017; Malinin & Gales, 2018: 8; Thulasidasan et al., 2019; Hendrycks et al., 2020b; Ovadia et al., 2019; Wenzel et al., 2020; Havasi et al., 2021; Rahaman & Thiery, 2020; Leathart & Polaczuk, 2020). In addition, we find that a recent zero-shot model, CLIP, is well-calibrated given its accuracy.
Temperature scaling reveals consistent properties of model families. The poor calibration of past models can often be remedied by post-hoc recalibration such as temperature scaling (Guo et al., 2017), which raises the question whether difference between models remains after recalibration. We find that the most recent architectures are better calibrated than past models even after temperature scaling (Figure 2, right).
More generally, temperature scaling reveals con- sistent trends in the calibration properties be-
Unscaled
Temperature-scaled
0.09
tween families that are obscured in the unscaled data by simple over- underconfidence mis- calibration. Before temperature scaling (Fig- ure 2, left), several families overlap in their ac- curacy/calibration properties (MLP-Mixer, ViT, BiT). After temperature scaling (Figure 2, right), clearer separation of families and consistent trends between accuracy and calibration within each family become apparent. Notably, temper- ature scaling reconciles our results for BiT (a ResNet architecture) with the results reported by Guo et al. for ResNets trained on IMAGENET. Furthermore, models pretrained without addi- tional labels (SimCLR) or with noisy labels (ResNeXt- WSLL) tend to be calibrated worse for a given accuracy than ResNets trained with supervision (BiT and the models studied by Guo et al.). Finally, non-convolutional model families like MLP-Mixer and VIT can perform just as well, if not better, than convolutional ones.
0.08
NLAM
0.07
Reahvaz WSL Sincut
0.06
0.05
Alashet Gue
0.04
0.03
0.02
0.01
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
ImageNet error
ImageNet error
Figure 2: Temperature scaling reveals consistent properties of model families. Left: ECE vs. clas- sification error as in Figure 1. Right: ECE after applying temperature scaling.
Differences between families are not explained by model size or pretraining amount. We next attempt to disentangle how the differences between model families affect their calibration properties. We focus on model size and amount of pretraining, both important trends in state of-the-art models.
We first consider model size. Prior work has suggested that larger neural networks are worse calibrated (Guo et al., 2017). We also find that within most families, larger members tend to have higher calibration error (Figure 2, right). However, at the same time, larger models have consistently lower classification error. This means that each model family occupies different Pareto set in the tradeoff between accuracy and calibration. For example, our results suggest that, at any given accuracy, ViT models are better calibrated than BiT models. Changing the size of BiT model cannot move it into the Pareto set of ViT models. Model size can therefore not fully explain the intrinsic calibration differences between these model families.?
We next consider model pretraining. Many current stat-oofthe-ar image models use transfer learning, in which model is pre-trained a large dataset and then fine-tuned to the task of interest (Kolesnikov et al., 2020; Chen et al., 2020; Xie et al., 2020). With transfer learning, large data sources can be
This relationship between model size, accuracy and calibration holds for all families we study except ResNeXt- WSL, for which increasing model sizes improves both accuracy and calibration. While investigating this difference was out of the scope of this work, it may be promising direction for future research.
5

---

exploited to train the model, even if little data are available for the final task. To test how the amount of pretraining affects calibration, we compare BiT models pretrained on IMAGENET (1.3M images), IMAGENET-21K (12.8M images), or JFT-300 (300M images; Sun et al. 2017).
More pretraining data consistently increases ac- curacy, especially for larger models. It has no
BiT-ImageNet
BiT-lmageNet21k
BiT-JFT
0:09
0:008
consistent effect on calibration (Figure 3). In particular, after temperature scaling, ECE is essentially unchanged across this 300-fold in-
0.07
00.06
0.05
0.04
0.003
crease in pretraining dataset size (e.g. for BiT- R50x1 pretrained on IMAGENET, IMAGENET- 21K and JFT-300, the ECEs are 0.0185, 0.0182, 0.0185, respectively; for BiT-R101x3, they are 0.0272, 0.0311, 0.0236; Figure 3, bottom).
0.02
5.001
0.00
0.05
(temp comneerauueee00
0.04
ECE
0.03
0.02
Therefore, regardless of the pretraining dataset, BiT always remains Pareto-dominant over Sim- CLR and Pareto-dominated by ViT and MLP- Mixer in our experiments.
0.01
0.00
0.1
0.2
0.3
0.1
0.2
0.3
0.1
0.2 0.3
ImageNet
ImageNet
ImageNet
Model family
The BiT models compared in Figure 3 differ in both the amount of pretraining data and the duration of pretraining (see Kolesnikov et al. (2020) for details). To further disentangle these variables, we trained BiT models on varying numbers of pretraining examples while holding the number of training steps constant, and vice versa. We find that pretraining dataset size has no significant effect on calibration, while pre- training duration only shifts the model within its accuracy/calibration Pareto set (longer-trained models more accurate and worse calibrated; Figure 10). These results suggest that pretrain- ing alone cannot explain the differences between model families that we observe.
MLP-Mixer
SiMCLR
Gue et
Figure 3: Family differences are not fully ex- plained by the amount of pretraining. Each column shows ECE vs. classification error on ImageNet for BiT models pre-trained with a different dataset: IMAGENET (1.3M images), IMAGENET- 2 21K (12.8M images), or JFT-300 (300M images). The values for other models provided for reference in light shading (same val- in Figure 2). Note how all BiT models remain in the same relative location between ViT and SimCLR across 300-fold difference in pre- training data size.
In summary, our results show that some modern neural network families combine high accuracy and state-of-the-an calibration on in-distribution data, both before and after post-hoc recalibration by temperature scaling. In Figure 8 and Appendices E and F, we show that these results generally hold for other measures of model calibration (other ECE variants, Brier score, and model likelihood). Our experiments further suggest that model size and pretraining amount do not fully explain the intrinsic calibration differences between model families. Given that the best-calibrated families (MLP-Min -Mxier speculate that model architecture, and in particular its spatial
and ViT) are non-convolutional, inductive bias, play an important role.
4.3 Accuracy and Calibration Under Distribution Shift
For safety-critical applications, the model should produce reasonable uncertainty estimates not just in-distribution, but also under distribution shifts that were not anticipated at training time. We first assess out-of-distribution calibration on the IMAGENET-C dataset, which consists of images that have been synthetically corrupted at five different severities. As expected, both classification and calibration error generally increase with distribution shift (Figure 4; Ovadia et al. 2019; Hendrycks & Dietterich 2019). Interestingly, this decay in calibration performance is slower for MLP-Mixe and ViT than for the other model families, both before and after temperature scaling.
Regarding the effect of model size on calibration, we observed some trend towards worse calibration of larger models on in-distribution data. However, the trend is reversed for most model families as we move out of distribution, especially after accounting for confidence bias by temperature scaling (note positive slope of the gray lines at high corruption severities in Figure 4, bottom row). In other words, the calibration of larger models is more robust to distribution shift (Figure 5).

---

MLP-Mixer
VIT
BiT
SimCLR
ResNeXt WSL
(temperature (rre-scaled) (no recalibration)
0.25
0.20
ECE
0.15
0.10
0.05
0.00
0.10
ECE
0.05
0.00
0.0
0.2 0.4 0.6 0.8
0.0 0.2 0.4 0.6 0.8
0.0 0.2
0.4 0.6 0.8
0.0 0.2 0.4 0.6 0.8
0.0 0.2 0.4 0.6 0.8 ImageNet-C
ImageNet-C error
ImageNet-C
ImageNet-C
ImageNet-C
ImageNet-C corruption severity
Figure 4: Calibration and accuracy on IMAGENET-C before (top) and after (bottom) temperature scaling IMAGENET. Severity 0 refers to the clean IMAGENET test set; marker size indicates relative model size within its family (see Table for model details). The calibration of some recent model families, e.g. MLP-Mixer and VIT, is more robust to distribution shift than past models.
We next consider to what degree the insights from in-distribution and IMAGENET-C calibration transfer to natural out-of-diitributtinn data. Previous work on IMAGENET-C C. suggests that, when comparing recalibration methods, better in-distribution calibration and accuracy do not usually predict better calibration under distribution shift (Ovadia al., 2019). Here, comparing model families, find that the performance on several natural out-of-distribution datasets largely consistent with that IMAGENET (Figure 6). In particular, models that are Pareto-optimal (i.e. other model is both more accurate and better calibrated) on IMAGENET remain Pareto-optimal on the OOD datasets. Further, we observe a strong correlation between accuracy and calibration the OOD datasets. This relationship is consistent across models within a family and across datasets, over a wide range of accuracies (Figure 11).
These results suggest that larger and more accu- rate models, and in particular MLP-Mixer and ViT, can maintain their good in-distribution cal- ibration even under severe distribution shifts. Based on the observed relationship between cal- ibration and accuracy, that good calibration on in-distribution data (and anticipated distribution shifts) generally trans- lates into good calibration on unanticipated out- of-distribution data, similar what has been observed for accuracy (Djolonga et al., 2020).
MLP-Mixer
VIT
BiT
Classification error
0.2
la laggst
reasonably hope
ECE to largest modell
0.02
0.00
4.4 Relating Accuracy and Calibration Within Model Families
Corruption severity
Corruption
Corruption severity
severity
Our data suggest that most model families lie on different Pareto sets in the accuracy/calibration space, which establishes a clear preference be- tween families. We next consider how pare individual models within a family (or more specifically, within Pareto set), where model is more accurate but worse calibrated, and the other is less accurate but better calibrated, Which model should practitioner choose for safety-critical application?
Figure 5: Classification error and ECE for the top three families on IMAGENET-C, relative to the largest model variant in each family. As distribu- tion shift increases, both errors tend to increase more slowly for larger models. Also note that changes in ECE are much smaller than changes in classification error.
7

---

ImageNet-v2
ImageNet-R
ImageNet-A
0.14
0.3
0.6
recalibration
0.12
0.4
ECE
0.08
0.06
0.04
*
0.1
0.2
Model family
MLP-Mixer
0.02
VIT
0.00
0.0
0.0
BIT
ResNeXt WSL
(temperatrre- sca-cee
SimCLR
0.12
CLIP
0.10
0.2
0.4
AlexNet
ECE
0.08
0.04
0.1
0.2
0.02
0.1
0.2
0.3
0.4
0.5
0.2
0.4
0.6
0.2
0.4
0.6
0.8
1.0
Classification error
Classification error
Classification error
Figure Calibration and accuracy before (top row) and after (bottom row) temperature scaling on out-of-distribution benchmarks. Marker size indicates relative model size within its family. IMAGENET-R and IMAGENET-A use a reduced subset of 200 classes; we follow literature and select the subset of the model logits for these classes before evaluation. Out-of cuttof--isttibuttin calibration tends to correlate with in-distribution calibration (Figure 1) and out-of-distribution accuracy.
The
depends on the cost structure of the specific application (Heránndez-Orall et al., 2012). As an example, consider the scenario of selective prediction, which is common in med- ical diagnosis. In this task, one can choose to ignore the model prediction ("abstain") at fixed cost if the prediction confidence is low, rather than risking (more costly) prediction error.
ImageNet
ImageNet-R
Abstention 1
0.8
Log relative cost lower EnerrECC EE model
0.6
10
100
10
100
Misdassifiicatin cost (relative to abstention cost)
for BiT variants, one with better classification er- (R152x4, by 0.08), and with better ECE (R50x1, by 0.009). For abstention rates up to 70% (which covers most practical scenarios with abstention rates low enough for the model to be useful), the model with better accuracy has lower overall cost than the model with bet- ter ECE. The same is true for all other model families we study (Appendix B.3). For these families and this cost scenario, a practitioner should therefore always choose the most accu- rate available model regardless differences calibration. Ultimately, real- cost structures are complex and may yield different results; Figure 7 presents one common scenario with downstream ramifications for the importance of the calibration differences compared to accuracy.
Figure 7 compares the expected
Figure 7: Relative cost of BīT-R152x4 and BiT- R50x1 models in selective prediction scenario, computed as a combination of the misclassifica- tion and abstention costs given cost ratio (x-axis) and abstention rate (y-axis). Blue indi- cates regions where the higher-accuracy model (R 152x4) achieves lower cost than the better- calibrated model (R50x1). The accuracy advantage outweighs the calibration advantage for practical rejection rates, across tested abstention costs.
Pitfalls and Limitations
For this study, approached calibration with simple, practical question: Given two models, one more accurate and the other better calibrated, which should a practitioner choose? While working towards answering this question, encountered several pitfalls that complicate the interpretation of calibration results.
Measuring calibration is challenging, and while the quantity we want to estimate is well specified, the estimator itself can biased. There are two sources of bias: (i) from estimating ECE by binning, and (ii) from the finite sample size used estimate the per-bin statistics.

---

ImageNet
ImageNet-v2
ImageNet-R
ImageNet-A
0.05
0,4
Itemp scaled)
0.07
0.04
0.15
0.03
0.06
0.10
0.2
0.05
0.1
[temp. -scaled
1.5
4-
NLL
1.0
2-
1-
1.5
0.10
NLL (residual)
0.05
0.5
0.0
0.0
0.0
-0.1
00.5
0.7
Drier score
(tempp. Itempp.scalee -ccaled
0.5
0.6
1.0
0.4
0.6
0.4
0.4
0.6
0.3
0.2
0.4
0.01
0.02
Brier (residual)
0.00
0.00
-0.005
-0.04
-0.02
-0.010
-0.06
0.4
0.4
Classffication error
Classification ervor
Classification error
Classification error
Model family ResNeXt: WSL
MLP-Mixer
VIT
BIT
SimCUR
CLIP
# AlexNet
negative log-likelihood (NLL) and Brier score. For comparison, the first row shows ECE as in Figure 6. Since NLL, Brier score, and classification error are all highly correlated, we also provide the residuals of NLL and Brier score after regressing out classification error (third and fifth row). Specifically, where x; is the classification error and is the calibration measure of model i. We then report the residual y/ (Bo + (14z)) on the y-axis of the plots in the third and fifth row. The residuals show which models have better (or worse) NLL and Brier score than what can be expected from their accuracy alone. The relationships between model families are largely similar across all calibration metrics.
Figure Alternative calibration metrics:
first fit linear regression y; 30 + 3,
The first of these biases is always negative (Kumar et al., 2019), while the second one is always positive. Thus, the estimator can both under- and over-estimate the value, and the magnitude of the bias can depend on multiple factors. In practice, this means that the ranking models depends which ECE variant is chosen estimate calibration (Nixon al., 2019). As show below, this is especially problematic for the positive bias, because this bias depends on the accuracy of the model. It is therefore possible to arrive at opposite conclusions about the relationship between accuracy and calibration, depending on the chosen bin size (Figure 9), especially when comparing models with widely varying accuracies.

---

Intuitively, a larger number of bins implies fewer points per bin and thus higher variance of the estimate of the model accuracy in each bin, which adds positive bias to the estimate of ECE. More formally, to estimate accuracy( (B.) precisely, we need a number of samples in- versely proportional to the standard deviation vp.(1 - pt)/ B, where p; is the expected ac- curacy in B, This indicates that the bias would be smaller for models with extreme average accuracies (i.e. close to O or 1) and larger for models with accuracy close 0.5. A de- tailed analysis reveals additional effects that fur- ther reduce the bias for higher-accuracy mod- els (Appendix C). In particular, if we estimate (Elaccuracyy (BarrrccyBB)]BB[cooffiiee [[[confidence(BBe]] (2 for any bin i with n, samples (using the sample means of the confidences and the accuracies), the bias can be shown to be equal to (conditioning on X B, omitted for brevity)
15 bins (2667 points/bin)
1 100 bins (400 points/bin)
5000 bins (8 points/bin)
(temp. -ccaeec
ECE
0.02
0.02
0.10
0.08
0.00
0.00
0.06
0.3 Classification on error
0.2
0.3 Classification crrer
Classification error
Figure 9: The effect of binning-induced bias in ECE depends on accuracy. Each dot represents BiT ResNet model. Plotted models differ in model size, pretraining dataset size, and pretraining dura- tion. All models are fine-tuned and evaluated on IMAGENET. After temperature scaling, there is a year-lineat relationship between ECE and classifi- cation error. However, whether this relationship positive negative depends the number of bins used for estimating ECE. This effect is explained by an securacy-depender bias that increases with the number of bins.
(V[A) + V[C] - 2Cov[C,
(5)
where A [Y E arg max [(X] and C max f(X). Hence, from Equation 5 we conclude that higher accuracy models have a lower bias not only due to higher accuracy (lower V[A]), but also because their outputs correlate more with the correct label (higher covariance).
In addition a careful choice of bin size, considering accuracy and calibration jointly mitigates this issue, because the Pareto-optimal models rarely change, even if the ranking based on ECE alone does (Appendix E). In Appendices E and F, we provide the main figures of the paper for other ECE variants (number of bins, binning scheme, normalization metric, top-label, all-label, class-wise).
Finally, metrics such Brier score (Brier, 1950) and likelihood provide alternative assessments of model calibration that do not require estimating expected calibration error. We find that the relationships between model families are consistent across ECE, NLL and Brier score (Figure 8). In particular, the same models (specifically the largest MLP-Mixer and ViT variants) remain Pareto- optimal with respect to the calibration metric and classification error in most cases. The relationship between models is visualized especially clearly after regressing out from the calibration metrics their correlation with classification error (Figure 8, third and fifth row).
6 Conclusion
We performed a large study of the calibration of recent state-of-the-ar image models and its relation- ship with accuracy. We find that modern image models are well calibrated across distribution shifts despite being designed with a focus on accuracy. Our results suggest that there no general trend for recent or highly accurate neural networks to be poorly calibrated compared to older or less accurate models.
Our experiments suggest that simple dimensions such as model size and pretraining amount do not fully account for the performance differences between families, pointing towards architecture as a major determinant of calibration. Of particular note is the finding that MLP-Mixer and Vi- sion Transformers-ttwe recent architectures that are not based on convolutions among the best-calibrated models both in-distribution and out-of-distre dit---f-ditiibtion.. Self-attention (which Vision Transformers employ heavily) has been shown previously to be beneficial for certain kinds of out- of-distribution robustness (Hendrycks et al., 2020a). Our work now hints at calibration benefits of non-convolutional architectures more broadly, for certain kinds of distribution shift. Further work on the influence of architectural inductive biases on calibration and out-of-distributie robustness will be necessary to tell whether these results generalize. If so, they may further hasten the end of the convolutional era in computer vision.
10

---

Acknowledgments and Disclosure of Funding
We thank Carlos Riquelme and Balaji Lakshminarayanan for valuable comments the manuscript. The authors declare no competing interests.
References
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp. B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. End to end learning for self-driving cars. arXiv: 1604.07316, 2016.
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 1950.
Bröcker, J. Reliability, sufficiency, and the decomposition of proper scores. Journal of the Royal Meteorological Society, 2009.
Chen, T., Kormblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning. 2020.
DeGroot, M. H. and Fienberg, S. E. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society, 1983.
Deng, J.. Dong, W. Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009,
Djolonga, J., Matthias, M., Nado, Z., Nixon, J., Romijnders, R., Tran, D., and Lucic, M. Robustness Metrics, 2020. URL httpps:/ggiiiee cwwpgooggeecceeeeoooooooooooocee wtt.:/oooogggggaciiiiiee
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16
words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Esteva, A., Kuprel, B., Novoa, A., Ko, J., Swetter, M., Blau, M., and Thrun, Dermatologist- level classification of skin cancer with deep neural networks. Nature, 2017.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M.. Wichmann, F. A., and Brendel, W. Imagenet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. Bayesian data analysis. CRC press, 2013.
Gneiting, T., Balabdaoui, F., and Raftery, A. E. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society, 2007.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. calibration modern neural networks. In International Conference Machine Learning, 2017.
Gupta, K., Rahimi, A., Ajanthan, T., Mensink, T., Sminchisescu, C., and Hartley, R. Calibration of neural networks using splines. International Conference on Learning Representations, 2021.
Havasi, M., Jenatton, R., Fort, S., Liu, J. Z., Snoek, Lakshminarayanan, B., Dai, A. M., and Tran, D. Training independent subnetworks for robust prediction. In International Conference on Learning Representations, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations, 2019.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E.,, Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: critical analysis of distiiiiittoooation generalization. arXiv: 2006 2006.16241, 16241, 2020a.

---

Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method improve robustness and uncertainty. In International Conference on Learning Representations, 2020b.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt. J., and Song, D. Natural adversarial examples. In Conference on Computer Vision and Pattern Recognition, 2021.
Hernández-Oralle J., Flach, P. A., and Ferri, C. A unified view of performance metrics: translating threshold choice into expected classification loss. Journal of Machine Learning Research, 2012.
Jiang, X., Osl, M., Kim, J., and Ohno- Machado, L. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 2012.
Kolesnikov, A., Beyer, L.. Zhai, X.. Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In European Conference on Computer Vision, 2020.
Kompa, B., Snoek, J., and Beam, A. L. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digital Medicine, 2021.
Krizhevsky, A. One weird trick for parallelizing convolutional neural networks. arXiv: 1404.5997, 2014.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012.
Kumar, A. Liang P. S., and Ma, T. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, 2019.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
Leathart, T. and Polaczuk, M. Temporal probability calibration. arXiv: 2002.02644, 2020.
2020.
Mahajan, D., Girshick, R. B., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision, 2018.
Malinin, and Gales, M. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, 2018.
Müller, R., Kornblith, S., and Hinton, G. When does label smoothing help? Advances in Neural Information Processing Systems, 2019.
Mustafa, B., Loh, A., Freyberg, J., MalWiliamss, P., Wilson, M.. McKinney, M., Sienick, M..
Winkens, J., Liu, Y., Bui, P., Prabhakara, S.. Telang, U.. Karthikesalingam, A., Houlsby, N., and Natarajan, V. Supervised transfer learning at scale for medical imaging. arXiv: 2101,05913, 2021.
Naeini, M. P., Cooper, G., and Hauskrecht, M. Ohtaining well calibrated probabilities using bayesian binning. In AAAI Conference on Artificial Intelligence, 2015.
Nixon, J., Dusenberry, M. W., Zhang, I., Jerfel, G., and Tran, D. Measuring calibration in deep learning. In IEEE Conference Computer Vision and Pattern Recognition Workshops, 2019.
Ovadia, Y., Fertig, E., Lakshminarayanan, B., Nowozin, S., Sculley, D., Dillon, J. V., Ren, J., Nado, Z., and Snock, J. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.. Sastry, G.. Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. Technical report, OpenAl, 2021.
Rahaman, and Thiery, A. H. Uncertainty quantification and deep ensembles. arXiv: 2007,0879) 2020.
12

---

Recht, B., Roelofs, R., Schmidt, L.. and Shankar, V. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, 2019.
Roelofs, R., Cain, N., Shlens, J., and Mozer, M. Mitigating bias in calibration error estimation. arXiv: 2012.08668, 2020.
Shafer, G. and Vovk, V. A tutorial conformal prediction. Journal of Machine Learning Research, 2008.
Sønderby, C. K., Espeholt, L., Heek, J., Dehghani, M., Oliver, A., Salimans, T., Agrawal, S., Hickey, J., and Kalchbrenner, N. Metnet: A neural weather model for precipitation forecasting, arXiv: 12140, 2020.
Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In International Conference on Computer Vision, 2017.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Conference Computer Vision and Pattern Recognition, 2016.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 2020.
Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., and Michalak, On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In Advances in Neural Information Processing Systems, 2019.
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, I.., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A. MLP-Mixer: An all-MLP architecture for vision. arXiv: 2105.01601, 2021.
Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J., and Schön, B. Evaluating model calibration in classification. In International Conference Artificial Intelligence and Statistics, 2019.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Wen, Y., Tran, D., and Ba, J. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In Advances in Neural Information Processing Systems, 2020.
Wen, Y., Jerfel, G., Muller, R., Dusenberry, M. W., Snoek, J., Lakshminarayanan, B., and Tran, D. Combining ensembles and data augmentation can harm your calibration. In International Conference on Learning Representations, 2021.
Wenzel, F., Snoek, J., Tran, D., and Jenatton, R. Hyperparameter ensembles for robustness and uncertainty quantification. In Advances in Neural Information Processing Systems, 2020.
Xie, Q., Luong, M.-T., Hovy, E., and Le, Q. V. Self-training with noisy student improves imagenet classification. In Conference on Computer Vision and Pattern Recognition, 2020.
13


Reviews:
[{"review_id": "jh0tiS_HvHZ", "reviewer": "Reviewer_Yhh5", "summary": "This paper provides a systematic benchmark of recent deep classifiers in terms of the level of calibration.\n\nOn the calibration side, this paper focus on the definition of top-label (argmax, confidence) calibration. As a result, the main evaluation metrics are based on the top-label (confidence) ECE. The authors also include accuracy, the proper scoring rules BS and NLL and visualise the reliability diagrams. Temperature scaling is selected as the post-hoc approach.\n\nVarious model architectures and training strategies are applied on the imagenet datasets, with additional experiments on out-of-distribution tasks. \n\nOther than using many popular recent approaches, the authors report findings over the relationships on (1) model size and in-distribution calibration, (2) model size and out-of-distribution calibration, (3) accuracy and top-label calibration, (4) model size, training settings and calibration. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As suggested by the summary above, this paper is in the position of benchmarking recent deep approaches in terms of calibration. As at the moment there are many ongoing works on the calibration of deep models, it is indeed interesting to have such a paper to provide a systematic comparison. In particular, the reasons for learning uncalibrated results still require further understand, so this paper should provide some empirical results that could be of good impact to other researchers.  \n\nThat being said, the quality of this paper, therefore, depends on the comprehensiveness of the benchmarks and empirical findings. \n\nOn the comprehensiveness side, I am impressed by the selection of deep models and training strategies, and I agree this should bring some nice insights to the deep learning community. Regarding the dataset, while it is mainly designed with the imagenet (hence CV domain), it is understandable given the number of experiments required for such benchmarks. \n\nHowever, I am less impressed by the choice of calibration definition hence some of the associated metrics. As indicated in the paper, this paper mainly investigates the definition of top-label calibration. While I agree this is the most popular definition since Guo et al 2016, there are indeed some pitfalls regarding that definition. For instance, there will not be any uncalibrated top-label probability smaller than $\\frac{1}{K}$ where $K$ is a number of classes. One can easily argue that you should not only focus on the biggest probability values as other small values also represent useful information on the label uncertainty.\n\nThis choice of definition further leads to an issue over the empirical findings. One of the contributions claimed by the authors is that the calibration and accuracy are correlated. I tend to think this only holds for the definition of top-label calibration, as accuracy is also a metric about the top-label. For multi-class calibration (df.1) or class-wise calibration, I don't think we can obtain the same results. Therefore, I would be more satisfied if the authors consider explicitly stating they are talking about top-label calibration in both the title and through the main texts. And it would be more than welcome if the authors can include experiments on the aforementioned definitions of calibration (multi-class, class-wise) as well, but it is understandable if the authors prefer not as these experiments are somehow expensive.\n\nOn the evaluation metric side, I am happy with the discussion on ECE and including metrics like BS and NLL. Although, there is also a trend of performing statistical tests over the hypothesis of calibration, and report the p-value. It would be nice to see at least some discussion over such methods (https://arxiv.org/abs/1910.11385).\n\nRegarding the post-hoc calibration approaches, while I would like to see more methods, it can easily make the number of experiments grow rather fast(N-models * N-calibration * N-datasets). But one thing I would appreciate is to consider adding (https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf), which suggests they can improve the level of calibration during the training time.\n\nIn general, I think this paper has a good value to be seen by the ML community. I am currently giving a 6 mainly due to the point that the paper doesn't differentiate top-label calibration / general calibration in terms of the correlation on accuracy. I am happy to change the score if upon a discussion with the authors.\n\n=============After author feedback============================\n\nThe authors addressed some of my concerns in the response, I am happy to increase the score to 7.\n\n(1) In response to my comments on the selected definition of calibration (hence ECE), the authors add some experiments to further justify their empirical findings.\n\n(2) The authors acknowledge some of the related work and indicate the benchmark should be accessible for others to add additional experiments.\n\nAt the moment I am not giving any higher score (e.g. 8) due to the fact that the overall paper still mainly reports empirical findings. But I agree it should provide a good reference for the community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitation: The authors discuss the limitations of the measures of ECE. So it might be good to include a discussion on calibration tests as mentioned above. It would also be nice to further mention that post-hoc calibration is limited as it depends on a good uncalibrated model. That is, if a model simply outputs the marginal distribution of the target, this isn't much a calibration method can do.\n\nSocial impact: I agree with the authors, poorly calibrated models can have a negative impact if applied in society. For instance, to have calibrated probability on things like credit card approval is non-trivial.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "about 4 hours"}, "extra_text": {}}, {"review_id": "g268vdCu7b", "reviewer": "Reviewer_Uy9F", "summary": "The paper systematically evaluates the relation between model calibration and accuracy. The main finding is that the most recent models, notably those not using convolutions, are among the best calibrated. In addition, the analysis shows that model size and amount of pretraining do not explain these differences. Experimental evaluation is extensive and very carefully performed.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper focuses on an important and timely question, namely the relation between model architecture, calibration and accuracy. The empirical evaluation performed in the paper is great and very insightful. The hypothesis that convolutions lead to poorly calibrated models is very interesting. Unfortunately, it is not further analysed. Also the paper is mainly empirical, a more theoretical analysis of the problem / interpretation of the results is lacking.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "5TpS_dMIpAE", "reviewer": "Reviewer_9HWK", "summary": "The paper evaluates numerous recent models in terms of uncertainty calibration. The models evaluated are recent advances in relation to a previous seminal paper on the topic. The findings of this work indicate that previous trends leading to overconfidence with larger models may not apply to the current state-of-the-art models.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Pros\n\n- Compares many SOTA models which have not been evaluated together in terms of uncertainty thus far. \n- Utilizes types of distributionally shifted data which are not considered in most previous calibration works.\n- The experiments are thorough, informative, and show that newer families of large non-convolutional models are not overconfident.  \n- The paper is clearly written and well designed. \n- The paper highlights relevant problems with Expected Calibration Error. The authors highlighted the positive bias on ECE of finite samples within bins. To my knowledge, this has not been discussed before.\n\n# Cons\n\n- As there are many different models evaluated, It would be beneficial to see some comparison of the total number of parameters across model families vs. calibration performance of each model. \n- Section 5 highlights problems with ECE, which is a crucial axis for all results in the main text and then only refers the reader to the appendix for other measures. The experiments in the appendix (Figure 16) should probably be moved to the main text to provide a measure besides only ECE.  \n- In the appendix (Figure 16), there is no ImageNet-C dataset. I would like to see the results for NLL and Brier score on this dataset, as it is included in the other results of the main text. The performance at each corruption level may be informative, given that ECE could be biased. \n- I am confused by the verbiage in a sentence of Figure 16. How exactly were the 3rd and 5th rows of residual errors calculated? \n    - \"we also provide the residuals of NLL and Brier score after regressing out\nclassification error (third and fifth row)\"\n\n# Addendum:\n\nMy final score derives from the following:\n\n- In terms of originality, the setting is not novel, but the results and conclusions are because the authors seem to have shown that the commonly held belief of DNN overconfidence may not be true anymore for recent models.\n- The paper is high quality, very clear and easy to read.\n- In terms of significance, I think this paper contributes in both upending the common mantra of overconfidence, and I also think the results here are immediately applicable to practitioners. Some of the models trained here are quite heavy and expensive, so being able to learn from these experiments presents an opportunity to both achieve better calibration and avoid costly and time consuming experimentation.  \n- The authors have adequately answered any concerns I raised.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they discussed the pitfalls of one of the main metrics used in the text. I am not aware of any potential negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the question of calibration in neural network image classifiers. Prior and widely cited empirical work in this area showed that state-of-the art models (at the time) could exhibit quite poor calibration properties. This paper empirically demonstrates that these same issues appear to be much less of a concern with recent, high-performing deep architectures including the MLP-Mixer and Vision Transformers in both the in-distribution setting and the out-of-distribution setting. The primary novelty in this work comes from re-visiting the question of calibration in light of recent advances in model architectures. The empirical results have strong practical significance as they indicate that past findings about calibration in convolutional models appear not to apply to recent deep architectures. The experiments conducted and metrics chosen are sound. Following the author response and discussion, the consensus is that the paper should be accepted. In revising the manuscript, the authors should include the additional results described during the discussion and take the reviewers' suggestions on the partitioning of material between the main body of the paper and the supplemental material into account. 
Decision: Accept (Poster)