Revisiting the Calibration of Modern Neural Networks
Year: 2022

Abstract:
Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.

heading: 1 Introduction
text: Neural networks, especially vision models, are increasingly used in safety-critical applications such as autonomous driving (Bojarski et al., 2016), medical diagnosis (Esteva et al., 2017; Jiang et al., 2012), and meteorological forecasting (Sønderby et al., 2020). For such applications, it is essential that model predictions are not just accurate, but also well calibrated. Model calibration refers to the accuracy with which the scores provided by the model reflect its predictive uncertainty. For example, in a medical application, we would like to defer images for which the model makes low-confidence predictions to a physician for review (Kompa et al., 2021). Skipping human review due to confident, but incorrect, predictions, could have disastrous consequences.
While intense research and engineering effort has focused on improving the predictive accuracy of models, less attention has been given to model calibration. In fact, over the last few years, there have been many reports that calibration of modern neural networks can be surprisingly poor, despite the advances in accuracy (e.g. Guo et al. 2017; Lakshminarayanan et al. 2017; Malinin & Gales 2018; Thulasidasan et al. 2019; Hendrycks et al. 2020b; Ovadia et al. 2019; Wenzel et al. 2020; Havasi et al. 2021; Rahaman & Thiery 2020; Leathart & Polaczuk 2020). Some works suggest a trend for larger, more accurate models to be worse calibrated (Guo et al., 2017).
These concerns are more relevant than ever, since the architecture size, amount of training data, and computing power used by state-of-the-art models continue to increase. At the same time, rapid advances in model architecture (Tolstikhin et al., 2021; Dosovitskiy et al., 2021) and training approaches (Chen et al., 2020; Mahajan et al., 2018; Radford et al., 2021) raise the question whether past results on calibration, largely obtained on standard convolutional architectures, extend to current state-of-the-art models. Since model advances are quickly translated to real-world, safety-critical applications (e.g. Mustafa et al. 2021), there is an urgent need to re-assess the calibration properties of current state-of-the-art models.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Contributions. To address this need, we provide a systematic comparison of recent image classification models, relating their accuracy, calibration, and design features. We find that:
1. The best current models, including the non-convolutional MLP-Mixer (Tolstikhin et al., 2021) and Vision Transformers (Dosovitskiy et al., 2021), are well calibrated compared to past models and their performance is more robust to distribution shift. 2. In-distribution calibration slightly deteriorates with increasing model size, but this is outweighed by a simultaneous improvement in accuracy. 3. Under distribution shift, calibration improves with model size, reversing the trend seen indistribution. 4. Accuracy and calibration are correlated under distribution shift, such that optimizing for accuracy may also benefit calibration. 5. Model size, pretraining duration, and pretraining dataset size cannot fully explain differences in calibration properties between model families.
Our results suggest that further improvements in model accuracy will continue to benefit calibration. They also hint at architecture as an important determinant of model calibration. We provide code and a large dataset of calibration measurements, comprising 180 distinct models from 16 families, each evaluated on 79 ImageNet-scale datasets and 28 metric variants.1

heading: 2 Related Work
text: Measures of model calibration. The losses that are commonly used to train classification models, such as cross-entropy and squared error, are proper scoring rules (Gneiting et al., 2007) and are therefore guaranteed to yield perfectly calibrated models at their minimum—in the infinite-data limit. However, in practice, due to model mismatch and overfitting, even losses based on proper scoring rules may result in poor model calibration. Miscalibration is commonly quantified in terms of Expected Calibration Error (ECE; Naeini et al. 2015), which measures the absolute difference between predictive confidence and accuracy. We focus on ECE because it is a widely used and accepted calibration metric. Nevertheless, it is well understood that estimating ECE accurately is difficult because estimators can be strongly biased and many estimator variants exist (Nixon et al., 2019; Roelofs et al., 2020; Vaicenavicius et al., 2019; Gupta et al., 2021). Section 5 discusses these issues and our approaches to mitigate them.
Alternatives to ECE include likelihood measures, Brier score (Brier, 1950), Bayesian methods (Gelman et al., 2013), and conformal prediction (Shafer & Vovk, 2008). Further, model calibration can be represented visually with reliability diagrams (DeGroot & Fienberg, 1983). Figure 8 and Appendix F provide likelihoods, Brier scores, and reliability diagrams for our main analyses.
Empirical studies of model calibration. There have been many recent empirical studies on the robustness (accuracy under distribution shift) of image classifiers (Geirhos et al., 2019; Taori et al., 2020; Djolonga et al., 2020; Hendrycks et al., 2020a). Several works have also studied calibration. Most notable is Guo et al. (2017), who found that “modern neural networks, unlike those from a decade ago, are poorly calibrated”, that larger networks tend to be calibrated worse, and that “miscalibration worsen[s] even as classification error is reduced.” Other works have corroborated some of these findings (e.g., Thulasidasan et al. 2019; Wen et al. 2021). This line of work suggests a trend that larger models are worse calibrated, which would have major implications for research toward bigger models and datasets. We show that for more recent models, this trend is negligible in-distribution and in fact reverses under distribution shift.
Ovadia et al. (2019) empirically study calibration under distribution shift and provide a large comparison of methods for improving calibration. They report that both accuracy and calibration deteriorate with distribution shift. While we observe the same trend, we find that the calibration of some recent model families decays so slowly under distribution shift that the decay in accuracy is likely more relevant in practice (Section 4.3).
Ovadia et al. also find that, across methods for improving calibration, improvements on in-distribution data do not necessarily translate to out-of-distribution data. This finding may suggest that there is little correlation between in-distribution and out-of-distribution calibration in general. However, our
1Available at https://github.com/google-research/robustness_metrics/tree/master/ robustness_metrics/projects/revisiting_calibration.
results show that, across model architectures, the models with the best in-distribution calibration are also the best-calibrated on a range of out-of-distribution benchmarks. The important implication of this result is that designing models based on in-distribution performance likely also benefits their out-of-distribution performance.
Improving calibration. Many strategies have been proposed to improve model calibration such as post-hoc rescaling of predictions (Guo et al., 2017), averaging multiple predictions (Lakshminarayanan et al., 2017; Wen et al., 2020), and data augmentation (Thulasidasan et al., 2019; Wen et al., 2021). Here, we focus on the intrinsic calibration properties of state-of-the-art model families, rather than methods to further improve calibration.
As a baseline on top of a model’s intrinsic calibration properties, we study temperature scaling (Guo et al., 2017). It is effective in improving calibration and so simple that it can be applied in many cases at minimal additional cost, in contrast to many more sophisticated methods. Temperature scaling re-scales a model’s logits by a single parameter, chosen to optimize the model’s likelihood on a held-out portion of the training data. This temperature factor changes the model’s confidence, i.e., whether the model predictions are on average too certain (overconfident), optimally confident, or too uncertain (underconfident). The classification accuracy of the model is not affected by temperature scaling. A large fraction of model miscalibration is typically due to average over- or underconfidence, e.g. due to suboptimal training duration (Guo et al., 2017). By normalizing a model’s confidence, temperature scaling not only improves calibration, but also removes a primary confounder that can hide trends in calibration between models (see Section 4.2 and Appendix D). Therefore, we study both unscaled and temperature-scaled predictions in the paper.

heading: 3 Definitions and Notation
text: We consider the multi-class classification problem, as analyzed by Bröcker (2009), where we observe a variable X and predict a categorical variable Y ∈ {1, 2, . . . , k}. We model our predictor f as a function that maps every input instanceX to a categorical distribution over k labels, represented using a vector f(X) belonging to the (k − 1)-dimensional simplex ∆ = {p ∈ [0, 1]k | ∑k y=1 py = 1}.
Intuitively, a model f is well-calibrated if its output truthfully quantifies the predictive uncertainty. For example, if we take all data points x for which the model predicts [f(x)]y = 0.3, we expect 30% of them to indeed take on the label y. Formally, the model f is said to be calibrated if (Bröcker, 2009)
∀p ∈ ∆: P (Y = y | f(X) = p) = py. (1)
We will focus on a slightly weaker, but more practical condition, called top-label or argmax calibration (Kumar et al., 2019; Guo et al., 2017). This requires that the above holds only for the most likely label, i.e., ∀p∗ ∈ [0, 1] P (Y ∈ arg max p | max f(X) = p∗) = p∗, (2) where the max and arg max act coordinate-wise.
The most common measure of the degree of miscalibration is the Expected Calibration Error (ECE), which computes the expected disagreement between the two sides of eq. (2)
E [ |p∗ − E[Y ∈ arg max f(X) | max f(X) = p∗| ] . (3)
Unfortunately, eq. (3) cannot be estimated without quantization as it conditions on a null event. Hence, one typically first buckets the predictions intom binsB1, . . . , Bm based on their top predicted probability, and then takes the expectation over these buckets. Namely, if we are given a set of n i.i.d. samples (x1, y1), . . . , (xn, yn) distributed as P (X,Y ), then we assign each j ∈ {1, . . . , n} to a bucket Bi based on max f(xj). Then, we compute in each bucket Bi the confidence(Bi) = 1 |Bi| ∑ j∈Bi max f(xj) and the accuracy(Bi) = 1 |Bi| ∑ j∈BiJyj ∈ arg max f(xj)K, where J·K is the Iverson bracket. Finally, we construct an estimator by taking the expectation over the bins
ÊCE = m∑ i=1 |Bi| n |accuracy(Bi)− confidence(Bi)| . (4)
In Section 5 we discuss the statistical properties of this estimator, possible pitfalls, and several mitigation strategies.

heading: 4 Empirical Evaluation

heading: 4.1 Experimental Setup
text: Model families. In this study, we consider a range of recent and some historic state-of-the-art image classification models. Our selection of models covers convolutional and non-convolutional architectures, as well as supervised, weakly supervised, unsupervised and zero-shot training. We follow the original publications in naming the model variants within each family (e.g. different model sizes). See Appendix A.1 for a detailed description of all used models.
1. MLP-Mixer (Tolstikhin et al., 2021) is based exclusively on multi-layer perceptrons (MLPs) and is pre-trained on large supervised datasets. 2. ViT (Dosovitskiy et al., 2021) processes images with a transformer architecture originally designed for language (Vaswani et al., 2017) and is also pre-trained on large supervised datasets. 3. BiT (Kolesnikov et al., 2020) is a ResNet-based architecture (He et al., 2016). It is also pre-trained on large supervised datasets. 4. ResNext-WSL (Mahajan et al., 2018) is based on the ResNeXt architecture and trained with weak supervision from billions of hashtags on social media images. 5. SimCLR (Chen et al., 2020) is a ResNet, pretrained with an unsupervised contrastive loss. 6. CLIP (Radford et al., 2021) is pretrained on raw text and imagery using a contrastive loss. 7. AlexNet (Krizhevsky et al., 2012; Krizhevsky, 2014) was the first convolutional neural network
to win the ImageNet challenge.
All models are either trained or fine-tuned on the IMAGENET training set, except for CLIP, which makes zero-shot predictions using IMAGENET class names as queries.
Datasets. We evaluate accuracy and calibration on the IMAGENET validation set and the following out-of-distribution benchmarks using the Robustness Metrics library (Djolonga et al., 2020):
1. IMAGENETV2 (Recht et al., 2019) is a new IMAGENET test set collected by closely following the original IMAGENET labeling protocol. 2. IMAGENET-C (Hendrycks & Dietterich, 2019) consists of the images from IMAGENET, modified with synthetic perturbations such as blur, pixelation, and compression artifacts at a range of severities. 3. IMAGENET-R (Hendrycks et al., 2020a) contains artificial renditions of IMAGENET classes such as art, cartoons, drawings, sculptures, and others. 4. IMAGENET-A (Hendrycks et al., 2021) contains images that are classified as belonging to IMAGENET classes by humans, but adversarially selected to be hard to classify for a ResNet50 trained on IMAGENET.
For the post-hoc recalibration of models, we reserve 20% of the IMAGENET validation set (randomly sampled) for fitting the temperature scaling parameter. All reported metrics are computed on the remaining 80% of the data. For evaluations on IMAGENET-C, we also exclude the 20% of images that are based on the IMAGENET images used for temperature scaling.
Calibration metric. Throughout the paper, we estimate ECE using equal-mass binning and 100 bins. Appendix E shows that our results hold for other ECE variants and are consistent with the Brier score and model likelihood.

heading: 4.2 In-Distribution Calibration
text: We begin by considering ECE on clean IMAGENET images (referred to as in-distribution). Figure 1 shows in-distribution ECE and reliability diagrams before any recalibration of the predicted probabilities. We find that several recent model families (MLP-Mixer, ViT, and BiT) are both highly accurate and well-calibrated compared to prior models, such as AlexNet or the models studied by Guo et al. (2017). This suggests that there may be no continuing trend for highly accurate modern neural networks to be poorly calibrated, as suggested previously (Guo et al., 2017; Lakshminarayanan et al., 2017; Malinin & Gales, 2018; Thulasidasan et al., 2019; Hendrycks et al., 2020b; Ovadia et al., 2019; Wenzel et al., 2020; Havasi et al., 2021; Rahaman & Thiery, 2020; Leathart & Polaczuk, 2020). In addition, we find that a recent zero-shot model, CLIP, is well-calibrated given its accuracy.
Temperature scaling reveals consistent properties of model families. The poor calibration of past models can often be remedied by post-hoc recalibration such as temperature scaling (Guo et al., 2017), which raises the question whether a difference between models remains after recalibration. We find that the most recent architectures are better calibrated than past models even after temperature scaling (Figure 2, right).
More generally, temperature scaling reveals consistent trends in the calibration properties between families that are obscured in the unscaled data by simple over- or underconfidence miscalibration. Before temperature scaling (Figure 2, left), several families overlap in their accuracy/calibration properties (MLP-Mixer, ViT, BiT). After temperature scaling (Figure 2, right), a clearer separation of families and consistent trends between accuracy and calibration within each family become apparent. Notably, temperature scaling reconciles our results for BiT (a ResNet architecture) with the results reported by Guo et al. for ResNets trained on IMAGENET. Furthermore, models pretrained without addi-
tional labels (SimCLR) or with noisy labels (ResNeXt-WSL) tend to be calibrated worse for a given accuracy than ResNets trained with supervision (BiT and the models studied by Guo et al.). Finally, non-convolutional model families like MLP-Mixer and ViT can perform just as well, if not better, than convolutional ones.
Differences between families are not explained by model size or pretraining amount. We next attempt to disentangle how the differences between model families affect their calibration properties. We focus on model size and amount of pretraining, both important trends in state of-the-art models.
We first consider model size. Prior work has suggested that larger neural networks are worse calibrated (Guo et al., 2017). We also find that within most families, larger members tend to have higher calibration error (Figure 2, right). However, at the same time, larger models have consistently lower classification error. This means that each model family occupies a different Pareto set in the tradeoff between accuracy and calibration. For example, our results suggest that, at any given accuracy, ViT models are better calibrated than BiT models. Changing the size of a BiT model cannot move it into the Pareto set of ViT models. Model size can therefore not fully explain the intrinsic calibration differences between these model families.2
We next consider model pretraining. Many current state-of-the-art image models use transfer learning, in which a model is pre-trained on a large dataset and then fine-tuned to the task of interest (Kolesnikov et al., 2020; Chen et al., 2020; Xie et al., 2020). With transfer learning, large data sources can be
2This relationship between model size, accuracy and calibration holds for all families we study except ResNeXt-WSL, for which increasing model sizes improves both accuracy and calibration. While investigating this difference was out of the scope of this work, it may be a promising direction for future research.
exploited to train the model, even if little data are available for the final task. To test how the amount of pretraining affects calibration, we compare BiT models pretrained on IMAGENET (1.3M images), IMAGENET-21K (12.8M images), or JFT-300 (300M images; Sun et al. 2017).
More pretraining data consistently increases accuracy, especially for larger models. It has no consistent effect on calibration (Figure 3). In particular, after temperature scaling, ECE is essentially unchanged across this 300-fold increase in pretraining dataset size (e.g. for BiTR50x1 pretrained on IMAGENET, IMAGENET21K and JFT-300, the ECEs are 0.0185, 0.0182, 0.0185, respectively; for BiT-R101x3, they are 0.0272, 0.0311, 0.0236; Figure 3, bottom). Therefore, regardless of the pretraining dataset, BiT always remains Pareto-dominant over SimCLR and Pareto-dominated by ViT and MLPMixer in our experiments.
The BiT models compared in Figure 3 differ in both the amount of pretraining data and the duration of pretraining (see Kolesnikov et al. (2020) for details). To further disentangle these variables, we trained BiT models on varying numbers of pretraining examples while holding the number of training steps constant, and vice versa. We find that pretraining dataset size has no significant effect on calibration, while pretraining duration only shifts the model within its accuracy/calibration Pareto set (longer-trained models are more accurate and worse calibrated;
Figure 10). These results suggest that pretraining alone cannot explain the differences between model families that we observe.
In summary, our results show that some modern neural network families combine high accuracy and state-of-the-art calibration on in-distribution data, both before and after post-hoc recalibration by temperature scaling. In Figure 8 and Appendices E and F, we show that these results generally hold for other measures of model calibration (other ECE variants, Brier score, and model likelihood). Our experiments further suggest that model size and pretraining amount do not fully explain the intrinsic calibration differences between model families. Given that the best-calibrated families (MLP-Mixer and ViT) are non-convolutional, we speculate that model architecture, and in particular its spatial inductive bias, play an important role.

heading: 4.3 Accuracy and Calibration Under Distribution Shift
text: For safety-critical applications, the model should produce reasonable uncertainty estimates not just in-distribution, but also under distribution shifts that were not anticipated at training time. We first assess out-of-distribution calibration on the IMAGENET-C dataset, which consists of images that have been synthetically corrupted at five different severities. As expected, both classification and calibration error generally increase with distribution shift (Figure 4; Ovadia et al. 2019; Hendrycks & Dietterich 2019). Interestingly, this decay in calibration performance is slower for MLP-Mixer and ViT than for the other model families, both before and after temperature scaling.
Regarding the effect of model size on calibration, we observed some trend towards worse calibration of larger models on in-distribution data. However, the trend is reversed for most model families as we move out of distribution, especially after accounting for confidence bias by temperature scaling (note positive slope of the gray lines at high corruption severities in Figure 4, bottom row). In other words, the calibration of larger models is more robust to distribution shift (Figure 5).
We next consider to what degree the insights from in-distribution and IMAGENET-C calibration transfer to natural out-of-distribution data. Previous work on IMAGENET-C suggests that, when comparing recalibration methods, better in-distribution calibration and accuracy do not usually predict better calibration under distribution shift (Ovadia et al., 2019). Here, comparing model families, we find that the performance on several natural out-of-distribution datasets is largely consistent with that on IMAGENET (Figure 6). In particular, models that are Pareto-optimal (i.e. no other model is both more accurate and better calibrated) on IMAGENET remain Pareto-optimal on the OOD datasets. Further, we observe a strong correlation between accuracy and calibration on the OOD datasets. This relationship is consistent across models within a family and across datasets, over a wide range of accuracies (Figure 11).
These results suggest that larger and more accurate models, and in particular MLP-Mixer and ViT, can maintain their good in-distribution calibration even under severe distribution shifts. Based on the observed relationship between calibration and accuracy, we can reasonably hope that good calibration on in-distribution data (and anticipated distribution shifts) generally translates into good calibration on unanticipated outof-distribution data, similar to what has been observed for accuracy (Djolonga et al., 2020).

heading: 4.4 Relating Accuracy and Calibration Within Model Families
text: Our data suggest that most model families lie on different Pareto sets in the accuracy/calibration space, which establishes a clear preference between families. We next consider how to compare individual models within a family (or more specifically, within a Pareto set), where one model is more accurate but worse calibrated, and the other is less accurate but better calibrated. Which model should a practitioner choose for a safety-critical application?
The answer depends on the cost structure of the specific application (Hernández-Orallo et al., 2012). As an example, consider the scenario of selective prediction, which is common in medical diagnosis. In this task, one can choose to ignore the model prediction (“abstain”) at a fixed cost if the prediction confidence is low, rather than risking a (more costly) prediction error.
Figure 7 compares the expected cost for two BiT variants, one with better classification error (R152x4, by 0.08), and one with better ECE (R50x1, by 0.009). For abstention rates up to 70% (which covers most practical scenarios with abstention rates low enough for the model to be useful), the model with better accuracy has a lower overall cost than the model with better ECE. The same is true for all other model families we study (Appendix B.3). For these families and this cost scenario, a practitioner should therefore always choose the most accurate available model regardless of differences in calibration. Ultimately, real-world cost structures are complex and may yield different results; Figure 7 presents one common scenario with downstream ramifications for the importance of the calibration differences compared to accuracy.

heading: 5 Pitfalls and Limitations
text: For this study, we approached calibration with a simple, practical question: Given two models, one more accurate and the other better calibrated, which should a practitioner choose? While working towards answering this question, we encountered several pitfalls that complicate the interpretation of calibration results.
Measuring calibration is challenging, and while the quantity we want to estimate is well specified, the estimator itself can be biased. There are two sources of bias: (i) from estimating ECE by binning, and (ii) from the finite sample size used to estimate the per-bin statistics.
The first of these biases is always negative (Kumar et al., 2019), while the second one is always positive. Thus, the estimator can both under- and over-estimate the true value, and the magnitude of the bias can depend on multiple factors. In practice, this means that the ranking of models depends on which ECE variant is chosen to estimate calibration (Nixon et al., 2019). As we show below, this is especially problematic for the positive bias, because this bias depends on the accuracy of the model. It is therefore possible to arrive at opposite conclusions about the relationship between accuracy and calibration, depending on the chosen bin size (Figure 9), especially when comparing models with widely varying accuracies.
0.2 0.3 Classification
error
0.00 0.02 0.04 EC E
(te m
p. -s
ca le
d)
15 bins (2667 points/bin)
0.2 0.3 Classification
error
0.00 0.02 0.04
100 bins (400 points/bin)
0.2 0.3 Classification
error
0.06 0.08 0.10 0.12
5000 bins (8 points/bin)
Figure 9: The effect of binning-induced bias in ECE depends on accuracy. Each dot represents a BiT ResNet model. Plotted models differ in model size, pretraining dataset size, and pretraining duration. All models are fine-tuned and evaluated on IMAGENET. After temperature scaling, there is a near-linear relationship between ECE and classification error. However, whether this relationship is positive or negative depends on the number of bins used for estimating ECE. This effect is explained by an accuracy-dependent bias that increases with the number of bins.
Intuitively, a larger number of bins implies fewer points per bin and thus higher variance of the estimate of the model accuracy in each bin, which adds positive bias to the estimate of ECE. More formally, to estimate accuracy(Bi) precisely, we need a number of samples inversely proportional to the standard deviation√ pi(1− pi)/|Bi|, where pi is the expected accuracy in Bi. This indicates that the bias would be smaller for models with extreme average accuracies (i.e. close to 0 or 1) and larger for models with an accuracy close to 0.5. A detailed analysis reveals additional effects that further reduce the bias for higher-accuracy models (Appendix C). In particular, if we estimate (E[accuracy(Bi)]−E[confidence(Bi)])2 for any bin i with ni samples (using the sample means of the confidences and the accuracies), the bias can be shown to be equal to (conditioning on X ∈ Bi omitted for brevity)
1
ni
( V[A] + V[C]− 2Cov[C,A] ) , (5)
where A = JY ∈ arg max f(X)K and C = max f(X). Hence, from Equation 5 we can conclude that higher accuracy models have a lower bias not only due to higher accuracy (lower V[A]), but also because their outputs correlate more with the correct label (higher covariance).
In addition to a careful choice of bin size, considering accuracy and calibration jointly mitigates this issue, because the Pareto-optimal models rarely change, even if the ranking based on ECE alone does (Appendix E). In Appendices E and F, we provide the main figures of the paper for other ECE variants (number of bins, binning scheme, normalization metric, top-label, all-label, class-wise).
Finally, metrics such as Brier score (Brier, 1950) and likelihood provide alternative assessments of model calibration that do not require estimating expected calibration error. We find that the relationships between model families are consistent across ECE, NLL and Brier score (Figure 8). In particular, the same models (specifically the largest MLP-Mixer and ViT variants) remain Paretooptimal with respect to the calibration metric and classification error in most cases. The relationship between models is visualized especially clearly after regressing out from the calibration metrics their correlation with classification error (Figure 8, third and fifth row).

heading: 6 Conclusion
text: We performed a large study of the calibration of recent state-of-the-art image models and its relationship with accuracy. We find that modern image models are well calibrated across distribution shifts despite being designed with a focus on accuracy. Our results suggest that there is no general trend for recent or highly accurate neural networks to be poorly calibrated compared to older or less accurate models.
Our experiments suggest that simple dimensions such as model size and pretraining amount do not fully account for the performance differences between families, pointing towards architecture as a major determinant of calibration. Of particular note is the finding that MLP-Mixer and Vision Transformers—two recent architectures that are not based on convolutions—are among the best-calibrated models both in-distribution and out-of-distribution. Self-attention (which Vision Transformers employ heavily) has been shown previously to be beneficial for certain kinds of outof-distribution robustness (Hendrycks et al., 2020a). Our work now hints at calibration benefits of non-convolutional architectures more broadly, for certain kinds of distribution shift. Further work on the influence of architectural inductive biases on calibration and out-of-distribution robustness will be necessary to tell whether these results generalize. If so, they may further hasten the end of the convolutional era in computer vision.
Acknowledgments and Disclosure of Funding
We thank Carlos Riquelme and Balaji Lakshminarayanan for valuable comments on the manuscript. The authors declare no competing interests.

Review summaries:
This paper provides a systematic benchmark of recent deep classifiers in terms of the level of calibration.  On the calibration side, this paper focus on the definition of top-label (argmax, confidence) calibration. As a result, the main evaluation metrics are based on the top-label (confidence) ECE. The authors also include accuracy, the proper scoring rules BS and NLL and visualise the reliability diagrams. Temperature scaling is selected as the post-hoc approach.  Various model architectures and training strategies are applied on the imagenet datasets, with additional experiments on out-of-distribution tasks.   Other than using many popular recent approaches, the authors report findings over the relationships on (1) model size and in-distribution calibration, (2) model size and out-of-distribution calibration, (3) accuracy and top-label calibration, (4) model size, training settings and calibration. 

The paper evaluates numerous recent models in terms of uncertainty calibration. The models evaluated are recent advances in relation to a previous seminal paper on the topic. The findings of this work indicate that previous trends leading to overconfidence with larger models may not apply to the current state-of-the-art models.  

The paper systematically evaluates the relation between model calibration and accuracy. The main finding is that the most recent models, notably those not using convolutions, are among the best calibrated. In addition, the analysis shows that model size and amount of pretraining do not explain these differences. Experimental evaluation is extensive and very carefully performed.