What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers?
Authors: Ido Galil
Mohammed Dabbah
Ran El-Yaniv
Venue: ICLR 2023 poster (2023)
Keywords: selective prediction
selective classification
reject option
risk coverage trade-off
deep learning
neural networks

Abstract: When deployed for risk-sensitive tasks, deep neural networks must include an uncertainty estimation mechanism.
Here we examine the relationship between deep architectures and their respective training regimes, with their corresponding selective prediction and uncertainty estimation performance. We consider some of the most popular estimation performance metrics previously proposed including AUROC, ECE, AURC as well as coverage for selective accuracy constraint. 
We present a novel and comprehensive study of selective prediction and the uncertainty estimation performance of 523 existing pretrained deep ImageNet classifiers that are available in popular repositories.
We identify numerous and previously unknown factors that affect uncertainty estimation and examine the relationships between the different metrics. We find that distillation-based training regimes consistently yield better uncertainty estimations than other training schemes such as vanilla training, pretraining on a larger dataset and adversarial training.
Moreover, we find a subset of ViT models that outperform any other models in terms of uncertainty estimation performance.
For example, we discovered an unprecedented 99% top-1 selective accuracy on ImageNet at 47% coverage
(and 95% top-1 accuracy at 80%) for a ViT model, whereas a competing EfficientNet-V2-XL cannot obtain these accuracy constraints at any level of coverage. 
Our companion paper, also published in ICLR 2023 (A framework for benchmarking class-out-of-distribution detection and its application to ImageNet), examines the performance of these classifiers in a class-out-of-distribution setting.

Published as a conference paper at ICLR 2023
WHAT CAN WE LEARN FROM THE SELECTIVE PREDIC- TION AND UNCERTAINTY ESTIMATION PERFORMANCE OF 523 IMAGENET CLASSIFIERS?
Ido Galil
Mohammed Dabbah
Ran El-Yaniv Technion, Deci.AI raniecs www.s..ccciionocooooooommoo ac. i1
Technion
Amazon
idogalil. gmailmmll....... all..cccoomm com
m. m. dabbah gmai com
ABSTRACT
When deployed for risk- tasks, deep neural networks must include an uncertainty estimation mechanism. Here we examine the relationship between deep architectures and their respective training regimes, with their corresponding selective prediction and uncertainty estimation performance. We consider some of the most popular estimation performance metrics previously proposed including AUROC, ECE, AURC as well as coverage for selective accuracy constraint. We present novel and comprehensive study selective prediction and the uncertainty estimation performance of 523 existing pretrained deep ImageNet classifiers that are available in popular repositories. We identify numerous and previously unknown factors that affect uncertainty estimation and examine the relationships between the different metrics. We find that distillation- yield better uncertainty estimations than other training schemes such vanilla training, pretraining on a larger dataset and ad adversarial training. Moreover, we find subset of ViT models that outperform any other models terms of uncertainty estimation performance. For example, we discovered an unprecedented 99% top-1 selective accuracy on ImageNet at 47% coverage (and 95% top-1 accuracy at 80%) for VIT model, whereas competing EfficientNet-V22 V2-XXL cannot obtain these accuracy constraints any level of coverage. Our companion paper, also published in ICLR 2023 (Galil et al., 2023), examines the performance of these classifiers in a -flss-out-of--istribuu setting.
training regimes consistently
INTRODUCTION
The excellent performance of deep neural networks (DNNs) has been demonstrated in range of applications, including computer vision, natural language understanding and audio processing. To deploy these models successfully, it is imperative that they provide uncertainty quantification of their predictions, either via some kind of selective prediction or probabilistic confidence score.
Notwithstanding, what metric should There are many and diverse ways so the answer to this question is not obvious, and to demonstrate the difficulty, consider the case of classification models for the stock market that predict whether a stock's 's value is about to increase, decrease, or remain neutral (three-class classification). Suppose that model A has 95% accuracy, and generates confidence score of 0.95 on every prediction (even on misclassified instances); model B has 40% true accuracy, but always gives confidence score of 0.6 on correct predictions, and 0.4 on incorrect ones. Model B can be utilized easily to generate perfect investment decisions. Using selective prediction (El-Yan (El-Yaniv& & Wiener, 2010; Geifman Yanv,, 2017), Model B will simply reject all investments on stocks whenever the confidence score is 0.4. While model A offers many more investment opportunities, each of its predictions carries a 5% risk of failure.
evaluate the uncertainty estimation performance?
Among the various metrics proposed for evaluating the performance of uncertainty estimation are: Area Under the Receiver Operating Characteristic (AUROC or AUC), Area Under the Tisk-Coverage curve (AURC) (Geifman al., 2018), selective risk coverage for selective accuracy constraint (SAC), Negative Log-likelinn likelhhoo (NLL), Expected Calibration Error (ECE), which often used for evaluating model's calibration (see Section and Brier score (Brier, 1950). All these metrics

---

Published as a conference paper at ICLR 2023
1
AUROC
84
82
Models
Various
+ ResMLPP
AlexNet
+ RasMLLF distilied
ResNet
EfficientNetV2
VIT
VIT-
distilled
78
MLP Mixer distilled
ReplietY
XOT distilled
KCIT
-Eog(ECE)
A comparison of 523 models by their AUROC (x100, higher is better) and -log(ECE) (higher is better) on ImageNet. Each marker's s size is determined by the model's 's number of parameters. full version graph given in Figure Distilled models are better than non-distilled ones. subset of ViT models is superior to all other models for all aspects of uncertainty estimation ("ViT" in the legend, marked as a red triangle facing upwards); the performance of EfficientNet- V2 and GENet models is worse.
Figure 1:
are well known and are often used for comparing the uncertainty estimation performance of models (Moon et al., 2020; Nado et al., 2021; Maddox et al., 2019; Lakshminarayanan et al., 2017). Somewhat surprisingly, NLL, Brier, AURC, and ECE all fail to reveal the uncertainty superiority of Model B in our investment example (see Appendix A for the calculations). Both AUROC and SAC, on the other hand, reveal the advantage of Model B perfectly (see Appendix A for details). It is not hard construct counterexamples where these two metrics fails and others (e.g., ECE) succeed. To sum up this brief discussion, we believe that the ultimate suitability of performance metric should be determined by its context. If there is no specific application in mind, there is strong incentive to examine a variety of metrics, as we choose to do in this study.
This study evaluates the ability of 523 models from the Torchvision and Timm repositories (Paszke et al., 2019; Wightman, 2019) to estimate uncertainty'.. Our study identifies several major factors that affect confidence rankings, calibration, and selective prediction, and lead to numerous empirical contributions important to selective predictions and uncertainty estimation. While no new algorithm or method introduced in our paper, our study generates many interesting conclusions that will help practitioners achieve more powerful uncertainty estimation. Moreover, the research questions that are uncovered by our empirical study shed light on uncertainty estimation, which may stimulate the development of new methods and techniques for improving uncertainty estimation. Among the most interesting conclusions our study elicits are:
(1) Knowledge distillation training improves estimation. Training regimes incorporating any kind of knowledge distillation (KD) (Hinton et al., 2015) lead to DNNs with improved uncertainty estimation performance evaluated by any metric, more than by using any other training tricks (such as pretraining on larger dataset, adversarial training, etc.). In Galil et al. (2023) we find similar performance boosts for class-out cutsssooooof oot--o--disteee -fdissiibbtttion (C-OOD) detection.
(2) Certain architectures are more inclined to perform better or worse at uncertainty estima- tion. Some architectures seem more inclined to perform well on all aspects of uncertainty estimation, e.g.. subset of vision transformers (ViTs) (Dosovitskiy et al., 2021) and the zero-shot language- vision CLIP model (Radford et al., 2021), while other architectures tend perform worse, e.g.. EfficientNet-V2 and GENet (Tan & Le, 2021; Lin et al., 2020). These results are visualized in Figure I. Galil et al. (2023) we find that ViTs and CLIPs are also powerful C-OOD detectors.
(3) Several training regimes result in a subset of ViTs that outperforms all other architectures and training regimes. These regimes include the original one from the paper introducing ViTs (Dosovitskiy et al., 2021; Steiner et al., 2022; Chen et al., 2022; Ridnik et al., 2021). These ViTs
Our code is available at http://ccchheoooocollateeeeeaeee
2

---

Published as a conference paper at ICLR 2023
achieve the best uncertainty estimation performance any aspect measured, both in absolute terms and per-model size (# parameters, see Figures 9 and 10 in Appendix B).
(4) Temperature scaling improves selective and ranking performance. The simple post-training calibration method of temperature scaling (Guo et al., 2017), which is known to improve ECE, for the most part also improves ranking (AUROC) and selective prediction meaning not only does it calibrate the probabilistic estimation for each individual instance, but it also improves the partial order of all instances induced by those improved estimations, pushing instances more likely to be correct to have a higher confidence score than instances less likely to be correct (see Section 3).
(5) The correlations between AUROC, ECE, accuracy and the number of parameters pendent on the architecture analyzed. Contrary to previous work by (Guo et al., 2017), we observe that while there is a strong correlation between accuracy/number of parameters and ECE or AUROC within each specific family of models of the same architecture, the correlation flips between a strong negative and a strong positive correlation depending on the type of architecture being observed. For example, as DLA (Yu et al., 2018) architectures increase in size and accuracy, their ECE deteriorates while their AUROC improves. The exact opposite, however, can be observed in XCiTs (Ali et al., 2021) as their ECE improves with size while their AUROC deteriorates (see Appendix L).
de-
(6) The best model in terms of AUROC or SAC is not always the best in terms of calibration, as illustrated in Figure I, and the trade-off should be considered when choosing a model based on its application.
2 HOW TO EVALUATE DEEP UNCERTAINTY ESTIMATION PERFORMANCE
Let X' be the input space and y be the label space. Let P(X: )) be an unknown distribution over X' x y. A model f is a prediction function f : X' - V, and its predicted label for an image x is denoted by Dy (x). The model's true risk w.r.t P is R(flP) = Ep(x,y) ( [f[((x) v)]. where (: x y - R is a given loss function, for example, 0/1 loss for classification. Given a labeled set Sm (.) C (X' x Y), sampled i.i.d. from P( the empirical risk of model is f(S]Sm) - - ((.(),) yt). Following Geifman et al. (2018), for given model f we define a confidence score function x(I,,) 9lf). where & E X, and ÿ E y is the model's 's prediction for x, as follows. The function should quantify confidence in the prediction of ý for the input x, based on signals from model f. This function should induce partial order over instances in X.
The most common and well-known & function for a classification model f (with sofumax at its last layer) is its softmax response values: s(x, f) A f(x) (Cordella al., 1995; De Stefano et al., 2000). We chose to focus on studying uncertainty estimation performance using softmax response as the models' function because of its extreme popularity, and its importance as baseline due its solid performance compared to other methods (Geifman & El- Yaniv, 2017: Geifman et al., 2018). While this is the main K we evaluate, we also test the popular uncertainty estimation technique of Monte Carlo dropout (MC dropout) (Gal & Ghahramani, 2016), which is motivated by Bayesian reasoning. Although these methods use the direct output from f. λ could be a different model unrelated to and unable to affect f's predictions. Note that to enable probabilistic interpretation, i can only be calibrated if its values reside in [0, 1] whereas for ranking and selective prediction any value in R can be used.
A selective model f (El-Yaniv & Wiener, 2010; Chow, 1957) uses a selection function g X > {0, 1} to serve binary selector for f. enabling it to abstain from giving predictions for certain inputs. g can be defined by a threshold on the values of a K function such that go(a) K. f) = [K(a, ūy (z)) where coverage, defined as o(f.g) = Er(g(z)), is the probability mass of the non-rejected instances in X. The selective risk of the selective model (f, g) is defined as R(f.g) g) A E E These quantities can be evaluated empirically over finite labeled set Sm. with the empirical coverage defined as ó(f.gl
0). The performance of selective model is measured using coverage and risk,
*(f.g)
Em (o.la))).. Similarly, SAC is defined as the largest coverage available for a specific 1 m Em g(xx),, and the empirical selective risk defined as ((fgg(mm) &
o(f..g 9m)
accuracy constraint. A way to visually inspect the behavior of K function for selective prediction can be done using the risk coverco (RC) curve (El- Yaniv & Wiener, 2010) a curve showing the selective risk as function of coverage, measured on some chosen test set; see Figure 2 for an
3

---

Published as a conference paper at ICLR 2023
ResNet1 8
0.0
0C4
AURC 6.53
0.2
201
Coverage
Figure 2: The RC curve made by ResNet18 trained on CIFAR-10, measured on the test set. The risk is calculated using a 0/1 loss (meaning the model has about 95% accuracy for 1.0 coverage); the used was softmax-ree softmax-response. The value of the risk at each point of coverage corresponds to the selective risk of the model when rejecting inputs that are not covered at that coverage slice. e-g.. the selective risk for coverage 0.8 is about 0.5%, meaning that an end user setting a matching threshold would enjoy a model accuracy of 99.5% on the 80% of images the model would not reject.
example. In general, though, two RC curves are not necessarily comparable if one does not fully dominate the other (Figure shows an example of lack of dominance).
The AURC and E-AURC metrics were defined by (Geifman et al., 2018) for quantifying the selective quality of functions via single number, with AURC being defined the area under the RC curve. AURC, however, is very sensitive to the model's ss accuracy, and in an attempt to mitigate this, E-AURC was suggested. The latter also suffers from sensitivity to accuracy, as we demonstrate in Appendix C. The advantage of scalar metrics such as the above is that they summarize the model's overall uncertainty estimation behavior by reducing however, these reductions could result in a loss of vital information about the problem (recall the investment example from Section 1, which is also discussed in Appendix A: reducing an RC curve to AURC does not show that Model B has optimal 0 risk if the coverage is smaller than 0.4). Thus, the choice of the "correct" single scalar performance metric unfortunately must be task-specific. When comparing the uncertainty estimation performance of deep architectures that exhibit different accuracies, we find that AUROC and SAC can effectively "normalize" accuracy differences that plague the usefulness of other metrics (see Figure 3). This normalization is essential in our study where we compare uncertainty performance of hundreds of models that can greatly differ in their accuracies.
single scalar. When not carefully chosen,
For risk-sensitive deployment, let us consider the two models in Figure 3 EfficientNett-2--XX (Tan & Le, 2021) and ViT-B/32-SAM (Chen et al., 2022). While the former model has better overall accuracy and AURC (metrics that could lead us to believe the model is best for our needs), it cannot guarantee Top-1 ImageNet selective accuracy above 95% for any coverage. ViT-B/32-SAM, on the other hand, can provide accuracies above 95% for all coverages below 50%.
In applications where risk (or coverage) constraints are dictated (Geifman & EI-Yaniv, 2017), the most straightforward and natural metric is SAC (or selective risk), which directly measures the coverage (resp., risk) given at the required level of risk (resp., coverage) constraint. We demonstrate this in Appendix I, evaluating which models give the most coverage for an ambitious SAC of 99%. If instead a specific range coverages is specified, we could measure the area under the RC curve for Σ f(.. with C being those required coverages.
those coverages:
AURCo (n. fl....
Often, these requirements are not known can change as a result of changing circumstances or individual needs. Also, using metrics sensitive to accuracy such as AURC makes designing architectures and methods to improve k very hard, since improvement in these metrics could be attributed to either an increase in overall accuracy (if such occurred) or to a real improvement in the model's 's ranking performance. Lastly, some tasks might not allow the model to abstain from making predictions at all, but instead require interpretable and cell-calibrated probabilities of correctness, which could be measured using ECE.
4

---

Published as a conference paper at ICLR 2023
Disk-Coverage Curve Comparison
0.74
EfficientNet-V244 (trained on 21k and finetuned on 1k)
0.25
VIT-8/32-SAM
VIT-L/16-384
Top-1 ImageNet Error (Risk)
0.20
Bigger AUROC (x100)
Smaller AURC 000) better
0.85
AUROC=87.52 AURC-76.18
0.10
AUROC AUR0C-77.77
Accuracy 0.87
AURC=68.45 68.45
Error Rate 0.13
0.05
AUROC=88.49 AURC-26.44
0.00
0.0
0.2
0.4
0.6
0.8
1.0
Coverage
Figure 3: A comparison of RC curves made by the best (VTT-L/16-384) and worst (EfficientNet- V2-XL) models we evaluated in terms of AUROC. Comparing ViT-B/32-SA EfficientNet- V2 exemplifies the fact that neither accuracy nor AURC reflect selective performance well enough,
2.1 MEASURING RANKING AND CALIBRATION
A function is not necessarily able to change the model's predictions. Therefore, it can improve the selective risk by ranking correct and incorrect predictions better, inducing a more accurate partial order over instances in X. Thus, for every two random samples (æ1, (I2, 122 ~ P(X, 3') and given that (x()) 81) >> ((((2) y2), the ranking performance of x is defined as the probability that K ranks T2 higher than 21:
Pr(s(011 /l (l
(1((2)) v2)l
We discuss this definition in greater detail in Appendix D. The AUROC metric is often used in the field of machine learning. When the 0/1 loss is in play, it is known that AUROC in fact equals the probability in Equation (1) (Fawcett, 2006) and thus a proper metric to measure ranking in classification (AKA discrimination). AUROC is furthermore equivalent to Goodman and Kruskal's 7-correlation (Goodman & Kruskal, 1954), which for decades has been extensively used to measure ranking (known "resolution") in the field of metacognition (Nelson, 1984). The precise relationship between 7-correlation and AUROC is 7 2 AUROC 1 (Higham & Higham, 2018). We note also that both the 7-correlation and AUROC are nearly identical or closely related various other correlations and metrics; 7-correlation (AUROC) becomes identical to Kendall's T (up to linear transformation) in the absence of tied values. Both metries are also closely related rank-biserial correlation, the Gini coefficient (not to be confused with the measure from economics) and the MhnnnWhinney U text, hinting at their importance and usefulness in a variety fields and settings. In Appendix E, we briefly compare the ranking performance of deep neural networks and humans in metacognitive research, and in Appendix F we address criticism using AUROC to measure ranking.
The most widely used metric for calibration is ECE (Nacini et al., 2015). For a finite test set of size N. ECE is calculated by grouping all instances into m interval bins (such that < N), each of size 1. (the confidence interval of bin B, is (2-1 i D). With acc(B,) being the mean accuracy in bin B, and cBn being its mean confidence, ECE is defined as
B N
(mx( = yal
s(x, (x:))) Byl
ECE
Σ B,
|Byl
B
|aco(BB) conf(B,)
5

---

Published as a conference paper at ICLR 2023
Since ECE is widely accepted use it here evaluate calibration, and follow (Guo et al., 2017) in setting the number of bins to m = 15. Many altematives to ECE exist, allowing an adaptive binning scheme, evaluating the calibration (Nixon et al., 2019; Vaicenavicius et al., 2019; Zhao et al., 2020). Relevant to our objective is that by using binning, this metric is not affected by the overall accuracy is the Brier score (mentioned in Section 1), for example.
the non-chosen labels as well, and other various methods
3 PERFORMANCE ANALYSIS
Method
AUROC Improvement over Vanilla
Adversanial Training
Pretraining imaçefiet21k
Somi Supervised Leamiing (.0)
Distillaiion Dn
Temperature Scaling
MC-Dropout (2)
-2
(a)
Method
Improvemmntt over Vanilla
0.2
Adversarial Training un
Pretraining ImageNet21k (u)
Serni-Superiieed Learning
0.15
Distillatoon ca
Temperature Scaling
00.05
sometmmee
(b)
Figure 4: comparison of different methods and their improvement in terms of (a) AUROC and (b) ECE, relative to the same model's performance without employing the method. Markers above the x-axis represent models that benefited from the evaluated method, and vice versa. The numbers legend to the right of each method indicate the number of pairs compared. Temperature scaling can sometimes harm ECE, even though its purpose to improve it.
In this section we study the performance of 523 different models (available in timm 0.4.12 and torchvision 00). Note that all figures from analysis available interactive plotly plots in the supplementary material, which provides information about every data point.
1) Among the training regimes evaluated, knowledge distillation improves performance the most. We evaluated several training regimes: (a) Training that involves KD in any form, including Touvron et al. (2021b), knapsack pruning with distillation (in which the teacher is the original unpruned model) (Aflalo et al., 2020) and a pretraining technique that employs distillation (Ridnik al., 2021): ; (b) adversarial training (Xie al., 2020a: Tramèr et al., 2018); (c) pretraining on ImageNet21k ("pure", with no additions) (Tan & Le, 2021; Touvron et al., 202 la; 2022); and (d) various forms of weakly or semi-supervised learning (Mahajan et al., 2018; Yalniz et al., 2019; Xie et al., 2020b). To make fair comparison, only compare pairs of models such that both models have identical architectures and training regimes, with the exception of the method itself being evaluated (e.g., training with without knowledge distillation). More information about each data point of comparison is available in the supplementary material. Note that the samples are of various sizes due the different number of potential models available for each.
Of the methods mentioned above, training methods incorporating distillation improve AUROC and ECE the most. For example, looking at Figure 4a, it is evident that distillation (purple box) almost

---

Published as a conference paper at ICLR 2023
Models
[ BiT
57
B BTT dettlled
A DeiT cietiled
TCVSst
ECMSet didille
* trficaentivee
EMfieencee distited
FTT disiilled
M Regiter-- tocer
ReeHLLP dieiileed
KOI chatiiled
64-
-lgg (ECE)
Comparing teacher models (yellow markers) to their KD students (represented by markers with thick borders and dot). The performance of each model is measured in AUROC (higher is better) and -log(ECE) (higher better).
Figure 5:
always improves AUROC, and moreover, its median improvement is the best all techniques evaluated. The same observation can be made with regards to improving ECE; Figure 4b. Distillation seems to greatly improve both metrics even when the teacher itself is much worse at both metrics. Figure 5 nicely shows this by comparing the teacher architecture and the students in each case. Additionally, in a pruning scenario that included distillation in which the original model was also the teacher (Aflalo al., 2020), the pruned models outperformed their teachers. The fact that KD improves the model over its original form is surprising, and suggests that the distillation process itself helps uncertainty estimation. In Galil et al. (2023) we find that KD also improves C-OOD detection performance, measured by AUROC. We discuss these effects in greater detail in Appendix G.
2) Temperature scaling greatly benefits AUROC and selective prediction. Evaluations of the simple post-training calibration method of temperature scaling (TS) (Guo et al., 2017), which widely known to improve ECE without changing the model's accuracy, also revealed several interesting facts: (a) TS consistently and greatly improves AUROC and selective performance (see Figure 4a) meaning not only does TS calibrate the probabilistic estimation for each individual instance, but it also improves the partial order of all instances induced by those improved estimations. While TS is well known and used for calibration, to the best of our knowledge, its benefits for selective prediction were previously unknown.
(b) While TS is usually beneficial, it could harm some models (see Figures 4a and 4b). While it is surprising that TS-a calibration method-would harm ECE, this phenomenon explained by the fact that TS optimizes NLL and not ECE (to avoid trivial solutions), and the two may sometimes misalign. (c) Models that benefit from TS in terms
Tempersture Sealing (Temp. Teripo stue Scatin (Tonp.
AUNDC improvematt
Temperature Scaling [Temp- 1)
Temperature Scaling (Temp. >
Figure 6: Out of 523 models evaluated, models that were assigned temperature higher than I by the calibration process tended to degrade in AUROC performance rather than improve. Markers above the x-axis represent models that benefited from TS, and vice versa.
of AUROC tend to have been assigned a temperature smaller than by the calibration process (see Figure 6). This, however, does not hold true for ECE (see Figure 14 in Appendix H). This example
7

---

Published as a conference paper at ICLR 2023
also emphasizes the fact that improvements in terms of AUROC do not necessarily translate into improvements in ECE, and vice versa. (d) While all models usually improve with TS, the overall ranking of uncertainty performance between families tends to stay similar, with the worse (in terms of ECE and AUROC) models closing most of the gap between them and the mediocre ones (see Figure 13 in Appendix H).
3) A subset of ViTs outperforms all other architectures in selective prediction, ranking and calibration, both in absolute terms and per-model size. Several training regimes (including the original regime from the paper introducing ViT) Dosovitskiy et al. (2021); Steiner et al. (2022); Chen al. (2022); Ridnik et al. (2021) result ViTs that outperform all other architectures and training regimes in terms of AUROC and ECE (see Figure I; Figure 13 in Appendix H shows this is true even after using TS) as well as for the SAC of 99% we explored (see Figure and Appendix I). These ViT' also outperform all other models in terms of C-OOD detection performance (Galil et al., 2023). Moreover, for any size, VIT models outperform their competition in all of these metrics (see Figures 9 and 10 in Appendix B and Figure 15 in Appendix I).
Accuracy
Models
Various
AlexNet
20
B/T
DLA
RegNettX
65
M RegNetY
ResNet
SWSL ResNeXt
ViT
10
20
30
40
50
Coverage for Accuracy 99
Figure 7: Comparison of models by their overall accuracy and the coverage they are able to provide given a selective accuracy constraint of Top-1 99% ImageNet. A higher coverage is better. Only ViT models are able to provide coverage beyond 30% for this constraint. They provide more coverage than any other model compared to their accuracy or size. "Various" refers all other models (out of the 523) that were not mentioned by name.
Further research into other training regimes, however, reveals that not all training regimes result in superb performance (Touvron et al., 202 2022; Singh et al., 2022; Paszke et al., 2019) (these ViTs are dubbed "ViT"" in the figures), even when similar amount of data is introduced into the training and strong augmentations are used. In fact, the models trained by Chen et al. (2022) were not pretrained at all and yet reach superb ranking. Even the largest model introduced by Tran et al. (2022), which is large modified VIT that was pretrained on JFT-4B (a dataset containing 4 billion images) with the aim of excelling in uncertainty estimation (and other areas), is outperformed by the best ViT we evaluated: Plex L achieves achieve an AUROC 87.4 .7.4 and 86.7, respectively), compared to 88.5 achieved by ViT-L/16-384 that has less parameters and was pretrained on ImageNet-21k. In total, 18 ViTs trained on ImageNet-221 outperform? Plex L. among which are two variations of small ViTs (each with 36 or 22 million parameters). In Appendix we analyze the different hyperparameters and augmentations used for training the ViT models evaluated in this paper. Unfortunately, no clear conclusions emerge to explain the advantage of the successful training regimes. There is, however, ample evidence show that advanced augmentations are unlikely to be part of such an explanation.
AUROC of 87.7 (while its smaller versions, Plex M and Plex S,
The above facts suggest that the excellent performance exhibited by some ViTs cannot be attributed to the amount of data or to the augmentations used during training. These observations warrant
TTh authors had not released clear results for Plex ECE performance on ImageNet, making a comparison of calibration difficult. The authors mentioned that the average ECE of Plex L in CIFAR- 10, CIFAR 100 and ImageNet is slightly below 0.01. Our evaluations revealed six ViTs that achieved the same results, with the most calibrated model heing VYT-T/16 with ECE of 0.0054 on ImageNet.
8

---

Published as a conference paper at ICLR 2023
additional research with the hope of either training more robust ViTs transferring the unidentified ingredient of the successful subset of ViT' into other models.
4) Correlations between AUROC, ECE, accuracy and the model's size could either be positive or negative, and depend on the family of architectures evaluated. This observation contradicts previous smaller scale studies calibration. While AUROC and ECE are (negatively) correlated (they have Spearman correlation of -0.44, meaning that generally, as AUROC improves, so does ECE), their agreement on the best performing model depends greatly on the architectural family in question. For example, the Spearman correlation between the two metrics evaluated on 28 undistilled XCITs is 0.76 (meaning ECE deteriorates as AUROC improves), while for the 33 ResNets (He et al., 2016) evaluated, the correlation is -0.74. Another general observation is that contrary to previous work by (Guo et al., 2017) concerning ECF, the correlations between ECE and the accuracy or the number of model parameters are nearly zero, although each family tends to have a strong correlation, either negative or positive. We include family-based comparison Appendix I. for correlations between AUROC/ECE and accuracy, number of parameters and input size. These results suggest that while some architectures might utilize extra resources to achieve improved uncertainty estimation capabilities, other architectures do not and are even harmed in this respect.
5) The zero-shot language-vision CLIP model is well-calibrated, with its best instance outper- forming 96 % of all other models. CLIP (Radford et al., 2021) enables zero-shot classification and demonstrates impressive performance. We find it is also inclined to be well-calibrated. See Appendix K for details about how with a linear-probe added to it, and obtains an ECE of 1.3%, which outperforms 96% of models evaluated. Moreover, for their size category, CLIP models tend to outperform their competition in calibration, with the exception of ViTs (see Figure 10 in Appendix B). While this trend is clear for zero-shot CLIPs, we note that some models' calibration performance deteriorates with the addition of a linear-prooee Further research is required to understand the ingredients of multimodal models' contribution to calibration, and to find ways to utilize them to get better calibrated models. For example, could multimodal pretraining regime be used to get better calibrated models?
CLIP. The most calibrated CLIP is based ViT-B/32
6) MC dropout does not improve selective performance, in accordance with previous works. We evaluate the performance of MC dropout using predictive entropy its confidence score and 30 dropout-enabled forward passes. We do not measure its effects on ECE since entropy scores do not reside in [0, 1]. Using MC dropout causes consistent drop in both AUROC and selective performance compared with using the same models with softmax as the k (see Appendix M and Figure 4a). MC dropout's underperformance was also previously observed in (Geifman & EI- -aniy, 2017). We note, however, that evaluations we have conducted in Galil et al. (2023) show that MC dropout performs well when dealing with C-OOD data.
4 CONCLUDING REMARKS
We presented comprehensive study of the effectiveness of numerous DNN architectures (families) in providing reliable uncertainty estimation, including the impact of various techniques on improv- ing such capabilities. Our study led to many new insights and perhaps the most important ones are: (1) architectures trained with distillation almost always improve their uncertainty estimation performance, (2) temperature scaling is very useful not only for calibration but also for ranking and selective prediction, and (3) no DNN (evaluated in this study) demonstrated an uncertainty estimation performance comparable-ing any metric tested-to a subset of ViT models (see Section 3).
Our work leaves open many interesting avenues for future research and we would like to mention a few. Perhaps the most interesting question why distillation is so beneficial in boosting uncertainty estimation. Next, is there an architectural secret in vision transformers (VIT) that enables their uncertainty estimation supremacy under certain training regimes? This issue especially puzzling given the fact that comparable performance is not observed in many other supposedly similar transformer- models that we tested. If the secret not in the architecture, what is the mysterious ingredient of the subset of training regimes that produces such superb results, and how can it be used to train other models? Finally, can we create specialized training regimes (e.g., Geifman & El- Yaniy (2019)), specialized augmentations, special pretraining regimes (such as CLIP's multimodal training regime) or even specialized neural architecture search (NAS) strategies that can promote superior uncertainty estimation performance?

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research was partially supported by the Israel Science Foundation, grant No. 710/18.
We thank Prof. Rakefet Ackerman for her help with understanding how uncertainty estimation performance evaluated for humans in the field of metacognition, and for her useful comments for Appendix E.
REFERENCES
Rakefet Ackerman, Avi Parush, Fareda Nassar, and Avraham Shtub. Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior, 14:1011113, January 2016. doi: 10.1016f.chb.2015.077. URL https:
//doi.org/ htt..//gg1..0116..cc.22010.....0
Rakefet Ackerman, Avigdor Gal, Tomer Sagi, and Roee Shraga. A cognitive model of human bias in matching. PRICAI 20/9: Trends Artificial Intelligecce pp. 632-646. Springer International Publishing, 2019. doi: 10. 0007/978-3-000-299000 8_50. URL https://ooi.or//10
1007/978-3 1777888--30-29900000 29908-8-500 8_50.
Yonathan Aflalo, Asaf Noy, Ming Lin. Itamar Friedman, and Lihi Zelnik-Manor. Knapsack pruning with inner distillation. CoRR, fts/2002.08258, 2020. URL https://anivvoor//ah/20022..22
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Xeit: Cross- covariance image transformers. Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neurul Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 20014-20027, 2021. URL https:/pooceeding..euii...co//o Abstract.html.
Alexandra Basile, Maggie E. Toplak, and Brendan F. Andrade. Using metacognitive methods to examine emotion recognition in children with ADHD. Journal of Attention Disorders, 25(2): 245-257, November 2018.
doi: 10.1177/107055771888666// URL https://doi.rr/10.1177
10870547188088
Glenn W. Brier. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1):1, January 1950. doi:
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resmets without pre-training strong data augmentations. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
C. K. Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-6(4))247-254, 1957, doi: 10.1109/TEC. 195.5222359......5.....
L. P. Cordella, C. De Stefano, F. Tortorella, and M. Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neural Networks, 6(5): 140-117,, 1995. doi: 10. 109/72.41033
C. De Stefano, C. Sansone, and M. Vento. To reject or not to reject: that is the question-an answer in case of neural classifiers. IEEE Transactions on Systems. Man, and Cybernetics, Part C (Applications and Reviews), 30(1):84-94, 2000. doi: 10. 10.1109/5326.827457
Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. Evaluation of neural network uncertainty estimation with application to resource-constranee platforms. CoRR, abs/1903 abs/1903,00005, 2019, URL http://arxiv.org/abs/ 020000
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http:://wppomeieeeeeee////oee/ee
10

---

Published as a conference paper at ICLR 2023
Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
Tom Fawcett. An introduction to roe analysis. Pattern Recognition Letters, 27(8):661-874, 2006. ISSN 0167-8655. doi: https://doi.ooo wrtp://0i../11111101110100000000 http::/..aatee......o.... URL http:://ww.
ROC Analysis in Pattern Recognition.
K. Fiedler, Rakefet Ackerman, and Chiara Scarampi. metacognition Monitoring and controlling one own knowledge, reasoning and decisions. 2019.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In 9th International Conference on Learning Repre- sentations, ICIR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
https://peeneviieeee/f//mu////ip wtmp://ddmmmmmmiiii..o Ippoirr..
Jonathan Frankle and Michael Carbin, The lottery ticket hypothesis: Training pruned neural networks. CoRR, 0oos/8803.03635 2018. URL http://arzxvvorr/abb// (0038883330333 006335.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1050-1059. JMLR.org, 2016. URL http://proceediing..ii www.s//rre//sse.... ht.ml.
Ido Galil, Mohammed Dabbah, and Ran El- Yaniv. A framework for benchmarking class-out-of- distribution detection and its application imagenet. In International Conference on Learning Representations, 2023. URL wttps///ppeneeiee.eeeffrooe http://wwww/ooon//dddduu/////aaa lid-luubbbWoJJ..
Yonatan Geifman and Ran EI- Yaniv. Selective classification for deep neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4878-4887, 2017. URL 4a8423d5e9 1dd442235e1110000b77774444444 1cta0006744644402226660000 Abstrfftf--stttttthhhh
www.r/ceddinn.eeurss.ccppppppppe //hss//
Yonatan Geifman and Ran El- Yaniv. Selectivenet: A deep neural network with an integrated reject option. CoRR, abs/1901 1901/991..00 1.001909,22 2019. URL http://arxxivorr//bbb/ 1911.09010000 192...
Yonatan Geifman, Guy Uziel, and Ran EI- Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations, 2018
Leo A. Goodman and William H. Kruskal. Measures of association for cross classifications. Journal of the American Statistical Association, 49(268) December 1954. doi: 10.1080/0162145, URL
1ttp://00100000001111111100000 19.5:/2111...91.1000000111 1954.10501231.
Thomas D. Griffin, Jennifer Wiley, and Keith W. Thiede. The effects of comprehension-test expectancies on metaconprehension accuracy. Journal of Experimental Psychology: Learn-
ing, Memory, and Cognition,
June 2019. doi: 10.1037/xlm00006344 URL
https://doi.org/10 100.////////////// 1037/x/m0000634.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-1 I August 2017 vol- ume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 2017. URL http://pocceedins. wttpppsees//7//o////a0
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEF Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2116
URL https:://oi.ogg/////tt wwwgg//.../////P......
Philip A. Higham and D. Paul Higham. New improved gamma: Enhancing the accuracy of good- man kruskal'ss gamma using ROC curves. Behavior Research Methods, 51 108)):008-11 125, September 2018. doi: 13488-088888818111111111 URL htt.:ttsss/ddoorrr/00..3////// 3428-018- 1125-5.
11

---

Published as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiehl Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3. CoRR, abs/1905.02244, 2019. URL http://arxxx.orr/abs// http:////aaa////0224....
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Isabelle Guyon, Ulriee von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6402-6413, 2017. URL https:
wwwooeedings.neuripsccooooo 77aa220000700009912222222 (04 hbte://aSfaaacccceeeesaatttttta
Ming Lin, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Neural architecture design for gpu-efficient networks, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL http:///wwpnrreeeeww.e/e//nn/nns
Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wil- son. A simple baseline for bayesian uncertainty in deep learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 13132-13143, 2019. URL 11892eeba23cc2296566277886666660 waw.:oe22-bbbrta.bbbbaa Abstract.html.
9/hash/
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II, volume 1 11206 of Lecture Notes in Computer Science, pp. URL http:///oooor//1....
185-201. Springer, 2018. doi: 10. 0100//978-3-0000022----- 978-3-030-0121--8 12.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7034-7044 PMLR, 2020. URL http://proceedings.mlrs
19/moon20ahttm.
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael W. Dusenberry, Sebastian Far- quhar, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim G. J. Rudner, Yeming Wen, Florian Wen- Jasper Snoek, Yarin Gal, and Dustin Tran. Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning, CoRR, abs/2106.04011 04015, 2021. URL https:/aaxxivorr//bb/222 060551
zel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAA/ Conference on Artificial Intelligence, AAAI 15. pp. 2901-2907. AAAI Press, 2015. ISBN 026251 1290.
Niv Nayman, Yonathan Aflalo, Asaf Noy, and Lihi Zelnik. Hardcore-nass Hard constrained differ- entiable neural architecture search. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 7979-7990. PMLR, 2021. URL http://prroceeding..mii wwr.s..sss................... 39/nayma22 ht.m.
Thomas O. Nelson. A comparison of current measures of the accuracy of beeling-of-knowiiee predictions. Psychological Bulletin, 95(1):109-133 1984. doi: 10.1037/0 29...99..999999. URL http:///wi.ay////000/3/003200000
12

---

Published as a conference paper at ICLR 2023
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In IEEE Conference on Computer Vision and Pat- ter Recognition Workshops, CVPR Workshops 2019, Long Beach, CA. USA, June 16-20, 2019, pp. 38-41, Computer Vision Foundation / IEEE, 2019. URL. http://openaccess.heeeee Com/content_CCPPR 2019//tm///nnne//e/ee Makon.MeasuringgCClbbara Cawibrationaannnnn in www.Deeeepeeanngccccc wewppp..aainngcccRW....... CWW. paper..im....mm.
www.stddsssnesssss.eeeeeea in Deep..isuall Lisual_Learning.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. ('lché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems 32, pp. 8024-808 8035, Curran Associates, Inc., 2019. URL http://papers.neurippe www.ine.uurccoooooooo cccppp/.cc//papr// 9015-pytorch- an- imperative-style-iiieee wewighheeefrommnc-eeeee www.ddeereeee-------eee learning-librayy.dd.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 202 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 2021. URL http://proocediings..o mir. press/v wttps//eee/ssaaeeeeee 139/radfor22 12.2tml.
Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik. Imagenet-22 21k pretraining for the masses. In Joaquin Vanschoren and Sai-Kit Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks I, NeurlPS Datasets and Benchmarks 2021, December 1,21, virtual, 2021. URL http::/dattasets-eeenihi weenchmarks-pooceddnng paper/2021/hash/999 /ttp://////n/1111100011444777770
ccc/
(oweeeeo6660688--e- Abstract roound .html.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 45200 Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00474. URL http://openaccess. www.vv..cm//coneee.cc evpr 201 8/html/Sandler MobileNetV2. L01iss_CCRRR22000 201PPR_2218_apperhma
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun ), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:///wwiii.rr//////////0//00
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross B. Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. CoRR, abs/2201 100371,, 2022. URL hutps//anxiv.og/aa//0000.00
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. URL https://openneeieww.ee//
forum
Mingxing Tan and Quoe V. Le. Efficientnetv2: Smaller models and faster training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. (009--10106. PMLR, 2021. URL http://procceeding..mii press// hatl//11//ttnt a.hmml.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. CoRR, abs/2105.03404, 202 la. URL https://arriv.oggaa///11.......
13

---

Published as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICMI 2021, 18-24 July 2021, I, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 2021b. URL http://ppooceedingg.liippeeee 139/oouvon1/// html.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit III: revenge of the vit. CoRR, abs/2204.077188 2022. doi: 0085500arXxi.2244..00 URL http://db...ogg/1144888800a1111
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference Learning Representations, 2018. URL
RZ.
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang. Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton,, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions. CoRR, abs/2207.00 17711, 2022. doi: 10.48550/rXiw.22077 07411. URL https://bbi.org/104885500 2207.07411.
Monika Undorf and Arndt Bröder. Cue integration in metamemory judgements strategic. Quarterly Journal of Experimental Psychology, 73(4):629-642, October 2019. doi: 10.1177/
1747021819882308. URL https://bb..rrg/1..1171/11111111 wttp://////011111770077000000000 11988330.
Juozas Vaicenavicius, David Widmann, Carl R. Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B. Schön. Evaluating model calibration in classification. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), The 22nd International Conference Artificial Intelligence and Statis- tics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 1459-34677 PMLR, 2019. URL http://proceedingg. mlr, press/v89/ press//// vaicenavicius ht.l
Ross Wightman. Pytorch image models. http://gihhhbb.om///viigaaaaaaaa http:///wwwwwweaoooooooeeeemeeee image- models, 2019.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoe V. Le. Ad- versarial examples improve image recognition. In 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 816-825. Computer Vision Foundation / IEEE, 2020a. doi: 10. 2.110//VP42200022000 00090. URL https://openaccess. www.saawwcccss.....eeeeeceem.ec. wttp://.co//ooeeetccoom CCw.ccCCP.R.... CVVPR_2020htm//// Adversarial Improve Impaoee_Imaae.aae Recognition CVPR_2020 200.22200ppppthtmm paper.html.
Adz0haaa/XieeAAleeraaaaall ExaeesarialEEEamples....
Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoe V. Le. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vi- sion and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 10684- 10695. Computer Vision Foundation / IEEE, 2020b. doi: 10.109/CVPR422002222 20.0000000000000000... URL https://openacces.tteee www.cccf..o//contenc CVPR 2020/html/Xie Self- ooainirain With Noisy Student Improves ImageNet Classification CVPR 2020 paper.html.
I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification, 2019.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2403-2412, 2018, doi: 10.1 10.1100/CVP..2018..000
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 11387-11397 PMLR. 2020. URL http://proceedings.lrrrees 19/zhao20e htmll
14

---

Published as a conference paper at ICLR 2023
A THE INVESTMENT EXAMPLE
Let us consider two classification models for the stock market that predict whether a stock's 's value about to increase, decrease or remain neutral (three-class classification). Suppose that Model A has a 95% true accuracy, and generates a confidence score of 0.95 on any prediction (even on misclassified instances); Model B has 40% true accuracy. but always gives a confidence score of 0.6 on correct predictions, and 0.4 on incorrect ones. We now try and evaluate these two models using the uncertainty metrics mentioned in Section to see which can reveal Model B's superior uncertainty estimation performance. AURC will fail due to its sensitivity to accuracy (the AURC of Model B is 0.12, more than twice as bad as the AURC for Model A, which is 0.05). NLL will rank Model A four times higher (Model A's NLL is 0.23 and Model B's is 0.93). The Brier score would also much prefer Model A (giving it a score of 0.096 while giving Model B score of 0.54). Evaluating the models' calibration with ECE will also not reveal Model B's advantages, since it is less calibrated than Model A, which has perfect calibration (Model A has an ECE of O, and Model B has a worse ECE of 0.4).
AUROC, on the other hand, would give Model B perfect score of 1 and terrible score of 0.5 to Model A. The selective risk for Model B would be better for any coverage of stock predictions below 40%, and for any SAC above 95% the coverage for Model A would be 0, but 0.4 for Model B.
Those two metrics are not perfect for any example. Let us instead compare two different models for the task of predicting the weather when we cannot abstain from making predictions. Accordingly, being required to provide an accurate probabilistic uncertainty estimation of the model's predictions, AUROC and selective risk would be meaningless (due to the model's 's inability to abstain in this task), but ECE or the Brier Score would better evaluate the performance the new task requires.
B RANKING AND CALIBRATION VISUAL COMPARISON
A comparison of 523 models by their AUROC (x 100, higher is better) and -log(ECE) (higher is better) on ImageNet is visualized in Figure 8. An interactive version of this figure provided as supplementary material. To compare models fairly by their size, we plot two graphs with the logarithm of the number of parameters as the X-axis, so that models sharing the same x value can be compared solely based on their y value. In Figure 9 we set the X axis to be AUROC (higher is better), and see ViTs outperform any other architecture with a comparable amount of parameters by large margin. We can also observe that using distillation creates consistent improvement in AUROC. In Figure 10 we set the X axis to be the negative logarithm of ECE (higher is better) and observe a very similar trend, with ViT outperforming its competition for any model size.
C DEMONSTRATION OF E-AURC'S DEPENDENCE ON THE MODEL'S S ACCURACY
EXCess-AURC (E-AURC) was suggested by Geifman et al. (2018) as an alternative to AURC (explained in Section 2). To calculate E-AURC, two AURC scores need to be calculated: (1) AU RC the AURC value of the actual model and (2) AU (C(odeel"), the AURC value of hypothetical model with identical predicted labels as the first model, but that outputs confidence values that induce a perfect partial order on the instances in terms of their correctness. The latter means that all incorrectly predicted instances are assigned confidence values lower than the correctly predicted instances.
E-AURC is then defined as AU (moddel)) - AU RC (model")) ). In essence, this metric acknowledges that given a model's iss accuracy, the area of AU RC ) is always unavoidable no matter how good the partial order is, but anything above that could have been minimized the a function was better at ranking, assigning correct instances higher values than incorrect ones and inducing better partial order over the instances.
This metric indeed helps to reduce some of the sensitivity to accuracy suffered by AURC, and for the example presented in Section 1, E-AURC would have given a perfect score of to the model inducing perfect partial order by its confidence values (Model B). It is easy, however, to craft examples showing that E-AURC prefers models with higher accuracy, even if they have lower or equal capacity to rank.
15

---

Published as a conference paper at ICLR 2023
GU Uhart the
NOS hosken
14. muraa
(ECE)
4
88
86
84
82
80
78
Figure 8: A comparison of 523 models by their AUROC (x 100, higher is better) and log(ECH) (lower better) on ImageNet. Each marker's size determined by the model's number of parameters. Each dotted marker represents distilled version of the original. An interactive version of this figure is provided as supplementary material.
16

---

Published as a conference paper at ICLR 2023
88
86
AUROC
84
Models
82
Various
ResMLP
AlexNet
+ ResMLP distilled
BIT
ResNet
distilled
80
EfficientNetV2
VIT
GENet
VIT*
MLP Mixer
VIT* distilled
MLP Mixer distilled
XCIT
78
RegNetY
XCIT distilled
6
6.5
7
7.5
8
8.5
9
log(# (arammarm))
Figure 9: A comparison of 523 models by their AUROC (x 100, higher is better) and log(number of model's parameters) on ImageNet. Each dotted marker represents distilled version of the original.
Models
Various
OllexNet
D BIT
B BIT distilled
Linear Probe
CLIP Zero-Shot
M
A EfficientNetV2
-OCg(ECE)
GENet
+ MLP
MLP Mixer distilled
MRegNetY
<>ResMLP
ResMLP distilled
ResNet
VIT
VIT
distiileed
XCIT
XCIT distilled
(aaaaParamtters)
Figure 10: A comparison of 523 models by their -log(ECE) (higher is better) and log(number of model's parameters) on ImageNet. Each dotted marker represents distilled version of the original.
To demonstrate this in simple way, let us consider two models with complete lack of capacity to rank correct and incorrect predictions correctly, always outputting the same confidence score. Model A has an accuracy of 20% (thus an eтror rate of 80%), and Model B has an accuracy of 80% (and an eтror rate of 20%). good ranking metric should evaluate them equally (the same way E-AURC gives the same score for two models that rank perfectly regardless of their accuracy). In Figure 11 we plot their RC curves with dashed lines, which are both straight lines due to their lack of ranking ability. We calculate both of these models' AURCs, AU (C(odeelA) 0.8, AU RC((modell))
0.2.
The next thing to calculate is the best AURC values those models could have achieved given the same accuracy if they had perfect partial order. We plot these hypothetical models' RC curves in Figure as solid lines. Their selective risk remains O for every coverage below their total accuracy, since these hypothetical models assigned the highest confidence to all of their correct instances first. As the coverage increases and they have no more correct instances to select, they begin to give instances that are incorrect, and thus their selective risk deteriorates for higher coverages.
17

---

Published as a conference paper at ICLR 2023
0.0
0.8
Model
Model
0.7
Hypothelical optimal curve RC surve
Risk
0.5
Coverage
Figure 11: The RC curves for Models A and B are plotted with dashed lines. The RC curves for the hypothetically optimal versions of Models A and are plotted with solid lines.
Calculating both these hypothetical models' AURCs gives us the following: AU AC 0.482, AU (B) 0.022. Subtracting our results we get: E-AURC(modelA) 0.8 0.482 0.318, E-AURC(modeel 0.2 0.022 0.178 Hence, E-AURC prefers Model B over Model A, even though both do not discriminate at all between incorrect and correct instances.
D MORE ON THE DEFINITION OF RANKING
Let us consider finite set Sm {(z,, =) Px,.. We assume that there two identical values given by & on Sn. Such an assumption is reasonable when choosing a continuous confidence signal.
We further denote e as the number of concordant pairs (i.e., pairs in Sm that satisfy the condition [m( flf) < n(z,, ūlf) n (xf(),) 34) ((f(z;) v,)) and d as the number of discordant pairs (i.e., pairs in Sm that satisfy the condition [k( 9 fll > k(w;, Gll) n ((J(s))) y1) > ((((z;),
We assume, for now, that there further develop Equation (1) from Section 2. using the definition of conditional probability.
identical values given by l on S.. Accordingly,
Pr|m(2,,01) < (f))((( (f) Pr|m(æ,, flf) (l) > (f((z)),)) Pr(((f(z:), > {(f(z;),
which can be approximated empirically, using the most likelihood estimator, as
C
m
2
We note that the last equation is identical to Kendall's 's up linear transformation, which equals
18

---

Published as a conference paper at ICLR 2023
C- d +
C
(
2
2c
(c d)
2c
c d
m
10
(13
2
2
2
1 = 2
[Equation 2] - 1.
2
Otherwise, if the loss assigns two identical values to pair of points in Sm, but x. does not, then we get:
C
(3)
c d'
which is identical to Goodman & Kruskal's c-orrelation up to linear transformation
C- d c c
(c+
C c+ d c+d
c+d
2c c
= 2. Eqquatin 3] 1.
c+d
c c+ d
D.1 INEQUALITIES OF THE DEFINITION
One might wonder why Equation (1) should have strict inequalities rather than non-strict ones to define ranking. As we discuss below, this would damage the definition:
(I) If the losses had non-strict incquality:
Pr[s(11,0)) < x( ()())(((()))))))))))))))))))))() v1) 2
Consequently, the case of classification, for example, this probability would increase for any pairs consisting of correct instances with different confidences. This would yield no benefit in ranking between incorrect and correct instances and motivates giving different confidence values for instances with the same loss a fact that would not truly add any value.
(2) If the values had non-strict inequality:
Pr(x(1 @I ((
> {(((2))
This probability would increase for any pair (I1, 22) such that 6(I1, Ølf) k(I2, and {(((x))) ((f(z))). although should have ranked 21 with lower value. Furthermore, if function were to assign the same confidence score to all I E X, then when there are no two identical values of losses, the definition's 's probability would be 1: otherwise, the more different values for losses there are, the larger the probability would grow. In classification with a 0/1 loss, for example, assigning the same confidence score to all instances would result in the probability being Accuracy(() (1 Accuracy((F) which is largest when Accuracy 0.5.
E RANKING CAPACITY COMPARISON BETWEEN HUMANS AND NEURAL NETWORKS
In the field of metacognition, interestingly, the predictive value of confidence is evaluated by two different aspects: by its ability to diseriminate between correct and incorrect predictions (also known as resolution in metacognition or ranking in our context) and by its ability to give well-calibrated confidence estimations, not being over- under-confident (Fiedler et al., 2019). These two aspects correspond perfectly with much of the research done in the deep learning field, with the nearly matching metric AUROC of y-correlation (see Section 2).
This allows us to compare how well humans rank predictions in various tasks versus how well models rank their own in others. Human AUROC measurements in various tasks (translated from
19

---

Published as a conference paper at ICLR 2023
y-correlation) tend range from 0.6 to 0.75 (Undorf & Bröder, 2019; Basile et al., 2018; Ackerman et al., 2016), but could vary, usually towards much lower values (Griffin et al., 2019). In our comprehensive evaluation on ImageNet, AUROC ranged from 0.77 to 0.88 (with the median value being 0.85), and in CIFAR-10 these measurements jump to the range of 0.92 to 0.94.
While such comparisons between neural networks and humans somewhat unfair due to the great sensitivity required for the task, research that directly compares humans and machine learning algorithms performing the same task exist. For example, in Ackerman et al. (2019), algorithms far surpass even the group of highest performing individuals in terms of ranking.
F CRITICISMS OF AUROC AS A RANKING METRIC
In this section, addressing the criticism of AUROC as a ranking metric, we show why AUROC does not simply reward models for having lower accuracy, The paper by Ding et al. (2019) presented a semi-artificial experiment to demonstrate that AUROC might get larger the worse the model's s accuracy becomes. They consider model f and its x function evaluated
a classification test set X' giving each a prediction ūy (x) and a confidence score <(2; 0j (x)).)) which in this case is the model's softmax response. Let X {xf E (l)y()))
y(x)) be the set of all instances correctly predicted by the model f, and define T(o) E X to be the correct instance that received
the i-lowest confidence score from K. Their example continues and considers an artificial model fd to be an exact clone of f with the following modification: for every i S m, the model f now predicts different, incorrect label for =(g; however, its given confidence score remains identical: x(2%) ūr (1())) = x( Uym (r())) for is exactly identical to f. by this definition, not changing any predictions. The paper shows how an artificially created model fr obtains a higher AUROC score the bigger its m. This happens even though "nothing" changed but hit to the model's accuracy performance (by making mistakes on more instances).
First, understand why this happens, let consider fl: AUROC for increases the more pairs of [m(x1) < x((2)) ((x2))0g((e (x1) + y(æ1), Ūy (x2) = y[æ2)] there are. The model f is now giving an incorrect classification to os1y. but this instance's 's position in the partial order induced by has remained the same (since m(2(1)) is unchanged); therefore, | correctly ranked pairs were added: (s(2(1)) < x(z(g))0 ( A (((r))) this does not guarantee an increase to AUROC by itself: if, previously, all pairs of (correct, instances were ranked correctly by K, AUROC would already be 1.0 for f and would not change for fl. If AUROC for f' is higher than it was for fo, this means there exists at least one instance x" that was incorrectly predicted by the original model få such that m(x(1)) < <(x"). Every such originally wrongly ranked pair (by fr) of [m(x[1)) ((=""))r(=") + y(x") "), (0())))) v(*(,)) ) has been climinated by fl wrongly predicting T(1y- This, therefore, causes AUROC to increase at the expense of the model's 's accuracy.
), dy (afo)) s(*;,)) for every 1 < i S 1. Nevertheless,
Such an analysis neglects many factors, which is probably why such an effect is only likely to be observed in artificial models (and not among the actual models we have empirically tested):
1. It is unreasonable to assume that the confidence score given by will remain exactly the same for an instance z%, given now has a different prediction. In the case of x being softmax, it assumes the model's logits have changed in a very precise and nontrivial manner. Additionally, by broad definition of K. which allows entirely different model than f. K receives the prediction and model as a given input (and cannot change affect either), and it is unlikely to assume changing its inputs will not change its output.
even be produced from an
2. Suppose we find the setting reasonable and assume we can actually create a model fr as described. Let us observe a model fr such that = min. (AUROC of f"==)), meaning that f ranks its predictions perfectly, unlike the original f Is it really true that f2 has no better uncertainty estimation than f Model f behaves very much like the investment in "Model B" from our example in Section 1, possessing perfect knowledge of when it is wrong and when it is correct, allowing its users risk-free classification. So, given a model f. we can use the above process to produce improved model fP, and then we can even calibrate its
20

---

Published as a conference paper at ICLR 2023
K to output 0% for all instances below its threshold and 100% for all those above to produce a perfect model, which might have a small coverage but correct every time, knows it and notifies its user when truly knows the prediction. The increase in AUROC reflects such an improvement.
Not only do we disagree with such an analysis and its conclusions, but we also have vast empirical evidence to show that AUROC does not prefer lower accuracy models unless there a good reason for it to do fact, out of the 523 models we tested, the model with the highest AUROC also has the 4th highest accuracy of all models, and the overall Spearman correlation between AUROC and accuracy of all the models we tested 0.03. Furthermore, Figure also exemplifies why AURC, which was suggested by the just mentioned paper the alternative AUROC, is bad choice single number metric, and might lead us to deploy a model that has a worse selective risk for most coverages only due to its higher overall accuracy.
demonstrate in Figure 3 (comparing
SencienntNet-2--X to VTT-B/32-SAM. In
G KNOWLEDGE DISTILLATION EFFECTS ON UNCERTAINTY ESTIMATION
88
Models
BIT
BiT distilled
DeiT
DeiT distilled
ECANet
ECANet distilled
EfficientNet
EfficientNet distilled
AUROC
MLP Mixer
MLP Mixer distillied
MobileNetV3
MobilioNett3 distilled
PIT
PIT distilled
ResMLP
84
ResMLPP distilled
TResNet
distilled
XCTT
83
distilled
(ECE)
Figure 12: Comparing vanilla models to those incorporating KD into their training (represented by markers with thick borders and dot). In pruning scenario that includes distillation, yellow markers indicate that the original model was also the teacher. The performance of each model is measured in AUROC (higher is better) and -log(ECE) (higher is better).
Figure 12 compares vanilla models to those incorporating KD into their training (represented by markers with thick borders and dot). In pruning scenario that includes distillation, yellow markers indicate that the original model was also the teacher (Aflalo et al., 2020). While distillation using different model tends to improve uncertainty estimation in both aspects, distillation by the model itself seems to improve only one suggestin it is generally more beneficial to use different model as teacher. The fact that KD improves the model over its original form, however, surprising, and implies that the distillation process itself helps uncertainty estimation. Note that although this specific method involves pruning, evaluations of models pruned without incorporating distillation (Frankle Carbin, 2018) revealed no improvement.
seems, morcover, that the teacher does not have to good in uncertainty estimation itself; Figure in Section 3 shows this by comparing the teacher architecture and the students in each case.
While the training method by Ridnik et al. (2021) included pretraining on ImageNet-211 and demonstrated impressive improvements, comparison of models that were pretrained on ImageNet2 (Tan & Le, 2021; Touvron et al., 2021a; 2022) with identical models that were not pretrained showed only slight improvement in ECE, and, in fact, exhibit degradation of AUROC (see
21

---

Published as a conference paper at ICLR 2023
Figures 4a and 4b in Section 3). This suggests that pretraining alone does not improve uncertainty
estimation.
H
MORE INFORMATION ABOUT TEMPERATURE SCALING
AUROC with Temperature Scaling
88
86
84
Models
82
Various
+ ResMLP
AlexNet
+ ResMLP distilled
ResNet
BIT distilled
EfficlentNetv2
HardCoReNAS
VIT+
MLP Mixer
VIT distilled
MLP Mixer distilled
1 RegNety
distilled
7
-Eog( with Temperature Scaling
Figure 13: A comparison of 523 models after being calibrated with TS, evaluated by their AUROC (x (x100, higher better) and -log(ECE) (higher better) on ImageNet. Each marker's size is determined by the model's number of parameters. VIT models are still among the best performing architectures for all aspects of uncertainty estimation.
In Figure 13 we see how temperature scaling (TS) affects the overall ranking of models in terms of AUROC and ECE. While the ranking between the different architecture remains similar, the poorly performing models are much improved and minimize the gap between them and the best models. One particularly notable exception is HardCoRe-NAS (Nayman et al., 2021), with its lowest latency versions becoming the top performers in terms of ECE. In addition, models that benefit from
0.4
Method
Temperature Scaling (Temp. 1)
0.35
Temperature Scaling (Temp. 1)
ECE Improvement over Vanilla
0.25
0.15
0.1
0
Temperature Scaling (Temp.
Temperature Scaling (Temp. > 1)
Figure 14: Here the relationship between temperature and the success of TS, unlike the case for AUROC, seems unrelated.
TS in terms of AUROC tend have been assigned temperature lower than by the calibration
22

---

Published as a conference paper at ICLR 2023
process (see Figure 6 in Section 3). The same, however, does not hold true for ECE (see Figure 14). This example also emphasizes the fact that models benefiting from TS terms of AUROC do not necessarily benefit in terms of ECE, and vice versa. Therefore, determining whether calibrate the deployed model with TS is, unfortunately, a task-specific decision.
We perform TS
suggested in Guo al. (2017). For each model we take random stratified sampling of 5,000 instances from the ImageNet validation set on which to calibrate, and reserve the remainder 45, 50000 instances for testing. Using the box-constrained L-BFGS Limited-Memorg Saaddee--eecche---Glaaa--------
algorithm, we optimize for 5,000 iterations (though fewer iterations usually converge into the same temperature parameter) using a learning rate of 0.01.
I ARCHITECTURE CHOICE FOR PRACTICAL DEPLOYMENT BASED ON SELECTIVE PERFORMANCE
As discussed in Section 2, when we know the coverage risk we require for deployment, the most direct metric to check is which model obtains the best risk for the coverage required (selective risk), or which model gets the largest coverage for the accuracy constraint (SAC). While each deployment scenario specifies its own constraints, for demonstration purposes we consider a scenario in which misclassifications are by far more costly than abstaining from giving correct predictions. An example of this could be classifying huge unlabeled dataset (or cleaning bad labels from labeled dataset). While desirable to assign labels to larger portion of the dataset (or to correct more the wrong labels), it is crucial that these labels are accurate as possible (or that correctly labeled instances not replaced with a bad label).
8.5
log (#Paammtters)
7.5
Models
Various
X RegNetY
AlexNet
ResNet
BiT
SWSL ResNeXt
DLA
VIT
RegNetX
VIT
10
20
30
40
50
Coverage for Accuracy 99
Figure 15: comparison of 523 models by their log(number of model's parameters) and the coverage they are able to provide for SAC of 99% (higher is better) on ImageNet.
To explore such a scenario, we evaluate all models on ImageNet see which ones give us the largest coverage for required accuracy of 99%. In Figure 7, Section 3 (paper's 's main body) we observe that of all the models studied, only VIT models are able to provide coverage beyond 30% for such extreme constraint. Moreover, we note that the coverage they provide is significantly larger than that given by models with comparable accuracy or size, and that ViT models that provide similar coverage to their counterparts do so with less overall accuracy.
In Figure 15 we see that not only do ViT models provide more coverage than any other model, but that they are also able to do in any size category. To compare models fairly by their size, present Figure 15, which sets the Y axis to be the logarithm the number of parameters, so that models sharing the same value can be compared solely based their x walue-which is the coverage they
23

---

Published as a conference paper at ICLR 2023
Table A comparison of different training regimes of ViTs. *The paper introducing ViTs (Dosovit- skiy et al., 2021) had also trained VIT models with the JFT-300M dataset; however, their weights are unavailable the general public. All evaluations of ViTs from that paper were conducted on ViTs pretrained on ImageNet-2Ik, which are publicly available. **Pretrained DeiT3 models were first pretrained with learning rate of 10- and then fine-tuned with a learning rate of 10-4
Regime
VT (original)
Steiner
Chen
DeiT
DeiT3
DeTT3 +Prerraining TouNron (2022)
Torchvision Paokke et al. (2019)
Reference
Dosociskly et (2021)
Steiner et (2022)
Coen al. (2022)
Tourson et (2021b)
Pretraining dataset
ImageNet 21k*
ImageNet 21k
ImageNNe 2 21448
Batch Saze
40%6
4196
4FF6
1024
2044
512 Adam
Optimizer
AdamW
Adamm
SAM
LAMB
LAMB
LAMB
LR
10"
3- 10
3-10--
1 1-10"'
3- 10
3- 10---e
3-10
1.R deaay
cosine
conine
cnsin2
ensine
Cosine
cosin2
cosine
Weight decay
0.1
0.3
0.1
0:05
BA12
0A2
0.3
Wamnup eepcchs
3.4
3,4
344
S
30
Label smoothing
0.1
0.1
0.11
Droporat
Stoch.
Repeased Aug
1.0
Resized Crop
Auguent
Adgpt
Adapt.
Augment
X
LayetScale
X
Mixup alpha
AdgL
02
Connas aighia
LA
1.0
10
Erning pruh.
0.25
X
X
Cobeliter
X
X
D.3
0.3
X
Test ratio
0.775
0.75
0,875
0875
1.0
1.0
0,775
Loos
CE
CE
CE
CE
BCE
CE
CE
Superb perforrancc
X
provide for SAC of 99%. We see that VIT models provide a larger coverage even when compared with models a similar size.
J COMPARISON OF VIT TRAINING REGIMES AND THEIR EFFECTS ON UNCERTAINTY ESTIMATION PERFORMANCE
In Table we compare the different hyperparameters and augmentations used for training the VIT models evaluated this paper, with the aim of revealing why some training regimes consistently result in superb ViTs, while others do not. An analysis of the various differences between these regimes, however, eliminates the obvious suspects.
1) Pretraining, its models (provided by Touvron et al. 2022) such that both models have identical architecture and training regimes, with the exception that Pretraining results in only a slight improvement 0.16 AUROC and 0.01 ECE. Moreover, as mentioned in detail in Section 3, ViT models trained JFT-4B (Tran et al., 2022) by the successful VIT models evaluated in this paper, most of which were pretrained on ImageNet-21k (and even by one ViT SAM model that not pretrained all). Second, we note that ViTs trained with the SAM optimizer (Chen et al., 2022), and not pretrained at all, reach superb ranking (AUROC) as well. These facts lead us conclude that pretraining, at least by itself, not the main contributor to training successful ViTss.
does not seem to offer explanation: First, analyze eight pairs of
pretrained ImageNet-211 and the other was not.
outperformed
2) The selection of optimizers and other hyperparameters (such learning rate, label smoothing etc.) does not seem to have a significant impact. For example, while AdamW (Loshchilov & Hutter, 2019) was used by two of the successful regimes, it was also used by Paszke et al. (2019), and on the other hand was replaced by SAM (Foret et al., 2021) in another successful training regime.
3) Advanced augmentations are unlikely to explain the gaps in uncertainty estimation performance, regimes producing superior VIT models (Dosovitskiy et al., 2021; Chen et al., 2022) did not use advanced augmentations (in comparison to Touvron et al. (2021b) and Touvron et al. (2022), for example).
For these reasons, for the moment, the explanation for the gap remains elusive. The only remaining "suspect" is the batch size used, with all successful regimes using a batch size of 4096. while others use a smaller batch size of 2048 or lower. One could argue, however, that a two-fold increase batch size is not sufficient explain the huge gaps in performance measured.
24

---

Published as a conference paper at ICLR 2023
Table 2: The relationship between uncertainty estimation performance and the model's attributes and resources (accuracy, number of parameters and input size), measured by Spearman correlation. Positive correlations indicate good utilization of resources for uncertainty estimation.
Architeoture
AUROC & Aocuncc
BCE AcCuEac
AUROC & iPaunnters
ECE PPaunetees
AUROC & leput Size
ECE & laput Size
Model Eviloazed
EftasenNee
0.16
-.29
-0.29
0.26
4.38
ReaNer
4044
0.50
XCT
XCT
0.33
014
0.14
SE, RoNN
4.46
000
43
0.30
4.02
4.35
18
0.700
445
060
0467
0.19
0.40
15
NFNNtt
0.56
0.78
0.6
0.8
0488
0.60
Incrption
4008
0.35
BezNert
0.7
Call dicillal
0.4
0.64
0.77
Mobik&NarV3
0.37
0.39
0.43
0.60
10
RectNet
0.700
0.2
016
040
CLIP Zec-Shat Shot
1,0
066
68
0.3
0.55
4.58
1P Linear Probe
0.88
0.11
0.39
4.27
0.11
0.71
4.33
ResNeXt
0.9
0.22
RozNei RS
0.00
0.7
4.15
0.82
4.30
0.82
MisCom
4.11
088
0.24
0.6
DenscNit
0.4
814
0.72
0.12
-100
0.2
0.89
0.71
0.77
641
BCANer
0.83
0.37
-0.6
SWSL ReoNet
0.94
0.77
GENS
0.50
-100
0.50
-1.00
0.87
4.87
SSL ResNet
0.14
110
0.36
409
TRooet
0.10
0.30
0.53
0.33
4.38
4.87
Coa'
-010
0.9
4110
divilled
0.50
0.60
0.20
0.15
VIT Diciiled
0,5
0.71
FIT divillal
1.00
-100
1.00
-1.00
PiT
0.90
100
01:0
1.00
WSP ReaNeli
1.00
0.80
1.00
0.50
ResMLP Paciiled
0.00
0.2
0.0
MassNer
0.40
0.30
0.1
K EVALUATIONS OF THE ZERO-SHOT LANGUAGE-VISION CLIP MODEL
In this section we describe how we use CLIP model and extract confidence signals from it during inference. To evaluate CLIP on ImageNet, we first prepare it following the code provided by its authors httt:ss/ggituu..cooopeaaa/i
The labels ImageNet- 1k are encoded into normalized embedding vectors. At inference time, the incoming image is encoded into another normalized embedding vector. A cosine similarity is then calculated between each label embedding vector and the image embedding vector, and lastly, softmax is applied. The highest score then taken as the confidence score for that prediction. We also evaluate the same models when adding a trained "linear-pree probe" to them (as described in Radford et al. (2021), which is essentially logistic regression head), that results in a large boost in their accuracy.
L EFFECTS OF THE MODEL'S ACCURACY, NUMBER OF PARAMETERS AND INPUT SIZE ON UNCERTAINTY ESTIMATION PERFORMANCE
Table shows the relationship between uncertainty estimation performance and model attributes and resources (accuracy, number of parameters and input size), measured by Spearman correlation. We measure uncertainty estimation performance by AUROC (higher better) and -ECE (higher is better). Positive correlations indicate good utilization of resources for uncertainty estimation (for example, a positive correlation between -ECE and the number of parameters indicates that as the number of parameters increases, the calibration improves). An interesting observation is that distillation can drastically change the correlation between a resource and the uncertainty estimation performance metrics. For example, undistilled XCITs have Spearman correlation of -0.79 between their number of parameters and AUROC, indicating that more parameters are correlated with lower ranking performance, while distilled XCTTs have correlation of 0.35 between the two.
25

---

Published as a conference paper at ICLR 2023
Table 3: Comparing using MC dropout to poltmax-responss (vanilla).
Architecture
Method
Accuracy
AUROC
Vanilla
74.04
86.88
MobileNetV3 Large
MC dropout
86.14
Vanilla
67.67
86.2
MobileNetV3 Small
MC dropout
67.55
84.54
Vanilla
71.88
86,05
MobileNetV2
MC dropout
71.81
84.68
Vanilla
70.37
86.31
VGG11
drupout
VGGII (no BatchNorm)
Vanilla
69.02
86.19
MC dropout
68.95
83.94
Vanilla
71.59
86.3
VGG13
MC drupout
71.43
84.37
Vanilla
69.93
86.24
VGG13 (no BatchNorm)
MC dropout
69.71
84.3
Vanilla
73.36
86,76
VGG16
MC dropout
73.33
85.02
Vanilla
71.59
86.63
VGG16 (no BatchNorm)
MC dropout
71.47
84.97
Vanilla
74.22
86.52
VGG19
MC dropout
74.17
85.06
Vanilla
72.38
86.55
VGG19 BatchNorm)
MC dropout
72.37
84.99
M EVALUATIONS OF MONTE CARLO DROPOUT RANKING PERFORMANCE
MC Dropout (Gal& & Ghahramani, 2016) is computed using several dropout-enabled forward passes to produce uncertainty estimates. In classification, the mean softmax score of these passes, is calculated, and then predictive entropy score is used the final uncertainty estimate. In our evaluations, 30 dropout-enable forward passes. We do not measure MC Dropout's effect on ECE since entropy scores do reside in [0, 1].
We test this technique using MobileNerV3 (Howard et al., 2019), MobileNetv2 (Sandler et al., 2018) and VGG (Simonyan & Zisserman, 2015), all trained ImageNet and taken from the PyTorch repository (Paszke et al., 2019).
The results comparing these models with and without using MC dropout are provided in Table 3.
The table shows that using MC dropout causes a consistent drop in both AUROC and selective performance compared with using the same models with softmax as the These results also visualized in comparison to other methods in Figure 4a in Section 3. MC dropout underperformance an ID setting was also previously observed in Geifman & El-Yaniv (2017).
26


Reviews:
[{"review_id": "o4A5CTXJAB", "reviewer": "Reviewer_GVar", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an extensive empirical study on the selective prediction and uncertainty estimation. The experiments include a large number of pretrained models and several metrics. Although there is no novel algorithmic novelty, the results presented are very intersting.", "summary_of_the_review": "Overall, the paper presents an interesting empirical study on selective classification and uncertainty estimation. Although the paper lacks algorithmic novelty, the results presented \nIt could have been better if the authors analyzed one of the presented insights and tried to find intuition for it.", "strength_and_weaknesses": "Strength:\n- The paper is well-written\n- The insights presented are very interesting \n- The experiments are extensive including several metrics and pretrained models\n- The community can benefit from the presented results by investigating why some architectures have low uncertainty and this can open the door designing better methods.\n\nWeaknesses:\n- The main point is the lack of algorithmic novelty ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mvq43W0XrV", "reviewer": "Reviewer_a33z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the ***relationship*** between ***deep models*** and their corresponding ***selective prediction and uncertainty estimation performance***. Specifically, this work considers several uncertainty estimation metrics, including AUROC, ECE, AURC and coverage for selective accuracy constraint. Using 523 models, this work provides some observations. For example, distillation-based training regimes consistently yield better uncertainty estimations; a subset of ViT models that outperform any other models in terms of uncertainty estimation performance.", "summary_of_the_review": "**I am around the borderline**. \n- Overall, this work is well-written and easy to follow. The study on the relationship between deep models and uncertainty estimation performance is useful.\n- While the analysis is interesting, I expect to see the observations on out-of-distribution datasets.\n- Some observations need more clarification (e.g., Figure 6).\n\n***------Post-Rebuttal------***\n\nThe replies did not answer initial concerns. I have been waiting for the required clarifications. If not offered, I would reject this paper and be willing to fight for my rating.", "strength_and_weaknesses": "**[Strength]**\n- Studying uncertainty estimation of deep models is important\n- This work considers commonly used uncertainty metrics. Also, diverse models with various architectures, training regimes, and learning schedules are included in the study.\n\n**[Weaknesses]**\n\n- The major question is: this work conduct analysis on ImageNet validation set, which is an in-distribution dataset. When testing models on ***out-of-distribution (OOD)*** datasets (e.g., ImageNet-R, ImageNet-S, and ObjectNet), the observations in this work ***might not hold***. In real-world applications, the dataset distribution typically changes. Thus, I would like to see the study on OOD datasets. Moreover, the models with the same accuracy can have significantly different performances on OOD datasets. Then, in addition to Figures 3 and 4, reporting the accuracy of models on OOD datasets would be more helpful to understand the effect of different methods. \n\n- Section 3 uses a condition that the models have the same classification accuracy. This seems reasonable. How about different architectures with the same accuracy? I would like to check the effect of architecture on uncertainty estimation accuracy. \n\n- Figure 6 (Comparing teacher models to their KD students ) is a little confusing. The student models have different accuracy from their teachers. In this case, it might not reasonable to compare them. Please clarify this.\n\n- Please clarify on which dataset the temperature scaling was learned.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mh8D_JmYbw_", "reviewer": "Reviewer_CSk1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work examines the uncertainty calibration of various image classifiers. A fairly comprehensive list of classification models, calibration techniques, and metrics are examined experimentally. From experimental results, this work draws several observations and recommendations in both model type and training techniques. \n", "summary_of_the_review": "See above", "strength_and_weaknesses": "From my understanding, this work relies on trained models released by various previous works, which precludes the authors from controlling optimization parameters such as learning rate, regularization parameters, and training time; these hyperparameters may have significant impact on calibration due to their effects on overfitting. Furthermore, the analysis in this work is limited to imagenet (and 1 plot with cifar 10?), which may limit the transferability of the observations made in this paper. \n\nThe way in which results are presented in the paper is also problematic at times. For example, the box plots in figure 4 and figure 5 did not include the number of pairs in each column, thereby limiting the reader’s ability to interpret the statistical significance of the results illustrated. In another example, while the authors repeatedly stated that they evaluated 523 models, not a figure, table, or graph in the supplementary materials explicitly lists what these 523 models consist of. For the sake of reproducibility, a machine-readable but human friendly list of models should accompany the supplementary materials (so that readers can see which models are used without prowling the html source). Since the authors have already grouped these models into tens of categories for plotting (different marker types in figure 1), tables summarizing the various models within each category would be very informative in the appendix. Furthermore, a distilled version of table 2 with the number of models within each category would be beneficial to understanding in the main text. The paper is also not fully self-contained; a review of related works that measure and benchmark uncertainty estimation and calibration is missing. This makes it difficult to discern which results in this paper are in accordance with current knowledge, and which results are surprising. Also missing are descriptions of the different methods applied to the models (e.g. adversarial training, KD, temperature scaling etc.). A concise and mathematical description of each method should at least be included in the appendix. The omission of this information diminishes the persuasiveness of the paper’s message. \n\nNonetheless, this work presents a trove of quantitative results from which interesting conclusions can be drawn. The six conclusions drawn from these results are salient and actionable. As such, I would recommend for acceptance, as I believe that the publication of this work will benefit the collective knowledge of our field. \n\nOne question I have is whether the observations from this work can transfer to any other dataset, using a more selective subset of models that had shown significant variation in performance on imagenet. \n\nOther minor notes:\nThere’s a kappa on page two that is not rendered correctly as the symbol. \nFigure placement and captioning can be significantly tightened (e.g. figure 4 and 5 has basically the same caption). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "UgEcSlAUHb", "reviewer": "Reviewer_5WoK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about quality of confidence predictions made by a bunch of pretrained models on ImageNet 1K, in terms of selective prediction (measured by the accuracy vs confidence plot), ranking (measured by AUROC), and calibration (using ECE).\n\nThe authors claim to evaluate 523 pretrained models from PyTorch and timm in terms of the above mentioned metrics, and explore issues like ranking and selective prediction performance (where a classifier can abstrain from making a prediction if its uncertainty is too high). There are some important conclusions and insights that can be drawn from this study that are summarized below and in the strengths.\n\nThe contributions are:\n- A large scale study of pretrained models on ImageNet 1K in terms of the quality of their predictive uncertainty.\n- Conclusion that knowledge distillation improves uncertainty estimation best as compard with other training regimes.\n- Conclusion that temperature scaling also improves AUROC, meaning better selective and ranking performance.\n- Finding that some vision transformer architectures, particularly ViT, have better uncertainty estimation than other comparable architectures.\n- Information about large scale correlations between AUROC, ECE, and model parameters like accuracy, number of parameters. There are good conclusions that not much can be inferred in terms of correlations from these metrics.\n", "summary_of_the_review": "I have mixed feelings about this paper, in one side there are some good contributions and conclusions, this is a large scale study which can have a major impact in future research, specially since it uses ImageNet and many models across different architectural patterns, but in the other side there are some known or dubious claims that weaken the paper, but the major issue is the lack of information and clarity about how the experiments were performed, particularly in some figures, and selection of models for each experiment. This paper has potential but needs some strong clarifications (these are documented in weaknesses and minor issues).\n\n~~I believe that right now this paper is borderline, this could definitely change with author feedback. My major issue is clarification about claims, previous work, and definition of AUROC.~~\n\nAfter the rebuttal, I can recommend this paper for acceptance if the clarifications made by the authors during rebuttal are implemented in the final version.", "strength_and_weaknesses": "Strengths\n- The paper is very well written and presented, I have no complaints about writing or presentation except some details on minor issues.\n- It makes sense to make a large scale analysis of pretrained models on ImageNet about their (uncalibrated) uncertainty quantification capabilities, specially considering the recent advances in the last years with vision transformers. While I have my reservations about metrics and models with uncertainty, at least this is not a bad idea to start with.\n- I like the analysis of risk-coverage, I have not seen this before at this level of depth, with so many models, and I believe this could be a novelty in this paper. The conclusion here is that some vision transformers produce better ranking and selective prediction, as measured by the coverage-accuracy plot.\n- I like the conclusion that some vision transformers have better quality of uncertainty, but I think this is also not new since there is previous research that shows vision transformers improve on out of distribution detection, which is also closely related to uncertainty estimation. For this you can view the paper \"Exploring the Limits of Out-of-Distribution Detection\" by Stanislav Fort at NeurIPS 2021. I do not think this completely invalidates the results but at least it is something to acknowledge.\n- One of the major results of this paper is an evaluation of different training regimes, including adversarial training, pretraining on ImageNet21K, semi-supervised learning, distillation, temperature scaling, and MC-dropout, on different uncertainty quantification metrics. Distillation seems to have the largest impact in improving AUROC and ECE over their standard variations. I believe this is a novel finding as I think it is not known at this scale.\n- Additionally, temperature scaling also has a good improvement on AUROC and ECE and since it is easy to implement, it is a worthy information for practitioners to try to improve basic uncertainty quantification of their models.\n- A good conclusion is that some particular vision transformers architectures (like the original ViT) clearly outperform all other architectures in terms of accuracy vs coverage, which I also think it is a novel finding.\n- Another good analysis is correlation between AUROC and ECE with other parameters, they could be positive, negative, or zero, so not many conclusions can be drawn from this. Note that I have some criticism in weaknesses about this, specially since correlation is linear and there could be non-linear effects between these metrics.\n\nWeaknesses\n- The major problem with this paper is that all the pretrained models evaluated in this paper have been trained only with empirical risk minimization (except the ones with alternate training methods), meaning they are not really tuned to produce high quality uncertainty. I recognize that the authors used some models that use monte carlo dropout, but this is a very basic method for uncertainty estimation, a much better comparison would have been using bayesian neural networks and similar methods aimed to produce high quality uncertainty (ensembles, flipout, etc).\n- The argument that ECE does not always point to the best model is well known, the ECE can be zero in degenerate cases like predicting mean accuracy as confidence, the ECE is not a proper scoring rule, so I think this is argument is not something new and severely weakens the paper. There are other calibration concepts that could have been used instead like group or adversarial calibration, for this see https://arxiv.org/abs/2006.10288\n\n~~- In Figures 4 and 5, there is a large variation in the number of samples for each category of comparison (adv training, semi supervised, ImageNet21K, etc), with temperature scaling having the largest number of samples by far, I think the paper should comment on this as the sample size has an effect on the quality of the comparison and the conclusions that follow it, ideally all methods should use similar samples. Here I also assume that each data point is one different model being used, the caption does not make this clear. And there seems to be no explanation why different number of models were used in each of these experiments, particularly for MC-dropout for example.~~\n\n~~- In Section 3, subpoint 4, the authors mention correlation of AUROC/ECE with several metrics are zero. The issue is the claim that this contradicts previous smaller scale studies, to me it is not clear which studies, and which exact claims. The paper later cites [Guo et al. 2017], but this paper does not evaluate AUROC, as that paper is about calibration and uses ECE for most experiments, so it is not clear which study claims to have found other correlations between AUROC and model parameters. This needs to be strongly clarified.~~\n\n~~- While I understand the idea of using the AUROC to gauge ranking of predicted probabilities, the paper makes references to AUROC for binary classification (this is the most common use), but the results are for multi-class classification, and the paper does not mention which formulation is used for this case, as it is not trivial to extend binary classification AUROC into multi-class AUROC, so I would expect that the authors describe this in some level of detail, currently it is completely missing.~~\n\nMinor Issues\n- For Reproducibility, I would expect a list or at least some summary or information on which ones are the 523 models used in the evaluation of this paper, the authors refer to versions of the timm library but from the documentation (at https://rwightman.github.io/pytorch-image-models/models/ ) it is not clear where the figure of 523 models comes from.\n- In Figure 8, what is the \"various\" models plotted with circles? This is not clear from the caption and should be clarified.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster