How to measure deep uncertainty estimation performance and which models are naturally better at providing it
Authors: Ido Galil
Mohammed Dabbah
Ran El-Yaniv
Venue: ICLR 2022 Submitted (2022)
Keywords: non-Bayesian uncertainty estimation
selective prediction
transformer
vision transformer
vit
risk-coverage curve
selective classification
classification with a reject option

Abstract: When deployed for risk-sensitive tasks, deep neural networks (DNNs) must be equipped with an uncertainty estimation mechanism. This paper studies the relationship between deep architectures and their training regimes with their corresponding uncertainty estimation performance. We consider both in-distribution uncertainties ("aleatoric" or "epistemic") and class-out-of-distribution ones. Moreover, we consider some of the most popular estimation performance metrics previously proposed including AUROC, ECE, AURC, and coverage for selective accuracy constraint. We present a novel and comprehensive study carried out by evaluating the uncertainty performance of 484 deep ImageNet classification models. 
We identify numerous and previously unknown factors that affect uncertainty estimation and examine the relationships between the different metrics. We find that distillation-based training regimes consistently yield better uncertainty estimations than other training schemes such as vanilla training, pretraining on a larger dataset and adversarial training. We also provide strong empirical evidence showing that ViT is by far the most superior architecture in terms of uncertainty estimation performance, judging by any aspect, in both in-distribution and class-out-of-distribution scenarios. We learn various interesting facts along the way. Contrary to previous work, ECE does not necessarily worsen with an increase in the number of network parameters. Likewise, we discovered an unprecedented 99% top-1 selective accuracy at 47% coverage (and 95% top-1 accuracy at 80%) for a ViT model, whereas a competing EfficientNet-V2-XL cannot obtain these accuracy constraints at any level of coverage.

Under review as a conference paper at ICLR 2022
HOW TO MEASURE DEEP UNCERTAINTY ESTIMATION PERFORMANCE AND WHICH MODELS ARE NATURALLY BETTER AT PROVIDING IT
Anonymous authors Paper under double-blind review
ABSTRACT
When deployed for risk-sensitive tasks, deep neural networks (DNNs) must be equipped with an uncertainty estimation mechanism. This paper stud- ies the relationship between deep architectures and their training regimes with their corresponding uncertainty estimation performance. We consider both in- distribution uncertainties ("alcatoric" or "epistemic") and fluss-out-of-distribut ones. Moreover, consider some of the most popular estimation performance metrics previously proposed including AUROC, ECE, AURC, and coverage for selective accuracy constraint. We present novel and comprehensive study carried out by evaluating the uncertainty performance of 484 deep ImageNet classification models. We identify numerous and previously unknown factors that affect uncer- tainty estimation and examine the relationships between the different metrics. We find that distillation- based training regimes consistently yield better uncertainty estimations than other training schemes such as vanilla training, pretraining on a larger dataset and adversarial training. We also provide strong empirical evidence showing that VIT is by far the most superior architecture in terms of uncertainty estimation performance, judging by any aspect, in both in-distribution and class- distii--istiittinnnn scenarios. We learn various interesting facts along the way. Contrary to previous work, ECE does not necessarily worsen with an increase the number of network parameters. Likewise, discovered unprecedented 99% top-1 selective accuracy at 47% coverage (and 95% top-1 I accuracy at 80%) for ViT model, whereas competing EfficientNet-V--XX cannot obtain these accuracy constraints at any level of coverage.
INTRODUCTION
Deep neural networks (DNNs) show great performance in wide variety of application domains including computer vision, natural language understanding and audio processing. Successful de- ployment of these models, however, critically dependent on providing an effective uncertainty estimation of their predictions in the form of some kind of selective prediction or providing a prob- abilistic confidence score for their predictions.
But how should we evaluate the performance of uncertainty estimation? Let us consider two classifi- cation models for the stock market that predict whether stock's value is about to increase, decrease or remain neutral (three-class classification). Suppose that model A has a 95% true accuracy, and generates confidence score of 0.95 on every prediction (even on misclassified instances); model B has 40% true accuracy. but always gives confidence score of 0.6 on correct predictions, and 0.4 on incorrect ones. Model B can be utilized easily to generate perfect investment decisions. Us- ing selective prediction (Geifman & El- Yaniv| 2017), Model B will reject all investments stocks whenever the confidence score is 0.4. While model offers many more investment opportunitics, each of its predictions carries 5 % risk of failure.
Among the various metrics proposed for evaluating the performance of uncertainty estimation are: Area Under the Receiver Operating Characteristic (AUROC AUC), Area Under the Risk- Coverage curve (AURC) (Geifman et al.) 2018), selective risk or coverage for a selective accuracy constraint (SAC), Negative log-liklldood (NLL)), Expected Calibration Error (ECE), which is of-

---

Under review as a conference paper at ICLR 2022
AUROC
Models
O
Various
+
AlexNet
4 ResMLP distilled
distilled
ResNet
ElficienNNet22
VIT
GENet
distilied
MMMP Mixer
VIT SAM
4 MLP Mixer distilled
A XCIT
M RogNety
distilled
5
-log(ECE)
Figure 1:
A comparison of 484 models by their AUROC 100, higher is better) and -log(ECE) (higher is better) ImageNet. Each marker's size is determined by the model's number of param- eters. A full version graph is given in Figure [ Distilled models are better than non-distilled ones. VIT models naturally better all aspects of uncertainty estimation, while EfficientNet-V and GENet models are worse.
ten used for evaluating a model's 's calibration (see Section 2 and Brier score (Brier 1950). All these metrics well known and often used for comparing the uncertainty estimation performance of models (Moon et al.) 2020; Nado et al. 202T Maddox et al. 2019 Lakshminarayanann et al. (017). Somewhat surprisingly, y, NLL., Brier, AURC, and ECH all fail reveal the uncertainty superiority of Model B in our investment example (see Appendix A for the calculations). Both AUROC and SAC, on the other hand, reveal the advantage of Model perfectly (see Appendix for details). is not hard to construct counter examples where these two metrics fails and others (e.g., ECE) succeed. The risk-coverage (RC) curve (EI- Yaniv Wiener 2010) perhaps one of the most informative and practical representations of the overall uncertainty profile of given model.
In general, though, two RC curves are not necessarily comparable if one does not fully dominate the other (see Figure 2 The advantage of scalar metrics such as the above is that they summarize the model's overall uncertainty estimation behavior by reducing it to a single scalar. When not carefully chosen, however, these reductions could result in loss of vital information about the problem (for example, reducing an RC curve to an AURC does not show that Model B has an optimal 0 risk the coverage is smaller than 0.4). Thus, the choice of the "correct" single scalar performance metric unfortunately must be task-specific. When comparing the uncertainty estimation performance of deep architectures that exhibit different accuracies, we find that AUROC and SAC effectively "normalize" accuracy differences that plague the usefulness of other metrics (see Section 2p.. This normalization is essential to our study where we compare uncertainty performance of hundreds of models that can greatly differ in their accuracies.
In applications where risk (or coverage) constraints are dictated (Geifiman El-Yaniv 2017), the most straightforward and natural metric is the SAC (or selective risk), which directly measures the coverage (resp., risk) given at the required level of risk (resp., coverage) constraint. We demonstrate this in Appendix ! evaluating which models give the most coverage for SAC of 99%. Sometimes, however, such constraints are unknown in advance, or even irrelevant, e.g., the constructed model should serve variety of risk constraint use cases, or the model may not be allowed to abstain from predicting all.
In this paper we conduct comprehensive study of DNNs' ability to estimate uncertainty by eval- uating 484 models pretrained on ImageNet (Deng et al. 2009). taken from the PyTorch and timm respositories (Paszke et al. 2019 Wightman 2019). ,. We identify the main factors contributing to or harming the confidence ranking of predictions ("ranking" for short), calibration and selective prediction. Furthermore, we also consider the source of uncertainty either internal (stemming from either the aleatoric or epistemic uncertainty the model (Kiureghian & Ditlevsen, (009)) or external (originating from unseen unknown tafs--ou-of-diitribiee (C-OOD) data) and evaluate
2

---

Under review as a conference paper at ICLR 2022
these models in multiple ways. After first evaluating models solely in-distribution (ID) data, then define and test two ways of evaluating C-OOD data, each of which also divides the data into different groups by how difficult it is for the model to distinguish instances as external.
Our study lead to quite a few new observations and conclusions; (1) Training regimes incorporating any kind of knowledge distillation (KD) (Hinton al. 2015) leads to DNNs with improved uncer- tainty estimation performance evaluated by any metric. in both internal and external settings (i.e., leading also to better C-OOD detection), more than by using any other training tricks (such as pre- training on a larger dataset, adversarial training, stc.). (2) Some architectures are naturally superb at all aspects of uncertainty estimation and in all settings, e.g., vision transformers (ViTs) (Dosovitskiy) iet al. 2020: Steiner et al. 2001)) while other architectures tend to perform worse, e.g., EfficientNet- V2 and GENet (Tan & Le|21021 Lin et al.) 2020). These results are visualized in Figure [ (3) The superiority of ViTs remains even when the comparison considers the models' phres-meaning that for any size, ViTs outperform the competition in uncertainty estimation performance, as visualized in Appendix B in Figures 9 and TO (4) The simple post-training calibration method of temperature scaling (Guo el al. 2017). which is known to improve ECE, for the most part also improves ranking (AUROC) and selective prediction meaning not only does it calibrate the probabilistic estimation for each individual instance, but it also improves the partial order of all instances induced by those improved estimations, pushing instances more likely to be correct have higher confidence than instances less likely to be correct (see Section 3 (5) Contrary to previous work by |uo et al.) (201)) (2017), we observe that while there is a strong correlation between accuracy/numbe of parameters and ECE or AUROC within each specific family of models of the same architecture, the correlation flips be- tween strong negative and strong positive correlation depending on the type of architecture being observed. For example, as ViT architectures increase in size and accuracy, their ECE deteriorates while their AUROC improves. The exact opposite, however, could be observed in XCiTs (EI-Nouby et al. 2021) their BCE improves with size while their AUROC deteriorates. (see Appendix G). (6) The best model in terms of AUROC or SAC is not always the best in terms of calibration, as illustrated in Figure D and the trade-off should be considered when choosing model based on its application. Due to lack of space, a number of additional interesting observations are briefly mentioned in the paper without supporting empirical evidence (which is provided in the appendix).
HOW TO EVALUATE DEEP UNCERTAINTY ESTIMATION PERFORMANCE
Let X' be the input space and y be the label space. Let P(X. y) be an unknown distribution over X' y. A model f is a prediction function f : X - V, and its predicted label for an image x is denoted by Ūs (z). The model's ' tre risk w.r.t. P is R) = E [f(f(r), { y)]. where f: y x y - R- is a given loss function, for example, 0/1 loss for classification. Given a laheled set Sm ((,,, mm C (X x V), sampled i.i.d. from P(X, the empirical risk of model f is P(f]Sm) a 1 - (.. Following Geifman et al. (2018), for given model f we define a confidence score function k<(z, flf), where x E X, and E y IS the model's 's prediction for x. as follows. The function x should quantify confidence the prediction of ý for the input x. based on signals from model f. This function should induce partial order over instances in X, and is not required to distinguish between points with the same score.
m
The most common and well-known function for a classification model f (with softmax at its last layer) is its softmax response values: k(z, ul.f) 2 f(z)z (Cordella et al. 1995 De Stefano et al, 2000). While this is the main x we evaluate, we also test the popular uncertainty estimation tech- nique of Monte- Carlo dropout (MC-Dropout) (Gal & Ghahramani] 2016). which is motivated by Bayesian reasoning. Although these methods use the direct output from f. k could be a differennt model unrelated to ff and unable to affect f's predictions. Note that to enable a probabilistic in- terpretation, & can only be calibrated its values reside in [0, 1] whereas for ranking and selective prediction any value in R can be used.
A selective model f YEl-Yaniv & Wiener 2010 Chow 1957 uses a selection function g X + {0, 1} to serve a binary selector for f. emabling it to abstain from giving predictions for certain inputs. 9 can be defined by a threshold on the values of a x function such that go (x/ f) = (s(a,, By (x))
0). The performance of a selective model is measured using coverage and risk, where coverage, defined as o(f.g) = Ep [x])]] is the probability mass of the non-rejected instances in X. The selective risk of the selective model (f. 9) is defined as R(f.g) A En(e(T()) These +(f-a)
3

---

Under review as a conference paper at ICLR 2022
quantities be evaluated empirically over finite labeled set Sm, with the empirical coverage defined @(s.g)Sm) Ei= g(;),, and the empirical selective risk defined as (() - 2. (()())))))))).)) Similarly, SAC defined the largest coverage available for specific o[S.g 9m)
accuracy constraint. A way to visually inspect the behavior of function for selective prediction can be done using an RC curve-a a curve showing the selective risk as a function of coverage, measured on some chosen test set; see Figure 2 for an example.
Coverroe Curve Comparison
Accuracy
EfficientNet-v2-XL (trained on 21k and finetuned on 1k)
0.25
VTT-B/32-SAM
VTT-L/6-384
ImageNet Error (Risk)
Bioger AUROC (x) better
Smaller (00)
Accuracy 0.85 Rate
0.15
AUROC 87.52
AURC- 76.18
0.10
AUROC=77.77 AURC-60 68.45
Accuracy 0.87
Erro Rate- 0.13
AUROC-888 AURC-26.48
0.00
100
Coverage
Figure 2: A comparison of RC-curves made by the best VIT-L/16-384) and worst (EfficientNet- V2-XL) models we evaluated in terms of AUROC. Comparing ViT-B/32-SAM EfficientNet-VV V2 exemplifies the fact that neither accuracy nor AURC reflect selective performance well enough.
The AURC and E-AURC metrics were defined by Geifman et al. (2018) for quantifying the selective quality of functions via a single number, with AURC being defined as the area under the RC curve. AURC, however, is very sensitive to the model's accuracy, and in attempt mitigate this, E-AURC was suggested. The latter also suffers from sensitivity to accuracy, as we demonstrate in Appendix C
Let us consider the two models in Figure risk-sensitive deployment; EfficientNet- V2-XL (Tan & Le 2021) and ViT-B/32-SAM (Chen el al. 2021a). While the former model has better overall accuracy and AURC (metrics that could lead us believe the model is best for our needs), it cannot guarantee Top-1 ImageNet selective accuracy above 95% for any coverage. ViT- B/32-SAM, on the other hand, can provide accuracies above 95% for all coverages below 50%.
When there are requirements for specific coverages, the most direct metric to utilize would be the matching selective risks, by which we can select the model offering the best performance for our task. If instead specific range of coverages is specified, we could measure the area under the RC curve for those coverages: AURC f S,)) с Σ F(f.gelSm. with being those required
cEC
coverages. Lastly, if certain accuracy constraint specified, the chosen model should be the one providing the largest coverage for that constraint (the largest coverage for certain SAC).
Often, these requirements are not known or can change as a result of changing circumstances or indi- vidual needs. Also, using metrics sensitive to accuracy such as AURC makes designing architectures and methods to improve K very hard, since an improvement these metries could be attributed to either increase in overall accuracy (if such occurred) or to real improvement in the model's "metacognition". Lastly, some tasks might not allow the model to abstain from making predictions at all, but instead require interpretable and well-calibrated probabilities of correctness, which could be measured using ECE.
MEASURING RANKING AND CALIBRATION
A function is not necessarily able to change the model's predictions. Thus, its means for improv- ing the selective risk by ranking correct and incorrect predictions better, inducing a more accurate partial order over instances in X. Thus, for every two random samples (11, (12,, y2) P(X,))
4

---

Under review as a conference paper at ICLR 2022
and given that (x)())) v1) > ((f(2)) y2), the ranking performance of K. is defined as the proba- bility that is ranks 22 higher than Ø1:
Pr[x(11, flf) < klz2, (l))
((f(x2), 22)l
We discuss this definition in greater detail in Appendix D The AUROC metric is often used in the field of machine learning. When the 0/1 loss is in play, it is known that AUROC in fact equals the probability in Equation ① (Fawceti 2006) and thus a proper metric to measure ranking in clas- sification (AKA discrimination). AUROC is furthermore equivalent to the Goodman and Kruskal's y-correlation Goodman & Kruskal (1954). which for decades has been extensively used to measure ranking (known as "resolution") in the field of metacognition Nelson (1984). The precise relation- ship between y-correlation and AUROC is 7 = 2 AUROC - (Higham & Higham 2018). We note also that both the yycorrelation and AUROC are nearly identical closely related to various other correlations and metrics; n-correlation (AUROC) becomes identical to Kendall's T (up to a linear transformation) in the absence of tied values. both metrics are also closely related to rank- biserial correlation, the Gini coefficient (not to be confused with the measure from economics) and the Mann-Whitney U test, hinting at their importance and usefulness in a variety of fields and set- tings, In Appendix E we briefly compare the ranking performance of neural networks and humans based on metacogmitive research and address criticism of using AUROC to measure ranking in Appendix
2015). For a finite test set of size N, ECE is calculated by grouping all instances into m interval bins (such that m < N), each of size 1 (the confidence interval of bin B, is (2=)) D. With acc(B,) being the mean accuracy in bin B, and conf(B,) being its mean confidence, ECE is defined as
The most widely used metric for calibration is ECE (Naeini et al.
ECE -
B N
(x,) = yal k(2, 9y (x.il)) [B]
Σ
B N
|acc(BB) (B)]
[B)
BEB
Since ECE is widely accepted we use it here to evaluate calibration, and follow Guo et al. (017))
in setting the number of bins to m 15. Many alternatives to ECE exist, to allow an adaptive binning scheme or to evaluate the calibration on the non-chosen labels as well (Nixon et al.) 2019
Vaicenavicius et al. 2019). Relevant to our objective that by using binning, this metric is not affected by the overall accuracy is the Brier score, for example.
3 IN-DISTRIBUTION ANALYSIS
While AUROC and ECE are (negatively) correlated (they have a Spearman correlation of -0.5, mean- ing that generally as AUROC improves so does ECE), their agreement on the best performing model depends greatly on the architectural family in question. For example, the Spearman correlation be- tween the two metrics evaluated on 28 undistilled XCiTs 0.76 (meaning ECE deteriorates as AUROC improves), while for the 33 ResNets (He et al. 2015) evaluated. the correlation is -0.74. Another general observation is that, contrary to previous work by /uo et al. (2017) concerning ECE, the correlations between AUROC ECF and the accuracy or the number of model parameters nearly zero, although each family tends to have a strong correlation, either negative or positive. We include family-basee comparison in Appendix Gor correlations between AUROC/ECE and accu- racy, number of parameters and input size. These results suggest that while some architectures might utilize extra resources achieve improved uncertainty estimation capabilities, other architectures do not and are even harmed in this respect.
We evaluated several training regimes: (1) Training that involves knowledge distillation in any form, including transformer-specift distillation (Touvron et al. 2020)) knapsack pruning with distillation (in which the teacher is the original unpruned model) (Aflalo et al. (020) and a pretraining technique
which employs distillation (Ridnik et al. 2021); (2) adversarial training (Xie et al. 2019a Tramer
ct al.
2018); (3) pretraining on ImageNet2 ("pure", with no additions) (Tan Le
2021 Touvron
let al. 2021a): and (4) various forms of weakly or semi-supervised learning (Mahajan et al. |0 8b)
Yalniz et al. 2019 Xie et al. 2019b). Of these methods, training methods incorporating distillation improve AUROC and ECE the most (see Figures 3 and 4). Moreover, distillation seems to greatly improve both metrics even when the teacher itself is much worse at both metrics. We discuss these
effects in greater detail in Appendix H
5

---

Under review as a conference paper at ICLR 2022
AUROC Improvement over Vanilla
Method
Adverserial Training
Pretraining ImageNet21k
Serni-Supervised Learning
Distillation
Temperature Scaling
MC-Dropout
Adversarial Training
Pretraining on ImageNor21k
Semi Subervised Learinng
Dietilation
Temperature Sceting
MC-Dropeud Dropprt
Figure 3: A comparison of different methods and their AUROC improvement relative to the same model's performance without employing the method. Markers above the x axis represent models that benefited from the evaluated method, and vice versa.
Method
ECE Improvement over Vanilla
Training
Pretraining ImageNett2k
Supervisd Learning
0.15
Distillation
Temperature Scaling
0.1
Temperature Scaling
ECE
Adversarial Training
Pretraining on Imageheetkk
Semi Super Superviaed viaed Learning
Disiilation
Temperature Scaling
Figure 4: A comparison of different methods and their ECE improvement relative to the same model's performance without employing the method. Markers above the x axis represent models that benefited from the evaluated method, and vice versa. Temperature scaling can sometimes harm ECE, although its purpose is to improve it.
Evaluations of the simple post-training calibration method temperature scaling (TS) (Guo et al, 2017). which is widely known to improve ECE without changing the model's accuracy, also vealed several interesting facts: (1) TS consistently and greatly improves AUROC and selective performance (see Figure 3) memaning not only does TS calibrate the probabilistic estimation for each individual instance, but it also improves the partial order of all instances induced by those im- proved estimations. While TS is well known and used for calibration, to the best of knowledge, its benefits for selective prediction were previously unknown. (2) While TS is usually beneficial, could harm some models (see TiguresBand would harm ECE, this phenomenon is explained by the fact that TS optimizes NLL and not ECE (to avoid trivial solutions), and the two may sometimes misalign. (3) Models that benefit from TS in terms of AUROC tend to have been assigned a temperature smaller than 1 by the calibration pro- cess. This, however, does not hold true for ECE (see Figures 16reesssT66] and Appendix []. (4) While all models usually improve with TS, the overall ranking of uncertainty performance between families tends to stay similar, with the worse (in terms of ECE and AUROC) models closing most of the gap between them and the mediocre ones.
While is surprising that TS-a calibration method-
The ViT architecture far surpasses any other family models in terms of AUROC and ECE (see Figure Figure 15 in Appendix I shows this is true after using TS) as well as for the SAC of 99% we explored (see Figure [ and Appendix J]. Moreover, for any size, VIT models outperform their competition in all of these metrics (see Figures and TO in Appendix B and Figure T8 in

---

Under review as a conference paper at ICLR 2022
80
Models
Accuracy
75
Various
O AlexNet
70
BIT
RegNetx
65
M RegNetY
ResNet
ResNeXt
60
VIT
VIT SAM
20
30
Coverage for Accuracy 99
Figure 5: Comparison of models by their overall accuracy and the coverage they able provide a selective accuracy constraint of Top-1 99% on ImageNet. A higher coverage is better. Only ViT models are able to provide coverage beyond 30% for this constraint. They provide more coverage than any other model compared to their accuracy or size.
Appendix D). While most ViT models we evaluated were pretrained ImageNet-21k and enjoyed augmentations tailored to them (Steiner et al., (021). comparing them with the con-pretrained and not strongly augmented ViT SAMs (Chen et al. (021a), which are consequently much worse in terms of accuracy than regular ViTs, confirms that the high performance in AUROC and SAC is not due to pretraining or augmentation. It does suggest that it is the architecture itself that is naturally superb at uncertainty estimation.
We also evaluate the AUROC performance of MC- MC-Dropout using predictive entropy as its confidence score and 30 dropout-enabled forward passes. We do not measure its affects on ECE since entropy
scores do not reside in [o, 1]. Using MC-Dropout causes a consistent drop in both AUROC and
selective performance compared with using the same models with softmax the x (see Appendix K and Figure 3). MC-Dropout's underperformance was also previously observed in (Geifman & El-
Yaniv 2017)
4 UNCERTAINTY DUE TO
When the underlying distribution P(x, y) used to train a model changes, may longer expect that the model will perform correctly. Changes P can be the result of many natural or adversarial processes such natural deviation in the input space X, noisy sensor reading of inputs, abrupt changes due to random events, newly arrived or refined input classes, etc. We distinguish between input distributional changes in Pxjy and changes in the label distribution. We focus the latter case and consider the -ffis-ootoo--disiibutii changes different set, YOOD. which contains new classes that were not observed in training. We note that both aspects of distributional deviations have been considered in the literature (Hendrycks) Dietterich_ 2019 [iang et al.) 2017] Hendrycks al. 2021), and number of methods have been introduced to deal with these cases (Liang et al. 2017 Lee et all 2018 Golan & EI- Yaniv 2018).
(C-OOD) scenario where the label support set y
We consider the following detection task, in which our model is required to distinguish between samples belonging classes it has seen in training, where x P(=ly ))), and novel classes, i.e.,
I P(rly E YOOD). We examine the detection performance of DNN classification models that their confidence rate function to detect OOD labels where the basic premise is that instances whose labels are in VOOD correspond to low x values.
A crucial question in any study of distributional deviations is what we choose as experimental data to proxy meaningful deviations. For our study of C-OOD we introduce a novel method for generating C-OOD data with controllable degree of severity. Let VOOD be a large set of OOD classes (e.g.. labels from ImageNet-21l 21kk, and let s(y) f, x) be a severity score that reflects the diffi- culty of model f, which to detect instances from class y E YooD. Having defined function

---

Under review as a conference paper at ICLR 2022
s(ylf, 6) (see details below) build multiple C-OOD datasets with progressively increasing severity levels. Importantly, the resulting C-OOD data is specifically tailored to model f itself (and its k).
Given a model f (and its K), we define s x) to be the average confidence given by k to samples from class y VOOD. When considering IDD instances confident predictions. Therefore, the larger syl.f, 66) is, the harder it is for x to detect the OOD class y among ID classes. We estimate s(ylf.k] for each class in ImageNet-2IK (not in ImageNet-IK) using a sample from the class and take as C-OOD data different sample from that class. Using we sub- sub-sample 11 groups of classes (severity levels) from VOOD, with increasing severities, such that severity level i is the tth percentile of all severities. We further expand on how we chose the severity levels, their statistical meaning and the construction of the multiple C-OOD datasets for our experiments in Appendix L
expect to give high values for highly
100
90
C-000 AUROC (dutacctio))
80
70
80
Models: -VTT-L/16-364
AlexNet
54
55
ImogeNet-O
Severity Levels
Figure 6: OOD performance across 11 severity levels. Note how the detection performance de- creases for all models as we increase the difficulty until it reaches near chance detection performance at the last severity (s (810). The top curve belongs to ViT-L/16-384, which beats all models at every severity level. We also observe how the previous C-OOD benchmark, ImageNet-O does not reflect the true OOD performance of the models, since it was designed to specifically fool ResNet-50, and so it is more difficult for models similar to ResNet-50 than other models.
Figure 6 presents the C-OOD detection performance of 484 models across 11 C-OOD severity levels (recall that severity levels are constructed individually for each model). The box plots clearly show a monotone AUROC performance degradation. In addition, the best model for each level (recall that this model is also the best for ID).
that ViT-L/16-33 3-84 is consistently
Most works OOD detection small scale datasets that generally do not resemble the training distribution and, therefore are, easy to detect. The use of such sets often causes C-OOD detectors to appear better than they truly in harder tasks. Motivated by this deficiency, [Hendrycks SeeficienyyHeenrrckkss introduced the ImageNet-O dataset as a solution. ImageNet-O, however, has two limitations. First, it lacks severity levels. Second, the original intent in the creation of ImageNet-O was to include only hard C-OOD instances.The definition of "OOD hardness", however, was carried out with respect ResNet-50's difficulty in detecting OOD classes. This property makes ImageNet-O strongly biased. Indeed, the right-most box in Figure corresoonds to the performance of the 484 models over ImageNet-O. The orange dot in that box corresponds
(2021)
ResNet-50, whose OOD detection perfromance is severely harmed by these data. Nevertheless. it is evident that numerous models perform quite well. this respect, can be argued that the proposed C-OOD dataset generator (see Appendix has better properties and is clearly, not biased toward single architecture.
The next question we ask is does ID uncertainty estimation ranking performance indicate better C-OOD detection performance (and vice versa)? Figure 7 shows scatter plot of ID C-OOD AUROC performance of all the tested models. The overall Spearman correlation is 0.43. The legend indicates correlations obtained by specific families. For instance, ViT's among the most
8

---

Under review as a conference paper at ICLR 2022
Models tipesiman conelation fumily, ouiuber of modes))
Various (0.40, 388)
ShufleNNer22 (1.00, 2)
AlexNett
SqueezeNet (1.00
ResNest (0.16, 33)
Swin (0.83, 6)
ViT (0.74, 21)
A ficientNerV2 (0.46, 15)
COOD AUROC (detection)
MLP Mixer (1.00,
+ BIT (0.32. 9)
80
Overall Spearman correlation 0.43
76
78
80
82
84
86
ID AUROC
Figure 7: OOD detection performance in severity vs. In-Distribution uncertainty estimation per- formance. We notice that the best performing network in one task is not the best in the other, but both belong to the same family, ViTs
correlated largest sample size family (0.74). This means that a ViT model is likely to perform well in both ID and C-OOD. Note that the worst performing models in C-OOD detection are small, optimized models. We further discuss correlations with other factors such as accuracy, model size, input size and embedding size in Appendix [M.7
Due to lack of space, number of additional interesting observations and results are presented in Appendix M We mention the most interesting ones here: (1) In accordance with the ID results (see Section 3 among all training regimes, distillation improves performance the most across all severities; see Figure2T]in Appendix M.3 (2) the outset could anticipated that ImageNet2 Ik pretraining will hinder C-OOD detection performance (due to its exposure to the OOD classes in training). Surprisingly, we observe that pretraining on ImageNet2 Ik somewhat helps performance at severity levels up to level 6; see Figure 21 in Appendix M.3 (3) ViTs appear to achieve the
best C-OOD detection performance per-model size (# parameters); see M. improves C-OOD detection performance in most cases; Appendix M.4 (5) The of
(4) Using entropy,
MC-dropout for C-OOD detection is investigated in Appendix[MS.
CONCLUDING REMARKS
We presented a comprehensive study of the effectiveness of numerous DNN architectures (fam- ilies) in providing reliable uncertainty estimation, including the impact of various techniques improving such capabilities. Moreover, we considered both in-distribution and novel (graded) class- out-of-distribution settings. Our study led to many discoveries and perhaps the most important are: (1) architectures trained with distillation almost always improve their uncertainty estimation performance, (2) temperature scaling very useful not only for calibration but also for ranking and selective prediction, and (3) no other DNN (evaluated in this study) had ever demonstrated an uncertainty estimation performance comparable-in any metric tested setting, in-distribution or class-out-of oft--oottt-------oooee distriibuiionn to the VIT architecture.
Our work leaves open many interesting avenues for future research and we would like mention few. Perhaps the most interesting question is why distillation is so beneficial in boosting uncertainty estimation. Next, what the architectural secret in vision transformers (ViT) that enables their uncertainty estimation supremacy. This question is even more puzzling given the fact that ViT supremacy not shared with many other supposedly similar transformer-basee models that we tested such Touvron et al. (202lb] 2020); Liu et al. (2021); Han et al. (2021); Graham et al. (2021) I d' Ascoli et al. (2021): : Heo et al. (2021); Xu et al. (2021) ;; El-Nouby et al. (2021): Zhang et al. 2021); Chu al. (2021) y Chen et al. (021bb. Finally,
create specialized training regimes (c.g., Geifman EI- Yaniy (2019), specialized augmentations, or even specialized neural architecture search (NAS) strategies that promote superior uncertainty estimation performance?

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
The weights for all models evaluated in this paper are publicly available, and taken from the Py- Torch and timm respositories (Paszke et al. 2019 Wightman, (019)). see Section [ The algorithm for constructing our C-OOD dataset out of ImageNet2 grouped into various severity levels, is detailed Appendix L
REFERENCES
Rakefet Ackerman, Avi Parush, Fareda Nassar, and Avraham Shtub. Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior, 54:101-113, January 2016. doi: 10. 10.1016/j.chb.00 2015..0h..01.....044.....
URL https:
//doi.org/10, 1016/3. chb. 20101010000 07.041
Rakefet Ackerman, Avigdor Gal, Tomer Sagi, and Roee Shraga. A cognitive model of human bias in matching. In PRICAI 20/9 Trends in Artificial Intelligence pp. 632 646. Springer Interna- tional Publishing, 2019, doi: 10.1007/978-3-098
00000000299988850 50. hRttts/s:///do
8_00
Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik-Manor, Knapsack pruning with inner distillation. CoRR, abs/2002.08255. 2020. URL https:://sxxiv.org///
2002.0822
Alexandra Basile, Maggie E. Toplak, and Brendan Andrade. Using metacognitive methods to examine emotion recognition in children with ADHD. Journal of Attention Disorders, 25(2): 245-257, November 2018. doi: (0.1177108705471
URI.https: orl/
Glenn W. Brier. Verification Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1):1, January 1950. doi: 10.1 10.11511880000000550000000000000
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pretraining or strong data augmentations. CoRR, abs/2106.01548, 2021a. URL https:
// www.ii..rggaass../.ooo.aoa 01154
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. CoRR, abs/2104. 12533, 2021b. URL dictp://arxiv..e
abs/2104, 2533
C. Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, 2C-6(4): 1957. doi: 10.1109/TEC.. 1957.522257555233
Xiangxiang Chu, Zhi Tian, Yuqing Wang. Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers, 2021.
L. P. Cordella, C. De Stefano, F. Tortorella, and M. Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neural Networks, 6(5): 100:1140-1144 1147, 1995. doi: 10. 1 00.1109772.413555
Stéphane d' Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. CORR, abs/2103. 10697, 2021. URL
C. De Stefano, Sansone, and M. Vento. To reject or not to reject: that is the question-an an- swer in case of neural classifiers, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(1):84-94, 2000. doi: 10. 827459766...775577
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 00.1109/CPRR.2009552266 5206848..
10

---

Under review as a conference paper at ICLR 2022
Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. Evaluation of neural network uncertainty estimation with application to resource-constrainee platforms. CoRR, abs/1903.02050, 2019.
http://// http//aaxiv.vm org/ aww.aas...3.. 02050
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. CoRR, abs/2010. 1929, 2020. URL https:: /ttpi//aarxiv... hwapg///bb///0
11929
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jegou. Xcit: Cross-covariance image transformers, 2021.
Ran Ylnva and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
Tom Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 2768):861-874, 2006. ISSN 0167-8655. doi: http:///.................///....
10.0100 URL https://ww/
sciencedirect. www.rreetccooscaiee ysis in Pattern Recognition.
ROC Anal-
K. Fiedler, Rakefet Ackerman, and Chiara Scarampi. metacognition Monitoring and controlling one knowledge, reasoning and decisions. 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural net- works. CoRR, abs/1803.0361 2018. RRLLittp::/aaxx..orr///aaa
03635
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. 2016.
Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr. Res2net: new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(2):655 662, Feb 2021. ISSN 1939-3539. doi: 10.1 10.1109/pami.2019 2938758. URL http: doi org/ 1109/TPAMT 201 9. 2938758
Yonatan Geifman and Ran El-Yanis Selective classification for deep neural networks. CoRR, abs/1705.00800, 2017. URL
http:////ggbbbbbbbbbbbabaab 08500
Yonatan Geifman and Ran EI- Yani.. Selectivenet: A deep neural network with an integrated reject option. CoRR, 1ttp//991..9112.. 2019. URL httpp://axxiv..r.pp/poooo wttp:///gggbbb//....b//// 09192
Yonatan Geifman, Guy Uziel, and Ran El- Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. International Conference on Learning Representations, 2018.
Izhak Golan and Ran El-Y Vaniv.. Deep anomaly detection using geometric transformations, 2018.
Leo A. Goodman and William H. Kruskal. Measures of association for cross
classifications*. Journal of the American Statistical Association, (96268) - -764, December1954. doi URL https: http:::://ooo..og////// 1080/ 000/001121155 621459119.. 954.
732
105C1231
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. CoRR, abs/2104.0113 2021. URL htttt::////eee http::////xxvvvoo/
www.iiii.rrrr..oooooo org/ abs 2204. 01136
Thomas D. Griffin, Jennifer Wiley, and Keith W. Thiede. The effects of comprehension-tee. ex- pectancies on metacomprehensioon accuracy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 45(6): 1066))0006-199990 June 2019. 10.1037/xlm0000634 URI https:
7/doi. 1037 10.37/x1m000000 00006334
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural 04599
networks. CoRR, abs/1706.00 706.04599 2017. URIhttp: htttt:://axxvv.rrrttt

---

Under review as a conference paper at ICLR 2022
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. CoRR, abs/2103.00112, 2021. httptttt/////i....oo www.xx...og//////////.e 00112
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common conrptiion and perturbations, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15262-15271, 2021.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. CoRR, abs/2103.16302, 2021. URL
hetps://arxiv.. org. abs / 2103. 16302
Philip A. Higham and D. Paul Higham. New improved gamma: Enhancing the accuracy of -roskal- gamma using ROC curves. Behavior Research Methods, 51(1):108- 125, September 2018. 0177588813322--18--111
kruskal's
URL
913428-018-1125-1
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, Quoc V. Lc, and Hartwig Adam. Searching for mobilenetv3. CoRR, 120444955.2244, 2019. URL http:://aaaa
org/ abs/ 1905. 02244
Armen Der Kiureghian and Ove Ditlevsen. Structural Safety, 31(2):105
Aleatory
epistemic? does ISSN 0167-4730.
matter? http://.o.ooyyll...////ssssss.s0 /ctp://ccaanne//tccccaa////////0
112, 2009.
URL http:/ /www. wwwencedireeeeeeeoeo.ee....oo
Risk Acceptance and Risk Commu-
nication.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributio samples and adversarial attacks, 2018
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability -ill----dittitttionn image detection in neural networks. arXiv preprint 1tti:::17060269900 2017.
Ming Lin, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Neural architecture design for gpu-efficient networks, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. CoRR, 0bs/22033.14030 2021.
220.3 14030
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. CoRR, abs/1902.02476, 2019. URL
http::/aax..v.o///abbb
02476
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pre- training, 2018a.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. CoRR, ftte/8805.009 2018b. Utthtt:://arre wttp://wwwwwww///.e.e/e//// 0005/
12

---

Under review as a conference paper at ICLR 2022
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. CoRR, abs/2007.01458, 2020. URL https://arxiv.on abs/ 2007.01458
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael W. Dusenberry, Sebastian Far- quhar, Angelos Filos, Marton Havasi, Rodolphe Jenatton. Ghassen Jerfel, Jeremiah Liu, Zelda Ma- riet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim G. J. Rudner, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, and Dustin Tran. Uncer- tainty baselines: Benchmarks for uncertainty & robustness in deep learning. CoRR, abs/2106.04015, 2021. URLhttps:/ http/axxiv..// org abs 2106. 04015
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI 15, pp.
AAAI Press, 2015. ISBN 0262511290.
Niv Nayman, Yonathan Aflalo, Asaf Noy, and Lihi Zelnik-Manor. Hardcore-nas: Hard constrained differentiable neural architecture search. CoRR, abs/2102.11646, 2021. URL https://arxxi>
org htt//ab//2102.. 211.. 11.02.1.664 1466
Thomas O. Nelson. comparison of current measures of the accuracy of feeling-of-knowing
predictions. Psychological Bulletin, 9591):00-113, 1984. 10. 10.1037/0033-299995..I https: / //doi
URL.
95.10.99........... 1.99.109
Jeremy Nixon, Mike Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration in deep learning. CoRR, abs/1904.01165, 2019. URL httt:://aaatt org/ abs/
1904. 01685
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 8034-8035. Curran Associates, Inc., 2019. URL http:/ httt:::///papps.... wetterrrs.eeurree nww. www..ip..cc/pppprr.. an oaapaaaaieeeetteeeeeiieaarrrrrrr
brary. libarry
pdf
Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining the masses. CoRR, hts/2104.10972 2021. htt.stttss:///rxii http:///ggaaab////// 10972
preptaiin for
abs
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1eXiv:1409 1556, 2014.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. CoRR, abs/2106.101 10270, 2021.
10270
Mingxing Tan and Quoc V. Le. Efficientnerv2: Smaller models and faster training. CoRR, abs/2104.00298, 2021. URL
00298
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. CoRR, 2ts/2012.12877, 2020.
12877
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Anmand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. CoRR, fass/205.0040.k 2021a. URL
nttps://arxiis http://axxiv..orr/////////////// 03404
13

---

Under review as a conference paper at ICLR 2022
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. CoRR, Stts/2103.172399 2021b. URL httpp::/axxiv.ont
Abs/2103.17233
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learn- ing Representations, 2018. [tttps:///peeeteeee
net forum Sisrruuse-ir
Monika Undorf and Arndt Bröder. Cue integration in metamemory judgements is strategic. Quarterly Journal of Experimental Psychology, 73(4):629-642, October 2019. 10.1177/174702181188883 /tt
Juozas Vaicenavicius, David Widmann, Carl R. Andersson, Fredrik Lindsten, Jacob Roll, and Thomas B. Schön. Evaluating model calibration in classification. CoRR, abx/1922.06977, 2019.
URL hwtphhtppttttite org/ / abs/ 19922069777
Ross Wightman. Pytorch image models.
bttp:///gitbbb.cmmmm.i.ggh.
pytorch-image-monn
2019.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang. Alan I. Yuille, and Quoc Le. Adversarial examples improve image recognition. CoRR, abs/1911.09665, 2019a. URL http://arxiv.
org org/abs/191n 11111.9966665
Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. Self-training with noisy student improves imagenet classification. CoRR, abs/1911.04252, 2019b. URL http:tttt///axii.rrgge. abs abs/1911. 04252
Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification, 2020.
Weijian Xu, Yifan Xu, Tyler A. Chang, and Zhuowen Tu. Co-scale conv-attentional image trans- formers. CoRR, abs/2104.06399, 2021. URLAxtss///axxxr.rrr////////1 06399
I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification, 2019.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R. Manmatha, Mu Li, and Alexander Smola. Resnest: Split-attention networks, 2020.
Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, and Tomas Plister. Aggregating nested trans- formers. CoRR, 102221055.222.... 2021. htLLttts:// httppps//axxi..ogg/bbb//////////
14

---

Under review as a conference paper at ICLR 2022
A THE INVESTMENT EXAMPLE
Let us consider two classification models for the stock market that predict whether a stock's 's value about to increase, decrease or remains neutral (three-class classification). Suppose that Model A has a 95% true accuracy, and generates a confidence score of 0.95 on any prediction (even on missclassified instances); Model B has a 40% true accuracy. but always gives a confidence score of 0,6 on correct predictions, and 0.4 on incorrect ones. We now try and evaluate these two models with the uncertainty metries mentioned uncertainty estimation performance. AURC will fail due to its sensitivity to accuracy (the AURC of Model is 0.12, more than twice as bad as the AURC for Model A, which is 0.05). NLL will rank Model A four times higher (Model A's NLL is 0.23 and Model B's is 0.93). The Brier score would also much prefer Model A (giving it a score of 0.096 while giving Model B a score of 0.54). Evaluating the models' calibration with ECE will also not reveal Model B's advantages, since it is less calibrated than Model A, which has perfect calibration (Model A has an ECE of O, and Model B has a worse ECE of 0.4).
Section [1] to see which can reveal Model B's superior
AUROC, on the other hand, would give Model B a perfect score of 1 and a terrible score of 0.5 to Model A. The selective risk for Model B would be better for any coverage of stock predictions below 40%, and for any SAC above 95% the coverage for Model A would be O, but 0.4 for Model B.
Those two metrics are not perfect for any example. If instead we were to compare two different models for the task of predicting the weather, such that we cannot abstain from making predictions but are required to provide an accurate probabilistic uncertainty estimation of the model's predic- tions, AUROC and selective risk would be meaningless (due to the model's inability abstain in this task), but ECE or the Brier Score would better evaluate the performance the new task requires.
B RANKING AND CALIBRATION VISUAL COMPARISON
A comparison of 484 models by their AUROC 100, higher is better) and -log(HCE) (higher is better) on ImageNet visualized in Figure X To compare models fairly by their size, we plot two graphs with the logarithm of the number of parameters
the X axis, so that models sharing the same x value can be compared solely based on their y value. In Figure 9 we set the X axis to be AUROC (higher is better), and ViTs outperform any other architecture with comparable amount of parameters by a large margin. We can also observe using distillation creates a consistent improvement in AUROC. In Figure [o] we set the X axis to be the negative logarithm of ECE (higher is better) and observe a very similar trend, with ViT outperforming its competition for any model size.
C DEMONSTRATION OF E-AURC'S DEPENDENCE ON THE MODEL'S S ACCURACY
EXcess-AURC (E-AURC) was suggested by Geifman et al. 2018) as an alternative to AURC (explained in Section [. To calculate E-AURC, two AURC scores need to be calculated: (I) AU RC (Codeell), the AURC value of the actual model and (2) AU RC(model"), the AURC value of a hypothetical model with identical predicted labels as the first model, but that outputs confidence values that induce a perfect partial order on the instances in terms of their correctness. The latter means that all incorrectly predicted instances are assigned confidence values lower than the correctly predicted instances.
F-AURC is then defined as AU RC(mddel) AU (Cmodell") In essence, this metrics acknowl- edges that given a model's accuracy, the area of AU (UURC(( (Coodee(") is always unavoidable matter how good the partial order is, but anything above that could have been minimized if the k function was better at ranking, assigning correct instances higher values than incorrect ones and inducing better partial order over the instances.
This metric indeed helps to reduce some of the sensitivity accuracy suffered by AURC, and for the example presented in Section [ E-AURC would have given a perfect score of 0 to the model inducing perfect partial order by its confidence values (Model B). It is easy, however, to craft
15

---

Under review as a conference paper at ICLR 2022
woneer
A SAM
Piutt
Ann
-EOg(ECE)
*
88
86
82
80
78
AUROC
Figure 8: A comparison of 484 models by their AUROC (x100, higher is better) and log(ECH)
(lower is better) on ImageNet. Each marker's size is determined by the model's 's number of parame- ters. Each dotted marker represents distilled version of the original.
16

---

Under review as a conference paper at ICLR 2022
88
86
AUROC
84
Models
82
Varlous
ResMLP
AlexNet
+ ResMLP distilled
BIT
ResNet
+
distilled
80
EfficientNetV2
VIT
GENet
VIT distilled
MLP Mixer
VIT SAM
Mixer distilled
XCIT
78
RegNetY
XCIT distilled
6.5
7
7.5
8
8.5
log # (arrammeteess)
Figure 9: A comparison 484 models by their AUROC (x 100, higher better) and log(number of model's 's parameters) on ImageNet. Each dotted marker represents distilled version of the original.
Models
Various
AlexNet
BIT distilled
EfficientNetv2
GENet
MLP Mixer
loggEECE)
MLP Mixer distillied
M RegNetY
ResMLP
ResMLP distilled
ResNet
VIT
distilled
VTT SAM
A XXIT
AXCT distilled
6.5
7.5
8.5
lag(#Parameters)
Figure 10: A comparison 484 models by their -log(ECE) (higher is better) and log(number of model's parameters) on ImageNet. Each dotted marker represents distilled version of the original.
examples showing that E-AURC prefers models with higher accuracy, even if they have lower or equal capacity to rank.
To demonstrate this in simple way, let us consider two models with complete lack of capacity to rank correct and incorrect predictions correctly, always outputting the same confidence score. Model has an accuracy of 10% (thus an error rate of 90%), and Model has an accuracy of 50%. A good ranking metric should evaluate them equally (the same way E-AURC gives the same models that rank perfectly regardless their accuracy). In Figure II we plot their RC curves, which both straight lines due to their lack of ranking ability. We calculate both of these models AURCs, AU (mooee)A = 0.9, AU RC(mdeelB) = 0.5.
for two
The next thing to calculate is the best AURC values those models could have achieved given the same accuracy they had a perfect partial order. We plot these hypothetical models' RC curves in Figure 12 Their selective risk remains for every coverage below their total aceu-
17

---

Under review as a conference paper at ICLR 2022
0.9
0.6
Risk
0.5
Model
0.2
0.1
0.1
0.5
0.4
0.5
0.6
Coverage
Figure 11: The RC curves for Models A and B.
racy, since these hypothetical models assigned the highest confidence all of their correct in- stances first. As the coverage increases and they have no more correct instances to select, they begin to give instances that are incorrect, and thus their selective risk linearly deteriorates for higher coverages. Calculating both of these hypothetical models' AURCs gives us the follow-
0.7
0.6
Risk
0.5
ModelA A
Model
0.2
0.1
Coverage
Figure 12: The RC curves for the hypothetically optimal versions of Models A and B.
ing: AURC SUJC (mmodelAA")
0.125. Subtracting our results we get: E RU (C(odeel))) = 0.9 - 0.405 = 0.495, E - AU (C(oddel = 0.5 0.125 0.375
0.405, AU (C
Hence, E-AURC prefers Model over Model A, even though both do not discriminate at between incorrect and correct instances.
D MORE ON THE DEFINITION OF RANKING
Let us consider finite set S. {(z,, w) Рx,Y We assume that there two identical values given by x on S,,. Such an assumption is reasonable when choosing a continuous confidence signal.
18

---

Under review as a conference paper at ICLR 2022
We further denote the number of concordant pairs (i.e., pairs in S. that satisfy the condition [ fllf < x(zz, f n (()()) y. > (.(())))))))) and as the number discordant pairs (i.e., pairs in S., that satisfy the condition [x( ff k(zz, ūlf) n ((f(x,). v:) > {(f(z)). v2)l
We assume, for now, that there are no two identical values given by on S.. Accordingly, we can further develop Equation (1 from 2.1 using the definition of conditional probability.
Pr|m(2,, ūlf) < (())))l)))))) (() (rlx( (l
(((rr)))))) Pr(e(f(z)), v)) > ((f(z)),,))
which can be approximated aepoxxiiateemmiriialllll using the most likelihood estimator, as
(2)
M
We notice that the last equation is identical to Kendall's up linear transformation, which equals
m
2
2c
+c++0 d)
2c
C d
2
2
1 = 2 [Equation 2
1.
2
Otherwise, if the loss assigns two identical values pair of points in S., but does not, then
get:
C
(3)
c d
which is identical to Goodman & Kruskal's 7-correlation to a linear transformation
c- d c d
2c (c d)
C c+ d c+d
c+d
2c c 2. [Equation] 3 1.
c+
d
c d
D.1 INEQUALITIES OF THE DEFINITION
One might wonder why Equation T should have strict inequalities rather than non-strict ones to define ranking. As we discuss below, this would damage the definition:
(1) If the losses had a non-strict inequality:
Pr[x(I1.01) < <( (l)))(((()))))) 91) 2
(2(())))
Consequently, in the case of classification, for example, this probability would increase for any pairs consisting of correct instances with different confidences, which yields benefit in ranking between incorrect and correct instances and motivates giving different confidence values for instances with the same loss-a fact that would not truly add any value.
(2) If the x. values had non-strict inequality:
Pris(11, fl < k(æ2, (1)))(((())()))))))) ((f(2)), 32)].
This probability would increase for any pair (#),22) such that s(21, f) 1(x22, @l.f) and (((())) {(((x2)), although should have ranked in with lower value. Furthermore, if
19

---

Under review as a conference paper at ICLR 2022
function were to assign the same confidence score to all E X, then when there are no two identical values of losses, the definition's 's probability would be 1; otherwise, the more different values for losses there are, the larger it would grow. In classification with 0/1 loss, for ex- ample, assigning the same confidence score to all instances would result in the probability being Accuracy( (1 A.curacy(()) which is largest when Accuracy( 0.5.
E RANKING CAPACITY COMPARISON BETWEEN HUMANS AND NEURAL NETWORKS
In the field of metacognition, interestingly, the predictive value of confidence is evaluated by two different aspects: by its ability to discriminate between correct and incorrect predictions (also known ranking in our context) and by its ability give well calibrated
resolution in metacognition confidence estimations, not being over- or underconfident (Fiedler et al. 2019). These two aspects correspond perfectly with much of the research done in the deep learning field, with the nearly matching metric to AUROC of y-correlation (see Section 2).
This allows us to compare how well humans rank predictions in various tasks and how models rank their own in others. Human AUROC measurements in various tasks (translated from y-correlation tends to range from 0.6 to 0.75 (Undort & Bröder 2019; Basile et al., 2018 Ackerman et al. 2016), but could vary, usually towards much lower values (Griffin et al. 2019). In our comprehensive evaluation on ImageNet, AUROC ranged from 0.77 to 0.88 (with the median value being 0.85), and in CIFAR-10 these measurements jump to the range of 0.92 to 0.94.
While such comparisons between neural networks and humans are somewhat unfair due to the great sensitivity required for the task, research that directly compares humans and machine learning algo- rithms on the same task exist. For example, in Ackerman et al. (2019), algorithms far surpass even the group of highest performing individuals in terms of ranking.
F CRITICISMS OF AUROC AS A RANKING METRIC
In this section we show why AUROC does not simply reward models for having lower accuracy, addressing such criticism. The paper by Ding et al. (201)) presented semi-artificial experiment to demonstrate that AUROC might get larger the worse the model's accuracy becomes. They considerr model f and its function evaluated and a confidence score <(ær, 0 (x)..), which in this case is the model's softmax response. Let X' = {- E X X10r(=*) = y(z)} be the set of all instances correctly predicted by the model f. and define zis) E X to be the correct instance that received the i-lowest confidence score from k. Their example continues to consider artificial model f to be an exact clone of with the following modification: for every i S m, the model fm now predicts a different, incorrect label for og however, its given confidence score remains identical: <(Z(t)y dy (1()))) = x(27 8ym (z()) (f))))))) f is exactly identical to f. by this definition, not changing any predictions. The paper shows how an artificially created model fm obtains a higher AUROC score the bigger its m. This happens even though "nothing" " changed but a hit to the model's accuracy performance (by making mistakes on more instances).
classification test set X, giving each prediction O (x()))
First, to understand why this happens, we note that ft: AUROC for k increases the more pairs of [m(21) x(x2))l (x1) * y(21), = y(*2)] there are. The model ft is now giving an incorrect classification to xf1y. but this instance's 's position in the partial order induced by k has re- mained the same (since 1a (((())) is unchanged); therefore, | correctly ranked pairs were added: (m(2f1)) k(2f)) )or (x())) A v(=(a)) ()(()) y(a( for every 1 < < | Nevertheless, this does not guarantee an increase AUROC by itself: if, previously, all pairs of (conrect.incorrect) instances were ranked correctly by K, AUROC would already be 1.0 for f and would not change for fi. If AUROC for f is higher than was for f", this means there exists at least one instance ". that was incorrectly predicted by the original model fo such that m(x())) < <<(x"). Every such originally wrongly ranked pair (by (a) of (s((2())) < ((="")))(x)) + y(=",, D (x())) = =(((,)) has been eliminated by f wrongly predicting =f1)- This, therefore, causes AUROC to increase at the expense of the model's accuracy.
20

---

Under review as a conference paper at ICLR 2022
Such analysis neglects many factors, which is probably why such an effect is only likely to be observed in artificial models (and not among the actual models we have empirically tested):
1. It is unreasonable to assume that the confidence score given by will remain exactly the same for an instance Tia) given it now has different prediction. In the case of being softmax, it assumes the model's logits have changed in very precise and nontrivial manner. Additionally, by our broad definition of K, which allows x to even be produced from an entirely different model than receives the prediction and model cannot change or affect neither), and it is unlikely to assume changing its inputs will not change its output.
given input (and
2. Suppose we find the setting reasonable and assume we can actually create a model f as described. Let observe model f such that p min,, (AUROC of f*==)), meaning that fP ranks its predictions perfectly, unlike the original fo. it really true that f has no better uncertainty estimation than fd? Model f" behaves very much like the investment "Model B" from our example in Section [ possessing perfect knowledge of when it is wrong and when it is correct, allowing its users risk-free classification. So, given model S. we can use the above process to produce an improved model f?, and then we can even calibrate its above to produce a perfect model, which might have small coverage but is correct every time, knows it and notifies its user when truly knows the prediction. The increase in AUROC reflects such an improvement.
output 0% for all instances below its threshold and 100% for all those
Not only do we disagree with such an analysis and its conclusions, but we also have vast empirical evidence to show that AUROC does not prefer lower accuracy models unless there a good reason for it to do so, as we demonstrate in Figure 2 (comparing EfficientNet V2-XL to ViT-B/32-SAMM In fact, out of the 484 models we tested, the model with the highest AUROC has also the 4th highest accuracy of all models, and the overall Spearman correlation between AUROC and accuracy of all the models we tested 0.03. Furthermore, Figure 2 also exemplifies why AURC, which was suggested by the mentioned paper as the alternative to AUROC, is bad choice single number metric, and might lead us to deploy a model that has a worse selective risk for most coverages only due to its higher overall accuracy.
G AFFECTS OF THE MODEL'S S ACCURACY, NUMBER OF PARAMETERS AND INPUT SIZE ON IN-DISTRIBUTION UNCERTAINTY ESTIMATION PERFORMANCE
Table T shows the relationship between uncertainty estimation performance and model's attributes and resources (accuracy, number of parameters and input size), measured by Spearman correlation. We measure uncertainty estimation performance by AUROC (higher is better) and -ECE (higher better). Positive correlations indiciate good utilization of resources for uncertainty estimation (for example, a positive correlation between -ECE and the number of parameters indicates that as the number of parameters increases, the calibration improves). An interesting observation that distillation can drastically change the correlation between a resource and the uncertainty estimation performance metries. For example, undistilled XCiT's have a Spearman correlation of -0.79 between their number of parameters and AUROC, indicating that more parameters are correlated with lower ranking performance, while distilled XCTTs have a correlation of 0.35 between the two.
H KNOWLEDGE DISTILLATION EFFECTS ON IN-DISTRIBUTION UNCERTAINTY ESTIMATION
compares vanilla models to those incorporating KD into their training (represented by markers with thick borders and a dot). In pruning scenario that included distillation, yellow mark- ers indicate that the original model was also the teacher (Aflalo et al. 2020). While distillation using different model tends to improve uncertainty estimation in both aspects, distillation by the model itself seems to improve only one mne-suggesting is generally more beneficial to use a different model teacher. The fact that KD improves the model over its original form, however, is surprising, and
Figure 13
21

---

Under review as a conference paper at ICLR 2022
The relationship between uncertainty estimation performance and the model's attributes and resources (accuracy, number of parameters and input size), measured by Spearman correlation. Positive correlations indicate good utilization of resources for uncertainty estimation.
Table I:
Architezton:
AUROC &
-BCE Auauny
AUROC 4Panmmccn
-RC & 4Parmncion
AUROC & legue Size
-FE & logue Size
Mascle Evalotol
HitiirreNer
-1
4:20
-0.32
-0.29
-0.26
40.5
ResNet
0.25
0.22
0.16
0.03
-0.00
0.44
XCI duailled
3.50
0.99
0.35
0.02
0.51
0.12
-065
EficicnNaeV2
..70
0.45
-040
NFNet
0.56
looption
-0.29
009
4.44
<008
0.23
RepNorY
-001
4.98
0.27
40.8
RegNexX
0.30
0.96
0.20
0.80
Cui disiiled
0.44
0.87
0.35
0.87
0.58
0.50
DLA
3.64
4.90
0.75
-0.90
WihileheeV3
0.37
033
0.42
20
0.21
4048
RepYGG
-.25
RoNNXi
0.30
RosNet
0.00
0.79
40.18
MixCCom
-0.11
0.89
4024
DenseNet
0.43
0..1
0.72
0.32
-0.00
0.26
40.49
0.37
Swin
0.71
0114
0.77
0.36
0.41
0.00
BCANet
0.20
0.60
0.43
0.37
0.53
0.37
Toins
0.26
0.4
0.14
ReaNet
-1000
TReeNat
0.30
0.87
0.9
0,30
L:WT didillal
0.60
4.900
0.60
+0.00
ResMLP
0.20
1.00
0.15
0.97
MchieNeeV2
01300
000
4021
0.30
WIl dinited
1.00
-100
1.00
-1.00
PST
0.40
1.00
0.40
1.00
WSP ReaVeek
1,00
0.30
1.00
0.50
Pdisnikkal
0.90
030
88
Models
BIT
BIT distilled
DeiT
87
DeiT distilled
ECANet
ECANet distilled
EfficientNet
86
EfficientNet distilled
AUROC
+ MLP Mixer distilled
MobileNNeVV3
85
MobileNettVs distilled
PIT
ResMLP
ResMLP distillie
TResNet
TResNet distilled
XCT distilled
(ECE)
Figure 13: Comparing vanilla models to those incorporating KD into their training (represented by markers with thick borders and dot). In a pruning scenario that included distillation, yellow markers indicate that the original model also the teacher. The performance of each model is measured in AUROC (higher is better) and -log(EEF) (higher is better).
suggests the distillation process itself helps uncertainty estimation. Note that although this specific method involves pruning, evaluations of models pruned without incorporating distillation (Frankle
& Carbin 2018) revealed improvement.
Moreover, it seems that the teacher does not have to be good uncertainty estimation itself; Fig- ure 14 shows this by comparing the teacher architecture and the students in each case.
22

---

Under review as a conference paper at ICLR 2022
Models
BBIT
87
BIT distilled
DefT distilled
ECANet
ECANet distilled
86
EfficientNet
EfficientNet distilled
PIT distilled
AUROC
RegNetY-16GF
ResMLP distilled
XCIT distilled
84
83
2.5
5.5
(EggEE
Figure 14: Comparing teacher models (yellow markers) their KD students (represented by mark- ers with thick borders and a dot). The performance each model measured in AUROC (higher better) and -log(ECE) (higher is better).
While the training method by Ridnik et al. (2021) included pretraining ImageNet-214 and demon- strated impressive improvements, comparison of models that were pretrained on ImageNe121K k (Tan)
& Le 2021 Touvron
2021a) with identical models that
pretrained showed clear
improvement in ECE, and, in fact, exhibit degradation of AUROC (see Figures 3 and in Sec- tion 3p. This suggests that pretraining alone does not improve uncertainty estimation.
I MORE INFORMATION ABOUT TEMPERATURE SCALING
AUROC with Temperature Scaling
88
86
C
84
Models
82
Various
ResMLP
AlexNet
ResMLP distilledi
BiT
ResNet
B/T distilled
EfficientNetV2
80
HardCoReNAS
VIT distilled
Mixer
VIT SAM
GMLP Mixer distilled
XCIT
RegNetY
XCIT distilled
4.5
5.5
6.5
-log(ECE) with Temperature Scaling
Figure 15: A comparison of 484 models after being calibrated with TS, evaluated by their AUROC (x 100, higher is better) and -log(ECE) (higher is better) ImageNet. Each marker's size is de- termined by the model's number of parameters. ViT models are still among the best performing architectures for all aspects of uncertainty estimation.
23

---

Under review as a conference paper at ICLR 2022
In Figure 15 see how temperature scaling (TS) affects the overall ranking of models in terms of AUROC and ECE. While the ranking between the different architecture remains similar, the poorly performing models much improved and minimize the gap between them and the best models. One particularly notable exception HardCoRe-NAS (Nayman et al. (021). with its lowest latency versions becoming the top performers in terms of ECE. In addition, models that benefit from TS in
Method
Temperature Scaling (Temp. < 1)
Temperature Scaling (Temp. > 1)
AUROC improvement over Vanilla
Temperature Scaling (Temp. < 1)
Temperature Scaling (Temp. > 1)
Figure 16: Out of 484 models evaluated, models that were assigned temperature higher than by the calibration process tended to degrade in AUROC performance rather than improve. Markers above the x axis represent models that benefited from TS, and vice versa.
Method
Temperature Scaling (Temp. 1)
0.35
Temperature Scaling (Temp. 1)
ECE Improvemer rerroveeenttt over Vanilla
0.25
0.2
0.1
Temperature Scaling (Temp.
Temperature Scaling (Temp. 1)
Figure 17: The relationship between temperature and the success of TS, unlike the case for AUROC, seems unrelated.
terms of AUROC tend have been assigned temperature lower than by the calibration process (see Figure [6]. The same, however, does not hold true for ECE (see Figure [7]. This example also emphasizes the fact that models benefitting from TS in terms AUROC do not necessarily benefit in terms of ECE, and vice versa. Therefore, determining whether calibrate the deployed model with TS is, unfortunately, a task-specific decision.
We conduct TS ified sampling of 5,000 instances from the ImageNet validation set to calibrate on, and reserve the remainder 45,000 instances for testing. Using the box-constrained L-BFGS (Limited-Memory
suggested in Guo al. (2017). For each model take a random strat-
24

---

Under review as a conference paper at ICLR 2022
Froyden-Fletcher Fletcher-Goldfart algorithm, optimize for 5,000 iterations (though fewer iter- ations usually converge into the same temperature parameter) using a learning rate of 0.01.
ARCHITECTURE CHOICE FOR PRACTICAL DEPLOYMENT BASED ON SELECTIVE PERFORMANCE
As discussed in Section when we know the coverage or risk we require for deployment, the most direct metric check is which model obtains the best risk for the coverage required (selective risk), or which model gets the largest coverage for the accuracy constraint (SAC). While each deployment scenario specifies own constraints, for demonstration purposes misclassifications are by far more costly than abstaining from giving correct predictions. An exam- ple for this could be classifying a huge unlabeled dataset (or for cleaning bad labels from labeled dataset). While it is desirable to assign labels to larger portion of the dataset (or to correct more of the wrong labels), crucial that these labels are as accurate as possible (or that correctly labeled instances are not replaced with bad label).
consider scenario which
8.5
log (#
7.5
Models
Various
X RegNetY
AlexNet
ResNet
BIT
SWSL ResNeXt
DLA
ViT
RegNetX
VIT SAM
10
20
Coverage for Accuracy 99
Figure 18: A comparison of 484 models by their log(number of model's parameters) and the cover- age they are able to provide for a SAC of 99% (higher is better) on ImageNet.
To explore such a scenario, we evaluate all models on ImageNet to see which ones give us the largest coverage for required accuracy of 99%. In Figure 5 Section 3 (paper's main body) observe that of all the models studied, only VIT models are able to provide coverage beyond 30% for such an extreme constraint. Moreover, we that the coverage they provide is significantly larger than that given by models with comparable accuracy or size, and that ViT models that provide similar coverage to their counterparts do so with less overall accuracy.
In Figure 18 we see that not only do VIT models provide more coverage than any other model, but they are also able to do so in any size category. To compare models fairly by their size, we present Figure T8] which sets the Y axis to be the logarithm of the number of parameters, so that models sharing the same y value can be compared solely based on their x value-whie is the coverage they provide for SAC of 99%. We see that VIT models provide a larger coverage even when compared with models similar size.
K EVALUATIONS OF MONTE-CARLO DROPOUT IN-DISTRIBUTION RANKING PERFORMANCE
MC-Dropout (Gal & Ghahramani
2016) computed using several dropout-enabled forward passes produce uncertainty estimates. In classification, the mean softmax score of these passes is caleu-
25

---

Under review as a conference paper at ICLR 2022
Table 2: Comparing using MC-Dropout softmax-response (vanilla).
Architecture
Method
Accuracy
AUROC
Vanilla
74.04
86.88
MobileNetV3 Large
MC-Dropout
74
86.14
Vanilla
67.67
86.2
MobileNeiV3 Small
MC- Dropout
67.55
84.54
Vanilla
71.88
86.05
MobileNetV2
MC-Dropout
71.81
Vanilla
70.37
VGG11
MC MC-Dropour
70.21
84.3
Vanilla
69.02
86,19
VGG11 (no BatchNorm)
MC-Dropout
68.95
83.94
Vanilla
71.59
86.3
VGG13
MC-Dropout
71.43
84.37
Vanilla
69.93
86.24
VGG13 (no BatchNorm)
MC MC-Dropour
69.71
84.3
Vanilla
73.36
86,76
VGG16
MC-Dropout
73.33
85 $5.02
Vanilla
71.59
86.63
VGG16 (no BatchNorm)
Mro-Dropuu
71.47
84,97
Vanilla
74.22
86.52
VGG19
MC-Dropout
74.17
85.06
Vanilla
72.38
86.55
VGG19 (no BatchNorm)
MC-Dropour
72.37
84,99
lated, and then predictive entropy score is used as the final uncertainty estimate. In our evaluations, we use 30 dropout-enabled forward passes. We do not measure MC-Dropout' effect on ECE since entropy scores not reside [0, 1].
We test this technique using MobileNetV3 (Howard et al. 2019). MobileNetv2 (Sandler et al. 2019) and VGG (Simonyan & Zisserman 2014), all trained on ImageNet and taken from Py Torch repository (Paszke et al. 2019).
The results comparing these models with and without using MC-Dropout are provided in Table
The table shows that using MC-Dropout causes consistent drop in both AUROC and selective performance compared with using the same models with softmax as the K. These results are also visualized in comparison other methods in Figure in Section MC-Dropout underperformance was also previously observed in (Geifman El- Yaniv 2017).
L CONSTRUCTING C-OOD DATASET PER MODEL.
Given model f, confidence function and large of classes VOOD (in case, ImageNet- 21K), our goal to build multiple datasets from VOOD that can used in evaluating the model's performance in C-OOD detection. For that we start by pre-processing O.
L. PRE-PROCESSING IMAGENET-2IK
Since ImageNet-2IK contains our ID dataset (ImageNet- the first step will be to remove all IK classes from ImageNet-2IK. After that, as a cautionary step. we remove all classes that are hypernyms or hyponyms of classes in ImageNet- because they might be unfair to include as an OOD class. For example, ImageNet-1K contains the class "brown bear", and ImageNet-21 K has the class "bear" which a hypernym for "brown bear" so it would not be fair to include it in a C-OOD detection test. After cleaning trivial classes, set our threshold to 200 samples. For classes with more than 200 samples we randomly select 200 samples and remove the rest. At the end of this process are left with 12697 classes containing 200 samples each. We divide the samples in each class e into 150 'estimation' samples (denoted by dll ( and 50 'test' samples (denoted by df The estimation samples will later be used estimate
remove classes with a low number of samples; we
26

---

Under review as a conference paper at ICLR 2022
the difficulty of the class and the test samples will be used to check the C-OOD performance if the class was chosen.
L.2 CONSTRUCTION
Having conducted the pre-processing, continue compute the 11 severity levels (C-OOD datasets) for the given model (with its given k).
We define the severity score of class y (ʃ, R) as follows:
1
stylf k)
<(zlf).
cV
We also define the severity score for given group of classes g
s s(gls,m)
g
WEg
Such definition of is intuitive because it assumes that samples that the model highly confident of are hard for it to distinguish from ID samples. We choose the size of each C-OOD dataset to be the same as the size of the of the ID dataset, 1000 classes. The number of subgroups in VOOD of size 1000 is huge ( ((12997 of classes, we choose to sort the classes by their severity score and then use sliding window of size 1000 to define resulting in 11698 groups classes with increasing severities. This choice for reducing the number of considered groups of classes was chosen for its simplicity. Using sgg),, k) we calculate the severity score of each group, and sort them accordingly. Finally we choose the 11 groups to be the groups that correspond the percentiles {10-i) (10-i){000 in the sorted groups array. This way we build the C-OOD) dataset of severity level from the test samples of classes in group i. This heuristic procedure for choosing groups allows to interpret the severity levels with percentiles. For example, severity level contains classes that have the median severity among the considered groups.
3.5 x 101518 groups), so instead of going over every possible group
The reason for choosing the number of classes in each group to be the same as the number of classes in the ID dataset (1000 classes per group) is because we wanted the C-OOD dataset to be equal in size to the ID dataset.
M LIST OF C-OOD OBSERVATIONS
In this section we list additional findings regarding C-OOD detection.
M.1 SIZE COMPARISON
The scatter plot in Figure 19 shows the relationship between the # of architecture parameters and its C-OOD AUROC performance. Overall, there a moderate Spearman correlation of 0.45 between #parameters and the C-OOD performance when considering all tested networks. When grouping the networks by architecture families, however, we see that some architectures have high correlation between their model size and their C-OOD AUROC. Architecture families that exhibit this behavior are, for example, ViTs, Swins, EffecientNetV22 and ResNets whose correlations are 0.91, 0.94, 0.89, and 0.79, respectively. Other families exhibit moderate correlations, e.g., EffecientNer(VI) with a 0.47 Spearman correlation. Some architectures, on the other hand, have strong negative correlation, e.g.. Twins (Chu et al. 2021), NesT (Zhang et al. 2020) and Res2Net (Gao et al.) 2021), whose correlations are -0.94, 1.0, and -0.85, respectively.
Additionally, we note that VIT models are also the best even when considering a model size limita- tion similar to Re ResNet-101.
M.2 HIGH CORRELATION WITH ACCURACY
Similarly, the scatter plot in Figure 20 shows the relationship between the architecture validation accuracy and its C-OOD AUROC performance. Based on their apparent correlation, increasing
27

---

Under review as a conference paper at ICLR 2022
Models
Spearman correlation 0.45
Venious (0.
AlscNiil 1)
ResNer (0.79, 53)
(0.47,55
C.00D AUROC (detectionn)
Toins ( (0.54,0 6)
VIT (0.9L, 21)
(-332,
NosT (-1. 5)
ShutteNeeV2
SquaszeAas (-1 11.000,21000000000
Swin (0.94, 6)
EfticcntNerr2 (0. 15)
BIT (0.54,
70
log10 Parameters
Figure 19: Number of architecture parameters C-OOD AUROC performance at severity level (median severity). The pair of numbers next to each architecture name at the legend correspond to its Spearman correlation and the number of models tested from that architecture (family), respectively. Note that ViT transformers are also the best when considering a model size limitation. Vertical lines indicate the sizes of ResNet-50 (left vertical line) and ResNet-101 (right vertical line).
Models
Spearman correlation
Various (0.61,327)
AlaxNeet (,
ReaiNer (0.73, 33)
C-OOD AUROC (detection)
Toins (-1 10, 6)
08,88,
85
Miver 80, 4)
NesT (-1 3)
ShulleNetv2 (1.00,
00
SqueezeNet (1.00,
Swin 89,
(0 (0.88,
BIT (0.63, 5)
75
70
55
60
65
70
75
80
85
90
Accuracy
Figure 20: Architecture accuracy vs. C-OOD AUROC performance. In the legend, the pair of numbers next to each architecture name correspond to the Spearman correlation and the number of networks tested from that architecture (family), respectively. Accuracy appears to have a high correlation with the C-OOD detection performance, with a Spearman correlation of 0.65. Most architectures also hold this general trend except for Nest and Twins. Next to each architecture we report the Spearman correlation value and the number of networks tested from that architecture.
accuracy could indicate better C-OOD detection performance. When grouping the networks by architecture, notice that most architectures also hold this trend. In Appendix M.7] M.7] examine how the correlation between C-OOD AUROC and accuracy (among other metrics) change with severity.
M.3 TRAINING REGIME EFFECTS
We evaluate the effect of several training regimes C-OOD performance various severity levels. The regimes we consider are: (1) Training that involves knowledge distillation in any form: (2) Ad- versarial training: (3) Pretraining ImageNet21k; (4) Various forms of weakly semi-supervised
28

---

Under review as a conference paper at ICLR 2022
Improvement
Training regimes:
vanilla Distillation
vanilla
vanilla
vanilla
Adversarial Training
vanilla
ImageNet21k Pretraining
Severity Levels
Figure 21:
The average relative improvement when using distillation, pretraining, semi-supervised learning and adversarial training. The shaded green area indicates the area of positive improvement.
learning, including noisy student (Xie et al. 2020) and semi-supervised pretraining Yalniz et al. 2019), which use extra unlabeled data; (5) Weakly supervised training (Mahajan et al. 201 8a), which uses billions of weakly labeled Instagram images. The relative improvement generated by each one of the training regimes depicted Figure[1]] We observe that distillation has a positive improvement on C-OOD detection performance all severity levels, adversarial training has mini- mal effects on performance, and, somewhat surprisingly, training regimes that include pretraining on ImageNet-21 -1 and WSP) does not hinder performance in detecting OOD classes at lower severity levels, but degrades performance on higher severities which is what originally expected (due exposure to the OOD classes in training).
larger datasets (pretraining
M.4 ENTROPY VS. SOFTMAX
Improvement by using entropy instead of softmax
Improvement t0 0O0 AUROC (eetetio))
Severity Levels
Figure 22: Relative improvement gain in C-OOD detection performance when using entropy instead of the softmax confidence signal. In median network terms, entropy offers positive improvement over softmax in most serverities except E {7,8.9} 9}. The green shaded area indicates the area of positive improvement.
In our research we evaluated entropy as an alternative confidence rate signal (k) for C-OOD detec- the algorithm described in Section for extracting the classes
tion For each network f
Entropy is maximal when the distribution given by the network for P(y]ar) is uniform, which implies high uncertainty. To convert entropy into confidence signal negative entropy.
29

---

Under review as a conference paper at ICLR 2022
for the 11 severity levels for (f, Kentropy) and them to benchmark the C-OOD detection for (f: Nentropy) (Note that using the same C-OOD groups produced when using softmax might yield an unfair advantage to entropy). We compare the performance gain from switching to using entropy instead of the softmax score. The results are depicted using box-plots in Figure 22 in most cases using entropy improves the detection performance.
We notice that
M.5 MONTE-CARE DROPOUT FOR C-OOD DETECTION
We evaluate MC-Dropout Gal & Ghahramani 2016) in the context of C-OOD detection. We use 30 dropout-enabled forward passes. The mean softmax score of these passes is calculated, and then predictive entropy score is used the final uncertainty estimate. We test this technique using MobileNetV3 (Howard et al. 2019), MobileNetv2 (Sandler et al.) 2019) and VGG (Simonyan & Zisserman 2014), all trained ImageNet and taken from the Pyforch repository Paszze et al. 2019).
For each of these three architectures re-run the algorithm described in Section L extract the classes for all 11 severity levels for (ʃ, dropoa) and use them to benchmark the C-OOD detection for (J, /iMC
We find that MC-Dropout improves perfor- mance at lower severity levels but degrades higher ones (with few exceptions). We further analyze MC-dropout and recall that it is composed of two main components: (1) dropout-enabled forward passes (2) entropy of the mean probability vector from the forward passes. To test which component contributes the most to the perceived gains, we compare the C-OOD detection perfor- mance when using MC-dropout the C-OOD detection performance when using entropy. We find that MC-dropout fails to improve upon entropy in most cases across all severity levels. This implies that MC-Dropout owes its C-OOD success entropy and not its forward passes. These results can be seen in Figure 24
The results are depicted using box-plots in Figure 23
Improvement by using MC-dropout instead softmax
10
Ing iproomment C-00 AUROC (tenecto)
-5
-10
-25
510
Saverity Levels
Figure 23: Relative improvement gain in C-OOD detection performance when using MC-Dropout instead of softmax confidence signal. We find that MC-dropout improves performance for most networks in severities up to 85. and degrades performance for most networks in higher ones. Some outlier networks get a significant improvement when switching to MC-dropout at high severity lev- els.
M.6 CORRELATION BETWEEN RANKINGS OF MULTIPLE SEVERITY LEVELS
Since we essentially have multiple benchmarks C-OOD datasets (i.e., the 11 severity levels) to test the performance models in C-OOD detection, and each severity level may rank the models differently. now consider the question of how does these ranking change across severity levels. To this end we calculated the correlations between the rankings obtained at different severity levels. Overall observe high correlations,
The resulting correlation matrix be seen in Figure 25
30

---

Under review as a conference paper at ICLR 2022
Improvement by using MC-dropout instead of entropy
ingrovement Win C-000 AUROC (minetion)
-5
-15
-20
-30
50
52 54
25
56
57
58
510
Severity Levels
Figure 24: Relative improvement gain in C-OOD detection performance when using MC-dropout entropy confidence signal. We see that MC-dropout fails to improve upon entropy in most cases across all severity levels. This suggests that the main component MC-dropout benefiting detection is the usage of entropy.
which means that different severity levels generally yield similar rankings of the models. We also notice that for each severity level s;, the correlation with sy is higher the closer j is to i. This is not surprising and might be anticipated because adjacent severity levels have close severity score by design.
$10
0.375
0.416
0.451
0.513
0.549
0.576
0.503
0.034
0.575
0.812
1.0
58
0.787
0.846
0.883
0.916
0.344
0.963
0.979
0.989
1.0
0.961
0.675
57
0.815
0.879
0.915
0.946
0.989
0.903
1.0
0.989
0.509
0.634
55
0.831
0.938
0.966
0.983
0.983
0.982
0.979
0.913
0.603
85
0.845
0.910
0.955
0.90
0.983
1.0
0.983
0.983
0.953
0.808
0.578
Spearman
0.968
0.983
0.969
0.543
63
0.882
0.955
0.985
1.0
0.991
0.98
0.366
0.946
0.915
0.824
0.513
s1
0.985
1.0
0.986
0.955
0.981
0.918
0.9
0.879
0.846
0.738
0.416
0.378
-0
62
64
85
67
68
610
Figure 25: Spearman correlation between the rankings of the models given by different severity level.
M.7 CORRELATIONS OF VARIOUS FACTORS WITH C-OOD DETECTION PERFORMANCE
We searched for factors that could be indicative or correlated with good performance in C-OOD detection. To this end we measure the correlations of various factors with the C-OOD detection AUROC performance across all severities. The results can be seen in the graphs of Figure 26 We observe that accuracy is typically a good indicator of the model's performance in C-OOD detection at most severity levels (so - 58), with Spearman correlation values in [0.6, 0.7] at those levels. The next best indicative factors are the ID-AUROC performance, number of parameters, and the input image size (moderate correlations). Finally, the embedding size is only weakly correlated. Inter- estingly. ID-AUROC exhibits slightly increasing correlation up to severity sy, and at $10 becomes the most indicative factor for C-OOD detection. In contrast, all other investigated factors lose their indicative power at the highest severity levels (8g, 810).
31

---

Under review as a conference paper at ICLR 2022
Accuracy ID AUROC
0.7
Embedding Size
0.5
Imput Sine
Spearman Correlation
0.4
0.2
0.1
Severity Levels
Figure 26: Spearman correlations between C-OOD detection AUROC and Accuracy, ID-AUROC,
#Parameters, Input size, Embedding size across all severity levels.
M.8 ANALYSING THE C-OOD CLASSES CHOSEN BY BACH MODEL
When analysing the classes chosen by the algorithm (described in Section L for the various models and severity levels, we found that for each severity level, the union the classes chosen for all models (for that severity level) spans all the classes in VOoD (namely, all the classes in ImageNet- 21K after filtration). This implies that hard OOD classes for one model can easy for others.
In addition, for each severity level (except $10) there is no single class that is shared between all models. However, in $10 we find that there are some shared classes between all models. Moreover, there are specific instances (images) in those classes that obtain the same wrong prediction from many models; one interesting example for this shared wrong prediction of 105 models appears in Figure 27 where the butterfly image is wrongly classified as fox squirrel.
Input image
Predicted class (Fox Squirrel)
Figure 27: An instance of viceroy butterfly predicted be a fox squirrel by 105 models.
32


Reviews:
[{"review_id": "o2BeGwlATQV", "reviewer": "Reviewer_tT7p", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really loved this paper.\nMy only real criticism is that I felt like the paper could use some extra editing and structuring to make sure it has the impact it deserves to.\nThe abstract felt slightly too long and didn't draw focused enough attention to the main contributions of the paper, dwelling too much on details.\nThe introduction dives into specific trade-offs for various metrics before the metrics have been properly introduced. I would recommend moving the second half of the third paragraph and the fourth  and fifth paragraphs to a later section *after* the metrics have been defined.\nI'd be tempted to slightly adjust the structure to: introduce the metrics; discuss the relative merits of the metrics; in-dist performance; introduce the novel OOD evaluation (which is a great contribution); ood performance; concluding remarks.\n\nI really like the example in paragraph 2. I've not seen this made so explicit before and I think it does a great job motivating the work.\n\nI also really like this paper as an example of how to make use of libraries of pre-trained models but nevertheless do extremely thorough empirical work.\n\nDo you think the architecture-dependent correlations between ECE and AUROC are 'real'? Would this extend to other datasets? It makes me wonder if there's something unreliable going on there. In general it might be good to comment more on the limitation of relying on ImageNet, and add some consideration of how this might differ for other datasets/data modalities.\n\n`both metrics are also' should be capitalized.\n\nTitle of Appendix G should be `Effects', not `Affects'", "summary_of_the_paper": "The authors make use of 484 models pre-trained on ImageNet to assess both in- and out-of-distribution uncertainties along a number of metrics.\nThey not only provide a critical evaluation of the different metrics and their task-dependent strengths and weaknesses, but also discover previously unknown empirical patterns in different architectures and techniques.", "summary_of_the_review": "This is a great paper. I have some quibbles about structure, but the basic results are extensive, interesting, and important. I would like to see this paper highlighted at the conference and would be disappointed if it were rejected (unless another reviewer points out to me prior work that I am not aware of).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gTme8EWcf0u", "reviewer": "Reviewer_i3hp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The capability of models to reflect a calibrated uncertainty is interesting for different applications that require a reliable confidence indicator for the model predictions. However, some points regarding clarity on the document can be commented on. \n \nThe paper makes many references to figures, and sections of the appendix making the discussion of the work feel split between both documents. The appendix should be complementary and the paper should be self-explanatory, providing only complementary information like demonstrations or additional results. In this regard, it is recommended to review the information and results included in the main paper, compared with the content of the appendix, and include the necessary introduction to the concepts and definition, making it less for readers necessary to jump into the appendix and back to the paper. \n\nMore details about how the severity level can be included, even though the details are included in one section of the appendix (L), some introductory details can be added to the main paper. \n\nSimilarly, indicate some details about the models used, categories, criteria for selection. ", "summary_of_the_paper": "The paper presents an evaluation of different models in their capacity to reflect epistemic and aleatoric uncertainty and reviews different methods for uncertainty performance measurement. Models are classifiers trained on ImageNet. The number of models reported is 484.  The analysis also considers the cases when the data is in-distribution and out-of-distribution. The conclusions lead to a group of models (trained with distillation), improving uncertainty estimation performance, and the vision transformers architecture having the best uncertainty estimation performance.   \n", "summary_of_the_review": "In general, the document presents an interesting review about uncertainty estimating performance related to the models, and training strategies, however, perhaps due to space limitations, the main paper can present a general description of the work and rely on the appendix to give details. This can make the paper complicated to follow since in some cases, it might be necessary to jump into the appendix before continuing with the lecture of the main document.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cVducdbOOxi", "reviewer": "Reviewer_Wgpg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n\n1. The authors provided a thorough comparison of different architectures and training strategies. \n2. The study compares 484 deep networks.\n3. Their main strength is the observations on knowledge distillation and how providing extra information in any form either as pre-training or learning from unlabelled data helps in getting better uncertainty estimates.\n4.  The authors compared different networks in OoD detection tasks of varying severity levels. \n5.  They provide a comprehensive experimental design that considers OoD as found in real-applications that are not necessarily very far from training distribution.\n6.  The authors demonstrate the strengths of the temperature scale (TS) method in proving ECE and uncertainty estimates.\n7.  The paper demonstrates the strength of ViT architecture in providing better uncertainty estimates and predictive performance. \n\n\n### Weakness\n1.  A comparison with existing strategies of improving uncertainty estimates for a single-pass neural network is missing.\nMany methods have been proposed which improves the uncertainty quantification for any network architecture. For examples\n\n*  Mix-up learning: It is a training strategy to improve uncertainty estimates and model calibration [1].\n* Strategies that impose a bi-Lipschitz constraint on feature extractors to learn a distance-aware feature representation for better OoD detection. A two-sided gradient penalty can be added to a network as an extra soft constraint [2]. \n*  Spectral normalization can be ​​combined with any architecture which has residual connections, resulting in better uncertainty estimates from that architecture [3].  \n* Adding extra regularization to learn an ordinal ranking relationship between different samples for better selective prediction [4]. \n* Temperature scaling as explored by the study\n\n2. Experiments in Figure 3 and 4 show an improvement in ECE and AUC-ROC. This improvement over vanilla training came at a cost, such as the cost of training on unlabelled data as in semi-supervised learning. \n\n3. Previous work, [4] has shown that Softmax entropy cannot capture epistemic uncertainty. Hence, the final output layer of the classifier, irrespective of the architecture, is usually not sufficient to capture epistemic uncertainty arising from OoD test samples.\n\n4.  In section 4, the authors used the average confidence given by κ to identify OoD samples. Average confidence is a measure of softmax entropy and hence, is not sufficient to capture epistemic uncertainty.\n\n5.  The figures 1, 5 and 7  are not much readable. \n\n6.  In Figure 2, it’s not clear how to choose the sample Sm, so that we get different coverage. For example, most of the models report results on 100% coverage over the test sets of benchmark datasets. Should we mix some OoD samples in Sm so that we expect the model to have less than 100% coverage?  Or given any Sm, we partially rank samples based on confidence score, and then use different thresholds of confidence score to obtain a specific coverage level?\nIn this case, what if the model makes overconfident predictions, and hence confidence score is not sufficient to obtain the desired coverage level?\n\n\n### Actions\n\n1.  I would like to see quantitative comparison among \n*  baseline models, \n*  baseline model + [extra constraints for improved uncertainty [1-4], \n*  Models with improved architecture\n\t\nThis quantitative comparison will help me understand what part of the improvement in uncertainty quantification can be achieved from a given baseline model (+ extra constraints), before adapting to a more complex architecture. \n\n2. I would like to see extended results in Fig. 3 and 4 to include quantitative comparison among \n* baseline models, \n* baseline model + [extra constraints for improved uncertainty [1-4]], \n* Baseline model with different KD methods.\n\t\nThis quantitative comparison will help me understand what part of the improvement in uncertainty quantification can be achieved from a given baseline model (+ extra constraints), before adding KD.\n\n3. For OoD detection, I would like to see a comparison of \n* baseline models such as ResNet\n* Models with advance architectures (such as ViT)\n*Feature-density based models that use data likelihood estimates to identify OoD samples.\n\t\nThis quantitative comparison will help me understand if advanced architectures reduce the need for extra feature-density based head such as gaussian mixture model to capture epistemic uncertainty.\n\n4. For results in Fig.3 and 4, the authors should add analysis to quantify the added cost of knowledge distillation vs gain in ECE/Accuracy.\nReport comparison on metrics such as extra training/pre-processing time.\n\n5. I would like to see a through discussion on the results, summarizing the key findings in the form of clear guidelines: \n* A guideline to follow when deciding for model architecture for improved uncertainty estimates.\n* A guideline to consider when choosing different knowledge distillation (KD) methods for learning better classifiers. Do KD methods perform superior to baseline models [extra constraints for improved uncertainty]?\n\n6. The authors should consider adding more text to explain the results in Figure 2.\n7. The authors should consider adding a table to summarize the different architectures explored in the study. \nProvide additional details on how different models with the same baseline architectures but different sizes or losses or knowledge distillation methods are considered. This table will greatly improve the readability of the current draft.\n\n\n[1] On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks\n\n[2] Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation\nusing a single deep deterministic neural network. In International Conference on Machine\nLearning, 2020.\n\n[3] Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In Advances in Neural Information Processing Systems 33, 2020.\n\n[4] Confidence-Aware Learning for Deep Neural Networks\n\n[5] Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric Uncertainty\n\n\n", "summary_of_the_paper": "The paper provides an empirical comparison of uncertainty estimates obtained from 484 deep neural networks (DNN), trained for image classification tasks on the ImageNet dataset. They compared uncertainty estimation performance of different architectures and training strategies (knowledge distillation) on many quantitative metrics such as AUC-ROC on classification tasks, Area under the risk coverage curve (AURC), etc. The authors finally summarize their findings on what architectures are best at providing uncertainty estimations and how to compare different methods on OoD detection and uncertainty in in-distribution samples.\n", "summary_of_the_review": "Overall the paper provides a thorough comparison of different networks on their uncertainty quantification. The paper largely did not consider the vast literature on improving uncertainty quantification of any given architecture. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N0MLVmjhUTb", "reviewer": "Reviewer_4Gca", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n- The paper focuses on uncertainty estimation in deep learning, which is an important research area that will enable risk aware and safe deployment of models.\n- The paper conducts extensive experiments with 484 models to gather insights into training schemes and model architectures for capturing uncertainty.\n\nNegatives:\n- I overall disagree with the classification of uncertainty used in the paper. Uncertainty due to class-out-of-distribution should be grouped into epistemic uncertainty, which by definition describes the uncertainty stemming from lack of knowledge.\n- Authors claim that ViT is a superior architecture for uncertainty quantification. I think this claim would have benefited by the distinction of aleatory and epistemic uncertainty. Aleatory uncertainty is related to the inherent randomness in data, and modeling choices do not affect it.\n- Most of the analysis in the paper uses confidence score function \\kappa to be the softmax values. This confidence score function will capture aleatory uncertainty but will not capture epistemic uncertainty. This distinction is important since there are evaluations performed with out of distribution data, which by definition stems from epistemic uncertainty. I believe the evaluations need to be consistent with these definitions.\n- Distillation is observed to yield better uncertainty estimates. I think this result was not surprising given knowledge distillation's connections to ensembling, which is a state-of-art method for uncertainty quantification that captures epistemic uncertainty. I recommend adding a discussion on this point.\n- Evaluations would benefit from reporting log likelihood numbers which is a common metric capturing the probabilistic distribution.\n\nMinor comments:\n- Typo in page 5. Use capital B in \"both\": \"... tied values. both metrics are also closely ..\"\n- Figure 4. For consistency, please report MC Dropout values here as well. \n- Figure 6. It is interesting to comment on the slope of the curves, indicating how quickly the performance degrades with respect to severity levels.", "summary_of_the_paper": "The paper presents an empirical study and focuses on evaluation of uncertainty with respect to different model architectures and training schemes. Using metrics capturing calibration and accuracy, the authors conclude that distillation based schemes are efficient in uncertainty estimation and ViT performs the best uncertainty estimates among the architectures considered.", "summary_of_the_review": "Overall I recommend a rejection of the paper. I think the definitions of sources of uncertainty are important in terms of what we are able to quantify and those definitions need to align with the evaluations, and I believe there are some mismatches in this paper. I think this distinction will immensely help the empirical evaluations performed in this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3ty5UU2YUR", "reviewer": "Reviewer_bFby", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengthes:\n- This work has extensive numerical studies on the metrics of uncertainty estimation performance and has drawn some observations. \n\nWeaknesses:\n- It is somewhat unclear to justify the usage of selective model for the evaluations of the uncertainty. The original work for selective model was using it when the task has the option of 'rejection' instead of performing 'prediction', but it is not clear why it is related to the measurement of uncertainty.\n- There is no new method, no new metric to be proposed in this work. In other words, the contribution of this work may be purely empirical. \n- There are a number of concerns on the experiments in terms of fairness: 1) it is unclear what Figure 2 means (see the first point in the weakness). 2) There are so many factors to determine the performance of a network on ImageNet that were not considered in this work such as optimization related parameters and practices, random initialization leading to different performance, possible differences in loss functions used and so on. However, a number of observations seem to assume that these are similar among all 484 trained models.\n- It is very hard to grasp the main theme of this work as a conference paper. Many observations were lined up, but what are the main contributions? It is not really surprising to see ViT performing better than EfficientNet-V2.\n\nComments:\n- Some of the important contents are in the Appendices. Unfortunately, in this manuscript, missing some of them in the main text seems to make this work to be very hard to understand. I strongly recommend to revise the manuscript to be accessible to more readers by writing key ideas and contents in the main body.\n- There are too many contents in the main body and it is very hard to grasp all of them. It will be great if the authors can write down clear contributions of this work in the beginning and some key conclusions / take-home messages of this work as remarks.\n- In selective model, the definition of coverage does not contain f while the symbol contains it (phi(f, g)). In addition, how did the authors implement g in experiments? It seems that g is important to define in details. \n- It is clear how MC dropout measures the uncertainty, but it is not clear how selective model does. Please elaborate more on this.\n- In Page 6, it says \"While TS is usually beneficial, it could harm some models (see Figures 3 and 4).\" How can we argue it since ECE is also measured empirically. In other words, the model could be ok while the measured ECE could contain some measurement errors.\n- In Page 9, how can we conclude that \"providing reliable uncertainty estimation\"? In other words, how one can measure reliability of uncertainty estimation and how were they measured in this work. It is not easy to see the supporting data / experiments for this sentence.\n\n", "summary_of_the_paper": "This work presents a number of empirical comparisons for the metrics on uncertainty estimation performance such as AUROC, ECE with 484 ImageNet classification models and has drawn some observations such as the superior properties of distillation over vanilla training, pre-training and adversarial learning, the superiority if ViT over other networks, and so on. ", "summary_of_the_review": "Some of the observations are interesting from extensive experiments, but there are a number of concerns on the metric to be used (e.g., selective model for uncertainty). There is no new method presented in this work, which makes this contribution somewhat weak for top conferences like ICLR (but could be an interesting workshop paper). The conclusions are not really surprising or are not easy to believe since there are other factors that were not considered in this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: As an empirical paper, this paper studies uncertainty estimations with respect to various architectures and learning schemes. Three reviewers suggested acceptance based on the strength of the paper (fairly extensive experiments were conducted, and some new observations were discovered, such as the superiority of ViT). On the other hand, two reviewers proposed rejection due to lack of rigor in writing and lack of novelty. No consensus was reached through additional discussion. In particular, the reviewer's point that the experiment was not well controlled-different models were trained with different hyperparameters etc- seems quite important, and it weakens the significance of the contribution of the paper. 

All reviewers agreed that it is a potentially interesting and important paper. I encourage the authors to resubmit in the future after carefully addressing the reviewers' concerns.
Decision: Reject