A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification
Authors: Paul F Jaeger
Carsten Tim Lüth
Lukas Klein
Till J. Bungert
Venue: ICLR 2023 notable top 5% (2023)
Keywords: failure detection
out-of-distribution detection
predictive uncertainty quantification
selective classification
robustness
method evaluation

Abstract: Reliable application of machine learning-based decision systems in the wild is one of the major challenges currently investigated by the field. A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores. This confidence may be obtained by either quantifying the model's predictive uncertainty, learning explicit scoring functions, or assessing whether the input is in line with the training distribution. Curiously, while these approaches all state to address the same eventual goal of detecting failures of a classifier upon real-world application, they currently constitute largely separated research fields with individual evaluation protocols, which either exclude a substantial part of relevant methods or ignore large parts of relevant failure sources. In this work, we systematically reveal current pitfalls caused by these inconsistencies and derive requirements for a holistic and realistic evaluation of failure detection. To demonstrate the relevance of this unified perspective, we present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions w.r.t all relevant methods and failure sources. The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation in the plethora of publicized research on confidence scoring. Code and trained models are at https://github.com/https://github.com/IML-DKFZ/fd-shifts

A CALL TO REFLECT ON EVALUATION PRACTICES FOR FAILURE DETECTION IN IMAGE CLASSIFICATION
Paul F. Jaeegr'? Carsten T. Lüth Lukas Klein & Till Bungert
'Interactive Machine Learning Group, German Cancer Research Center, Heidelberg, Germany Heelmholtz Imaging, DKFZ, Heidelberg, Germany
IInstitute for Machine Learning, ETH Zürich, Zürich, Switzerland
P. p.jaege reckerrkkkoo fde de lberg.errd..
ABSTRACT
Reliable application of machine learning- decision systems in the wild is one of the major challenges currently investigated by the field. A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores. This confidence may be obtained by either quantifying the model's predictive uncertainty, learning explicit scoring functions, or assessing whether the input is in line with the training distribution. Curiously, while these approaches all state to address the same eventual goal of detecting failures of classifier upon real-world application, they currently constitute largely sepa- rated research fields with individual evaluation protocols, which either exclude substantial part of relevant methods ignore large parts of relevant failure sources. In this work, we systematically reveal current pitfalls caused by these inconsistencies and derive requirements for holistic and realistic evaluation of failure detection. To demonstrate the relevance of this unified perspective, we present large-scale empirical study for the first time enabling benchmarking confidence scoring functions w.r.t. all relevant methods and failure sources. The revelation of simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation in the abundance of publicized research on confidence scoring. Code and trained models are https: /tt/://gtttt com IML DKFZ/EE Ed- -tsiiffts
INTRODUCTION
"Neural network- -ased classifiers may silently fail when the test data distribution differs from the training data. For critical tasks such as medical diagnosis or autonomous driving, it is thus essential to detect incorrect predictions based on an indication of whether the classifier is likely to fail".
Such similar mission statements prelude numerous publications in the fields of misclassification
detection (MisD) (Corbière et al. 2019 Hendrycks and Gimpel 2017) Malinin and Gales) 2018).
detection (OoD-D) (Fort et al.) [021;; Winkens al.) 2020 Lee et al. 2018
Hendrycks and Gimpel, 2017 | DeVries and Taylor] 2018 Liang et al. 2018). selective classification
(SC) (Liu | [2199] Geifman and El- Yaniv 2019; 2017). and predictive uncertainty quantification
(PUQ) (Ovadia et al. 2019; Kendall and Gal 2017), hinting at the fact that all these approaches aim towards the same eventual goal: Enabling safe deployment of classification systems by means of failure detection, i.c. the detection or filtering of erroncous predictions based on ranking of associated confidence scores. In this context, any function whose continuous output aims to separate classifier's ss failures from correct predictions can be interpreted as confidence scoring function (CSF) and represents valid approach to the stated goal. This holistic perspective on failure detection reveals extensive shortcomings in current evaluation protocols, which constitute major bottlenecks in progress toward the goal of making classifiers suitable for application in real-world scenarios. Our work is appeal to corresponding communities to reflect current practices and provides technical deduction of a unified evaluation protocol, a list of empirical insights based on a large-scale study, as well as hands-on recommendations for researchers catalyze progress in the field.

---

Relevant classification failure sources:
"li.d" Misclasolliations
(e.g. comuption shifl, subpopuiation shift, domann shift
Covariate shifts
New-class shifts
Task 1: Prevent Failures
by definition
Eoslation subject: Classiter (Robustrese")
(eg. WILDS, BREEDS, MetaShifts, ImageNett C, Deapmind DSF)
Task 2: Detect (Remaining) Fallures
MisD
Evalustion subject:
8C
Confidence Scoring Function (CSF)
0o0-0
Considered evaluztion
Proposed
Unity failure detection banchmark relevant solutions relevant shifts
Not considarod
(incl. MisO, SC, PUQ, OoD-D)
Figure Holistic perspective on failure detection. Detecting failures should be seen in the context of the overarching goal of preventing silent failures of a classifier, which includes two tasks: preventing failures in the first place as measured by the "robustness" of a classifier (Task I), and detecting the non-prevented failures by means of CSFs (Task 2, focus of this work). For failure prevention across distribution shifts, a consistent task formulation exists (featuring accuracy as the primary evaluation metric) and various benchmarks have been released covering a large variety of realistic shifts (e.g. image corruption shifts, sub-class shifts, or domain shifts). In contrast, progress in the subsequent task of detecting the non-prevented failures by means of CSFs is currently obstructed by three pitfalls: 1) A diverse and inconsistent set of evaluation protocols for CSFs exists (MisD, SC, PUQ. OoD-D) impeding comprehensive competition. 2) Only fraction of the spectrum of realistic distribution shifts and thus potential failure sources is covered diminishing the practical relevance of evaluation. 3) The task formulation in OoD-D fundamentally deviates from the stated purpose of detecting classification failures. Overall, the holistic perspective failure detection reveals an obvious need for a unified and comprehensive evaluation protocol, in analogy to current robustness benchmarks, to make classifiers fit for safety-critical upplications. Abbreviations: CSF: Confidence Scoring Function, OoD-D: Out-of- of-----iiibutiinn Detection, MisD: Misclassification Detection, PUQ: Predictive Uncertainty Quantification, SC: Selective Classification.
PITFALLS OF CURRENT EVALUATION PRACTICES
Figure gives overview of the current state of failure detection research and its relationship to the preceding failure prevention task, which is measured by classifier robustness. This perspective reveals three main pitfalls, from which derive three requirements RI-R3 for comprehensive and realistic evaluation in failure detection:
Pitfall 1: Heterogeneous and inconsistent task definitions. To achieve meaningful evaluation, all relevant solutions toward the stated goal must be part of the competition. In research on failure detection, currently, four separate fields exist each evaluating proposed methods with their individual metrics and baselines. Incomplete competition is first and foremost an issue of historically evolved delimitations between research fields, which go so far that employed metrics are by design restricted to certain methods. MisD: Evaluation in Misl) (see Section B for formal task definition) exclusively measures discrimination
classifier's success versus failure cases by means of ranking metrics such as AUROC (Hendrycks and Gimpel 2017 Jiang et al. 2018 Corbière al. 2019 Bernhardt et al. 2022). This protocol excludes a substantial part of relevant CSFs from comparison, because any CSF that affects the underlying classifier (e.g. by introducing dropout alternative loss functions) alters the set of classifier failures, i.c. ground truth labels, and thus creates their individual test set (for visualization of this pitfall Figure 4). As example, CSF that negatively affeets the accuracy of a classifier might add easy-to-detect failures to its test set and benefit in the form of high AUROC; scores. As depicted in Figure T argue that the task of detecting failures is self-purpose, but preventing and detecting failures are two sides of the same coin when striving to avoid silent classification failures. Thus, CSFs should be evaluated part of symbiotic system with the associated classifier. While additionally reporting classifier accuracy associated with each CSF
where the "f" denotes "failure" (see Appendix B for details), not to be confused with the classification AUROC.
2

---

renders these effects transparent, it requires nontrivial weighting of the two metries when aiming to rank CSFs based on a single score. PUQ: Research in PUQ often remains vague about the concrete application of extracted uncertainties stating the purpose of "meaningful confidence values" (Ovadia et al. 2019; Lakshminarayanan et al., 2017) (see Appendix [B223] for a formal task definition), which conflates the related but independent use-cases of failure detection and confidence calibration. This (arguably vague) goal is reflected in the evaluation, where typically strictly proper scoring rules (Gneiting and Raftery 2007) such as negative log-likelihood assess combination of ranking and calibration of scores. However, for failure detection use cases, arguably an explicit assessment of failure detection performance is desired (see Appendix[[ffor a discussion on how calibration relates to failure detection). Furthermore, these metrics are specifically tailored towards probabilistic predictive outputs such as softmax classifiers and exclude all other CSFs from comparison.
Requirement I (RI): Comprehensive evaluation requires single standardized score that applies to arbitrary CSFs while taking into account their effects on the classifter.
Ignoring the major part of relevant failure sources. As stated in the introductory quote, research on failure detection typically expects classification failures to occur when inputs upon application differ from the training data distribution. As shown in Figure [] we distinguish "covariate shifts" Llabe--presevvin shifts) versus "new-class shifts" Olabel-alterngg detailed formulation of different failure sources, see Appendix [A] The fact that in the related task of preventing failures a myriad of nuanced covariate shifts have been released on various data sets and domains (Koh et al., [2021] Santurkar al. [02T] Henddyykk and Dietterich| 2019: Liang and Zou 2022 Wiles et al. (002) to catalyze real-world progress of classifier robustness begs the question: If simulating realistic classification failures is such a delicate and extensive effort, why are there no analogous benchmarking efforts in the research on detecting failures? In contrast, CSFs are currently almost exclusively evaluated on i.i.d. test sets (MisD, PUQ, SC). Exceptions (see hatched areas in Figure [I) are PUQ study that features corruption shifts (Ovadia et al.. 2019). or SC evaluated on sub-class shift (comparing a fixed CSF under varying classifiers) (Tran et al. 2022) and applied to question answering under domain shift (Kamath et al. 2020). Further, research in OoD-D (see Section [B2.2 for a formal task definition) exclusively evaluates methods under one limited fraction of failure sources: new-class shifts (see images 7 and 8 in Figure 2 (right panel)). A recent trend in this area is to focus on "near OoD" scenarios, i.e. shifts affecting semantic image features but leaving the context unchanged (Winkens et al.) 2020 Fort et al., [02 Ren et al. (02T). While the notion that nuanced shifts might bear more practical relevance compared to vast context switches seems reasonable, the term "near" is misleading, it ignores the whole spectrum of even "nearer" and thus potentially more relevant covariate shifts, which OoD-D methods are not tested against. We argue that for most applications. it is not realistic to exclusively assume classification failures from label-altering shifts and no failures caused by label- bebee-preservin shifts.
Pitfall 2:
shifts). For a
Requirement (R2): Analogously to robustness benchmarks, progress in failure detection requires to evaluate on nuanced and diverse set of failure sources.
Pitfall 3: Discrepancy between the stated purpose and evaluation. The described limitations of
OoD-D evaluation are only symptoms of a deeper rooted problem: Methods are not tested to predict failures of classifier, but instead to predict an external, i.e. classífier-agnosti "outlier" label. In some cases, this formulation reflects the inherent nature of the given problem, such as in anomaly detection, where no underlying task is defined and the data sets are potentially unlabeled (Ruff et al.) 2021). However, the majority of work on OoD-detection comes with defined classification task, including training labels and states detecting failures of the classifier as their main purpose (Fort al. 2021 Winkens et al. 2020 Lee et al.; 2018 Hendryckk and Gimpel 2017 DeVrsss and Taylor 2018 Liang et 2018). Yet, this line of work falls short of justifying why associated methods subsequently not shown to detect the said failures but are instead tested on the surrogate task of detecting distribution shifts in the data. Figure [2 shows that the outlier label constitutes a poor tool to define which cases we wish to filter because the question "what is an outlier?" highly subjective for covariate shifts (see purple question marks). The ambiguity of the label extends to the concept of "inliers" (what extent of data variation is still considered i.i.d??), which the protocol rewards to retain irrespective of whether they cause the classifier to fail (see purple lightning).
+ Requirement (R3): If there is a defined classifier whose incorrect predictions are to be detected, its respective failure information should be used to assess CSFs w.e. the stated purpose instead of surrogate task such as distribution shift detection.
3

---

Fallrrss
Purpese: Dotest
Pred
Cornet
Covertata
New Class thiths
Lid Tost Caass
Image Compptions
diaso Brin
"Noor Oup "Sa Oup
Madias- Fration
aia
680
Falure Labet
thet
Success foop
Evaluation: Detoot
Outler
Correct Prediction
/or Trshh
Ace/
Ape
Dverr/ Ape
Ape
Bear
Apw/X
Fallure Label
Keep
Filter
Keep
Filter
Keep
Filter
+ Filter
Filter
New-cles S1
Outlier
Outiet Label:
Covarizz
Labal
Amtiquuas ourtlertaee label
Outber bbsi charpen from kr detection taak
unioen class
Figure 2: Left: The discrepancy between the commonly stated purpose and evaluation in OoD- Detection. The stated eventual purpose of detecting incorrect predictions of classifier is represented by the binary "failure label" and its associated event space (top). However, in practice this goal is merely approximated by instead evaluating the detection of distribution shift, i.c. separating cases according to a binary "outlier label" irrespective of the classifier's 's correctness (bottom). Right: Ex- emplary failure detection study under different types of failure sources. A hypothetical classifier trained to distinguish "ape" from "bear" is evaluated on 8 images under a whole range of relevant dis- tribution shifts: For instance, images 5 and depict apes, but these were not among the breeds in the training data and thus constitute sub-class shifts. Images 7 and 8 depict entirely unseen categories, but while "meerkat" stays within the task context ("semantic", "near OoD"), "house number" represents a vast context switch (" "noonssmmatti", "far OoD"). See Appendix A for a detailed formulation of shifts,
3 UNIFIED TASK FORMULATION
Parsing the quoted purpose statement at the start of Section Tresults in the following task formulation:
Given data set {(z,, Sel,s) 1 of size N with (r;, Met.a) independent samples from X' x y and yet the class label, and given a pair of functions (m, g). where g X' - IR is a CSF and m(: w) : X - y is the classifier including model parameters w, the classification output after failure detection is defined as:
(x) P (Vall,, w), if g( T
(1)
filterr,
otherwise.
Filtering ("detection") ") is triggered when g1x) falls below threshold r. In order to perform meaningful failure detection, CSF g(r)) is required to output high confidence scores for correct predictions and low confidence scores for incorrect predictions based on the binary failure label
yr (r(z, w, y) = 11(yet + fm(2z (z, w)),
(2)
where im = argmax.cy Pm = c|r, w) and is the identity function (1 for true events and 0 for false events).
Despite accurately formalizing the stated purpose of numerous methods from MisD, OoD-D, SC, and PUQ, and allowing for evaluation of arbitrary CSFs g(z). this generic task formulation is currently only stated in SC research (see Appendix B for a detailed technical description of all protocols considered in this work). To deduct appropriate evaluation metric for the formulated task, start with the ranking requirement on g(r), which is assessed e.g. by AUROC; in MisD leading to Pitfalls described in Section 2 Following R/ and modifying AUROC, to take classifier performance into account lets us naturally converge (see Appendix B.2.5 for the technical process) to a metric that has previously been proposed in SC as a byproduct, but is not widely employed for evaluation (Geifman et al. 2019): the Area under the-Risk Meae-Risk-ovvei Siisk-Coverage Coverage-Curve (AURC, see Equation 3T). We propose to use AURC as the primary metric for all methods with the stated purpose of failure detection, as fulfills all three requirements R/-R3 in single score. The inherently joint assessment of classifier accuracy and CSF ranking performance comes with a meaningful weighting between the two aspects, eliminating the need for manual (and potentially arbitrary) score aggregation. AURC measures the risk or eиror rate (1 - Accuracy) on the non- non-filtered cases averaged over all filtering thresholds (score range: [0.1], lower is better) and can be interpreted as directly assessing the rate of silent failures occurring in a classifier. While this metric enables a general evaluation of CSFs, depending on the case, more specific assessment of certain coverage regions (i.e. the ratio of remaining cases
4

---

after filtering) single covercee working points might be appropriate. In Appendix Fiee provide an open source implementation of AURC fixing several shortcomings of previous versions.
3.1 REQUIRED MODIFICATIONS FOR CURRENT PROTOCOLS
The general modifications necessary to shift from current protocols comprehensive and realistic evaluation of failure detection, i.e. to fulfill requirements RI-R3, are straightforward for considered fields (SC, MisD, PUQ. OoD-D): Researchers may simply consider reporting performance in terms of AURC and benchmark proposed methods against relevant baselines from all previously separated fields as well as on a realistic variety of failure sources (i.e. distribution shifts).
An additional aspect needs to be considered for SC, where the task is to solve failure prevention and failure detection at the same time (see Task 1 and Task 2 in Figure I]. i.e. the goal is to minimize absolute AURC scores. This setting includes studies that compare different classifiers while fixing the CSF (Tran et al.) 2022). On the contrary, the evaluation of failure detection implies a focus on the performance of CSFs (Task 2 in Figure while monitoring the classifier performance as a requirement (R/) to ensure a fair comparison of arbitrary CSFs. This shift of focus is reflected in the fact that the classifier architecture, as well as training procedure, are be fixed (with some exceptions as described in Appendix E..4) across all compared CSFs. This way, external variations in classifier configuration are removed as a nuisance factor from CSF evaluation and the direct effect of the CSF on the classifier training is isolated to enable relative comparison of AURC scores.
For evaluation of new-class shifts (as currently performed OoD-D), further modification is required: The current OoD-D protocol rewards CSFs for not detecting inlier misclassifications (see Figure 2). On the other hand, penalizing CSFs for not detecting these cases (as handled by AURC) would dilute the desired evaluation focus on new-class shifts. Thus, we propose to remove inlier misclassification from evaluation when reporting a CSF's performance under new-class shift. Figure 5 visualizes the proposed modification. Notably, the proposed protocol does still consider the CSF's effect on classifier performance (i.e. does not break with R/), since higher classifier accuracy will remain cause higher AURC scores (see Equations 29 31.
3.2 OWN CONTRIBUTIONS IN THE PRESENCE SELECTIVE CLASSIFICATION
Given the fact that the task definition in Equation I as well as AURC, the metric advocated for in this paper, have been formulated before in SC literature (see Appendix [B.2.4 for technical details on current evaluation in SC)), it is important to highlight that the relevance of our work is not limited to advancing research in SC, but, next to the shift of focus on CSFs described in Section 3.1 we articulate a call to other communities (MisD, OoD-D, PUQ) to reflect current practices. In other words, the relevance of our work derives from providing evidence for the necessity of the SC protocol in previously separated research fields and from extending their scope of evaluation (including the current scope of SC) w.r.t. compared methods and considered failure sources.
4 EMPIRICAL STUDY
To demonstrate the relevance of a holistic perspective on failure detection, we performed a large-scale empirical study, which we refer as FD-shifts [ For the first time, state-of-the-ar CSFs from MisD, OoD-D, PUQ, and SC are benchmarked against each other. And for the first time, analogously to recent robustness studies, CSFs are evaluated on a nuanced variety of distribution shifts to cover the entire spectrum of failure sources.
4.1 UTILIZED DATA SETS
Appendix features details of all used data sets, and Appendix A describes the considered distribution shifts. FD-Shift benchmarks CSFs on CAMELYON-I7-Wiid (Koh et al. 2021), iWildCam-2020 Wilds (Koh et al. (02T), and FREEDS-ENTITE been proposed to evaluate classifier robustness (Task 1 in Figure 1} under sub-class shift in various do-
13 (Santurkar et al., 2021). which have originally
mains. Further sub-class shifts are considered in the form of super-classes of CIFAR- 100 (Krizhevsky)
*odde httpps://gg thub. com/ wttp.////......//////////////
fts
5

---

2009 where one random class per super-class is held out during training. For studying corruption shifts, we report results on the 15 corruption types and intensity levels proposed by Heendrycks and Dietterich 2019) based on CIFAR- 10 as well as SIFAR-100. Regarding new-class shifts, we test on SVHN (Netzer et al. 2011). CIFAR- 101000 and Tinylmagenet (Le and Yang) 2015) in a rotating fashion while considering the shift between CIFAR data sets as semantic and other shifts as non-semantic. Finally, we create additional semantic new-class shift scenarios by testing on held-out training classes (randomly sampled 40% of all training classes) on SVHN and iWildCam-20200 Widd..
4.2 COMPARED METHODS
We compare the following CSFs: The maximum softmax response (MSR) calculated from the classifier's 's softmax output. PUQ: Predictive entropy based on softmax output (PE) and three predictive uncertainty measures based on Monte Carlo Dropout (MCD) (Gal and Ghahramani 2016): mean softmax (MCD-MSR), predictive entropy (MCD-PE), and expected entropy (MCD-EE) (for technical formulations see Appendix D). For MCD we take 50 samples at test time. MísD: We include ConfidNet (Corbière et al., 2019), which is trained as an extension the classifier and uses its regressed true class probability as a CSF. SC: We include DeepGamblers (DG), which uses loss attenuation based on portfolio theory learn confidence-like reservation score (DG-RES) (Liu et al., 2019). As the loss attenuation of DG's training paradigm might have a positive effect on the classifier itself, we additionally evaluate the softmax output (DG-MCD-MSR), OoD-D: We include the work of Vriies and Taylor (2018) 3) Notably, ConfidNet, DG, and the work by Devries et al. are excellent examples of artificial separation of previous evaluations, as all three have not been compared before, despite their strong conceptual and technical similarities. We evaluate Maximum Logit Scores (MLS) as proposed by (Vaze et al., (022) for semantic new-class shifts who argue that the softmax operation cancels out feature magnitudes relevant for OoD-D (we also add MLS scores averaged over MCD samples to the benchmark: MCD-MLS). Finally, we include the recently reported sfate-of-the-ard approach: Mahalanobis Distance (MAHA) measured
representations of a Vision Transformer (ViT) that has been pretrained on ImageNet (Fort et al. 2021). Classifiers: Because this change in the classifier biases the comparison of CSFs, we additionally report the results for selected CSFs when trained in conjunction with a ViT classifier. For implementation and training details, see Appendix E Since drawing conclusions from re- re-implemented baselines has to be taken with care, we report reproducibility results for all baselines including justifications for all hyperparameter deviations from the original configurations in Appendix T
4.3 RESULTS
The broad scope of this work reflects in the type of empirical observations we make: We view the holistic task protocol as an enabler for future research, thus we showcase the variety of research questions and topics that are now unlocked rather than providing an in-depth analysis on a single observation. Appendix G.I features a discussion on how this study empirically confirms R1-R3 stated in Section 2
Table shows the results of the FD-Shifts benchmark measured AURC scores. The repro- ducibility study in Appendix | confirms that none of the observed effects are caused by faulty re-implementations.
None of the evaluated methods from literature beats the simple Maximum Softmax Response baseline across a realistic range of failure sources. For both classifiers (CNN and ViT) the softmax baselines (either MSR or MCD-MSR) show the best close the best performance on all i.i.d. test sets and all shifts except new class shifts [ This is surprising given the claims of literature in MisD, SC, and OoD-D: All three tested methods based on the CNN-classifee (DG-Res, Devries, and ConfidNet) fail to generalize beyond the scenarios they have been proposed on, i.c. to more complex data sets (like iWildCam or BREEDS) and covariate distribution shifts (corruptions and sub-class
We aimed to further include OpenHybrid (Zhag al. 2020), but were not able reproduce their results despite running the original code.
NNotably, the loss attenuation of DG seems to have positive effects on the softmax for i.i.d. settings leading DG-MCD-MSR being the top-performing i.i.d. method with the CNN classifier on 3 out of data sets.

---

Table 1: FD-Shifts henchmark results measured as AURC * *10 (score range:[O, 1000], lower is better 4). The color heatmap is normalized per column and classifier (separately for CNN and ViT), while whiter colors depict better scores. "cor" is the average over 5 intensity levels of image corruption shifts. AURC scores are averaged over runs on all data sets with exceptions for the CNN: 10 rums on CAMELYON-I7-WiLE second version of this table featuring CSF rankings is provided as Table IT Abbreviations: nes: new-class shift (s for semantic, ns for non-semantic), iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR- ti: TinyImagene
(due to high volatility in results) and 2 runs on BREEDS.
/WildCam Idd
BRERDS
CAMELYON
CHIRR-100
CIFAR- 10
SVHN
soidy
sub
sincs
1id
sub
iid
sub
lid
sub
20T
ICC
155 DCS
lid
CDE
5 DCS
05 DCS
iid 5-DCS
D5 Dcs
rxtedaaa
c10
c100
wohn
<10 c100
MSR
CNN
62.3
69.5
212
8.39
175
10.1
145
70.2
270
289
312
533
321
5.62
93.0
226
429
195
4.85
1322
55.1
55.8
55.7
ML.8
UNN
90.2
67.8
740
11.
186
10.1
14%
87.66
215
312
51.8
564
270
6
6.16
94.0 0
221
421
191
5.09
1.11
52.3
53.0
$7.9
PE
CNN
62.7
69.9
21.5
8.21
174
10.3
145
77.0
234
308
519
567
249
5.59
92.
225
427
195
4.85
132
54.3
55.0
$4.9
MCD-MSR
CNN
52.6
65.0
180
7.79
175
6.93
151
67.0
217
260
311
546
318
5.25
61.4
226
435
211
4.75
164
54.1
54.9
$4.0
MCD MLS
CNN
$4.1
78.5
189
11.5
185
7.22
156
19.0
220
279
315
525
295
6.25
44.0
221
427
195
5.37
160
51.7
52.4
$2.1
MCD-99
CNN
53.5
67.1
175
173
6.93
151
71.4
217
512
81.3
414
4.87
163
52.8
57.7
MCD-EB
CNN
56.1
67.6
175
7.94
175
6.94
151
72.5
218
214
513
564
916
5.49
80.6
22.
429
195
4.85
162
52.5
53.3
52.8
MCD-MI
CNN
71.0
216
176
76.9
221
513
502
5.71
88.2
453
212
54.5
$3.7
ConbdNer
CNN
214
176
132
72.7
232
521
285
5.32
427
55.4
55.8
MSR
CNN
54.8
72.7
224
167
273
66.3
216
311
327
82.3
230
443
209
54.4
$1.4
CNN
246
316
52.4
Dewicou
CNN
234
179
242
07.5
226
420
7.27
55.7
35.4
70.6
265
105
0.00
70.0
69.0
255
0.95
9.77
192
404
708
52.7
$2.0
MISS
111
68.4
271
1.20
21.5
86
206
193
401
705
90.2
55.8
70.5
266
105
0.09
70.0
68.6
0.95
9.75
192
404
201
4002
52.5
51.8
MSR
113
105
MLSS
143
173
123
0.10
24.1
17.3
190
20.4
49.9
MCD-PE
$2.0
MCD-EE
161
105
236
228
17.7
400
197
50.6
MCD-MI
161
108
93.3
244
234
200
52.5
$1.4
MAHA
183
360
159
95.1
15%
219
16.0
410
185
48.8
50.0
shifts). Even on the test data they have been proposed on, all three struggle to outperform simple baselines.
These findings indicate a pressing need to evaluate newly proposed CSFs for failure detection in a wide variety of data sets and distribution shifts in order to draw general methodological conclusions.
Prevalent OoD-D methods are only relevant in a narrow range of distribution shifts. The proposed evaluation protocol allows, for the first time, to study the relevance of the predominant OoD-D methods in a realistic range of distribution shifts. While for non-semantic new class shifts ("far OoD"), prevalent methods from OoD-D (MLS, MCD-MLS, MAHA) show the best performance across both classifiers, their superiority vanishes already on semantic new class shifts (only ViT-based MAHA on SVHN shows best performance). On the broad range of more nuanced (and arguable more realistic) covariate shifts, however, OoD-D methods are widely outperformed by softmax baselines.
This finding points out an interesting future research direction of developing CSFs that are able to detect failures across the entire range of distribution shifts.
AURC is able to resolve previous obscurities between classifier robustness and CSF performance.
The results of ConfidNet provide vivid example of the relevance of assessing classifier performance single score when evaluating CSFs. The original publication reports
and confidence ranking superior results on CIFAR-10 10 and CIFAR-100 compared to the MCD-MSR baseline as measured by the MisD metric AUROC, These results are confirmed in Table S but we observe a beneficial effect of MCD classifier training that leads to improved accuracy (see Table B). This poses the question: Which of the two methods (ConfidNet or MCD-MSR) will eventually lead to fewer silent failures of the classifier? One directly aids the classifier to produce fewer failures and the other seems better at detecting the existing ones (at least on its test set with potentially more easily preventable failures)? AURC naturally answers this question by expressing the two effects in single score that directly
7

---

Table 2: Rounding errors occurring during the softmax operation negatively affect the ranking performance of MSR. The table shows, for different floating point precisions, error rates at which rounding errors occur, affected ranking metrics AURC and AUROC;, as well as Accuracy for i.i.d. test set performance of MSR. The default setting of 32-bit precision leads to substantial ranking performance drops on the ViT classifier. Note that not all rounding errors necessarily decrease ranking performance, especially in high-accuracy settings where failure cases are rare.
Round-to-ond to-ome
AURC 100
AUROC,
200
Accuracy
ernor rate 100
10bit
32hir 64hit
16hit
32bit
64hir
10bit
32hir
64hit
WWidCam
47,30
1,802 0.000
82.22
69.20
69.00
85.80
87.50
87.50
76.01
BREEDS
35.25
4.268
0.003
18.81
12.89
12.84
89.89
92.22
92.22
90.72
CNN CAMELYON
10.12
89.21
93.99
CIFAR-100
22.75 2,264
0.001
87.22
77.43
77.39
86.38
87.29
87.29
73.26
CIFAR-10
41.54
1,455
0.000
8.346
5.620
5.617
91,98
93.73
93.73
94.35
SVHN
41.76
0.001
4,902
4.850
89.59
92.81
92.87
96.09
IWIMCam
44,41
0.000
177.8
75.97
80.38
BREEDS
80.19
52.59
0.423
11:43
4.559
1.893
72.65
88.88
91.35
97.92
CAMELYON
82.03
14:52
0.000
4.661
1.007
1.007
88.59
96.42
96.42
97.95
VIT
CIFAR-100
68.65
30.27
0.000
14.95
14.23
79.29
90.10 90.70
91.62
CIFAR- 10
92.16
$1.79
1.883
7.614
3,480
0.950
69.30
85.85 94.90
98.76
SVHN
69.02 47.17 0:305
16.94
8.757
5.475
68.75
83.55
88.14
relates to the overarching goal of preventing silent failures. This reveals that the MCD-MSR baseline is in fact superior to ConfidNet on the i.i.d. test sets of both CIFAR- 10 and CIFAR-100.
VIT outperforms the CNN classifier on most data sets. Figure 8 shows a comparative analysis between ViT and CNN classifier across several metrics. As for AURC, VIT outperforms CNN on all datasets except iWildCam, indicating that the domain gap of (mageNet-pretraines representations might be too large for this task. This is an interesting observation, given that CAMELYON featuring images from the biomedical domain could intuitively represent a larger domain gap. Further looking at Accuracy and AUROC; performance,
see that performance gains clearly stem from improved classifier accuracy but the CSF ranking performance is on par for ViT and CNN (although the failure detection task might be harder for ViT given fewer detectable failures compared to CNN).
Different types of uncertainty are empirically not distinguishable. Considering the associations made in the literature between uncertainty measures and specific types of uncertainty (see Appendix D}. we are interested in the extent to which such relations can be confirmed by empirical evidence from our experiments. As an example, we would expect mutual information (MCD-MI) to perform well on new class shifts where model uncertainty should be high and expected entropy (MCD-EE) to perform well on i.i.d. cases where inherent uncertainty in the data (seen during training) is considered the prevalent type of uncertainty. Although, as expected, MCD-EE performs generally better than MCD-MI on i.i.d. test sets, the reverse behavior can not be observed for distribution shifts. Therefore, no clear distinction can be made between aleatoric and epistemic uncertainty based on the expected benefits of the associated uncertainty measures. Furthermore, no general advantages of entropy-basee based uncertainty measures over the simple MCD-MSR baseline are observed.
CSF' beyond Maximum Softmax Response yield well-calibrated scores. We advocate for a clear purpose statement in research related to confidence scoring, which for most scenarios implies separation of the tasks of confidence calibration and confidence ranking (see Section 2). Nevertheless, to demonstrate the relevance of our holistic perspective, extend FD-Shift to assess the calibration error, a measure previously exclusively applied to softmax outputs, of all considered CSFs. Platt scaling is used to calibrate CSFs with natural output range beyond [o, 1] (Platt 1999). Calibration errors of CSFs are reported in Table indicating that currently neglected CSTs beyond MSR provide competitive calibration (e.g. MCD-PE on CNN MAHA on ViT) and thus constitute appropriate confidence scores to be interpreted directly by the user.
*Our CNN results are not representative for state-of-the-art CNN performance, since we employ small models such VGG-13 3 ResNet-50, see Appendix E.2
8

---

This observation points out a potential research direction, where, analogously to the quest for CSFs that outperform softmax baselines in confidence ranking, it might be possible to identify CSFs that yield better calibration compared to softmax outputs across wide range of distribution shifts.
The Maximum Softmax Response baseline is disadvantaged by numerical errors in the standard setting. Running inference of our empirical study yields terabytes of output data. When tempted to save disk space by storing logits as 16-bit precision floats instead of 32-bit precision, we found the confidence ranking performance of MSR baselines drop substantially (reduced AURC and AUROC; scores). This effect is caused by a numerical error, where high logit scores are rounded during the softmax operation. thereby losing the ranking information between rounded scores. Surprisingly. when returning to 32-bit precision, we found the rate at which rounding emrors occur to still be substantial, especially on the ViT classifier (which has higher accuracy and confidence scores compared to the CNN). Table shows error rates well affected metrics for different floating point precisions. Crucially, confudence ranking on VIT classifiers is still affected by rounding errors even in the default 32-bit precision setting (effects for the CNN are marginal as seen in AURC scores), see for instance AUROC, drops 9% on CIFAR-IO and 5.47% on BREEDS (i.e. ImageNet data). This finding has far-reaching implications affecting any ViT-based MSR baseline used for confidence ranking tasks (including current OoD-D literature).
We recommend either casting logits 64-bit precision (as performed for study) performing temperature scaling prior to the softmax operation in order to minimize the rounding errors.
Further results. Despite its relevance for application, the often final step of failure detection, i.e. the definition of decision threshold on the confidence score, is often neglected in research. In Appendix Dee present an approach that does not require the calibration of scores and analyze its reliability under distribution shift. In addition, Appendix features Accuracy and AUROC; results for all experiments. For a qualitative study of failure cases, see Appendix H
5 CONCLUSION & TAKE-AWAYS
This work does not propose novel method, metric, or data set. Instead, following the calls for more rigorous understanding of existing methods (Lipton and Steinhardi, 2018 Sculley al. 2018) and evaluation pitfalls (Reinke et al. 2021). its relevance comes from providing compelling theoretical and empirical evidence that a review of current evaluation practices necessary for all research aiming to detect classification failures. Our results demonstrate vividly that the need for reflection in this field outweighs the need for novelty: None of the prevalent methods proposed in the literature is able to outperform softmax baseline across a range of realistic failure sources.
Therefore, our take-home messages are:
Research on confidence scoring (including MisD, OoD-D, PUQ, SC) should come with a clearly defined use case and employ meaningful evaluation protocol that directly reflects this purpose.
2. If the stated purpose is detect failures of classifier, evaluation needs to take into account potential effects on the classifier performance. We recommend AURC as the primary metric, as combines the two aspects in a single score.
Analogously to failure prevention saaiiiionn""oolrr ("robustness"), evaluation of failure detection should include realistic and nuanced set of distribution shifts covering potential failure sources.
4. Comprehensive evaluation in failure detection requires comparing all relevant solutions towards the same goal including methods from previously separated fields.
5. The inconsistency of our results across data sets indicates the need to evaluate failure detection on variety of diverse data sets.
Logits should be cast 64-bit precision or temperature-scaled prior to the softmax operation for any aankingrrelated tasks to avoid subpar softmax baselines.
7. Calibration of confidence scoring functions beyond softmax outputs should be considered as an independent task.
Our open-source framework features implementations of baselines, metrics, and data sets that allow researchers to perform meaningful benchmarking of confidence scoring functions.

---

ACKNOWLEDGEMENTS
This work was funded by Helmholtz Imaging (HI), a platform of the Helmholtz Incubator on Infor- mation and Data Science. We thank David Zimmerer and Fabian Isensee for insightful discussions and feedback.
REFERENCES
Ahmed and A. Courville. Detecting Semantic Anomalies. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 3404)331544-1112 Apr. 2020. ISSN 2374-3468. doi: 10. 10.1609/ aaai.v34i04.5711 /
5712 Number: 04.
M. Bernhardt, F. D. S. Ribeiro, and B. Glocker. Failure detection in medical image classification: A reality check and benchmarking testhed. arXiv preprint arXiv:2205. /4094, 2022.
C. Chow. An optimum character recognition system using decision functions. IRE Trans. Electron. Comput., 1957.
C. Corbière, N. Thome, A. Bar-Hen, M. Cord, and P. Pérez. Addressing Failure Predic- tion by Learning Model Confidence. Advances in Neural Information Processing Sys- tems, 32, 2019. URL hatpps://proceeding: newreiiiiinnnnnnnnoooo.em.m....r neurips wwt.rrips.cccpppppeeee wwtpppppr///1//aaaat... 757f843al 005T15500000050000012111111-1a00
html
S. Depeweg. J. M. Hernándzz-Lobato, F. Doshi-Velez, and S. Udluft. Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and sensivvee Learning. arXiv: /7 1710.07283 [cs, stat], June 2018.
arXiv: 1710.07283.
T. Veiries and G. W. Taylor. Learning Confidence for Out- Distributtt Detection in Neural Networks. arXiv:1802. 152.046502 [cs, stat], Feb. 2018. URL http: 04865 arXiv: 1802.04865.
abs
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, Gelly, J. Uszkoreit, and N. Houlsby, An Image is Worth 16x16
Words: Transformers for Image Recognition at Scale. ICLR 2021, Oct. 2020. URL https
//arxiv. //axxiv.rg//bs
wttp:///.rrabbb///1100 awwaaa/b...0.... 11929v2
R. Elniv and Y. Wiener. On the Foundations of Noise-free Selective Classification. Jounral of Machine Learning Research, page 37, 2010.
Fort, Ren, and Lakshminarayanann Exploring the Limits of tion. Neurips 2021, July 2021. URL. http::/aaxxvv..ooo /20 2106. 03004] 2 2106.03004.
Detec- arXiv:
Y. Gal and Z. Ghahramani. Dropout as Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML, page 10, 2016.
Y. Geifman and R. El-Yaniv. Selective Classification for Deep Neural Networks. NIPS, 2017.
Y. Geifiaan and El-ya..v. SelectiveNet: Deep Neural Network with an Integrated Reject Option. ICML, page 9, 2019.
Y. Geifman, G. Uziel, and R. El-Yvn Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers. ICLR, 2019.
R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 (011):6656733 Nov. 2020. ISSN 2522-5839. com/
doi: 10. 00257-2
URL https: / www. nature Number: 11 Publisher: Nature Publishing Group.
T. Gneiting and A. E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477):359-377 Mar. 2007. ISSN 0162- 1459. doi: 10. 119// 0162145060000014377 URL
11tp://oooo......11/11111011.. Publisher: Taylor & Francis wtw.://w.ww....o..o///////////oo
10

---

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On Calibration of Modern Neural Networks. ICML, page 10, 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. arXiv:1512.03360 [cs], Dec. 2015. URL /tt.L//////axxx...ooo///////22.. 03388 arXiv: 1512.03385.
D. Hendrycks and T. Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR, 2019.
D. Hendrycks and K. Gimpel. A Baseline for Detecting Misclassified and Out-of-Dissribution Examples in Neural Networks. ICLR, 2017. /tt/:tttee///arvv..or//////////0 arXiv: 1610.02136.
H. Jiang, B. Kim, M. Guan, and M. Gupta. To Trust Or Not To Trust Classifier. Neurips 2018, page 12, 2018.
A. Kamath, Jia, and Liang. Selective Question Answering under Domain Shift. Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 5684-5696, Online, July 2020. Association for Computational Linguisties. doi: 503. URL
main.
wwtp:/oggo.oog//oo wwtp:///.///////// main.
A. Kendall and Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? NIPS, 2017.
W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga,
R. L. Phillips. I. Gao, T. Lee. E. David, I. Stavness, W. Guo, B. A. Earnshaw, I. S. Haque, S. Beery, Leskovec, A. Kundaje, E. Pierson, Levine, C. Finn, and P. Liang. WILDS:
A Benchmark of Wnithe-Wild Distribution Shifts. arXiv:2012.07421 [cs], Mar. 2021. URL http://arxix www.org///ao...
07421 arXiv: 2012.07421.
A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. page 60, 2009. URL https: www. cs. http::/wwww...tooooo.ee.eee////. aopronto.edd-krrizz.
html
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NIPS, 2017. arXiv: 1612.01474.
Y. Le and X. Yang. Tiny ImageNet Visual Recognition Challenge. CS (2015): 7., page 2015.
K. Lee, K. Lee, Lee, and Shin. Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. Neurips, 2018.
Liang, Y. Li, and Srikant. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. ICLR, 2018.
W. Liang and J. Zou. MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts. Technical report, Feb. 2022. URL http:tttpp/aaxxxorr//////// 06523 arXiv:2202.06523 [cs] type: article.
Z. C. Lipton and J. Steinhardt. Troubling Trends in Machine Learning Scholarship. ICML: The Debates, 2018. (tRt:ttp: httpp/////wwww..o////aaas/ass /1807. 03341 arXiv: 1807.03341.
Z. Liu, Z. Wang, P. P. Liang, R. R. Salakhutdinov, L.-P. Morency, and M. Ueda. Deep Gam- blers: Learning to Abstain with Portfolio Theory. Advances in Neural Information Pro- cessing Systems, 32, 2019. URL
nips cc cccppaper/219//aall Abs
I. Loshchilov and F. Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR, 2017.
A. Malinin and M. Gales, Predictive Uncertainty Estimation via Prior Networks. Neurips, page 12, 2018.
11

---

Y. Netzer, Т. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, page 9, 2011.
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. Neurips, 2019.
J. C. Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In Advances in Large Margin Classifiers, pages 61 61-74. MIT Press, 1999.
J. Cuidnero-Candela, M. Sugiyama, A. Schwaighofer, and N. URL /paper/
Lawrence.
Dataset Shift in Machine Learning.
Mit Press, 2009.
A. Reinke, M. Fisemmann, M. D. Tizabi, C. H. Sudre, T. Rädsch, M. Antonelli, T. Arbel, S. Bakas, J. Cardoso, V. Cheplygina, al. Common limitations of image processing metries: picture story. arXiv preprint arXiv:2104 arXiv:2104.06642, 05642,, 2021.
J. Ren, S. Fort, J. Liu, G. Roy, Padhy, and B. Lakshminarayanan. A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection. arXiv:2106.000 [cs], June 2021. hRLhttp?
06 09022
arXiv: 2106.09022.
L. Ruff. J. R. Kaufffman, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller A Unifying Review of Deep and Shallow Anomaly Detection. Proceedings of the IEEE, 109(5):756-799) May 2021. ISSN 1558-2256. doi: 10.1109/PRRC.2221.13554444 Conference Name: Proceedings of the IEEE.
S. Santurkar, D. Tsipras, and A. Madry. BREEDS: Benchmarks for Subpopulation Shift. ICLR, 2021.
D. Sculley, J. Snoek, A. Wiltschko, and A. Rahimi. Winner's Curse? On Pace, Progress, and Empirical Rigor. ICLR Workshop, 2018. URL httpps://penrrevlew.eeee forummif
EJJF
K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Scale Image Recogni- tion. 1409. [cs], Apr. 2015. 1409.1556.
1556 arXiv:
L. Smith and Y. Gal. Understanding Measures of Uncertainty for Adversarial Example Detection. UAI, 2018.
C. Tomani, S. Gruber, M. E. Erdem, D. Cremers, and F. Buettner. Post-hoc Uncertainty Calibration for Domain Drift Scenarios. CVPR 2021, June 2021. URL http::::aaa...../////aaaaa 10988 arXiv: 2012.10988.
D. Tran, J. Liu, M. W. Dusenberry, D. Phan, M. Collier, J. Ren, K. Han, Z. Wang, Z. Mariet, H. Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.07411, 2022.
S. Vaze, K. Han, A. Vedaldi, and A. Zisserman. Open-Set Recognition: a Good Closed- Classifier is All You Need? ICLR, Apr. 2022. htLLhttp://aaae wwg. abs/ aabs/211 22100. 06207
O. Wiles, S. Gowal, F. Stimberg, S.-A. Rebuffi, I. Ktena, and T. Cemgil. A FINE-GRAINED ANALYSIS ON DISTRIBUTION SHIFT. ICLR, page 15, 2022.
J. Winkens, R. Bunel, A. G. Roy. R. Stanforth, V. Natarajan, J. R. Ledsam, P. MacWilliams, P. Kohli, A. Karthikesalingam, S. Kohl, T. Cemgil, S. M. A. Eslami, and O. Ronneberger. Contrastive Training for Improved Out-of-Distributtion Detection. arXiv:2007.055: 055666 [cs, stat], July 2020. URL http:htt////iiii org/ abs/2007 05566)
arXiv: 2007.05566
H. Zhang, A. Li, J. Guo, and Y. Guo. Hybrid Models for Open Set Recognition. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision - ECCV 2020, volume 12348, pages 102-117. Springer International Publishing, Cham, 2020. ISBN 077--33000-8877-2000 978-3- 030-58580-8, doi: 10.10077978-3-030 58580-8 hRLpatts::/ //iink......///////////////// springer. com/ 10. 10.1007/978-20 030 -
Series Title: Lecture Notes in Computer Science.
12

---

A FORMULATION OF FAILURE SOURCES
In general, we distinguish three sources of error that can cause image classification systems to output false predictions (see Figure 2 for exemplary visualizations). Inlier Misclasiffccatios:: This type of failure source is defined by occurring cases that sampled i..d. with respect to the training distribution and is commonly addressed by work in MisD and SC. Possible reasons for occurrence are missing evidence in the image related the ground truth category, poor model fitting, or data variations that are not considered as distribution shifts yet in a specific use case. Covariate Shift: Images subject to a covariate shift (Quionero-Candela Candela al., 2009) can still be assigned to one of the training categories. Various examples are investigated by recent robustness benchmarks: Image corruptions shift (Hendryeks and Dietterich 2019), domain shift such medical images from different scanners and clinical sites or satellite images from different seasons (Koh et al. (021), or subpopulation shif( where unseen semantic variations of the training categories occur such as unseen breeds of animal category (Santurkar et al. (021). We summarize domain shift and subpopulation shift under the term sub-class shift. To the best our knowledge. these relevant sub-class shifts have not been studied in the context of confidence scoring before. New-class Shift: Images subject to new a class shift can not be assigned to any of the training categories. This type of failure is commonly addressed by work in Ool detection. We follow Ahmed and Courville 2020) in further distinguishing semantic (only foreground object is subject to semantic variations, e.g. previously unseen classes, a.k.a "near OoD" (Winkens al. 20200) and non-semantic new-class shifts (context changes, e.g. images from a new data set and classification task, a.k.a "far OoD").
B TASK FORMULATIONS ADDRESSING FAILURE DETECTION
This section provides a technical exposition of current evaluation protocols for all task formulations stating address the goal of preventing failures of related classifier by means of CSFs. Figure gives an overview of relevant metrics in this context.
B. CLASSIFICATION
To set the context we describe the standard evaluation protocol of a classification task, which is denoted in the notation with Yel for the class label, distinguishing it from the failure detection label yr. Given a data {<14, j=1 of size N with (zy, yel,x) independent samples from X' x V, and including the discrete class label Yet E y = {0, 1, 2, .... Telaases - 1}. On this data set a classifier m(: w) -> y maps from the input images x to the predicted labels ý with model parameters w. Im, is the classification decision obtained as the maximum class probability from the classification model's probability output vector Pm (yel xr w) E [0, 1]moasmm:::
Me1,s)
N
im = arg max (yal = w)x,w), eEY
with c E {0.,... - 1} for the respective class. According to this notation, arg max.ey Pm (y c/z,, w). The performance of such classifier is in multi-class setups often evaluated via accuracy,
1 NV N
Accuracy =
Σ
I(Jet, = Im,s):
(4)
where ((Eventt) is defined as indicator function:
I(Event)) = f1. if the event Event occurs
(5)
o, otherwise.
Another way is to evaluate classes separately by computing the binary label ye. E [0, 1] as
ye = [[ c).
(6)
'The term subpopulation shift has different meaning in Koh et al. 2021 where it describes variations of category frequencies as opposed unseen variations
13

---

Required for generic essassment
failure detection
Classifier Performance
A) Confidence Ranking
Confidence Callbration
Metric
Formulation
Class.
Accuracy
Class.
AUROC
Class.
AP
Class.
Sens//Prec...
MisD
AUROC
MisD
APf
OoD
AUROC Out
Risk Coverage
SC
e-AURC
(SC)
AURC
Calibration
ECE
PUQ
NLL
PUQ
Brier-Score
Considered
Not Considered
Does not evaluate CSF, but Predicted cless probabiittee
Figure Overview of Evaluation Metrics in the Context of Failure Detection. The three performance aspects related to confidence scoring are classification performance, confidence ranking, and confidence calibration. As argued in this work, the former two are required for evaluating CSFs in the context of failure detection, while confidence calibration usually constitutes separate task and use case (see Appendix C.. Appendix B provides a definition of all metrics as well as their relations and showcases how naturally converged to considering AURC as the primary metric for evaluation in failure detection.
Following this notation ye, = I(yet, = c) is the binary class label for sample i. Subsequently, the confusion matrix can be computed counting the four possible evaluation outcomes per case and class
TPa(0,c) Σ Ue.s I(PmiVet c/as, w) 2
N
FPa(0,c) Σ Ve.i) I[Pm(ue clz,, w) 2 8)
(8)
N
(Na(0.c) Σ (1 Ve,i) I(Pm c|z,, w) < 8)
(9)
CN, Σ ye,t I(P (yel c/2,, w) 0).
(10)
These cardinalities defined depending the cut-off the provided predicted class probabilities (PCP). Subsequently, various counting metrics can be applied, for instance Sensitivity (also called recall true positive rate). False Positive Rate (FPR, or Specificity), or Precision per class
(P,(f.c)
Sensitivity,.(e.c) TPa (0,
(11)
(P. c)
FP.c(00c) TNa(0, FP c0,c)
(12)
14

---

TP (0,,) Procisiona (Pa(0, + CP)((,c)
(13)
Next to evaluating these metrics on certain cut-off on PCPs, often employed, which scan over cut-off's from all PCP values present in the data set to obtain ROC-curves (Sensitivity plotted over FPR) or Precision- CecisionnRReaal-CCCe Recall-Curves (PRC, Precision plotted over Sensitivity). Model performance in the form of single score is extracted via computing the respective area under the curve, ie. the AUROC for ROC-curves. Here the multi-threshold list {#r of length T are the cut-off values obtained the unique values of the descending ranking of all CSF values. The class-wise AUROC values can then be computed as follows:
metrics are
T
AUROCa(C)
Σ (FPRa((0.)) c) - FP c))
(14)
(Sensitivity. c) + Sensitivitya((0==))
The AUPRC for PRC-curves is defined similarly, and commonly approximated by average precision (AP) score due to interpolation issues:
T
APa(c) Σ
(Sensitivityy.((e..)) Sensitivity,(((0-n))))) Precisiona c).
(15)
Both areas under the curves can be interpreted ranking metrics, i.e. they require cases of the class e to be separated from cases of other classes based on a ranking of PCP values.
B.2 FAILURE DETECTION
Concrete application of CSFs for the task of detecting failures in order to prevent incorrect predictions of classifier are formulated in Equation [ in Section 3 Based on the failure label in Equation 2 which is I for classification failure and for success, a confusion matrix can be determined for cutoff value follows:
N TP((T) = Σ vl,)) 1(g(xx)) 2 r)
(16)
FP((T) Σ ye. I[g(x;)) 2
(17)
TN((T) = Σ Ve. T(g(x;) <T)
(18)
N FN (1- I(g(x;) < <)).
(19)
Analogous the evaluation of the classification performance, one can compute different counting metries for the confidence ranking task:
TPr(T)
Sensitivity:(7)
(20)
+
FP((T) (Nr(T) + FP((T)
FPR((7)
(21)
TP((T)
Precisione(1 (T)
(22)
T FPr(T)
15

---

B.2.1 MISCLASSIFICATION DETECTION
This evaluation protocol for CSFs directly sticks to the binary classification task between correct and incorrect predictions defined above, and uses ranking metrics analogous the area under the curves defined for classification evaluation based on a multi-threshold list {Te fre) of length T, which are the cut-off values obtained as the unique values of ascending ranking of confidence scores. Most commonly used are the failure detection AUROC:
AUROC,
(FPRr(T)) FPR((((-11))) (Sens)iiiitiyi (Tr) (ansitivity((((==))))
N UL. (I(g(1)) Te) I(g(x)) 2 T(t-))))
(23)
Σx Vr,
1=1
Σ (1 vr.)) (I(g(2)) Tτ) H((()) 2 T(-1))
2 Σ N yr,)
and the failure detection AUPRC or AP-score:
AP, (e) Sensitivityy Precision((T))
Σn (1 vr.) (T(g(az) 2 Te) - T(g((;) T(-1)))) E vr,)) [(g((;)) 2 Tc) N Σ (1 yt,s)
Σ I(g(x;) 2 Ti)
(24)
The data set in these tasks is often biased towards many correct predictions and few incorrect predictions since one does not usually think about preventing failures if the classification performance very poor in the first place. Consequently, this biases the failure detection AP-score resulting in higher scores. The reverse form defining errors as positive samples, which is not biased in the aforementioned way, is also evaluated:
T Σn ve. (U(g(a;) < - 1(g(x)) < T(-1)))
Σ yess I(g(x;) < <n)) Σ I(g(a,) < T))
APr.rr
(25)
Σ Yr.i
t=1
As described in Section this protocol comes with the pitfall of not considering the classifier performance, thus potential effects CSFs on the classifier performance caused by joined training go unnoticed (for visualization of this pitfall Figure 4). This effect can be nailed down to the (1-yt,i factor of Equation 24 The precision score (second factor of the product) only gets a weight for correct predictions // U. so by perfect separation, it is possible traverse all correct predictions (all "weights") while the precision score is one, and any subsequent incorrect predictions are not considered by the metric. This behaviour prevents meaningful comparison of arbitrary CSFs in practice. For instance, consider a comparison between two very standard CSFS, the confidence scores derived from the maximum softmax response (MSR) of classifier against scores derived from the softmax mean over Monte Carlo Dropout (MCD) sampling of the same classifier. This comparison can already lead to considerable evaluation bias because the classification decision based on MCD means approach most likely produces different set of failure cases. What's 's more, associated metrics are notoriously sensitive to such label biases: While it is well-known, that high scores failure detection AUROC; can be achieved via bias towards true negatives (such as background objects in object detection). in failure detection APr,err with failure as the positive label is used (Hendrycks and Gimpel 2017; 2017) Corbière et al.) 2019), which can be equally hacked by biases towards pasy-to-dee easy-to-detect true positives. As a consequence, this metric typically gives the highest scores at the beginning of model training, when large amounts of failures produced by the classifier.
16

---

Ciassifeer Evaluation
Resultigg
Evclustion Marsring CSF sanking with tukan
Setup
falture lubels In CSP sonkastiont
hig actifiom dropoct
lunctont
Conccit:
X
X
X
X
0.1
0.3
1.0
0.1
CSFT
Predi:
Door Doar
Door
=1
Ape
Ape
Comeect:
X
X
Which CSF best preventing
CSF2
0.9
AIFROG - 0.75
0.1
0.3
1.0
0.1
silent fallures classifier?
Predi:
Ape Ade
Douar
Commoch
X
CSF3
0.9
0.1
0.3
1.0
0.1
CSFS
Predi:
AUTOOC = 0.583
Apu
Disut
Cunnutt
X
X
Predi:
Boao
Ape
AOB
Baor
Daar
Figure 4: Visualizing the pitfall of the MisD protocol for failure detection: This figure demon- strates the need behind Requirement 1 ("Comprehensive evaluation requires single standardized score that applies to arbitrary CSFs while accounting for their effects on the classifier"). Many CSFs affect the classifier training (e.g. CSFs based dropout additional loss functions resulting in distinctive classifiers per CSF. Evaluating the hypothetical task of binary image classification between apes and bears (see "task 1: Failure prevention" in Figure 1). joint training with CSF1 decreases the accuracy of the classifier (compared to training without CSF setting), CSF2 leads to altered predictions with equivalent accuracy, and CSF3 leaves the predictions
the classifier unchanged. CSF evaluation (see "task 2: Failure detection" in Figure m For simplicity,
Crucially, the results from this evaluation ("failure labels") will be used as reference labels for the
assume that all three CSFs output identical confidence scores. However, ranking metrics such as AUROC, (as applied in MisD) yield vastly different results across CSFs because the evaluation is based different sets of reference labels. In this case CSFI shows the best failure detection performance, but is also the one causing additional failures the first place. This shows that isolated evaluation of failure detection (as done in MisD) is flawed, because CSF effects on the classifier training not accounted for. To serve the overarching purpose preventing silent failures of a classifier ("taskl + task2" in Figure D. one would need select CSF based both, rankings of Accuracy and AUROC,- But what is a natural way to weigh the two rankings? The metric proposed in Section 3 directly reflects the purpose of preventing silent failures and can be interpreted providing natural weighting between Accuracy and A AUROC, rankings.
B.2.2 OUT-OF-DISTRIBUTION DETECTION
In out of distribution detection, the label yout signals whether sample outlier yout 1 inlier yout = 0. Based on this label the AUROC value is primarily used as a performance metric:
AUROCout
(FPRom (T.) FPR (T(+-)))) -Sens-ttiittyyynnnmn.... out (Tc) + Sensitivity (T)+2))))2
T Σ ))ut..) (I(g(2;) 2 TX) 1(9(xx)) T(-1))))
Σ yout,;)
Σ (Nout.s) (I(g((;)) Te) M(g(x;) 2 T(-1))
2 N yout,s
(26)
This formulation and the thresholds t, are equivalent to the AUROC used for MisD Equation 23 except for the different label yout that is employed (for implications and pitfalls of using this label
17

---

Due to the vast overlap of task formulations FD FD-Shifts enables seamless integration of OoD-dctection methods for holistic comparison (see Section 3).
Figure 2}.
B.2.3 PREDICTIVE UNCERTAINTY ESTIMATION
Common evaluation in PUQ (e.g. Ovadia al. (2019
Lakshminarayanan al. (2017) employs
proper scoring rules like Negative- Log-Likelihood
NLL
Σ log( (mn = yets w)),
(27)
and Brier Score
1 N
BrierScore
(yla = c/2;, w) ye,,)?
(28)
Both directly assess the classifier output and are not applicable to other CSFs. Each metric requires both ranking and calibration of confidence scores which is interpreted assessing the "general meaningfulness" of scores (see also Figure However, in case there is a clear use case defined requiring either ranking calibration of confidence scores, evaluation with proper scoring rules might dilute this focus and hinder the practical relevance of results.
B.2.4 SELECTIVE CLASSIFICATION
The idea of equipping a classifier with the option to abstain from certain decisions based on a selection function has been around for long time (Chow 1957 Yaniv and Wiener 2010). SC directly evaluates the process of detecting failures of a classifier via CSFs as described by Equation | essentially tries to optimize the trade-off between achieving low risk and high coverage. Thereby, risk is defined as the error rate of cases remaining after selection:
SC
FP((7) Risk(r) = Procision,(77 = TP((F) + FP((T)
ΣN yts T(g(a)) 2T)
(29)
Ig((())) 2))
And coverage as the ratio of cases remaining after selection:
+ +Pr()) Coverage(T) = TP;(T) + +
Σ I(((x)) 2T)
(30)
N
Typical evaluation is performed for single cut-offs of r reporting and comparing resulting risk and coverage scores. One can also calculate the Area under the Lark-Coverage-uure (AURC):
AURC- Coverage(T:) Coverage(T(--)) (Riik((T) + Risk(T(-1)))/2
E EN (I(g(z)) 7) - I(g(x:) T(-1)))
(31)
N
ΣN Vr. 1(((x,) Tb) +
ΣN Mis I(g(x:) 2 T(-
2 Σ. 1((())) T() 2 Σ I(g(xx)) 2 T(-1))
which uses the thresholds T. equivalently to the failure detection AUROC for MisD Equation 23 as the unique values of the ascending ranking of confidence scores. However, AURC is currently not used as the primary metric or part of an established evaluation protocol SC. Instead, e-AURC has been proposed (Geifman et al. 2019):
e-AURC AURC+ Risk Risk(ro))) ro)) (os((-Rs((TT)))) Risk(ro))
log(Accuracy), (32)
where is used that Risk(7)) = (1 Prevalencer) equal to the Negative Rate. Further, Prevalencer is equal to the Accuracy of the base classifier Therefore e-AURC effectively subtracts the classifier performance aspect from AURC for exclusive focus on evaluating the ranking power of CSFs. This process essentially collapses AURC back to pure ranking metric such as employed in MisD. However, as laid out in Section evaluating CSFs while being agnostic to the classifier comes with significant pitfalls preventing comprehensive method assessment.
18

---

B.2.5 UNIFICATION TASK FORMULATION TECHNICAL DETAILS
In this work, we propose establish AURC the primary metric for failure detection, as it fulfills requirements RI-R3 defined in Section 2 Comparing Equation 31 to Equation 24 deviation is observed in technical details such as the conservative interpolation of the AP-score versus trapezoidal interpolation in AURC, or the fact that AURC is defined with "reverse precision" in the form of Risk requiring minimization of the score. However, the one crucial conceptual difference is the fact that "weight" is put the risk score (second factor of the product) in Equation 31 for all steps of coverage (first factor). This in contrast to the (1 - Vr,a) factor in Equation 24] which ignores steps of incorrect predictions in sensitivity (first factor in the equation), and essentially implies that incorrect predictions of the classifier are penalized irrespective of how perfect the CSF separates these.
We argue that this aspect of evaluation is essential part of evaluating CSFs in the context of failure detection, because to truly evaluate the property of a CSF to "prevent silent failures of a classifier" don't only need to check whether CSFs able filter existing failures. We also need to check whether they might have caused new failures (or prevented further failures) by altering the training compared a neutral classifier trained without the CSF, thereby creating their own test set.
In Appendix we provide a revised implementation of AURC fixing various shortcomings of prior open-source implementations.
Figure visualizes how OoD-D protocols need to be adapted in order to follow the unified task formulation
Current
Correct Prediction
Proposed
Correct Prediction
Inlier Misclass- ification
Inlier Misclass- ification
New-class shift
New-class shift
Outlier Label:
Covariate shift
Covariate shift
Failure label Outlier label
"Inlier" Keep
"Outlier" Filter
Figure 5: Modification of the current OoD-Detection protocol for seamless integration into uni- fied evaluation. Intuitively, data samples labelled as "outlier" can by definition be set to "failure" "filter"), because the trained classifier has no possibility to correctly predict unseen classes. However, not all "inlier" are correct classifier decisions. The current OoD-D protocol rewards CSFs for not detecting inlier misclassifications (see Figure On the other hand, penalizing CSFs for not detecting these cases would dilute the desired evaluation focus on new-class shifts. Thus, propose to remove inlier misclassifications from evaluation when reporting CSF's performance under new-class shift. While taking those cases out of the confidence ranking assessment, we argue that at the same time is required to fulfil R1, i.e. assess the effect of CSFs on classifier performance. Thus, we propose to report AURC even on new-class shifts instead of AUROC,, which follows R1 because more inlier failures taken out lead to a higher ratio of OoD cases (failure cases by definition) in the test set and thus a lower accuracy.
C RELATION BETWEEN CONFIDENCE RANKING AND CONFIDENCE CALIBRATION
Calibration in the context of CSFs describes the requirement of predicted class scores to match the empirical accuracy of associated cases (e.g. cases with confidence score 0.8 are correct 80% of the time), which is useful for use cases like interpretable decision making (e.g. human assessment a need for interpretable cutoffs like "risk >> 0.8"), or assessing the applicability a trained classifier to entire data sets (Guo al. 2017). Figure 6 shows how confidence calibration and confidence ranking are two independent requirements, i.e. either of the tasks can be solved without solving the other. A CSF can theory have zero calibration error (perfect calibration) but provide ranking of
19

---

single cases and vice versa, raising the questions: Which requirement do actually want CSF to satisfy given a concrete use case? What can we do with calibrated scores in practice if there is no ranking? What concrete cases where both tasks are required i.c. a "general meaningfulness" is measured such as by proper scoring rules like negative log-likelihood or Brier Score?
We argue that for all cases where any sort of selection, cut-off, separation between cases with lower or higher confidence performed that does not rely on interpretation of raw confidence values, only confidence ranking and calibration is required: Given well-separating CSF (good scores according to ranking metrics), one can define a cut-off value on a validation set according to practical requirements such as "filter only 20% of cases" (coverage guarantee) or "filter such that maximum of 5% eπrors remain" (risk guarantee) and select cases according to this cut-off in subsequent application. Importantly, calibrated scores are not required any stage in this process. Appendix D demonstrates realistic example of how to do this under distribution shifts.
While Tomani al. (202T) studied calibration under variety of distribution shifts, to the best of our knowledge recent work in the context of neural networks only considers single CSF: the maximum softmax response (MSR) of the classifier. Applying our holistic perspective, the question arises: If other CSFs beat MSR in confidence ranking tasks, is it not likely that they could also provide better-calibratee scores? To this end, we extend our empirical study and analyze the calibration performance of all studied CSFs under distribution shifts (see Appendix G
No calibration
Perfect calibration
Poor Accuracy
Good Accuracy (e.g. "under-oonfideet)
Poor Accuracy
Good Accuracy
(eg. 'Over-contident")
No Ranking
0.5
0.5
0.5
Perfect Ranking
0.5
0.5
correct prediction
false prediction faillure
confidence score
Figure 6: This figure is a rough sketch of the results of hypothetical classifier to illustrate that the two requirements of calibration and ranking entirely independent of each other. Note that the confidence scores drawn for the combination of "perfect ranking" and "perfect calibration' (bottom right panels) reachable for MSR, because MSR has lower bound at 1/Delasses.
D DEFINING A FINAL DECISION THRESHOLD ON CONFIDENCE SCORES
The idea exists that despite a well-ranked confidence scoring function one still requires calibration of scores in order to set meaningful decision threshold the end. In Figure we explore selection under guaranteed risk (SGR) as an alternative decision making tool under distribution shift by exemplary studying ConfidNet under image corruptions on CIFAR-100. Again, comparing the practical applicability to the calibration approach: When specific application and a potential practitioner observe 0.11 ECE on the i.i.d. set, what guarantees for reliable decision making can be derived from this score? The results of the SGR study show how given desired risk y" the thresholds determined on the validation set provide reliable risk guarantees on the i.i.d. test set. Under image corruption shift, expected, these guarantees start to break in the order of confidence (here in the statistical sense) parameters 8. While providing risk guarantees under unknown distribution shifts is certainly open problem, SGR might help practitioners to develop general feeling for risk excesses under potential shifts in their application domain and to understand how choices might be able to counteract certain levels of expected shift.
20

---

For the experiments on SGR, the algorithm proposed by Geifman and El- Yaniv (2017) which computes the threshold т that maximizes coverage, while satisfying the following condition:
Ps. [R(m, g) r 6
(33)
where ö a defined confidence parameter, is the desired risk, It the risk and S are i.i.d samples from a training or validation set. m is the classifier and the confidence scoring function defined in Section B
pr -3% 1-5.4% 17.1%
ref 3.4% 1-0.1% 1.3%
N- 0.6% 4.5% [0.8%
M 13.4% 0.5% (6.1
lid-study
Corupt-shift11
Comupt-shitt-2
Comupt-shit--3
10
correct prediction
Incorrect prediction
75=10-3 7h=10-7
Figure 7: Risk guarantee analysis for ConfidNet CIFAR-100 under the increasing intensity of image corruptions (logarithmic Y-axis). Thresholds т are determined on the validation set via the SGR algorithm from Geifman and EI- Yaniv (207)) according Equation 33 for different confidence parameters 6. The desired risk r* set to 0.15 and the risk excess Risk r* under distribution shift is measured. This figure demonstrates how the risk excess between validation set i.i.d. test set is neglectable small, enabling robust decision making without the requirement for calibration. Negative excess values the i.i.d. study caused by the confidence on the risk guarantees.
E TRAINING DETAILS
E.
DATA SET SPLITS
Table 3 shows the number of images per utilized data set and associated splits. We used official splits for all data sets, except that followed Liu et al. (2019) in splitting small amount (in our case 1000 images) the i.i.d. test set for validation and tuning. Because there was no i.i.d. test set for BREEDS-Enne split for validation). The number of images in new-class shifts are sums over the original i.i.d. test splits of SVHN (26032), CIFAR-10 (10000), CIFAR-100 (10000) and TinyImagene (10000). The model was trained on each case while holding out the test split of the data set. When training on CIFAR-101 100 super-classes, we discarded the super-class "vehicle 2" because of its obvious semantic overlap with the "vehicle 1" super-class in order to mitigate evaluation noise due to label ambiguities. The sub-class shift test of the super-classes training was composed by adding up training and testing cases of each held out sub-class per super-class. For the semantic new-class shift on SVHN and iWildCam we sample 40% classes randomly to held back and train on the remaining 60% of classes (each of the runs per experiment receives sample of held-out classes).
we held 10000 training cases for this purpose (1000 of which were again
Table 3: Number of images per data set and associated splits.
Data
train
val
i..dd. test
corruptions
sub-class shift new-class shift
SVHN
73257
1000
25032
30000
CIFAR-10
50000
1000
9000
750000
46032
CIFAR-10
50000
1000
9000
750000
46032
CIFAR- 100 (super-classes)
38000
11400
iWildCam
129809
1000
7154
42791
Wilds-Camelyon
302436
1000
32560
85054
BREEDS-Enttti-13
157120
1000
9000
167592
We used the 32x32 resized data from http::///hhhha.....//////haaaaaa
21

---

E.2 CLASSIFIER ARCHITECTURES
For the CNN training. we used small convolutional network on SVHN (following Corbière et al. (2019)) ). VGG- 13 (Simonyan and Zisserman 2015) on CIFAR-10/100 (following Deeess and Taylor
(0188) ), and ResNet-50 (He et 2015) for the three robustness benchmarks. For ViT training we use the ViT-B/16 architecture and scale up all images to 384x384 pixels following Dosovitskiy et al.
(2020).
E.3 HYPERPARAMETERS
On SVHN and CIFAR- we sticked as close possible to configurations of original publications CCrriière et al. 2019 De Vries and Taylor, 20188 Liu et al., 2019) while at the same time converging to identical classification model configurations. On robustness benchmarks, configurations of respective baseline experiments (Santurkar er al. 2021: Koh et al. 2021). Thus, data augmentation following De Vries and Taylor (2018)) and on BREEDS F (horizontal flip and color jitter following Santurkar et al. (2021)) D. We used cosine decay schedule without restarts following Loshchilov and Hutter (2017) for all methods and data sets, as we found this schedule to robustly generate high-quality results (see Appendix [] ablations). All classifiers are trained with SGD and momentum 0.9. ConfidNet is trained with Adam, learning rate 10 and finetuned with learning rate 10-6 following the original configuration. Table 4 training parameters that have been modified on each data set. Table 7 shows training parameters used for finetuning the ViT.
stuck to the proposed
only used on CIFAR-10/100) (slight rotation, horizontal flip, and cutout
Table 4: Training parameters per data set. init-Ir: Initial learning rate of the cosine scheduler. wd: L.2 weight-decay. DG-CE-epochs DeepGamblers CrossEntropy pretraining epochs (as proposed in the original implementation (Liu et al. 2019)). ConfidNet-epochs: ConfidNet training epochs (on top of trained classifier) "+" ConfidNet finetuning epochs (for details see Corbière al. (2019)).
Data set
init-Ir
wd
batch size
epnchs
DGGCCE-epochs
ConfidNee-epocchs
SVHN
10
15 10
128
100
50
200+20
CIFAR-10
10
5 10
128
250
50
200+20
CIFAR-100
10-1
10-1
250
200+20
iWildCam
10-3
0
16
12
6
S+3
Wilds-Camelyong
10-2
10-2
32
5
3
3+1
BREEDSS-nnity-i
10-1
10-4
128
300
50
200+20
E.4 MODEL SELECTION
Using the validation set we selected whether or not to use dropout during training for all deterministic confidence scoring functions well as the best hyperparameter o per DeepGambler-
confidence
scoring function. For the latter, we repeated all experiments using o = 2,2,3,, 6, 10). As we found higher o to be heneficial for more complex data sets, we additionally ran [12, 15, 20) on CIFAR- 100 and o = 15 on (WildCam and BREEDS- BREEDD--Entty-15 13 (additional runs on the last two data sets had to be reduced due to computing resource constraints). Table 5 shows selected o per method and data set, 6 set. Table whether or not dropout has been used and what learning rate was selected when finetuning the VIT. To select the learning rate do single of training for all learning rates in [3 10-2 10 3 10 3 10 3 3 10 10 10-41 and chose the lowest AURC on the validation set. If dropout used to finetune the ViT the dropout rate 0.1.
whether not dropout has been used for training per method and data
22

---

Table 5: Selected hyperparameters based the validation set for all confidence scoring functions trained with the DeepGamblers objective.
Method
iWildCam
BREEDDS-Entty-1
Wilds-Camelyon-17
CIFAR- 100
CIFAR-JU 10
SVIIN
DG-MCD-EE
10
6
10
20
3
DG-MCD-MSB
15
10
10
20
20
DG-MCD-PE
10
6
10
20
DG-MSR
15
DG-PE
2.2
DG-Res
2.2
2.2
2.2
2.2
DG-Res-MCD
15
3
10
20
2.2
2.2
Table 6: Whether not dropout-training has been selected based on the validation set. This selection only done for deterministic confidence scoring methods (no MCD). "1" denotes dropout training and "0" denotes training without dropout.
Method
iWildCam
SRAFDD--nniivee
Wilds Filds-Camelyon-17
CIFAR-100
CIFAR-I0
SVHN
ConfidNet
DG-MSR
0
0
DG-PE
0
0
0
DG-Res
0
0
0
Devries et
1
0
MSR
0
PE
I 1
Table 7: Training parameters and dropout selection per data set and method for VIT. init-Ir: Initial learning rate of the cosine scheduler as selected. do: "0" denotes no dropout was used, "1" denotes dropout was used. wd: L2 weight-decay. steps: Number of batches that trained
Dataset
Method
imit-Ir
wd batch size
steps
MAHA
3 10-2
MCD
10-2
SVHN
0
128
40000
MSR
10-2
PE
10-2
MAHA
10-2
MCD
10-2
CIFAR-10
0
128
40000
MSR
3 10
PE
3 1=10-
MAHA
10-2
MCD
10 2
CIFAR-100
512
10000
MSR
10
PE
10
MAHA
3* 10
MCD
10
CIFAR-100 (super-classes)
0
512
10000
MSR
10
PE
10-3
MAHA
3x 10
MCD
3 10-3
iWildCam
512
40000
MSR
10-3
PE
10-3
MAHA
10-3
MCD
3x 10-3
Wilds Bolds-Camelyon-11 17
0
128
40000
MSR
10-3
PE
10
MAHA
3 10
MCD
10-2
COFFDD--Enitty:
128
40000
MSR
10-3
PE
10-3
23

---

F REVISED AURC IMPLEMENTATION
Our implementation of AURC is based on two implementations we found by Geifman et al and Corbiere et a The two existing implementations have several shortcomings. for instance, in Geifman et al. steps in the RC curve are not defined as new unique values in the sorted list of confidence scores, but instead each data sample, including the ones with equal confidence scores, is considered as leading to an individual classification decision with an associated risk-coverage pair. This effectively leads to a random interpolation of the RC-curve between unique confidence scores adding considerable noise to the result especially in dense singular points such confidence scores of O or A shortcoming in the implementation by Corbiere et al. is that there is no well-defined endpoint of the curve meaning the risk just drops to zero after the last RC-curve step (i.e. the coverage value corresponding to thresholding at the lowest confidence score), which effectively favours methods with higher "lowest coverages". This can make a substantial difference in practice because methods often assign equal confidence scores to more than one case especially at O and 1. Thus, analogously to scikit-learn's PRC-eurve implementation, we add final point at zero coverage and the risk remaining the risk of the last RC-curve step.
G ADDITIONAL RESULTS
This section contains additional results for Accuracy (Table 8p. AUROC, (Table 9 as well as Expected Calibration Error (Table 10). Table shows rankings of confidence scoring functions based on AURC scores (i.e. based on Table
G.1 EMPIRICAL CONFIRMATION OF THE IMPORTANCE OF REQUIREMENTS R1-R3
R1: Comprehensive evaluation requires a single standardized score that applies to arbitrary CSFs while accounting for their effects on the classifier. The arguments that lead to RI are stated in Section 2 and visualized in Figure 4 In Table 12 we provide evidence for the importance of this argument: Not following RI and instead evaluating CSFs with pure rankings metrics like AUROC, (as common in MisD) leads to substantially differing rankings of CSFs. This means that the effect of CSFs on the classifier is a crucial factor to consider in practice when ranking CSFs. The finding "AURC is able to resolve previous obscurities between classifier robustness and CSF performance" in Section 4.3 describes a concrete example of how neglecting this factor has led to misleading results in the literature.
Equivalently, we argue that conflating evaluation of failure detection with calibration, as long as the purpose for this dual-purpose assessment is not clearly stated, is shortcoming of current practices. Empirically showcasing the effects of this conflation on the ranking of CSFs analogously to Table [2] is not possible, because common metrics in PUQ (proper scoring rules) exclusively operate the predicted class scores and are not compatible with arbitrary CSFs. Instead, this restriction in itself acts exclusion criterion for these metrics for the comprehensive evaluation of CSFs.
The experiment shown in Table underlines that fulfilling R1 is essential for meaningful comparison of arbitrary CSFs. Thus, the general findings of our study described in Section 4.3 are all made possible by the proposed protocol following R1 and can be seen as further empirical confirmation of the importance of this requirement.
R2: Analogously to robustness benchmarks, progress in failure detection requires to evaluate on a nuanced and diverse set of failure sources. As described in Section 4.3) our study reveals that "none of the evaluated methods from literature beats the simple Maximum Softmax Response baseline across a realistic range of failure sources" and "prevalent OoD-D methods are only relevant in a narrow range of distribution shifts".. These findings are direct result of fulfilling R2 in our evaluation protocol and demonstrate how current protocols (without R2) lead to the proposition of
Micertainty_toole.
https://qi thub. com wwm/valllee http://t.....aaaaaaa///////////a
netrics.py
24

---

def AURC (residuals, confidence);
coverages []
risks []
n = len (residuals)
idx_scrted argargt (confidence)
cov
error_sum sum (residuulls [idx__crtedde ewveragesaapenddee n),
risks.appent (error_sum / n)
weights
tmp_weight 12
13
for i in range len (idx__e 1):
cov-1
16
error_sum error_sun residuals sict oclective_risk crror_sum / (n
17
18
tmp_wcight +=
if
21
confidence idx_sorted !- confidence(idx_sortee 111:
22
coverages, append (cov n)
risks append (selective risk)
25
weights. append weights.append (tmp_weight, n) tmp_weight
26
27
if tmp_ wep_wwiihht.. 0:
30
coverages. (p)
31
risks append weights. app (tmp_xeightt n)
32
aurc sum ( (risksssss risks(i+1]) 0.5
35
weights [i] for i in range (1en (weights)) ) 1)
36
curve (coverages, risks) return curve, aure
Listing 1: Code for AURC calculation.
25

---

Table 8: Classification performance measured as Accuracy [%] (higher is better). The color heatmap is normalized per column and classifier (separately for CNN and ViT), while whiter colors depict better scores. "cor" is the average over 5 intensity levels of image corruption shifts. Accuracy scores averaged over 5 runs per experiment on all data sets except 10 runs on CAMELYON- 17-Wilds (as recommended by the authors due to high volatility in results) and 2 runs on BREEDS. Abbreviations: nes: new-class shift (s for semantic, for non-semantic), iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR-10/100, ti: Tmnylmagenet.
PAEMCam
CAMELYON
CIFAR-100
SVHN
suidy
DCS
sub
DCS
B2S
BCS
MSR
CNN
76.2
73.2
55.0
92.7
63.6
94.0
73.9
75.1
55.5
47.4
20.3
20.6
40.3
94.3
72.5
45.9
24.6
45.9
96.1
58.7
70.6
70,6
70,6
ML.S
UNN
16.2
73.2
55.0
92.7
63.6
94.0
73.9
55.0
46
39.7
2012
72.8
45.9
24.0
65.9
96.1
70.0
70.6
20.6
PB
CNN
76.
73.2
55.0
92.7
63.0
73.9
55.0
39.7
20.2
98.3
72.5
45.9
24.6
45.9
96.1
58.7
70.6
70.6
70.6
MCD-MSR
CNN
63.6
73.6
20.7
94.5
75.0
24
06.
53.0
70.6
CNN
63.6
73.0
55.5
20.7
75.0
24.6
53.0
MCD.PE
CNN
63.6
73.6
55.8
20.7
75.0
53.0
70.6
73.6
20.7
75.0
24.0
MCD-MI
CNN
75.0
24.6
96.
53.0
70.6
ConAdNet
CNN
55.0
73.9
24.6
58.7
7066
70.6
DG-MCD-MSR
CNN
64.4
65.2
20.7
75.2
24.6
59.8
70.6
DGi-Res
UMN
73,
09.5
20.6
95.0
14.2
24.7
96.1
59.7
70.6
20.6
et
CNN
76.0
69.8
16.4
93.0
64.1
89.7
66.9
72.6
54.8
44.8
30.5
20.1
39.5
94.7
13.4
46.01
24.7
06.0
59.2
70.6
70.6
70.6
49L1
75.9
83.4
77.0
24,1
93.2
25.4
47.3
57.4
71.0
71.0
75.9
83.4
24.1
93.2
25.4
57.4
MCD-M5R
23.9
$0.9
25.4
57.3
MLS
29.4
60.5
23.9
59.9
57.3
MCD-99
VIT
99.7
74.1
99.5
00.5
90
76.7
72.6
45.0
23.9
98.5
60.9
47.0
25.4
47.0
97.7
579
71.0
71.0
71.0
MCD-EE
VI
68.0
59.7
61.1
97.6
74.1
99.6
60.5
90.9
76.7
72.6
45.0
23.9
45.0
98.5
59.9
47.0
25.4
47.0
97.7
57.3
71.0
71.
71.0
MCD-MI
VIT
68.0
99.7
61.1
97.6
74.1
99.6
60.5
90.9
76.7
72.6
45.0
23.9
45.0
98.5
50.9
47.0
25.4
47.0
97.7
57.3
71.0
71,0
71.0
MARA
VIT
70.9
72.0
49.1
97.9
75.9
99.6
83.4
91.5
79.9
77.0
45,2
24.1
45.2
95.8
93.2
47.1
25.4
47.1
97.7
57.4
710
7L.
71.0
CSFs with lack of generalization ability that often directly contradicts their purpose statement: to enable the safe application of classifiers by detecting silent failures.
R3: If there a defined classifier whose incorrect predictions are to be detected, its respective failure information should be used to assess CSF' w.r. the stated purpose instead of a surrogate task such as distribution shift detection. The fact that evaluation w.r.t failure labels, as opposed to OoD-labels, enables testing methods important argument for R3. This importance is empirically confirmed in the finding "prevalent OoD-D methods are only relevant in narrow range of distribution shifts" (see Section 4.3) which reveals clear contradiction of many of these methods' purpose statements and their actual functionality.
broad and realistic range of distribution shifts is the most
Besides the general importance of following R3, would like to empirically highlight the importance of dismissing inlier misclassifications from CSF evaluation on new-class shifts (the proposed protocol for new-class shifts is visualized in Figure 5). We argue that when assessing confidence ranking w.r.t failure detection on new class shifts, penalizing the detection of failures from the i.i.d. set is an undesired behavior of the current OoD-D protocol and subsequently propose dismiss these cases from ranking (they still affect the classifier performance considered by AURC thus fulfilling R1). Table T demonstratee the importance of this modification revealing considerable effects on the resulting CSF rankings (as measured by AUROC,).
26

---

Table 9; Confidence Ranking results measured as AUROC, [%]] (higher is better). The color heatmap is normalized per column and classifier (separately for CNN and ViT), while whiter colors depict better scores. "cor" is the average over intensity levels of image corruption shifts. AUROC, scores are averaged over 5 runs per experiment on all data sets except 10 runs on CAMELYON- 17-Wilds (as recommended by the authors due to high volatility in results) and 2 runs on BREEDS. Abbreviations: nes: new-class shift (s for semantic, ns for non-semantic), iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR-10/100, ti:
Imagnnt
BREEDS
CAMELYON
CIFAR-100
CIFAR-10
SVHN
sauds
sub
lid
sub
lid
sub
lid
sub
CDE
DCS
I5 B25
Uid
COT
S 5:555
DS 8C5
jid BCS
BCS
sc/n
vohn
c100
MSR
CNN
59.1
90.3
85.5
93.5
73.0
89.2
71.2
57.4
77.1
80.8
95.4
85.4
$4.3
95.7
56.5
31.7
95.8
96.9
92.9
93.9
97.5
97.3
U7.4
MLS
CNN
52.3
56.5
82.8
90.2
76.2
69.2
71.2
54.5
75.0
11
86.5
87.9
93.9
92.6
56.2
92.9
97.6
96.5
91.5
94.3
98.6
98.3
98.4
PE
UNN
69.0
90.2
36.
93.8
78.2
89
21.2
67.4
77.9
81
85.0
81
910
95.8
56.8
92.11
96.4
97.5
92.9
98.0
97.5
97.5
M1.1
MCD-MSR
CNN
00.2
91.4
85.6
94.3
78.2
90.6
70.0
67.5
78.4
82.
86.3
54.8
94.1
66.6
01.25
04.9
05.0
09.0
94.6
02.9
90.7
95.0
MCD MLS
CNN
$1.8
59.2
34.7
90.7
76.3
90.4
09.8
54.
75.0
00.4
86.
88.7
92.3
55.5
92.9
96.2
98.0
91.7
95.4
98.7
98.5
95.6
MCD.PE
CNN
59.8
90.9
36.3
94.2
75.4
90.6
70.0
55.6
75.3
31.
86.2
87.4
87.1
93.7
56.5
92.4
95.0
96.9
92.8
94.9
98.4
98.2
95.4
MCD-EB
CNN
90.7
94.1
788
20.0
78.2
83.9
85.5
93.6
66.8
96.2
97.5
90.7
95.1
98.2
MCD-MI
59.4
66.8
88.6
90.4
CantdNaa
CNN
73.6
86.
$7.8
96.1
94.0
DGR
CNN
91.1
62.5
86.7
93.4
94.7
31.00
63.1
74.8
50.0
MSR
56.5
$9.0
36.1
94.3
77822
95.1
78.6
90.3
81.8
87.5
93.8
89.7
95.9
92.8
97.2
98.8
95.3
90.2
93.88
98.1
98.0
08.4
64.8
78.9
91.9
99.5
78.6
98.8
98.1
64.4
80.0
97.6
MLSS
55.91
MCD-9E
MCD-MI
00.5
09.2
90.01
97.6
97.3
99.2
27

---

Table 10: Confidence Calibration Results measured as Expected Calibration Error (lower is better). To our knowledge, this table shows the most comprehensive study of confidence calibration to date featuring CSFs beyond MSR and including broad range of distribution shifts. Importantly, only the CSFs with natural output range beyond [0, 1] have undergone Platt-scaling, so the data does not provide a fair comparison of calibration errors between CSFs. Instead, the purpose is to demonstrate that CSFs beyond MSR can be calibrated and could thus considered an appropriate competition to MSR in future research. The color heatmap is normalized per column and classifier (separately for CNN and ViT), while whiter colors depict better scores. "cor" is the average over intensity levels of image corruption shifts. ECE scores are averaged over 5 runs per experiment all data sets except 10 runs on CAMELYON-177 1i-Widss (as recommended by the authors due to high new-class shift for semantic,
volatility in results) and 2 runs BREEDS. Abbreviations: ns for non-semantic), iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR- ti: TinyImagenet
IWEMCam
CIFAR-100
suidy
923
5-nCS
MSR
CNN
0.14
0.11
0:23
0.03
0.22
0.01
0.14
0.05
0.22
0.11
0.25
0.41
0.31
0.09
0.16
0.43
0.57
0.35
0.02
0:34
0.19
0.10
0.19
MLS
CNN
0.07
0.10
0.10
0.01
0.15
0.01
0.15
0.01
0.02
0.11
0.21
0.34
0.15
0.01
0.07
0.34
0.41
0.23
0.01
0.02
0.13
0.13
0.15
CNN
0.10
0.15
0.09
0.42
0.16
0.00
0.11
0.42
0.20
0.02
0.12
0.12
0.12
0.12
MCD-MSR
CNN
0.49
0:33
MCD-MLS
CNN
0.16
0.49
CNN
0.44
0.32
CanfiNNa
CNN
0.23
0.05
0.47
DG MCD MSR
CNN
0.09
0.34
0.28
0.14
DG
CNN
0.25
0:25
0.57
0:40
Devries
CNN
0.09
0.20
0.14
0.09
0.16
0.28
0.57
0.10
0.00
0.04
0.11
0.08
0.37
0.17
MCD-MSR
0.22
0.63
0.36
0.21
MLS
0.10
0.36
0,40
0.04
0.39
0.02
0.12
0.37
0.31
MCD-ER
VTT
0.12
0.14
0.16
0:01
0.17
0.00
0:37
0.05
0.98
0.25
0.41
0.22
0100
0:33
0.43
0.34
0.01
0.10
0.13
0.11
0.12
MCD-MI
VIT
0.09
0.11
0.16
0.01
0.17
0.20
0.39
0.04
0.06
0.10
0.36
0.57
0,31
0.01
0.06
6.47
0.69
D.47
O.D1
0.00
0.18
0.15
0.17
MARA
VIT
0.05
0.05
0.16
0.00
0.12
0.20
0.13
0.02
0.03
0.08
0.25
0.55
0.24
0.00
0.04
0.33
0.60
0.27
0.01
0:03
0.11
0.11
0.10
28

---

Table 11: Failure Detection Results shown as Rankings according to AURC. While providing results in the form of rankings might facilitate parsing the information quickly, it should be said that rankings hide important information in method assessment. Specifically, the notion to distinguish between "similar performance" and "substantially different" performance is lost. The color heatmap normalized per column and classifier (separately for CNN and ViT), while whiter colors depict better scores. "cor" is the average over intensity levels of image corruption shifts. Scores are (as
averaged over 5 runs per experiment on all data sets except 10 runs on CAMELYON-17-WIT recommended by the authors due to high volatility in results) and 2 runs on BREEDS. Abbreviations: nes: new-class shift (s for semantic, ns con-semanti iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR- 10/100 ti: Tiny Tinglnteenet.
:wilCCam
BREDDS
CAMELYON
CIEAR
CHARR-I 10
SVHN
sub
üd sub
nee-bra
cIO wihn
c100 wchn
<10 c100
MCD-MBR
MCD
MCT-P8
CNN
MC1-BB
CNN
MCD-MI
CNN
12
12
12
ContiiVer
CNN
12
20
11
DG. MSR
CNN
11
DG-Ree
CNN
10
10
11
10
Devsies etal.
CNN
12
12
12
12
12
MSR
VIT
MI.S
VIT
FE
VIT
MCD MIR
VIT
MCD MLLS
VIT
MCT-PE
VIT
MCD-EE
VIT
MCD-MI
VIT
MARA
VIT
29

---

Table 12: Comparing Rankings of AURC + a versus AUROC, - 3. The fact that these two metrics result in substantially different rankings of CSFs demonstrates the important effect of the CSFs on classifier performance and thus the relevance of the pitfall visualized in Figure [ Section we argue that the effects of CSFs on the classifier need to be taken into account for fair comparison (RI), i.c. the ability to prevent silent failures should be assessed by AURC instead of AUROC,- The color heatmap is normalized per column and classifier (separately for CNN and ViT), while whiter colors depict better scores. "cor" is the average over 5 intensity levels of image corruption shifts. Scores are averaged over 5 runs per experiment on all data sets except 10 runs CAMELYON MELLYON-17-A 17-Wilds (as recommended by the authors due to high volatility in results) and runs on BREEDS. Abbreviations: nes: new-class shift (s for semantic, ns for non-semantic), iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR-10/100, ti: Tinylmagenet.
BAFITTS
CAMPLVON
CIPAR 000
CIAAE II
SVEN
C30
600D
ada
ERI
C:00
3BR
CNN
CNN
16 11
MED-NLS
CNN
10
11
31
12
MED
MED-ME
CarkdNe:
30
DOANCDSER CNN
Deviacaaa al ONN
In
11
10
17
MEDAES
VIT
MIDDEE

---

Table 13: Comparing Rankings of AUROC, based on the current OoD-protoeol ("original"
+ O) versus the proposed modification of dismissing inlier misclassifications ("proposed"
+ P). The fact that these two protocols lead considerably different rankings of CSFs in many scenarios demonstrates the importance of the effect of inlier misclassifications and thus the importance of the proposed modification (visualized in Figure 55. The color heatmap is normalized per column and classifier (separately for CNN and ViT), while whiter colors depict better scores. Scores are averaged over 5 runs per experiment on data sets. Abbreviations: nes: new-class shift for semantic, for non-semantic), c10/100: CIFAR- 10/R-000000,,,,0 ti: Tinylmagenet.
OoD-D evaluation
(WilCam
CIFAR-100
CIFAR-I0
SVHN
nos-datt
cto
c100
sxfn
<10
0
ONN
CNN
MCD
CNN
MCD
CNN
MCD-M
CNN
12
CunfidNer
CNN
12
12
DG MCD MSK CNN
11
11
DG Res
CNN
12
2
Devrios al
ONN
12
12
12 17
12
11
10
MSR
PE
VIT
MCD
VII
8
MCD-MLS
VII
2
2
MC
VIT
4
MCD-EE
VIT
3
MCD
VII
MAHA
VII
H QUALITATIVE ANALYSIS OF FAILURE CASES
We categorize two types of errors for confidence scores: Overconfidence, i.e. assigning high confidence to wrong predictions, and underconfidence, i.e. assigning low confidence to correct predictions. Figure 9h) shows the three most overconfident and underconfident test cases of the MSR confidence score CIFAR-100 per distribution shift (since test cases subject to new-class shift are incorrect by default, there are no underconfident predictions). Looking at overconfident predictions, the i..d. test set categories, Images under corruption shift are subjectively hard to classify even for the human eye. The sub-class shift reveals ambiguity in the labels rather than failure of the confidence containing objects two valid categories, fish and people, indicating that superclasses CIFAR-100 100 have not been specifically designed for sub-class shift by e.g. excluding such cases (Figure 9p looks at failure cases of BREEDS-Enniti-13 which has been curated for meaningful sub-class shift). On the new-class shifts, observe the expected behaviour of predicting the training categories most semantically similar to the unknown categories. Looking at underconfident cases reveals some interesting shorteut learning (Geirhos et al. 2020): The fact that the two racoon images received low confidence despite showing their faces indicates that the model relies on spurious features perhaps related the background. Similarly, the fox image on the corruption tests may have received low confidence due to the artificial frame. Further, the colored pixel noise in the hamster image source of confusion hints at the mode relying on local texture patterns rather than global semantic features.
some expected confusions between semantically close
While the previous qualitative study only showcases confidence based on MSR, the proposed unified protocol facilitates study arbitrary confidence scoring functions in detail and across different distribution shifts. Thus, Figure 1b) we study failure cases the Predictive Entropy obtained from Monte Carlo Dropout (MCD-PB) the BREEDS-EEtity data set. Entropy scores reversed and
31

---

Canoterr
JURC
CANBSTO
CHA 120
190
50W
3,800
14
DECEDS
CANCJON
DIN--1
Figure 8: Comparison of CNN versus ViT-based Classifier. Boxplots show median (black line) and quartiles, and include all CSFs and studies shown Table
scaled [0, for interpretabilityy Note, how an "external" confidence score like entropy can be zero even on the predicted class (such decoupling would be inherently impossible with MSR). Looking at overconfident cases again reveals label ambiguities of confidence scoring. Examples are a bird in front the i.i.d. test set rather than actual failures monitor ("monitor" is a training sub-class "equipment"), a woman wearing a scarf on top of poncho ("poncho" training sub-class to "garment"), , or vehicle behind a fence ("fence" is a sub-class to man-made structure). Further, this study reveals interesting shortcut learning such as focusing on knitting patterns ("garment" vs. "accessory"), or a dog being predicted as unlikely to appear in front of the water. A really hard categorization task posed by the image showing hot air balloons, they are confused with volleyballs (training sub-class to "equipment").
1 TECHNICAL FORMULATION OF CLASSIFIER-OUTPUS BASED CONFIDENCE SCORING FUNCTIONS
The majority of confidence scoring functions explored in this study is based on one the well-established methods for quantifying the uncertainty of classifier's prediction. While strictly
32

---

a)
Overconfidence
Underconfidence
L..d. study
Corruption-1
cub-olass--hiie
New-class-shitt
L.d. study
Corruption-1
Sub-class-sin
CIFAR-10
SVHN
frue: cup prod: 1,000
true: bed corfidi: 1.000
tue: fish contal: 1.000
true: automablee
truc:
frue: rassoon pred. racocon
Irus: fox pred. fox
Irue: food cont pred:
pred
pred: Worm
confis::0.04
5
true: bowl pred: cup 1,000
truy imushroom 1.000
true: horee confid::
Irue. prod: worm
thue: raccocn pred, reczoon
thae: hemster pred. harmater
Irue: non insec prodd: Indeo 0.229
confid:: 1.000
mushroom confist: 1.000
bue automobile
true
true: shark pred
truc: fox pred:
frue: large, om
prod: 0.232
conlid: 1,000
confidd 1. 0.0
confid:: 1,000
contd: 1.000
5
Overconfidence
Underconfidence
l..dd study
Sub-clasa-shift
I.d. study
Sub-class-shift
true: accessory
true: garment pred.: accessory confid.: 1,000
true: produce pred.:: produce
true: mammal pred.: mammal
pred.: garment
confid.: 1,000
confid.: 0.000
confid.: 0.029
true: bird
true: craft pred.: equipment confid.: 1,000
true: reptile pred.: reptile
true: instrument. pred.: instrument confid.:
pred.: equipment confid.: 1.000
confid.: 0.004
true: furniture pred.: mammal confid.: 1.000
true: man-made
true: craft
furniture pred. furniture confid.: 0.072
pred.: wheeled v confid.: 1,000
pred.: craft confid.: 0.089
Figure 9: Qualitative study of failure cases in failure detection. a) The three most overconfident and underconfident test cases of MSR for different distribution shifts on CIFAR-I00, 100. b) The three most overconfident and underconfident test cases of Predictive Entropy obtained by Monte Carlo Dropout (MCD-PE) on BREEDS. BREEDS-Entity-13:
33

---

proper scoring rules such as NLL Brier Score only assess performance related failure detection for one of those measures, MSR, the protocol proposed in this study enables comparison across arbitrary measures. While PE is often associated with capturing prediction's 's total uncertainty ((Smith and Gal 2018 Depeweg et al. 2018 Malinin and Gales) 2018), MCD-EE is said to capture the uncertainty inherent in the data (or aleatoric uncertaintty. Consequentially, subtracting the latter from the former, i.c. the mutual information (MCD-MI) between the parameters O and the categorical label y, is often associated with only capturing the model uncertainty (a form of epistemic uncertainty). The Maximum Logit Score (MLS), proposed by Vaze et al. (2022). uses the magnitude of the logits which they argue carries information relevant for OoD-D, where greater logits should correspont to more certain predictions.
Maximum Softmax Response (MSR):
P max P(y = c|<"; ;D),
(34)
where с represents the different classes.
Predictive Entropy (PE):
; = Σ Ply = d) In(((y = cl="; D)
(35)
Maximum Logit Score (MLS)):
max f(x",, D);
(36)
where f is the function producing the logits that used to create the probability vector P(y|æ" D) = (f2t D)).
Maximum Softmax Response obtained from Monte Carlo Dropout (MCD-MSR):
P = max D) PPweel> :0),,
(37)
where 0 are the model parameters.
Predictive Entropy obtained from Monte Carlo Dropout (MCD-PE):
(Pylpa" 0)]],
(38)
Expected Entropy obtained from Monte Carlo Dropout (MCD-EE):
(M(P)/= O)]
(39)
Mutual Information obtained from Monte Carlo Dropout (MCD-MI); Tyy; 01) = H (P 0)]] p) (M(p))
(40)
Maximum Logit Score obtained from Monte Carlo Dropout (MCD-MLS):
max (f 0)),
(41)
where of is the function producing the logits that are used to create the probability vector P(y|=", softmax (x(x" 0)).
J REPRODUCIBILITY AND DEVIATIONS FROM ORIGINAL BASELINE CONFIGURATIONS
meaningful evaluation of confidence scoring methods for failure detection requires deviating from original configurations of baselines in order to assimilate the underlying classifier across compared methods. Since re-implementing and re-configuring baseline methods is prone to inducing biases in comparative studies and respective conclusions (Lipton and Steinhardd 2018). in this section, we want to explain all deviations in detail and provide evidence that baselines perform par with or superior the original versions under our configuration.
As described in Section
34

---

J.1. DEEPGAMBLERS
In Table we compare our configuration against the original configuration using the results reported on CIFAR-10. We deviate from the original configuration in using additional data augmentation in the form of cutout, decaying the learning rate via cosine scheduler, training a VGG-13 13 model instead of VGG-16, and not using dropout for training (as selected by model selection). The hyperparameter o has been selected as 2.2 on this data set, which is identical the original configuration. The comparison shows, that our results are substantially better than the original configuration, mainly due to better classifier accuracy.
Table 14: Comparing our DeepGamblers configuration to the original configuration (as proposed in Liu et al. (2019)) on CIFAR-10. The table shows risk scores at predefined coverages. Lower is better.
Coverage
configuration
original configuration
1.00
4.94
6.12
0.95
2.69
3,49
0.90
1.37
2.19
0.85
0.82
1.09
0.80
0.54
0.66
0.75
0.43
0.52
DEVRIES ET AL. J.2
The only substantial deviation from the original protocol is the replacement of the proposed MultiStep learning rate scheduler with our cosine scheduler (the latter is more aggressive with overall higher learning rates). As Figure [0 shows, this deviation has a negative effect on the OoD-detection performance only when training CIFAR-10 and testing on EinyImagenet. There, our configuration only achieves 0.965 AUROC instead 97.0 AUROC from the original paper. In all other settings, the cosine scheduler results in superior performance. As part of the assimilation of the training configuration across compared methods, we increased the number of training epochs for Devries et al. from 200 to 250. Since performance did not benefit from this increase in training time, we believe there is no fairness issue with DeepGamblers and ConfidNet requiring additional training stages leading to overall more training exposure (see Table We did not include additional confidence measures such as MSR, PE, or any MCD variations based on the Devries et al. learning objective, as found no interesting synergies occur (this is in contrast to DeepGamblers and ConfidNet).
35

---

15
L.d.d.
200
130
690
New class shift
TinylmageNet scuncc
C903
6998
646
CMR
201
2014
New class shift
6402
CIFAR-100
200
440
New class shift
3305
42-
SVHN
320
60-
Figure 10: Ablation study comparing the originally proposed Multistep scheduler against the Cosine scheduler for the confidence scores from Devries et trained on CIFAR- 10. One dot denotes one run, the thin line denotes the mean across runs and the thick line denotes the median across runs.
J.3 CONFIDNET
The original authors propose three-stage epoch selection, first selecting the best classifier according to validation accuracy, second selecting the best ConfidNet training epoch according to fail-average- precision (with failure defined the positive label), and third selecting the best ConfidNet finetuning epoch according to thil-average-precision (Corbière et al. 2019). We found results to be very volatile across single epochs, especially for fail-avera fvil-average precision. Since reducing the validation set to 1000 cases in the process of assimilating training protocols (the original configuration used 5000 cases from the training data) amplified volatility further, we ran an ablation study comparing the proposed model selection to simply taking the last epoch at each stage. Figure [1 shows the results, indicating that (at least for our smaller validation set), choosing the latest epoch performs superior to epoch selection. We further deviate from the original configuration by using VGG-13 instead of VGG-16 and applying cutout in addition for data augmentation. Comparing our results to the ones in the original publication,
vastly improve classifier accuracy and thus AURC scores on SVHN and CIFAR- 10/ 10/100, although is unclear how much of this increase is to be attributed to our configuration and how much the different training and validation splits.
36

---

0.0
80
0146
0964
0.42
0941
Figure 11: Ablation study for the three-staged model selection proposed in the original ConfidNet configuration. One dot denotes one and evaluation the CIFAR-10 i.i.d. test set, the thin line denotes the mean across runs and the thick line denotes the median across runs. MS: Model selection, latest: select the latest epoch each training stage.
J.4 MAXIMUM LOGIT SCORE
Since Vaze et al. 2022)) do not include the ViT architecture in their evaluation, we only compare CNN results. While Vaze et al. (2022) use the VGG-32 classifier, we use different classifiers depending on the data set as discussed in Appendix F.2. On each data set we use the same classifier for MLS evaluation as for the other methods. The authors employ a variety of improvements to their baseline strategy of which do not employ label smoothing or ensembling to ensure fair comparison with other CSFs. Thus, expect results to lie between their baseline and their improved baseline. Table (5bll[[5]comppre our configuration to theirs as measured by A AUROC, on the SVHN dataset in the original open-set setting, and confirms our expected performance relations.
only some: we do use cosine annealing to schedule learning rates, but
Table 15: Comparing our classifier configuration to the configuration proposed by Vaze et al. (022) measured by AUROC, on the SVHN dataset in the open-set setting. Baseline MLS and Baseline MSR+ use the aforementioned improvements to the training strategy.
Method
AUROC,
100 t
Baseline MSR (Vaze al. 2022
88.6
MSR (ours)
94.0
MLS (ours)
94.3
MSR (improved baseline)
Vaze et al.
2022
96.0
MI.S (improved baseline)
Vaze et
2022
97.1
J.5 MAHALANOBIS SCORE
Fort et al. (2021 compare different vision transformer architectures, due to computational constraints we only consider the VTT-B/16 architectue. Since the original publication does not report how results have been obtained, i.e. whether multiple runs have been performed and how final results were selected from these runs, we report both our best run as well as an average over five runs, selected by best accuracy. Table 16 compares our implementation to theirs as measured by AUROC, trained on CEAAR-1O/CIFAR.. 100 using 1ORAR--O//CIARRR
as the -itto--distribution dataset respectively. We that average is slightly below the original results, but results are very volatile with high standard deviations over runs. Our best run results are en par with the original results. Equivalently original results, MAHA score consistently higher compared MSR.
37

---

Table 16: Comparing ViT-B/16 implementation the by Fort et al. (002 measured by AUROC, when fine-tuned on CIFAR-1OCIIAR- 100 using CIFAR- 10O//IFAARAAIIII 10 as the out-of- distribution dataset respectively.
ID to OD
MAHA AUROC, 100 1
MSR AUROC, * 100 t
CIFAR 10 - CIFAR-100 (ours, mean)
97.99 = 0.31
96.71 0.39
CIFAR- 10 CIFAR-100 (ours, best)
98.49
97.69
CIFAR-10 - CIFAR-100 (Fort et al.
2021
98.42
97.68
CIFAR 100 - CIFAR-10 (ours, mean)
91.13 + 7.86
89.02 + 4.96
CIFAR- 100 -> CIFAR-10 (ours, best)
95.23
92.50
CIFAR- 100 -> CIFAR-10
(Fort et al.
2021
95.53
91.89
38


Reviews:
[{"review_id": "gpWWfuE8T05", "reviewer": "Reviewer_7dx4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "With the advancements of machine learning and boosting availability of compatible hardware as well as cloud services, machine learning-based classification systems have been deployed in numerous scenarios. Regardless of the specific application — some more critical than others — it is essential to detect the failure cases in a timely manner in order to quickly iterate on targeted improvements.\n\nThe authors pointed out that currently the dominant means for such failure detection is through checking some type of confidence scores, either directly produced by the model as an output, or indirectly calculated through some other heuristics (e.g., recall the notion of uncertainty in Focal Loss). The authors claimed that these different approaches are aiming to solve the same failure detection problem yet through distinctive methods which led to diverging protocols of various subfields (misclassification detection, out-of-distribution detection, selective classification, predictive uncertainty quantification). The authors then systematically analyzed the existing methods to reveal their pitfalls, and proposed a holistic and realistic approach for failure detection.\n\nThe proposed metric is the Area under the Risk-Coverage Curve (AURC), which is proposed in Geifman et al. This metric assess the rate of silent failures of a classifier across different failure filtering thresholds. The authors claimed that the metric avoids all pitfalls they revealed, and recommended wide adoption across all subfields of research within classification failure detection. The authors further delineated the modification of protocols needed to adapt to the new evaluation approach from these aforementioned subfields.", "summary_of_the_review": "This paper encouraged researchers to view multiple subfields (misclassification detection, out-of-distribution detection, selective classification, predictive uncertainty quantification) under the umbrella of classification failure detection through a unified perspective, and proposed improvements to the problem statements and walked through the design of a generic evaluation protocol. More remarkably, the authors showed that a simple baseline can match or beat the state-of-the-arts in these subfields on the unified benchmark, since these SOTA techniques only perform in their selective subfield and miss the generalizability in the bigger picture. I like the scope, vision and insight from the authors, and would recommend for acceptance of this paper.", "strength_and_weaknesses": "Strengths\n1. I strongly encourage the intention stated by the authors: to articulate a call to various interrelated yet isolated communities to acknowledge the shortcomings of current practices and adapt to a more reasonable practice that not only better fit their alleged purpose but also help bring the isolated fields together.\n2. Section 2 is a deliberate and eye-catching section. The authors discussed three pitfalls of the existing methods and derived the corresponding point-by-point requirements for a desired evaluation protocol for failure detection. Among them, I particularly like (1) the second pitfall where the authors drew out attention towards different sources of classification failure and pointed out that existing methods often cover a subset of them (also illustrated in Figure 1). (2) the third pitfall where most relevant research claim failure detection as their purpose but in fact are evaluated on outlier detection, causing problems due to the mismatch (also illustrated in Figure 2).\n3. I also like that the authors pointed out an easily neglected issue: rounding errors during softmax operation may largely affect the resulting evaluation metrics.\n\nWeaknesses\n1. Personally I believe reorganizing section 4.2 would be beneficial. For example, consider organizing the CSFs to be compared by their subfields (MisD, SC, PUQ, OoD-D) if possible?\n2. Table 1 is a bit difficult for me to quickly process — I understand what is displayed but my brain is not well trained to instantly make sense of this representation. Is this just me or is it a common inconvenience? If it’s that latter, I would suggest adding a column for each dataset showing the “sum of rank order for the current method across all evaluations in this dataset”.\n\nMinor Things\n1. Typo? Section 2, Pitfall 2, last sentence. “…it is not realistic to exclusively assume classification failures from label-altering shifts **an** no failures caused by label-preserving shifts.”\n2. Typo? Section 4.3, Under heading “Different types of uncertainty are empirically not distinguishable.“ 3rd line. “…we are interested in the **extend** to which such relations can be confirmed…”\n3. Is there any reason $y_{f,i}$ in Eq. 24 is highlighted?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Kep6kesNts", "reviewer": "Reviewer_JKia", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper offers a set of criteria (and a metric area under risk coverage curve) to compare different flavors of failure detection. It gives experiments  that compare different methods according to these criteria.", "summary_of_the_review": "The proposed criteria are thoughtful and offer, agree or disagree, a good point of departure for discussion.\n\nThe \"step back and look in the mirror\" approach is welcome.\n\nThe paper combines many methods that are usually silo'ed.\n\nThe findings section is super interesting.\n\nThe Conclusions paragraph is wonderful.", "strength_and_weaknesses": "(strengths)\n\nThe paper evidently springs from a substantial understanding of and experience with the topic.\n\nThe proposed criteria are thoughtful and offer, agree or disagree, a good point of departure for discussion.\n\nThe \"step back and look in the mirror\" approach is welcome.\n\nThe findings section is super interesting.\n\n(weaknesses)\n\nSometimes the paper covers ground too fast for me (too condensed).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ABazAe-mHzO", "reviewer": "Reviewer_poSv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper summarizes various types of classification failures and confidence scoring metrics and identifies their shortcomings with respect to the softmax baseline. Extensive experiments on public image benchmarks (CIFAR-10, CIFAR-100, SVHN, CAMELYON-17-Wilds, iWildCam-2020- Wilds, BREEDS-ENTITY-13) indicate that confidence scoring techniques are sensitive to the dataset choice, integer precision, and calibration.\n", "summary_of_the_review": "A useful paper offering insightful, novel analysis, but very difficult to follow.\n", "strength_and_weaknesses": "The key strength of the paper is the thorough discussion of evaluation of model classification failures, which is an important and often overlooked part of ML research. The key conclusion is concisely stated in the conclusion: a need for greater introspection and possibly less novelty. \n\nThe key weakness is that the paper is difficult to follow. It is not clear how the pitfalls discussed in Section 2 are evaluated or studied in the remainder of the paper. A number of relevant method definitions are found in the appendix, rather than the main paper. In addition, definitions of techniques is often found after the fact. The paper would benefit from greater organization and extraction of key findings and results that support these conclusions. In Table 2, a number of results are reported, however, it is difficult to parse the metrics, color code, and main results from the paper. One of key findings is that MSR outperforms other techniques, but (a) this is not obvious from the results as its not clear what trends to look for in the metrics (a simple up/down arrow would help) (b) no analysis and discussion of other techniques is provided.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%