Overcoming Common Flaws in the Evaluation of Selective Classification Systems
Authors: Jeremias Traub
Till J. Bungert
Carsten T. Lüth
Michael Baumgartner
Klaus Maier-Hein
Lena Maier-hein
Paul F Jaeger
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Selective Classification
Method evaluation
Failure Detection
Primary area: evaluation
TL;DR: We reveal common flaws of Selective Classification evaluation metrics such as the AURC and propose a suitable alternative based on the Generalized Risk.

Abstract: Selective Classification, wherein models can reject low-confidence predictions, promises reliable translation of machine-learning based classification systems to real-world scenarios such as clinical diagnostics. While current evaluation of these systems typically assumes fixed working points based on pre-defined rejection thresholds, methodological progress requires benchmarking the general performance of systems akin to the $\mathrm{AUROC}$ in standard classification. In this work, we define 5 requirements for multi-threshold metrics in selective classification regarding task alignment, interpretability, and flexibility, and show how current approaches fail to meet them. We propose the Area under the Generalized Risk Coverage curve ($\mathrm{AUGRC}$), which meets all requirements and can be directly interpreted as the average risk of undetected failures. We empirically demonstrate the relevance of $\mathrm{AUGRC}$ on a comprehensive benchmark spanning 6 data sets and 13 confidence scoring functions. We find that the proposed metric substantially changes metric rankings on 5 out of the 6 data sets.

Overcoming Common Flaws in the Evaluation of Selective Classification Systems
Jeremias Traub 1.2 Till J. Bungert 12.6 Carsten Lüth 1.2 Michael Baumgartner
Klaus H. Maier-Hein 2,3,5,6,7 Lena Maier-Hein 2.4,6,7 Paul F. Jäger
German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group, Germany
2Helmholtz Imaging, DKFZ Heidelberg, Germany
DKFZ Heidelberg, Division of Medical Image Computing (MIC), Germany
DKFZ Heidelberg, Division of Intelligent Medical Systems (IMSY), Germany
Pattern Analysis and Learning Group. Department Radiation Oncology,
Heidelberg University Hospital, 69120 Heidelberg. Germany
FFaullyy of Mathematics and Computer Science, University of Heidelberg, Germany
National Center for Tumor Diseases (NCT) Heidelberg jeremias. wwubbbkkfz-heidelle de
Abstract
Selective Classification, wherein models can reject low-confidence predictions, promises reliable translation of machine-leaminng based classification systems real-world scenarios such as clinical diagnosties. While current evaluation of these systems typically assumes fixed working points based on pre-defined rejection thresholds, methodological progress requires benchmarking the general perfor- mance of systems akin the AUROC in standard classification. In this work, we define 5 requirements for multi-threshold metrics in selective classification regarding task alignment, interpretability, and flexibility, and show how current approaches fail to meet them. We propose the Area under the Generalized Risk Coverage curve (AUGRC), which meets requirements and can be directly in- terpreted as the average risk of undetected failures. We empirically demonstrate the relevance of AUGRC on a comprehensive benchmark spanning data sets and 13 confidence scoring functions. We find that the proposed metric substantially changes metric rankings on 5 out of the data sets.
1 Introduction
Selective Classification (SC) is increasingly recognized as crucial component for the reliable de-
ployment of machine lcarning-basee based classification systems real-world scenarios, such as clinical
diagnostics [Dvijotham et al. 2023 Leibig et al. 2022 Dembrower et al. 2020 Yala et al. 2019).
The core idea of SC is to equip models with the ability to reject low-confidence predictions. thereby
enhancing the overall reliability and safety of the system [Geifman and El- Yaniv 2017 Chow 1957 EI- Yaniv et al. 2010). This typically achieved through three key components: classifier that makes the predictions, confidence scoring function (CSF) that assesses the reliability of these predictions, and rejection threshold т that determines when to reject a prediction based on its confidence score.
In evaluating SC systems, two primary concepts are essential: risk and coverage. Risk expresses the classifier's ' егror potential and is typically measured by its misclassification rate. Coverage, on the other hand, indicates the proportion of instances where the model makes prediction rather than rejecting it. An effective SC system aims to minimize risk while maintaining high coverage, ensuring that the model provides accurate predictions for as many instances as possible.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Current evaluation of SC systems often focuses on fixed working points defined by pre-set rejection
thresholds (Geifmmn and EI- Yaniv 2017 Liu al. 2019, Geifman and EI- Yaniv 2019] For
stance, a common evaluation metric might be the selective risk at given coverage of 70% [Geifman and Yaniv 2019), which communicates the potential risk associated with a specific confidence score с and selection threshold т to a patient or end-user. While this method is useful for commu- nicating risk in specific instances, it does not provide a comprehensive evaluation of the system's overall performance. In standard classification, this is analogous to the need for the Area Under the Receiver Operating Characteristic (AUROC) curve rather than evaluating sensitivity at a specific specificity, which can be highly misleading when assessing general classifier capabilities [Maier- Hein et al., 2022]. The AUROC provides a holistic view of the classifier's performance across all possible thresholds, thereby driving methodological progress. Similarly, SC requires multi- threshold metric that aggregates performance across all rejection thresholds to fully benchmark the system's is capabilities.
Several current approaches attempt to address this need for multi-threshold metries in SC. The Area Under the Risk Coverage curve (AURC) is the most prevalent of these Greefmmn et al. [2018). How- ever, we demonstrate in this work that AURC has significant limitations as it fails to adequately translate the risk from specific working points into meaningful aggregated evaluation score. This shortcoming hampers the ability to benchmark and improve SC methodologies effectively.
In our work, we aim to bridge this gap by providing a robust evaluation framework for SC. Our contributions are as follows:
We are the first to provide a comprehensive SC evaluation framework, explicitly deriving meaningful task formulations for different evaluation and application scenarios such as working point versus multi-threshold evaluation.
Refined Task Formulation:
Formulation of Requirements: We define five critical requirements for SC, focusing on task suitability, interpretability, and flexibility. We assess current multi-threshold metries against our five requirements and demonstrate their shortcomings.
metrics in
We introduce the Area Under the Generalized Risk Coverage (AUGRC) curve, new metric designed to overcome the flaws of current multi-threshen metrics for SC. AUGRC meets all five requirements, providing comprehensive and interpretable measure of SC system performance.
Proposal of AUGRC:
Empirical Validation:
We empirically demonstrate the relevance and effectiveness of AUGRC through a comprehensive benchmark spanning 6 datasets and 13 confidence scoring functions.
In summary, our work presents a significant advancement in the evaluation of SC systems, offering a more reliable and interpretable metric that can drive further methodological progress in the field.
2 Refined Task Formulation
Selective Classification (SC) systems consist of a classifier X + 3r which outputs prediction and a confidence scoring function (CSF) g : X' -> IR, which outputs a confidence score associated with the prediction. Assuming supervised training setup, let {(x,, Ue) 3.) be dataset contain- ing N independent samples from the source distribution P(X, Y) over X' x y. Given a rejection threshold r, the model prediction is accepted only if the corresponding score is larger than 7:
m(x), if g()()) 2 T reject, otherwise
(x) =
(1)
Given an error function E: )' x + R+, the overall risk of the classifier m is given by R(m)
N. Ei= ((((x),)). The error function thereby contains the measure of classification performance suitable for the task at hand. Commonly, SC literature assumes a 0/1 error corresponding to the failure indicator variable represents the probability of misclassification P(Y: = 1). By rejecting low-confidence predictions, the selective classifier can decrease the risk of misclassifications at the cost of letting the classifier make predictions on only a fraction of the dataset. This fraction is denoted as the coverage = P(g(z) 2 r), with the respective empirical estimator & En I(g(x,) 2 r). Evaluation of SC
with yr, (I, Vi, Vi. m) == Ily * m(x;)]. In this case, the overall risk

---

Selective Classifictton System
Melhod Development ("Mutt-hrreshold eval.)
Application ("Working point eval.")
Current practide
Input Data
1. Scan over
Reject pred.
rejection thresholds
Risk
(AURC)
Classifier
CSF
Selactive P/fall accept)
Rp Ta
Ours
Coverage
(AUGRC)
Prediction
Cont. score
Patient
Gorrect prediction
2. Get area under Risk-coverage
Generalized
Accept pred.
Incomrct prediction ("Sient" failure)
Pifail, accepl)
Reject if
Generalized risk: "The general system produce "silent" failure for prediction"
Selective risk: "Given and f, the risk of a "silent" fallure occuring for an accepted prediction"
b)
AUGRC (ours)
NLL Brier
Requirement
Description
AURC
C-AURC
AUROC,
Score
sensitive both factors: classification performance and CSF ranking quality
R1
Task Alignment
4
4
x
v
improving factor while keeping the other one fixed leads to better metric value
Monotonicity
R3
Ranking Interpretability
ranking quality distance to the optimum
on permutation
x
v
Flexibility
fexxible CSF
RS
Enor Flexibility
Tlexible w.r.t. error function (e.g. beyond accuracy)
x
Figure 1: The AUGRC metric based on Generalized Risk overcomes common flaws in cur- rent evaluation of Selective classification (SC). a) Refined task definition for SC. Analogously to standard classification, we distinguish between holistic evaluation for method development and benchmarking using nulti-threshold metrics versus evaluation of specific application scenarios at pre-determined working points. The current most prevalent multi-threst tulti-threshold metric in SC, AURC, is based on Selective Risk, concept for working point evaluation that is not suitable for aggregation over rejection thresholds (red arrow). To fill this gap, we formulate the new concept of Generalized Risk and corresponding metric, AUGRC (green arrow). b) We formalize our perspective on SC evaluation by identifying five key requirements for multi-threshold metrics and analyze how previ- ous metrics fail to fulfill them. Abbreviations, CSF: Confidence Scoring Function, AU(G)RC: Area Under the (Generalized) Risk Coverage curve.
systems often focuses on preventing "silent", i.c. undetected failures for which both yr, = 1 and g(x,)= 2 T.
Deploying SC system in practice requires the selection fixed rejection threshold т. In the fol-
lowing refined evaluation protocol, we distinguish between the well-established iepliiatioonspeciin task formulation where SC models are evaluated individual working points (Section 2.1) and SC
method development where evaluation is independent of individual working points (Section
2.1 Evaluating SC systems in applied settings
Current Selective Classification evaluation commonly reports the selective risk of the system at given coverage ("risk @ @ooerage"", which implies pre-determined cutoff on the risk coverage (e.g. "maximum risk of 5%") leading to a working point the system defined by a rejection thresh- old The selective risk is defined as:
N {(m(xs) vi) 1(g(x;) 7)
Selective Risk(m.g))
Σ I(g(x;) 2 T)
This formulation only considers accepted predictions with > r. For binary failure error, this
risk is an empirical estimator for the conditional probability P(Y 11y(x) 2 7). Thus, this metric effectively communicates the risk of "silent" failure for a specific prediction of classifier m that has been accepted (c т) at pre-defined threshold T. This information may be useful in applied medical settings, e.g. to communicate the risk of misclassification for patient whose associated

---

prediction has been accepted by a given SC system (Cp > T). Aside from the selective risk, other performance measurements for working point selection include for example Classification quality and Rejection quality [Condessa et al. 2017].
2.2 Evaluating SC systems for method development and benchmarking
Evaluating the general performance of Selective Classification system requires metric for a more comprehensive view compared to a working point analysis based on a fixed singu- lar rejection threshold, analogous binary classification tasks, where the Area Under the Receiver Operating Characteristic (AUROC) is used to assess the entire system performance across multiple classifier thresholds. However, the working point analysis can not be simply translated to multi- threshold aggregation, as it is based on Selective Risk, which only considers the risk w.r. accepted predictions, assuming that a specific selection has already occurred (P(Y)- This evaluation ignores rejected cases and thus contradicts the holistic assessment, where we are interested in the general risk for any prediction causing silent failure independent of the future selection decision of individual predictions. This holistic assessment is reflected by the joint proba- bility P(Y, = 1, ((x) 2 T), which reflects the risk of silent failure for any prediction processed by the system including cases that are potentially rejected in the future.
g(x) 7)).
To address the discrepancy between the need for holistic assessment versus the selective risk's focus on already selected cases, we formulate the generalized risk:
Generalized Risk g)()) =
Σ {(m(xx)))) T[g(s;) 2 r].
This metric reflects the joint probability of misclassification and acceptance by the confidence score threshold. By aggregating this risk over multiple rejection thresholds, we can evaluate the overall performance of the SC system.
2.3 Requirements for Selective Classification multi-threshold metries
Based on the refined task definition given above, we can formulate concrete five requirements R1-R5 for multi-threshold metrics in Selective Classification:
R1: Task Alignment. The general goal in SC is to prevent silent failures either by preventing fail- ures in the first place (via classifier performance), or by detecting the remaining failures (via CSF ranking quality). As argued in Jäger et al. [2023]. it is crucial to jointly evaluate both aspects of an SC system, since the choice of CSF generally affects the underlying classifier. R2: Monotonic- ity. The metric should be monotonic w.r.t both evaluated factors stated in R1, i.e. improving on one of the two factors while keeping the other one fixed results in an improved metric value. Note that R2 does not make assumptions about how the two factors are combined, but it represents a minimum requirement for meaningful comparison and optimization of the metric. R3: Ranking Interpretability. Interpretability is crucial component of a metric [Maier-Hein et al. [022]. AU- ROC is the de facto standard ranking metric for binary classification tasks, providing an intuitive assessment of ranking quality which is proportional to the number of permutations needed for es- tablishing an optimal ranking. This can also be interpreted as the "probability of a positive sample having higher score than a negative one". Ideally, the SC metric should follow this intuitive as- sessment of rankings. R4: CSF Flexibility. The metric should be applicable to arbitrary choices of CSFs. This includes external CSFs, i.e. scores that are not based directly
the classifier output. RS: Error Flexibility. Current SC literature largely focuses on 0/1 error (i.e. 1- accuracy) in their risk computation. However, in many real-world scenarios, accuracy is not an adequate classification metric, such as in the presence of class imbalance and for pixel-level classification. Thus, the SC metric should be flexible w.r.t the choice of error function.
2.4 Current multi-threshold metrics in SC do not fulfill requirements R1-R5
AURC: Geifman et al. 2018 derive the AURC as the Area under the Selective
curve. This metric is the most prevalent multi-threshold metric in SC [Geifman et al. 2018 Jäger
al. 2023 Bungert et al. 2003 [Cheng et al. 2023 Zhu al., 2023a] Varshney al. 2020
Naushad and Voiculescu 2024 Van Landeghem et al. 2024
2023
7hu et al. 2022 Xin et al.

---

2021 Yoshikawa and Okazaki 2023 Ding et al. 2020 Zhu et al. 2023b Galil and El- Yaniv, 2021
Franc et al., [023 Cen et al. [203] Xia and Bouganis 2022 Cattelan and Silva [2023] Tran et al.
2022 Kim et al., 2023] Ashukha et al.) 2020 Xia et al. 2024]. For the 0/1-error, AURC can be
expressed through the following integral:
AURC = L. P(V)= 2 (P((()) 27)
(4)
This integral effectively aggregates the Selective Risk (Equation 2 over all fractions of accepted pre- dictions (i.e. coverage). However, as discussed in Section 2.2 the Selective Risk is not suitable for aggregation over rejection thresholds to holistically assess a SC system. This inadequacy effectively leads to an excessive over- weightinng of high-confidence failures in AURC and in return to breach- ing the requirements of monotonicity (R2) and ranking interpretability (R3). We provide empirical evidence for these shorteomings on toy data (Section 3) and real-world data (Section 4.2). Further, current SC. literature usually employs the 0/1 error function, even though the related Accuracy metric is well known to be an unsuitable classification performance measure for many applications. For ex- ample, Balanced Accuracy is used to address class imbalance and metrics such as the Dice Score employed for segmentation. Moving beyond the 0/1 error yields a higher variance in distinct error values and may thus amplify the shortcoming of over verrweighting high-confidence failures. As more sophisticated error functions being are an important direction of future SC research, it is crucial to ensure the compatibility of metrics in SC evaluation.
c-AURC: Geifman et al. 2018 further introduce the excess-A AURS as the difference to the AURC of an optimal confidence scoring function (denoted as AURC*)
-AURC- AURC - AURC"
(5)
It relies on the intuition that subtracting the optimal AURC eliminates the contribution of the over- all classification performance and collapses it to pure ranking metric [Geifman et al. 2018 Galil let al. 2023 Jäger et al. 2023]. However, several works demonstrated that this intuition does not hold and that the e-AURC is still sensitive to the overall classification performance (Galil et al.) 2023 Cattelan and Silva 2023]. We attribute this deviation from the intended behavior to the short- comings of the underlying AURC, as we find the desired isolation of classifier performance in our improved metric formulation in Scction[3 c-AURC further inherits the shorteomings of AURC w.r.t monotonicity (R2), ranking interpretability (R3), and error flexibility (R5).
and the Brier Score are technically not thhei-t metrics. Yet, we include them here as they also aim for a holistic performance assessment, i.e. assessment beyond individual working points, by evalu- ating a general meaningfulness of scores based on the softmax-output of the classifier [Ovadia et al.) 2019]. Thereby, they jointly assess ranking and calibration of scores, which dilutes the focus on ranking quality in the context of SC [Täger et al. 2023]. In our formulation, the calibration aspect in these metries breaks the required monotonicity w.I.t SC evaluation (R2). Further, they are not ap- plicable to CSFs beyond those that are directly based on the classifier output (R4). AUROG; The "failure version" of the standard AUROC assesses the correctness of predictions with binary failure label Däger et al. 2023). Based on the AUROC, it provides an intuitive ranking quality assessment. However, ignores the underlying classifier performance (R1, R2), and is restricted to binary error functions (R5). OC-AUC:
NLL Brier Score: Importantly, proper scoring rules such as the Regative-Log-Liklline
(CCAUUC:Kivlichan Kivlichan et al. [202T] introduce the Oracle-Model Collaborative AUC, where first a fixed threshold is applied on the confidence scores, above which error values are set to zero Then, the AUROC (or the Average Precision) are evaluated. This metric is also reported in (Dehghani et al., 2023 Tran al. 2022). with review fraction of 0.5%. OC-AUC is subject to the same pitfalls as A AUROC, (RI, R2, R5). AUROC-AURC: Pugnana and Ruggieri [2023 propose the AUROC-AURC where the Selective Risk is defined as the classification (not failure) AUROC computed over the set of accepted predictions. However, it is only applicable to binary classification models with binary error functions (R5). Further, in employing Selective Risk it inher- its the pitfalls described for A AURC regarding monotonicity (R2) and ranking interpretability (R3). NAURC: Cattelan and Silva 2023] propose the -OrRa, a min-mar scaled version of the C-AURC, and claim that this modification eliminates its lack of interpretability. However, given the linear relationship with the AURC, it does not fulfill the requirements R2 and R3. FI-AUC:
Malinin and Gales (2020, Malinin et al.) (021] introduce the notion of Error-Retention Curves, which corresponds to that of Generalized Risk Coverage Curves. However, Malinin et al. [021] the authors propose an F1-AUC metric which is only applicable to binary error functions (R5) and
5

---

breaks monotonicity (R2), as it decreases with increasing accuracy for accuracy values above z 0.56 (see Appendix A.1.3 for a detailed explanation.) ARC: 2009 Ashukha et al. 2020 Condessa et al. 2017] and the associated AUC directly correspond to the AURC and are therefore subject to the same pitfalls (R2, R3).
Curves [Nadeem al.]
[s]
fnllure contbrrk xID1
AURC inducedd weighting nunking characteristic (AUGRC)
200 (descending confidence scores, 1000)
620
1000
(b)
(6)
coneatt
CSF-1
AUDRD 50
equal rarieing g quaiity equal acouracy
Saloiie Kisk
CSP-1
CSF-2
AUROC 0.5
aczuracy a 0.9
+-UURC
AURC
0.5
CSF-3
rank Jeon high (98)
R2
(6)
0.0
comect
CSF-2
AUGRC = 50
0.0
0.2
0.4
0.6
0.8
1,0
fakure
accuracy - 0.9
AURG 69
Coverage
koont high (Ow)
AUROG in 0.5
Coneralaned Makk
02
CSF-1
CSF-2
CSR3
(d)
worse ranking quality worse accuracy aay
0.1
conecc
CSF-3
AUDRD 110
failure
AUROC
0.11
AURC 169
0.0
0.2
0.4
0.6
0.8
1.0
high
Coverage
The proposed AUGRC metric resolves shortcomings of AURC. All figures are based on rankings of predictions according to descending associated confidence scores induced by CSF. All AURC, e-AURC, and AUGRC values are scaled by x 1000. a) shows the contribution of an individual failure case on the AURC and AUGRC metries depending on its ranking position (for technical details, see Section A.I.I. While AUGRC reflects the intuitive behavior of weighing the failure cases proportional to their ranking position, the AURC puts excessive weight on high- confidence failures. b-d) Toy example of three CSFs ranking 20 predictions to show how AUGRC resolves the broken monotonicity requirement (R2) of AURC. Despite equal AUROC, and equal acc in CSF-1 and CSF-2, the AURC improves. And AURC even improves in CSF-3, which features lower AUROC, and lower acc compared to CSF-1. e-f) The corresponding risk-coverage curves reveal that the non-intuitive behavior of AURC is due to the excessive effect of the high-confidence failure of CSF-1 on the Selective Risk, which is resolved in the Generalized Risk.
Figure 2:
3 Area under the Generalized Risk Coverage Curve
we illustrate that aggregating SC performance across working points requires shift the perspective from the Selective Risk to the Generalized Risk (Equation 3) as an holistic assess- ment of the risk of silent failures for all predictions, before the rejection decision is made. We pro- pose to evaluate SC methods via the Area under the Generalized Risk Coverage Curve (AUGRC). For the binary failure error becomes an empirical estimator of the following expression:
In Section 2.2
AUGRC = 1.'
[ P(Y
1, g(z) 2 dP(g(z) 2T)
(6)
This metric evaluates SC models in terms of the rate of silent failures averaged over working points and thus provides a practical measurement that is directly interpretable. It is bounded to the (0. 1/2] interval, whereby lower AUGRC corresponds to better SC performance. The AUGRC is not subject the shortcomings of the AURC, and we can derive a direct relationship to AUROC; and acc (the derivation and visualizations are shown in Equation 8 and Figure
5)
1
AUGRC = (1 AUROC;) acc (1 (1- + 2
acc)²
To illustrate, when drawing two random samples, it corresponds to the probability that either both are failure cases that one is failure and has higher confidence score than the non noilfrilure. This

---

is reflected by the second and first term of Equation 7 respectively, As an example, a CSF that outputs random scores yields AUROC; 2. and hence AUGRC- ( acc). The AUGRC for an optimal CSF (AUROC; = 1) is given by the second term in Equation 7 hence subtracting the optimal SC performance yields a pure ranking measure re-scaled by the probability of finding a positive/negative pair. This overcomes the lack of interpretability of AURC and e-AURC (R3). Monotonicity (R2) is ensured since the partial gradients w.r.t both AUROC and acc are always negative. Further, it can accommodate arbitrary classification metrics via the error function e (R4) as well as arbitrary CSF (R5).
Figure 2 depicts the metric contribution of individual failure cases depending on their ranking position. This shows empirically how the excessive wver-weighting of high-confidence failures in AURC is resolved by AUGRC, and how the intuitive ranking assessment of AUROC; is established (see Section 1. for a detailed derivation). We further showcase how AUGRC resolves the broken monotonicity requirement (R2) of AURC in Figures 2 b-d. Despite equal AUROC; and equal acc in CSF-1 and CSF-2, the AURC improves. And AURC even improves in CSF-3, which features lower AUROC; and lower acc compared to CSF-1. Figures@eett depict the associated risk-coverage curves and reveal that the non-intuitive behavior of AURC is due to the excessive effect of the high- confidence failure of CSF-1 on the Selective Risk, which is resolved in the Generalized Risk. In we demonstrate implications of AURC's shortcomings on real-world data.
Section 4.2
4 Empirical Study
We conduct an empirical study based on the existing FD-Shifts benchmarking framework [Jäger et al. [2023, which compares various CSFs across broad range of datasets. The focus of our study is not on the performance of individual methods (CSFs) but rather on the ranking of methods based on AUGRC evaluation as compared to AURC. We utilize the same experimental setup as in Jäger
et al. (2023) with the addition of CSF scores based on temperature-scaled classifier logits. Extending the benchmark by recent SC methods, e.g. |Feng et al. 22022) is interesting direction of future
work. A detailed overview of the datasets and methods used can be found in Appendix [A.2] The
code for reproducing our results and PyTorch implementation of the AUGRC are available at: btttp:://githut http://ggttttacooooo////////a//a
4.1 Comparing Method Rankings of AUGRC and AURC
To study the relevance of AUGRC, we investigate the changes induced by AUGRC on rankings of CSFs compared to AURC method rankings.
CSF Rankings with AUGRC substantially deviate from AURC rankings. Figure 3 illustrates the method ranking differences for all i.i.d. test datasets, showing changes in ranks across all CSFs. Notably, AUGRC induces changes ensure that these ranking changes are not due to variability in the test data, we evaluate the metrics on 500 bootstrap samples from the test dataset and derive the compared method rankings based on the average rank across these samples trank-then--gggregatee corresponds to the size of the test dataset. Metric values for each bootstrap sample are averaged over 5 training initializations (2 for BREEDS and 10 for CAMELYON-I7-Wilds) We analyze the robustness of the resulting method rankings for AURC and AUGRC separately, based on statistical significance. To that end, we perform pairwise CSF comparisons in form of one-sided Wilcoxon signed-rank test [Wilcoxon] 1992] on the bootstrap samples. To control the family-wise error rate at a 5% significance level, we apply the Holm correction for multiple testing per metric and dataset, following |Wiesenfarth et al. [021] (see Figure 7 for the results without correction for multiple testing). The resulting significance maps displayed in Figure 3 both AURC and AUGRC. This suggests that the observed ranking differences are induced by the conceptual difference of AUGRC compared to AURC and, as consequence, that the shortcomings of current metrics and respective solutions discussed in Section 2 are not only conceptually sound, but also highly relevant in practice. For the results in Figure we select the DeepGamblers reward parameter and whether to use dropout for both metrics separately on the validation dataset (details in Table [ and Table 2).
the top-3 methods (out of 13) on 5 out of 6 datasets. To
The size of cach bootstrap sample
indicate stable method rankings for
The shortcomings of AURC affect CSF comparison across datasets and distribution shifts. The method rankings for all datasets and distribution shifts, based on the original test datasets, are shown
7

---

in Table which also includes results for equal hyperparameters for AURC and AUGRC. Results for the bootstrap-base based method ranking analysis for distribution shifts are displayed in Figure The substantial differences in method rankings across all datasets and distribution shifts underline the relevance of AUGRC for SC evaluatiion. The complete AUGRC results on the FD-Shifts benchmark are shown in Table K
In few cases, as shown in Figure we observe that CSF A ranked worse than a neighboring CSF B even though CSF A's ranks are significantly superior to those of CSF B based on the Wilcoxon statistic over the bootstrap samples. This discrepancy can occur the rank variability in CSF A is larger than in CSF B. While this does not affect our conclusions regarding the relevance of the AUGRC, it indicates that selecting a single best CSF for application based solely on method rank- ings should be approached with caution. We recommend closer inspection of the individual CSF performance in such cases, although this not the focus of our study.
BREEDS EREEDD-Entity-33 13
CIFAR-10
AURC
CSF rankings
AUGRC
AURC
CSF rankings
AUGRC
13
12
4
CAMELYON-17-48
iWildCam
AURC
CSF rankings
AUGRC
AURO
CSF rankings
AUGRC
10
CIFAR-100
SVHN CSF rankings
AURO
CSF rankings
AUGRC
AURC
AUGRC
C 01 Ae h)
00
3
10
11:0110
3 10 11
12 30 5
CSFs:
MSR
MLS
MCD-MSR
DG-TEMP-MLS
DG-Res
ConfidNet
MCD-EE
PE
TEMP-MLS
MCD-PE
DG-MCD-MSR
DG-PE
Devries al.
ranks
Substantial differences in method rankings for AUGRC and AURC. On 5 out of datasets, the top-3 CSFs (out of 13 compared methods) change when employing the proposed AUGRC instead of AURC. This demonstrates the practical relevance of the AUGRC metric for Se- lective Classification evaluation. CSFs are color-coded and sorted from top (best) to bottom (worst) by average rank based 500 bootstrap samples from the test dataset to ensure ranking stability. Ranking changes are reflected changes in the color sequence and highlighted by red arrows. We assess the stability of the method rankings for each metric individually using one-sided Wilcoxon signed-rank tests based on the bootstrap samples at 5% significance level with adjustment for multi- ple testing according to Holm. Adjacent to each ranking, we present the resulting significance maps for the pairwise CSF comparisons. These maps can be interpreted as follows: At each grid position w). filled entries indicate that metric values of CSF y are ranked significantly better than those from CSF æ (across bootstrap samples), cross-marks indicate no significant superiority. An ideal ranking exhibits only filled entries above the diagonal.
Figure 3:
8

---

4.2 Showeasing Implications of AURC Shorteomings on Real-world Data
Figure gives an in-depth look at how the conceptual shortcomings of AURC affect method assess- ment on real-world data. The example uses the confidence like reservation score of the DeepGam- blers method (DG-Res) [Liu et al. 2019] and the Monte Carlo Dropout-base based predictive entropy (MCD-PE) [Gal and Ghahramani 2016 as CSFs on the CIFAR-10 test dataset. Despite DG-Res having higher classification performance and ranking quality than MCD-PE, the AURC erroneously favors MCD-PE over DG-Res, This violates the monotonicity requirement (R2). This can be at- tributed the excessive contributions of only few high-confidence failures, which aligns with the theoretical findings failure contributions shown in Figure 2 (R3). In this example, the high- confidence failures are associated with high label ambiguity or incorrect labeling, suggesting that the AURC may exacerbate the influence of label noise in practice.
(a)
(b)
DG-Res
DG-Res
MCD-FF
truck
truck
truck
CT.
Pred: automobile Pred: automoobile Pred: automobile
Pred: frog
AUROC; 0.9340
AUROC: 0.9338
Confid: 1.0000
Confid: 0,9999
Cordid: 0.5998
Confid: 0.9998
Sclociee Rok
Accuracy = 0,950
Accuray D.946
AURO-
AURO
0.01
MCD-PE
GT cal
GE cat
GE truck
GT. truck automobile
0.00
frog Confidt: 0.9896
dog
Pred: autuenobile Corfid: 0.9891
Confid:: 0.9892
Confid: 0.9889
Generalbod Beck
0.44-
0.02
000
0.
02
0166
0.8 1.0
Coverage
Figure 4: The conceptual shortcomings of AURC affects method assessment in practice. We illustrate the practical effects of excessive weight high-confidence failures in AURC by comparing the performance of two CSFs, DG-Res and MCD-PE, on the CIFAR-10 test dataset. (a) shows the coverage curves based on Selective Risk and Generalized Risk for both CSFs. The AURC violates the monotonicity requirement (R2) in practice, favoring MCD-PE despite lower classification per- formance and ranking quality compared to DG-Res. (b) displays the images associated with the top-k high-confidence failures. For DG-Res, the four failures correspond to the first four peaks in the Selective Risk curve, up to coverage 0.27 (the total number failures 446). Only a few high-confidence failures significantly increase the AURC. For both CSFs, the images associated with high-confidence failures exhibit high label ambiguity or are incorrectly labeled, indicating that the AURC may amplify the influence of label noise in practice. AURC and AUGRC values scaled by x 1000.
Conclusion
Despite the increasing relevance SC for reliable translation of machine learning systems to real- world application, we find that the current metrics have limitations in providing the comprehensive assessment needed to guide the methodological progress of SC systems.
In this work, we establish systematic SC evaluation framework, thereby promoting the adoption of more comprehensive, interpretable, and task-aligned metrics for comparative benchmarking of SC systems.
We find that none of the existing multi-threshold metrics, particularly the AURC, meet the key re- quirements we identified for comprehensive SC evaluation, leading to deviations from intended and intuitive performance assessment behaviors. To address this, we introduce the AUGRC as a suit- able metric for comprehensive SC method evaluation. Substantial differences in method rankings

---

between AURC and AUGRC, demonstrated through extensive empirical studies, highlight the im- portance of selecting the right SC metric. Thus, we propose the adoption of AUGRC for meaningful SC performance evaluation.
When evaluating specific applications for which certain risk coverage intervals known to be ir- relevant, adaptations such as a partial AUGRC (analogous to the partial AUROC) may be considered. Further, investigating the relation between confidence ranking and calibration and on evaluating SC in settings where calibrated scores are of interest is an interesting direction of future work. Overall, our proposed evaluation framework provides a solid basis for future work in Selective Classification, including developing novel methods as well as analyzing the properties of individual methods.
Acknowledgments and Disclosure of Funding
This work was partly funded by Helmholtz Imaging (HI), platform of the Helmholtz Incubator on Information and Data Science. We thank Lukas Klein, Maximilian Zenk and Fabian Isensee for insightful discussions and feedback.
References
Krishnamurthy Dvijotham, Jim Winkens, Melih Barsbey, Sumedh Ghaisas, Robert Stanforth, Nick Pawlowski, Patricia Strachan, Zahra Ahmed, Shekoofeh Azizi, Yoram Bachrach, et al. Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity-diive deferral to clinicians, Nature Medicine, 29(7): 1814-1820, 2023.
Christian Leibig, Moritz Brehmer, Stefan Bunk, Danalyn Byng. Katja Pinker, and Lale Umutlu. Combining the strengths of radiologists and ai for breast cancer screening: a retrospective analysis. The Lancet Digital Health, 4(7))e507 c519, 2022.
Karin Dembrower, Erik Wählin. Yue Liu, Mattie Salim, Kevin Smith, Peter Lindholm. Martin Eklund, and Fredrik Strand. Effect of artificial intelligence-base triaging of breast cancer screening mammograms on cancer detection and radiologist workload: retrospective simulation study. The Lancet Digital Health, 2 (9):c46--c744 2020.
Adam Yala, Tal Schuster, Randy Miles, Regina Barzilay, Constance Lehman. deep learning model to triage screening mammograms: a simulation study. Radiology, 293(1):38-46, 2019.
Yonatan Geifman and Ran El-van Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Chi-Keung Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, (4):247-254, 1957.
Ran El-Yaniv al. On the foundations of noise-free selective classification. Journal of Machine Learning Rexearch, 11(5), 2010.
Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. Advances in Neural Information Processing Systems, 32, 2019.
Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning, pages 2155121199. PMIR, 2019.
Lena Maier-Hein, Bjoemn Menze, et al. Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv. org, (2206.01653), 2022.
Yonatan Geifman, Guy Uziel, and Ran EI- Yaniv. Bias-reduced uncertainty estimation for deep neural classi- fiers. arXiv preprint /oXiv:/805.0020 2018.
Filipe Condessa, José Bioucas-Dias and Jelena Kovačević. Performance measures for classification systems with rejection. Pottern Recognition, 63:437-450, 2017.
Paul F Jäger, Carsten Lüth, Lukas Klein, and Till Bungert. A call to reflect evaluation practices for failure detection in image classification. In ICLR 2023, 2023.
10

---

Till J Bungert, Levin Kobelke, and Paul F Jaeger. Understanding silent failures in medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 400- 410. Springer, 2023.
Zhen Cheng, Xu- Yao Zhang, and Cheng-Lin Liu. Unified classification and rejection: A one-versus-all frame- work. arXiv preprint arXiv:231 1.,355, 2023.
Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-Lin Liu. Openmix: Exploring outlier samples for misclassifi- cation detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023a.
Neeraj Varshmey, Swaroop Mishra, and Chitta Baral. Towards improving selective prediction ability of nlp systems, arXiv preprint arXiv:2008.09.17 2020.
Junayed Naushad and ID Voiculescu. Super-trustscore: reliable failure detection for automated skin lesion diagnosis. 2024.
Jordy Van I Landeghem, Sanket Biswas, Matthew Blaschko, and Marie-Francine Moens. Beyond document
page classification: Design, datasets, and challenges. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2962-2972, 2024.
Jordy Van Landeghem, Rubên Tito, Lukasz Borchmann, Michal Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Valveny, et al. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17528-19540, 2023.
Fei Zhu, Zhen Cheng, Yu-Yao Zhang, and Cheng-Lin Liu. Rethinking confidence calibration for failure pre- diction. In European Conference on Computer Vision, pages 518-536. Springer, 2022.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lnguage Processing (Volume 1: Long Papers), pages 1040-1051, 2021.
Hiyori Yoshikawa and Naoaki Okazaki. Selective-lama Selective prediction for confidence-awee aware evaluation of language models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2017-200 2028, 2023.
Yukun Ding. Jinglan Liu, Jinjun Xiong, and Yiyu Shi. Revisiting the evaluation of uncertainty estimation and its application explore model complexity-uncertane uncerraintty trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 4-5, 2020.
Fei Zhu, Yao Zhang, Zhen Cheng, and Cheng-Lin Liu. Revisiting confidence estimation: Towards reliable failure prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023b.
Ido Galil and Ran El-Yaniv. Disrupting deep uncertainty estimation without harming accuracy. Advances in Neural Information Processing Syxtems, 34:21285-21296 2021.
Vojtech Franc, Daniel Prusa, and Vaclav Voracek. Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24(11):1 2023.
Jun Cen, Di Luan, Shiwei Zhang, Yixuan Pei, Yingya Zhang, Deli Zhao, Shaojie Shen, and Qifeng Chen. The devil is in the crongly-classifedd samples: Towards unified open-set recognition. arXiv preprint arXiv:2302.04000 2023.
Guoxuan Xia and Christos-Saveer Bouganis. Augmenting softmax information for selective classification with out-of-distribution data. In Proceedings of the Asian Conference on Computer Vision, pages 1995-2012, 2022.
Luis Felipe Prates Cattelan and Danilo Silva. How to broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. 2023.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang. Zelda Mariet, Huiyi Hu, el al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint
2022.
Jihyo Kim, Jiin Koo, and Sangheum Hwang. A unified benchmark for the unknown detection capability of deep neural networks. Expert Systems with Applications, 229:120461, 2023.
11

---

Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dimityy Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling deep learning. arXiv preprint urXiv:2002.06470 2020.
Guoxuan Xia, Olivier Laurent, Gianni Franchi, and Christos-Savvas Bouganis. Understanding why label 2024.
smoothing degrades selective classification and how to fix it. arXiv preprint arXiv:2403. 14715, 2024.
Ido Galil, Mohammed Dabbah, and Ran EI- Yani.. What can we learn from the selective prediction and uncer- tainty estimation performance of 523 imagenet classifiers. arXiv preprint arXiv:2302. 1874, 2023.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Bal-
aji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances neural information processing systems, 32, 2019.
Ian Kivlichan, Zi Lin, Jeremiah Liu, and Lucy Vasserman. Measuring and improving modee-tr collaboration using uncertainty estimation. arXiv preprint urXiv:2107.04212 2021.
Mostafa Dehghani. Josip Djolonga, Basil Mustafa, Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Pe- ter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
Andrea Pugnana and Salvatore Ruggieri. Auc-based selective classification. In International Conference on Artificial Intelligence and Statistics, pages 2494-2514. PMLR, 2023.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv: 2020.
Andrey Malinin, Neil Band, German Chesnokav, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkow, Vatsal Raina, al. Shifts: A dataset of real dis- tributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455. 2021.
Malik Sajjad Ahmed Nadeem, Jean-Daniel Zucker, and Blaise Hanczar. Accuracy-rejection curves (arcs) for comparing classification methods with reject option. In Machine Learning Systems Biology, pages 65-81. PMLR, 2009.
Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir Abdi. Towards better selective classi- fication. arXiv preprint arXiv:2206.0000 2022.
Frank Wilcoxon. Individual comparisons by ranking methods. Breakthroughs statistics: Methodology and distribution, pages 196-202. Springer, 1992.
Manuel Wiesenfarth, Annika Reinke, Bennett A Landman, Matthias Eisenmann, Laura Aguilera Saiz, M Jorge Cardoso, Lena Maier Hein, and Annette Sopp-Schncider. Methods and open-source toolkit for analyzing and visualizing challenge results. Scientific reports, 11(1):2369, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059, PMLR, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference machine learning, pages 1321-1330. PMLR, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian Approximation: Representing Model Uncertainty in Deep Learning.
Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philippe Morency, and Masahito Ucda. Deep Gamblers: Learning to Abstain with Portfolio Theory.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction learning model confidence. Advances in Neural Information Processing Systems, 32, 2019.
Terrance DeVries and Graham W Taylor. Learning confidence for out-o--distributiin detection in neural net-
works, arXiv preprini arXiv:1802.04880 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning. volume 2011, page Granada, Spain, 2011.
Alex Krizhevsky, Leamingg Multiple Layers of Features from Tiny Images.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N. 7(7):3, 2015.
12

---

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang. Akshay Balsubramani,
Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the- wild distribution shifts. International conference on machine learning, pages 5637-5664. PMLR, 2021.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds:
Benchmarks for subpopulation shift.
arXiv preprint arXiv:2008,0483 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 19t:::9933.220 12261, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
A Appendix
A.1 Technical Details
A.1.1 AURC Failure Contribution
Let (Te) provide total order on the predictions (relaxing the assumption to partial order leads to more complicated but very similar formulation). Then, for rejection threshold Tr, a failure case at rank contributes to the Selective Risk (Equation 2 by 1 if S Averaging over all thresholds yields contribution through the failure at rank 1. to the AURC of & E +. For the Generalized Risk, the contribution at t* < t is always 1 resulting in an overall contribution of N-t"-1 to the AUGRC. The failure contribution curves are displayed in Figure 2
A.1.2 Relationship between AUGRC and failure AUROC
Let Y = [I(m(x) + y) be binary indicator variable for classification failures. Then, the General- ized Risk corresponds to the joint probability that a sample is accepted and wrongly classified. We
write the AUGRC as follows:
AUGRC- = P(Y = 12](x),)(x) 2 2)
(1 (
/ P(g(x) 2 -|y; = 0)P = d))
2
(8)
1
acc (1 acc) / P(g(a) 2 +|Y = dP -- 1)]
2 2
acc acc + - (1-acc) AUROCI
2
2
1
AUROC;) acc(1 acc) + 7 acc))
At (+), we use that P(g(x) r) P(g(x) 2 T 0) acc P(g(2) 2 - 1). -1- On choosing two random samples, the second term the final equation represents the probability that both are failures and the first term represents the probability that one is failure, the other is not, and that the failure has higher confidence score than the other.
The AUGRC is monotonic in acc and AUROC; as both partial derivatives are negative Vacc (0, 1), AUROC, E [0, 1]:
8AUGRC
@AUROC -acc (1 acc)
(9)
ĐAUGICC 2 AUROC, acc acc AUROC,
(10)
Racc
13

---

1.0
Selective Risk
Generalized Risk
0.8
0.375
0.6
0.300
Accuracy
AUGRC
Rösk
0.225
0.4
0.4
0.150
02
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.000
Coverage
AUROC
Figure 5: Visualization of the relationship between AUGRC and AUROC, (a) The Selective Risk curve be transformed into the Generalized Risk curve via multiplication by the respective coverages. The resulting curve is monotonically increasing and bounded by the diagonal; decreas- ing Selective Risk corresponds to plateau in Generalized Risk. The AUGRC corresponds to the AUGRC of an optimal CSF (shaded red) plus the re-scaled AUROC₁ (shaded in green). The AUROC; corresponds to the fraction of the area (parallelogram) enclosed by the green dashed line that lies above the Generalized Risk curve. (b) AUGRC (color-coded) and its negative gradients (arrows) in the Accuracy-AUROC, space.
A.1.3 FI-AUC does not fulfill R2
For the optimal CSF, all failure cases are attributed to lower confidence scores than correct predic- tions. Hence, the precision P 0lg(x) 2 is one for coverages up to P(Y; = 0) and decreases as P(Y, 0)/ (a)( 2 7) above. Following the task formulation as defined in Malinin et al. 2021 we can derive the following analytical expression for the optimal FI-AUC*:
[ F1(T) 2 T)
FI-AUC*
P(V=0)
2. P(g(x) 2)
dP(g(z) 2T)
P(Y 0) + P(g(x) TT
(11)
2 P-Y 0)
P(Y=0) P(Y, = 0) + P(g(x) dP(g(z) 21)
2 In 1
1
acc
F1-AUC" increases with increasing accuracy up to P(Y; 0) ~ 0.56, then it decreases, favoring models with lower classification performance. Thus, the FI-AUC does not fulfill the monotonicity requirement (R2).
A.2
Experiment Setup
A.2.1 Datasets and Methods
We compare the following CSFs: From the classifier's logits we compute the Maximum Softmax Response (MSR), Maximum Logit Score (MLS), Predictive Entropy (PE), and the MLS based on temperature-scaal
logits, for which scalar temperature parameter is tuned based on the validation set Guoo et al.] 2017]. Three predictive uncertainty measures are based on Monte Carlo Dropout (Gal and Ghahramani]: mean softmax (MCD-MSR), predictive entropy (MCD-PE), and mutual formation (MCD-MI). We additionally include the DeepGamblers method Liu et al. which learss
14

---

confidence like reservation score (DG-Res), ConfidNet Corbière et al. 2019 which is trained as an extension the classifier, and the work VeVisss and Taylor 20188
We evaluate SC methods on the FD-Shifts benchmark [Jäger et al. 2023 which considers broad range of datasets and failure sources through various distribution shifts: SVHN (Netzer al. 2011],
CIFAR-10 10, and CIFAR-I00 [Krizhevsky] are evaluated on semantic and non-semantic new-class
shifts in rotating fashion including Tiny ImageNet [Le and Yang 2015). Additionally, we con-
sider sub-class shifts on iWildCam Koh et al. 2021]. BREEDS-Enit-
[Santurkar et al.] 2020].
on CIFAR-100 (based on super-classes) as well as corruption shifts based on Hendryckks and Dietterich [2019] on CIFAR-I00, The data preparation
CAMELYON-17- 1--N-1 [Koh et al., 2021],
and splitting are done as described in Däger al. [2023] For the corruption shifts, we reduce the test split size to 75000 (subsampled within each corruption type and intensity level).
The following classifier architecture used in the benchmark: small convolutional network for SVHN, VGG-13 [Simonyan and Zisserman, 2014) on CIFAR-10/100, ResNet-50 [He et al.) 2016 on the other datasets.
the distribution shift not mentioned explicitly, we evaluate on the respective i.idd test datasets.
Our method ranking study focuses on the evaluation of CSF performance based on the existing FD-Shifts benchmark, hence we required no GPU's for the analysis Section. $ As both AURC and AUGRC can be computed efficiently. (CPU) evaluation time for single test set is less than minute; evaluation on 500 bootstrap samples on single CPU core take around hours.
A.2.2 Hyperparameters and Model Selection
The experiments are based on the same hyperparameters reported Table 4 in Jäger al. [2023] Based on the performance on the validation set, we choose the DeepGambler reward hyperparam- eter and whether to use dropout (for non-MCD-based CSFs). For the former, we select from [2.2, 3, 6, 10] Wilds-Camelyon-I11 CIFAR-10, and SVHN, from [2.2, 3, 6, 10, 15) on iWildCam and BRREDDS-Entty- selection based the AURC metric, obtain the same configurations reported in Jäger et al. [2023]. When performing model selection based on the AUGRC metric, we obtain the parameters reported in Tab. T and Tab. For temperature scaling, we optimize the NLL on the validation set using the L-BFGS algorithm with 0.01.
and from [2.2, 3, 6, 10, 12, 15, 20] on CIFAR-100. When performing model
Table 1:
Selected DeepGambler reward hyperparameter based on the AUGRC on the validation set for confidence scoring functions trained with the DeepGamblers objective.
Method
iWildCam
BREEDS Entitsy--
Wilds-Camelyont
CIFAR-
CIFAR- 10
SVHN
MG-MCD-MSR
20
3
DG-PE
2.2
10
3
DG-Res
2.2
12
2.2
2.2
DG-TEMP-MLS
15
2.2
15
3
Table 2: Whether or not dropout-trainin has been selected based on the AUGRC the validation set. This selection is only done for deterministic confidence scoring methods (no MCD). "1" denotes dropout training and "0" denotes training without dropout.
Method
WWilCCam
BREEDS-EaEE Eati00---tttt133 Wilds-Camelyon-11
CIFAR-100
CIFAR-I -0
SVHN
MSR
0
MLS
1
PE
0
0
ConfidNet
0
1
DG-Res
I
0
Devries et al.
0
0
TEMP MLS
0
0
DG-PE
0
0
DG- TEMP-MIS
0
O
0
15

---

A.3 AUGRC Computation Time
The main bottleneck of the AUGRC computation is the sort operation on the confidence scores. This is the same for the AURC and there is no computational overhead compared to the AURC. For small benchmark of the computational time, we evaluate the AURC, AUGRC, and AUROC; on 1000 random scores and failure labels. We obtain the following results (averaged over 5000 cases): 564μs + 2,s (AURC), 562 + 2us (AUGRC), 730μe 13js (AUROC)).
A.4 Additional Results
Table shows the AUGRC results for the 13 compared CSFs across all datasets and distribution shifts. While the MSR baseline is not consistently outperformed across all experiments by any of the compared CSFs, MCD improves MSR scores in most settings. Temperature scaling does not exhibit consistent improvement of MLS scores.
Table 3: FD-Shifts benchmark results measured as AUGRC x10 (score range: [0, 500], lower is better 4). The color heatmap is normalized per column, whereby whiter colors depict better scores. "cor" is the average over 5 intensity levels of image corruption shifts. AUGRC scores are averaged over 10 runs on CAMELYON- 1I--LON----Widd over runs on BREEDS, and over 5 runs on all other datasets. Abbreviations: nes: new-class shift (s for semantic, ns for non-semantic), iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR-10/100, ti: Tiny Imagenett...
iWildCam
BREEDS
CAMELYON
CIFAR-100
CIFAR-I0
SVHN
study
lid sub
sub
iid
sub
sub
DOC
5-aCS
25 BCS
iid
COT
5-DCS
D2-DDS
nos-data
c10
syhn
c100
s9hn
c10 c100
MSR
48.0
55.0
6.86
117
8.13
91.3
54.6
150
187
210
334
2/11
4.95
67.9
166
291
154
3.44
48.5
4A
48.5
MLS
62.4
9:28
121
8.13
91.3
65.6
155
213
337
195
5.56
68.1
163
288
150
3.97
46.5
46.1
46.6
48.0
MCD-MSR
42.8
54.3
6.89
116
3.64
95.1
53.1
146
176
210
316
2012
4.56
60.4
166
204
1.5
3.40
47.5
47.4
47.9
MCD-PE
55.1
6.98
5.64
95.1
56.6
146
210
335
198
4.73
60.6
161
293
153
3.47
46.3
46.1
46,9
MCD-MI
47.7
ConfdNNr
82.0
91.2
6.89
117
4.36
86.1
57.55
1.54
192
2144
340
2000
4.701
65
165
201
1.5
3.43
48.6
4R
48.8
DG-MCD-MSR
43.9
59.6
6.31
112
3.47
149
12.1
145
175
2110
9.66
2011
4.75
60.8
168
2066
1.59
3.38
47.6
47..7
48.0
Devties et al.
61.7
74.7
8.24
120
24.3
145
69.6
150
214
202
3442
211
4.57
70.8
166
287
152
5.56
48.3
48.5
49.9
TEMP-MIS
48.2
55.1
6.86
116
8.13
91.3
548
1.50
187
210
333
2013
4.92
67.4
166
291
1.5
3.44
4%
47.
48.44
DG-PF
6.33
55.2
28H
DG-TEMP MLS
52.2
69.9
6.35
114
3.46
123
55.0
222
184
211
199
4,09
63.1
167
288
3.41
49.0
48.5
48.8
16

---

Table 4: Comparing Rankings of AURC a versus AUGRC - 3. Differences in the method rankings between AURC and AUGRC demonstrate the relevance of the pitfalls of the AURC discussed in Section 2.4 Upper half: Model selection for dropout and DG hyperparameter was done for both metrics separately. Lower half: Model selection was done based on AUGRC for both and B. The selected hyperparameters are reported in Appendix A.2.2 The color heatmap is normalized per column, whereby whiter colors depict better scores. "cor" is the average over intensity levels of image corruption shifts. AUGRC scores are averaged over 10 runs on CAMELYON-I7-WIT non-semantic), iid: independent and identically distributed, sub: sub-class shift, cor: image corruptions, c10/100: CIFAR-10/100, Tinylmagenet.
over 2 runs on BREEDS, and over 5 runs on all other datasets. Abbreviations: nes: new-class shift (s for semantic, ns for
(WildCam
BREEDS
CAMELYON
CIFAR-100
CIFAR-I0
SVHN
studl
suth
i8
suh
iid
suh
idd
sub
DY
S-R0X
SA-RES
idd
Dos data set
cI0
sYan
C100
sYha
C1U
C100
metiie
02
optimized (wai dropout reward parametes)
MSR
L1
30
12
12
11
11
30
MI.S
00
10
13
13
11
12
13
12
11
PC
10
12
10
MCD MSR
MCD
MCD-ER
DG-MCD-MISR
10
TEMP-MES
10
DG-FF
DG-TEMP-MIL
AUGRC
dropout reward parmmeter)
MSR
L1
12
11
6
30
MLS
10
10
13
12
13
12
13
12
10
B1
12
12
1O
30
31
13
12
31
10
30
6
6
MCD-MSR
10
12
10)
S
MCD PC
2
2
E1
MCD EB
5
5
ConRdNet
23
13
133
13
20
DO
DO
20
TO
TO
20
11
11
12
12
30
10
DG-MCD-MSR
12:
11
I1
10
13
13
DG-Res
12
12
12
12
13
12
13
12
Devries
12
13
13
13
13
13
13
13
13
13
13
TUMP MLS
LO
10
8
DG-TEMP-MEL

---

CIFARI-10 CIFAR.10 p-noo
CIFAR-100 sub-class shitt
CIFAR-10 CIFAR &
BREEDS- EEDDD--- sub-class shift
AURC
CSF ranidngs
AUGRC
AURC
rankings
AUGRC
AURG
CSF rankings
NURC
CSP rankings
AUGRC
19111122
12
DIFAR-100 Teyimaginnt (na-nea))
SVHN CIFAR-10
CIFAR-10 Tinfmmggnnt
INldCam-
shift
rankings
rarkings
rankings
AUGRC
CSF rankings
AUGRC
CIFAR-100 - SVHNN (n5-noo)
SVHN CIFR-100 100 (ns-ncs)
CIFAR-10 -+ comuptionn shift rankings
CAMELYON--7-WIA sub-dess shift
CSF rankings
AUGRO
CSF rankings
AUGRC
CSF rankings
18
CIFAR-100 compption shift
BWHN - Thyyiaggnnt (15-n05)
CIFAR-10 SVHN (ns-ncs)
AURC
CSF rankings
AUGRC
AURC
CSF rankings
AUGRO
rankings
CSFx
MSR
DG-TEMP-MLB
MLS
DG-MCD-MSR
PF
DG-Rws
DG-PE
MCD-EE
MCD-PE
beed ranker
Figure 6: Method ranking analysis for distribution shifts. This figure displays the results for the method ranking analysis for AURC and AUGRC for evaluation under distribution shifts, analogous to Figure 3 The subfigures are titled as follows: "training dataset" "evaluation dataset". Details on the distribution shifts are given in Section A CSFs are color-coded and sorted from top (best) to bottom (worst) by average rank based on 500 bootstrap samples from the test dataset to ensure ranking stability. Ranking changes are reflected in changes in the color sequence and highlighted by red arrows. We assess the stability of the method rankings for each metric individually using one-sided Wilcoxon signed-rank tests based on the bootstrap samples at 5% significance level with adjustment for multiple testing according to Holm. Adjacent to each ranking, we present the resulting significance maps for the pairwise CSF comparisons. These maps can be interpreted as follows: At each grid position (I, y). filled entries indicate that metric values of CSF y are ranked significantly better than those from CSF (across bootstrap samples), cross-man marks indicate no significant superiority. Abbreviations: nes; new-class shift (s for semantic, ns for non- non-semantic).

---

BREEDS-Entiyy rankings
CIFAR-10
AURC
AUGRC
AURC
CSF rankings
AUGRC
30
10121
120160
11
10
GAMELYON- 17-Wilds
iWlldCam
AURO
CSF rankings
AUGRC
AURC
CSF rankings
AUGRC
12
19
13
13
12
12
2
5
12
12
CIFAR-100
SVHN
CSF rankings
AUGRC
AURC
CSF rankings
AUGRC
01
00000
CSFs:
MSR
MLS
DG-TEMP-MLS
ConfidNet
MOD-EE
PE
TEMP-MLS
MCD-PE
DG-MCD-MSR
DG-PE
Devries al.
tied ranks
Figure 7: Method ranking analysis without correction for multiple testing. contrast to the results shown in Figure which include the Holm correction for multiple testing, this figure displays the results without correction. On 5 out of 6 datasets, the top-3 CSFs (out of 13 compared methods) change when employing the proposed AUGRC instead of AURC. This demonstrates the practical relevance of the AUGRC metric for Selective Classification evaluation. CSFs are color-coded and sorted from top (best) bottom (worst) by average rank based 500 bootstrap samples from the test dataset to ensure ranking stability. Ranking changes are reflected in changes in the color sequence and highlighted by red arrows. We assess the stability the method rankings for each metric individually using one-sided Wilcoxon signed-rank tests, each with a 5% significance level, based on the bootstrap samples. Adjacent to each ranking, we present the resulting significance maps for the pairwise CSF comparisons. These maps can be interpreted as follows: At each grid position (x, y), filled entries indicate that metric values of CSF y are ranked significantly better than those from CSF x (across bootstrap samples), cross-marks indicate no significant superiority. An ideal ranking exhibits only filled entries above the diagonal.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The described pitfalls of existing metrics affect Selective Classification evalu- ation in general. Hence, thoroughly discuss all common metrics in the field and demon- strate the relevance on a state-of-the-an benchmark.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The empirical study is based on an extensive, ofate-of-the-arr, yet naturally limited benchmark. However, the scope is clearly defined in the paper.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "L.mmitttoos" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification:
For the analytical relationships, we provide the assumptions and derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The evaluation protocol as well as all experiment setups are described. Addi- tionally, we provide the code for reproducing the results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
also be provided via
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code for reproducing the experimental results published on github. All datasets used are freely accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ww.geeeeeeeeeeubbmm.mm Comeeuumieieeeeeeceeeeeeo ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The details of the experiments setups are described in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
We validate the significance of our results via statistical tests based on boot- strapping.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g,, Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are mentioned in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aarr....c.ooo......ccttta
Answer: [Yes]
Justification:
There are no conflicts with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ye]]
Justification: We don't see any potential negative socictal impacts. We address enhanced reliability of real-world decision systems through the studied methods as a potential posi- tive impact. However, we propose no new Selective Classification method in our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our experiments do not involve any new datasets or classification models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We added proper credit for all datasets and models used in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We provide the code for our proposed evaluation protocol including an im- plementation of our proposed metric. Documentation is added for reproducibility.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer:[NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "4jdwRbIAUC", "reviewer": "Reviewer_BBe8", "summary": "The paper tackles the problem of Selective Classification (SC). The authors show problem with the existing metrics used in evaluating SC and propose to use the Area under the Generalized Risk Coverage curve (AUGRC). Empirical results provide useful insights about the effect of using this metric and shows how this changes  the relative ordering of state-of-the art methods.", "questions": "See the previous section", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem tacked is very important, especially for the safe deployment of ML models which requires the knowledge of when not to trust the model.\n- The proposed metric is well-motivated.\n- The experiments are extensive.\n- It is important for the community to know about this work as this changes the perception of SC and the best method to use.", "weaknesses": "- It is important to add the details of the methods in the main paper. For example, DeepGamblers (DG) is referred to as DG in the main paper and this abbreviation is only explained in the appendix. Similarly for other baselines.\n- Although the experiments are extensive, it is good to include the recent state-of the art SC methods such as Feng et al (2023) to understand which method is the best to use.\n\nReferences:\nFeng et al: Towards Better Selective Classification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:\n\n---\nW1. “It is important to add the details of the methods in the main paper. For example, DeepGamblers (DG) is referred to as DG in the main paper and this abbreviation is only explained in the appendix. Similarly for other baselines.”\n* Thank you for pointing out that the abbreviations DG and MCD-PE were used in the main text without reference to the explanation in the appendix. This was particularly the case in Section 4.2, where we explicitly compare these two confidence scoring functions. In response to your feedback, we adapted Section 4.2 to properly introduce the abbreviations DG and MCD-PE.\n* While the main paper is now fully self-contained and all abbreviations are introduced, we would like to comment on the general decision to provide the details of utilized confidence scoring functions in the appendix. This is because the focus of our work is not on the confidence methods like in other studies, but on the metric, which is why we find it important to allocate sufficient space in the main paper to metric-related descriptions.\n\n---\nW2. “Although the experiments are extensive, it is good to include the recent state-of the art SC methods such as Feng et al (2023) to understand which method is the best to use.”\n* Please note that the focus of our experiments is to show the relevance of the proposed metric rather than the performance of individual methods. We argue that this metric relevance can be demonstrated with a diverse and representative set of prevalent confidence scoring functions, but it is not necessary to include all existing methods.\n* The main conclusion in [2] is that “[...] selecting according to classification scores is the SOTA selection mechanism for comparison” (based on experiments with three SC methods, including DG). We want to point out that with DG-MCD-MSR, DG-PE, and DG-TEMP-MLS, our empirical evaluation already includes multiple CSFs that are trained with DG loss attenuation but are based on the classifier’s logits, aligning with the recommendation in [2].\n* We agree that testing and comparing further novel methods with our proposed AUGRC metric is an interesting direction for future work. However, running all experiments required for this method within one week is not feasible for us.\nIn response to your feedback, we extended the first paragraph in Section 4 to include the method of Feng et al. in the discussion about future work.\n\n[2] Feng et al., “Towards better Selective Classification”\n\n---\nThank you once more for your constructive feedback. As we believe to have resolved your comments, please let us know in case you have remaining suggestions to further increase the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yXyCIBQ5CJ", "reviewer": "Reviewer_xFt7", "summary": "The authors propose 5 requirements that should be satisfied by selective classification (SC) metrics such that they can be successfully used to rank SC models for a task. They then propose a new metric, called AUGRC, which is shown to satisfy all 5 requirements. Finally, the authors show empirically that their method performs better rankings thank previous metrics.", "questions": "1. A similar task to SC is classification with rejection of OOD samples (sometimes called open-set recognition). In such cases, there isn't a ground truth to evaluate the risk on (in practice). Can the authors envision a way to adapt their metric to these scenarios?\n2. The metrics were scaled by ×1000 in Figure 2. Were they also rescaled for all the other results?\n3. Regarding interpretability, the authors say that \"The optimal AUGRC is given by the second term in Equation 7\". What does the optimal AUGRC mean? What would be the AUGRC of the Bayes-optimal model for a task? Is there an AUGRC value associated to a random classifier, as in AUROC?\n5. Any intuitions about why the rankings of AURC and AUGRC where the same for iWildCam?", "limitations": "Yes.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-motivated, clearly written, and has an excellent coverage of previous works.\n2. The 5 requirements were well-chosen and a metric that satisfies them is likely to be a good choice for SC ranking.\n3. The metric is simple, yet effective, and the maths seems sound.\n4. Results are convincing and statistically analysed; the toy dataset was didactic.\n5. There's enough information in the main body of the paper to fully understand the experiments and corresponding results, i.e. it's not necessary to read the appendix, even if further details are available there.", "weaknesses": "1. Limited discussion of future works.\n2. NLL and Brier Score are listed among the multi-threshold metrics, but there's no thresholding involved in their evaluation. In fact, it should be possible to use them as risk measures, which would cover more for than just 0/1 loss, especially where probabilities are important for decision making.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:\n\n---\nW1. “Limited discussion of future works.”\n* Thank you for this helpful comment. In response to your feedback, we extended the Conclusion Section to point out directions of future work: Our proposed evaluation framework provides a solid basis for future work in Selective Classification, including developing novel methods as well as analyzing the properties of individual methods.\n\n---\nW2. “NLL and Brier Score are listed among the multi-threshold metrics, but there's no thresholding involved in their evaluation. In fact, it should be possible to use them as risk measures, which would cover more for than just 0/1 loss, especially where probabilities are important for decision making.”\n* Thank you for raising this point. Indeed, the NLL and Brier Score do not involve thresholding of confidence scores. In response to your feedback, we rephrased line 167 to: “Importantly, proper scoring rules such as the Negative-Log-Likelihood and the Brier Score are technically not multi-threshold metrics. Yet, we include them here as they also aim for a holistic performance assessment, i.e. assessment beyond individual working points.”\n* Regarding your proposal to use NLL or Brier Score as risk measure: NLL and BS both assess the calibration of confidence scores in addition to their ranking. Generally, the calibration task can be viewed as orthogonal to that of Selective Classification where rejection is solely based on the ranking of scores. For a detailed discussion on the relation between calibration and confidence ranking we refer to Appendix C in [1]. Further investigating the relation between confidence ranking and calibration and on evaluating SC in settings where calibrated scores are of interest is an interesting direction of future work, which we added to the conclusion section in the updated manuscript.\n\n[1] Jäger et al., “A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification”\n\n---\nQ1. “A similar task to SC is classification with rejection of OOD samples (sometimes called open-set recognition). In such cases, there isn't a ground truth to evaluate the risk on (in practice). Can the authors envision a way to adapt their metric to these scenarios?”\n* We understand you are referring to settings where the SC model is validated on data containing classes that were not part of the training data. The AUGRC metric based on the binary failure label is directly applicable to these scenarios, as any prediction on unknown classes is per definition a misclassification (for a visualization, see Figure 5 in [1]). In Table 3, we report AUGRC values evaluated on various distribution shifts, also containing semantic and non-semantic new-class shifts.\n* In case you are referring to a different kind of open-set recognition, we would ask for  a more concrete task formulation for us to comment on.\n\n---\nQ2. “The metrics were scaled by ×1000 in Figure 2. Were they also rescaled for all the other results?”\n* Thank you for pointing out the missing information. We rescaled all results apart from Figure 5. More precisely, we scaled the AURC and AUGRC values by 1000 in Figure 4 and Table 3. For the color-coded AUGRC in Figure 5, no rescaling was done. In response to your feedback, we updated the caption of Figure 4 to clearly state which values were rescaled.\n\n---\nQ3. “What does the optimal AUGRC mean? What would be the AUGRC of the Bayes-optimal model for a task? Is there an AUGRC value associated to a random classifier, as in AUROC?”\n* The optimal AUGRC refers to the AUGRC for an optimal CSF, i.e. one that ranks all failure cases lower than correctly classified cases (AUROC$_f = 1$), but is conditioned on the performance of the underlying classifier. Please note that the CSF is not directly linked to the classifier output, but can be any (external) function that provides continuous confidence scores per classified sample. As such, in the general formulation, classifier and CSF are two independent entities and AUGRC assesses the performance of both jointly.\n* The Bayes-optimal classifier by itself does not directly define a CSF and hence does not correspond to a specific AUGRC value. We believe that investigating the relation between Bayes-optimality and SC is an interesting direction of future work.\n* On the level of detecting failure cases, i.e. given a fixed classifier with accuracy $acc$, assigning random confidence scores yields a AUROC$_f=0.5$. Following Equation 8, this corresponds to AUGRC $=(1-acc)/2$.\nThe same holds if the classification itself is random. There, however, the accuracy depends on the number of classes $K$, i.e. $acc=1/K$.\n* In response to your questions, we re-formulated line 206 to: “The AUGRC for an optimal CSF (AUROC$_f=1$) is given by the second term in Equation 7”. We also added the AUGRC of the random classifier to the same section.\n\n---\nQ4. “Any intuitions about why the rankings of AURC and AUGRC where the same for iWildCam?”\n* For the iWildCam dataset, we observe very robust method rankings for both AURC and AUGRC which don’t change between the two metrics. Overall, the volatility of method rankings varies from dataset to dataset (Figure 3).\n* We agree that the investigation of specific methods and datasets is an interesting direction of future work, which we added to the conclusion section in the updated manuscript.\n\n---\nThank you once more for your constructive feedback. As we believe to have resolved your comments, please let us know in case you have remaining suggestions to further increase the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MJx9cPynNy", "reviewer": "Reviewer_izgv", "summary": "The paper presents 5 requirements for multi-threshold metric for Selective Classification and a novel metric to evaluate selective classifiers called AUGRC. The proposed metric satisfies 5 requirements that are not met by current approaches. The proposed metric changes the rankings on 5 out of 6 datasets considered by the authors.", "questions": "I have a few questions for the authors:\n\n* [Q1] can the authors clarify my concerns regarding the empirical evaluation?\n* [Q2] can the authors discuss further my concern regarding interpretability? Is there a straightforward interpretation of the AUGRC score?\n* [Q3] when considering a degenerate abstaining classifier that always abstains, i.e., whose confidence score is always zero, is the AUGRC 0 (as the one obtained by a perfect classifier)? If so, is AUGRC favoring classifiers that tend to be underconfident and asbtain too much? Is there a way to avoid such a behaviour? \n* [Q4] if I correctly understand how AUGRC works, why do not the authors consider the _classification quality_ by [Condessa et al, 2017] instead of the Generalized Risk Score? Such a metric also considers the failures that are correctly rejected and could avoid favoring abstaining classifiers that over reject.\n\n\nTypos:\n\nline 128: schore  $\\to$ score", "limitations": "The authors do not directly discuss limitations of their proposed approach in the main paper. For instance, I think a brief discussion regarding the computational time required to compute AURGC should be included in the main paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper's main strengths are:\n\n\n1. the paper aims to tackle a long-standing concern in abstaining classifiers literature, i.e. how to evaluate with a single measure these classifiers\n2. the theoretical derivations seem sound\n3. the contribution is well framed within current literature", "weaknesses": "The main concerns of the paper are:\n\n1. The empirical evaluation can be improved\n2. The interpretability of the proposed measure is not that straightforward\n3. The paper presentation can be improved. \n\nRegarding the empirical evaluation, I have a few remarks.\n* there are some contradictory lines, which should be double-checked: for instance, in lines 268-269, it is not clear to me why you claim that AURC erroneously favors DG-RES over MCD-PE, as DG-RES's performance seems to be better than MCD-PE (accuracy-wise and ranking-wise). Similarly, it is not clear to me why in Figure 4 the authors state DG-RES is favored \"despite a lower classification performance and ranking quality\", while in Figure 4.a is reported a higher accuracy and higher $AUROC_f$ for DG-RES;\n* second, the authors never specify whether they correct for multiple outcomes testing (e.g. using Bonferroni correction). I think since the authors are performing multiple pairwise ranking tests, they also have to account for this. \n\nRegarding the interpretability requirement, the authors claim (correctly) that the AUROC can be interpreted as \"the probability of a positive sample having a higher score than a negative one\". I did not fully grasp what is the straigthforward interpretation of the AUGRC score in this context (lines 206-208).\n\nRegarding paper presentation, the paper heavily relies on acronyms. I would personally reconsider this choice to improve the overall readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:\n\n---\nW1. “The empirical evaluation can be improved.”\n\nW1.1. “there are some contradictory lines [...]”\n* Thank you for pointing out the mistakes in Section 4.2. It should read “erroneously favors MCD-PE over DG-RES” in line 268-269, and it should read “favoring MCD-PE [...] compared to DG-Res” in the caption of Figure 4. We corrected the sentences in the manuscript.\n\nW1.2. “the authors never specify whether they correct for multiple outcomes testing [...]”\n* Thank you for pointing this out. Please find our detailed response in the general response to all reviewers above.\n\n---\nW2. “[...] straigthforward interpretation of the AUGRC score in this context (lines 206-208).”\n* Arguably most intuitively, the AUGRC corresponds to the rate of silent failures averaged over working points. In the suggested context, the interpretation is: On drawing two random samples, it corresponds to the probability that either both are failure cases or that one is a failure and has a higher confidence score than the non-failure. This can be read from Equation 7:\n$$AUGRC = (1 − AUROC_f ) · acc · (1 − acc) + (1 − acc)^2 / 2$$\n  The second term is the probability of drawing two failures. The first term is the probability that one is a failure times the probability that the failure is ranked higher (1 − AUROC$_f$). As an example, a CSF that outputs random scores yields AUROC$_f=0.5$, and hence AUGRC $=(1-acc)/2$.\n* In response to your feedback, we changed “expected risk of silent failures across working points” in lines 201-202 to “rate of silent failures averaged over working points”. Further, we extended the explanation on Equation 7 to include the AUROC-analogous interpretation as well as the example case of a random CSF.\n\n---\nW3. “[...] the paper heavily relies on acronyms”\n* Thank you for raising this point. In response to your feedback, we reintroduced the abbreviations at important places in the paper (e.g. beginning of Section 2) or used the full term instead. We further updated Section 2.4 to introduce all acronyms with the full term for improved readability.\n\n---\nW4. “[...] a brief discussion regarding the computational time required to compute AURGC should be included in the main paper.”\n* Thank you for bringing up this point. The main bottleneck of the AUGRC computation is the sort operation on the confidence scores (O(n log n)). This is the same as for the AURC and there is no computational overhead as compared to the AURC.\n* Benchmarking the computational time for evaluating the AURC, AUGRC, and AUROC$_f$ for 1k random scores and failure labels, we obtain the following results:\\\nAUROC$_f$: 730μs ± 13μs (sklearn implementation)\\\nAURC: 564μs ± 2μs (ours)\\\nAUGRC: 562μs ± 2μs (ours)\n* In response to your feedback, we added a note on the AURC computation time in the beginning of Section 4, and details and benchmark results to a new section in the Appendix: “A.2.3 Computation Time”.\n\n---\nQ1. & Q2. \nAddressed in W1-W3.\n\n---\nQ3. “[...] confidence score is always zero, is the AUGRC 0 [...]? If so, is AUGRC favoring classifiers that tend to be underconfident and asbtain too much? [...]”\n* To clarify, whether a SC model abstains given a predicted confidence score depends on a threshold which is fixed on application. However, a multi-threshold evaluation, like in AUGRC, does not involve a fixed threshold but aggregates across all working points. More precisely, no explicit abstention decision is made in the AUGRC, the same as no classification decision is made in the classification AUROC.\n* Illustrating the latter with an example: A binary classifier that always outputs $P(Y=1)=0$ will yield an AUROC of ½ being equivalent to the random classifier. Analogously, if the confidence scores are always zero, the AUROC$_f$ is ½ and following Equation 7 the AUGRC is $(1-acc)/2$. Thus, the AUGRC does not favor underconfident classifiers.\n* We extended the explanation on Equation 7 (see our response to W2).\n\n---\nQ4. “[...] why do not the authors consider the classification quality [...]? [...]”\n* [2] define the Classification quality (Q) as the fraction of rejected failures plus the fraction of non-rejected non-failures.\n* Using 1-Q as a replacement of Generalized Risk in the AUGRC deviates from the intention of evaluating the silent failure risk, as this assigns a loss of 1 to rejected but correctly classified samples. As an example of resulting unintended behavior, two classifiers with accuracy 0 and 1, respectively, both yield a Q-based AUGRC of ½. This breaks requirement R2.\n* Please note that instead of a holistic evaluation suitable for method benchmarking, [2] derives “performance measures [...] [that] correspond to a reference operating point”. Thus, Q may be used for working point selection but is not suitable for replacing Generalized Risk in the AUGRC.\n* We added to Section 2.1: “Aside from the selective risk, other performance measurements for working point selection include for example Classification quality and Rejection quality [2]”.\n\n[2] Condessa et al., “Performance measures for classification systems with rejection”\n\n---\nThank you for your constructive feedback. As we believe to have resolved your comments, please let us know if there are any remaining concerns that would hinder a recommendation for acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "roBQlgXiY1", "reviewer": "Reviewer_QM33", "summary": "The authors introduce a new metric AUGRC for evaluating classifiers under the Selective Classification framework, whereas the classifier\nhas an option to reject low-confidence predictions. The authors introduce desirable properties for evaluating the Selection Classifiers, and show that the new metric has all those properties, unlike any of the currently used metrics. Experiments show that AUGRC produces significanly different rankings of Confidence Scoring Functions compared with currently used metrics.", "questions": "The authors claim that Selective Classification is a crucial components for clinical diagnostics ML systems (among other application domains). Can you provide examples of classifiers in clinical use that incorporate Selective Classification?\n\nWhat is AUROC_f? It is introduced on line 172, however the description is unclear and there are no references.", "limitations": "There is no limitations section. I don't have a good idea regarding limitations of this manuscript.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors address relevant question and make useful contribution. The paper is well written, clear and easy to read. The experimental analyses are, to my judgement, sound. The code is provided.", "weaknesses": "Fig. 4: the risk vs. coverage curves for AURC vs. AUGRC do not look that different. Sure, there is non-monotonicity in AURC, but that is\nin the low-coverage region, which presumably is not of practical interest. The rest looks monotonic and fairly similar. It is also true\nthat the two metrics suggest different CSFs, but again, they aren't that different. To be clear, I still think AUGRC is favorable, just\nsaying that the difference appears modest.\n\nI think the key conclusion, which is the recommendation to adopt AUGRC, is valid, however the language in the section is a bit too strong (there is \"substantial\" twice and \"significant limitations\"). I recommend to tone-it-down.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you again for your valuable comments, and for taking the time to read our general reply, as well as considering our point-by-point comments here:\n\n---\nW1. “Fig. 4: the risk vs. coverage curves for AURC vs. AUGRC do not look that different. Sure, there is non-monotonicity in AURC, but that is in the low-coverage region, which presumably is not of practical interest. The rest looks monotonic and fairly similar. It is also true that the two metrics suggest different CSFs, but again, they aren't that different. To be clear, I still think AUGRC is favorable, just saying that the difference appears modest.”\n* While the overall shape of the Selective Risk-Coverage is indeed similar to that of the Generalized Risk-Coverage curve, we demonstrate in our experiments that even smaller visual changes in the curves lead to alterations in method rankings and are thus highly relevant.\n* Further, we want to point out that the low-coverage region may also be relevant in practice, for instance considering a clinical deferral-system where a large fraction of cases may be deferred to ensure very high accuracy on the accepted cases. When evaluating specific applications for which certain risk or coverage intervals are known to be irrelevant, adaptations such as a partial AUGRC (analogous to the partial AUROC) may be considered. In response to your feedback, we added the note on partial evaluation of the AUGRC to the conclusion section.\n\n---\nW2. “I think the key conclusion, which is the recommendation to adopt AUGRC, is valid, however the language in the section is a bit too strong (there is \"substantial\" twice and \"significant limitations\"). I recommend to tone-it-down.”\n* Thank you for this helpful comment. In the updated manuscript, we toned down the conclusion, as you proposed, removing the word “significant” in “the current metrics have significant limitations” (line 276) and “substantial” in “substantial deviations from intended and intuitive performance assessment behaviors” (line 282-283).\n\n---\nQ1. “The authors claim that Selective Classification is a crucial components for clinical diagnostics ML systems (among other application domains). Can you provide examples of classifiers in clinical use that incorporate Selective Classification?”\n* Three interesting examples would be the following:\n  * Dvijotham et al., “Enhancing the reliability and accuracy of AI-enabled diagnosis via complementarity-driven deferral to clinicians”: The authors introduce the SC-based CoDoC deferral system for breast cancer screening and TB triaging, showing “that combined AI-clinician performance using CoDoC exceeds that currently possible through either AI or clinicians alone”.\n  * Leibig et al., “Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis”: The authors propose an SC-based decision-referral approach for breast-cancer screening based on mammography data.\n  * Bungert et al., “Understanding Silent Failures in Medical Image Classification”: The authors comprehensively evaluate SC in the biomedical field and develop an interactive tool that facilitates identifying silent failures.\n\n\n---\nQ2. “What is AUROC_f? It is introduced on line 172, however the description is unclear and there are no references.”\n* The “failure AUROC” AUROC$_f$ is the standard AUROC but computed on the binary failure labels (correctly classified vs. failure, denoted $Y_f$ in this paper) and the confidence scores. We added a corresponding reference in Section 2.4.\n\n---\nThank you for your constructive feedback. As we believe in having resolved your comments, please let us know in case there are remaining concerns that would prevent you from recommending a stronger acceptance of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their valuable comments. The reviewers generally agreed on the added value of our work , noting that “The paper is well written, clear and easy to read” (QM33) , “The metric is simple, yet effective” (xFt7), “The experiments are extensive” (BBe8), and “It is important for the community to know about this work as this changes the perception of SC and the best method to use” (BBe8).

However, one reviewer (izgv) did not yet recommend acceptance of the paper. Thus, additionally to the point-by-point responses below, we would like to address izgv’s main point of criticism here:

---
**Empirical evaluation: Missing correction for multiple testing in Section 4.1.** Reviewer izgv noted that a correction for multiple testing may be required when performing multiple pairwise ranking tests.
* We thank reviewer izgv for bringing up this point. We agree that when deducing overall stability of method rankings based on multiple pairwise tests, a correction for multiple testing such as the Bonferroni [1] or Holm [2] method is necessary.
* To address this concern, we conducted the empirical evaluation with both the Holm correction and the more conservative Bonferroni correction for multiple testing.
* Please find the detailed **results in the additional PDF**. Across the whole study, 913/936 pairwise differences were significant at alpha=0.05 without correction, 906/936 with the Holm correction, and 906/936 with the Bonferroni correction. In summary, the corrections did not alter our original statement that the results of the pairwise signed-rank tests “indicate stable method rankings for both AURC and AUGRC”.
* In response to the feedback we updated Figure 3 in the manuscript to include the Holm correction for multiple testing, following [3]. Further, we added the results without correction and with the Bonferroni correction to the appendix in section A.3 Additional Results.

[1] Dunn, “Multiple comparisons among means”\
[2] Holm, “A simple sequentially rejective multiple test procedure”\
[3] Wiesenfarth et al., “Methods and open‑source toolkit for analyzing and visualizing challenge results”

---
We have thoroughly revised our manuscript to address the provided feedback. The list of changes we performed  includes:
* We improved the explanations on the interpretability of the AUGRC in Section 3 based on the comments and questions by the reviewers xFt7 and izgv.
* We corrected two mistakes in the main text in Section 4.2, pointed out by izgv.
* We updated the significance maps displayed in Figure 3 to include the Holm correction for multiple testing. (izgv)
* We toned down strong formulations in the conclusion. (QM33)
* We renamed “multi-threshold metrics” to “holistic metrics” in several parts of the manuscript. (xFt7)
* We extended the Conclusion Section to point out directions of future work. (xFt7)
* We added details on the AUGRC computation time in the Appendix and Section 4. (izgv)
* We reduced the amount of acronyms used to improve clarity and readability. (izgv & BBe8)

---
We believe that these updates resolve the stated concerns of all reviewers. Please find our point-by-point answers in the respective reviewer sections.

Meta-review: The paper addresses the problem of performance measurement in selective classification (SC). The authors introduce five key requirements that SC metrics should meet and demonstrate that none of the existing metrics fully satisfies all of these requirements. They propose a new metric, called the Area under the Generalized Risk Coverage curve (AUGRC), and show that it meets all five criteria. Additionally, empirical studies provide further evidence that this new metric can be more effective in evaluating SC algorithms.

All reviewers agree that the paper is worth publishing. In their rebuttal and during the discussion phase, the authors successfully addressed the reviewers' concerns and committed to making the necessary revisions.
Decision: Accept (spotlight)