Overcoming Common Flaws in the Evaluation of Selective Classification Systems

Authors: jeremias  traub
till j bungert
carsten t lüth
michael  baumgartner
klaus h maier-hein
lena  maier-hein
paul f jäger
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:15:32

Abstract: Selective Classification, wherein models can reject low-confidence predictions, promises reliable translation of machine-learning based classification systems to real-world scenarios such as clinical diagnostics. While current evaluation of these systems typically assumes fixed working points based on pre-defined rejection thresholds, methodological progress requires benchmarking the general performance of systems akin to the $\mathrm{AUROC}$ in standard classification. In this work, we define 5 requirements for multi-threshold metrics in selective classification regarding task alignment, interpretability, and flexibility, and show how current approaches fail to meet them. We propose the Area under the Generalized Risk Coverage curve ($\mathrm{AUGRC}$), which meets all requirements and can be directly interpreted as the average risk of undetected failures. We empirically demonstrate the relevance of $\mathrm{AUGRC}$ on a comprehensive benchmark spanning 6 data sets and 13 confidence scoring functions. We find that the proposed metric substantially changes metric rankings on 5 out of the 6 data sets.

Main paper: 
 introduction: 
 Selective Classification (SC) is increasingly recognized as a crucial component for the reliable deployment of machine learning-based classification systems in real-world scenarios, such as clinical diagnostics [Dvijotham et al., 2023 , Leibig et al., 2022 , Dembrower et al., 2020 , Yala et al., 2019] . The core idea of SC is to equip models with the ability to reject low-confidence predictions, thereby enhancing the overall reliability and safety of the system [Geifman and El-Yaniv, 2017 , Chow, 1957 , El-Yaniv et al., 2010] . This is typically achieved through three key components: a classifier that makes the predictions, a confidence scoring function (CSF) that assesses the reliability of these predictions, and a rejection threshold τ that determines when to reject a prediction based on its confidence score.In evaluating SC systems, two primary concepts are essential: risk and coverage. Risk expresses the classifier's error potential and is typically measured by its misclassification rate. Coverage, on the other hand, indicates the proportion of instances where the model makes a prediction rather than rejecting it. An effective SC system aims to minimize risk while maintaining high coverage, ensuring that the model provides accurate predictions for as many instances as possible.Current evaluation of SC systems often focuses on fixed working points defined by pre-set rejection thresholds [Geifman and El-Yaniv, 2017 , Liu et al., 2019 , Geifman and El-Yaniv, 2019] . For instance, a common evaluation metric might be the selective risk at a given coverage of 70% [Geifman and El-Yaniv, 2019 ], which communicates the potential risk associated with a specific confidence score c and selection threshold τ to a patient or end-user. While this method is useful for communicating risk in specific instances, it does not provide a comprehensive evaluation of the system's overall performance. In standard classification, this is analogous to the need for the Area Under the Receiver Operating Characteristic (AUROC) curve rather than evaluating sensitivity at a specific specificity, which can be highly misleading when assessing general classifier capabilities [Maier-Hein et al., 2022] . The AUROC provides a holistic view of the classifier's performance across all possible thresholds, thereby driving methodological progress. Similarly, SC requires a multithreshold metric that aggregates performance across all rejection thresholds to fully benchmark the system's capabilities.Several current approaches attempt to address this need for multi-threshold metrics in SC. The Area Under the Risk Coverage curve (AURC) is the most prevalent of these Geifman et al. [2018] . However, we demonstrate in this work that AURC has significant limitations as it fails to adequately translate the risk from specific working points into a meaningful aggregated evaluation score. This shortcoming hampers the ability to benchmark and improve SC methodologies effectively.In our work, we aim to bridge this gap by providing a robust evaluation framework for SC. Our contributions are as follows:Refined Task Formulation: We are the first to provide a comprehensive SC evaluation framework, explicitly deriving meaningful task formulations for different evaluation and application scenarios such as working point versus multi-threshold evaluation.Formulation of Requirements: We define five critical requirements for multi-threshold metrics in SC, focusing on task suitability, interpretability, and flexibility. We assess current multi-threshold metrics against our five requirements and demonstrate their shortcomings.We introduce the Area Under the Generalized Risk Coverage (AUGRC) curve, a new metric designed to overcome the flaws of current multi-threshold metrics for SC. AUGRC meets all five requirements, providing a comprehensive and interpretable measure of SC system performance.We empirically demonstrate the relevance and effectiveness of AUGRC through a comprehensive benchmark spanning 6 datasets and 13 confidence scoring functions.In summary, our work presents a significant advancement in the evaluation of SC systems, offering a more reliable and interpretable metric that can drive further methodological progress in the field.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The described pitfalls of existing metrics affect Selective Classification evaluation in general. Hence, we thoroughly discuss all common metrics in the field and demonstrate the relevance on a state-of-the-art benchmark.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
refined task formulation: 
 Selective Classification (SC) systems consist of a classifier m : X → Y, which outputs a prediction and a confidence scoring function (CSF) g : X → R, which outputs a confidence score associated with the prediction. Assuming a supervised training setup, let {(x i , y i )} N i=1 be a dataset containing N independent samples from the source distribution P (X, Y ) over X × Y. Given a rejection threshold τ , the model prediction is accepted only if the corresponding score is larger than τ :EQUATIONGiven an error function ℓ : Y × Y → R + , the overall risk of the classifier m is given by R(m) :=1 N N i=1 ℓ(m(x), y). The error function thereby contains the measure of classification performance suitable for the task at hand. Commonly, SC literature assumes a 0/1 error corresponding to the failure indicator variable Y f with y f,i (x i , y i , m) := I[y i ̸ = m(x i )]. In this case, the overall risk represents the probability of misclassification P (Y f = 1). By rejecting low-confidence predictions, the selective classifier can decrease the risk of misclassifications at the cost of letting the classifier make predictions on only a fraction of the dataset. This fraction is denoted as the coverage := P (g(x) ≥ τ ), with the respective empirical estimator 1 N N i=1 I(g(x i ) ≥ τ ). Evaluation of SC Figure 1 : The AUGRC metric based on Generalized Risk overcomes common flaws in current evaluation of Selective classification (SC). a) Refined task definition for SC. Analogously to standard classification, we distinguish between holistic evaluation for method development and benchmarking using multi-threshold metrics versus evaluation of specific application scenarios at pre-determined working points. The current most prevalent multi-threshold metric in SC, AURC, is based on Selective Risk, a concept for working point evaluation that is not suitable for aggregation over rejection thresholds (red arrow). To fill this gap, we formulate the new concept of Generalized Risk and a corresponding metric, AUGRC (green arrow). b) We formalize our perspective on SC evaluation by identifying five key requirements for multi-threshold metrics and analyze how previous metrics fail to fulfill them. Abbreviations, CSF: Confidence Scoring Function, AU(G)RC: Area Under the (Generalized) Risk Coverage curve. systems often focuses on preventing "silent", i.e. undetected failures for which both y f,i = 1 and g(x i ) ≥ τ . Deploying a SC system in practice requires the selection of a fixed rejection threshold τ . In the following refined evaluation protocol, we distinguish between the well-established application-specific task formulation where SC models are evaluated at individual working points (Section 2.1) and SC method development where evaluation is independent of individual working points (Section 2.2).Current Selective Classification evaluation commonly reports the selective risk of the system at a given coverage ("risk@coverage"), which implies a pre-determined cutoff on the risk or coverage (e.g. "maximum risk of 5%") leading to a working point of the system defined by a rejection threshold τ . The selective risk is defined as:Selective Risk (m,g) (τ ) := N i=1 ℓ(m(x i ), y i ) • I(g(x i ) ≥ τ ) N i=1 I(g(x i ) ≥ τ ) (2)This formulation only considers accepted predictions with c > τ . For a binary failure error, this risk is an empirical estimator for the conditional probability P (Y f = 1|g(x) ≥ τ ). Thus, this metric effectively communicates the risk of a "silent" failure for a specific prediction of classifier m that has been accepted (c > τ ) at a pre-defined threshold τ . This information may be useful in applied medical settings, e.g. to communicate the risk of misclassification for a patient p whose associated prediction has been accepted by a given SC system (c p > τ ). Aside from the selective risk, other performance measurements for working point selection include for example Classification quality and Rejection quality [Condessa et al., 2017] .Evaluating the general performance of a Selective Classification system requires a multi-threshold metric for a more comprehensive view compared to a working point analysis based on a fixed singular rejection threshold, analogous to binary classification tasks, where the Area Under the Receiver Operating Characteristic (AUROC) is used to assess the entire system performance across multiple classifier thresholds. However, the working point analysis can not be simply translated to multithreshold aggregation, as it is based on Selective Risk, which only considers the risk w.r.t accepted predictions, assuming that a specific selection has already occurred (P(Y f = 1 | g(x) ≥ τ )).This evaluation ignores rejected cases and thus contradicts the holistic assessment, where we are interested in the general risk for any prediction causing a silent failure independent of the future selection decision of individual predictions. This holistic assessment is reflected by the joint probability P (Y f = 1, g(x) ≥ τ ), which reflects the risk of silent failure for any prediction processed by the system including cases that are potentially rejected in the future.To address the discrepancy between the need for holistic assessment versus the selective risk's focus on already selected cases, we formulate the generalized risk:EQUATIONThis metric reflects the joint probability of misclassification and acceptance by the confidence score threshold. By aggregating this risk over multiple rejection thresholds, we can evaluate the overall performance of the SC system.Based on the refined task definition given above, we can formulate concrete five requirements R1-R5 for multi-threshold metrics in Selective Classification:R1: Task Alignment. The general goal in SC is to prevent silent failures either by preventing failures in the first place (via classifier performance), or by detecting the remaining failures (via CSF ranking quality). As argued in Jäger et al. [2023] , it is crucial to jointly evaluate both aspects of an SC system, since the choice of CSF generally affects the underlying classifier. R2: Monotonicity. The metric should be monotonic w.r.t both evaluated factors stated in R1, i.e. improving on one of the two factors while keeping the other one fixed results in an improved metric value. Note that R2 does not make assumptions about how the two factors are combined, but it represents a minimum requirement for meaningful comparison and optimization of the metric. R3: Ranking Interpretability. Interpretability is a crucial component of a metric [Maier-Hein et al., 2022] . AU-ROC is the de facto standard ranking metric for binary classification tasks, providing an intuitive assessment of ranking quality which is proportional to the number of permutations needed for establishing an optimal ranking. This can also be interpreted as the "probability of a positive sample having a higher score than a negative one". Ideally, the SC metric should follow this intuitive assessment of rankings. R4: CSF Flexibility. The metric should be applicable to arbitrary choices of CSFs. This includes external CSFs, i.e. scores that are not based directly on the classifier output. R5: Error Flexibility. Current SC literature largely focuses on 0/1 error (i.e. 1accuracy) in their risk computation. However, in many real-world scenarios, accuracy is not an adequate classification metric, such as in the presence of class imbalance and for pixel-level classification. Thus, the SC metric should be flexible w.r.t the choice of error function.2.4 Current multi-threshold metrics in SC do not fulfill requirements R1-R5 AURC: Geifman et al. [2018] derive the AURC as the Area under the Selective Risk-Coverage curve. This metric is the most prevalent multi-threshold metric in SC [Geifman et al., 2018 , Jäger et al., 2023 , Bungert et al., 2023 , Cheng et al., 2023 , Zhu et al., 2023a , Varshney et al., 2020 , Naushad and Voiculescu, 2024 , Van Landeghem et al., 2024 , 2023 , Zhu et al., 2022 , Xin et al., 2021 , Yoshikawa and Okazaki, 2023 , Ding et al., 2020 , Zhu et al., 2023b , Galil and El-Yaniv, 2021 , Franc et al., 2023 , Cen et al., 2023 , Xia and Bouganis, 2022 , Cattelan and Silva, 2023 , Tran et al., 2022 , Kim et al., 2023 , Ashukha et al., 2020 , Xia et al., 2024] . For the 0/1-error, AURC can be expressed through the following integral:EQUATIONThis integral effectively aggregates the Selective Risk (Equation 2) over all fractions of accepted predictions (i.e. coverage). However, as discussed in Section 2.2, the Selective Risk is not suitable for aggregation over rejection thresholds to holistically assess a SC system. This inadequacy effectively leads to an excessive over-weighting of high-confidence failures in AURC and in return to breaching the requirements of monotonicity (R2) and ranking interpretability (R3). We provide empirical evidence for these shortcomings on toy data (Section 3) and real-world data (Section 4.2). Further, current SC literature usually employs the 0/1 error function, even though the related Accuracy metric is well known to be an unsuitable classification performance measure for many applications. For example, Balanced Accuracy is used to address class imbalance and metrics such as the Dice Score are employed for segmentation. Moving beyond the 0/1 error yields a higher variance in distinct error values and may thus amplify the shortcoming of over-weighting high-confidence failures. As more sophisticated error functions being are an important direction of future SC research, it is crucial to ensure the compatibility of metrics in SC evaluation.e-AURC: Geifman et al. [2018] further introduce the excess-AURC as the difference to the AURC of an optimal confidence scoring function (denoted as AURC * )e-AURC = AURC -AURC * (5)It relies on the intuition that subtracting the optimal AURC eliminates the contribution of the overall classification performance and collapses it to a pure ranking metric [Geifman et al., 2018 , Galil et al., 2023 , Jäger et al., 2023] . However, several works demonstrated that this intuition does not hold and that the e-AURC is still sensitive to the overall classification performance [Galil et al., 2023, Cattelan and Silva, 2023] . We attribute this deviation from the intended behavior to the shortcomings of the underlying AURC, as we find the desired isolation of classifier performance in our improved metric formulation in Section 3. e-AURC further inherits the shortcomings of AURC w.r.t monotonicity (R2), ranking interpretability (R3), and error flexibility (R5).NLL / Brier Score: Importantly, proper scoring rules such as the Negative-Log-Likelihood and the Brier Score are technically not multi-threshold metrics. Yet, we include them here as they also aim for a holistic performance assessment, i.e. assessment beyond individual working points, by evaluating a general meaningfulness of scores based on the softmax-output of the classifier [Ovadia et al., 2019] . Thereby, they jointly assess ranking and calibration of scores, which dilutes the focus on ranking quality in the context of SC [Jäger et al., 2023] . In our formulation, the calibration aspect in these metrics breaks the required monotonicity w.r.t SC evaluation (R2). Further, they are not applicable to CSFs beyond those that are directly based on the classifier output (R4). AUROC f : The "failure version" of the standard AUROC assesses the correctness of predictions with a binary failure label [Jäger et al., 2023] . Based on the AUROC, it provides an intuitive ranking quality assessment. However, it ignores the underlying classifier performance (R1, R2), and is restricted to binary error functions (R5). OC-AUC: Kivlichan et al. [2021] introduce the Oracle-Model Collaborative AUC, where first a fixed threshold is applied on the confidence scores, above which error values are set to zero. Then, the AUROC f (or the Average Precision) are evaluated. This metric is also reported in [Dehghani et al., 2023 , Tran et al., 2022] , with a review fraction of 0.5%. OC-AUC is subject to the same pitfalls as AUROC f (R1, R2, R5). AUROC-AURC: Pugnana and Ruggieri [2023] propose the AUROC-AURC where the Selective Risk is defined as the classification (not failure) AUROC computed over the set of accepted predictions. However, it is only applicable to binary classification models with binary error functions (R5). Further, in employing Selective Risk it inherits the pitfalls described for AURC regarding monotonicity (R2) and ranking interpretability (R3). NAURC: Cattelan and Silva [2023] propose the Normalized-AURC, a min-max scaled version of the e-AURC, and claim that this modification eliminates its lack of interpretability. However, given the linear relationship with the AURC, it does not fulfill the requirements R2 and R3. F1-AUC: Malinin and Gales [2020], Malinin et al. [2021] introduce the notion of Error-Retention Curves, which corresponds to that of Generalized Risk Coverage Curves. However, in Malinin et al. [2021] the authors propose an F1-AUC metric which is only applicable to binary error functions (R5) and breaks monotonicity (R2), as it decreases with increasing accuracy for accuracy values above ≈ 0.56 (see Appendix A.1.3 for a detailed explanation.) ARC: Accuracy-Rejection-Curves [Nadeem et al., 2009 , Ashukha et al., 2020 , Condessa et al., 2017] and the associated AUC directly correspond to the AURC and are therefore subject to the same pitfalls (R2, R3). And AURC even improves in CSF-3, which features lower AUROC f and lower acc compared to CSF-1. e-f) The corresponding risk-coverage curves reveal that the non-intuitive behavior of AURC is due to the excessive effect of the high-confidence failure of CSF-1 on the Selective Risk, which is resolved in the Generalized Risk.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The empirical study is based on an extensive, state-of-the-art, yet naturally limited benchmark. However, the scope is clearly defined in the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
area under the generalized risk coverage curve: 
 In Section 2.2, we illustrate that aggregating SC performance across working points requires to shift the perspective from the Selective Risk to the Generalized Risk (Equation 3) as an holistic assessment of the risk of silent failures for all predictions, before the rejection decision is made. We propose to evaluate SC methods via the Area under the Generalized Risk Coverage Curve (AUGRC).For the binary failure error it becomes an empirical estimator of the following expression:EQUATIONThis metric evaluates SC models in terms of the rate of silent failures averaged over working points and thus provides a practical measurement that is directly interpretable. It is bounded to the [0, 1 /2] interval, whereby lower AUGRC corresponds to better SC performance. The AUGRC is not subject to the shortcomings of the AURC, and we can derive a direct relationship to AUROC f and acc (the derivation and visualizations are shown in Equation 8and Figure 5 ):AUGRC = (1 -AUROC f ) • acc • (1 -acc) + 1 2 (1 -acc) 2 (7)To illustrate, when drawing two random samples, it corresponds to the probability that either both are failure cases or that one is a failure and has a higher confidence score than the non-failure. This is reflected by the second and first term of Equation 7, respectively. As an example, a CSF that outputs random scores yields AUROC f = 1 2 , and hence AUGRC = 1 2 (1acc). The AUGRC for an optimal CSF (AUROC f = 1) is given by the second term in Equation 7, hence subtracting the optimal SC performance yields a pure ranking measure re-scaled by the probability of finding a positive/negative pair. This overcomes the lack of interpretability of AURC and e-AURC (R3). Monotonicity (R2) is ensured since the partial gradients w.r.t. both AUROC f and acc are always negative. Further, it can accommodate arbitrary classification metrics via the error function ℓ (R4) as well as arbitrary CSF (R5).Figure 2a depicts the metric contribution of individual failure cases depending on their ranking position. This shows empirically how the excessive over-weighting of high-confidence failures in AURC is resolved by AUGRC, and how the intuitive ranking assessment of AUROC f is established (see Section A.1.1 for a detailed derivation). We further showcase how AUGRC resolves the broken monotonicity requirement (R2) of AURC in Figures 2 b-d . Despite equal AUROC f and equal acc in CSF-1 and CSF-2, the AURC improves. And AURC even improves in CSF-3, which features lower AUROC f and lower acc compared to CSF-1. Figures 2 e-f depict the associated risk-coverage curves and reveal that the non-intuitive behavior of AURC is due to the excessive effect of the highconfidence failure of CSF-1 on the Selective Risk, which is resolved in the Generalized Risk. In Section 4.2 we demonstrate implications of AURC's shortcomings on real-world data.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: For the analytical relationships, we provide the assumptions and derivations. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
empirical study: 
 We conduct an empirical study based on the existing FD-Shifts benchmarking framework [Jäger et al., 2023] , which compares various CSFs across a broad range of datasets. The focus of our study is not on the performance of individual methods (CSFs) but rather on the ranking of methods based on AUGRC evaluation as compared to AURC. We utilize the same experimental setup as in Jäger et al. [2023] with the addition of CSF scores based on temperature-scaled classifier logits. Extending the benchmark by recent SC methods, e.g. Feng et al. [2022] , is an interesting direction of future work. A detailed overview of the datasets and methods used can be found in Appendix A.2. The code for reproducing our results and a PyTorch implementation of the AUGRC are available at: https://github.com/IML-DKFZ/fd-shifts.To study the relevance of AUGRC, we investigate the changes induced by AUGRC on rankings of CSFs compared to AURC method rankings.CSF Rankings with AUGRC substantially deviate from AURC rankings. Figure 3 illustrates the method ranking differences for all i.i.d. test datasets, showing changes in ranks across all CSFs. Notably, AUGRC induces changes in the top-3 methods (out of 13) on 5 out of 6 datasets. To ensure that these ranking changes are not due to variability in the test data, we evaluate the metrics on 500 bootstrap samples from the test dataset and derive the compared method rankings based on the average rank across these samples ("rank-then-aggregate"). The size of each bootstrap sample corresponds to the size of the test dataset. Metric values for each bootstrap sample are averaged over 5 training initializations (2 for BREEDS and 10 for CAMELYON-17-Wilds). We analyze the robustness of the resulting method rankings for AURC and AUGRC separately, based on statistical significance. To that end, we perform pairwise CSF comparisons in form of a one-sided Wilcoxon signed-rank test [Wilcoxon, 1992] on the bootstrap samples. To control the family-wise error rate at a 5% significance level, we apply the Holm correction for multiple testing per metric and dataset, following [Wiesenfarth et al., 2021] (see Figure 7 for the results without correction for multiple testing). The resulting significance maps displayed in Figure 3 indicate stable method rankings for both AURC and AUGRC. This suggests that the observed ranking differences are induced by the conceptual difference of AUGRC compared to AURC and, as a consequence, that the shortcomings of current metrics and respective solutions discussed in Section 2, are not only conceptually sound, but also highly relevant in practice. For the results in Figure 3 , we select the DeepGamblers reward parameter and whether to use dropout for both metrics separately on the validation dataset (details in Table 1 and Table 2 ).The shortcomings of AURC affect CSF comparison across datasets and distribution shifts. The method rankings for all datasets and distribution shifts, based on the original test datasets, are shown in Table 4 , which also includes results for equal hyperparameters for AURC and AUGRC. Results for the bootstrap-based method ranking analysis for distribution shifts are displayed in Figure 6 . The substantial differences in method rankings across all datasets and distribution shifts underline the relevance of AUGRC for SC evaluation. The complete AUGRC results on the FD-Shifts benchmark are shown in Table 3 .In few cases, as shown in Figure 3 , we observe that a CSF A is ranked worse than a neighboring CSF B even though CSF A's ranks are significantly superior to those of CSF B based on the Wilcoxon statistic over the bootstrap samples. This discrepancy can occur if the rank variability in CSF A is larger than in CSF B. While this does not affect our conclusions regarding the relevance of the AUGRC, it indicates that selecting a single best CSF for application based solely on method rankings should be approached with caution. We recommend closer inspection of the individual CSF performance in such cases, although this is not the focus of our study. Ranking changes are reflected in changes in the color sequence and highlighted by red arrows. We assess the stability of the method rankings for each metric individually using one-sided Wilcoxon signed-rank tests based on the bootstrap samples at 5% significance level with adjustment for multiple testing according to Holm. Adjacent to each ranking, we present the resulting significance maps for the pairwise CSF comparisons. These maps can be interpreted as follows: At each grid position (x, y), filled entries indicate that metric values of CSF y are ranked significantly better than those from CSF x (across bootstrap samples), cross-marks indicate no significant superiority. An ideal ranking exhibits only filled entries above the diagonal.Figure 4 gives an in-depth look at how the conceptual shortcomings of AURC affect method assessment on real-world data. The example uses the confidence like reservation score of the DeepGamblers method (DG-Res) [Liu et al., 2019] and the Monte Carlo Dropout-based predictive entropy (MCD-PE) [Gal and Ghahramani, 2016] as CSFs on the CIFAR-10 test dataset. Despite DG-Res having higher classification performance and ranking quality than MCD-PE, the AURC erroneously favors MCD-PE over DG-Res. This violates the monotonicity requirement (R2). This can be attributed to the excessive contributions of only few high-confidence failures, which aligns with the theoretical findings on failure contributions shown in Figure 2 (R3). In this example, the highconfidence failures are associated with high label ambiguity or incorrect labeling, suggesting that the AURC may exacerbate the influence of label noise in practice. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The evaluation protocol as well as all experiment setups are described. Additionally, we provide the code for reproducing the results. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code for reproducing the experimental results is published on github. All datasets used are freely accessible. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 Despite the increasing relevance of SC for reliable translation of machine learning systems to realworld application, we find that the current metrics have limitations in providing the comprehensive assessment needed to guide the methodological progress of SC systems.In this work, we establish a systematic SC evaluation framework, thereby promoting the adoption of more comprehensive, interpretable, and task-aligned metrics for comparative benchmarking of SC systems.We find that none of the existing multi-threshold metrics, particularly the AURC, meet the key requirements we identified for comprehensive SC evaluation, leading to deviations from intended and intuitive performance assessment behaviors. To address this, we introduce the AUGRC as a suitable metric for comprehensive SC method evaluation. Substantial differences in method rankings between AURC and AUGRC, demonstrated through extensive empirical studies, highlight the importance of selecting the right SC metric. Thus, we propose the adoption of AUGRC for meaningful SC performance evaluation.When evaluating specific applications for which certain risk or coverage intervals are known to be irrelevant, adaptations such as a partial AUGRC (analogous to the partial AUROC) may be considered. Further, investigating the relation between confidence ranking and calibration and on evaluating SC in settings where calibrated scores are of interest is an interesting direction of future work. Overall, our proposed evaluation framework provides a solid basis for future work in Selective Classification, including developing novel methods as well as analyzing the properties of individual methods.  
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Details are mentioned in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We don't see any potential negative societal impacts. We address enhanced reliability of real-world decision systems through the studied methods as a potential positive impact. However, we propose no new Selective Classification method in our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our experiments do not involve any new datasets or classification models. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We added proper credit for all datasets and models used in our experiments. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Technical DetailsLet {τ t } N t=1 provide a total order on the predictions (relaxing the assumption to a partial order leads to a more complicated but very similar formulation). Then, for a rejection threshold τ t , a failure case at rank t * contributes to the Selective Risk (Equation 2) by 1 t if t * ≤ t. Averaging over all thresholds yields a contribution through the failure at rank t * to the AURC of 1N N t=t * 1 t . For the Generalized Risk, the contribution at t * ≤ t is always 1 N , resulting in an overall contribution of N -t * -1 N 2to the AUGRC. The failure contribution curves are displayed in Figure 2 .Let Y f = I(m(x) ̸ = y) be a binary indicator variable for classification failures. Then, the Generalized Risk corresponds to the joint probability that a sample is accepted and wrongly classified. We can write the AUGRC as follows:EQUATIONAt ( * ), we use thatP (g(x) ≥ τ ) = P (g(x) ≥ τ |Y f = 0) • acc + P (g(x) ≥ τ |Y f = 1) • (1 -acc).On choosing two random samples, the second term in the final equation represents the probability that both are failures and the first term represents the probability that one is a failure, the other is not, and that the failure has higher confidence score than the other.The AUGRC is monotonic in acc and AUROC f as both partial derivatives are negative ∀ acc ∈ (0, 1), The Selective Risk curve can be transformed into the Generalized Risk curve via multiplication by the respective coverages. The resulting curve is monotonically increasing and bounded by the diagonal; decreasing Selective Risk corresponds to a plateau in Generalized Risk. The AUGRC corresponds to the AUGRC of an optimal CSF (shaded red) plus the re-scaled AUROC f (shaded in green). The AUROC f corresponds to the fraction of the area (parallelogram) enclosed by the green dashed line that lies above the Generalized Risk curve. (b) AUGRC (color-coded) and its negative gradients (arrows) in the Accuracy-AUROC f space.EQUATIONFor the optimal CSF, all failure cases are attributed to lower confidence scores than correct predictions. Hence, the precision P (Y f = 0|g(x) ≥ τ ) is one for coverages up to P (Y f = 0) and decreases as P (Y f = 0)/P (g(x) ≥ τ ) above. Following the task formulation as defined in Malinin et al.[2021], we can derive the following analytical expression for the optimal F1-AUC * :EQUATIONF1-AUC * increases with increasing accuracy up to P (Y f = 0) ≈ 0.56, then it decreases, favoring models with lower classification performance. Thus, the F1-AUC does not fulfill the monotonicity requirement (R2).We compare the following CSFs: From the classifier's logits we compute the Maximum Softmax Response (MSR), Maximum Logit Score (MLS), Predictive Entropy (PE), and the MLS based on temperature-scaled logits, for which a scalar temperature parameter is tuned based on the validation set [Guo et al., 2017] . Three predictive uncertainty measures are based on Monte Carlo Dropout [Gal and Ghahramani] : mean softmax (MCD-MSR), predictive entropy (MCD-PE), and mutual information (MCD-MI). We additionally include the DeepGamblers method [Liu et al.] , which learns a confidence like reservation score (DG-Res), ConfidNet Corbière et al. [2019] , which is trained as an extension to the classifier, and the work of DeVries and Taylor [2018] .We evaluate SC methods on the FD-Shifts benchmark [Jäger et al., 2023] , which considers a broad range of datasets and failure sources through various distribution shifts: SVHN [Netzer et al., 2011] , CIFAR-10, and CIFAR-100 [Krizhevsky] If the distribution shift is not mentioned explicitly, we evaluate on the respective i.i.d. test datasets.Our method ranking study focuses on the evaluation of CSF performance based on the existing FD-Shifts benchmark, hence we required no GPU's for the analysis in Section. 4. As both AURC and AUGRC can be computed efficiently, (CPU) evaluation time for a single test set is less than a minute; evaluation on 500 bootstrap samples on a single CPU core take around 3 hours.The experiments are based on the same hyperparameters as reported in 0 0 1 0 1 1 MLS 0 0 1 1 1 1 PE 0 0 1 0 1 1 ConfidNet 1 0 1 1 1 1 DG-Res 0 1 0 0 0 1 Devries et al. 1 0 1 0 0 1 TEMP-MLS 0 0 1 0 1 1 DG-PE 1 0 0 0 0 1 DG-TEMP-MLS 1 0 0 0 0 1The main bottleneck of the AUGRC computation is the sort operation on the confidence scores. This is the same as for the AURC and there is no computational overhead compared to the AURC.For a small benchmark of the computational time, we evaluate the AURC, AUGRC, and AUROC f on 1000 random scores and failure labels. We obtain the following results (averaged over 5000 cases): 564µs ± 2µs (AURC), 562µs ± 2µs (AUGRC), 730µs ± 13µs (AUROC f ).Table 3 shows the AUGRC results for the 13 compared CSFs across all datasets and distribution shifts. While the MSR baseline is not consistently outperformed across all experiments by any of the compared CSFs, MCD improves MSR scores in most settings. Temperature scaling does not exhibit consistent improvement of MLS scores. Figure 6 : Method ranking analysis for distribution shifts. This figure displays the results for the method ranking analysis for AURC and AUGRC for evaluation under distribution shifts, analogous to Figure 3 . The subfigures are titled as follows: "training dataset" → "evaluation dataset". Details on the distribution shifts are given in Section A.2.1. CSFs are color-coded and sorted from top (best) to bottom (worst) by average rank based on 500 bootstrap samples from the test dataset to ensure ranking stability. Ranking changes are reflected in changes in the color sequence and highlighted by red arrows. We assess the stability of the method rankings for each metric individually using one-sided Wilcoxon signed-rank tests based on the bootstrap samples at 5% significance level with adjustment for multiple testing according to Holm. Adjacent to each ranking, we present the resulting significance maps for the pairwise CSF comparisons. These maps can be interpreted as follows: At each grid position (x, y), filled entries indicate that metric values of CSF y are ranked significantly better than those from CSF x (across bootstrap samples), cross-marks indicate no significant superiority. Abbreviations: ncs: new-class shift (s for semantic, ns for non-semantic). In contrast to the results shown in Figure 3 , which include the Holm correction for multiple testing, this figure displays the results without correction. On 5 out of 6 datasets, the top-3 CSFs (out of 13 compared methods) change when employing the proposed AUGRC instead of AURC. This demonstrates the practical relevance of the AUGRC metric for Selective Classification evaluation. CSFs are color-coded and sorted from top (best) to bottom (worst) by average rank based on 500 bootstrap samples from the test dataset to ensure ranking stability. Ranking changes are reflected in changes in the color sequence and highlighted by red arrows. We assess the stability of the method rankings for each metric individually using one-sided Wilcoxon signed-rank tests, each with a 5% significance level, based on the bootstrap samples. Adjacent to each ranking, we present the resulting significance maps for the pairwise CSF comparisons. These maps can be interpreted as follows: At each grid position (x, y), filled entries indicate that metric values of CSF y are ranked significantly better than those from CSF x (across bootstrap samples), cross-marks indicate no significant superiority. An ideal ranking exhibits only filled entries above the diagonal.Justification: There are no conflicts with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper addresses the problem of performance measurement in selective classification (SC). The authors introduce five key requirements that SC metrics should meet and demonstrate that none of the existing metrics fully satisfies all of these requirements. They propose a new metric, called the Area under the Generalized Risk Coverage curve (AUGRC), and show that it meets all five criteria. Additionally, empirical studies provide further evidence that this new metric can be more effective in evaluating SC algorithms.

All reviewers agree that the paper is worth publishing. In their rebuttal and during the discussion phase, the authors successfully addressed the reviewers' concerns and committed to making the necessary revisions.

Reviews: review: paper_summary: paper_summary: The paper tackles the problem of Selective Classification (SC). The authors show problem with the existing metrics used in evaluating SC and propose to use the Area under the Generalized Risk Coverage curve (AUGRC). Empirical results provide useful insights about the effect of using this metric and shows how this changes  the relative ordering of state-of-the art methods.
strength_weakness: strength_weakness: - It is important to add the details of the methods in the main paper. For example, DeepGamblers (DG) is referred to as DG in the main paper and this abbreviation is only explained in the appendix. Similarly for other baselines.
- Although the experiments are extensive, it is good to include the recent state-of the art SC methods such as Feng et al (2023) to understand which method is the best to use.

References:
Feng et al: Towards Better Selective Classification
questions: questions: See the previous section
limitations: limitations: NA
paper_summary: paper_summary: The authors propose 5 requirements that should be satisfied by selective classification (SC) metrics such that they can be successfully used to rank SC models for a task. They then propose a new metric, called AUGRC, which is shown to satisfy all 5 requirements. Finally, the authors show empirically that their method performs better rankings thank previous metrics.
strength_weakness: strength_weakness: 1. Limited discussion of future works.
2. NLL and Brier Score are listed among the multi-threshold metrics, but there's no thresholding involved in their evaluation. In fact, it should be possible to use them as risk measures, which would cover more for than just 0/1 loss, especially where probabilities are important for decision making.
questions: questions: 1. A similar task to SC is classification with rejection of OOD samples (sometimes called open-set recognition). In such cases, there isn't a ground truth to evaluate the risk on (in practice). Can the authors envision a way to adapt their metric to these scenarios?
2. The metrics were scaled by ×1000 in Figure 2. Were they also rescaled for all the other results?
3. Regarding interpretability, the authors say that "The optimal AUGRC is given by the second term in Equation 7". What does the optimal AUGRC mean? What would be the AUGRC of the Bayes-optimal model for a task? Is there an AUGRC value associated to a random classifier, as in AUROC?
5. Any intuitions about why the rankings of AURC and AUGRC where the same for iWildCam?
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents 5 requirements for multi-threshold metric for Selective Classification and a novel metric to evaluate selective classifiers called AUGRC. The proposed metric satisfies 5 requirements that are not met by current approaches. The proposed metric changes the rankings on 5 out of 6 datasets considered by the authors.
strength_weakness: strength_weakness: The main concerns of the paper are:

1. The empirical evaluation can be improved
2. The interpretability of the proposed measure is not that straightforward
3. The paper presentation can be improved. 

Regarding the empirical evaluation, I have a few remarks.
* there are some contradictory lines, which should be double-checked: for instance, in lines 268-269, it is not clear to me why you claim that AURC erroneously favors DG-RES over MCD-PE, as DG-RES's performance seems to be better than MCD-PE (accuracy-wise and ranking-wise). Similarly, it is not clear to me why in Figure 4 the authors state DG-RES is favored "despite a lower classification performance and ranking quality", while in Figure 4.a is reported a higher accuracy and higher $AUROC_f$ for DG-RES;
* second, the authors never specify whether they correct for multiple outcomes testing (e.g. using Bonferroni correction). I think since the authors are performing multiple pairwise ranking tests, they also have to account for this. 

Regarding the interpretability requirement, the authors claim (correctly) that the AUROC can be interpreted as "the probability of a positive sample having a higher score than a negative one". I did not fully grasp what is the straigthforward interpretation of the AUGRC score in this context (lines 206-208).

Regarding paper presentation, the paper heavily relies on acronyms. I would personally reconsider this choice to improve the overall readability.
questions: questions: I have a few questions for the authors:

* [Q1] can the authors clarify my concerns regarding the empirical evaluation?
* [Q2] can the authors discuss further my concern regarding interpretability? Is there a straightforward interpretation of the AUGRC score?
* [Q3] when considering a degenerate abstaining classifier that always abstains, i.e., whose confidence score is always zero, is the AUGRC 0 (as the one obtained by a perfect classifier)? If so, is AUGRC favoring classifiers that tend to be underconfident and asbtain too much? Is there a way to avoid such a behaviour? 
* [Q4] if I correctly understand how AUGRC works, why do not the authors consider the _classification quality_ by [Condessa et al, 2017] instead of the Generalized Risk Score? Such a metric also considers the failures that are correctly rejected and could avoid favoring abstaining classifiers that over reject.


Typos:

line 128: schore  $\to$ score
limitations: limitations: The authors do not directly discuss limitations of their proposed approach in the main paper. For instance, I think a brief discussion regarding the computational time required to compute AURGC should be included in the main paper.
paper_summary: paper_summary: The authors introduce a new metric AUGRC for evaluating classifiers under the Selective Classification framework, whereas the classifier
has an option to reject low-confidence predictions. The authors introduce desirable properties for evaluating the Selection Classifiers, and show that the new metric has all those properties, unlike any of the currently used metrics. Experiments show that AUGRC produces significanly different rankings of Confidence Scoring Functions compared with currently used metrics.
strength_weakness: strength_weakness: Fig. 4: the risk vs. coverage curves for AURC vs. AUGRC do not look that different. Sure, there is non-monotonicity in AURC, but that is
in the low-coverage region, which presumably is not of practical interest. The rest looks monotonic and fairly similar. It is also true
that the two metrics suggest different CSFs, but again, they aren't that different. To be clear, I still think AUGRC is favorable, just
saying that the difference appears modest.

I think the key conclusion, which is the recommendation to adopt AUGRC, is valid, however the language in the section is a bit too strong (there is "substantial" twice and "significant limitations"). I recommend to tone-it-down.
questions: questions: The authors claim that Selective Classification is a crucial components for clinical diagnostics ML systems (among other application domains). Can you provide examples of classifiers in clinical use that incorporate Selective Classification?

What is AUROC_f? It is introduced on line 172, however the description is unclear and there are no references.
limitations: limitations: There is no limitations section. I don't have a good idea regarding limitations of this manuscript.

Comments: title: Thank you for your response
Thank you for your response
Author-Reviewer Discussion Phase
Response to the Rebuttal
comment: Thank you very much for taking the time to read our response and re-considering your assessment based on the provided updates.
Thank you very much for taking the time to read our response and re-considering your assessment based on the provided updates.
I appreciate your responses. I'll raise my rating to 7.
Dear Authors and Reviewers!

Thank you for your reviews, rebuttal, additional comments, questions and responses!

We have the last two days of the discussion phase! Please use this time as efficiently as possible :)

Thank you,

NeurIPS 2024 Area Chair
I thank the authors for their clarifications and the new experiments.
A few comments:

regarding Q1, I thank the reviewers for implementing my suggested changes.

regarding Q2,
> the interpretation is: On drawing two random samples, it corresponds to the probability that either both are failure cases or that one is a failure and has a higher confidence score than the non-failure. This can be read from Equation 7:

I think clarifying the interpretation of AUGRC (meaning adding these lines to the paper) is very important and can help the reader improving its understanding.

Regarding Q3 and Q4, I see your points, and I have no further questions.

Finally, I thank the authors for providing a small analysis regarding the required time for computing AUGRC.

Hence, after the authors' rebuttal and clarifications, I will increase my score to a weak accept.