A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Authors: Han Zhou
Jordy Van Landeghem
Teodora Popordanoska
Matthew B. Blaschko
Venue: ICML 2025 poster (2025)
Keywords: Selective classifier; Area Under the Risk-Coverage Curve
Primary area: applications->computer_vision
TL;DR: Revisit the characterization of the Area Under the Risk Coverage Curve (AURC)

Abstract: The selective classifier (SC) has been proposed for rank based uncertainty thresholding, which could have applications in safety critical areas such as medical diagnostics, autonomous driving, and the justice system. The Area Under the Risk-Coverage Curve (AURC) has emerged as the foremost evaluation metric for assessing the performance of SC systems. In this work, we present a formal statistical formulation of population AURC, presenting an equivalent expression that can be interpreted as a reweighted risk function. Through Monte Carlo methods, we derive empirical AURC  plug-in estimators for finite sample scenarios. The weight estimators associated with these plug-in estimators are shown to be consistent, with low bias and tightly bounded mean squared error (MSE). The plug-in estimators are proven to converge at a rate of $\mathcal{O}(\sqrt{\ln(n)/n})$ demonstrating statistical consistency. We empirically validate the effectiveness of our estimators through experiments across multiple datasets, model architectures, and confidence score functions (CSFs), demonstrating consistency and effectiveness in fine-tuning AURC performance.

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Han Zhou Jordy Van Landeghem 2 Teodora Popordanoska Matthew B. Blaschko
Abstract
and El- Yaniv,,v, 2017; Geifiman et al., 2019; Ding et al., 2020; Galil et al., 2023). Specifically, these classifiers employ cost-based models (Chow, 1970; Cortes et al., 2016; Hen- drickx et al., 2024) with a reject region to balance the risks of wrong predictions against the non-decision costs. The goal of an effective SC system is to minimize the expected misclassification costs-termmed selective risk-while max- imizing coverage, ensuring the model provides accurate predictions for as many instances as possible. This dual focus on selective risk and coverage motivates the develop- ment and evaluation of SC systems.
The selective classifier (SC) has been proposed for rank based uncertainty thresholding, which could have applications in safety critical areas such as medical diagnostics, autonomous driv- ing, and the justice system. The Area Under the Cosk-Coveregge Curve (AURC) has emerged as the foremost evaluation metric for assessing the performance of SC systems. this work, we present formal statistical formulation of popula- tion AURC, presenting an equivalent expression that can be interpreted as a reweighted risk func- tion. Through Monte Carlo methods, we derive empirical AURC plug-in estimators for finite sam- ple scenarios. The weight estimators associated with these plug-in estimators are shown to be con- sistent, with low bias and tightly bounded mean squared error (MSE). The plug-in estimators are proven to converge at rate of (I)(()))/)) demonstrating statistical consistency. We em- pirically validate the effectiveness of our estima- tors through experiments across multiple datasets, model architectures, and confidence score func- tions (CSFs), demonstrating consistency and ef- fectiveness in fine-tuning AURC performance.
Prominent evaluation metrics in SC systems, such as the area under the risk-coverage curve (AURC) and the normal- ized AURC (E-AURC) (Geifman et al., 2019), are widely used to assess model performance based on selective risk and coverage. While most studies interpret and improve (Ao et al., 2023; Traub et al., 2024) these metrics from the per- spective of risk and coverage, relatively little attention has been given to directly optimizing SC models by treating AURC loss function. In addition, these metrics are typi- cally computed empirically from the given datasets, making them susceptible to biases and variances, particularly in the context of small finite sample rather than the underlying population. Franc et al. (2023) proposed the Selective Clas- sifier Learning (SELE) loss as a lower bound of empirical AURC and is designed to optimize uncertainty scores by minimizing both the regression and SELE losses using batch training strategies. This approach only learns uncertainty scores top of pre-trained model within the selective classifier framework, and does not directly optimize the clas- sifier itself based on the loss. Franc et al. (2023) motivate the SELE loss by the fact that it is "a close approximation of the AuRC and, at the same time, amenable to optimization." " We demonstrate here through analysis of the computational complexity and statistical properties of direct AURC estima- tion, that approximation by lower bound is unnecessary and both AURC and SELE are equally amenable to opti- mization.
1. Introduction
In safety-critical scenarios such as autonomous driving, medical diagnostics, and the justice system (Berk et al., 2021; Leibig et al., 2022; Dvijotham et al., 2023; Franc et al., 2023; Groh et al., 2024), selective classifiers (SC) are promising for their ability to withhold predictions un- der conditions of uncertainty, thereby mitigating associated risks and enhancing the reliability of the models (Geifman
'Processing Speech and Images, Department of Electrical En- gineering, KU Leuven, Belgium IInstabase, San Francisco, USA. Correspondence to: Han Zhou <han. zhoun esat. kuleuveeee....>
We establish a formal definition of AURC at the population level based on the underlying data distribution and derive an equivalent expression that explicitly represents it a reweighted risk function, where the weights are determined
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
solely by population rankings according the CSFs. This formulation allows us treat AURC as loss function in more theoretically grounded manner. Building upon these findings, we introduce two plug-in estimators with weight estimators derived from Monte Carlo method. We show that both can provide good estimation and come with theoretical guarantees. Specifically, we analyze the statistical properties of the weights estimators, including their MSE and bias, and establish the convergence rate of the plug-in in estimators. Finally, we validate their efficacy through evaluations and fine-tuning experiments across various model architectures, CSFs, and datasets, demonstrating their practical ad vanntagess in AURC estimation.
main categories: ensemble approaches and post-hoc meth- ods. Ensemble methods (Lakshminarayanan et al., 2017; Teye et al., 2018; Liu et al., 2023; Xia and Bouganis, 2023; Hou et al., 2023) require multiple forward passes to approx- imate the posterior predictive distribution, exemplified by Monte Carlo Dropout (MCD) techniques (Gal and Ghahra- mani, 2016). However, recent works (Cattelan and Silva, 2024; Xia and Bouganis, 2022) suggest that ensembles may not be crucial for enhancing uncertainty estimation but rather serve to improve the predictions through a set of diverse classifiers. Thus such methods are not consid- ered further in this paper. In contrast, post-hoc estimators leverage the logits produced by the model to evaluate its performance. Popular methods include Maximum Softmax Probability (MSP) (Hendrycks and Gimpel, 2022), Max- imum Logit Score (MaxLogit) (Hendrycks et al., 2022), Softmax Margin (Belghazi and Lopez-Paz. 2021), and Neg- ative Entropy (Liu et al., 2020). Furthermore, Cattelan and Silva (2023) show that the maximum p-norm of the logits (MaxLogit-
2. Related Work
Evaluation Metrics: The Area Under the Risk Cover- age curve (AURC) and its normalized counterpart Excess- AURC (E-AURC) (Geifman et al., 2019) are the most preva- lent evaluation metrics for SC systems that compute the risk or the error with accepted predictions at different confidence thresholds. Furthermore, Cattelan and Silva (2024) have proposed a min-max scaled version of E-AURC, designed to maintain a monotonic relationship with AURC, thereby enhancing its consistency in performance assessment. How- ever, Traub et al. (2024) argues that these metrics related to the selective risk, which only focus on the risk w.r.t. ac- cepted predictions, do not suffice for holistic assessment. To address this limitation, they developed the Area under the Generalized Risk Coverage curve (AUGRC). which quantifies the average risk of undetected failures across all predictions, thereby providing comprehensive measure of system reliability. Despite these achievements, most stud- ies directly employ the empirical AURC as a proxy for the population AURC, even in finite sample scenarios, without thoroughly examining the effectiveness of these estimators under such conditions. Franc et al. (2023) introduced the SELE score, lower bound for AURC. However, their study did not explore important statistical properties of this es- timator, such as its bias or MSF, when compared to the population AURC. In contrast to previous studies, our work focuses on formally defining the population AURC in terms of the underlying data distribution and offering a reliable approximation for it in finite sample settings. Our goal is to propose an effective estimator with theoretical guaran- tees and perform an empirical analysis to compare it with existing AURC estimators.
can work better than MSP in uncertainty estimation when dealing with some models. Specifically, normalizing the logits with &g norm has been shown to yield more distinct confidence scores, as evidenced in (Wei et al., 2022). Gomes et al. (2022) propose the Negative Gini Score, which utilizes the squared Eg norm of the softmax probabil- ity. In this study, we examine the impact of the post-hoc CSFs on the AURC, aiming to offer a thorough evaluation of AURC estimators in finite sample scenarios.
3. Performance Evaluation of Selective Classifiers
3.1. Problem Setting
Let X' S Rdd be the input space, y S {0, 13 be the label space, and P(x.,) v) be the unknown joint distribution over X' V. We consider a classifier f X + A which maps to k-dimentional probability simplex, and confidence scoring function (CSF) X - [o, 1]. the selective classi- fication system (f.g) at an input x can then be described by
f(z) if 2T, "abstain" otherwise.
(f.g)(t) =
(I)
where "abstain" is triggered when g(x) falls below a de-
cision threshold 7 E R. Given loss function é A x y + R. the true risk of f w.r.t. P(z, y) is R(f) Ep(a.)) [e(f Given the finite sample dataset D,, = { )1 (X' x y) sampled i.i.d. from P(r, y), the true risk can be inferred from the empirical risk R(() ==
Uncertainty estimation: There has been a large number of works (Geifman et al., 2019; Abdar et al., 2021; Zhu et al., 2023) that highlight the importance of confidence scoring and uncertainty quantification associated with pre- dictions. In practice', commonly used CSFs fall into two
1 DE {(f(z)), w:). For practical purposes, we define the
0. 1] for easier determination of selection thresholds, this is not a strict requirement.
'hile it is preferable for the output domain of CSFs be in
2

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
selection function 9 as ((x) [z)( r]. The choice of T depends the specific scenario and the evaluation metric being used. It can either be pre-defined constant or adapt dynamically based on the predicted uncertainty of the observations.
order to better understand the population AURC, we study the behavior of the weight o(x) in Eq. (5). The following proposition provides equivalent expression for axx).
Proposition 3.1 (An equivalent expression of a(x)). Define function G(z) as the cumulative distribution, function distributionffunciionnCCC (DF) of the CSF g(x) such that
3.2. Evaluation Metrics
G(z) = (r(g(x)) ( = [ S (x)]
One common way to assess the performance of selective classifiers is the risk-coverage curve (RC curve) (El-Yaniv et al., 2010), where coverage measures the probability mass of the input space that is not rejected in Eq. (I), denoted by [x)]] And the selective risk w.r.t. P(x, y) then defined as
(6)
Under this definition, the axz) in Eq. (5) is equivalent to
a(z) = - -n G(x))
Proof. Since the expectation in the denominator in Eq. (5) is the CDF of 1 G(r), we have:
R(S,9)= = Ep(rs)) (e([(z), v)ğ(a)
(2)
Ep(x) [[x)]
/ I [g((x) 2 g [ 1- G(ā) dP(t).
is typically the 0/1 error, making R(f.g) the selective error. As indicated by the equation above, risk and cover- age are strongly dependent, where rejecting more examples reduces selective risk but also results in lower coverage. This relationship revealed by the curve motivates the devel- opment of more nuanced evaluation metrics for selective classifiers. Additionally, accuracy alone often fails in cases of class imbalance or pixel-level tasks (Ding et al., 2020), so evaluation metrics should accommodate different loss functions for more comprehensive assessment.
a(x)
This implies that integrating over the domain 3 where g(ā) S g(x). Hence, we can rewrite it as:
a(xx) = - g(()(S(()) - G (P))..
To proceed, note that G(ã) the CDF of gā),, and since G(ø) is monotonically increasing in g(²), eterize the integral in terms of G(ā). Specifically, we know that G(ă) takes values between 0 and 1 as it is a CDF, thus integration can be rewritten as:
reparam-
3.3. Equivalent Expressions of AURC
Driven by the aforementioned considerations, the AURC metric (Geifman et al., 2019) is designed offer robust evaluation framework for classifiers by effectively capturing performance across varying rejection thresholds that are determined based on the distribution of samples within the population. The AURC is typically specified as an empirical quantity from a finite sample (Franc et al., 2023, Eq. (27)), from which we derive the population AURC as
(xx) Jcceyaaces 1- G(a)
G(x)
dP(
dP(t).
1 G(a)
Now, this integral straightforward to compute:
G(a)
1
d-
(Baaa)) Pl..))
(0()) 2 ((*) ExrPP 7 (T((')) 2 s()]
Thus, we have derived the desired result:
axx) = In(1 G(x)).
Noticing that the expectation in the numerator can be swapped with the expectation outside, the equation above can then be written as:
Here G(x) can be interpreted the population rank per- centile based on the CSF sorted in ascending order. This proposition motivates the following formulation, which can considered equivalent to the population AURC Eq. (3).
AURC,(S) = Th ((a (a(x)) (f()) (x).v)
(4)
where
Definition 3.2 (An equivalent expression of AURC,). Given G(x²) as the CDF of the random variable g(æ), the population AURC in Eq. (3) is equivalent to:
I [g(x) 2 [z)
a(x) = Ep-
(5)
E I [g(x') 2 9()))
This expression shows that the population AURC can be interpreted as the expectation of the risk function weighted by axx) that accounts for the importance of each point. In
AURC, (f) (l)))(((())))))P(((
(8)
where a(x) = - In (1-G(x)).
3

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
We also provide empirical evidence supporting the equiv- alence in Appendix A.5. Notably, the following integral holds:
Continuous Mapping Theorem. Let H, : Ei= 1 denote the rath harmonic number, and define the digamma function
&
as (b(n) := T(a) The relationship between these two func- tions is given by H,, 1 1) + Y. where y ~ 0.577 is the Euler- Masschennnr constant. Setting Ho = 0, we can express 0; in terms of harmonic numbers or digamma functions:
['m
L - - (1 1.
This result indicates that, in the limit of infinite data, the integral of axx) computed using this formula converges to one. Consequently, the population AURC can be interpreted as a redistribution of the risk.
à Hn, H -n 1) - (n 1), (13)
which enables efficient computation of the weight estimator. The computation cost of Eq. (11) is O(n In(n)) due to the sorting operation required for rank computation. Addition- ally, for the finite sample case, we have
3.4. Plug-in Estimator of AURC
Given finite sample of size n, namely Dn = {(e,, y:) S (X x )), which is sampled i.i.d. from the joint probability distribution P(x, y), following Eq. (3), the empirical AURC can be defined based on the sample as
á =
Hn Hn-1)= =1
(14)
AURC,(F) 1
1
(xf())), (.)x)))))) 2 s(æ;)] Li=1
indicating the plug-in estimator with this weight å can be viewed as a redistribution of the risk. Each individual loss is weighted by à, which depends on the rank of the corresponding sample point.
E [g(xx) 2 [(,))]
(10)
This formulation represents the widely used AURC metric for evaluating the SC system. However, guarantees on the relationship to population-level AURC has not been con- sidered, even though relying on this empirical estimator may introduce error, particularly when assessing the SC system with a small sample size. The naive implementa- tion of this estimator incurs quadratic computational cost of Onn²) due to the nested loops. However, some pack- ages e.g. torch-uncertuinty² decrease this complexity to On (n) by replacing redundant subset evaluations with a single sorting step followed by cumulative summation that efficiently computes error rates across all coverage levels. Here, we present derivation of method that achieves computational complexity of (n In (n)). By leveraging the approach used to transform (3) into (4), the empirical AURC can be reformulated as a plug-in estimator:
Franc et al. (2023) gave another expression of the empirical AURC in Eq. (10) that can be interpreted as an arithmetic mean of the empirical selective risks corresponding to the coverage spread evenly over the interval [0, 1] with step 1. In addition, they proposed the SELE score, served as a coarse lower bound for empirical AURC. Details about this estimator are provided in Appendix A.2.
n'
3.5. Alternate Derivation of Plug-in Estimators via Monte Carlo
In this section, we explore an alternative derivation of plug- in estimators using the Monte Carlo method. For our popu- lation AURC, in Eq. (8), we aim to estimate this quantity using Monte Carlo integration. Since the cumulative dis- tribution score Gzx)) is unknown, we require an estimator for Eq. (7), which can be achieved by taking the conditional expectation
AURC,
Σ (f
(11)
E - In(1 - G(z)))(())())))) (o))
(15)
where
Since (G(z:))sisn behave as i.i.d. samples from a uni- form distribution 20 1] when o, are i.i.d. from P(x), we sample i.i.d. (9. ~ 2/00 1]. then sort this set in ascending order. Let r; be the rank for B, and set a, - In(1 B.). Consequently,
[[(())) 2 g(و;)]
a(x.) Σ 1 Ei [g(())) 2 g(,,)]
Σ j=1
j +1
(12)
can find lower variance estimate with the same bias by repeating the pro- cess and averaging the obtained estimates. The 8, are order statistics of the uniform distribution, and have known distribution (Jones, 2009, Section 2)
where r; denotes the rank of when the data is sorted in as- cending order according to the CSE, such that a larger r; cor- responds a higher CSF value. For simplicity, we use å; as shorthand for (..)). This estimator is consistent estimator af; for the population AURC, as directly established by the
17 m-
httpps: thw.g com cow/ENSTA- EEEEENSTAAUU AAAA--22--AAAA /II/
Bg Beta(r;, a + 1-117 rr:) B(r,, + 1 -rrr)) r.)
(16)
4

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
where B denotes the beta function. Consequently, the limit expectation of our estimator with repeatedly resampled (a will yield
Proposition 3.3 (Bias of á; The bias of the in is given by
Bias (ā: (G, = (4a)
(21)
6 = RR, 1 In(1 - [33]
Hn 1 (1-
3,)= m
In(1 - B.).
In (1 - x) x0111 (1 2)"=r de (17)
B(ri, n + r:)
Proof. The conditional expectation of à; associated with (xi, yk) is given by
- Hn Hn-r;
which leads to the weight estimator in Eq. (13). This indi- cates that the plug-in estimator of Sec. 3.4 is in fact quite principled. Furthermore, we demonstrate the consistency of this weight estimator in Prop. A.4 and 3.6. In the above procedure, we have used & = [- In(1 - 3.)], but we can utilize the expectation of 3., leading to another weight estimator
E [( 3.1 Σ (H,, 154--H-1) Hn..)
(x.) = B.).
We notice that Pr(ri Pr(ri]G(aa)) = Be) is binomial distribution Bin(n - 1, 1;) given by:
Pr (r.- = iG(()) = =B)) -
3.)=
á = - in (1-1 Es, [3.)) = =-Ih -In (1
Ts vu +1
E3
(18)
because the probability of any i.i.d. sample being ranked below x; is 3, and there must be (i (i- 1) of them for the sample to be ranked ith, while the remaining (n-i) samples must be ranked higher, each with an independent probability of (1 Be). Combining these results gives us:
where the last equation is due to the fact that the expecta- tion of Bota(a, b) is afc. This estimator is consistent, as
Pi
lim
= Br. In addition, we have
11000
+ 1
Bias (á:]G(x:) = 3,) = [[a = [b:] - ai
1 Σ 1=1
d
In
To n2
11,
(19)
Hn - -.h-- -1m-CH1-180 3.)= + +In(1 B.).
but it approaches one as n + cc. Since the function m(t)= =
- -In(1 - t) is convex, applying Jensen's 's inequality gives:
which concludes our proof.
Proposition 3.4 (Bias of a(). The bias of the âf, is given by
- In (1 - Es, [a1) S E,, I- Im(1 - -.)].
(20)
indicating the first estimator å; upper bounds the a4. Thus this weight estimator oę will lead to a consistent plug-in estimator that lower bounds the plug-in estimator with á; In the next section, we will analyze the statistical properties of these two plug-in estimators, incorporating the weight estimators discussed above.
Bias (a' G(a;) = B.)
(22)
In
n ch (1)) -,,)
+ In(1 - B.).
Proof. The expected a; associated with (z), ys) is given by
B [al]G((z)) = &]]
3.6. Statistical Properties
i =- In 1- n+1
Pr(r. a|G(x.) B.).
In this section, we show that empirical estimators of AURC are in general biased (Prop. 3.3, 3.4), but we also show con- sistency at favorable convergence rate (Prop. 3.6) indicat- ing the soundness of empirical estimators even at relatively small batch sizes. Given the fact that the weight estimator and the losses are typically not independent as they both depend on the model logits, it is difficult to directly derive the statistical properties of the plug-in estimators. There- fore, we begin by examining the properties of the weight estimators a; and â based on finite samples. For specific data pair (ara. v/) E (X,3') with an unknown population rank percentile B,, we consider randomly sampling n - i..d. samples repeatedly from the population. Our analy- sis focuses on assessing the bias and MSE of the weight estimators associated with this data pair.
Since Pr(rs|G(x:)) = Bb) follows Bin(n - 1c))) we obtain:
Bias (al(G(zz) = B.)) = [ad G(Tz)) = [::] a,
Σ In 1n(1-1 1- n +
c ch 3.) In(1 - Ba)
which concludes our proof.
For the above mentioned weight estimators, the SELE Weight estimator 6ee exhibits the largest bias compared with & and á', as indicated in Fig 1. Due to this significant bias in weight estimation, SELE is not a reliable estimator for population AURC.
5

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
By analogous reasoning determining a lower bound on the MSE, we achieve the result
Expected Expected a
Diws Biss ot
Espected 6
Dlas Blss Biss 6
Espected of
3
MSE( n B.)
vB, (0, 1).
(24)
-2
-2
which is visualized Fig 2.
0.6
(a) n=8
(b) 128
15
Figure 1. The bias of the weights estimator as a function of 3, based the results in Propositions 3.3, 3.4 and A.2. The bias of these weight estimators is not equal to zero in general, but being positive for smaller 3; and negative for larger 3. as indicated. As sample size increases, the expected bias decreases significantly for larger B, The complete plots for different sample size be found in Appendix S8.
05
04 809
Proposition 3.5 (MSE of (;)) The MSE of the âg is
Figure 2. The bound in Eq. (24) as a function of n and 8
MSE((;) = 25 (n + 11-111
-1-r, rs)) - 1)
Bg
(23)
We also demonstrate in Appendix A.3 that the MSE of á is tightly upper bounded by Eq. (24), though it remains larger than the MSE of âG.
n(1 - 3.) +1
Proof. From result in 1rop.3.1, calculate:
Proposition 3.6 (Convergence Rate of the plug-in esti- mators with & or &' ). Assume that the loss function é is square-integrable, i.e., J ((((()) dP(x, v) < co. Then, the plug-in in estimators with â or â as the weight estimator, converges at rate of O (O(vn(nn/m))
MSF();) = Es; [(ás + In(1 (8))
((H, - -Hn-r;) + In (1- (B)))²dP(())
1 In(1 - (.)))P((3))) - (H,, Hn-r,)²
Proof. We first analyze the difference between the plug-in estimator with â; and the population expected value:
:=M
where the second equality is led by So In(1 - (.,.)
(H,, - H, And dP((3)) is taken to mean integration with respect the measure induced by 3, Beta(r,, + 1 r:). Focusing the remaining integral, we have the closed form:
Σ a.l(f(())) v.) - Ela((f(z). v)].
This can decomposed as the sum of the following two terms:
M = (Hn - H Hnnn.) s6 (n 1-11- 1-rr)) (n
and the result:
A =
(a. ac)((((s)).
MSF(()) = (n +1 1- - 4// v/nn + 1).
-(ae((())))
This term involves the first derivative of the digamma func- tion for which the inequality 1 + -/2 I (n(n)) < 1 # is well known. Applying these inequalities, we obtain
where term (1) captures the error caused by the bias in estimating o and term (2) represents the error introduced by approximating the expected value [[al(f()) y)] with the empirical average.
1/2
(SE < 1
(n - r2)2 n n+1
(n + 1)²
3
Pt +1 n+1)
0 n(1 8.) +1
Making use of the Cauchy- Schwarz inequality, we obtain
6

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
the following result:
the test set. In the finite-sample setting, we evaluate the plug-in estimators with á or or the SELE score (Franc et al., 2023), and 2 SELE proposed by the original authors (see Appendix A.2 for a discussion). Beyond the 0/1 loss, we incorporate these metrics with the Cross-Entropy (CE) loss that serves as a complementary measure for assessing the classifier's performance.
1 7
{ ((((s))
A <
(6:
a,,2
1=1
(25)
(ā: a=)² E [(((f(=).)))]
Experimental setup. We evaluate the metries using sev- eral pre- treitrained models on the test set, which is randomly divided into various batch sizes (8, 12, ....................... 1024). We use MSP as our confidence score function, compute the metrics for these batch samples, and subsequently calculate the mean and standard deviation of these finite sample es- timators. The population AURC, is computed across all samples in the test set. For the CIFAR 10/100 datasets, we evaluate the mean and standard deviation of the Mean Ab- solute Error (MAE) across five distinct pre-trained models. For the ImageNet and Amazon datasets, we compute the mean, standard deviation, and MSE for different estimators of the pre-trained model across batch samples.*
From Proposition A.4, n Ei= (á;) is bounded by
In(x) which means
7
a a. ai)²=
In(n) n
(26)
By combining this with Eq. (25) and the square-integrable assumption of the loss function l. term A asymptotically converges at a rate (n)lnn))/n) Term B, correspond- ing to the Monte Carlo method, is well-known to converge at a rate On 1/2) (Caflisch, 1998), which is faster than O( (n))nn))/n). Thus, our overall convergence rate is dom- inated by the rate derived for term A. Similarly, the same convergence rate applies to the estimator with dz-
AUTC,
AUYC,
prite prite suncle entinnte
S012
4. Experiments
Datasets. We use images datasets such as CI- IAR10/100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009), and a text dataset ie Amazon Reviews (Ni et al., 2019). The Amazon dataset contains review text inputs paired with l-out-of-5 star ratings as labels.
12
10
10
(a) BERT (0/1)
(b) BERT (CE)
Models. For experiments on CIFARIO/100, we report the results on the VGG13, VGG16, VGG19 (Simonyan and Zisserman, 2014) model with batch norm layers, WideResNet28x 10 (Zagoruyko and Komodakis, 2016), and ResNet (He et al., 2016) models with different depths (20, 56, 110, 164). For each model architecture, we have 5 different models that are pre-trained on the CIFAR dataset. For experiments on Amazon dataset, we use pre- trained transformer-based models BERT (Kenton and Toutanova, 2019). RoBERTa (Liu et al., 2021a), Distill- Bert (Sanh, 2019) (D-BERT), and Distill-Roberta (D- RoBERTa) For experiments on the ImageNet dataset, we use the pre-trained models from timm (Wightman, 2019) package, including two vision transformer (ViT) (Dosovit- skiy et al., 2020) variants, ViT-Small and ViT-Large; and two Swin transformer-based models (Liu et al., 2021b), Swin-Base and Swin-Tiny. All these models are configured with standard image resolution - 224.
Figure 3. (Amazon) Finite sample estimators with 0/1 or CE loss. We utilize pre-trained model and randomly divide the test set into batch samples of size n. Subsequently, we compute the mean and std of various estimators applicd to these batch samples.
Measurement of the statistical properties of the estima- tors. From Fig. 3, it is observable that with 0/1 loss (accu- racy) and increasing sample size, the SELE score tends to un- derestimate the population AURC,. Conversely, 2 x SELE tends to overestimate the population AURC,. The plug-in estimator with á' empirically serves lower bound for that with á, supporting the correctness of our theoretical results. As the sample size grows, both estimators progres- sively converge to the population AURCp. Similar trends can also be observed regardless of the 0/1 loss or CE loss in Fig. S15-S19. Furthermore, comparison between Fig. 3(a) and 3(b) reveals that using CE loss rather than 0/1 loss re- sults in a different magnitude of variance and bias in the estimators. The bias plots in Figures $9-S12 show similar findings. In Fig.4, the MAE of the plug-in estimators con-
Metries. For our comparative analysis, we evaluate several metrics, including the population AURC, and finite-sample estimators. The AURC, is computed using Eq. (11) across
'Code available https: //gi:///iit tttpg/tttthtt.gg///g////gg com/han678/ SyymptoticAJRRs
OOtaiined from RoBERTa with procedure Sanh (2019)
7

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Table 1. Summary of population A (expressed as mean + standard deviation, scaled by 10 of the test set for models fine- tue-tuned with various loss functions. The AURC, calculated for each model architecture based on the fine-tunce results aggregated from five different seeds, each using the same pre-trained model.
CIFARIO
CIFARI00
Model
CE
SELE
Est.
a' Est.
CE
SELE
& Est.
a' Cst.
ResNet18
1.967 +0. oss
4. =4.0000000
4.473=0.030
4.471 +0.050
6, 4.648
0.577
6.532
6.533
ResNet34
G.4644g.coo cos
5.GG1-20.03
5.652g0.036
5.651120.osm ose
6.023
5.862 012
5.825 0111
5.826:2.001
ResNet50
8.318 0012
20.66
7.921200047
7.918 049
G. 6.225
6.043 015
6.007 ::0 008
6.007 :: 009
VGG16BN
7.922 2.222u.co oote
20.018
7. o.0
7.060 o15
10.790
10.58 00.586,g 2. 02a
10. 40.559-m. -0 0229
10.56 (0.5600.m 2o 030
VGG19BN
9.813 1912
8.
10.601
8.528 o oo.0se
8.524. ose
10,633 20.001
10.421 026
10. 30. 025
10.33 10.391 1o 024
WideResNet28x10
4.137 1.0. 046
3.867 10.049
3. 3.864_gooom 0. 049
3.86310 049
20.62
5.607 - 707
5.836 10 652
5.607 10 707
0000
0433
4 Est.
4 Cat.
0755
te:.
Ext.
SELE
0.022
SELE
0.03
0.23
SELE
0102
0000
0005
00030
0.00
0002
0000
6.03
0.000
0.03
log;
lng,
(a) Swin Base (0/1)
(b) Swin Base (CE)
(a) VGGI6BN (CIFARIO)
(b) VGG16BM (CIFARI00)
Figure 5. (ImageNet) MSE of finite sample estimators with 0/1 or CE loss. For each model architecture, we calculate the MSE of the estimators using a pre-trained model on batch samples derived from the test set.
Figure 4. (CIFARI0/100) MAE of different finite sample estima- tors evaluated with 0/1 loss. For each model architecture, we compute the mean and std of the MAE across five distinct pre- trained models. The MAE for each model is calculated using batch samples divided from the test set. More results can be found in Figs. S13-S14.
SCe
CDI
S0E
sistently decreases as the sample size increases. However, the SELE score does not always exhibit this trend. Its per- formance lacks stability compared to the plug-in estimators and can even be worse as sample size increases. Results shown in Fig. 5 also indicate declining trend in the MSE of the plug-in estimators on the ImageNet dataset as the sam- ple size increases, regardless of whether 0/1 or CE loss is used. This convergence is not reflected in the SELE scores, as expected. Similar MSE results are also observed across other model architectures for the 1IFAR10/100 and Ama- zon datasets (see Fig. S20-S25). Although à theoretically exhibits lower MSE in weight estimation compared á', its corresponding plug-in estimator empirically achieves even higher MSE than that of à.
Lag
(a) BERT (0/) loss)
(b) BERT (CE loss)
Figure 6. (Amazon) Finite sample estimators with different CSFs.
the case, as shown in Fig, 6(b). The SELE score is sub- stantially lower than the population AURC, across various CSFs in our evaluations. We can also observe that compared to CSFs, these finite sample estimators are more sensitive to the choice of loss functions. When using 0/1 loss, they display lower variance than CE loss.
Influence of the CSFs, We also examine the impact of vari- ous CSRs on the estimators to provide thorough evaluation of the metrics. Specifically, we consider MSP, Negative En- tropy, MaxLogit, Softmax Margin, MaxLogit-22 norm, and Negative Gini Score, as outlined in Table S2. The sample size is set to 128. We report the results for Amazon and ImageNet datasets in Fig. S26-S28, As indicated in Fig. 6, both plug-in estimators exhibits lower bias compared to other estimators across various CSFs. The 2x SELE score is more likely to overestimate AURC,, but this is not always
Training a selective classifier.
We can finetune our pre-trained model using these finite sample estimators as loss function. The MSP is employed as the CSF when applying the metrics. The models in Ta- ble 1 are fine-tuned on the training set using these estimators incorporated with CE loss over 30 epochs, using a learn- ing rate of 10-3 We set the training batch size to be 128. Additionally, we present the results for both CE loss and SELE score, as detailed in Table for the CIFAR10/100
8

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
dataset. As indicated by Table I, training with these estima- tors can effectively optimize the AURC, compared with the CE. Moreover, training with SELE loss also accelerates the optimization of A compared with CE optimization.
(Italy) and the LEONARDO consortium.
References
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D.,
Liu, L.., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021). A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243-297.
5. Conclusion and Future Work
In this work, we revisit the definition of empirical AURC and propose the population AURC from statistical per- spective, along with equivalent expression that can be interpreted as reweighted risk function. Subsequently, we introduce a plug-in estimator for population AURC, characterized by a biased weight estimator. Additionally, we provide an alternative derivation of this and another plug-in estimator using the Monte Carlo method. We rig- orously analyze the statistical properties of these Monte Carlo-derived weight estimators, including their bias, MSE, and consistency, and establish their convergence rate to be O( ((/inn))/n) To validate our theoretical results, we eval- uate the estimator across various sfate-of-the-art neural net- work models and widely-used datasets. Both plug-in in estima- tors exhibit better performance compared to the SELE score. Finally, we have demonstrated that the combination of good statistical convergence and efficient computation make them suitable training objectives for directly fine-tuning networks to minimize AURC.
Ao, S., Rüger, S., and Siddharthan, A. (2023). Empirical optimal risk to quantify model trustworthiness for failure detection. In CEUR Workshop Proceedings, volume 3505, CEUR-WS.
Belghazi, M. I. and Lopez-Paz, D. (2021). What classifiers know what they don't' 't? arXiv preprint
Berk, R., Heidari, H., Jabbari, S., Kearns, M., and Roth, A. (2021). Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50(1):3-44.
Caflisch, R. E. (1998). Monte carlo and quasi-monte carlo methods. Acta numerica, 7:1-49.
Cattelan, L. F. P. and Silva, D. (2023). On selective classi- fication under distribution shift. In NeurlPS 2023 Work- shop on Distribution Shifts: New Frontiers with Founda- tion Models.
In this paper, our primary focus is on the estimation of AURC for a fixed model. For completeness, we discuss in the appendix the scenario in which a Bayesian model is con- sidered, specifically when ~~ (P) We anticipate that these directions will inspire further research. Additionally, investigating the performance of estimators under distribu- tion shift or in the context of imbalanced datasets represents a promising avenue for future work. We also encourage studies that adapt estimators of the form developed in this
Cattelan, L. F. P. and Silva, D. (2024). How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks. In The 40th Conference on Uncertainty in Artificial Intelligence.
Chow, C. (1970). On optimum recognition error and re- ject tradeoff. IEEE Transactions on information theory, 16(1):41-46.
paper to these settings.
Cortes, C., DeSalvo, G., and Mohri, M. (2016). Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 67-82. Springer.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Deng, J.. Dong, W., Socher, R., Li, Li, K., and Fei-Fei, I. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee.
Acknowledgement
This research received funding from the Flemish Govern- ment (AI Research Program) and the Research Foundation Flanders (FWO) through project number GOG2921N. HZ is supported by the China Scholarship Council. We acknowl- edge EuroHPC JU for awarding the project ID EHPC-BEN- access to the EuroHPC supercomputer LEONARDO, hosted by CINECA
Ding, Y., Liu, J., Xiong, J.. and Shi, Y. (2020). Revisiting the evaluation of uncertainty estimation and its application explore model complexity-uncertiing trade-off. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 4-5.
2024B 10-050 and
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
9

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations.
Hendryeks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., and Song, D. (2022). Scal- ing out-of-distribution detection for real-world settings. In International Conference on Machine Learning, pages 7773. PMLR.
Dvijotham, K., Winkens, J., Barsbey, M., Ghaisas, S., Stanforth, R., Pawlowski, N., Strachan, P., Ahmed, Z., Azizi, S., Bachrach, Y., et al. (2023). Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity-diee driven deferral to clinicians. Nature Medicine, 29(7):1814-188)
Hendrycks, D. and Gimpel, K. (2022). A baseline for de- tecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learn- ing Representations.
Hou, B., Liu, Y., Qian, K., Andreas, J., Chang, S., and Zhang. Y. (2023). Decomposing uncertainty for large lan- guage models through input clarification ensembling. In Forty-first International Conference Machine Learn- ing.
El- Yaniv, et al. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5).
Franc, V., Prusa, D., and Voracek, V. (2023). Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24(11):1-49,
Jones, M. (2009). Kumaraswamy's s distribution: A beta-type distribution with some tractability advantages. Statistical Methodology, 6(1):70-81.
Gal, Y. and Ghahramani, Z. (2016). Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learn- ing, pages 1050-1059. PMLR.
Kenton, J. D. M.-W. C. and Toutanova, L. K. (2019). Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Galil, I., Dabbah, M., and Yl-Yaniv, R. (2023). What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? In The Eleventh International Conference on Learning Rep- resentations.
Krizhevsky, A., Hinton, G., et al. (2009). Learning mul- tiple layers of features from tiny images. Toronto, ON, Canada.
Lakshminarayannan B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
Geifman, Y. and El-vaniv, R. (2017). Selective classification for deep neural networks. Advances in neural information processing systems, 30.
Geifman, Y., Uziel, G., and El- Yaniv, R. (2019). Bias- reduced uncertainty estimation for deep neural classi- fiers. In International Conference on Learning Represen- tations.
Leibig, C., Brehmer, M., Bunk, S., Byng, D., Pinker, K., and Umutlu, L. (2022). Combining the strengths of radiolo- gists and ai for breast cancer screening: a retrospective analysis. The Lancel Digital Health, 4(7)xe00-e551,
Gomes, E. D. C., Romanelli, M., Granese, F,, and Piantanida, P. (2022). A simple training- free method for rejection option.
Liu, T., Cheng, J., and Tan, S. (2023). Spectral bayesian uncertainty for image super-resolution In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18166-18175.
URI.
Groh, M., Badri, O., Daneshjou, R., Koochek, A., Harris, C., Soenksen, L. R., Doraiswamy, M., and Picard, R. (2024). Deep learning-aidee decision support for diagno- sis of skin disease across skin tones. Nature Medicine, 30(2):573-5833
Liu, W., Wang, X., Owens, J., and Li, Y. (2020). Energy- based -fistrriittiibuttion detection. Advances in neural information processing systems, 33:21464 21475.
Liu, Z., Lin, W., Shi, Y., and Zhao, (2021a). A robustly optimized bert pre-training approach with post-training. In China National Conference on Chinese Computational Linguistics, pages 471-484. Springer.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, 7., Lin, S., and Guo, B. (2021b). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages (0012-1002)
Hendrickx, K., Perini, L.,, Van der Plas, D., Meert, W., and Davis, J. (2024). Machine learning with a reject option: survey. Machine Learning, 113(6)30033-311..
10

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
J., Li, and McAuley, (2019). Justifying recommen- dations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on em- pirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) pages 188-197.
Sanh, V. (2019). Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of Thirty-third Conference on Neural Information Process- ing Systems (NIPS2019).
Simonyan, K. and Zisserman, A. (2014). Very deep convolu- tional networks for large-scale image recognition. arXiv preprint urXiv:1409 1556.
Teye, M., Azizpour, H., and Smith, K. (2018). Bayesian certainty estimation for batch normalized deep networks. In International conference on machine learning, pages 4907-4916. PMLR.
Traub, J., Bungert,
Luth, C. T., Baumgartner, M., Maier-Hein, K. H., Maier-Hein, L., and Jaeger, P. F. (2024). Overcoming common flaws in the evaluation of selective classification systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Wei, H., Xie, R., Cheng, H., Feng, L., An, B., and Li, Y. (2022). Mitigating neural network overconfidence with logit normalization. In International conference on machine learning, pages 23631-23644 PMLR.
Wightman, R. (2019). Pytorch image mod- els.
hctps://qithir com/ wwtmatn pytorch- image-modeeeeee
Xia, G. and Bouganis, C.-S. (2022). On the usefulness of deep ensemble diversity for out-of-distribttion detection. arXiv preprint arXiv:2207.07517.
Xia, G. and Bouganis, C.-S. (2023). Window-based early- exit cascades for uncertainty estimation: When deep sembles more efficient than single models. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 17368-17380.
Zagoruyko, S. and Komodakis, N. (2016). Wide residual networks. In Procedings of the British Machine Vision Conference 2016, pages 87-1. British Machine Vision Association.
Zhu, F., Zhang, X.-Y., Cheng, Z., and Liu, C.-L. (2023). Revisiting confidence estimation: Towards reliable failure prediction. IEEE Transactions on Puttern Analysis and Machine Intelligence.

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
A. Appendix
A.I. Additional Proofs
Proposition A.1 (Consistency â;). Assume 31 is the population rank percentile of the observation (I4, yi) ranked by CSF. Under this definition, the parameter ā; consistent, converging to the limit
lim å = - B.). R-000
Proof. Given the sample size n and sample rank ri let set 1/ for 8 E (0, and take the limit
lim [Hn
H Ann]
lim (n + 1) 1p(n 8in + 1)]
1
lim n-x
In(n 1)
In((n - B/ + 1)
2(n + 1)
2(n - In + 1)
1
1
3
(27)
lim
+
11nn111111 - {8 +
n-70c
2(n + 1)
2(n B,, +1)
n +
lim
In(1 - 81
n-+00
= =- =IIn 33))
where the 2nd equation was obtained using the asymptotic result that (nn) + In + 00. Proposition A.2 (Bias of a. The bias of the the weight estimator 0 corresponding to the SELE score is given by
Bias (ā.))G(xx) = B.) n
- (1)1-)))----- 3,)= In(1 - (b).
Proof. From Sec. A.2, the expected age associated with (#t, yi) is given by
E G((")G(()) B:]
Pr(r: = i[((x, 3:).
n
i=1
Since Pr(r:[G(x;) = B:) is a binomial distribution Bin(n - 1, B,), we obtain:
Bias (a/lG(z,) = B.) [aj]G((a. 3.] a
Ci- 1 11 8,)= In(1 8%)
(28)
n
which conclude our proof.
Proposition A.3 (MSE of â().. The MSE of a; is given by
2 x n(1 - B.) + (S /// (n + r.) (' ( 1)+ + In (1111_1)) n nn+1 + Hn Hn-mi
3
(29)
Proof. From the result in Proposition 3.1, we calculate the MSE as follows:
MSE(a() = Es, [(a) + In(1 - =;)))
which becomes:
MSE(6() = [// (- (-In 1
Ti + In(1 - (b)) dP(B.).
+
We break this into two parts, denoted M and N:
MSE(a.) M N,
S12

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
where
M / In2 dP)dP(.) (Hn, Hn + 2 (1 - 1),
and
N = ln? (1 (1- Ti
+ 2 (1-
Fi + (Hn Hn-r;)
Here, dP,).) refers the integration with respect the probability measure induced by 8r Beta(r;, 1-r;) rz). Now, by combining M and N, we obtain the MSE:
MSE(a)) = & (n 1 1-) - vn +)+
where
Q = in (1
n2 + + Hn - Hn-r..
P3
Using the inequalities of the harmonic numbers, + In(n) S Hn, S In(n + 1) + 7, we obtain the following bounds:
5 H,, Hn-r; In(n 1) In(n - rz).
This leads to the upper bound for Q:
Q In 1
) S n1 r;
since it holds that In(1 x)) for x > -1 For the remaining term, we use the same approach as in Proposition 3.5 and
obtain the following estimate:
3
v (n + 1-r ve (n +1 1
n(1 - B₂) +
Combining this with the upper bound for Q, we obtain the final bound for the MSE:
B1
MSE(a() x
43, E (0
(30)
n(1-8))
as the MSE is dominated by this remaining term.
og or if is tightly bounded by O In(n) Proposition A.4 & E. MSE(aa) or 1 n
E MSE(a.)). The average of the sum of the MSEs of the weight estimators
n
Proof. From result in Proposition 3.5, we derive:
1 n
MSF(aa) - Σ
72
71
n11
3.) + 1
i=1
8
d3
J0
n(1 3) +1
(31)
(n + 1) In(n + 1)
1
n²
o In(n)
We could obtain the same results for á', thereby concluding our proof.
S13

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
A.2. SELE score
Franc al. (2023) gave another expression of the empirical AURC in Eq. (10) that can be interpreted as an arithmetic mean of the empirical selective risks corresponding the coverage spread evenly over the interval [0, with step J In addition, they proposed a coarse lower bound, referred to as the SELE score given by
n elf (2.) v1) lg 2 g(x,)),
Aeee (f)
ΣΣ
(32)
n2
as an alternative to the empirical AURC. They claim that 22ee (f) is an upper bound to the empirical AURC, but in the Appendix A.2, we demonstrate that this is not the case. This naive implementation this metric requires O(n²) operations but we can rewrite it in form that can computed (n In(n)) using the trick for empirical AURC:
Ti Asele (f) = Σ
( (24), n
(33)
For this metric, d.t = serves as the estimate for a.(). For the SELE score in Eq. (33), we can show that 2 (f) does not always serve as an upper bound for empirical AURC, meaning that (Franc et al., 2023, Theorem 8) does not hold. We could find a simple counterexample s.t. the following inequality holds
AURC,(F)
2A (f).
(34)
Given dataset of 5 observations sorted according to the CSF {aa, Vi ) it is possible to find classifier f s.t. (f .)) 0 for = 1 4 4 and E (f (xs), ys) > 0. Then we would have:
âs Hs Ho x 2.2833 2 20 = 2,
(35)
which leads to
5
AURC, = Σ agl (z)) 2 2
26 e (f (I))..) = 2A (f).
(36)
is a non-decreasing sequence so they cannot use Lemma 15 to derive their results. In their proof of Thcorem 8, since bg
A.3. A Generalized Approach to Epistemic Risk for Bayesian Models
In this work, we mainly focus on the estimation of AURC for a fixed model. However, when considering a Bayesian model with f ~ P(f(D), a natural approach is to consider the following expectation:
By-EUID Ey>P((|
[R (T g(x)]]
(37)
where F is the threshold value. This can be computed directly using our method with Monte Carlo sampling. By applying Fubini's Theorem, we can exchange the order of the two expectations, leading to
E, EA-CC/D))ARRC(() E, (E) [R (T
(38)
Since g(x) depends on f() under the posterior distribution P(f and the predictions are made through full Bayesian framework, this formulation allows the evaluation of AURC in way analogous to the standard AURC for fixed model. We can envision several ways to define potential quantities of interest based on model uncertainty. Many of these quantities could be potentially connected to AURC and epistemic risk, and exploring these relationships could open up valuable avenues for further investigation.
A.4. Confidence Score Functions (CSFs)
The CSFs are generally defined as functions of the predicted probabilities p;. which are the outputs by passing the logits z produced by the model for the input x through the softmax function o--). expressed as (z)) E RK The specific forms of these CSFs are outlined as follows:
S14

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Table S2, Commonly Used CSFs
Method
Equation
MSP
g(x) max Pi
MaxLogit
g(z) mmxx Z
Softmax Margin
g(z) P; max Pj
with i arg max
Pi
Negative Entropy
g(z) Σ '
Ei=1 Z; log 2;
MaxLogit-4, Norm
g I p
Negative Gini Score
(Zz) - 1 + viml p
cfar10
cifar10
ofar100
cifar100
amazon
amazon
170
imagenet
imagenet
AURC, AURC,
AURC, AURC,
122-
AURC.
AURC
1200
AURCCp
AURCp
Figure S6. (a) 0Vl Loss
Figure S6. (b) CE Less
Figure Comparison of AURC, and AURC, under different loss functions. The AURC, and AURC, are computed across the test set using Eqs. (4) and (8), respectively, Subfigure (a) shows the 0/1 loss, while subtigure (b) depicts the CE loss.
A.5. Empirical Comparison Between AURC, and AURC,
In Section 3.4, we demonstrate the theoretical equivalence of these two metrics, and here, we aim provide an empirical validation of this equivalence. We evaluate the population AURC metries using either 0/1 CE loss across 30 different models on test sets from 10/1000 We also assess these metrics for the previously mentioned models on the test sets from Amazon and ImageNet datasets. The results are reported in Fig. S7, where these two population AURC metrics are shown to be identical to each other. We also assessed the results using a two-sided t-test, which yielded p-values of 0.9981 and 0.998 for the 0/1 and CE loss, respectively. These values suggest that we should accept the null hypothesis that these two metrics are identical.
A.6. Figures
S15

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Expected Expected &
Bias Blas of
Expected à
Blas
Depected
Bla Blaz
Expected
Blas
Espacted &
Slas
Expected &
Expocted
Blas di
Expected
Expacted
Expected
Blaz d
Expocted True dI
True
True
True d
1.0
02
0.4
0.6
0.8
1.0
(a) 11
(b) 12
(c) n
(d) 6-1
Expected
Expacted à
Expected
Bias
Expacted
Expected &
Bas o
Expocted &
Blas
Expected &
Blas
Expected
Blas 6
Expected
Expected
Expected
Expected
True
True
True d
True
1.0
(6) 12
n 256
(e) 512
12 1024
Figure The bias of the weights estimator function / different sample size n. The bias computed based the results Prop. 3.3, 3.4 and A.2.
S16

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
380
000
FM
3.0
031
014
SELE
SELE
0.0
0.2
031
3.0
0.0
8.0
-000
-0.03
-0.2
-032
PreResNer20 (0/1)
(b) PreReaNett (0/1)
6re (0/1)
WideRenNee28K
3.5
ESt
ESt.
2.4
SELE
0.2
MELE
0.2
08
-02
-01
-02
082
-42
(ng:0n)
Ig.ini
l00.00)
log.in In]
(c) PreResNet20 (CE)
(1) PreResNet110 (CE)
PreResNet164 (CE)
WideResNee28k1000 (CE)
ESt
Est.
G
ES4
0.3
tat
3.4
2x3EL
QU.E
0.2
S2LE
SELE
3.5
0.
0.22
0.
3.90
-0.
92
-02
-0:4
(00.99)
[n]
(00-00)
log. in]
Pre5esNet56 (0/1)
PreResNet56 (CE)
VGG16 (W/T
VGGI6(CE)
Figure S9. (CIFAR10) Bias of different finite sample estimators evaluated with 0/1 CE loss. For each model architecture, We utilize pre pre-trained model and randomly divide the test set into batch samples of size n. Subsequently, we compute the mean and std of bias for various estimators applied these batch samples.
S17

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
032
0.80
030
Ex..
1.3
3E.5
SELE
SELE
0.00
0.0
3.0
0.0
0.0
8.0
10
10
10
10
PreResNer20 (0/1)
PreReaNett (0/1)
PreResNet164 (0/1)
WidcResNer28x10
123
1.25
1.0
190
8 tee
5F1
SELL
0.7
2
0.50
0.4
025
0.70
+0.20
-035
-0.2
-69
-099
(0099)
100.00
(0g
lgg.nn
(c) PreResNet20 (CE)
(1) P0 (CE)
(g) PreResNet164 (CE)
(b) WideResNet28s 10 (CE)
1.2
0.25
0.2
$E.5
1.00
SELE
E2
030
SE.5
1.5
SELE
Ex
SELE
0.5
2 SELE
2ELE
2 SELE
0.50
0.55
0.00
4.0
+0.30
-085
-0.0%
-03
10
In]
PreResNet166 (0/1)
(i) PreResNet56 (CE)
(k) VGG16 (M/)
VGGI6CF)
Figure S10. (CIFARI00) Bias of different finite sample estimators evaluated with 0/1 or CE loss. For each model architecture, We utilize a pre-trained model and randomly divide the test set into batch samples of size n. Subscquently, we compute the mean and std of bias for various estimators applied to these batch samples.
Est.
à Est 8' Est. SELE
Ext
Ex
SELE
SELE
2x SELE
3.15
2 SELE
0.15
2 SELE
03
3.50
0.10
Elas
3.00
0.00
-0.9
-32
-130
-0.00
-0.12
269
(a) BERT (Q/1)
(b) D-BERT (0/1)
(c) RoBERTa (0/1)
(d) D-RoBERTa (0/1)
Stst.
tal.
dEst.
SELE
Es
12
SELE
6' Est.
3.
d Est.
SELE
1.0
SELE
8
STFF
2
0.5
0.5
32
-05
02
-03
-10
10g
340
102
(e) BERT (CE)
(1) BERT (CE)
(g) RoBERTa (CE)
(h) D. RoBERTR (CE)
Figure SIL. (Amazon) Bias of different finite sample estimators evaluated with 0/1 or CE loss. For each model architecture, We utilize a pre-trained model and randomly divide the test set into hatch samples of size Subsequently, we compute the and std of bias for various estimators applied to these batch samples.
S18

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Ett
dbt
Est.
Ett
STLE
SELE
SELE
0.13
889
1.0
0000
-180
-000
-0.0
10g:01
Irgs
10g
log.
(a) ViT-Small (W/1)
(h) Large (0/1)
(c) Swin Tiny (0/1)
(d) Swin Buse (0/1)
EELL
00
04
088
30
-02
-02
-0.
-04
Togu In!
100u
00,991
(c) VIT-Small (CE)
( arge (CE)
(g) Swin-Tiny (CF)
(h) Swin-Base (CE)
Figure S12. (ImageNet) Bias different finite sample estimators evaluated with loss. each model architecture, utilize pre-trained model and randomly divide the test set into batch samples of size Subsequently, compute the mean and std of bias for various estimators applicd to these batch samples.
S19

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
Db
0.066
For
0.00
0.005
0005
8005
0000
0.044
6004
0002
6002
0227
0.01
0.099
0.000
(og,0n)
logunn
log,0n)
loguini
(a) PreResNer20 (0/1)
(b) PreResNett 1D (I/1)
(c) PreResNet164 64 (0/1)
(d) WideRessNe28k (0/)
SILE
0.00
6.3
8603
0000
0.0
10
(c) PreResNer20 (CE)
(f) PreResNerI (CE)
(g) PreResNet164 (CE)
WideResNer28x (CE)
0.44
MAL
2.5
Est.
SELE
1007
3.22
0.0
3.00
2055
A02
3,00
007
1805
1511
600
0330
logo In]
logo
logo [rd
VGGIGBN (CE)
PreResNet56 (CE)
(k) PreResNet56 (0/1)
Figure $13. (CIFARIO) MAE of different finite sample estimators evaluated with 0/1 or CE loss. For each model architecture. we compute mean and of MAE across five distinct pre-trained models. The MAE each model calculated using batch samples divided from the test set.
S20

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
SELC
FDc
0023
0003
0.00
0009
5.00
0000
0003
0005
0.000
0039
6003
0089
EC00
0.000
(og,0n)
Ipgint
10g,09)
legu
(a) PreResNer20 (0/1)
(b) PreResNetl (I/1
(c) 6reResNet164 (0/1)
(d) WideResNee28k (0/T)
032
033
6.10
6.06
6.0
604
687
000
6.0
0.2
830
10
(c) PreResNer20 (CE)
(f) PreResNerl (CE)
(g) PreResNet164 (CE)
WideResNer28kk (CE)
0.2
MLR
0.300
332
3.255
2.0
0.100
3.05
382
3.00
logu mn]
(1)
logo [m]
VGGIGBN (CE)
PrcResNet56 (CE)
(k) PreResNet56 (W/1)
Figure S14. (CIFARIOO) MAE of different finite sample estimators evaluated with 0/1 CE loss. For each model architecture, we compute mean and of MAE across five distinct pre-trained models. The MAE model calculated using batch samples divided from the test set.
S21

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
AURC,
AURC,
AURC, Est.
0.20
0.9
6.6
Rinte sample estimato
Fintec sample estmator
Finite sample estimator
0.06
SILE
SELE
0.0
SELE
SELE
0.4
SALE
2 SELE
0.04
0.2
0.22
0.20
0.00
0.00
-0.02
-0.02
10
10
10
[gg [n]
log; (1)
log:(00 (n)
(a) VGGIGBN
(b) PreReaNer20
(c) PreResNet36
0..0
AURC,
0.05
AURC,
0.05
Est.
Hnite sample estimator
ST1
Est.
Hnntc cample aztimator
0.04
SFIP SELE
Esc.
Fintte sample estimetor
0.04
Est
SELE
043
0:83
0.52
0.82
0.01
0.00
-0.03
-002
10
9
10
10
1ogg
log: (n)
logo (K)
(d) PreResNet110
(c) PreResNer164
(f)
WideResNNe18x110
Figure S15. Finite sample estimators on CIFAR10 dataset with 0/1 loss. We utilize a pre-trained model and randomly divide the test set into batch samples of size n. Subsequently, compute the mean and variance of various estimators applied these batch samples. Additionally, the population AURCp is computed across all samples in the test set.
AURC,
AURC,
AURG,
Fintte samp'e estimtor
SELE
Est.
Finte sample calmator
04
SELE
Es..
Finite sample estimator
SELE
SELE
2 SELE
63
2 SELE
01
-63
-02
-02
-60
10
10
16
(ng
logu [n]
loga
(a) VGGI6RN
(b) PreResNer20
(c) PreResNet56
0.4
AURC,
AURC,
AURC,
0.3
Est.
Hnite sample estinator
0.3
Fnnte cample cstmator
Finite sample estimator
6,2
SILE
SELE
SELE
0.2
2 SELE
2 SELE
2 SELE
0.3
0.0
0.0
-01
10
10
16
log 01)
logu [n]
logu (n)
(d) 1r
(c) PreResNet164
(1) WideResNer188110
Figure S16. Finite sample estimators on CIFARIO dataset with CE loss. We utilize a pre-trained model and randomly divide the test set into batch samples of size Subsequently, compute the and variance of various estimators applied these batch samples. Additionally, the population AURC, is computed across all samples in the test set.
$22

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
AURG
0.02
AURC
0.30
AURG
tst.
Rinte sample estimato
ESt.
Finte cample extimator
Finte sample estimwtor
222
SELE
0.00
SELE SELF
SELE
SPLF
0.9
2 SELF
0.19
0.15
0.02
9.19
605
0.00
0.05
0.00
10
10
10
10g2
loga (n)
10g
(a) VGGI6BN
(b) PreRexNe1220
(c) PreRexNei156
AURC,
AURCa
AURC,
0.25
Frite samble estmetrr
Frite stmple catmattr
Esc.
Finite sample estimator
SELE x SELE
0.20
SELE SELE
SELE
6.55
2 SELE
0.5
0.19
6.00
0.03
0.55
0.00
10
10
10
logg mm)
logu (n)
10g
(d) PreResNet110
(c) PreResNet164
(f) WideResN:228110
Figure S17. Finite sample estimators on the CIFARI00 dataset with 0/1 loss. utilize pre-trained model and randomly divide the test set into batch samples of size Subsequently, compute the and variance of various estimators applied these batch samples. Additionally, the population AURC, computed across all samples in the test set.
A0SG
EXt
042
AURC
Eah
AURG Ext Eai.
Eb..
035
DiL
to cample CEmPORT
Freste sumpee endrmator
SELE
2
025
028
035
038
666
00
(a) D-BERT (0/1)
(b) D-RoBERTa (0/1)
(c) D-BERT (CE)
(d) D-RoBERTa (CE)
Figure S18. (Amazon) Finite sample estimators with 0/1 or CE loss. We utilize a pre-trained model and randomly divide the test set into batch samples size n. Subsequently, we compute the mean and variance of various estimators applied to these batch samples. Additionally, the population AURC, is computed across all samples in the test set.
$23

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
230
Frite pomelepeeeeeoo estmaaor
1.2
SF1F
S1L5
2 SELF
2.00
SELE 2
0.13
0.30
230
Eitte
2.3
899
000
-0.0
10
logo 250
Iogo 10
10g:00
ViT-Small (0/1)
(h) ViT-Large Large (0/1)
(c) Twin-Tiny (0/1
(d) (W1)
Frite nimble minttt
Rhitc sample cramorr
estmmor
comple artINatt
SELE
SCLE
2 SELE
SELE
-03
Togu mn]
VIT-Small (CF)
(f) arge (CE)
( Swin-Tiny (CF)
(h) (CE)
Figure S19. (ImageNet) Finite sample estimators with 0/1 CE loss. We utilize pre-trained model and randomly divide the set into batch samples of size n. Subsequently, we compute the mean and variance of various estimators applied to these batch samples. Additionally, the population AURC, computed across all samples in the set.
6,00035
Estt
Est.
Est.
0.0025
0,0000
5,00225
0.0034
6.0015
0.0000
0.00133
0.00005
0.00003
0.0000
0.00000
(b) PreResNet20
(c) PreResNe156
0.002200
Fst.
6' Est.
05003315
0.001755
& Est.
SELE
2 SELE
2 SELE
0.001500
2 SELE
0.00115
0.003000
0000075
6920075
0:00005
8.000000
6.000500
0.00002
0.000000
0.000003
10gu
(d) PreResNet110
(c) PreResNet164
(f) WideResNe28810
Figure $20. (CIFARIO) MSE of different finite sample estimators with 0/1 loss. For each model architecture, calculate the MSE of the estimators using a pre-trained model on batch samples derived from the test set.
S24

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
6.100
Est. Est.
0.012
Est. Est.
Est.
FM..
6.00
x SELE
2 SELE
SELE
0.00
0.000
25
0.003
0.004
0.002
0.03
10
10
logo
log; [n]
log (1)
(a) VGG16BN
(b) PreRexNer220
(c) PreBeeNei156
à FAT.
F4t.
F4t.
0.0225
8'
0.0000
Est.
SELE
SELE
SELE
0.8%
xx SELE
0.0025
2 SELE
2 SELE
0.0020
0.0315
6.002
0.0010
6.001
0.005
0.0025
20
10
10
(ng
log; (n)
logg [n]
(d) PreResNed11
(c) PreResNet164
(f) WideResNe228x10
Figure S21. (CIFARIO) MSE of different finite sample estimators with CE loss. For each model architecture, we calculate the MSE the estimators using pre-trained model batch samples derived from the test set.
6.010
0.006
8' Est.
0.032
Est.
0.037
8 Est.
x SELE
2 SELE
SELE
5.00
0.003
0.055
0.00
0.04
0.004
0.002
0.000
0.000
0.000
loga
log, [n]
logs (n)
(a) VGGI6BN
(b) PreReaNet20
(c) PreResNe156
à Est.
Eat.
0.200
Est
Est.
SELE
SELE
SELE
0.005
x SELE
0.005
2 SELE
0.003
0.004
0.001
0.002
0.002
0.031
0.001
0.002
10
10
10
log2/h)
1o0g [n]
logo (n)
(d) 11
(c) ProResNer164
(f) WideResNe228110
Figure S22. (CIFAR100) MSE different finite sample estimators with 0/1 loss. each model architecture, we calculate the MSE the estimators using a pre-trained model on batch samples derived from the test set.
$25

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
6 Est.
Est.
Est.
SELE
SELE
0.12
SELE
x SELE
0.225
2 SELE
SELE
0.2
0.100
0.08
0.075
0.05
0.000
0.44
0.22
0.00
0.000
0.00
10
10
3
16
logy (61
log; [n]
100 (R)
(a] VGG16BN
(b) PreResNe120
(c) PreResNe156
à FAT.
0.2
FAR.
0.07
Fat.
Est.
0.10
SELE
SELE
0.05
SELE
SELE
2 SELE
0.0
0.00
0.04
0.76
0.00
0.03
0.07
0.52
0.03
0.00
0.00
10
10
logu [m]
logo (n)
10g:
(d) PreRecNet110 10
(c) N6t
(f) WideResNNe18x100
Figure S23. (CIFARIOO) MSE of different finite sample estimators with CE loss. For each model architecture, we calculate the MSE of the estimators using a pre-trained model on batch samples derived from the set.
0064
0.07
3.300
DM.
Et
tte
tet.
1.25
0.00
SELE
2 SELE
0066
SELE
SELE
2.34
1.34
uses
1.32
133
0007
1371
10g
kg2
log,
(a) BERT (0/1)
(b) D-BERT (0/1)
(c) RoBERTa (0/1)
(d) D-RoBERTa (0/1)
2.49
3.55
3.35
3.50
0.12
1.34
1.25
ESt
Est
ER
Ext
3200
SELE
SELE
135
2x SE1E
2 SILE
132
2 SELE
830
131
135
1.9
3.0
0:5
log
10g000
(e) BERT (CE)
(1) D-BERT (CE)
(g) RoBERTa (CE)
(h) D.RoBERTaa (CE)
Figure $24. (Amazon) MSE of different finite sample estimators with 0/1 CE loss. For each model architecture, we calculate the MSE of the estimators using a pre-trained model on batch samples derived from the test set.
$26

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
0.04
1030
SELE
5 0.6
36 400
1.34
0000
044
0000
642
0.31
3.000
0.03
1.00
0.00
10
(ng
logu (P)
(a) ViT-Small (0/1)
(b) Swin-Tiny (0/1)
(c) VIT-Small (CE)
(d) Swin-Tiny (CE)
Cut.
1.355
2.4
Gat.
SELE
SELE
2 SELE
2 SELE
203
10310
1.1
290
3005
(0) ViT-Large (CE)
(1) VIT-Large (0/1)
Figure S25. (ImageNet) MSE of finite sample estimators with 0/1 or CE loss. For each model architecture, we calculate the MSE of the estimators using a pre-trained model on batch samples derived from the test set.
2-00Z
(a) D.BERT (0/1)
(b) D.RoRERTA (0/1)
(c) D.BERT (CF)
(d D.RoBERRT (CE)
(e) RoBERTa (0/1 loss))
(1) RoBERTa (CE losss
Figure S26 (Amazon) Finite sample estimators that utilize 0/1 or CE loss with different CSFs.
SOLE
ARG
(4) SmalS
(b) ViT-I
(c) Swin-Tiny
(d) Swin-Base
Figure S27, (ImageNet) Finite sample estimators that utilize 0/1 loss with different CSFs.
S27

---

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators
APA
OFF
JAN
JPN
OFT
Small
(b) VingL
Swin Tiny
(d) Swin Base
Figure $28. (ImageNet) Finite sample estimators that utilize CE loss with different CSFs.
S28


Reviews:
[{"review_id": "Xgqt6l7pGr", "reviewer": "Reviewer_eX5V", "summary": "This paper proposes a new approach for learning classifiers for selective classification tasks where classifiers need not only make predictions, but also decide if it wants make a prediction or to abstain. It reformulates the area under the risk coverage curve (AURC) as an optimization objective during training of selective classifiers. Accompanying this, is an approach for estimating these classifiers using Monte Carlo methods The results show that the proposed approach outperforms the existing approach of performing uncertainty estimation on top of pre-trained models for selective classification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***General response:***  \nWe thank all reviewers for the valuable feedback. We hope our response has addressed all the concerns.\n\n**Supplementary Material:**  \nWe indeed included the appendix as supplementary material; however, while two reviewers did not see it, the other two did and were able to check it. The appendix contains extensive additional empirical results as well as detailed theoretical proofs.\n\n**Main Contribution:**  \nThe primary goal of this work is to characterize the population AURC metric, with an emphasis on its interpretation as a redistribution of the risk function. To study this, we employ Monte Carlo methods and introduce two estimators based on weights $\\hat{\\alpha}\\_{i}$ and $\\hat{\\alpha}\\_{i}^{\\prime}$. Notably, the Monte Carlo estimator using $\\hat{\\alpha}\\_{i}$ corresponds exactly to the commonly used empirical AURC. This is to say, the weight estimators in Eqs. (12), (13), and (17) are equivalent, leading to the empirical AURC formulation given in Eq. (10) or Eq. (11). Our statistical analysis of this estimator—including its bias, MSE, and convergence rate—shows that this estimator is, in fact, principled from a theoretical perspective. The estimator with $\\hat{\\alpha}_{i}^{\\prime}$ is a novel contribution of this paper, and we establish a connection between the two estimators in Eq. (20). The statistical analysis of both estimators is new and, to our knowledge, has not been addressed in the existing literature.\n\n***Response to Reviewer eX5V:***  \nThanks very much for your comments. We direct you to our general reviewer response, in particular the section about supplementary material.\n\n***Response to Reviewer cuHV:***\n\n**About Claims and Evidence:**  \nAlthough we show that the population AURC metric can be optimized using batch training, this is only a minor aspect of the paper and not its central focus. The core contribution of this work lies in characterizing the statistical properties of these two estimators. We not only establish their convergence rates theoretically in Proposition 3.6, but also empirically validate these results across multiple models (i.e. VGG13/16, ResNet 20/56/110/164, Bert, RoBERTa, ViT and Swin models) and diverse datasets (i.e. CIFAR10/100, ImageNet, Amazon).\n\nThe reviewer states that \"Needs more evidence evaluating AURC optimization (convergence, computational difficulty, etc.)\" We wholeheartedly disagree. As shown in Figures 3–6 and Figures S9–S28 (in total **152** Figures), we have provided an overwhelming amount of evidence of the optimization and convergence of the estimators. The computational complexity is $\\mathcal{O}(n\\log n)$ due to the need to sort the points by the CSF, and requires only a linear number of forward passes. Theoretical convergence rates are rigorously proved and demonstrated empirically.\n\n**Experimental Designs Or Analyses:**  \nIn terms of AURC estimation, [1] proposed the SELE and claimed $2\\times$SELE serves as an upper bound for the empirical AURC. Both SELE and its upper bound are used as baselines in our comparison. In terms of estimation performance, our two proposed plug-in estimators significantly outperform the SELE, which we show to be inconsistent. We demonstrate that they possess strong statistical properties—such as asymptotic unbiasedness and provable convergence rates—and validate these findings empirically through extensive experiments. These results provide strong evidence that our plug-in estimators are reliable for estimating the population AURC, whereas SELE and $2\\times$SELE are not. In terms of AURC optimization, our estimators achieve performance comparable to the SELE score and both outperform models trained with the standard cross-entropy loss. While this optimization result is not the main focus of the paper, it serves to demonstrate that the population AURC can, in fact, be optimized using our estimators as loss functions within a batch training framework.\n\n**Relation To Broader Scientific Literature:**  \nOur paper shows that AURC can be interpreted as a redistribution of the risk function, where the redistribution is guided by the confidence score function used to rank the dataset. This interpretation is the key characteristic of the population AURC. While we acknowledge that many other uncertainty estimation methods could potentially be integrated with AURC, this is not the main focus of our study. Instead, our work centers on AURC estimation and the statistical properties associated with it. Moreover, we empirically examine the AURC combined with uncertainty estimation methods such as MSP, Negative Entropy, MaxLogit, Softmax Margin, MaxLogit-$\\ell_2$ norm, and Negative Gini Score. Our results, presented in Figure 6 and Figures S26–S28, show that the plug-in estimators consistently outperform SELE and $2\\times$SELE, regardless of the chosen uncertainty estimation method. This is sufficient to demonstrate that SELE and $2\\times$SELE are not reliable AURC estimators.", "claims_and_evidence": "The submission claims that Selective Classifier Learning (SELE) has some drawbacks – specifically, it learns to predict uncertainty scores based on outputs of pretrained models, and that the proposed approach of joint selective classification learning and uncertainty estimation can alleviate this issue. The results show that their framework outperforms SELE in terms of MAE.", "methods_and_evaluation_criteria": "The risk-coverage curve is a standard tool for selective classification. When we need to abstain from predicting, coverage becomes an important metric of interest. Pairing coverage with performance on the test set in the form of risk, gives us the risk-coverage curve, the area under which is a suitable metric for evaluating selective classifiers.", "theoretical_claims": "I took a high-level look at the proofs, but they are beyond the area of my expertise. It would be best if someone with more expertise would kindly check them.", "experimental_designs_or_analyses": "The paper formulates a selective classifier based on the area under the risk coverage curve (AURC), and estimation via Monte Carlo methods. Instead of treating the AURC as a metric, the proposed approach suggests directly optimizing this metric during training.", "relation_to_broader_scientific_literature": "Seamless integration between safety critical system and machine learning models depends on frameworks such as selective classification. Understanding (and estimating) the metrics that govern these types of frameworks is a step-forward towards deployment of machine learning models in real-world safety critical system.", "essential_references_not_discussed": "The related works section details evaluation metrics used for selective classification, especially focusing on metrics related to the area under the risk coverage curve. This section also discusses approaches to uncertainty estimation and why the logit scaling over ensembling is chosen.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BIeOQjuAe4", "reviewer": "Reviewer_cuHV", "summary": "Introduces population Area Under the Risk-Coverage Curve (AURC) for selective classifiers. Statistical properties of these estimators are analyzed and evaluated on CIFAR datasets with VGG models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response to Reviewer cuHV: (continued): Please refer to the initial part of this response above.***\n\n**Essential Reference Not Discussed:**  \nWe agree that learning with rejection is related to AURC in relation to selective classifiers, where the model includes a reject option. However, this line of work is not directly relevant to the literature on AURC estimation, which is the primary focus of our study. As highlighted in [3], in the learning-with-rejection setting, the rejection threshold is learned by optimizing an explicit cost function (e.g., Eq. (9) in [3], Eq.(1) in [2]. This work primarily focuses on learning the rejector and evaluating models equipped with a rejector. In contrast, the objective of AURC is fundamentally different—it is an evaluation metric that measures the expected selective risk across all possible threshold values induced by the data. Optimizing AURC is equivalent to finding a model that achieves consistently better accuracy across the entire range of rejection thresholds. Although [3] uses the Area Under the Accuracy-Reject Curve (referred to as AURC in their Sec 3.2) to evaluate the trade-off between model accuracy and rejection, the intuition behind their Fig.7 aligns with the Area Under the Risk-Coverage Curve (AURC) used in our paper. However, they treat AURC purely as an evaluation metric and do not attempt to estimate or optimize it, which is the central goal of our work. While these papers provide interesting background on selective classification, they are not directly relevant to AURC estimation and analysis. Nevertheless, we will cite them in the final version in the introduction section as related work.\n\n**Weakness:**  \nAs noted in [3], AURC is an important evaluation metric for analyzing the risk-coverage trade-off. We believe it is valuable to understand the characteristics of this metric more deeply. In our work, we provide a novel definition of the population AURC (see Definition 3.2), framing it as a reweighted risk function, where the weighting term admits a closed-form expression (see Eq. (7)). To estimate the population AURC, we introduce two plug-in estimators derived via Monte Carlo methods. One of these corresponds to the widely used empirical AURC found in existing literature, while the other—based on the novel weight estimator $\\hat{\\alpha}^{\\prime}_i$—is proposed in this paper. As highlighted in this paper, this novel formula achieves a computation cost $\\mathcal{O}(n\\log(n))$, offering a significant improvement over the naïve empirical AURC implementation. Importantly, we also establish convergence rates for both estimators, which, to our knowledge, have not been addressed in prior work. While [1] claims that “SELE loss is a close approximation of the AURC and, at the same time, amenable to optimization,” our results show that SELE is not a reliable estimator for the population AURC when compared with our estimators. Furthermore, our findings show that AURC can be effectively optimized via batch training using our estimators as loss functions, yielding significant improvements over standard cross-entropy risk minimization. We believe these results are important for the application of AURC estimation and optimization.\n\n**Other comments:**  \nThanks! These are indeed the directions we're considering for future work. However, our current goal is to first publish this paper, which provides a solid foundation for AURC estimation, independent of those other directions. AURC is an interesting metric in its own right and deserves a proper analysis—both in terms of its characteristics and how it should be estimated.\n\n**Q1:** These are not marginal improvements—they provide precise guidance on how AURC should be estimated and optimized. While SELE may achieve results comparable to our proposed estimator in terms of optimization, it is still not a reliable estimator of the population AURC. As we have emphasized, optimization is only a minor aspect of this paper, and we are comfortable with the fact that SELE performs similarly in that regard. Our primary focus is on the estimation side, for which we provide extensive theoretical and empirical analysis. In particular, our paper shows what convergence rates can be expected for these plug-in estimators, providing a theoretical underpinning for further analysis of the AURC objective.\n\n**Q2:** Estimation under distribution shift is an crucial and interesting direction, but it is beyond the scope of this paper. Our theoretical characterization of AURC and the convergence rates we establish provide a solid foundation for future work on understanding the effects of distribution shift and related challenges.\n\n**Reference**\n\n[1] Franc, V., Prusa, D., & Voracek, V. (2023). *Optimal strategies for reject option classifiers*. JMLR.\n\n[2] Cortes, C., DeSalvo, G., & Mohri, M. (2016). *Learning with rejection*. In ALT 2016.\n\n[3] Hendrickx, K., et al. (2024). *Machine learning with a reject option: A survey*. Machine Learning.", "claims_and_evidence": "Needs more evidence evaluating AURC optimization (convergence, computational difficulty, etc).\n\n\n\n\nWhile mathematically sound, the practical benefits of the proposed approach over existing methods are not convincingly demonstrated through compelling real-world examples.", "methods_and_evaluation_criteria": "Only one baseline (SELE) is compared. Other selective classification methods should also be evaluated as baselines.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Improvements over SELE baselines are marginal. Analysis or discussion of computational overhead of the proposed estimators should be compared to baselines.", "relation_to_broader_scientific_literature": "Connection to broader literature on uncertainty estimation in deep learning seems superficial", "essential_references_not_discussed": "Missing discussion and comparion of previous work in learning with rejection:\n* https://cs.nyu.edu/~mohri/pub/rej.pdf \n* https://arxiv.org/html/2107.11277v3", "other_strengths_and_weaknesses": "Limited practical applicability and novelty. The novelty compared to existing work on AURC seems incremental.", "other_comments_or_suggestions": "How to generalize beyond classification accuracy. What about calibration, robustness to distribution shift, or performance on minority classes?", "questions_for_authors": "* What are the real-world implications of these marginal improvements?\n* How do your estimators perform under distribution shift or with imbalanced datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DlIbeq0ZnL", "reviewer": "Reviewer_3mMn", "summary": "This work addresses estimation of AURC risk aware classification by Selective Classifiers (SC). \nIt considers the theoretical properties of a pre existing AURC estimators, specifically SELE and proposes several new population based estimators with improved convergence properties. \nFurthermore, they conduct empirical analysis of their findings to confirm the properties predicted by theory.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response to Reviewer 3mMn:***\n\n**Claims and Evidence:**  \nWe largely agree that our analysis focuses on the two proposed Monte Carlo estimators. In fact, one of them is equivalent to the widely used plug-in estimator, and our results confirm the effectiveness of this plug-in estimator from a theoretical perspective. Moreover, our analysis provides practical guidelines on which estimator to use and demonstrates that the novel estimator based on $\\hat{\\alpha}^{\\prime}_i$ achieves the same convergence rate as the plug-in estimator.\n\n**Comments.**  \nThanks and we'll correct this format issue in the final version.\n\n**Question.**\n\n**Q1:** A straightforward way to address this issue is to consider the following expression:\n\n$$\n\\mathbb{E}\\_{f\\sim \\mathbb{P}(f|\\mathcal{D})}[\\text{AURC}(f)] = \\mathbb{E}\\_{f\\sim \\mathbb{P}(f|\\mathcal{D})}\\left[ \\mathbb{E}\\_{x} \\left[ \\mathcal{R}(f,g) \\mid \\tau = g(x)\\right] \\right]\n$$\n\nwhere $\\tau$ is the threshold value. This can be computed directly using our method with Monte Carlo sampling. By applying Fubini’s Theorem, we can exchange the order of the two expectations, leading to:\n\n$$\n\\mathbb{E}\\_{f\\sim \\mathbb{P}(f|\\mathcal{D})}[\\text{AURC}(f)] = \\mathbb{E}\\_{x}\\left[ \\mathbb{E}\\_{f\\sim \\mathbb{P}(f|\\mathcal{D})} \\left[ \\mathcal{R}(f,g) \\mid \\tau = g(x)\\right] \\right].\n$$\n\nSince $g(x)$ depends on $f(x)$ under the posterior distribution $\\mathbb{P}(f|\\mathcal{D})$, and the predictions are made through a full Bayesian framework, this formulation allows the evaluation of AURC in a way analogous to the standard AURC for a fixed model. We can envision several ways to define potential quantities of interest based on model uncertainty. Many of these quantities could be potentially connected to AURC and epistemic risk, and exploring these relationships could open up valuable avenues for further investigation.\n\n**Q2:** Given an explicit cost function, it can naturally be incorporated into a loss function. If a cost is explicitly assigned to abstention, then evaluating AURC becomes unnecessary. Instead, with appropriate analysis—such as that in [2]—one can identify the optimal operating point that determines a threshold on the confidence score function (CSF). This is a well-studied area in the literature. However, it is outside the scope of this paper, as our focus is on analyzing the entire risk-coverage curve. Unlike approaches that rely on a fixed cost, AURC evaluates performance across all possible cost trade-offs, providing a broader view of model behavior under selective prediction.\n\n[2] Cortes, Corinna, Giulia DeSalvo, and Mehryar Mohri. \"Learning with rejection.\" Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27. Springer International Publishing, 2016.", "claims_and_evidence": "The claims are that the two proposed estimators have improved convergence properties compared to established approaches.\nThe claims are backed theoretically and largely check out empirically.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense.", "theoretical_claims": "The theoretical section is large, the proofs were superficially checked and appear to be in good order.", "experimental_designs_or_analyses": "The experiments were set up reasonably. Several datasets were used, effect of 0/1 as well as CE loss were investigated. \nThe proposed population estimators were also used to tune the models successfully.", "relation_to_broader_scientific_literature": "This paper falls into the broader work on risk aware machine learning.", "essential_references_not_discussed": "None to extent of my knowledge.", "other_strengths_and_weaknesses": "The strength of the paper is in depth analysis of statistical properties of AURC estimators.", "other_comments_or_suggestions": "Minor point: formulas in Proof 3.4 overflow into the page margins.", "questions_for_authors": "1. The theoretical results derived apply to a fixed classifier. Can this approach be generalized to the epistemic risk with respect to multiple modes of the posterior distribution $p(\\mathbf{w} \\mid \\mathcal{D})$ ?\n2. How would introducing a cost function $c$ affect the properties of the estimators?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lmVdqOOnEF", "reviewer": "Reviewer_BrTV", "summary": "This paper concerns evaluating the performance of *selective classifiers*, where the classifier has the option of abstaining from making a prediction when the confidence is low. For such classifiers, the Area Under the Risk Coverage curve (AURC) has been a commonly used (population) metric for evaluating their performances. This paper finds an equivalent representation of AURC, based on which an estimator is provided. Asymptotic properties of the estimator are analyzed and the proposed method is evaluated in simulations and real data examples.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Response to Reviewer BrTV:***\n\n**Q1:** See general response from \"Notably, the Monte Carlo estimator using $\\hat{\\alpha}_{i}$...from a theoretical perspective.\"\n\n**Q2:** Take Fig. 3 as an example. When the confidence intervals are wide, it is typically due to the estimator being evaluated with a small sample size. This is expected, as small $n$ naturally leads to higher variability. In our analysis, we show that the convergence rate for our estimators is $\\mathcal{O}(\\sqrt{\\ln(n)/n})$, which explains the wider intervals at smaller sample sizes. Nevertheless, as the sample size increases, the variance—and hence the confidence intervals—of the estimators converge as expected. With a sufficiently large sample size, both Monte Carlo estimators demonstrate consistency.\n\n**Q3:** In Table 1, we show that our estimators perform comparably to SELE and outperform standard cross-entropy risk minimization. While [1] describes the SELE score as a “computationally manageable proxy of AuRC”, we demonstrate that it is possible to directly optimize consistent estimators of AURC using our formulations—rather than relying on a lower bound, as SELE does. In general, optimizing a lower bound is often less effective than optimizing the true objective or an upper bound. We are not concerned by the fact that SELE achieves similar results in terms of optimization—indeed, optimization is not the primary contribution of this paper. Rather, our findings illustrate that population AURC can, in fact, be optimized via batch training using our estimators as loss functions. The main focus of this work lies in the estimation of AURC and the theoretical foundation that supports it—not in advocating for AURC optimization. Regarding SELE, although [1] claims that $2\\times$SELE serves as an upper bound for the empirical AURC (Theorem 8), we show in Appendix Section A.2 that this is not the case. Furthermore, our empirical results indicate that SELE is not a consistent estimator of the population AURC.\n\n[1] Franc, Vojtech, Daniel Prusa, and Vaclav Voracek. \"Optimal strategies for reject option classifiers.\" *Journal of Machine Learning Research* 24.11 (2023): 1-49.", "claims_and_evidence": "The major claim of the paper is that it provides a new estimator for AURC that is consistent asymptotically and computationally efficient. The claim is supported by proofs and numerical evidence.", "methods_and_evaluation_criteria": "The estimator is constructed by first establishing an equivalent formulation of the population quantity and then plugging in sample-version estimates. I wonder, what if one uses the estimator in Equation (10) or (11) directly (for example, in the numerical studies)? What is the advantage of the proposed method over it?", "theoretical_claims": "I briefly went over the outline of the proof for the consistency.", "experimental_designs_or_analyses": "Yes, I have checked the experimental design. My questions are:\n1. What if one uses the estimator in Equation (10) or (11) directly (for example, in the numerical studies)? What is the advantage of the proposed method over it? \n2. In some of the plots, the confidence intervals are quite wide and the performance of the candidate methods is not quite distinguishable. I wonder if more accurate results can be presented.\n3. What is the take-away of Table 1? There do not seem to be significant differences across different methods.", "relation_to_broader_scientific_literature": "This paper finds an equivalent representation of AURC, based on which an estimator is provided. Asymptotic properties of the estimator are established.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "NA.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "1. What if one uses the estimator in Equation (10) or (11) directly (for example, in the numerical studies)? What is the advantage of the proposed method over it?\n2. In some of the plots, the confidence intervals are quite wide and the performance of the candidate methods is not quite distinguishable. I wonder if more accurate results can be presented.\n3. What is the take-away of Table 1? There do not seem to be significant differences across different methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors study the Area Under the Risk-Coverage Curve (AURC), an important metric in the context of selective classification. They  present a formal statistical formulation of population AURC and derive empirical AURC plug-in estimators for finite sample scenarios using Monte Carlo methods. Statistical properties such as consistency are shown for these estimators, and convergence rates are analysed. Empirically, the effectiveness of the estimators is evaluated in several experimental studies. 

Overall, this is a reasonably strong paper with a solid contribution to selective classification. The reviewers had some comments and made some minor suggestions for improvement, which the authors could address in their rebuttal.
Decision: Accept (poster)