Weighted ROC Curve in Cost Space: Extending AUC to Cost-Sensitive Learning
Authors: Huiyang Shao
Qianqian Xu
Zhiyong Yang
Peisong Wen
Gao Peifeng
Qingming Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: AUC
Cost Learning
Bilevel
machine learning

Abstract: In this paper, we aim to tackle flexible cost requirements for long-tail datasets, where we need to construct a (a) cost-sensitive and (b) class-distribution robust learning framework. The misclassification cost and the area under the ROC curve (AUC) are popular metrics for (a) and (b), respectively. However, limited by their formulations, models trained with AUC cannot be applied to cost-sensitive decision problems, and models trained with fixed costs are sensitive to the class distribution shift. To address this issue, we present a new setting where costs are treated like a dataset to deal with arbitrarily unknown cost distributions. Moreover, we propose a novel weighted version of AUC where the cost distribution can be integrated into its calculation through decision thresholds.  To formulate this setting, we propose a novel bilevel paradigm to bridge weighted AUC (WAUC) and cost. The inner-level problem approximates the optimal threshold from sampling costs, and the outer-level problem minimizes the WAUC loss over the optimal threshold distribution. To optimize this bilevel paradigm, we employ a stochastic optimization algorithm (SACCL) to optimize it. Finally, experiment results show that our algorithm performs better than existing cost-sensitive learning methods and two-stage AUC decisions approach.

Weighted ROC Curve in Cost Space: Extending AUC to Cost-Sensitive Learning
Huiyang Shao 1.2
Qianqian Xu
Zhiyong Yang²
Peisong Wen 1.2
Peifeng Gao²
Qingming Huang 1,33 11.
Key Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS
School of Computer Science and Tech., University of Chinese Academy of Sciences
BDKM, University of Chinese Academy of Sciences shachuiyang211maill amaiil ucas lusssacc.om en xugianiianiict.ace wenpeisong20220cct.ee ac gaopeifeng21 @mails. wca..ca..... ac. rangzhiyong2@@uca. cn guhuangOucas. cn
Abstract
In this paper, aim to tackle flexible cost requirements for long-tail datasets, where we need to construct a (1) cost-sensitive and (2) class-distribution robust learning framework. The misclassification cost and the area under the ROC curve (AUC) are popular metrics for (1) and (2), respectively. However, limited by their formulations, models trained with AUC are not well-suited for cost-sensitive decision problems, and models trained with fixed costs are sensitive to the class distribution shift. To address this issue, present new setting where costs are treated like a dataset to deal with arbitrarily unknown cost distributions. Moreover, we propose novel weighted version of AUC where the cost distribution be integrated into its calculation through decision thresholds. To formulate this setting, we propose novel bilevel paradigm to bridge weighted AUC (WAUC) and cost. The inner-level problem approximates the optimal threshold from sampling costs, and the outer-level problem minimizes the WAUC loss over the optimal threshold distribution. To optimize this bilevel paradigm, we employ stochastic optimization algorithm (SACCL) which enjoys the same convergence rate (O(e +)) with the SGD. Finally, experiment results show that our algorithm performs better than existing cost-sensitive learning methods and two-stage AUC decisions approach.
1 Introduction
Receiver Operating Characteristics (ROC) is a popular tool to describe the trade-off between the True Positive Rate (TPR) and False Positive Rate (FPR) of a scoring function. AUC is defined by the area under the ROC curve [17, 18]. This metric naturally measures the average classification
performance under different thresholds and is widely used (e-g., disease prediction [19], and anomaly
detection [29]). Compared with accuracy, AUC is insensitive to the threshold and cost [7], making it
be a popular metric for long-tail learning [32] and achieve remarkable success [24, 44, 26].
Similar to AUC optimization, cost-sensitive learning is a common data mining method [10, 2, 4]. The main goal is to incorporate the misclassification costs in the model, which is more compatible with realistic scenarios (e.g., the cost of misdiagnosing a disease as healthy is greater than the counterexample). Over the past transferred to cost space by utilizing a threshold choice method, this is equivalent to computing the area under the convex hull of the ROC curve [21]. In this way, AUC can be seen the performance of the model with a uniform cost distribution [16]. However, AUC considers all situations, which not focus more hard samples, Partial AUC (PAUC) is proposed extension of AUC with
decades, researchers have pointed out that the ROC curve be
Cooresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Comparison with existing classification settings. Cost distribution represents the cost condition of each setting.
Different setting
Formulation
A
Attr.2
Cost distribution
Vie
Cost learning
Ea x p(l]=) +1 (1 (1- m) -1-=)))(0(x))
+ 211
AUC/PAUC
B. (T()))))))R((T)) T~ ((,
V
Ulo.))
WAUC
([TPR())TPR((WWG)) ~(7)~belal(
J
betels, F)
Our method
E (PPR())FPR(()
T' E aig CCOST
fis)
truncated uniform cost distribution [3]] Recently, some studies extend PAUC using parameterized cost distributions and propose WAUC to real-world applications [16, 30].
Whether we use AUC or cost learning, our main purpose is to train models with these attributes: Attr.1: The trained model can be robust to class distribution shift in the test without class prior.
Attr.2: The trained model can be robust to cost distribution in the test without cost prior.
However, the best of knowledge, there few methods train model have both of these attributes. According to Tab. 1, both AUC-related methods and cost-sensitive learning require a strong prior knowledge of the cost distribution; (1) The cost learning mainly considers a specified cost and class imbalanced radio Models trained under this method are sensitive to class distribution, which does not apply the scenario where data distribution with offset. (2) AUC (PAUC)
assumes that the cost distribution belongs to (truncated) uniform cost distribution U(a, b). Models trained with them will have poor performance when the true cost distribution is not uniform [16]. (3) WAUC considers optimizing models based on more complex forms of cost distribution, such as beta(u, (a, b). However, we can not obtain the cost prior real problem scenarios, e.g., financial market prediction [14]. Considering the weakness comes from the existing settings, we will explore the following question in this paper:
Can bridge AUC and complicated cost distribution to training robust model on desired cost-sensitive and arbitrary class imbalanced decision scenarios?
To answer this question, we propose a view that, in some real applications [14], the cost, like the instance data, not available prior but can be obtained by sampling. Therefore, we choose to sample desired cost to approximate the true cost distribution. Different from previous settings, ours is closer to real world, the main process can be divided into three parts:
Firstly, we sample some desired costs to construct the empirical cost set. Step.2 Data Sampling: Next, we sample some instance data to construct the empirical dataset. Step.3 Build Formulation: Finally, construct the appropriate formulation to maximize the per- formance in different desired costs and ensure model is robust to distribution shift.
Step.1 Cost Sampling:
It is natural for us to ask the question: Can we use the existing methods to realize this process? It's clear the answer is no. For AUC-related methods, they can not perform Step. and for cost-sensitive learning, they fail to achieve robust distribution shift and multiple costs in Step.3 (as shown in Fig. I (orange line). Hence, we propose novel bilevel formulation combining the advantages of WAUC and cost learning. The inner-level process calculates the optimal threshold from sampling costs, and the luter-level process minimizes the WAUC loss over the optimal threshold distribution. The method can help the model improve robustness to class distribution in cost-sensitive decision problems. The main process is shown Fig. (green line). We summarize our contributions below:
We propose a setting that focuses on the robustness of the model to the class distribution and cost distribution simultaneously. This setting treats cost as data that can be sampled, not prior information, which is closer to the real-world cost-sensitive scenario.

---

Existing setting
ftl
f(4)
Set prior cost
model
(c) minn (.,r)
Our
Sample empirical
optimal threshold
Train model
Dosparanari ditrtbutios)
135 niight
polns
(4) Population cost distribution
(b) Diserete cost distribution
(c) Threshold distribution
(d) Train metries
(c) Test model
Figure 1:
The comparison of our proposed setting with the previous setting. The orange line represents
the previous cost-sensitive learning approach, and the green line represents our method.
We present a bilevel paradigm where the inner cost function is an inner constraint of reformulate this paradigm into outer WAUC optimization. For sake of optimization, nonconvex-stronggl convex bilevel form. Moreover, we employ a stochastic optimization algorithm for WAUC (SACCL), which can solve this problem efficiently.
We conduct extensive experiments on multiple imbalanced cost-sensitive classification tasks.
The experimental results speak to the effectiveness of our proposed methods.
2 Observation and Motivation
In Tab. we compare the existing methods with ours from different views. However, the table comparison does not have very visual presentation. In this section, will analyze the disadvantages of existing settings and explain our motivation. We train the model with all methods on a Cifar- 10- Long- Joil training set under the different imbalanced ratios and cost distribution. We visualize the feature representation (the last layer's 's output of the trained model) in test data by t-SNE. The blue point represents negative samples predicted by the optimal threshold, and the orange point represents positive samples. The smaller the overlap between them, the better the performance. From Fig. 2, make the following remarks: (1) According to Fig. (a), AUC is robust changes in the imbalance ratio but completely not applicable with the cost distribution. (2) According to Fig. 2 (b), cost learning can process different cost distributions, but is sensitive to imbalance ratios.
(a) Tine revalt of reperoentation (ABC)
(I) Taae noult of ruponsentation (COST)
(c) Tane roult of repeasentation (Ours)
Figure 2: The feature representations comparison among different methods. (a) AUC optimization.
(b) Cost learning. (c) Ours result. We solve the optimal threshold with the LCoST (defined in Sec. 3). æ denotes the probability of positive class, e denotes the cost ratio misclassification (U denotes Uniform, N denotes Normal). For example,
0.5, N means model tested on dataset which has imbalanced ratio π and cost set sampled from N.
Hence, our motivation is propose new approach solve the problems in AUC and Cost- sensitive learning. As shown in Fig. 2 (c), our proposed method can have better learning results in various complex cost distributions and imbalance ratios. That means our method can overcome the shortcomings of traditional AUC and cost-learning, which perfectly fits the proposed setting.
3 Preliminaries
Notations. In this section, give definitions and preliminaries about AUC. We denote (ær y) be an instance, where is drawn from feature space X C dd (d is feature number) and is drawn from
3

---

label space y = =00 (0. Let Dp (DN resp.) be positive (negative resp.) instance distribution. Let x ~ Dp (x ~ DN resp.) be positive (negative resp.) instance. We denote S+ = {(x) (.) 3)) (S_ = [(2, [i:], resp.) as set of training data drawn from Dp (DN resp.), where n (n- resp.) denotes the instance number of S+ (S- resp.). Let (() be the indicator function, which returns 1 when the condition is true and 0 otherwise. In this paper,
focus on the deep neural network scoring function 8(w, a)) X' + [0, 1). where parameterized by w on an input x:
AUC & WAUC. For specified threshold the TPR of classifier derived from s(w, .)) measures the likelihood that accurately predicts a positive instance when getting a random positive instance from Dp. Formally, we have:
1
(Pop.) (PR,(T)= = PataD (w, xt)
r] (Emp.) (PR.
(wwx))>
n4+
In similar spirit, the classifier's ' FPR on threshold 7 refers to the probability that it predicts positive when it gets negative instance from Px.
(Pop.) FT) Pa Pa--DDlss (w, x > 7| (Emp.) FPR.(T)
71 Σ
AUC measures a scoring function's 's trade-off between TPR and FPR under uniform thresholds. Denote drawn from the distribution D,, WAUC utilizes the threshold distribution explicitly based on AUC. FPR denotes the probability density function of s(w, x:
AUC
(P),(T))PR(A
(3a)
WAUC (PP((())PP(())))))))))))
(3b)
Cost function [2]. In some real application scenarios, we need to consider the misclassification
cost. We denote C(-) as misclassification cost for class (-), cost drawn from D.. Since we could
obtain the cost distribution De, we sample empirical set S. {cr [e n. denotes the sample number
of cost c. Given a scoring function s and parameter w, the cost function LcOST is (the empirical version of cost function, LoOST contains the empirical forms of TPR and FPR):
CCosT(w,co c, T (c)) = c. n. TPR, (T" (c))) + +1 - c) (1 11-a1 x) (F))))
(4)
where π n + n-), c = c+/(c+ + c_) and (c) is optimal threshold for score function under specified e [21], the sample number n,, ne-
4 Problem Formulation
In this section, we introduce how to link the ROC curve to the cost space. First, we reformulate Eq.(3) into expectation:
AUC = E' T-U
(T).(T) -FPR(
(5a)
WAUC-
(T), (F).
(5b)
T-D
If threshold T is drawn from the uniform distribution U, WAUC will degrade to the standard AUC formulation. However, AUC only describes the global mean performance under all possible costs. If we want to extend AUC to cost-sensitive problems, maybe lifting the restriction on the uniform distribution of 7 is a good solution. Hence, we release the D, 's restriction make it belongs to complicated distribution (e.g., normal distribution, exponential distribution). Then we can extend AUG WAUC. However, using WAUC raises another question: how do we get D,? We find that ('(c) is one of parameters of LcosT(mn (w 7 (c)) ). A natural idea is to use LCOST to solve for the optimal 7" (c) and to combine the 7* (c) solved for at different e obtain D,.
(c) arg min Lcosr T.C) = =
TPR. (7)) (1 -- c) (1 - 4) FPR, (6)

---

If we couple (5)( and (6) together so that WAUC can enjoy the optimal threshold distribution in CCOST, then we can break the barrier between the ROC curve and the cost space. With the help of the threshold as bridge, we can extend the AUC metric to achieve the WAUC cost-sensitive learning. Then we give the problem formulation (intuitively, from the result of (c), (OPO) satisfies both Attr. 1 and Attr. simultaneously):
(OPO) (outer.)
WAUC = E (T) [T) Tw-T
(7)
(inner.)
T = (T = arg min (oosr(w, T, c)le De}
Nevertheless, there are still three main challenges in WAUC cost-sensitive learning:
Given the scoring function 8 and negative dataset S_, how to estimate (T) in WAUC?
The inner problem nonconvex, which hard to give a theoretical convergence guarantee.
(3) How to design a formulation that can bridge WAUC and LCOST so that WAUC can be optimized over the cost distribution of the desired problem scenario?
We will address the challenge (1) in Sec. 5.1, challenge (2) in Sec. 5.2 and challenge (3) in Sec. 5.3.
5 Methodology
5.1 The Estimation of False Positive Rate
For challenge (1), choose the kernel density estimation (KDE) estimate FPR and denote as Kxx) (please see definition in Sec. C.I). Then we can address the density estimation problem. However, (5b) still exists oi-differreniable and non-smooth term [[]), which is hard to optimize. Hence, we propose the following smooth and differentiable WAUC estimator to approximate (5b).
Definition 5.1. Denote K (x) be statistics kernel with bandwidth m and S = {s(w, 23 ))
With Lemma 5.2, have the approximate estimator and loss function for WAUC:
WAÚC [
ZwAuo(w,7) 1
h(w,7)
(8)
nt
where + {
Mrr and the point loss h is defined by
R_
h(w,t)=1- =1 n4n Σ
(s(ww, ((((w,xt))) T)) mT) K((s(w, m1/m)//m.
(9)
(xx) = 1/(1 + exp(-Bx)), 8 is smooth parameter and we have (xx) 8-3 L.
Lemma 5.2. Given a scoring function 8, if is known, when the number of instances is large enough, WAUC almost surely converges to WAUC.
lim WAUC - WAUC| a.S. 0.
(10)
With KDE consistency [41], when the negative sample size large enough, Lemma 5.2 provides theoretical approximation guarantees for our proposed WAUC estimator in Prop. 5.1.
5.2 The Estimation of Threshold Weighting
For challenge (2). natural idea is ΣGoST solve for the optimal threshold #* when given the cost set S, and the scoring function 8. Then we can use the optimal threshold set 4* to calculate WAUC. Firstly, we define the solution for 4* be
4* = { 4" (c) (c) arg min Êcosr(w, c), sc}.
(11)
5

---

However, it's noticed that the arg min, EcoST in Eq.(1I) is conve. As we analyzed before, T" the inner constraint of WAUC. To the best our knowledge, there few studies on optimizing two coupled non-convex problems simultaneously with theoretical convergence guarantees. Most studies coupled optimization assume that the inner problems have good properties, such strong convexity. Hence, we propose the approximated convex formulation of the inner problem for f"
Theorem 5.3. When we set K, M large positive numbers and M < M 6nt
then have (1*+138.
the approximated convex formulation for CCOST
min r. PERR "NNER T,C) = C -=) C.
(1- 11--111 -
1 Σ N₃)
n4
n+
Pi(s(w, xxt) (xt -
7)) + Mrb(P, - + - 1)
(12)
n4
M' ('bws(w -Nj(s(w,,,,) - + MMNN - O
where (2) log(1 exp(kx))) Eeg in Eq. is pg-strongly convex w.r.t. Eq.(12) has same solution as min, ECOST when the parameters satisfy the conditions of the penalty.
Thm. 5.3 provides an optimization method with good properties. Eq. (12) adopts the penalty function to convert inequality constraints into a part of the objective function. When these inequality constraints are not satisfied, the objective function will increase to infinity. Otherwise, we will get P: = [s ww, ",, +) > r] and N, [[s(w, xz)) ) > r]. then we will get the same formulation as LCOST. When the parameters meet the requirements, (1. has the same solution as LOOST. We give the proof of Thm. 5.3 and the definition ş in Sec. C.4. Moreover, we give the analysis of approximation error between CCOST and Thm. 5.3 in Sec. B.7.
5.3 Bilevel Optimization for WAUC learning
After answering questions in challenge (1) and (2), we have solved most of the problems in WAUC cost-sensitive learning. However, there remains challenge (3) in optimization: How do design learning paradigms to solve the coupled optimization problem of WAUC and LcosT? In recent years, bilevel optimization has achieved remarkable success. This approach combine related optimization problems to form a coupled optimization formulation. Hence, with Prop. 5.1 and Thm.5.3, we propose a bilevel paradigm formulate this coupled optimization problem.
(OP1) (outer.) min F(w) == f(w, T") = ZwAuC(w,
(13)
(inner.) 7" = arg min g(w, T) = - H T T.P..No
(e (w
The
where Pa E X and N E xn (OP1) describes bilevel optimization formulation for WAUC cost-sensitive learning, where the inner-level provides a threshold optimization process, and the outer-level minimizes the WAUC loss over the optimal threshold distribution. Moreover, this formulation is consistent with the mainstream bilevel optimization problem (outer-level is smooth and non-convex, inner-level is convex and smooth), which enjoys a faster convergence rate.
Optimization Algorithm
In this section, we focus on optimizing (OP1) in an end-to-end manner. Hence, we propose stochastic algorithm for WAUC cost-sensitive learning shown Alg. which is referred to SAACL.
6.1 Main Idea of SAACL
We provide some intuitive explanations of our algorithm. each iteration k SACCI alternates between the inver-level gradient update and the outer-level gradient update on During

---

Algorithm Stochastic Algorithm for WAUC Cost-sensitive Learning
Input: training data S+ and S_, iteration numbers and T' batch size B. Initialize: parameters wy E R" To R"r, stepsizes ogky Skg for k 0 to K do
set T Tk-
for t = 0 to T do
drawn B {(rb, [b)] from S+ and S_ uniformly.
Va E Se. 7 kk1t11 TR,t" 5x V q;; B²).
end for
set Th+1 = TR.T
drawn B₁ = {(1g, yb )= from S and S_ uniformly.
Wk ax. V w f Tk+1 Bx) V g( Wk, Th+1; Ba).
Wk+1
Lg.1 n= [I
N N'
1 ( Bk+1: -1)) V V (wk,, Th- BB)
Lng
end for
iteration k, we update Te, with standard SGD T steps to ensure that Tk+1 is as optimal as possible. After updating inner-level variables, we perform outer-level optimization with Tk+11 the parameter to update Wn. Notice that will not take a large value to ensure the validity of the coupling update of t and Let Or and B& be stepsizes of and that have the same decrease rate SGD). We denote n be the number of elements in deep neural network parameters wt
6.2 Convergence Analysis of SAACL
In this subsection, we present the convergence analysis for SAACL. We give some Lipschitz continuity assumptions that are common in bilevel optimization problems [11, 28].
Assumption 6.1. (Lipschitz continuity) Assume that f, Vf, and Vg are respectively Ly,., Lips-hipsc continuous.
Assumption 6.2. (Bounded stochastic derivatives) The variance of stochastic derivatives VSW,, T) and Vg(w,, (3) are bounded by f,1. 2 respectively.
Based on Assumption 6. and Assumption 6.2, following [5], Thm. 6.3 indicates that we can optimize (OP1) with the same convergence rate the traditional SGD algorithm.
Theorem 6.3. Suppose Assumption 6.1 and 6.2 hold. We define
pl. 0 (u+ 1 (8T by + 2n G3n1)
α = 2LF + L + 2L Lyz/(LL)))
(14)
where LF Ly Lp, Lgg, Ly and C7 come from Lem. 2 and Lem. 4 in [5]. We select the following stepsize as
8L, + 2n C7a1
min ④1 02
BR
O
(15)
VK
4Tp
For any T 2 1, the iteration sequence {wk) ) k=1 and 1K
{K generated by Algorithm 1 satisfy
K-1
(3MRe* /aakee//( (e* + +1)² 1.g.1
2
Σ
+0
(16)
K
24M + 1)2
TVK
VK
Ly where = 2002 g.1 To
1 1+551Lyā + 19
9 at
(8L L, + 2n
Remark 6.4. When and M are large enough positive integers, according Eq. (16), Alg. 1 is still guaranteed to find a e-stationary point within O(e-a) iterations (e is error tolerance).
Experiments
In this section, we conduct series of experiments for WAUC cost-sensitive learning common long-tail benchmark datasets. Due to space limitations, please refer to Sec. B for the details of our experiments. The source code is available in supplemental materials.
7

---

7.1 Dataset Details
We use three datasets: Binary CIFAR-1O-Long-Tee Dataset [23], Binary CIFAR-100-Long-T11 Dataset [23]. and Jane Street Market Prediction [14]. Binary CIFAR-10-Long Tail Dataset and Binary CIFAR-I00-Long 100-Long-Taal Tail Dataset are common datasets in long-tail learning, and we construct their cost distributions. Jane Street Market Prediction is data from real cost-sensitive learning application scenarios. For all datasets, we divide them into the training set, validation set, and test set with proportion 0.7:0. 17::0.15:0.15 15. All image data is normalized to ensure more stable training process.
Table 2: Performance comparisons benchmark datasets with different metrics. The first and second best results are highlighted with bold text and underline, respectively.
Subsetl
Subse12
Subset3
AUC
dataset
type
methods
WAUC
Ecosr
WACC
EcosT
WAUC1
Goosr
Subeet
Subset?
Subeet3
BCE
0.525
0.1127
0.533
0.015
0.318
(1009
0.822
(1566)
0,870
ELAUIC
0.516
0.029
0.518
0.013
0.366
0.028
0.845
0.963
0.855
S4AUC
0.407
0,028
0.548
0.012
0.327
0.031
0.811
0.923
0,867
NWAUC
0:565
0.030
0.574
0.017
0.396
01027
0.786
0.385
0.827
Competitors
PAUC
0.549
0.029
0.508
0.015
0.354
0.028
0.650
0.501
0.736
CIFAR-I0LLT
PALCC-poll
0.526
0.028
0.420
0.029
0.651
0.742
PATICT
0.027
0.088
0.704
0.734
hinge
0.026
0.02
0.675
0.762
AdaCTS
0.576
0:025
0.559
0.391
0.023
0.758
0.742
ECL.
0.589
0.026
0.561
0.014
0.388
0.000
0.694
0.918
0.763
WALCC-Gal
0.679
0.024
0.660
0.12
0:467
0.015
0.787
0.934
0.843
method
WAL Log
0.653
0.023
0.674
0.011
0.468
0.014
0.820
0.988
0.802
BCE
0.556
0.022
0.463
0.012
0.512
0.019
0.912
0.957
0.806
EXAUC
0,522
0,019
0,502
0.011
0.506
0.017
0,933
0.967
0.833
SqAUC
0-483
0.024
0.367
0.015
0.474
0.018
0.889
0,955
E.855
NWAUC
0.654
0.025
0.511
0.016
0.631
0.019
0.867
0.925
0.807
Competitors
PAUC
0.464
0.020
0.016
0.787
CHARR-00-LT
PAITC-poly
0.022
0.017
0.791
PAUCI
0.018
0.188
0.812
0.822
C.-hhing
0.525
1:017
0.457
0.515
0014
0.734
0.10
0.716
AdsCCSS
0.590
0.018
0:474
0011
0.587
0.16
0.769
0.919
0.727
ECL
0,583
0.017
0.497
0.009
0.595
0.015
0.863
0.939
0.794
WALTC-Gsm
0.745
0.015
0.509
0.005
0.728
0.013
0.842
0 07978
0.745
method
WAUC-Log
0.719
0.012
0,560
0,003
0.745
0.810
0.905
0.960
0,875
Table 3: Performance comparisons on benchmark datasets in real world cost-sensitive problem. Profit means represents the money earned by the model over the entire trading period.
Medaed
ECALC
SwAtTC
PABIC 4133
PAIIC pely
PAECT
CS-aing
WALCC Gas
WAlTE-1ag
W-UCC
p-tea x65
567
57t
0:508
6-190 160
E1653 805
007
0.67
Foner
0375 + 20
0:366 663
4.761 10"
0.730 011
(1741 (m))
0:33 - 106
0779 AM
0:300
107
8.389 011
01711 100
ADO1
0.22-2.000 .85
0,59 2 JMM
B223-0 JC3
0.331 z 194
0.221 am
0.30 .a8
0210: 200
0223 = 100
02346 2: 802
0,30 = 103
0,336 012
0.336.1. /90
7.2 Overall Performance
In Tab. 2 and Tab. 3, we collect all the methods' performance on test sets of three types of datasets. For cost distribution of we sample some data from normal distribution (00.5, 1) to construct a dataset S. (we clip all data to [0, IJ). We also conduct numerous experiments for other types of distribution of and please see Appendix B for the details. From the results, we make the following observations:
(1) For WAUC and ŻooST metric, Our proposed algorithm achieves superior performance in most benchmark datasets compared to other methods. This demonstrates that our proposed WAUC cost- sensitive learning can extend the ROC curve into the cost space. Models trained with our proposed bilevel optimization formulation can enjoy high WAUC and cost-related metrics.
(2) AUC and cost-related metrics are inconsistent. From the high-performing heatmap of Tab. 2, it can be noticed that ECOST and AUC have two completely diffferent highlight regions. This indicates that the assumption of uniform distribution of AUC does not match the realistic scenario.
(3) We also find that AUC-related and traditional classification algorithms do not perform well in cost-sensitive problems. This means that we first train the model with the classification algorithm, subsequently using the cost function to solve for the optimal threshold for decision does not work well. Meanwhile, the algorithm that can learn from scratch has better scalability. Two-stage decision method Therefore, designing stage algorithms for WAUC cost- sestssensitive learning is necessary.
8

---

7.3 Sensitivity Analysis
In this subsection, we show the sensitivity of 3, T, and bandwidth on test data.
Effect of B.. In Fig. (a) and (d), we observe that for both WAUC and cost metrics, when 8 closes to 7, the model will have the largest performance improvement and the lowest variance. This can be explained in two ways: (1) When the 3 is too small, the error between the (x) and the 0-1 loss function is large, resulting in large approximation error between the WAUC and the WAUC. (2)
When the 33 is too large, the gradient also tends to be O. Therefore, choosing a beta value that trades off the approximation error and the gradient is essential.
11
10
0.5
bandwwett
beta
0.3
0.1
0.25
0.30
0.35
0.45
0.20
0.25
0:35
0.40
0.3
0.4
Performance (WAUC)
Performance IWAUC!
Por Performance (WALC)
(a) The effect of 3 on WAUC
(b) The effect of T WAUC
The effect of m WAUC
bandwicth
0.4
0.2
0,050 0,075
0,100 0.125
0.025
0.050 0,075
0.100
0.125
0.025
0.050
0.075
0.100 0.125
Performance (COST)
Performance (COST)
Performance (COST)
(d) The effect of 3 on cost
(e) The effect of 'T' on cost
(I) The effect of vn cost
Figure 3: Sensitivity analysis on test data where WAUC and cost for WAUC-Gau with respect to 3, T, and bandwidth. The other two variables are fixed for each box in the plots, and the scattered points along the box show the variation.
Effect T. As we mentioned in the Section 6.1, choosing a smaller T can effectively improve the performance of the model. However, shown in Fig. 3(e), larger T value can reduce the variance. Hence, as set in our experiments, T = 3 is good choice ensure the average performance and variance of the model.
Effect of m. From Fig. 3(c), we find that the kernel's bandwidth strongly influences the model's performance. The model's bandwidth and performance are almost proportional; the closer the bandwidth is [0.4. 0.5]. the better the effect: otherwise, the effect is worse. This indicates that our proposed method is sensitive to the bandwidth parameter, which also compounds the bandwidth characteristics in the KDE method.
8 Conclusion
This paper focuses on extending the traditional AUC metric to associate with misclassification costs. Restricted by the assumption of cost distribution, existing settings could not describe the model's performance in the complicated cost-sensitive scenario. To address this problem, we propose novel setting that treats the cost as sampled data. We employ the WAUC metric and propose a novel estimator to approximate it. With the help of threshold weighting, we establish the correspondence between WAUC and the cost function. To describe this connection, we present bilevel optimization formulation to couple them, where the inner-level problem provides a threshold optimization process, and the outer-level minimizes the WAUC loss based on the inner thresholds. This paradigm ensures that the WAUC can always be optimized at the optimal threshold value based on the complicated cost distribution in reality. Moreover, we propose stochastic algorithm to optimize this formulation. We prove that our algorithm enjoys the same convergence rate as standard SGD. Finally, numerous

---

experiments have shown that our method can extend AUC to cost-sensitive scenarios with significant performance.
Acknowledgements
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, part by National Natural Science Foundation of China: 62236008, U21B2038, U2001 202, 61931 62122075, 61976202, and 62206264, in part by the Fundamental Research Funds for the Central Universities, part by Youth Innovation Promotion Association CAS, in part by the Strategie Priority Research Program of Chinese Academy of Sciences (Grant No. XDB28000000) and in part by the Innovation Funding of ICT, CAS under Grant No. E000000.
References
[1] J A, Hanley, B J, and McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 1982.
[2] Daniel Andrade and Yuzuru Okajima. Efficient bayes risk estimation for cost-sensitive classi- fication. In The 22nd international conference on artificial intelligence and statistics, pages 3372-3381. PMLR, 2019.
[3] Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations Research, 21(1):37-44, 1973.
[4] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classifi- cation with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pages 1507-1517. PMLR, 2021.
[5] Tianyi Chen, Yuejiao Sun, and Wotao Yin. Tighter analysis of alternating stochastic gradient method for stochastic nested problems. arXiv preprint arXiv:2106. 13781, 2021.
[6] Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153(1):235-2566 2007.
[7] Corinna Cortes and Mehryar Mohri. AUC optimization error rate minimization. In Advances in Neural Information Processing Systems, pages 313-320, 2003.
[8] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9268-9277 2019.
[9] Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 27(8):861-874, 2006.
[10] Alberto Fernández, Salvador García, Mikel Galar, Ronaldo Prati, Bartosz Krawezyk, Fran- cisco Herrera, Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, et al. Cost- sensitive learning. Learning from Imbalanced Data Sets, pages 63-78, 2018
[11] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pages 1568-1577. PMLR, 2018.
[12] Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933-969, 2003,
[13] Thore Graepel, Klaus Obermayer, et al. Large margin rank boundaries for ordinal regression.
In Advances in Large Margin Classifiers, pages 115-132. 2000.
[14] Jane Street Group. Jane street market prediction. https://wwu http://wwwwwkggl.e/e com/ 2021.
competitioons/jaeettteeeeee
[15] Zhishuai Guo, Mingrui Liu, Zhuoning Yuan, Li Shen, Wei Liu, and Tianbao Yang. Communication-effioiie distributed stochastic auc maximization with deep neural networks. In International Conference Machine Learning, pages 3864-3874, 2020.
10

---

[16] David J Hand. Measuring classifier performance: a coherent alternative to the area under the roc curve. Machine learning, 77(1):103-123, 2009.
[17] James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29 36, 1982.
[18] James A Hanley and Barbara McNeil. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148(3):839 843, 1983.
[19] Huaying Hao, Huazhu Fu, Yanwu Xu, Jianlong Yang, Fei Li, Xiulan Zhang, Jiang Liu, and Yitian Zhao. Open-narrow-synechiae anterior chamber angle classification in as-oct sequences. arXiv preprint www.ix:2006055666 2020.
[20] José Hernández-Orallo, Peter Flach, and César Ferri Ramírez. A unified view of performance metrics:
Translating threshold choice into expected classification loss. Journal of Machine Learning Research, 33:2813-8869, 2012.
[21] José Hernández-Oralle Peter Flach, and César Ferri. Roe curves in cost space. Machine learning, 93(1):71-91,
[22] Thorsten Joachims. A support vector method for multivariate performance measures. In International Conference on Machine Learning, pages 377-384, 2005.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Yunwen Lei and Yiming Ying. Stochastic proximal auc maximization. JMLP, 22(61): 2021.
[25] Jialiang Li and Jason Fine. Weighted area under the receiver operating characteristic curve and its application to gene selection. Journal of the Royal Statistical Society: Series C (Applied Statistics), (9(4):673-69) 2010.
[26] Mingrui Liu, Xiaoxuan Zhang, Zaiyi Chen, Xiaoyu Wang, and Tianbao Yang. Fast stochastic auc maximization with o(1/n) converecneeeeeeece rate. In ICML, pages 3189-3197. PMLR, 2018.
[27] Mingrui Liu, Zhuoning Yuan, Yiming Ying, and Tianbao Yang. Stochastic AUC maximization with deep neural networks. In International Conference Learning Representations, 2020.
[28] Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In International Conference on Machine Learning, pages 6305-6315. PMLR, 2020.
[29] Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a new bascline. In CVPR, pages 6536-6545, 2018.
[30] Andreas Maurer and Massimiliano Pontil. Estimating weighted areas under the roc curve. Advances in Neural Information Processing Systems, 15:7733-77422 2020.
[31] Donna Katzman McClish. Analyzing portion of the roc curve. Medical decision making, 9()): 190-195, 1989.
[32] Charles E Metz. Basic principles of roc analysis. In Seminars in nuclear medicine, volume 8, pages 283-298. Elsevier, 1978.
[33] Harikrishna Narasimhan and Shivani Agarwal. Support vector algorithms for optimizing the partial area under the roc curve. Neural Computation, 29(7):1919-1963 2017.
[34] Sakrapee Paisitkriangkrai, Chunhua Shen, and Anton Van Den Hengel. Efficient pedestrian detection by directly optimizing the partial area under the roc curve. In Proceedings of the IEEE international conference on computer vision, pages 1057-1064, 2013.
[35] Margaret Sullivan Pepe and Mary Lou Thompson. Combining diagnostic test results to increase accuracy. Biostatistics, 1(2):123-140. 2000.

---

[36] Alain Rakotomamonjy. Support vector machines and area under roc curve. PSI-INSA de Rouen: Technical Report, 2004.
[37] Shoham Sabach and Shimrit Shtern. first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization, 27(2):640-660, 2017.
[38] HuiYang Shao, Qianqian Xu, Zhiyong Yang. Shilong Bao, and Qingming Huang. Asymptoti- cally unbiased instance- wise regularized partial auc optimization: Theory and algorithm. In Advances in Neural Information Processing Systems, 2022.
[39] Luis N Vicente and Paul H Calamai. Bilevel and multilevel programming: bibliography review. Journal of Global optimization, 5(3):291-306, 1994.
[40] Sam Wieand, Mitchell H Gail, Barry R James, and Kang L James. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika, 76(3): 585-592, 1989.
[41] Dominik Wied and Rafael Weißbach. Consistency of the kernel density estimator: survey. Statistical Papers, 53(1):1-21, 2012.
[42] Lian Yan, Robert Dodier, Michael Mozer, and Richard Wolniewicz. Optimizing classifier performance via an approximation to the pelloxon-mann-whiee statistic. In International Conference on Machine Learning, pages 848-855, 2003.
[43] Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey, arXiv preprint arXiv:2203. 15046, 2022.
[44] Zhenhuan Yang. Wei Shen, Yiming Ying, and Xiaoming Yuan. Stochastic auc optimization with general loss. CPAA, 19(8), 2020.
[45] Zhiyong Yang. Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. When all we need is piece of the pie: A generic framework for optimizing two-way partial auc. In International Conference Machine Learming, pages 11820-11829, 2021.
[46] Yao Yao, Qihang Lin, and Tianbao Yang. Large-scale optimization of partial auc in range of false positive rates. arXiv preprint arXiv:2203.00 0055, 2022.
[47] Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online auc maximization. Advances in Neural Information Processing Systems, 29:451-4553 2016
[48] Zhuoning Yuan, Zhishuai Guo, Nitesh Chawla, and Tianbao Yang. Compositional training for end-to-end deep auc maximization. In International Conference Learning Representations, 2021.
[49] Adriano Z Zambom and Dias Ronaldo. A review of kernel density estimation with applications to econometrics. International Econometric Review, 5(1):20-444 42, 2013.
[50] Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, and Tianbao Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. arXiv preprint arXiv:2003.00176, 2022.
12


Reviews:
[{"review_id": "FK2VpxZ2Jq", "reviewer": "Reviewer_uhz4", "summary": "The authors propose a method that combines WAUC (weighted Area under ROC curve) learning with cost-sensistive learning. They propose a bilevel optimization algorithm to solve the formulated problem and provide theoretical analysis for convergence. According to their experiments on three datasets, the practical performance is good. The idea is interesting and the method should be novel.", "questions": "Please see weaknesses.", "limitations": "Besides of the weaknesses, it would be better if authors could provide running time report for the proposed method and other baselines.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The presentation is good. The authors raise their motivation (combines AUC with cost learning) at the beginning. And the writing flow is clear for the paper. They have several good figures to illustrate the motivations.\n2) Overall, the soundness is good. Apart from basic writings, authors also provide both practical comparison with some baselines and theoretical analysis for their algorithm convergence.", "weaknesses": "1) The method is designed to be both class distribution robust and cost distribution robust, but doesn't demonstrate better AUC performance in Table 2. Comparing with $\\widehat{WAUC}$, the $\\widehat{AUC}$ is the more suitable metric for class distribution robustness (besides, the $\\widehat{WAUC}$ is defined and only optimized by the proposed method).\n2) Miss an ad-hoc baseline that optimizes both AUC and cost sensitive learning objectives simultaneously by simply assigning different weights. \n3) The experiments should be repeated multiple times independently and report the mean&standard deviation values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Author response to Reviewer uhz4\n\nThank you for your detailed and constructive feedback on the paper. We value your insights and have taken your suggestions into consideration. Here are our responses to your specific comments.\n\n**Q(1-1) WAUC method doesn't demonstrate better AUC performance in Table 2.**\n\n**A(1-1)** We conducted three sets of experiments targeting different cost distributions (uniform, normal, beta). It can be observed that when the cost distribution follows a normal or beta distribution (see Tab.2, Tab.6), our method exhibits a lower AUC value. This is because AUC does not align with cost-sensitive scenarios. **However, when the cost distribution is uniform (see Tab.5 in the appendix), our method achieves state-of-the-art.** WAUC is equivalent to AUC when the cost ratio follows a uniform distribution $U(0,1)$. \n\n**Q(1-2) Comparing with $\\hat{WAUC}$, the $\\hat{AUC}$ is the more suitable metric for class distribution robustness.**\n\nIn fact, AUC can be considered as special versions of WAUC (we point it in introduction, line 30). WAUC is equivalent to AUC when the cost ratio follows a uniform distribution $U(0,1)$. **We have conduct experiment in Tab. 5 which demonstrate that WAUC achieves state-of-the-art.**\n\nTherefore, by definition, $\\hat{WAUC}$ emerges as the more appropriate metric for class distribution robustness. Moreover, we conducted relevant experiments in Cifar-10-Subset-1 to compare the differences between AUC and WAUC by altering various levels of class imbalance.  Assuming that we have two models (for both of them, we solve the optimal posterior threshold by $\\hat{\\mathcal{L}}_{COST}$ after training): \n - [a] model trained on AUC method, $c\\sim U(0,1)$; \n \n - [b] model trained on WAUC method, $c\\sim U(0,1)$;\n\nGiven different $\\pi(n_+/(n_++n_-))$ of data, The results are shown in the following table.\n|  method   |   $\\pi=0.1$  |   $\\pi=0.2$  |   $\\pi=0.3$  |   $\\pi=0.4$  |   $\\pi=0.5$  |   $\\pi=0.6$  |   $\\pi=0.7$  |   $\\pi=0.8$  |   $\\pi=0.9$  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|  [a]   |  0.285  | 0.271 |  0.253 |  0.217 |  0.201 |   0.192 |  0.175 |  0.186 |   0.199 |\n|  [b]   |  0.280  |  0.268  |  0.241  |  0.213  |  0.200  |  0.193  |  0.171  |  0.182  |  0.190 |\n\nIt's clear that there is no discernible disparity in class robustness between AUC and WAUC.\n\n**Q(2) Miss an ad-hoc baseline that optimizes both AUC and cost sensitive learning objectives simultaneously by simply assigning different weights.**\n\n**A(2)** Thank you very much for your suggestion. We conducted additional experiments incorporating AUC and cost-sensitive learning weighted baselines. The specific results are listed in the table below ($c\\sim \\mathcal{N}(0.5, 1)$):\n\n|  method   |   Cifar-10-Subset-1 / $\\hat{AUC}$  |   Cifar-10-Subset-1 / $\\hat{WAUC}$  |   Cifar-10-Subset-1 / $\\hat{\\mathcal{L}}_{COST}$  |   Cifar-100-Subset-1 / $\\hat{AUC}$  |   Cifar-100-Subset-1 / $\\hat{WAUC}$  |   Cifar-100-Subset-1 / $\\hat{\\mathcal{L}}_{COST}$  | \n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|0.5*ExAUC+0.5*ECL|0.815|0.524|0.026|**0.917**|0.543|0.016|\n|0.5*SqAUC+0.5*ECL|0.795|0.477|0.027|0.876|0.497|0.021|\n|0.8*ExAUC+0.2*ECL|**0.839**|0.519|0.025|0.921|0.465|0.018|\n|0.8*SqAUC+0.2*ECL|0.804|0.487|0.025|0.880|0.503|0.025|\n|0.2*ExAUC+0.8*ECL|0.810|0.506|0.027|0.903|0.509|0.019|\n|0.2*SqAUC+0.8*ECL|0.768|0.518|0.025|0.854|0.503|0.019|\n|WAUC-Gau|0.787|**0.679**|0.024|0.842|**0.745**|0.015|\n|WAUC-Log|0.820|0.653|**0.023**|0.906|0.719|**0.012**|\n\nThe results demonstrate that our algorithm has achieved state-of-the-art performance in terms of cost.\n\n**Q(3) The experiments should be repeated multiple times independently and report the mean&standard deviation values.**\n\n**A(3)** In our experimental setup, we indeed run each method multiple times and take the average. However, due to space limitation in the paper, we omitted the inclusion of standard deviation in the experimental table. We have included standard deviation in Table 3, please click [clickable url](https://anonymous.4open.science/r/WAUC-9B9B/error_bar.png) to open. We plan to include the detailed information in future revisions of the paper. \n\n\n\n**Q(4) It would be better if authors could provide running time report for the proposed method and other baselines**\n\n**A(4)** Thank you for your suggestion! \nFirstly, we analysis the time complexity (one iteration) of our methods and baselines.\n\n - WAUC-Gau (WAUC method): $O(n_\\tau n_+ + n_\\tau n_-)$\n - ExAUC (AUC method): $O(n_+n_-)$\n - ECL (cost-sensitive learning method): $O(n_\\tau n_+ + n_\\tau n_-)$\n\nWe conduct some experiments for time complexity with a fixed epoch with varying $n_+$ and $n_-$. All experiments are conducted on an Ubuntu 16.04.1 server with an Intel(R) Xeon(R) Silver 4110 CPU (to get rid of the affect of parallel computing). For every method, we repeat running 10,000 times and record the average running time. We only record the loss calculation time and use the python package time.time() to calculate the running time. \n\n|method/unit:s|$n_+,n_-=128$|$n_+,n_-=256$|$n_+,n_-=512$|$n_+,n_-=1024$|$n_+,n_-=2048$|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|BCE|1.352|1.856|3.285|6.195|11.957|\n|ExAUC|1.481|1.952|3.951|7.592|12.903|\n|SqAUC|1.380|1.988|4.041|7.813|12.203|\n|NWAUC|1.648|2.268|4.241|8.853|16.947|\n|PAUC-exp|1.380|1.968|3.741|7.748|13.374|\n|PAUC-poly|1.402|2.075|4.013|7.983|14.183|\n|PAUCI|2.085|3.597|6.592|10.967|22.571|\n|CS-hinge|1.880|4.193|7.893|11.213|23.846|\n|AdaCOS|2.197|3.896|6.871|13.414|20.487|\n|ECL|1.974|3.268|5.862|10.831|17.127|\n|WAUC-Gau|1.980|2.975|4.587|8.681|16.976|\n|WAUC-Log|1.897|2.790|4.924|8.487|16.891|\n\n**The results indicate that there is no significant difference in the running time of the WAUC method compared to other binary classification methods.**"}, {"review_id": "Wya86nVXtD", "reviewer": "Reviewer_mY4P", "summary": "In this paper, a bi-level optimization method is proposed for binary classification with unknown cost distributions. The motivation is to propose an adaptive method to deal with different class and cost distributions, getting rid of the assumption of traditional AUC which assumes the uniform cost distribution. The key idea lies in treating the prediction threshold as an learnable parameter, and to utilize bi-level optimization to learn prediction threshold and model parameters jointly. The proposed method is test under several benchmark datasets for verifying its usefulness.", "questions": "In most cost-sensitive binary classification tasks, it is sufficient to assume that the cost for one class of instances remain the same. While in this paper, it seem to assume that the cost can be varied. It would be nice to discuss in what kind of real applications this assumption is necessary.\n\n------\nAcknowledgement:\n\nI would like to thank the authors for the efforts on the responses and the improvements on the paper. Even though my concerns are not fully addressed, in special the real-world applicability, it would be very nice to include the improvements in the future versions of the paper.", "limitations": "I didn't find out potential negative societal impact of the paper. While I encourage to include more discussions on technical limitations of the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is mostly well-written so that it is easy to grasp the key ideas and major results.\n\n- The proposed method is companioned with theoretical guarantees.", "weaknesses": "- Only binary classification is studied. For binary classification, cost-sensitive learning and the AUC metric are both thoroughly studied. Therefore, the contribution is not quite significant.\n\n- Bi-level optimization is a standard technique to optimize hyper-parameters like prediction threshold. Thus the technical contribution is somehow limited. \n\nFurther suggestions:\n\nIn fig. 2 and the experiments, the benchmark datasets such as CIFAR-10/100 are multi-class ones. While the paper makes use of binary class versions of the datasets. It is necessary to describe the classes chosen in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Author response to Reviewer mY4P\n\nThank you for your detailed and constructive feedback on the paper. We value your insights and have taken your suggestions into consideration. Here are our responses to your specific comments.\n\n**Q(1) Only binary classification is studied. For binary classification, cost-sensitive learning and the AUC metric are both thoroughly studied**\n\n**A(1)** While there have been many studies on binary classification problems, both AUC and cost-sensitive learning have their own limitations. The main goal of this paper is to address the shortcomings of existing binary classification methods.\n\n - Cost-sensitive learning: The trained model is not robust to class distribution shift in the test.\n - AUC learning: The trained model is not robust to cost distribution in the test.\n\nWAUC learning: The trained model can be robust to cost distribution and class distribution shift simultaneously in the test. **Furthermore, the problem we investigate, namely cost-sensitive robust learning, is grounded in real-world context. Please refer to Q(4) for more details.**\n\n**Q(2) Bi-level optimization is a standard technique to optimize hyper-parameters like prediction threshold. Thus the technical contribution is somehow limited**\n\n**A(2)** Bilevel optimization is a classical approach to optimization. In our methodology, the utilization of bilevel optimization allows for an elegant solution to optimization problems. Our primary contribution lies in the proposal of the WAUC form, which addresses the drawbacks of existing AUC optimization and cost-sensitive learning techniques, along with providing a well-grounded algorithm for its optimization. Bilevel optimization can be viewed as an essential tool for problem-solving.\n\n**Q(3)  CIFAR-10/100 are multi-class  It is necessary to describe the classes chosen in the experiments**\n\n - Binary CIFAR-10-Long-Tail Dataset. \nThe CIFAR-10 dataset contains 60,000 images, each of 32 * 32 shapes, grouped into 10 classes of 6,000 images. The training and test sets contain 50,000 and 10,000 images, respectively. We construct the binary datasets by selecting one super category as positive class and the other categories as negative class. We generate three binary subsets composed of positive categories, including 1) birds, 2) automobiles, and 3) cats.\n\n - Binary CIFAR-100-Long-Tail Dataset. The original CIFAR-100 dataset has 100 classes, with each containing 600 images. In the CIFAR-100, there are 100 classes divided into 20 superclasses. By selecting a superclass as a positive class example each time, we create CIFAR-100-LT by following the same process as CIFAR-10-LT. The positive superclasses consist of 1) fruits and vegetables, 2) insects, and 3) large omnivores and herbivores, respectively.  \n\n**A(3)** We have provided the methodology for creating the dataset, with specific details outlined in Appendix B.2 Dataset Details.\n\n**Q(4)  It would be nice to discuss in what kind of real applications this assumption is necessary**\n\n**A(4)** In reality, some applications like financial markets prediction ([real-world scenario, click this link to open](https://www.kaggle.com/competitions/jane-street-market-prediction/overview)) involve investment issues. Every investment has a cost involved, including but is not limited to:\n\n - When the company is well-funded and the market conditions are relatively good, missing investment opportunities are more costly (true action: trade, prediction: pass).\n\n - When a company is short of fund and the market performance is poor, blind investment is more costly (true action: pass, prediction: trade).\n\nThe real cost situation must be more complex than the above and is far beyond a simple beta distribution. Developing trading strategies to identify and take advantage of inefficiencies is challenging. Even if a strategy is profitable now, it may not be in the future, and market volatility makes it hard to predict the profitability of any given trade with certainty. Hence, we propose our method to build the WAUC estimator over non-parametric cost distribution for fit this type of application.\n"}, {"review_id": "PfRfSFH8G3", "reviewer": "Reviewer_uAig", "summary": "This paper considers usage of the AUC, in a cost sensitive setting, i.e. where miss classification cost is not uniform. Extensions of the AUC have been considered on parametrised cost distributions, such as the WAUC. In this paper the authors aim to develop a cost sensitive extension to the AUC that does not depend on prior information of the cost distribution. To do this the authors propose a bilevel optimisation problem, where the inner loop estimates the optimal threshold of the scoring function, according to the cost, and then the outer loop estimates the WAUC. The performance of their method is evaluated on several real world datasets, along with sensitivity analysis.", "questions": "There is no proof of Theorem 6.3, are we to assume it follows immediately from [5]? I do not see how the results of [5] lead immediately to Theorem 6.3.", "limitations": "Potential limitations are given good consideration.  The authors use a convex estimator of the none convex cost function when solving for the optimal threshold, the theoretical convergence of said estimator holds only when certain constants, $M, \\kappa, M'$ are sufficiently large, leading to a potential limitation when used in practice. The authors explore this, observing the difference between the optimal threshold, when using the original and convex estimator, of the cost function, for several fixed values of $M, \\kappa, M'$.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors carry out experiments on a wide variety of data sets with a rich set of benchmarks.", "weaknesses": "I do not understand the statement of Proposition 5.1. To me it is not  a proposition but rather the definition of the estimator $\\hat{WAUC}$. The proof of proposition 5.1 is similarly confusing and seems to be a reformulation of the estimator   $\\hat{WAUC}$ which is then used in the proof of Lemma 5.2. Also I assume $\\tau_k$ in eq 22 is a typo and should be $\\tau$? And that Lemma 5.2 holds when $\\tau^*$ is known, as opposed to $\\tau$, another typo. \n\nLemma 5.2 itself is vague, it does not specify how the total number of instances must grow with the negative instances, only that it must be “large enough”. There are no results on the rate of convergence. Also there is no comment on what happens when $\\tau^*$ is not known and we instead use the estimator, as is the case in the bilevel optimisation. \n\nTheorem 5.3 is also vague, the estimator for the convex formulation of the empirical loss is said to have the same minimum when the “parameters satisfy the penalty”. The exact penalty in question is not clear to me. \n\nThere is no proof of Theorem 6.3, are we to assume it follows immediately from [5]? I do not see how the results of [5] lead immediately to Theorem 6.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Author response to Reviewer uAig\n\n**Q(1-1) Proposition 5.1 is not a proposition**\n\n**A(1-1)** We present it as a proposition because by Lemma 5.2, we can derive a convergence result for WAUC, showing that $\\|\\hat{WAUC}-WAUC\\|$ converges at a rate of $O(\\sqrt{\\frac{\\log n_-}{n_- m}})$. Please refer to Q(2-1) for more details.\n\n**Q(1-2)   $\\tau_k$ in eq 22 is a typo and should be $\\tau$?**\n\n**A(1-2)** Thank you very much for your keen observation, this is a typo.\n\n**Q(1-3) Lemma 5.2 holds when $\\tau\\^\\*$ is known, as opposed to $\\tau$, another typo?**\n\n**A(1-3)** Thank you for pointing that out, this is not a typo. The meaning of Lemma 5.2 is that for any given $\\tau$, there is asymptotic convergence, and $\\tau^*$ is not necessary.\n\n**Q(2-1) Lemma 5.2 itself is vague.**\n\n**A(2-1)** \nIn fact, Lemma 5.2 expresses the asymptotic convergence of $\\hat{WAUC}$ and $WAUC$, which can be decomposed into the convergence proof of KDE (please refer to term (c) and term (d) in the appendix). Therefore, the convergence rate of Lemma 5.2 is the same as the convergence rate of KDE. Specifically, the convergence result can be found in Theorem 7 of [1], with a convergence rate of $O(\\sqrt{\\frac{\\log n_-}{n_- m} })$ (where $m$ represents the bandwidth, and in our problem, the KDE dimension $d=1$).\n\n[1] Jiang H. Uniform convergence rates for kernel density estimation[C]//International Conference on Machine Learning. PMLR, 2017: 1694-1703.\n\n**Q(2-2) What happens when $\\tau^\\*$ is not known**\n\n**A(2-2)** When the threshold is not optimal, the outer objective function, $WAUC$, is optimized based on the current threshold $\\tau$. However, since the threshold $\\tau$ is continuously optimized and converges to $\\tau^*$, as the number of iterations increases, the outer $WAUC$ also converges to the optimal value $WAUC\\^\\*$ along with $\\tau$.\n\n**Q(3) Theorem 5.3 is also vague.**\n\n**A(3)** We apologize for our unclear expression. We have provided the penalty function condition in lines 543-544 of the proof of Theorem 5.3: $\\kappa, M, M'\\rightarrow \\infty$ where $\\kappa$ comes from $\\psi(x)=\\frac{\\log(1+\\exp(\\kappa x))}{\\kappa}$.\n\n**Q(4) There is no proof of Theorem 6.3.**\n\n**A(4)** According to [5]'s result (As quoted from [5], page 21) , we have the following convergence rate for bilevel optimization.\n\nIn our paper, we change some symbols, which can be describes by following tables\n\n|  Symbol   |   Symbol in [5]  |   Symbol in ours |\n|:---:|:---:|:---:|\n|   Lipschitz continuity of $f$  |  $\\ell_{f,0}$  | $L_{f,0}$ |\n|   Lipschitz continuity of $\\nabla f$  |  $\\ell_{f,1}$  | $L_{f,1}$ |\n|   Lipschitz continuity of $\\nabla g$  |  $\\ell_{g,1}$  | $L_{g,1}$ |\n|   Strong convexity of $g$ w.r.t. $\\tau$  |  $\\mu_g$  | $\\mu$ |\n\nIn our problem, $\\rho_g:=\\frac{2 \\mu L_{g, 1}}{\\mu+L_{g, 1}}$, then we have:\n\n$$\n\\bar{\\alpha}\\_1=\\frac{1}{2 L\\_F+4 L\\_f L\\_y+2 L\\_f L\\_{y x} \\left(L\\_y \\eta\\right)}, \\quad \\bar{\\alpha}\\_2=\\frac{16 T \\mu L\\_{g, 1}}{\\left(\\mu+L\\_{g, 1}\\right)^2\\left(8 L\\_f L\\_y+2 \\eta L\\_{y x} \\tilde{C}\\_f^2 \\bar{\\alpha}\\_1\\right)}\n$$\n\nwe select the following stepsizes as\n\n$$\n\\alpha\\_k=\\min \\left\\\\{\\bar{\\alpha}\\_1, \\bar{\\alpha}\\_2, \\frac{1}{\\sqrt{K}}\\right\\\\} \\quad \\beta\\_k=\\frac{8 L\\_f L\\_y+2 \\eta L\\_{y x} \\tilde{C}\\_f^2 \\bar{\\alpha}\\_1}{4 T \\mu} \\alpha\\_k\n$$\n\nWith the above choice of stepsizes, (53) in [5] can be simplified as\n\n$$\n\\mathbb{E}\\left[\\mathbb{V}\\^{k+1}\\right]-\\mathbb{E}\\left[\\mathbb{V}\\^k\\right] \\leq-\\frac{\\alpha\\_k}{2} \\mathbb{E}\\left[\\left\\|\\nabla F\\left(\\boldsymbol{w}\\_k\\right)\\right\\|^2\\right]+c\\_1 \\alpha_k^2 \\sigma\\_{g, 1}^2+\\alpha\\_k b\\_k\\^2+c\\_2 \\alpha\\_k\\^2 \\tilde{\\sigma}\\_f\\^2\n$$\n\nwhere the constants $c_1$ and $c_2$ are defined as\n\n$$\n\\begin{aligned}\nc\\_1 & =\\frac{L\\_f}{L\\_y}\\left(1+2 L\\_f L\\_y \\bar{\\alpha}\\_1+\\frac{\\eta L\\_{y x} \\tilde{C}\\_f^2}{4} \\bar{\\alpha}\\_1\\^2\\right)\\left(\\frac{8 L\\_f L\\_y+\\eta L\\_{y x} \\tilde{C}\\_f^2 \\bar{\\alpha}\\_1}{4 \\rho_g}\\right)^2 \\frac{1}{T} \\\\\\\\\nc\\_2 & =\\left(\\frac{L\\_F}{2}+L\\_f L\\_y+\\frac{L\\_{y x} L\\_f}{4 \\eta L\\_y}\\right) .\n\\end{aligned}\n$$\n\nThen telescoping leads to\n\n$$\n\\frac{1}{K} \\sum_{k=0}\\^{K-1} \\mathbb{E}\\left[\\left\\\\|\\nabla F\\left(x\\^k\\right)\\right\\\\|\\^2\\right]  \\leq \\begin{matrix}\n&\\frac{2M_0}{K \\min\\{\\bar{\\alpha}\\_1,\\bar{\\alpha}\\_2\\}}\\quad +&\\frac{2\\mathbb{V}\\_0}{\\alpha \\sqrt{K}}\\quad+&2 b\\_k\\^2\\quad+&\\frac{2c\\_1\\alpha}{\\sqrt{K}}\\sigma\\_{g,1}^2\\quad+&\\frac{2c\\_2\\alpha}{\\sqrt{K}}\\tilde{\\sigma}\\_f\\^2\\\\\\\\\n&\\downarrow &\\downarrow &\\downarrow &\\downarrow &\\downarrow \\\\\\\\\n&O\\left(\\frac{1}{K}\\right) & O\\left(\\frac{1}{\\sqrt{K}}\\right) & O\\left(\\frac{1}{K}\\right) & \\text{term} (1) & O\\left(\\frac{1}{\\sqrt{K}}\\right)\n\\end{matrix}\n$$\n\n$$\n\\begin{aligned}\n    \\text{term} (1) &\\overset{(a)}{=} \\underbrace{2\\alpha \\sigma\\_{g, 1}^2 \\frac{L\\_f}{L\\_y}\\left(1+2 L\\_f L\\_y \\bar{\\alpha}\\_1+\\frac{\\eta L\\_{y x} \\tilde{C}\\_f\\^2}{4} \\bar{\\alpha}\\_1\\^2\\right)\\left(8 L\\_f L\\_y+\\eta L\\_{y x} \\tilde{C}\\_f\\^2 \\bar{\\alpha}\\_1\\right)\\^2 \\left(\\frac{\\mu+L\\_{g,1}}{8\\mu L\\_{g,1}}\\right)\\^2}_{\\gamma } \\frac{1}{T\\sqrt{K}}\\\\\\\\\n    &\\overset{(b)}{=}  \\gamma \\left(\\frac{3 M \\kappa e\\^\\kappa \\/\\left(e\\^\\kappa+1\\right)\\^2+L\\_{g, 1}}{24 M \\kappa e\\^\\kappa \\/\\left(e\\^\\kappa+1\\right)\\^2 L\\_{g, 1}}\\right)\\^2\\frac{1}{T\\sqrt{K}}\n\\end{aligned}\n$$\n\nwhere $(a)$ comes from $\\rho\\_g=\\frac{2 \\mu L\\_{g, 1}}{\\mu+L\\_{g, 1}}$ and $(b)$ comes from $\\mu=\\frac{3M\\kappa e\\^\\kappa}{(e\\^\\kappa+1)\\^2}$. Then we have:\n\nHowever, $O(1/\\sqrt{K})$ is slower than $O(1/K)$, hence, we adopt $O(1/\\sqrt{K})$ as the final result.\n\n$$\n\\frac{1}{K} \\sum\\_{k=0}^{K-1} \\mathbb{E}\\left[\\left\\\\|\\nabla F\\left(\\boldsymbol{w}\\_{k}\\right)\\right\\\\|\\^{2}\\right] \\leq \\gamma\\left(\\frac{3 M \\kappa e\\^{\\kappa} \\/ \\left(e^{\\kappa}+1\\right)\\^{2}+L\\_{g, 1}}{24 M \\kappa e\\^{\\kappa} \\/ \\left(e\\^{\\kappa}+1\\right)\\^{2} L\\_{g, 1}}\\right)\\^{2} \\frac{1}{T \\sqrt{K}}+O\\left(\\frac{1}{\\sqrt{K}}\\right)\n$$\n"}, {"review_id": "i8Of69slf0", "reviewer": "Reviewer_N1ih", "summary": "This paper proposes a weighted AUC (WAUC) loss that is robust to both class distribution shift and cost distribution without class and cost priors. A bilevel optimization paradigm is proposed to bridge WAUC and cost. The authors propose a stochastic optimization algorithm for WAUC, and prove its convergence rate. Extensive experiments are conducted to evaluate the proposed approach.", "questions": "- How are the time and space complexities of Algorithm 1 compared with AUC optimization or other comparable approaches?", "limitations": "NA", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is well organized and easy to read. The main goals and core challenges are listed clearly, and solved one by one.\n- A novel cost-sensitive setting is proposed in this paper, where the cost is obtained by sampling instead of an available prior.\n- The proposed WAUC is robust to both distribution shift and cost distribution, whereas AUC/PAUC and cost learning fail to achieve the two robustness simultaneously.\n- Sound theoretical analysis is presented, including the convergence of the WAUC estimation and the convergence rate of the proposed bilevel optimization. ", "weaknesses": "- Complexity analysis of the proposed optimization approach is lacking. \n- The error bars are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n## Author response to Reviewer N1ih\n\nThank you for your detailed and constructive feedback on the paper. We value your insights and have taken your suggestions into consideration. Here are our responses to your specific comments.\n\n**Q(1) Complexity analysis of the proposed optimization approach is lacking**\n\n**A(1)** Thank you for your suggestion! \nFirstly, we analysis the time complexity (one iteration) of our methods and baselines.\n\n - WAUC-Gau (WAUC method): $O(n_\\tau n_+ + n_\\tau n_-)$\n - ExAUC (AUC method): $O(n_+n_-)$\n - ECL (cost-sensitive learning method): $O(n_\\tau n_+ + n_\\tau n_-)$\n\nWe conduct some experiments for time complexity with a fixed epoch with varying $n_+$ and $n_-$. All experiments are conducted on an Ubuntu 16.04.1 server with an Intel(R) Xeon(R) Silver 4110 CPU (to get rid of the affect of parallel computing). For every method, we repeat running 10,000 times and record the average running time. We only record the loss calculation time and use the python package time.time() to calculate the running time. \n\n|method/unit:s|$n_+,n_-=128$|$n_+,n_-=256$|$n_+,n_-=512$|$n_+,n_-=1024$|$n_+,n_-=2048$|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|BCE|1.352|1.856|3.285|6.195|11.957|\n|ExAUC|1.481|1.952|3.951|7.592|12.903|\n|SqAUC|1.380|1.988|4.041|7.813|12.203|\n|NWAUC|1.648|2.268|4.241|8.853|16.947|\n|PAUC-exp|1.380|1.968|3.741|7.748|13.374|\n|PAUC-poly|1.402|2.075|4.013|7.983|14.183|\n|PAUCI|2.085|3.597|6.592|10.967|22.571|\n|CS-hinge|1.880|4.193|7.893|11.213|23.846|\n|AdaCOS|2.197|3.896|6.871|13.414|20.487|\n|ECL|1.974|3.268|5.862|10.831|17.127|\n|WAUC-Gau|1.980|2.975|4.587|8.681|16.976|\n|WAUC-Log|1.897|2.790|4.924|8.487|16.891|\n\n**The results indicate that there is no significant difference in the running time of the WAUC method compared to other binary classification methods.**\n\n**Q(2) The error bars are not provided**\n\n**A(2)** In our experimental setup, we indeed run each method multiple times and take the average. However, due to space limitation in the paper, we omitted the inclusion of standard deviation in the experimental table. We have included standard deviation in Table 3, please click [clickable url](https://anonymous.4open.science/r/WAUC-9B9B/error_bar.png) to open. We plan to include the detailed information in future revisions of the paper. \n"}, {"review_id": "tIxtRn2TEN", "reviewer": "Reviewer_Trb9", "summary": "This paper proposed more robust learning using combining the WAUC and cost learning. The authors claim that it can be vital to the shifts of cost function and covariate distribution. The algorithm is constructed using bi-level optimization with inner and outer parts. Experiments show high performance, especially in earning money in cost-sensitive situations. \n", "questions": "There are some questions. \n\ni) The definition of $\\mathcal{K} (S_w^{-}, \\tau) $ should be clarified. \n\nii) Is it the right notations $\\nabla \\hat{f}$ and $\\nabla \\hat{g}$ in Alg. 1? \n\niii) The effect of $T$ is strange. Usually, a large $T$ can achieve better performance. Can you explain this phenomenon correlated to $\\alpha_k$ and $\\beta_k$?    \n", "limitations": "No detailed limitations. Maybe, the algorithm can work well in a dynamic situation. However, the static situation is not thoroughly examined.  ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach is somewhat similar to Bayesian, considering the randomness. The motivation is clear, and the algorithm is well-established. Furthermore, this paper is successful in providing the appropriate cases. ", "weaknesses": "The presentation is not good. For example, the notation of $\\hat{L}_{WAUC}$ is isolated in proposition 5.1. Also, there are many typos concerning the indexes in formulation and algorithm. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Author response to Reviewer Trb9\n\nThank you for your detailed and constructive feedback on the paper. We value your insights and have taken your suggestions into consideration. Here are our responses to your specific comments.\n\n**Q(1) What the meaning of $\\mathcal{K}(S_w^-,\\tau)$**\n\n**A(1)** We apologize for our unclear expression. $\\mathcal{K}(S,\\tau)$ is defined in the KDE definition in Appendix.C.1.\n\n$$\n\\mathcal{K}(S, \\tau)=\\frac{1}{|S| m} \\sum_{x_i \\in S} K\\left(\\frac{x_i-\\tau}{m}\\right)\n$$\n\n**Q(2) Is it the right notations $\\nabla \\hat{f}$ and $\\nabla \\hat{g}$ in Alg.1?**\n\n**A(2)** Accoring to the Eq.13, $\\hat{f}$ and $\\hat{g}$ are defined by:\n\n$$\n\\hat{f}\\left(\\boldsymbol{w}, \\boldsymbol{\\tau}\\^*\\right):=\\hat{\\mathcal{L}}\\_{WAUC}\\left(\\boldsymbol{w}, \\hat{\\boldsymbol{\\tau}}^\\* \\right) \\\\\\\\\n\\hat{g}(\\boldsymbol{w}, \\boldsymbol{\\tau}):=\\frac{1}{n\\_\\tau} \\sum\\_{l=1}\\^{n\\_\\tau} \\hat{\\mathcal{L}}\\_{e q}\\left(\\boldsymbol{w}, \\tau\\_l, c\\_l\\right)\n$$\n\nHence, $\\nabla \\hat{f}$ and $\\nabla \\hat{g}$ are gradient of $\\hat{f}$ and $\\hat{g}$.\n\n**Q(3) The effecte of T is strange. Usually, a large T can achieve better performance**\n\n**A(3)** In fact, if the optimization problem is a deterministic one (where all samples are processed in each iteration), then indeed a larger T would be better as it allows the inner loop to directly get the optimal state. \n\n$$\n \\hat{\\boldsymbol{\\tau}}\\^\\*=\\underset{\\boldsymbol{\\tau}, \\boldsymbol{P}\\_a, \\boldsymbol{N}\\_a}{\\arg \\min } \\hat{g}(\\boldsymbol{w}, \\boldsymbol{\\tau}):=\\frac{1}{n\\_\\tau} \\sum\\_{l=1}\\^{n\\_\\tau} \\hat{\\mathcal{L}}\\_{e q}\\left(\\boldsymbol{w}, \\tau\\_l, c\\_l\\right)\n$$\n\nHowever, the optimization problem we are studying is a stochastic one (where only a subset of samples is processed in each iteration), so a larger T is not necessarily better. This is because with a larger T, the inner optimization only selects a subset of samples, resulting in the optimal value of the inner function being only the optimal parameters for the current set of samples, rather than the global optimal parameters. As a result, it is easy to get stuck in a local optimum, leading to poor generalization performance.\n\n$$\n \\hat{\\boldsymbol{\\tau}}\\^\\*(\\mathcal{B})=\\underset{\\boldsymbol{\\tau}, \\boldsymbol{P}\\_a, \\boldsymbol{N}\\_a}{\\arg \\min } \\hat{g}(\\boldsymbol{w}, \\boldsymbol{\\tau};\\mathcal{B}):=\\frac{1}{n\\_\\tau} \\sum_{l=1}\\^{n\\_\\tau} \\widehat{\\mathcal{L}}\\_{e q}\\left(\\boldsymbol{w}, \\tau\\_l, c\\_l;\\mathcal{B}\\right)\n$$\n\n$$\n \\hat{\\boldsymbol{\\tau}}\\^\\*\\neq \\hat{\\boldsymbol{\\tau}}\\^\\*(\\mathcal{B})\n$$\n\nwhere $\\mathcal{B}=\\{\\boldsymbol{x}\\_i,y\\_i\\}\\_{i=1}\\^B$ is a sampled batch of data, $ \\hat{\\boldsymbol{\\tau}}\\^\\*(\\mathcal{B})$ only suits for data distribution of $\\mathcal{B}$.\n\nIt's notable that when batch size $B$ converges to data size $n$, $T$ shoule increase, that means $B\\rightarrow n$, $T\\rightarrow \\infty$.\n\n**Q(4) the notation of $\\hat{L}_{WAUC}$ is isolated in proposition 5.1.**\n\n**A(4)** We use $\\hat{L}\\_{WAUC}$ in Eq.13 $\\hat{f}(\\boldsymbol{w},\\boldsymbol{\\tau}^\\*):=\\hat{L}\\_{WAUC}(\\boldsymbol{w},\\boldsymbol{\\tau}^\\*)$."}]

Author rebuttal:
### Dear the ACs, and the Reviewers, Thank you so much for your valuable comments! They really helped us improve our manuscript!

In order to facilitate reviewers' comprehension of our paper, we want to summarize our contributions again:

 - **We propose a setting that focuses on the robustness of the model to the class distribution and cost distribution simultaneously.** This setting treats cost as data that can be sampled, not as prior information, which is closer to the real-world cost-sensitive scenario.
 - **We present a bilevel paradigm where the inner cost function is an inner constraint of outer WAUC optimization.** For sake of optimization, we reformulate this paradigm into a nonconvex-strongly convex bilevel form. Moreover, we employ a stochastic optimization algorithm for WAUC (SACCL), which can solve this problem efficiently.
 - We conduct extensive experiments on multiple imbalanced cost-sensitive classification tasks. The experimental results speak to the effectiveness of our proposed methods.

Meta-review: The paper is motivated by the observation that models that perform well on standard AUC may not be best-suited for cost-sensitive learning tasks. The authors propose a new objective that intuitively incorporates both robustness towards class prior shifts and arbitrary deployment costs. They optimize this objective using bi-level optimization, and present convergence guarantees and empirical results.

The reviewers seem generally happy with the algorithmic contribution and theoretical guarantees. However, there were concerns about the scope and motivation. On reading the paper, I concur that the paper does have some issues with how it motivates its proposal and is sparse in how it discusses some important connections to prior work. While we are happy to accept this paper, we strongly urge the authors to incorporate the following comments in the camera-ready version.

- **Making statements about AUC more precise:** Some of the statements the paper makes about AUC are not entirely accurate. For example, statements such as "models trained with AUC *cannot* be applied to cost-sensitive decision problems" are strong and imprecise. I think what the authors are trying to say is that AUC is not well-suited for real-world applications as it does not offer the *flexibility* to incorporate a specific cost distribution, and does not consider optimal thresholds. 

- **Robustness to class prior shifts needs to shown formally:** The authors need to be more clear about why their formulation (OP0) is robust to class prior shifts. I understand that standard AUC is invariant to class prior shifts (because it simply measures the fraction of positive-negative pairs that are ranked correctly). It's not obvious that this "invariance" property is retained with the optimal threshold choice used in OP0. The authors need to *show formally why their approach satisfies both attr 1 and attr 2*. If this is only an intuitive observation and cannot be shown formally, the authors need to state this clearly in the paper.

- **Objective optimized same as area under the convex hull?** The formulation in OP0 essentially computes the AUC with the optimal-threshold choice method [Hernández-Orallo et al. 2012, ref. 20], when $\mathcal{D}_c$ is uniform. This is equivalent to computing the area under the **convex hull** of the ROC curve when $\mathcal{D}_c$ is uniform [20-21]. This connection to the prior literature is worth mentioning.

- **Definitions and appropriateness of the eval metrics need to be discussed**: It appears from the reviewer-author discussions that the main aim of the experiments is to showcase robustness towards both class distribution and cost distribution shifts. My understanding is that the authors do this by evaluating two metrics: (i) the WAUC with optimal threshold (the proposed metric) and (ii) the expected cost-sensitive error at optimal threshold $\mathcal{L}_{\rm avg}$ (averaged over costs?). The definition of $\\mathcal{L}\_{\\rm avg}$ (or $\\mathcal{L}\_{\\rm COST}$ in some places) is not explicit, and its unclear if the metric is computed for a fixed cost or averaged over multiple costs. The authors are also encouraged to read the related literature on the connections between "the area under the convex hull of the ROC curve" (equivalent to (i) under uniform costs) and the "area under the optimal cost curve" (equivalent to (ii) under uniform costs) [20-21].
Decision: Accept (poster)