Title: Weighted ROC Curve in Cost Space: Extending AUC to Cost-Sensitive Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: AUC

Cost Learning

Bilevel

machine learning
Abstract: In this paper, we aim to tackle flexible cost requirements for long-tail datasets, where we need to construct a (a) cost-sensitive and (b) class-distribution robust learning framework. The misclassification cost and the area under the ROC curve (AUC) are popular metrics for (a) and (b), respectively. However, limited by their formulations, models trained with AUC cannot be applied to cost-sensitive decision problems, and models trained with fixed costs are sensitive to the class distribution shift. To address this issue, we present a new setting where costs are treated like a dataset to deal with arbitrarily unknown cost distributions. Moreover, we propose a novel weighted version of AUC where the cost distribution can be integrated into its calculation through decision thresholds.  To formulate this setting, we propose a novel bilevel paradigm to bridge weighted AUC (WAUC) and cost. The inner-level problem approximates the optimal threshold from sampling costs, and the outer-level problem minimizes the WAUC loss over the optimal threshold distribution. To optimize this bilevel paradigm, we employ a stochastic optimization algorithm (SACCL) to optimize it. Finally, experiment results show that our algorithm performs better than existing cost-sensitive learning methods and two-stage AUC decisions approach.

Weighted ROC Curve in Cost Space: Extending AUC to Cost-Sensitive Learning

Abstract

In this paper, we aim to tackle flexible cost requirements for long-tail datasets, where we need to construct a (1) cost-sensitive and (2) class-distribution robust learning framework. The misclassification cost and the area under the ROC curve (AUC) are popular metrics for (1) and (2), respectively. However, limited by their formulations, models trained with AUC are not well-suited for cost-sensitive decision problems, and models trained with fixed costs are sensitive to the class distribution shift. To address this issue, we present a new setting where costs are treated like a dataset to deal with arbitrarily unknown cost distributions. Moreover, we propose a novel weighted version of AUC where the cost distribution can be integrated into its calculation through decision thresholds. To formulate this setting, we propose a novel bilevel paradigm to bridge weighted AUC (WAUC) and cost. The inner-level problem approximates the optimal threshold from sampling costs, and the outer-level problem minimizes the WAUC loss over the optimal threshold distribution. To optimize this bilevel paradigm, we employ a stochastic optimization algorithm (SACCL) which enjoys the same convergence rate ( O ( ϵ -4 ) ) with the SGD. Finally, experiment results show that our algorithm performs better than existing cost-sensitive learning methods and two-stage AUC decisions approach.

1 Introduction

Receiver Operating Characteristics (ROC) is a popular tool to describe the trade-off between the True Positive Rate (TPR) and False Positive Rate (FPR) of a scoring function. AUC is defined by the area under the ROC curve [17, 18]. This metric naturally measures the average classification performance under different thresholds and is widely used ( e . g ., disease prediction [19], and anomaly detection [29]). Compared with accuracy, AUC is insensitive to the threshold and cost [7], making it be a popular metric for long-tail learning [32] and achieve remarkable success [24, 44, 26].

Similar to AUC optimization, cost-sensitive learning is a common data mining method [10, 2, 4]. The main goal is to incorporate the misclassification costs in the model, which is more compatible with realistic scenarios ( e . g ., the cost of misdiagnosing a disease as healthy is greater than the counterexample). Over the past two decades, researchers have pointed out that the ROC curve can be transferred to cost space by utilizing a threshold choice method, this is equivalent to computing the area under the convex hull of the ROC curve [21]. In this way, AUC can be seen as the performance of the model with a uniform cost distribution [16]. However, AUC considers all situations, which can not focus more on hard samples, Partial AUC (PAUC) is proposed as an extension of AUC with

truncated uniform cost distribution [31]. Recently, some studies extend PAUC using parameterized cost distributions and propose WAUC to fit real-world applications [16, 30].

Whether we use AUC or cost learning, our main purpose is to train models with these attributes:

Attr.1 : The trained model can be robust to class distribution shift in the test without class prior .

Attr.2 : The trained model can be robust to cost distribution in the test without cost prior .

However, to the best of our knowledge, there are few methods can train a model to have both of these attributes. According to Tab. 1, both AUC-related methods and cost-sensitive learning require a strong prior knowledge of the cost distribution; (1) The cost learning mainly considers a specified cost c and class imbalanced radio π . Models trained under this method are sensitive to class distribution, which does not apply to the scenario where test data distribution with offset. (2) AUC (PAUC) assumes that the cost distribution belongs to (truncated) uniform cost distribution U ( a, b ) . Models trained with them will have poor performance when the true cost distribution is not uniform [16]. (3) WAUC considers optimizing models based on more complex forms of cost distribution, such as beta ( a, b ) . However, we can not obtain the cost prior in real problem scenarios, e . g ., financial market prediction [14]. Considering the weakness comes from the existing settings, we will explore the following question in this paper:

Can we bridge AUC and complicated cost distribution to training robust model on desired cost-sensitive and arbitrary class imbalanced decision scenarios?

To answer this question, we propose a view that, in some real applications [14], the cost, like the instance data, is not available prior but can be obtained by sampling. Therefore, we choose to sample desired cost to approximate the true cost distribution. Different from previous settings, ours is closer to real world, the main process can be divided into three parts:

Step.1 Cost Sampling: Firstly, we sample some desired costs to construct the empirical cost set.

Step.2 Data Sampling: Next, we sample some instance data to construct the empirical dataset.

Step.3 Build Formulation: Finally, we construct the appropriate formulation to maximize the performance in different desired costs and ensure model is robust to distribution shift.

It is natural for us to ask the question: Can we use the existing methods to realize this process? It's clear the answer is no. For AUC-related methods, they can not perform Step.1, and for cost-sensitive learning, they fail to achieve robust distribution shift and multiple costs in Step.3 (as shown in Fig. 1 (orange line). Hence, we propose a novel bilevel formulation combining the advantages of WAUC and cost learning. The inner-level process calculates the optimal threshold from sampling costs, and the outer-level process minimizes the WAUC loss over the optimal threshold distribution. The method can help the model improve robustness to class distribution in cost-sensitive decision problems. The main process is shown in Fig. 1 (green line). We summarize our contributions below:

We propose a setting that focuses on the robustness of the model to the class distribution and cost distribution simultaneously. This setting treats cost as data that can be sampled, not as prior information, which is closer to the real-world cost-sensitive scenario.

(a) Population cost distribution

(b) Discrete cost distribution

(c) Threshold distribution

Sample empirical cost

Solve the optimal threshold

Existing setting

Ours

FPR

FPR weight

Train model

(d) Train metrics

TPR

w

Cost

Train model

(e) Test model

c = 0.4

c = 0.6

c = 0.8

prior cost

(

Nonparametric distribution)

positive

negative

decision

plane

Set prior cost

We present a bilevel paradigm where the inner cost function is an inner constraint of outer WAUC optimization. For sake of optimization, we reformulate this paradigm into a nonconvex-strongly convex bilevel form. Moreover, we employ a stochastic optimization algorithm for WAUC (SACCL), which can solve this problem efficiently.

We conduct extensive experiments on multiple imbalanced cost-sensitive classification tasks. The experimental results speak to the effectiveness of our proposed methods.

2 Observation and Motivation

In Tab. 1, we compare the existing methods with ours from different views. However, the table comparison does not have a very visual presentation. In this section, we will analyze the disadvantages of existing settings and explain our motivation. We train the model with all methods on a Cifar-10Long-Tail training set under the different imbalanced ratios and cost distribution. We visualize the feature representation (the last layer's output of the trained model) in test data by t-SNE. The blue point represents negative samples predicted by the optimal threshold, and the orange point represents positive samples. The smaller the overlap between them, the better the performance. From Fig. 2, we can make the following remarks: (1) According to Fig. 2 (a), AUC is robust to changes in the imbalance ratio but completely not applicable with the cost distribution. (2) According to Fig. 2 (b), cost learning can process different cost distributions, but is sensitive to imbalance ratios.

(a.1) π = 0.5, c ~ U

(a.3) π = 0.5, c ~ N

(a.2) π = 0.2, c ~ U

(a.4) π = 0.2, c ~ N

(b.1) π = 0.5, c ~ U

(b.3) π = 0.5, c ~ N

(b.2) π = 0.2, c ~ U

(b.4) π = 0.2, c ~ N

(c.1) π = 0.5, c ~ U

(c.2) π = 0.2, c ~ U

(c.3) π = 0.5, c ~ N

(c.4) π = 0.2, c ~ N

(a) Tsne result of  repensentation (AUC)

(b) Tsne result of  repensentation (COST)

(c) Tsne result of  repensentation (WAUC)

(Ours)

0

20

L0

10-

01=

=10

10

=20

=10

t9

-30

10

=30

=20

=10

20

29

=10

10

29

-10

99

29

30

29

-10

=10

Hence, our motivation is to propose a new approach to solve the problems in AUC and Costsensitive learning. As shown in Fig. 2 (c), our proposed method can have better learning results in various complex cost distributions and imbalance ratios. That means our method can overcome the shortcomings of traditional AUC and cost-learning, which perfectly fits the proposed setting.

3 Preliminaries

Notations. In this section, we give definitions and preliminaries about AUC . We denote ( x , y ) be an instance, where x is drawn from feature space X ⊆ R d ( d is feature number) and y is drawn from

label space Y = { 0 , 1 } . Let D P ( D N resp .) be positive (negative resp .) instance distribution. Let x + ∼ D P ( x -∼ D N resp .) be positive (negative resp .) instance. We denote S + = { ( x + i , y i ) } n + i =1 ( S -= { ( x -j , y j ) } n -j =1 resp .) as a set of training data drawn from D P ( D N resp .), where n + ( n -resp .) denotes the instance number of S + ( S -resp .). Let I ( · ) be the indicator function, which returns 1 when the condition is true and 0 otherwise. In this paper, we focus on the deep neural network scoring function s ( w , x ) : X ↦→ [0 , 1] , where parameterized by w on an input x .

AUC & WAUC. For specified threshold τ , the TPR of a classifier derived from s ( w , x ) measures the likelihood that it accurately predicts a positive instance when getting a random positive instance from D P . Formally, we have:

In a similar spirit, the classifier's FPR on threshold τ refers to the probability that it predicts positive when it gets a negative instance from D N .

AUCmeasures a scoring function's trade-off between TPR and FPR under uniform thresholds. Denote τ drawn from the distribution D τ , WAUC utilizes the threshold distribution explicitly based on AUC. FPR ′ s denotes the probability density function of s ( w , x -) .

Cost function [2]. In some real application scenarios, we need to consider the misclassification cost. We denote c ( · ) as misclassification cost for class ( · ) , cost c drawn from D c . Since we could not obtain the cost distribution D c , we sample empirical set S c = { c l } n c l =1 , n c denotes the sample number of cost c . Given a scoring function s and parameter w , the cost function L COST is (the empirical version of cost function, ̂ L COST contains the empirical forms of TPR and FPR ):

where π = n + / ( n + + n -) , c = c + / ( c + + c -) and τ ∗ ( c ) is optimal threshold for score function s under specified c [21], the sample number n τ = n c .

4 Problem Formulation

In this section, we introduce how to link the ROC curve to the cost space. First, we reformulate Eq.(3) into expectation:

If threshold τ is drawn from the uniform distribution U , WAUC will degrade to the standard AUC formulation. However, AUC only describes the global mean performance under all possible costs. If we want to extend AUC to cost-sensitive problems, maybe lifting the restriction on the uniform distribution of τ is a good solution. Hence, we release the D τ 's restriction to make it belongs to complicated distribution ( e . g ., normal distribution, exponential distribution). Then we can extend AUC to WAUC . However, using WAUC raises another question: how do we get D τ ? We find that τ ∗ ( c ) is one of parameters of L COST ( w , c, τ ∗ ( c )) . A natural idea is to use L COST to solve for the optimal τ ∗ ( c ) and to combine the τ ∗ ( c ) solved for at different c to obtain D τ .

If we couple Eq.(5b) and Eq.(6) together so that WAUC can enjoy the optimal threshold distribution in L COST , then we can break the barrier between the ROC curve and the cost space. With the help of the threshold as a bridge, we can extend the AUC metric to achieve the WAUC cost-sensitive learning. Then we give the problem formulation (intuitively, from the result of 2 (c), (OP0) satisfies both Attr.1 and Attr.2 simultaneously):

Nevertheless, there are still three main challenges in WAUC cost-sensitive learning:

Given the scoring function s and negative dataset S -, how to estimate FPR ′ s ( τ ) in WAUC ?

The inner problem is nonconvex, which is hard to give a theoretical convergence guarantee.

How to design a formulation that can bridge WAUC and L COST so that WAUC can be optimized over the cost distribution of the desired problem scenario?

We will address the challenge (1) in Sec. 5.1, challenge (2) in Sec. 5.2 and challenge (3) in Sec. 5.3.

5 Methodology

5.1 The Estimation of False Positive Rate

For challenge (1) , we choose the kernel density estimation (KDE) to estimate FPR ′ s ( τ ) and denote it as K ( x ) (please see definition in Sec. C.1). Then we can address the density estimation problem. However, Eq.(5b) still exists non-differentiable and non-smooth term I ( · ) , which is hard to optimize. Hence, we propose the following smooth and differentiable WAUC estimator to approximate Eq.(5b).

Definition 5.1. Denote K ( x ) be statistics kernel with bandwidth m and S w -= { s ( w , x -j ) } n -j =1 . With Lemma 5.2, we have the approximate estimator and loss function for WAUC:

where τ = { τ l } n τ l =1 and the point loss ˆ h is defined by

σ ( x ) = 1 / (1 + exp( -βx )) , β is smooth parameter and we have σ ( x ) β →∞ -→ I x .

Lemma 5.2. Given a scoring function s , if τ is known, when the number of instances is large enough, ̂ WAUC almost surely converges to WAUC .

With KDE consistency [41], when the negative sample size is large enough, Lemma 5.2 provides theoretical approximation guarantees for our proposed WAUC estimator in Prop. 5.1.

5.2 The Estimation of Threshold Weighting

For challenge (2) , a natural idea is to use ̂ L COST to solve for the optimal threshold set ˆ τ ∗ when given the cost set S c and the scoring function s . Then we can use the optimal threshold set ˆ τ ∗ to calculate ̂ WAUC . Firstly, we define the solution for ˆ τ ∗ be

However, it's noticed that the arg min τ ̂ L COST in Eq.(11) is non-convex. As we analyzed before, τ ∗ is the inner constraint of ̂ WAUC . To the best of our knowledge, there are few studies on optimizing two coupled non-convex problems simultaneously with theoretical convergence guarantees. Most studies on coupled optimization assume that the inner problems have good properties, such as strong convexity. Hence, we propose the approximated convex formulation of the inner problem for ˆ τ ∗ .

Theorem 5.3. When we set κ, M are large positive numbers and M ′ 2 < M 2 6 κ 2 e 3 κ ( e κ +1) 6 , then we have the approximated convex formulation for ˆ L COST

where ψ ( x ) = log(1 + exp( κx )) /κ . ̂ L eq in Eq.(12) is µ g -strongly convex w.r.t. τ . Eq.(12) has same solution as min τ ̂ L COST when the parameters satisfy the conditions of the penalty.

Thm. 5.3 provides an optimization method with good properties. Eq. (12) adopts the penalty function to convert inequality constraints into a part of the objective function. When these inequality constraints are not satisfied, the objective function will increase to infinity. Otherwise, we will get P i = I [ s ( w , x + i ) > τ ] and N j = I [ s ( w , x -j ) > τ ] , then we will get the same formulation as ̂ L COST . When the parameters meet the requirements, Eq.(12) has the same solution as ̂ L COST . We give the proof of Thm. 5.3 and the definition µ in Sec. C.4. Moreover, we give the analysis of approximation error between ̂ L COST and Thm. 5.3 in Sec. B.7.

5.3 Bilevel Optimization for WAUC learning

After answering questions in challenge (1) and (2) , we have solved most of the problems in WAUC cost-sensitive learning. However, there remains a challenge (3) in optimization: How do we design learning paradigms to solve the coupled optimization problem of WAUC and L COST ? In recent years, bilevel optimization has achieved remarkable success. This approach can combine two related optimization problems to form a coupled optimization formulation. Hence, with Prop. 5.1 and Thm.5.3, we propose a bilevel paradigm to formulate this coupled optimization problem.

where P a ∈ R n τ × n + and N a ∈ R n τ × n -. ( OP 1) describes a bilevel optimization formulation for WAUC cost-sensitive learning, where the inner-level provides a threshold optimization process, and the outer-level minimizes the WAUC loss over the optimal threshold distribution. Moreover, this formulation is consistent with the mainstream bilevel optimization problem (outer-level is smooth and non-convex, inner-level is convex and smooth), which enjoys a faster convergence rate.

6 Optimization Algorithm

In this section, we focus on optimizing ( OP 1) in an end-to-end manner. Hence, we propose a stochastic algorithm for WAUC cost-sensitive learning shown in Alg. 1, which is referred to SAACL.

6.1 Main Idea of SAACL

We provide some intuitive explanations of our algorithm. At each iteration k , SACCL alternates between the inner-level gradient update on τ and the outer-level gradient update on w . During

Algorithm 1

iteration k , we update τ k,t with standard SGD T steps to ensure that τ k +1 is as optimal as possible. After updating inner-level variables, we perform outer-level optimization with τ k +1 as the parameter to update w k . Notice that T will not take a large value to ensure the validity of the coupling update of τ and w . Let α k and β k be stepsizes of w and τ that have the same decrease rate as SGD. We denote n be the number of elements in deep neural network parameters w .

6.2 Convergence Analysis of SAACL

In this subsection, we present the convergence analysis for SAACL. We give some Lipschitz continuity assumptions that are common in bilevel optimization problems [11, 28].

Assumption 6.1. (Lipschitz continuity) Assume that f , ∇ f , and ∇ g are respectively L f, 0 , L f, 1 , L g, 1 -Lipschitz continuous.

Assumption 6.2. (Bounded stochastic derivatives) The variance of stochastic derivatives ∇ f ( w , τ ; B ) and ∇ g ( w , τ ; B ) are bounded by σ 2 f, 1 , σ 2 g, 1 , respectively.

Based on Assumption 6.1 and Assumption 6.2, following [5], Thm. 6.3 indicates that we can optimize ( OP 1) with the same convergence rate as the traditional SGD algorithm.

Theorem 6.3. Suppose Assumption 6.1 and 6.2 hold. We define

where η = L F /L y , L F , L f , L y and ˜ C 2 f come from Lem. 2 and Lem. 4 in [5]. We select the following stepsize as

For any T ≥ 1 , the iteration sequence { w k } K k =1 and { τ k } K k =1 generated by Algorithm 1 satisfy

Remark 6.4 . When κ and M are large enough positive integers, according to Eq. (16), Alg. 1 is still guaranteed to find a ϵ -stationary point within O ( ϵ -4 ) iterations ( ϵ is error tolerance).

7 Experiments

In this section, we conduct a series of experiments for WAUC cost-sensitive learning on common long-tail benchmark datasets. Due to space limitations, please refer to Sec. B for the details of our experiments. The source code is available in supplemental materials.

7.1 Dataset Details

We use three datasets: Binary CIFAR-10-Long-Tail Dataset [23], Binary CIFAR-100-Long-Tail Dataset [23], and Jane Street Market Prediction [14]. Binary CIFAR-10-Long-Tail Dataset and Binary CIFAR-100-Long-Tail Dataset are common datasets in long-tail learning, and we construct their cost distributions. Jane Street Market Prediction is data from real cost-sensitive learning application scenarios. For all datasets, we divide them into the training set, validation set, and test set with a proportion 0.7:0.15:0.15. All image data is normalized to ensure a more stable training process.

dataset

type

methods

Subset1

Subset2

Subset3

̂

AUC

↑

̂

WAUC

↑

̂

L

COST

↓

̂

WAUC

↑

̂

L

COST

↓

̂

WAUC

↑

̂

L

COST

↓

Subset1

Subset2

Subset3

CIFAR-10-LT

Competitors

BCE

0.525

0.027

0.533

0.015

0.318

0.029

0.822

0.960

0.870

ExAUC

0.516

0.029

0.518

0.013

0.366

0.028

0.845

0.963

0.858

SqAUC

0.407

0.028

0.548

0.012

0.327

0.031

0.811

0.933

0.867

NWAUC

0.565

0.030

0.574

0.017

0.396

0.027

0.786

0.885

0.827

PAUC-exp

0.549

0.029

0.508

0.015

0.354

0.028

0.650

0.801

0.736

PAUC-poly

0.526

0.028

0.470

0.015

0.354

0.029

0.661

0.812

0.742

PAUCI

0.516

0.027

0.520

0.015

0.382

0.028

0.704

0.847

0.734

CS-hinge

0.566

0.026

0.633

0.010

0.377

0.022

0.675

0.782

0.762

AdaCOS

0.576

0.025

0.559

0.014

0.391

0.023

0.758

0.873

0.742

ECL

0.589

0.026

0.561

0.014

0.388

0.020

0.694

0.918

0.762

Our method

WAUC-Gau

0.679

0.024

0.660

0.012

0.467

0.015

0.787

0.934

0.843

WAUC-Log

0.653

0.023

0.674

0.011

0.468

0.014

0.820

0.958

0.869

CIFAR-100-LT

Competitors

BCE

0.556

0.022

0.463

0.012

0.512

0.019

0.912

0.957

0.806

ExAUC

0.522

0.019

0.502

0.011

0.506

0.017

0.933

0.967

0.833

SqAUC

0.483

0.024

0.367

0.015

0.474

0.018

0.889

0.955

0.855

NWAUC

0.654

0.025

0.511

0.016

0.631

0.019

0.867

0.925

0.807

PAUC-exp

0.464

0.020

0.282

0.014

0.469

0.016

0.826

0.811

0.787

PAUC-poly

0.461

0.022

0.262

0.017

0.473

0.017

0.828

0.887

0.791

PAUCI

0.549

0.018

0.439

0.016

0.514

0.018

0.812

0.843

0.822

CS-hinge

0.523

0.017

0.457

0.010

0.515

0.014

0.734

0.910

0.716

AdaCOS

0.590

0.018

0.474

0.011

0.587

0.016

0.769

0.919

0.727

ECL

0.583

0.017

0.497

0.009

0.595

0.015

0.863

0.939

0.794

Our method

WAUC-Gau

0.745

0.015

0.589

0.005

0.728

0.013

0.842

0.928

0.745

WAUC-Log

0.719

0.012

0.560

0.003

0.745

0.010

0.906

0.960

0.875

7.2 Overall Performance

In Tab. 2 and Tab. 3, we collect all the methods' performance on test sets of three types of datasets. For cost distribution of c , we sample some data from a normal distribution N (0 . 5 , 1) to construct a dataset S c (we clip all data to [0 , 1] ). We also conduct numerous experiments for other types of distribution of c , and please see Appendix B for the details. From the results, we make the following observations:

(1) For ̂ WAUC and ̂ L COST metric, Our proposed algorithm achieves superior performance in most benchmark datasets compared to other methods. This demonstrates that our proposed WAUC costsensitive learning can extend the ROC curve into the cost space. Models trained with our proposed bilevel optimization formulation can enjoy high WAUC and cost-related metrics.

(2) AUC and cost-related metrics are inconsistent. From the high-performing heatmap of Tab. 2, it can be noticed that ̂ L COST and ̂ AUC have two completely different highlight regions. This indicates that the assumption of uniform distribution of AUC does not match the realistic scenario.

(3) We also find that AUC-related and traditional classification algorithms do not perform well in cost-sensitive problems. This means that if we first train the model with the classification algorithm, subsequently using the cost function to solve for the optimal threshold for decision does not work well. Meanwhile, the algorithm that can learn from scratch has better scalability. Two-stage decision method Therefore, designing one-stage algorithms for WAUC cost-sensitive learning is necessary.

7.3 Sensitivity Analysis

In this subsection, we show the sensitivity of β , T , and bandwidth on test data.

Effect of β . In Fig. 3 (a) and (d), we observe that for both WAUC and cost metrics, when β closes to 7, the model will have the largest performance improvement and the lowest variance. This can be explained in two ways: (1) When the β is too small, the error between the σ ( x ) and the 0-1 loss function is large, resulting in a large approximation error between the ̂ WAUC and the WAUC . (2) When the β is too large, the gradient also tends to be 0. Therefore, choosing a beta value that trades off the approximation error and the gradient is essential.

0.25

0.30

0.35

0.40

0.45

Performance (WAUC)

3

5

7

9

11

beta

Mean

Median

(a) The effect of

β

on WAUC

0.20

0.25

0.30

0.35

0.40

0.45

Performance (WAUC)

2

4

6

8

10

T

Mean

Median

(b) The effect of

T

on WAUC

0.3

0.4

0.5

0.6

Performance (WAUC)

0.1

0.2

0.3

0.4

0.5

bandwidth

Mean

Median

(c) The effect of

m

on WAUC

0.050

0.075

0.100

0.125

Performance (COST)

3

5

7

9

11

beta

Mean

Median

(d) The effect of

β

on cost

0.025

0.050

0.075

0.100

0.125

Performance (COST)

2

4

6

8

10

T

Mean

Median

(e) The effect of

T

on cost

0.025

0.050

0.075

0.100

0.125

Performance (COST)

0.1

0.2

0.3

0.4

0.5

bandwidth

Mean

Median

(f) The effect of

m

cost

Effect of T . As we mentioned in the Section 6.1, choosing a smaller T can effectively improve the performance of the model. However, as shown in Fig. 3(e), a larger T value can reduce the variance. Hence, as set in our experiments, T = 3 is a good choice to ensure the average performance and variance of the model.

Effect of m . From Fig. 3(c), we find that the kernel's bandwidth strongly influences the model's performance. The model's bandwidth and performance are almost proportional; the closer the bandwidth is to [0.4, 0.5], the better the effect; otherwise, the effect is worse. This indicates that our proposed method is sensitive to the bandwidth parameter, which also compounds the bandwidth characteristics in the KDE method.

8 Conclusion

This paper focuses on extending the traditional AUC metric to associate with misclassification costs. Restricted by the assumption of cost distribution, existing settings could not describe the model's performance in the complicated cost-sensitive scenario. To address this problem, we propose a novel setting that treats the cost as sampled data. We employ the WAUC metric and propose a novel estimator to approximate it. With the help of threshold weighting, we establish the correspondence between WAUC and the cost function. To describe this connection, we present a bilevel optimization formulation to couple them, where the inner-level problem provides a threshold optimization process, and the outer-level minimizes the WAUC loss based on the inner thresholds. This paradigm ensures that the WAUC can always be optimized at the optimal threshold value based on the complicated cost distribution in reality. Moreover, we propose a stochastic algorithm to optimize this formulation. We prove that our algorithm enjoys the same convergence rate as standard SGD. Finally, numerous

experiments have shown that our method can extend AUC to cost-sensitive scenarios with significant performance.

References

J A, Hanley, B J, and McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology , 1982.

Daniel Andrade and Yuzuru Okajima. Efficient bayes risk estimation for cost-sensitive classification. In The 22nd international conference on artificial intelligence and statistics , pages 3372-3381. PMLR, 2019.

Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations Research , 21(1):37-44, 1973.

Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning , pages 1507-1517. PMLR, 2021.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Tighter analysis of alternating stochastic gradient method for stochastic nested problems. arXiv preprint arXiv:2106.13781 , 2021.

Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research , 153(1):235-256, 2007.

Corinna Cortes and Mehryar Mohri. AUC optimization vs. error rate minimization. In Advances in Neural Information Processing Systems , pages 313-320, 2003.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

Tom Fawcett. An introduction to roc analysis. Pattern recognition letters , 27(8):861-874, 2006.

Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, Bartosz Krawczyk, Francisco Herrera, Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, et al. Costsensitive learning. Learning from Imbalanced Data Sets , pages 63-78, 2018.

Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning , pages 1568-1577. PMLR, 2018.

Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research , 4:933-969, 2003.

Thore Graepel, Klaus Obermayer, et al. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers , pages 115-132. 2000.

Jane Street Group. Jane street market prediction. https://www.kaggle.com/ competitions/jane-street-market-prediction/overview , 2021.

Zhishuai Guo, Mingrui Liu, Zhuoning Yuan, Li Shen, Wei Liu, and Tianbao Yang. Communication-efficient distributed stochastic auc maximization with deep neural networks. In International Conference on Machine Learning , pages 3864-3874, 2020.

David J Hand. Measuring classifier performance: a coherent alternative to the area under the roc curve. Machine learning , 77(1):103-123, 2009.

James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology , 143(1):29-36, 1982.

James A Hanley and Barbara J McNeil. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology , 148(3):839-843, 1983.

Huaying Hao, Huazhu Fu, Yanwu Xu, Jianlong Yang, Fei Li, Xiulan Zhang, Jiang Liu, and Yitian Zhao. Open-narrow-synechiae anterior chamber angle classification in as-oct sequences. arXiv preprint arXiv:2006.05367 , 2020.

José Hernández-Orallo, Peter Flach, and César Ferri Ramírez. A unified view of performance metrics: Translating threshold choice into expected classification loss. Journal of Machine Learning Research , 13:2813-2869, 2012.

José Hernández-Orallo, Peter Flach, and César Ferri. Roc curves in cost space. Machine learning , 93(1):71-91, 2013.

Thorsten Joachims. A support vector method for multivariate performance measures. In International Conference on Machine Learning , pages 377-384, 2005.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yunwen Lei and Yiming Ying. Stochastic proximal auc maximization. JMLP , 22(61):1-45, 2021.

Jialiang Li and Jason P Fine. Weighted area under the receiver operating characteristic curve and its application to gene selection. Journal of the Royal Statistical Society: Series C (Applied Statistics) , 59(4):673-692, 2010.

Mingrui Liu, Xiaoxuan Zhang, Zaiyi Chen, Xiaoyu Wang, and Tianbao Yang. Fast stochastic auc maximization with o (1 /n ) -convergence rate. In ICML , pages 3189-3197. PMLR, 2018.

Mingrui Liu, Zhuoning Yuan, Yiming Ying, and Tianbao Yang. Stochastic AUC maximization with deep neural networks. In International Conference on Learning Representations , 2020.

Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In International Conference on Machine Learning , pages 6305-6315. PMLR, 2020.

Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a new baseline. In CVPR , pages 6536-6545, 2018.

Andreas Maurer and Massimiliano Pontil. Estimating weighted areas under the roc curve. Advances in Neural Information Processing Systems , 33:7733-7742, 2020.

Donna Katzman McClish. Analyzing a portion of the roc curve. Medical decision making , 9(3): 190-195, 1989.

Charles E Metz. Basic principles of roc analysis. In Seminars in nuclear medicine , volume 8, pages 283-298. Elsevier, 1978.

Harikrishna Narasimhan and Shivani Agarwal. Support vector algorithms for optimizing the partial area under the roc curve. Neural Computation , 29(7):1919-1963, 2017.

Sakrapee Paisitkriangkrai, Chunhua Shen, and Anton Van Den Hengel. Efficient pedestrian detection by directly optimizing the partial area under the roc curve. In Proceedings of the IEEE international conference on computer vision , pages 1057-1064, 2013.

Margaret Sullivan Pepe and Mary Lou Thompson. Combining diagnostic test results to increase accuracy. Biostatistics , 1(2):123-140, 2000.

Alain Rakotomamonjy. Support vector machines and area under roc curve. PSI-INSA de Rouen: Technical Report , 2004.

Shoham Sabach and Shimrit Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

HuiYang Shao, Qianqian Xu, Zhiyong Yang, Shilong Bao, and Qingming Huang. Asymptotically unbiased instance-wise regularized partial auc optimization: Theory and algorithm. In Advances in Neural Information Processing Systems , 2022.

Luis N Vicente and Paul H Calamai. Bilevel and multilevel programming: A bibliography review. Journal of Global optimization , 5(3):291-306, 1994.

Sam Wieand, Mitchell H Gail, Barry R James, and Kang L James. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika , 76(3): 585-592, 1989.

Dominik Wied and Rafael Weißbach. Consistency of the kernel density estimator: a survey. Statistical Papers , 53(1):1-21, 2012.

Lian Yan, Robert H Dodier, Michael Mozer, and Richard H Wolniewicz. Optimizing classifier performance via an approximation to the wilcoxon-mann-whitney statistic. In International Conference on Machine Learning , pages 848-855, 2003.

Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey. arXiv preprint arXiv:2203.15046 , 2022.

Zhenhuan Yang, Wei Shen, Yiming Ying, and Xiaoming Yuan. Stochastic auc optimization with general loss. CPAA , 19(8), 2020.

Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. When all we need is a piece of the pie: A generic framework for optimizing two-way partial auc. In International Conference on Machine Learning , pages 11820-11829, 2021.

Yao Yao, Qihang Lin, and Tianbao Yang. Large-scale optimization of partial auc in a range of false positive rates. arXiv preprint arXiv:2203.01505 , 2022.

Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online auc maximization. Advances in Neural Information Processing Systems , 29:451-459, 2016.

Zhuoning Yuan, Zhishuai Guo, Nitesh Chawla, and Tianbao Yang. Compositional training for end-to-end deep auc maximization. In International Conference on Learning Representations , 2021.

Adriano Z Zambom and Dias Ronaldo. A review of kernel density estimation with applications to econometrics. International Econometric Review , 5(1):20-42, 2013.

Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, and Tianbao Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. arXiv preprint arXiv:2203.00176 , 2022.

Peer reviews:
comments: SUMMARY:
This paper proposed more robust learning using combining the WAUC and cost learning. The authors claim that it can be vital to the shifts of cost function and covariate distribution. The algorithm is constructed using bi-level optimization with inner and outer parts. Experiments show high performance, especially in earning money in cost-sensitive situations.

STRENGTHS:
The approach is somewhat similar to Bayesian, considering the randomness. The motivation is clear, and the algorithm is well-established. Furthermore, this paper is successful in providing the appropriate cases.

WEAKNESSES:
The presentation is not good. For example, the notation of $\hat{L}_{WAUC}$ is isolated in proposition 5.1. Also, there are many typos concerning the indexes in formulation and algorithm.

QUESTIONS:
There are some questions. 

i) The definition of $\mathcal{K} (S_w^{-}, \tau) $ should be clarified. 

ii) Is it the right notations $\nabla \hat{f}$ and $\nabla \hat{g}$ in Alg. 1? 

iii) The effect of $T$ is strange. Usually, a large $T$ can achieve better performance. Can you explain this phenomenon correlated to $\alpha_k$ and $\beta_k$?

LIMITATIONS:
No detailed limitations. Maybe, the algorithm can work well in a dynamic situation. However, the static situation is not thoroughly examined.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes a weighted AUC (WAUC) loss that is robust to both class distribution shift and cost distribution without class and cost priors. A bilevel optimization paradigm is proposed to bridge WAUC and cost. The authors propose a stochastic optimization algorithm for WAUC, and prove its convergence rate. Extensive experiments are conducted to evaluate the proposed approach.

STRENGTHS:
- This paper is well organized and easy to read. The main goals and core challenges are listed clearly, and solved one by one.
- A novel cost-sensitive setting is proposed in this paper, where the cost is obtained by sampling instead of an available prior.
- The proposed WAUC is robust to both distribution shift and cost distribution, whereas AUC/PAUC and cost learning fail to achieve the two robustness simultaneously.
- Sound theoretical analysis is presented, including the convergence of the WAUC estimation and the convergence rate of the proposed bilevel optimization.

WEAKNESSES:
- Complexity analysis of the proposed optimization approach is lacking. 
- The error bars are not provided.

QUESTIONS:
- How are the time and space complexities of Algorithm 1 compared with AUC optimization or other comparable approaches?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers usage of the AUC, in a cost sensitive setting, i.e. where miss classification cost is not uniform. Extensions of the AUC have been considered on parametrised cost distributions, such as the WAUC. In this paper the authors aim to develop a cost sensitive extension to the AUC that does not depend on prior information of the cost distribution. To do this the authors propose a bilevel optimisation problem, where the inner loop estimates the optimal threshold of the scoring function, according to the cost, and then the outer loop estimates the WAUC. The performance of their method is evaluated on several real world datasets, along with sensitivity analysis.\n\nSTRENGTHS:\nThe authors carry out experiments on a wide variety of data sets with a rich set of benchmarks.\n\nWEAKNESSES:\nI do not understand the statement of Proposition 5.1. To me it is not  a proposition but rather the definition of the estimator $\\hat{WAUC}$. The proof of proposition 5.1 is similarly confusing and seems to be a reformulation of the estimator   $\\hat{WAUC}$ which is then used in the proof of Lemma 5.2. Also I assume $\\tau_k$ in eq 22 is a typo and should be $\\tau$? And that Lemma 5.2 holds when $\\tau^*$ is known, as opposed to $\\tau$, another typo. \n\nLemma 5.2 itself is vague, it does not specify how the total number of instances must grow with the negative instances, only that it must be “large enough”. There are no results on the rate of convergence. Also there is no comment on what happens when $\\tau^*$ is not known and we instead use the estimator, as is the case in the bilevel optimisation. \n\nTheorem 5.3 is also vague, the estimator for the convex formulation of the empirical loss is said to have the same minimum when the “parameters satisfy the penalty”. The exact penalty in question is not clear to me. \n\nThere is no proof of Theorem 6.3, are we to assume it follows immediately from [5]? I do not see how the results of [5] lead immediately to Theorem 6.3.\n\nQUESTIONS:\nThere is no proof of Theorem 6.3, are we to assume it follows immediately from [5]? I do not see how the results of [5] lead immediately to Theorem 6.3.\n\nLIMITATIONS:\nPotential limitations are given good consideration.  The authors use a convex estimator of the none convex cost function when solving for the optimal threshold, the theoretical convergence of said estimator holds only when certain constants, $M, \\kappa, M'$ are sufficiently large, leading to a potential limitation when used in practice. The authors explore this, observing the difference between the optimal threshold, when using the original and convex estimator, of the cost function, for several fixed values of $M, \\kappa, M'$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a bi-level optimization method is proposed for binary classification with unknown cost distributions. The motivation is to propose an adaptive method to deal with different class and cost distributions, getting rid of the assumption of traditional AUC which assumes the uniform cost distribution. The key idea lies in treating the prediction threshold as an learnable parameter, and to utilize bi-level optimization to learn prediction threshold and model parameters jointly. The proposed method is test under several benchmark datasets for verifying its usefulness.\n\nSTRENGTHS:\n- The paper is mostly well-written so that it is easy to grasp the key ideas and major results.\n\n- The proposed method is companioned with theoretical guarantees.\n\nWEAKNESSES:\n- Only binary classification is studied. For binary classification, cost-sensitive learning and the AUC metric are both thoroughly studied. Therefore, the contribution is not quite significant.\n\n- Bi-level optimization is a standard technique to optimize hyper-parameters like prediction threshold. Thus the technical contribution is somehow limited. \n\nFurther suggestions:\n\nIn fig. 2 and the experiments, the benchmark datasets such as CIFAR-10/100 are multi-class ones. While the paper makes use of binary class versions of the datasets. It is necessary to describe the classes chosen in the experiments.\n\nQUESTIONS:\nIn most cost-sensitive binary classification tasks, it is sufficient to assume that the cost for one class of instances remain the same. While in this paper, it seem to assume that the cost can be varied. It would be nice to discuss in what kind of real applications this assumption is necessary.\n\n------\nAcknowledgement:\n\nI would like to thank the authors for the efforts on the responses and the improvements on the paper. Even though my concerns are not fully addressed, in special the real-world applicability, it would be very nice to include the improvements in the future versions of the paper.\n\nLIMITATIONS:\nI didn't find out potential negative societal impact of the paper. While I encourage to include more discussions on technical limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

comments: SUMMARY:
The authors propose a method that combines WAUC (weighted Area under ROC curve) learning with cost-sensistive learning. They propose a bilevel optimization algorithm to solve the formulated problem and provide theoretical analysis for convergence. According to their experiments on three datasets, the practical performance is good. The idea is interesting and the method should be novel.

STRENGTHS:
1) The presentation is good. The authors raise their motivation (combines AUC with cost learning) at the beginning. And the writing flow is clear for the paper. They have several good figures to illustrate the motivations.
2) Overall, the soundness is good. Apart from basic writings, authors also provide both practical comparison with some baselines and theoretical analysis for their algorithm convergence.

WEAKNESSES:
1) The method is designed to be both class distribution robust and cost distribution robust, but doesn't demonstrate better AUC performance in Table 2. Comparing with $\widehat{WAUC}$, the $\widehat{AUC}$ is the more suitable metric for class distribution robustness (besides, the $\widehat{WAUC}$ is defined and only optimized by the proposed method).
2) Miss an ad-hoc baseline that optimizes both AUC and cost sensitive learning objectives simultaneously by simply assigning different weights. 
3) The experiments should be repeated multiple times independently and report the mean&standard deviation values.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Besides of the weaknesses, it would be better if authors could provide running time report for the proposed method and other baselines.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-22