SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huiyang Shao $^{1,2}$  Qianqian Xu $^{1*}$  Zhiyong Yang $^{2}$  Peisong Wen $^{1,2}$  Peifeng Gao $^{2}$  Qingming Huang $^{2,1,3*}$ $^{1}$  Key Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS  
 $^{2}$  School of Computer Science and Tech., University of Chinese Academy of Sciences  
 $^{3}$  BDKM, University of Chinese Academy of Sciences  
shaohuiyang21@mails.ucas.ac.cn xuqianqian@ict.ac.cn  
wenpeisong20z@ict.ac.cn gaopeifeng21@mails.ucas.ac.cn  
yangzhiyong21@ucas.ac.cn qmhuang@ucas.ac.cn
In this paper, we aim to tackle flexible cost requirements for long-tail datasets, where we need to construct a (1) cost-sensitive and (2) class-distribution robust learning framework. The misclassification cost and the area under the ROC curve (AUC) are popular metrics for (1) and (2), respectively. However, limited by their formulations, models trained with AUC are not well-suited for cost-sensitive decision problems, and models trained with fixed costs are sensitive to the class distribution shift. To address this issue, we present a new setting where costs are treated like a dataset to deal with arbitrarily unknown cost distributions. Moreover, we propose a novel weighted version of AUC where the cost distribution can be integrated into its calculation through decision thresholds. To formulate this setting, we propose a novel bilevel paradigm to bridge weighted AUC (WAUC) and cost. The inner-level problem approximates the optimal threshold from sampling costs, and the outer-level problem minimizes the WAUC loss over the optimal threshold distribution. To optimize this bilevel paradigm, we employ a stochastic optimization algorithm (SACCL) which enjoys the same convergence rate  $(O(\epsilon^{-4}))$  with the SGD. Finally, experiment results show that our algorithm performs better than existing cost-sensitive learning methods and two-stage AUC decisions approach.
Receiver Operating Characteristics (ROC) is a popular tool to describe the trade-off between the True Positive Rate (TPR) and False Positive Rate (FPR) of a scoring function. AUC is defined by the area under the ROC curve [17, 18]. This metric naturally measures the average classification performance under different thresholds and is widely used (e.g., disease prediction [19], and anomaly detection [29]). Compared with accuracy, AUC is insensitive to the threshold and cost [7], making it be a popular metric for long-tail learning [32] and achieve remarkable success [24, 44, 26].
Similar to AUC optimization, cost-sensitive learning is a common data mining method [10, 2, 4]. The main goal is to incorporate the misclassification costs in the model, which is more compatible with realistic scenarios (e.g., the cost of misdiagnosing a disease as healthy is greater than the counterexample). Over the past two decades, researchers have pointed out that the ROC curve can be transferred to cost space by utilizing a threshold choice method, this is equivalent to computing the area under the convex hull of the ROC curve [21]. In this way, AUC can be seen as the performance of the model with a uniform cost distribution [16]. However, AUC considers all situations, which can not focus more on hard samples, Partial AUC (PAUC) is proposed as an extension of AUC with
Table 1: Comparison with existing classification settings. Cost distribution represents the cost condition of each setting.
Different setting	Formulation	Attr.1	Attr.2	Cost distribution
Cost learning	Ex[c·π·p(1|x) + (1-c)·(1-π)·p(0|x)]	×	×	f(c)c0δ(c=c0)c
AUC/PAUC	Eτ[TPR(τ)FPR'(τ)] τ~U(a,b)	√	×	f(c)U(a,b)c
WAUC	Eτ[TPR(τ)FPR'(τ)W(τ)] W(τ) ~ beta(a,b)	√	×	f(c)beta(a,b)c
Our method	Eτ~τ* [TPR(τ)FPR'(τ)] τ* ∈ arg minτLCost	√	√	f(c)Sampled cost distributionc
truncated uniform cost distribution [31]. Recently, some studies extend PAUC using parameterized cost distributions and propose WAUC to fit real-world applications [16, 30].
Whether we use AUC or cost learning, our main purpose is to train models with these attributes: Attr.1: The trained model can be robust to class distribution shift in the test without class prior.
Attr.2: The trained model can be robust to cost distribution in the test without cost prior.
However, to the best of our knowledge, there are few methods can train a model to have both of these attributes. According to Tab. 1, both AUC-related methods and cost-sensitive learning require a strong prior knowledge of the cost distribution; (1) The cost learning mainly considers a specified cost  $c$  and class imbalanced radio  $\pi$ . Models trained under this method are sensitive to class distribution, which does not apply to the scenario where test data distribution with offset. (2) AUC (PAUC) assumes that the cost distribution belongs to (truncated) uniform cost distribution  $U(a,b)$ . Models trained with them will have poor performance when the true cost distribution is not uniform [16]. (3) WAUC considers optimizing models based on more complex forms of cost distribution, such as  $beta(a,b)$ . However, we can not obtain the cost prior in real problem scenarios, e.g., financial market prediction [14]. Considering the weakness comes from the existing settings, we will explore the following question in this paper:
Can we bridge AUC and complicated cost distribution to training robust model on desired cost-sensitive and arbitrary class imbalanced decision scenarios?
To answer this question, we propose a view that, in some real applications [14], the cost, like the instance data, is not available prior but can be obtained by sampling. Therefore, we choose to sample desired cost to approximate the true cost distribution. Different from previous settings, ours is closer to real world, the main process can be divided into three parts:
Step.1 Cost Sampling: Firstly, we sample some desired costs to construct the empirical cost set.
Step.2 Data Sampling: Next, we sample some instance data to construct the empirical dataset.
Step.3 Build Formulation: Finally, we construct the appropriate formulation to maximize the performance in different desired costs and ensure model is robust to distribution shift.
It is natural for us to ask the question: Can we use the existing methods to realize this process? It's clear the answer is no. For AUC-related methods, they can not perform Step.1, and for cost-sensitive learning, they fail to achieve robust distribution shift and multiple costs in Step.3 (as shown in Fig. 1 (orange line). Hence, we propose a novel bilevel formulation combining the advantages of WAUC and cost learning. The inner-level process calculates the optimal threshold from sampling costs, and the outer-level process minimizes the WAUC loss over the optimal threshold distribution. The method can help the model improve robustness to class distribution in cost-sensitive decision problems. The main process is shown in Fig. 1 (green line). We summarize our contributions below:
- We propose a setting that focuses on the robustness of the model to the class distribution and cost distribution simultaneously. This setting treats cost as data that can be sampled, not as prior information, which is closer to the real-world cost-sensitive scenario.
Figure 1: The comparison of our proposed setting with the previous setting. The orange line represents the previous cost-sensitive learning approach, and the green line represents our method.
- We present a bilevel paradigm where the inner cost function is an inner constraint of outer WAUC optimization. For sake of optimization, we reformulate this paradigm into a nonconvex-strongly convex bilevel form. Moreover, we employ a stochastic optimization algorithm for WAUC (SACCL), which can solve this problem efficiently.
- We conduct extensive experiments on multiple imbalanced cost-sensitive classification tasks. The experimental results speak to the effectiveness of our proposed methods.
2 Observation and Motivation
In Tab. 1, we compare the existing methods with ours from different views. However, the table comparison does not have a very visual presentation. In this section, we will analyze the disadvantages of existing settings and explain our motivation. We train the model with all methods on a Cifar-10-Long-Tail training set under the different imbalanced ratios and cost distribution. We visualize the feature representation (the last layer's output of the trained model) in test data by t-SNE. The blue point represents negative samples predicted by the optimal threshold, and the orange point represents positive samples. The smaller the overlap between them, the better the performance. From Fig. 2, we can make the following remarks: (1) According to Fig. 2 (a), AUC is robust to changes in the imbalance ratio but completely not applicable with the cost distribution. (2) According to Fig. 2 (b), cost learning can process different cost distributions, but is sensitive to imbalance ratios.
(a) Tsne result of repensentation (AUC)
Figure 2: The feature representations comparison among different methods. (a) AUC optimization. (b) Cost learning. (c) Ours result. We solve the optimal threshold with the  $\hat{\mathcal{L}}_{COST}$  (defined in Sec. 3).  $\pi$  denotes the probability of positive class,  $c$  denotes the cost ratio of misclassification ( $U$  denotes Uniform,  $N$  denotes Normal). For example,  $\pi = 0.5$ ,  $c \sim N$  means model tested on dataset which has imbalanced ratio  $\pi$  and cost set sampled from  $N$ .
(b) Tsne result of repensation (COST)
(c) Tsne result of repensentation (Ours)
Hence, our motivation is to propose a new approach to solve the problems in AUC and Cost-sensitive learning. As shown in Fig. 2 (c), our proposed method can have better learning results in various complex cost distributions and imbalance ratios. That means our method can overcome the shortcomings of traditional AUC and cost-learning, which perfectly fits the proposed setting.
Notations. In this section, we give definitions and preliminaries about AUC. We denote  $(\pmb{x},y)$  be an instance, where  $\pmb{x}$  is drawn from feature space  $\mathcal{X} \subseteq \mathbb{R}^d$  ( $d$  is feature number) and  $y$  is drawn from
label space  $\mathcal{V} = \{0,1\}$ . Let  $\mathcal{D}_{\mathcal{P}}(\mathcal{D}_{\mathcal{N}}\text{resp.})$  be positive (negative resp.) instance distribution. Let  $\pmb{x}^{+}\sim \mathcal{D}_{\mathcal{P}}(\pmb{x}^{-}\sim \mathcal{D}_{\mathcal{N}}\text{resp.})$  be positive (negative resp.) instance. We denote  $S_{+} = \{(x_{i}^{+},y_{i})\}_{i = 1}^{n_{+}}$  ( $S_{-} = \{(x_{j}^{-},y_{j})\}_{j = 1}^{n_{-}}$  resp.) as a set of training data drawn from  $\mathcal{D}_{\mathcal{P}}(\mathcal{D}_{\mathcal{N}}\text{resp.})$ , where  $n_+(n_-)$  denotes the instance number of  $S_{+}$  ( $S_{-}\text{resp.}$ ). Let  $\mathbb{I}_{(\cdot)}$  be the indicator function, which returns 1 when the condition is true and 0 otherwise. In this paper, we focus on the deep neural network scoring function  $s(\pmb {w},\pmb {x}):X\mapsto [0,1]$ , where parameterized by  $\pmb{w}$  on an input  $\pmb{x}$ .
AUC & WAUC. For specified threshold  $\tau$ , the TPR of a classifier derived from  $s(\boldsymbol{w}, \boldsymbol{x})$  measures the likelihood that it accurately predicts a positive instance when getting a random positive instance from  $\mathcal{D}_{\mathcal{P}}$ . Formally, we have:
$$
\left(\operatorname {P o p .}\right) \operatorname {T P R} _ {s} (\tau) = \mathbb {P} _ {\boldsymbol {x} ^ {+} \sim \mathcal {D} _ {\mathcal {P}}} [ s (\boldsymbol {w}, \boldsymbol {x} ^ {+}) > \tau ] \quad \left(\operatorname {E m p .}\right) \widehat {\operatorname {T P R}} _ {s} (\tau) = \frac {1}{n _ {+}} \sum_ {i = 1} ^ {n _ {+}} \mathbb {I} _ {s (\boldsymbol {w}, \boldsymbol {x} _ {i} ^ {+}) > \tau}. \tag {1}
$$
In a similar spirit, the classifier's FPR on threshold  $\tau$  refers to the probability that it predicts positive when it gets a negative instance from  $\mathcal{D}_{\mathcal{N}}$ .
$$
\left(\operatorname {P o p .}\right) \operatorname {F P R} _ {s} (\tau) = \mathbb {P} _ {\boldsymbol {x} ^ {-} \sim \mathcal {D} _ {\mathcal {N}}} [ s (\boldsymbol {w}, \boldsymbol {x} ^ {-}) > \tau ] \quad \left(\operatorname {E m p .}\right) \widehat {\operatorname {F P R}} _ {s} (\tau) = \frac {1}{n _ {-}} \sum_ {j = 1} ^ {n _ {-}} \mathbb {I} _ {s (\boldsymbol {w}, \boldsymbol {x} _ {j} ^ {-}) > \tau}. \tag {2}
$$
AUC measures a scoring function's trade-off between TPR and FPR under uniform thresholds. Denote  $\tau$  drawn from the distribution  $\mathcal{D}_{\tau}$ , WAUC utilizes the threshold distribution explicitly based on AUC.  $\mathrm{FPR}_s^\prime$  denotes the probability density function of  $s(\boldsymbol {w},\boldsymbol{x}^{-})$
$$
\mathrm {A U C} = \int_ {\infty} ^ {- \infty} \mathrm {T P R} _ {s} (\tau) \mathrm {F P R} _ {s} ^ {\prime} (\tau) d \tau \tag {3a}
$$
$$
\mathrm {W A U C} = \int_ {\infty} ^ {- \infty} \mathrm {T P R} _ {s} (\tau) \mathrm {F P R} _ {s} ^ {\prime} (\tau) p (\tau) d \tau , \tag {3b}
$$
Cost function [2]. In some real application scenarios, we need to consider the misclassification cost. We denote  $c_{(\cdot)}$  as misclassification cost for class  $(\cdot)$ , cost  $c$  drawn from  $\mathcal{D}_c$ . Since we could not obtain the cost distribution  $\mathcal{D}_c$ , we sample empirical set  $S_c = \{c_l\}_{l=1}^{n_c}$ ,  $n_c$  denotes the sample number of cost  $c$ . Given a scoring function  $s$  and parameter  $w$ , the cost function  $\mathcal{L}_{COST}$  is (the empirical version of cost function,  $\widehat{\mathcal{L}}_{COST}$  contains the empirical forms of TPR and FPR):
$$
\mathcal {L} _ {C O S T} (\boldsymbol {w}, c, \tau^ {*} (c)) = c \cdot \pi \cdot \left(1 - \operatorname {T P R} _ {s} \left(\tau^ {*} (c)\right)\right) + (1 - c) \cdot (1 - \pi) \cdot \operatorname {F P R} _ {s} \left(\tau^ {*} (c)\right), \tag {4}
$$
where  $\pi = n_{+} / (n_{+} + n_{-})$ ,  $c = c_{+} / (c_{+} + c_{-})$  and  $\tau^{*}(c)$  is optimal threshold for score function  $s$  under specified  $c$  [21], the sample number  $n_{\tau} = n_{c}$ .
In this section, we introduce how to link the ROC curve to the cost space. First, we reformulate Eq.(3) into expectation:
$$
\mathrm {A U C} = \underset {\tau \sim U} {\mathbb {E}} \left[ \mathrm {T P R} _ {s} (\tau) \cdot \mathrm {F P R} _ {s} ^ {\prime} (\tau) \right] \tag {5a}
$$
$$
\operatorname {W A U C} = \underset {\tau \sim \mathcal {D} _ {\tau}} {\mathbb {E}} \left[ \mathrm {T P R} _ {s} (\tau) \cdot \mathrm {F P R} _ {s} ^ {\prime} (\tau) \right]. \tag {5b}
$$
If threshold  $\tau$  is drawn from the uniform distribution  $U$ , WAUC will degrade to the standard AUC formulation. However, AUC only describes the global mean performance under all possible costs. If we want to extend AUC to cost-sensitive problems, maybe lifting the restriction on the uniform distribution of  $\tau$  is a good solution. Hence, we release the  $\mathcal{D}_{\tau}$ 's restriction to make it belongs to complicated distribution (e.g., normal distribution, exponential distribution). Then we can extend AUC to WAUC. However, using WAUC raises another question: how do we get  $\mathcal{D}_{\tau}$ ? We find that  $\tau^{*}(c)$  is one of parameters of  $\mathcal{L}_{COST}(\boldsymbol{w}, c, \tau^{*}(c))$ . A natural idea is to use  $\mathcal{L}_{COST}$  to solve for the optimal  $\tau^{*}(c)$  and to combine the  $\tau^{*}(c)$  solved for at different  $c$  to obtain  $\mathcal{D}_{\tau}$ .
$$
\tau^ {*} (c) = \arg \min  _ {\tau} \mathcal {L} _ {C O S T} (\boldsymbol {w}, \tau , c) = c \cdot \pi \cdot \left(1 - \operatorname {T P R} _ {s} (\tau)\right) + (1 - c) \cdot (1 - \pi) \cdot \operatorname {F P R} _ {s} (\tau), \tag {6}
$$
If we couple Eq.(5b) and Eq.(6) together so that WAUC can enjoy the optimal threshold distribution in  $\mathcal{L}_{COST}$ , then we can break the barrier between the ROC curve and the cost space. With the help of the threshold as a bridge, we can extend the AUC metric to achieve the WAUC cost-sensitive learning. Then we give the problem formulation (intuitively, from the result of 2 (c), (OPO) satisfies both Attr.1 and Attr.2 simultaneously):
$$
\begin{array}{l} (O P 0) \quad (\text {o u t e r .}) \quad \mathrm {W A U C} = \underset {\tau \sim \tau^ {*}} {\mathbb {E}} \left[ \mathrm {T P R} _ {s} (\tau) \cdot \mathrm {F P R} _ {s} ^ {\prime} (\tau) \right] \\ (\text {i n n e r .}) \quad \tau^ {*} = \left\{\tau^ {*} (c) = \arg \min  _ {\tau} \mathcal {L} _ {C O S T} (\boldsymbol {w}, \tau , c) | c \sim \mathcal {D} _ {c} \right\} \end{array} \tag {7}
$$
Nevertheless, there are still three main challenges in WAUC cost-sensitive learning:
(1) Given the scoring function  $s$  and negative dataset  $S_{-}$ , how to estimate  $\mathrm{FPR}_s^\prime (\tau)$  in WAUC?
(2) The inner problem is nonconvex, which is hard to give a theoretical convergence guarantee.
(3) How to design a formulation that can bridge WAUC and  $\mathcal{L}_{COST}$  so that WAUC can be optimized over the cost distribution of the desired problem scenario?
We will address the challenge (1) in Sec. 5.1, challenge (2) in Sec. 5.2 and challenge (3) in Sec. 5.3.
5.1 The Estimation of False Positive Rate
For challenge (1), we choose the kernel density estimation (KDE) to estimate  $\mathrm{FPR}_s^\prime (\tau)$  and denote it as  $K(x)$  (please see definition in Sec. C.1). Then we can address the density estimation problem. However, Eq.(5b) still exists non-differentiable and non-smooth term  $\mathbb{I}_{(\cdot)}$ , which is hard to optimize. Hence, we propose the following smooth and differentiable WAUC estimator to approximate Eq.(5b).
Definition 5.1. Denote  $K(x)$  be statistics kernel with bandwidth  $m$  and  $S_{-}^{w} = \{s(\boldsymbol{w},\boldsymbol{x}_{j}^{-})\}_{j = 1}^{n - }$ . With Lemma 5.2, we have the approximate estimator and loss function for WAUC:
$$
\widehat {\mathrm {W A U C}} = \int_ {\infty} ^ {- \infty} \operatorname {T P R} _ {s} (\tau) \mathcal {K} \left(S _ {-} ^ {\boldsymbol {w}}, \tau\right) p (\tau) d \tau , \quad \widehat {\mathcal {L}} _ {\mathrm {W A U C}} (\boldsymbol {w}, \tau) = \frac {1}{n _ {\tau}} \sum_ {l = 1} ^ {n _ {\tau}} \hat {h} (\boldsymbol {w}, \tau_ {l}) \tag {8}
$$
where  $\tau = \{\tau_{l}\}_{l = 1}^{n_{\tau}}$  and the point loss  $\hat{h}$  is defined by
$$
\hat {h} (\boldsymbol {w}, \tau) = 1 - \frac {1}{n _ {+} n _ {-}} \sum_ {i = 1} ^ {n _ {+}} \sum_ {j = 1} ^ {n _ {-}} \sigma \left(s \left(\boldsymbol {w}, \boldsymbol {x} _ {i} ^ {+}\right) - \tau_ {l}\right) \cdot K \left(\left(s \left(\boldsymbol {w}, \boldsymbol {x} _ {j} ^ {-}\right) - \tau_ {l}\right) / m\right) / m. \tag {9}
$$
$\sigma(x) = 1 / (1 + \exp(-\beta x))$ ,  $\beta$  is smooth parameter and we have  $\sigma(x) \stackrel{\beta \to \infty}{\longrightarrow} \mathbb{I}_x$ .
Lemma 5.2. Given a scoring function  $s$ , if  $\tau$  is known, when the number of instances is large enough,  $\widehat{\mathrm{WAUC}}$  almost surely converges to WAUC.
$$
\lim  _ {n _ {-} \rightarrow \infty} | \widehat {\mathrm {W A U C}} - \mathrm {W A U C} | \xrightarrow {a . s .} 0. \tag {10}
$$
With KDE consistency [41], when the negative sample size is large enough, Lemma 5.2 provides theoretical approximation guarantees for our proposed WAUC estimator in Prop. 5.1.
5.2 The Estimation of Threshold Weighting
For challenge (2), a natural idea is to use  $\widehat{\mathcal{L}}_{COST}$  to solve for the optimal threshold set  $\hat{\tau}^*$  when given the cost set  $S_{c}$  and the scoring function  $s$ . Then we can use the optimal threshold set  $\hat{\tau}^*$  to calculate  $\widehat{\mathrm{WAUC}}$ . Firstly, we define the solution for  $\hat{\tau}^*$  be
$$
\hat {\boldsymbol {\tau}} ^ {*} = \left\{\hat {\tau} ^ {*} (c) | \hat {\tau} ^ {*} (c) \in \arg \min  _ {\tau} \widehat {\mathcal {L}} _ {C O S T} (\boldsymbol {w}, c), c \in S _ {c} \right\}. \tag {11}
$$
However, it's noticed that the arg  $\min_{\tau} \widehat{\mathcal{L}}_{COST}$  in Eq.(11) is non-convex. As we analyzed before,  $\pmb{\tau}^{*}$  is the inner constraint of WAUC. To the best of our knowledge, there are few studies on optimizing two coupled non-convex problems simultaneously with theoretical convergence guarantees. Most studies on coupled optimization assume that the inner problems have good properties, such as strong convexity. Hence, we propose the approximated convex formulation of the inner problem for  $\hat{\pmb{\tau}}^{*}$ .
Theorem 5.3. When we set  $\kappa, M$  are large positive numbers and  $M'^2 < M^2 \frac{6\kappa^2 e^{3\kappa}}{(e^\kappa + 1)^6}$ , then we have the approximated convex formulation for  $\hat{\mathcal{L}}_{COST}$
$$
\begin{array}{l} \min  _ {\tau , \boldsymbol {P} \in \mathbb {R} ^ {n _ {+}}, \boldsymbol {N} \in \mathbb {R} ^ {n _ {-}}} \widehat {\mathcal {L}} _ {e q} (\boldsymbol {w}, \tau , c) := c \cdot \pi \cdot \left(1 - \frac {1}{n _ {+}} \sum_ {i = 1} ^ {n _ {+}} P _ {i}\right) + (1 - c) \cdot (1 - \pi) \cdot \left(\frac {1}{n _ {-}} \sum_ {j = 1} ^ {n _ {-}} N _ {j}\right) \\ + \frac {1}{n _ {+}} \sum_ {i = 1} ^ {n _ {+}} M ^ {\prime} \psi \left(s \left(\boldsymbol {w}, \boldsymbol {x} _ {i} ^ {+}\right) - \tau\right) - P _ {i} \left(s \left(\boldsymbol {w}, \boldsymbol {x} _ {i} ^ {+}\right) - \tau\right)) + M \psi \left(P _ {i} - 1\right) + M \psi (\tau - 1) \tag {12} \\ + \frac {1}{n _ {-}} \sum_ {j = 1} ^ {n _ {-}} M ^ {\prime} \psi (s (\boldsymbol {w}, \boldsymbol {x} _ {j} ^ {-}) - \tau) - N _ {j} (s (\boldsymbol {w}, \boldsymbol {x} _ {j} ^ {-}) - \tau)) + M \psi (N _ {j} - 1) 0 \leq \tau , P _ {i}, N _ {j} \\ \end{array}
$$
where  $\psi(x) = \log(1 + \exp(\kappa x)) / \kappa$ .  $\widehat{\mathcal{L}}_{eq}$  in Eq.(12) is  $\mu_g$ -strongly convex w.r.t.  $\pmb{\tau}$ . Eq.(12) has same solution as  $\min_{\tau} \widehat{\mathcal{L}}_{COST}$  when the parameters satisfy the conditions of the penalty.
Thm. 5.3 provides an optimization method with good properties. Eq. (12) adopts the penalty function to convert inequality constraints into a part of the objective function. When these inequality constraints are not satisfied, the objective function will increase to infinity. Otherwise, we will get  $P_{i} = \mathbb{I}[s(\boldsymbol{w}, x_{i}^{+}) > \tau]$  and  $N_{j} = \mathbb{I}[s(\boldsymbol{w}, x_{j}^{-}) > \tau]$ , then we will get the same formulation as  $\widehat{\mathcal{L}}_{COST}$ . When the parameters meet the requirements, Eq.(12) has the same solution as  $\widehat{\mathcal{L}}_{COST}$ . We give the proof of Thm. 5.3 and the definition  $\mu$  in Sec. C.4. Moreover, we give the analysis of approximation error between  $\widehat{\mathcal{L}}_{COST}$  and Thm. 5.3 in Sec. B.7.
5.3 Bilevel Optimization for WAUC learning
After answering questions in challenge (1) and (2), we have solved most of the problems in WAUC cost-sensitive learning. However, there remains a challenge (3) in optimization: How do we design learning paradigms to solve the coupled optimization problem of WAUC and  $\mathcal{L}_{COST}$ ? In recent years, bilevel optimization has achieved remarkable success. This approach can combine two related optimization problems to form a coupled optimization formulation. Hence, with Prop. 5.1 and Thm.5.3, we propose a bilevel paradigm to formulate this coupled optimization problem.
$$
\begin{array}{l} \left(O P 1\right) \quad \text {(o u t e r .)} \quad \min  _ {\boldsymbol {w}} \hat {F} (\boldsymbol {w}) := \hat {f} (\boldsymbol {w}, \boldsymbol {\tau} ^ {*}) := \widehat {\mathcal {L}} _ {\mathrm {W A U C}} (\boldsymbol {w}, \hat {\boldsymbol {\tau}} ^ {*}) \\ \text {(i n n e r .)} \quad \hat {\boldsymbol {\tau}} ^ {*} = \underset {\boldsymbol {\tau}, \boldsymbol {P} _ {a}, \boldsymbol {N} _ {a}} {\arg \min } \hat {g} (\boldsymbol {w}, \boldsymbol {\tau}) := \frac {1}{n _ {\tau}} \sum_ {l = 1} ^ {n _ {\tau}} \widehat {\mathcal {L}} _ {e q} (\boldsymbol {w}, \tau_ {l}, c _ {l}), \tag {13} \\ \end{array}
$$