SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $P_{a} \in \mathbb{R}^{n_{\tau} \times n_{+}}$  and  $N_{a} \in \mathbb{R}^{n_{\tau} \times n_{-}}$ . (OP1) describes a bilevel optimization formulation for WAUC cost-sensitive learning, where the inner-level provides a threshold optimization process, and the outer-level minimizes the WAUC loss over the optimal threshold distribution. Moreover, this formulation is consistent with the mainstream bilevel optimization problem (outer-level is smooth and non-convex, inner-level is convex and smooth), which enjoys a faster convergence rate.
6 Optimization Algorithm
In this section, we focus on optimizing (OP1) in an end-to-end manner. Hence, we propose a stochastic algorithm for WAUC cost-sensitive learning shown in Alg. 1, which is referred to SAACL.
6.1 Main Idea of SAACL
We provide some intuitive explanations of our algorithm. At each iteration  $k$ , SACCL alternates between the inner-level gradient update on  $\tau$  and the outer-level gradient update on  $w$ . During
Algorithm 1 Stochastic Algorithm for WAUC Cost-sensitive Learning
Input: training data  $S_{+}$  and  $S_{-}$ , iteration numbers  $K$  and  $T$ , batch size  $B$ .
Initialize: parameters  $\pmb{w}_0\in \mathbb{R}^n$ $\pmb {\tau}_0\in \mathbb{R}^{n_\tau}$  , stepsizes  $\alpha_{k},\beta_{k}$
for  $k = 0$  to  $K$  do
set  $\pmb{\tau}_{k,0} = \pmb{\tau}_k$
for  $t = 0$  to  $T$  do
drawn  $\mathcal{B}_t = \{(x_b,y_b)\}_{b = 1}^B$  from  $S_{+}$  and  $S_{-}$  uniformly.
$\forall c_{l}\in S_{c},\tau_{k,t + 1}^{l} = \tau_{k,t}^{l} - \beta_{k}\nabla_{\pmb{\tau}}\hat{g} (\pmb {w}_{k},c_{l};\mathcal{B}_{t}).$
set  $\pmb{\tau}_{k + 1} = \pmb{\tau}_{k,T}$
drawn  $\mathcal{B}_k = \{(x_b,y_b)\}_{b = 1}^B$  from  $S_{+}$  and  $S_{-}$  uniformly.
$\pmb{w}_{k + 1} = \pmb{w}_k - \alpha_k[\nabla_{\pmb{w}}\hat{f} (\pmb {w}_k,\pmb{\tau}_{k + 1};\mathcal{B}_k) - \nabla_{\pmb{w}\tau}^2\hat{g} (\pmb {w}_k,\pmb{\tau}_{k + 1};\mathcal{B}_k)\cdot$
$\left[\frac{N}{L_{g,1}}\prod_{n = 1}^{N^{\prime}}\left(I - \frac{1}{L_{g,1}}\nabla_{\pmb{\tau}\pmb{\tau}}^{2}\hat{g} (\pmb {w},\pmb{\tau}_{k + 1};\mathcal{B}_{k})^{-1}\right)\right]\nabla_{\pmb{\tau}}\hat{f} (\pmb {w}_k,\pmb {\tau}_{k + 1};\mathcal{B}_k)$
iteration  $k$ , we update  $\pmb{\tau}_{k,t}$  with standard SGD  $T$  steps to ensure that  $\pmb{\tau}_{k+1}$  is as optimal as possible. After updating inner-level variables, we perform outer-level optimization with  $\pmb{\tau}_{k+1}$  as the parameter to update  $\pmb{w}_k$ . Notice that  $T$  will not take a large value to ensure the validity of the coupling update of  $\pmb{\tau}$  and  $\pmb{w}$ . Let  $\alpha_k$  and  $\beta_k$  be stepsizes of  $\pmb{w}$  and  $\pmb{\tau}$  that have the same decrease rate as SGD. We denote  $n$  be the number of elements in deep neural network parameters  $\pmb{w}$ .
6.2 Convergence Analysis of SAACL
In this subsection, we present the convergence analysis for SAACL. We give some Lipschitz continuity assumptions that are common in bilevel optimization problems [11, 28].
Assumption 6.1. (Lipschitz continuity) Assume that  $f$ ,  $\nabla f$ , and  $\nabla g$  are respectively  $L_{f,0}, L_{f,1}$ ,  $L_{g,1}$ -Lipschitz continuous.
Assumption 6.2. (Bounded stochastic derivatives) The variance of stochastic derivatives  $\nabla f(\boldsymbol{w},\tau;\mathcal{B})$  and  $\nabla g(\boldsymbol{w},\tau;\mathcal{B})$  are bounded by  $\sigma_{f,1}^2$ ,  $\sigma_{g,1}^2$ , respectively.
Based on Assumption 6.1 and Assumption 6.2, following [5], Thm. 6.3 indicates that we can optimize (OP1) with the same convergence rate as the traditional SGD algorithm.
Theorem 6.3. Suppose Assumption 6.1 and 6.2 hold. We define
$$
\bar {\alpha} _ {1} = \frac {1}{2 L _ {F} + 4 L _ {f} L _ {y} + 2 L _ {f} L _ {y x} / \left(L _ {y} \eta\right)}, \quad \bar {\alpha} _ {2} = \frac {1 6 T \mu L _ {g , 1}}{\left(\mu + L _ {g , 1}\right) ^ {2} \left(8 L _ {f} L _ {y} + 2 \eta L _ {y x} \tilde {C} _ {f} ^ {2} \bar {\alpha} _ {1}\right)}, \tag {14}
$$
where  $\eta = L_F / L_y$ ,  $L_F$ ,  $L_f$ ,  $L_y$  and  $\tilde{C}_f^2$  come from Lem. 2 and Lem. 4 in [5]. We select the following stepsize as
$$
\alpha_ {k} = \min  \left\{\bar {\alpha} _ {1}, \bar {\alpha} _ {2}, \frac {1}{\sqrt {K}} \right\} \quad \beta_ {k} = \frac {8 L _ {f} L _ {y} + 2 \eta L _ {y x} \tilde {C} _ {f} ^ {2} \bar {\alpha} _ {1}}{4 T \mu} \alpha_ {k} \tag {15}
$$
For any  $T\geq 1$  , the iteration sequence  $\{\pmb {w}_k\}_{k = 1}^K$  and  $\{\pmb {\tau}_k\}_{k = 1}^K$  generated by Algorithm 1 satisfy
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \| \nabla F (\boldsymbol {w} _ {k}) \| ^ {2} \right] \leq \gamma \left(\frac {3 M \kappa e ^ {\kappa} / (e ^ {\kappa} + 1) ^ {2} + L _ {g , 1}}{2 4 M \kappa e ^ {\kappa} / (e ^ {\kappa} + 1) ^ {2} L _ {g , 1}}\right) ^ {2} \frac {1}{T \sqrt {K}} + O \left(\frac {1}{\sqrt {K}}\right). \tag {16}
$$
where  $\gamma = 2\alpha \sigma_{g,1}^{2}\frac{L_{f}}{L_{y}}\left(1 + 5L_{f}L_{y}\bar{\alpha}_{1} + \frac{\eta L_{yx}\tilde{C}_{f}^{2}}{4}\bar{\alpha}_{1}^{2}\right)$ $(8L_{f}L_{y} + 2\eta L_{yx}\tilde{C}_{f}^{2}\bar{\alpha}_{1})^{2}$
Remark 6.4. When  $\kappa$  and  $M$  are large enough positive integers, according to Eq. (16), Alg. 1 is still guaranteed to find a  $\epsilon$ -stationary point within  $O(\epsilon^{-4})$  iterations ( $\epsilon$  is error tolerance).
In this section, we conduct a series of experiments for WAUC cost-sensitive learning on common long-tail benchmark datasets. Due to space limitations, please refer to Sec. B for the details of our experiments. The source code is available in supplemental materials.
7.1 Dataset Details
We use three datasets: Binary CIFAR-10-Long-Tail Dataset [23], Binary CIFAR-100-Long-Tail Dataset [23], and Jane Street Market Prediction [14]. Binary CIFAR-10-Long-Tail Dataset and Binary CIFAR-100-Long-Tail Dataset are common datasets in long-tail learning, and we construct their cost distributions. Jane Street Market Prediction is data from real cost-sensitive learning application scenarios. For all datasets, we divide them into the training set, validation set, and test set with a proportion 0.7:0.15:0.15. All image data is normalized to ensure a more stable training process.
Table 2: Performance comparisons on benchmark datasets with different metrics. The first and second best results are highlighted with bold text and underline, respectively.
dataset	type	methods	Subset1	Subset2	Subset3	AUC↑
WAUC↑	\( \widehat{\mathcal{L}}_{COST} \)↓	WAUC↑	\( \widehat{\mathcal{L}}_{COST} \)↓	WAUC↑	\( \widehat{\mathcal{L}}_{COST} \)↓	Subset1	Subset2	Subset3
CIFAR-10-LT	Competitors	BCE	0.525	0.027	0.533	0.015	0.318	0.029	0.822	0.960	0.870
ExAUC	0.516	0.029	0.518	0.013	0.366	0.028	0.845	0.963	0.858
SqAUC	0.407	0.028	0.548	0.012	0.327	0.031	0.811	0.933	0.867
NWAUC	0.565	0.030	0.574	0.017	0.396	0.027	0.786	0.885	0.827
PAUC-exp	0.549	0.029	0.508	0.015	0.354	0.028	0.650	0.801	0.736
PAUC-poly	0.526	0.028	0.470	0.015	0.354	0.029	0.661	0.812	0.742
PAUCI	0.516	0.027	0.520	0.015	0.382	0.028	0.704	0.847	0.734
CS-hinge	0.566	0.026	0.633	0.010	0.377	0.022	0.675	0.782	0.762
AdaCOS	0.576	0.025	0.559	0.014	0.391	0.023	0.758	0.873	0.742
ECL	0.589	0.026	0.561	0.014	0.388	0.020	0.694	0.918	0.762
Our method	WAUC-Gau	0.679	0.024	0.660	0.012	0.467	0.015	0.787	0.934	0.843
WAUC-Log	0.653	0.023	0.674	0.011	0.468	0.014	0.820	0.958	0.869
CIFAR-100-LT	Competitors	BCE	0.556	0.022	0.463	0.012	0.512	0.019	0.912	0.957	0.806
ExAUC	0.522	0.019	0.502	0.011	0.506	0.017	0.933	0.967	0.833
SqAUC	0.483	0.024	0.367	0.015	0.474	0.018	0.889	0.955	0.855
NWAUC	0.654	0.025	0.511	0.016	0.631	0.019	0.867	0.925	0.807
PAUC-exp	0.464	0.020	0.282	0.014	0.469	0.016	0.826	0.811	0.787
PAUC-poly	0.461	0.022	0.262	0.017	0.473	0.017	0.828	0.887	0.791
PAUCI	0.549	0.018	0.439	0.016	0.514	0.018	0.812	0.843	0.822
CS-hinge	0.523	0.017	0.457	0.010	0.515	0.014	0.734	0.910	0.716
AdaCOS	0.590	0.018	0.474	0.011	0.587	0.016	0.769	0.919	0.727
ECL	0.583	0.017	0.497	0.009	0.595	0.015	0.863	0.939	0.794
Our method	WAUC-Gau	0.745	0.015	0.589	0.005	0.728	0.013	0.842	0.928	0.745
WAUC-Log	0.719	0.012	0.560	0.003	0.745	0.010	0.906	0.960	0.875
Table 3: Performance comparisons on benchmark datasets in real world cost-sensitive problem. Profit means represents the money earned by the model over the entire trading period.
Methods	BCE	ExAUC	SqAUC	NWAUC	PAUC-exp	PAUC-poly	PAUCI	CS-hinge	AdaCOS	ECL	WAUC-Gau	WAUC-Log
VAUC †	0.5427	0.594 ± .003	0.508 ± .005	0.562 ± .002	0.576 ± .004	0.481 ± .005	0.529 ± .006	0.592 ± .002	0.6527 ± .005	0.625 ± .004	0.698 ± .002	0.675 ± .001
LcostT ↓	0.254 ± .004	0.269 ± .005	0.246 ± .003	0.251 ± .002	0.270 ± .004	0.246 ± .004	0.243 ± .001	0.237 ± .006	0.229 ± .004	0.226 ± .007	0.209 ± .003	0.213 ± .002
AUC ↑	0.528 ± .005	0.539 ± .004	0.526 ± .005	0.520 ± .004	0.529 ± .002	0.519 ± .005	0.510 ± .003	0.522 ± .005	0.5246 ± .002	0.530 ± .003	0.526 ± .002	0.5235 ± .003
Profit ↑	4955 ± 20.14	5468 ± 17.90	5183 ± 30.91	5395 ± 22.48	5418 ± 14.06	4862 ± 28.04	4963 ± 15.09	5583 ± 30.05	5839 ± 34.92	5764 ± 25.09	6526 ± 15.98	6308 ± 16.09
7.2 Overall Performance
In Tab. 2 and Tab. 3, we collect all the methods' performance on test sets of three types of datasets. For cost distribution of  $c$ , we sample some data from a normal distribution  $\mathcal{N}(0.5,1)$  to construct a dataset  $S_{c}$  (we clip all data to [0,1]). We also conduct numerous experiments for other types of distribution of  $c$ , and please see Appendix B for the details. From the results, we make the following observations:
(1) For WAUC and  $\widehat{\mathcal{L}}_{COST}$  metric, Our proposed algorithm achieves superior performance in most benchmark datasets compared to other methods. This demonstrates that our proposed WAUC cost-sensitive learning can extend the ROC curve into the cost space. Models trained with our proposed bilevel optimization formulation can enjoy high WAUC and cost-related metrics.
(2) AUC and cost-related metrics are inconsistent. From the high-performing heatmap of Tab. 2, it can be noticed that  $\widehat{\mathcal{L}}_{COST}$  and  $\widehat{\mathbf{AUC}}$  have two completely different highlight regions. This indicates that the assumption of uniform distribution of AUC does not match the realistic scenario.
(3) We also find that AUC-related and traditional classification algorithms do not perform well in cost-sensitive problems. This means that if we first train the model with the classification algorithm, subsequently using the cost function to solve for the optimal threshold for decision does not work well. Meanwhile, the algorithm that can learn from scratch has better scalability. Two-stage decision method Therefore, designing one-stage algorithms for WAUC cost-sensitive learning is necessary.
7.3 Sensitivity Analysis
In this subsection, we show the sensitivity of  $\beta, T$ , and bandwidth on test data.
Effect of  $\beta$ . In Fig. 3 (a) and (d), we observe that for both WAUC and cost metrics, when  $\beta$  closes to 7, the model will have the largest performance improvement and the lowest variance. This can be explained in two ways: (1) When the  $\beta$  is too small, the error between the  $\sigma(x)$  and the 0-1 loss function is large, resulting in a large approximation error between the WAUC and the WAUC. (2) When the  $\beta$  is too large, the gradient also tends to be 0. Therefore, choosing a beta value that trades off the approximation error and the gradient is essential.
(a) The effect of  $\beta$  on WAUC
(b) The effect of  $T$  on WAUC
(c) The effect of  $m$  on WAUC
(d) The effect of  $\beta$  on cost
(e) The effect of  $T$  on cost
(f) The effect of  $m$  cost
Figure 3: Sensitivity analysis on test data where WAUC and cost for WAUC-Gau with respect to  $\beta$ ,  $T$ , and bandwidth. The other two variables are fixed for each box in the plots, and the scattered points along the box show the variation.
Effect of  $T$ . As we mentioned in the Section 6.1, choosing a smaller  $T$  can effectively improve the performance of the model. However, as shown in Fig. 3(e), a larger  $T$  value can reduce the variance. Hence, as set in our experiments,  $T = 3$  is a good choice to ensure the average performance and variance of the model.
Effect of  $m$ . From Fig. 3(c), we find that the kernel's bandwidth strongly influences the model's performance. The model's bandwidth and performance are almost proportional; the closer the bandwidth is to [0.4, 0.5], the better the effect; otherwise, the effect is worse. This indicates that our proposed method is sensitive to the bandwidth parameter, which also compounds the bandwidth characteristics in the KDE method.
This paper focuses on extending the traditional AUC metric to associate with misclassification costs. Restricted by the assumption of cost distribution, existing settings could not describe the model's performance in the complicated cost-sensitive scenario. To address this problem, we propose a novel setting that treats the cost as sampled data. We employ the WAUC metric and propose a novel estimator to approximate it. With the help of threshold weighting, we establish the correspondence between WAUC and the cost function. To describe this connection, we present a bilevel optimization formulation to couple them, where the inner-level problem provides a threshold optimization process, and the outer-level minimizes the WAUC loss based on the inner thresholds. This paradigm ensures that the WAUC can always be optimized at the optimal threshold value based on the complicated cost distribution in reality. Moreover, we propose a stochastic algorithm to optimize this formulation. We prove that our algorithm enjoys the same convergence rate as standard SGD. Finally, numerous
experiments have shown that our method can extend AUC to cost-sensitive scenarios with significant performance.
[1] J A, Hanley, B J, and McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 1982.
[2] Daniel Andrade and Yuzuru Okajima. Efficient bayes risk estimation for cost-sensitive classification. In The 22nd international conference on artificial intelligence and statistics, pages 3372-3381. PMLR, 2019.
[3] Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations Research, 21(1):37-44, 1973.
[4] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pages 1507-1517. PMLR, 2021.
[5] Tianyi Chen, Yuejiao Sun, and Wotao Yin. Tighter analysis of alternating stochastic gradient method for stochastic nested problems. arXiv preprint arXiv:2106.13781, 2021.
[6] Benoit Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153(1):235-256, 2007.
[7] Corinna Cortes and Mehryar Mohri. AUC optimization vs. error rate minimization. In Advances in Neural Information Processing Systems, pages 313-320, 2003.
[8] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-9277, 2019.
[9] Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 27(8):861-874, 2006.
[10] Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, Bartosz Krawczyk, Francisco Herrera, Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C Prati, et al. Cost-sensitive learning. Learning from Imbalanced Data Sets, pages 63-78, 2018.
[11] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pages 1568-1577. PMLR, 2018.
[12] Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933-969, 2003.
[13] Thore Graepel, Klaus Obermayer, et al. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers, pages 115-132. 2000.
[14] Jane Street Group. Jane street market prediction. https://www.kaggle.com/competitions/jane-street-market-prediction/overview, 2021.
[15] Zhishuai Guo, Mingrui Liu, Zhuoning Yuan, Li Shen, Wei Liu, and Tianbao Yang. Communication-efficient distributed stochastic auc maximization with deep neural networks. In International Conference on Machine Learning, pages 3864-3874, 2020.
[36] Alain Rakotomamonjy. Support vector machines and area under roc curve. PSI-INSA de Rouen: Technical Report, 2004.
[37] Shoham Sabach and Shimrit Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization, 27(2):640-660, 2017.
[38] HuiYang Shao, Qianqian Xu, Zhiyong Yang, Shilong Bao, and Qingming Huang. Asymptotically unbiased instance-wise regularized partial auc optimization: Theory and algorithm. In Advances in Neural Information Processing Systems, 2022.
[39] Luis N Vicente and Paul H Calamai. Bilevel and multilevel programming: A bibliography review. Journal of Global optimization, 5(3):291-306, 1994.
[40] Sam Wieand, Mitchell H Gail, Barry R James, and Kang L James. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika, 76(3): 585-592, 1989.
[41] Dominik Wied and Rafael Weibach. Consistency of the kernel density estimator: a survey. Statistical Papers, 53(1):1-21, 2012.
[42] Lian Yan, Robert H Dodier, Michael Mozer, and Richard H Wolniewicz. Optimizing classifier performance via an approximation to the wilcoxon-mann-whitney statistic. In International Conference on Machine Learning, pages 848-855, 2003.
[43] Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey. arXiv preprint arXiv:2203.15046, 2022.
[44] Zhenhuan Yang, Wei Shen, Yiming Ying, and Xiaoming Yuan. Stochastic auc optimization with general loss. CPAA, 19(8), 2020.
[45] Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. When all we need is a piece of the pie: A generic framework for optimizing two-way partial auc. In International Conference on Machine Learning, pages 11820-11829, 2021.
[46] Yao Yao, Qihang Lin, and Tianbao Yang. Large-scale optimization of partial auc in a range of false positive rates. arXiv preprint arXiv:2203.01505, 2022.
[47] Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online auc maximization. Advances in Neural Information Processing Systems, 29:451-459, 2016.
[48] Zhuoning Yuan, Zhishuai Guo, Nitesh Chawla, and Tianbao Yang. Compositional training for end-to-end deep auc maximization. In International Conference on Learning Representations, 2021.
[49] Adriano Z Zambom and Dias Ronaldo. A review of kernel density estimation with applications to econometrics. International Econometric Review, 5(1):20-42, 2013.
[50] Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, and Tianbao Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. arXiv preprint arXiv:2203.00176, 2022.