SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
D R A U C _ {\epsilon_ {+}, \epsilon_ {-}} ^ {D a} \left(f _ {\boldsymbol {\theta}}, \widehat {P}\right) = 1 - \max  _ {\substack {\widehat {Q} _ {+}: \mathcal {W} _ {c} \left(\widehat {Q} _ {+}, \widehat {P} _ {+}\right) \leq \epsilon_ {+} \\ \widehat {Q} _ {-}: \mathcal {W} _ {c} \left(\widehat {Q} _ {-}, \widehat {P} _ {-}\right) \leq \epsilon_ {-}}} \mathbb {E} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(x _ {i} ^ {+}\right) - f _ {\boldsymbol {\theta}} \left(x _ {j} ^ {-}\right)\right) \right]. \tag{17}
$$
For simplicity, let us denote
$$
\widehat {Q} = \left\{\widehat {Q} \mid \mathcal {W} _ {c} \left(\widehat {Q} _ {+}, \widehat {P} _ {+}\right) \leq \epsilon_ {+}, \mathcal {W} _ {c} \left(\widehat {Q} _ {-}, \widehat {P} _ {-}\right) \leq \epsilon_ {-} \right\} \tag {18}
$$
Similar to DRAUC-Df, we construct our reformulation as follows:
$$
(\mathbf {D a}) \quad \min  _ {\mathbf {w}} \max  _ {\alpha \in \Omega_ {\alpha}} \max  _ {\widehat {Q} \in \widehat {Q} _ {+}, \widehat {Q} _ {-}} \mathbb {E} [ g (a, b, \alpha , \boldsymbol {\theta}, z _ {i}) ]. \tag {19}
$$
Moreover, we conduct a similar derivation as DRAUC-Df, to construct a tractable upper bound:
$$
\left(\boldsymbol {D} \boldsymbol {a} \star\right) \min  _ {\mathbf {w}} \max  _ {\lambda_ {+}, \lambda_ {-} \geq 0} \max  _ {\alpha \in \Omega_ {\alpha}} \left\{\lambda_ {+} \epsilon_ {+} + \lambda_ {-} \epsilon_ {-} + \widehat {p} \underset {\widehat {P} _ {+}} {\mathbb {E}} \left[ \phi_ {\mathbf {w}, \lambda_ {+}, \alpha} (z) \right] + (1 - \widehat {p}) \underset {\widehat {P} _ {-}} {\mathbb {E}} \left[ \phi_ {\mathbf {w}, \lambda_ {-}, \alpha} (z) \right] \right\} \tag {20}
$$
where  $\phi_{\mathbf{w},\lambda_+,\alpha}(z) = \max_{z'\in \mathcal{Z}}[g(\mathbf{w},\alpha ,z) - \lambda_+c(z,z')]$  and  $\phi_{\mathbf{w},\lambda_{-},\alpha}(z) = \max_{z'\in \mathcal{Z}}[g(\mathbf{w},\alpha ,z) - \lambda_{-}c(z,z')]$ . Please see Appendix A for the details.
Algorithm 2 Algorithm for optimizing DRAUC-Da:
1: Input: the training data  $\mathcal{Z}$ , step number  $K$ , step size for inner  $K$ -step gradient ascent  $\eta_z$ , learning rates  $\eta_{\lambda}, \eta_w, \eta_\alpha$  and maximal corrupt distance  $\epsilon_+$ ,  $\epsilon_-$ .  
2: Initialize: initialize  $a^0, b^0, \alpha^0 = 0, \lambda_+^0 = \lambda_-^0 = \lambda_0$ .  
3: for  $t = 1$  to  $T$  do  
4: Sample a batch of example  $z$  from  $\mathcal{Z}$ .  
5: Generate Local Worst-Case Examples:  
6: Initialize  $z'_+ = z_+$ ,  $z'_- = z_-$ .  
7: for  $k = 1$  to  $K$  do  
8:  $z'_+ = \Pi_{\mathcal{Z}}(z'_+ + \eta_z \cdot \nabla_z \phi_{\lambda_+^t, a, b, \alpha}(\theta, z'_+))$ .  
9:  $z'_- = \Pi_{\mathcal{Z}}(z'_- + \eta_z \cdot \nabla_z \phi_{\lambda_-^t, a, b, \alpha}(\theta, z'_-))$ .  
10: end for  
11: Update Parameters:  
12: Update  $\alpha^{t+1} = \Pi_{\Omega_\alpha}(\alpha^t + \eta_\alpha \cdot (p\nabla_a g^t(z'_+) + (1 - p)\nabla_a g^t(z'_-)))$ .  
13: Update  $\lambda_+^{t+1} = \Pi_{\Omega_\lambda}(\lambda_+^t - \eta_l \cdot \nabla_{\lambda_+}[\lambda_+ \epsilon_+ + \phi_{\lambda_+^t, a, b, \alpha}(\theta, z'_+)]$ .  
14: Update  $\lambda_-^{t+1} = \Pi_{\Omega_\lambda}(\lambda_-^t - \eta_l \cdot \nabla_{\lambda_-}[\lambda_- \epsilon_+ + \phi_{\lambda_-^t, a, b, \alpha}(\theta, z'_-])$ .  
15: Update  $\mathbf{w}^{t+1} = \Pi_{\Omega_{\mathbf{w}}}(\mathbf{w}^t - \eta_{\mathbf{w}} \cdot (\widehat{p}\nabla_{\mathbf{w}}g^t(z'_+) + (1 - \widehat{p})\nabla_{\mathbf{w}}g^t(z'_-)))$ .  
16: end for
4.4 Algorithm
4.4.1 DRAUC Optimization
Motivated by the above reformulation, we propose our DRAUC optimization framework, where we solve this optimization problem alternatively.
Inner maximization problem:  $K$ -step Gradient Ascent: Following [39], we consider accessing  $K$ -step gradient ascent with learning rate  $\eta_z$  to solve the inner maximization problem, which is widely used in DRO and can be considered as a variance of PGM. For  $\alpha$ , we use SGA with a step size  $\eta_{\alpha}$ .
Outer minimization problem: Stochastic Gradient Descent: On each iteration, we apply stochastic gradient descent over  $w$  with learning rate  $\eta_w$  and over  $\lambda$  with learning rate  $\eta_\lambda$ .
See Algorithms 1,2 for more details.
4.5 Generalization Bounds
In this section, we theoretically show that the proposed algorithm demonstrates robust generalization in terms of DRAUC-Da metric, even under local worst-case distributions. That is, we show that a model sufficiently trained under our approximate optimization  $(Da\star)$  enjoys a reasonable performance guarantee in DRAUC-Da metric. Our analysis based on the standard assumption that the model parameters  $\theta$  are chosen from the hypothesis set  $\Theta$  (such as neural networks of a specific structure). To derive the subsequent theorem, we utilize the results analyzed in Section 4.3 and perform a Rademacher complexity analysis of DRAUC-Da. The proof for DRAUC-Df follows a similar proof and is much simpler, thus we omit the result here. For additional details, please refer to Appendix A.
Theorem 3 (Informal Version). For all  $\theta \in \Theta, \lambda_{+}, \lambda_{-} \geq 0, (a,b) \in \Omega_{a,b}, \alpha \in \Omega_{\alpha}$  and  $\epsilon_{+}, \epsilon_{-} > 0$ , the following inequality holds with a high probability
$$
\underbrace {D R A U C _ {\epsilon_ {+} , \epsilon_ {-}} ^ {D a} \left(f _ {\boldsymbol {\theta}} , P\right)} _ {(a)} \leq \underbrace {\widehat {\mathcal {L}}} _ {(b)} + \underbrace {\mathcal {O} (\sqrt {1 / \tilde {n}})} _ {(c)} \tag {21}
$$
where  $\tilde{n}$  is some normalized sample size and  $\hat{\mathcal{L}} = \min_{\mathbf{w}}\min_{\lambda_+,\lambda_- \geq 0} \max_{\alpha \in \Omega_\alpha} \{\lambda_+ \epsilon_+ + \lambda_- \epsilon_- + \hat{p}\mathbb{E}_{\hat{P}_+}[\phi_{\mathbf{w},\lambda_+, \alpha}(z)] + (1 - \hat{p})\mathbb{E}_{\hat{P}_-}[\phi_{\mathbf{w},\lambda_-, \alpha}(z)]\}$ .
In Thm.3,  $(a)$  represents the robust AUC loss in terms of expectation,  $(b)$  denotes the training loss that we use to optimize our model parameters, and  $(c)$  is an error term that turns to zero when the
(a) Im ratio 0.01
(b) Im ratio 0.05
Figure 1: Overall Performance of ResNet32 Across Perturbation Levels on CIFAR10. This graph illustrates the performance of various methods at different corruption levels, with Level 0 indicating no corruption and Level 5 representing the most severe corruption. In each figure, the seven lines depict the test AUC for CE, AUCMLoss, FocalLoss, ADVShift, WDRO, DROLT, GLOT, AUCDRO, DRAUC-Da and DRAUC-Df, respectively. Best viewed in colors.
(c) Im ratio 0.1
(d) Im ratio 0.2
sample size turns to infinity. In this sense, if we train our model sufficiently within a large enough training set, we can achieve a minimal generalization error.
In this section, we demonstrate the effectiveness of our proposed framework on three benchmark datasets with varying imbalance ratios.
5.1 Experiment Settings
We evaluate our framework using the following approach. First, we conduct a binary, long-tailed training set. Then, we proceed to train the model on the long-tailed training set with varying imbalance ratios, tune hyperparameters on the validation set, and evaluate the model exhibiting the highest validation AUC on the corrupted testing set. For instance, we train our model on binary long-tailed MNIST [22], CIFAR10, CIFAR100 [18], and Tiny-ImageNet [21], and evaluate our proposed method on the corrupted version of corresponding datasets [30, 13, 14]. Furthermore, we compare our results with multiple competitors including the baseline (CE), typical methods for long-tailed problems [24, 52, 56] and DRO methods [55, 20, 37, 34]. Please see Appendix B for more details.
5.2.1 Overall Performance
The overall performances on CIFAR10 and Tiny-ImageNet are presented in Table 1 and Table 2, respectively. We further compare model performances by altering the level of perturbation, with results displayed in Figure 1. Due to the space limitation, we attach results on MNIST and CIFAR100 in Appendix B. Based on these findings, we make the following observations:
Effectiveness. Our proposed method outperforms all competing approaches across Corrupted MNIST, CIFAR10, CIFAR100 and Tiny-ImageNet datasets for all imbalance ratios, thereby substantiating its effectiveness. Additionally, our approach exhibits enhanced performance as the level of perturbation intensifies, indicating its robustness in challenging testing scenarios.
Ablation results. Given that our method is modified on AUCMLoss [52], the results presented in Figure 1 can be treated as ablation results. Under the same hyperparameters of AUCMLoss, our method exhibits significant improvement over the baseline, indicating enhanced model robustness.
Advantage of Distribution-awareness. As presented in Table 1, DRAUC-Da attains higher scores than DRAUC-Df across almost all corrupted scenarios. This supports our hypothesis that a strong attack on tail-class examples can potentially compromise model robustness.
Performances on non-corrupted data. Within non-corrupted datasets, our approach continues to exhibit competitive performance under conditions of extreme data imbalance, specifically when the imbalance ratio equals to 0.01. However, with less imbalanced training data, our method may suffer performance degradation, attributable to the potential trade-off between model robustness and clean performance, which is an unavoidable phenomenon in Adversarial Training [54].
Table 1: Overall Performance on CIFAR10-C and CIFAR10-LT with different imbalance ratios and different models. The highest score on each column is shown with **bold**, and we use darker color to represent higher performance.
Model	Methods	CIFAR10-C	CIFAR10-LT
0.01	0.05	0.10	0.20	0.01	0.05	0.10	0.20
ResNet20	CE	62.48	75.87	83.13	86.20	65.43	84.12	92.32	95.68
AUCMLoss	63.93	76.77	81.75	85.26	68.88	84.74	90.97	94.40
FocalLoss	56.56	74.44	81.81	84.97	57.63	81.62	91.33	94.62
ADVShift	61.36	75.97	83.78	87.35	64.97	82.91	87.87	95.46
WDRO	63.19	78.90	80.59	86.02	68.80	88.54	91.04	94.04
DROLT	59.92	77.51	81.09	86.46	60.99	85.76	91.35	95.17
GLOT	63.98	77.19	83.33	87.57	65.95	88.37	90.51	94.62
AUCDRO	63.35	76.19	81.82	85.96	67.14	84.00	90.92	94.88
DRAUC-Df	65.58	80.18	85.71	88.83	68.12	86.47	90.57	94.17
DRAUC-Da	66.06	80.13	85.91	89.51	68.71	84.43	90.30	93.76
ResNet32	CE	64.43	78.79	83.12	86.89	66.05	84.40	90.44	95.61
AUCMLoss	64.00	76.98	81.87	85.66	68.90	84.94	91.52	95.16
FocalLoss	56.96	76.53	83.82	87.42	58.04	82.99	91.02	95.16
ADVShift	55.74	72.42	83.47	88.32	56.73	79.36	87.88	94.95
WDRO	64.51	78.45	83.87	88.03	68.16	86.48	90.11	95.23
DROLT	63.66	76.71	83.93	88.42	65.40	84.68	90.11	95.51
GLOT	62.59	77.21	83.67	87.30	64.53	82.62	89.59	94.62
AUCDRO	65.10	71.23	81.45	86.23	68.69	78.51	90.67	95.07
DRAUC-Df	65.44	80.27	85.70	90.62	67.11	85.03	90.63	94.86
DRAUC-Da	65.50	80.57	86.25	90.15	68.51	85.03	90.98	94.27
Table 2: Overall Performance on Tiny-ImageNet-C and Tiny-ImageNet-LT with different imbalance ratios and different models. The highest score on each column is shown with **bold**, and we use darker color to represent higher performance.
Model	Methods	Tiny-ImageNet-C	Tiny-ImageNet-LT
Dogs	Birds	Vehicles	Dogs	Birds	Vehicles
ResNet20	CE	78.46	85.19	87.53	93.72	94.49	97.72
AUCMLoss	77.35	85.98	82.37	93.35	94.11	97.34
FocalLoss	78.34	81.48	86.55	93.25	92.87	97.66
ADVShift	81.20	80.94	86.65	93.70	93.53	97.66
WDRO	82.20	85.23	85.92	94.46	95.50	98.19
DROLT	80.44	86.91	86.76	93.89	96.40	97.86
GLOT	81.96	85.89	86.80	94.67	96.14	98.05
AUCDRO	75.97	83.26	79.46	92.58	93.04	96.29
DRAUC-Df	84.11	87.30	88.67	93.39	95.58	97.50
DRAUC-Da	83.96	87.61	89.06	93.76	95.94	97.25
ResNet32	CE	82.55	84.64	86.26	94.31	94.49	97.76
AUCMLoss	77.25	85.20	81.12	93.19	95.19	97.57
FocalLoss	77.96	79.80	85.33	93.41	92.85	97.78
ADVShift	84.30	84.56	86.43	92.92	94.71	97.59
WDRO	80.08	85.58	86.94	94.39	95.51	97.67
DROLT	79.25	85.75	86.79	91.68	96.06	97.82
GLOT	81.70	83.09	88.24	94.08	95.16	97.92
AUCDRO	78.21	80.55	85.26	91.56	93.15	96.33
DRAUC-Df	85.79	88.00	88.32	94.43	95.29	97.37
DRAUC-Da	84.56	87.60	88.46	94.03	95.96	97.65
(a) Effect of  $\epsilon$  on  $\mathbf{0.01}$
(e) Effect of  $\eta_{\lambda}$  on 0.01
(b) Effect of  $\epsilon$  on 0.05
(f) Effect of  $\eta_{\lambda}$  on 0.05
Figure 2: Sensitivity analysis of  $\epsilon$  and  $\eta_{\lambda}$  on different imbalance ratios.
(c) Effect of  $\epsilon$  on  $\mathbf{0.1}$
(g) Effect of  $\eta_{\lambda}$  on 0.1
(d) Effect of  $\epsilon$  on  $\mathbf{0.01}$
(h) Effect of  $\eta_{\lambda}$  on 0.2
5.2.2 Sensitivity Analysis
The Effect of  $\epsilon$ . In Figure 2-(a)-(d), we present the sensitivity of  $\epsilon$ . The results demonstrate that when the training set is relatively balanced (i.e., the imbalance ratio  $p \geq 0.1$ ), the average robust performance improves as  $\epsilon$  increases. Nonetheless, when the training set is highly imbalanced, the trend is less discernible due to the instability of the training process in these long-tailed settings.
The Effect of  $\eta_{\lambda}$ . In Figure 2-(e)-(h), we present the sensitivity of  $\eta_{\lambda}$ .  $\eta_{\lambda}$  governs the rate of change of  $\lambda$  and serves as a similar function to the warm-up epochs in AT. When  $\eta_{\lambda}$  is small,  $\lambda$  remains large for an extended period, so the adversarial example is regularized to be less offensive. In cases where the training set is extremely imbalanced, a large  $\eta_{\lambda}$  introduces strong examples to the model while it struggles to learn, increasing the instability of the training process and explaining why the smallest  $\eta_{\lambda}$  performs best with an imbalance ratio of 0.01. Conversely, when the model does not face difficulty fitting the training data, an appropriately chosen  $\eta_{\lambda}$  around 0.1 enhances the model's robustness.
6 Conclusion and Future Works
This paper presents an instance-wise, end-to-end framework for DRAUC optimization. Due to the pairwise formulation of AUC optimization, a direct combination with DRO is intractable. To address this issue, we propose a tractable surrogate reformulation on top of the instance-wise formulation of AUC risk. Furthermore, through a theoretical investigation on the neural collapse feature space, we find that the distribution-free perturbation is a scheme that might induce heavy label noise into the dataset. In this sense, we propose a distribution-aware framework to handle class-wise perturbation separately. Theoretically, we show that the robust generalization error is small if both the training error and  $(1 / \sqrt{\tilde{n}})$  is small. Finally, we conduct experiments on three benchmark datasets employing diverse model structures, and the results substantiate the superiority of our approach.
Owing to space constraints, not all potential intersections between AUC optimization and distributionally robustness can be exhaustively explored in this paper. Numerous compelling aspects warrant further investigation. We offer a detailed, instance-wise reformulation of DRAUC, primarily evolving from an AUC optimization standpoint. Future discussions could benefit from initiating dialogue from the angle of DRO. Additionally, integrating various formulations of AUC such as partial AUC and AUPRC with distributional robustness presents a fertile ground for exploration. The existence of a potentially overly-pessimistic phenomenon is yet to be conclusively determined, which paves the way for future inquiries and discoveries.
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, U21B2038, U2001202, 61931008, 62122075, 61976202, and 62206264, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDB28000000) and in part by the Innovation Funding of ICT, CAS under Grant No. E000000.
[1] K. Baker, E. Dall'Anese, and T. Summers. Distribution-agnostic stochastic optimal power flow for distribution grids. In 2016 North American Power Symposium (NAPS), pages 1–6. IEEE, 2016.
[2] A. Ben-Tal, D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013.
[3] J. Blanchet, Y. Kang, and K. Murthy. Robust Wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56(3):830-857, 2019.
[4] J. C. Duchi, T. Hashimoto, and H. Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2:1, 2019.
[5] A. Feizi. Hierarchical detection of abnormal behaviors in video surveillance through modeling normal behaviors based on auc maximization. Soft Computing, 24(14):10401-10413, 2020.
[6] C. Frogner, S. Claici, E. Chien, and J. Solomon. Incorporating unlabeled data into distributionally robust learning. arXiv preprint arXiv:1912.07729, 2019.
[7] R. Gao and A. Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance. Mathematics of Operations Research, 2022.
[8] W. Gao and Z.-H. Zhou. On the consistency of auc pairwise optimization. arXiv preprint arXiv:1208.0645, 2012.
[9] N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pages 297-299. PMLR, 2018.
[10] X. Han, V. Papyan, and D. L. Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.02073, 2021.
[11] J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29-36, 1982.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
[14] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
[15] W. Hou, Q. Xu, Z. Yang, S. Bao, Y. He, and Q. Huang. Adauc: End-to-end adversarial auc optimization against long-tail problems. In International Conference on Machine Learning, pages 8903-8925. PMLR, 2022.
[16] W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pages 2029-2037. PMLR, 2018.
[17] V. Kothapalli, E. Rasromani, and V. Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041, 2022.
[18] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.