SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Siran Dai $^{1,2}$  Qianqian Xu $^{3*}$  Zhiyong Yang $^{4}$  Xiaochun Cao $^{5}$  Qingming Huang $^{4,3,6*}$
$^{2}$  School of Cyber Security, University of Chinese Academy of Sciences
<sup>3</sup> Key Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS
$^{4}$  School of Computer Science and Tech., University of Chinese Academy of Sciences
<sup>5</sup> School of Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University
$^{6}$  BDKM, University of Chinese Academy of Sciences
daisiran@ie.ac.cn xuqianqian@ict.ac.cn
yangzhiyong21@ucas.ac.cn caoxiaochun@mail.sysu.edu.cn
qmhuang@ucas.ac.cn
The Area Under the ROC Curve (AUC) is a widely employed metric in long-tailed classification scenarios. Nevertheless, most existing methods primarily assume that training and testing examples are drawn i.i.d. from the same distribution, which is often unachievable in practice. Distributionally Robust Optimization (DRO) enhances model performance by optimizing it for the local worst-case scenario, but directly integrating AUC optimization with DRO results in an intractable optimization problem. To tackle this challenge, methodically we propose an instance-wise surrogate loss of Distributionally Robust AUC (DRAUC) and build our optimization framework on top of it. Moreover, we highlight that conventional DRAUC may induce label bias, hence introducing distribution-aware DRAUC as a more suitable metric for robust AUC learning. Theoretically, we affirm that the generalization gap between the training loss and testing error diminishes if the training set is sufficiently large. Empirically, experiments on corrupted benchmark datasets demonstrate the effectiveness of our proposed method. Code is available at: https://github.com/EldercatSAM/DRAUC.
The Area Under the ROC Curve (AUC) is an essential metric in machine learning. Owing to its interpretation equivalent to the probability of correctly ranking a random pair of positive and negative examples [11], AUC serves as a more suitable metric than accuracy for imbalanced classification problems. Research on AUC applications has expanded rapidly across various scenarios, including medical image classification [40, 51], abnormal behavior detection [5] and more.
However, current research on AUC optimization assumes that the training and testing sets share the same distribution [46], a challenging condition to satisfy when the testing environment presents a high degree of uncertainty. This situation is common in real-world applications.
Distributionally Robust Optimization (DRO) as a technique designed to handle distributional uncertainty, has emerged as a popular solution [38] in various applications, including machine learning [19], energy systems [1] and transportation [25]. This technique aims to develop a model that performs well, even under the most adversarial distribution within a specified distance from the original training distribution. However, existing DRO methods primarily focus on accuracy as a
metric, making it difficult to directly apply current DRO approaches to AUC optimization due to its pairwise formulation. Consequently, it prompts the following question:
Can we optimize the Distributionally Robust AUC (DRAUC) using an end-to-end framework?
This task presents three progressive challenges: 1): The pairwise formulation of AUC necessitates simultaneous access to both positive and negative examples, which is computationally intensive and infeasible in online settings. 2): The naive integration of AUC optimization and DRO leads to an intractable solution. 3): Based on a specific observation, we find that the ordinary setting of DRAUC might lead to severe label bias in the adversarial dataset.
In this paper, we address the aforementioned challenges through the following techniques: For 1), we employ the minimax reformulation of AUC and present an early trail to explore DRO under the context of AUC optimization. For 2), we propose a tractable surrogate loss that is proved to be an upper bound of the original formulation, building our distribution-free DRAUC optimization framework atop it. For 3), we further devise distribution-aware DRAUC, to perform class-wise distributional perturbation. This decoupled formulation mitigates the label noise issue. This metric can be perceived as a class-wise variant of the distribution-free DRAUC.
It is worth noting that [56] also discusses the combination of DRO techniques with AUC optimization. However, the scope of their discussion greatly differs from this paper. Their approach focuses on using DRO to construct estimators for partial AUC and two-way partial AUC optimization with convergence guarantees, whereas this paper primarily aims to enhance the robustness of AUC optimization.
The main contributions of this paper include the following:
- Methodologically: We propose an approximate reformulation of DRAUC, constructing an instance-wise, distribution-free optimization framework based on it. Subsequently, we introduce the distribution-aware DRAUC, which serves as a more appropriate metric for long-tailed problems.
- Theoretically: We conduct a theoretical analysis of our framework and provide a generalization bound derived from the Rademacher complexity applied to our minimax formulation.
- Empirically: We assess the effectiveness of our proposed framework on multiple corrupted long-tailed benchmark datasets. The results demonstrate the superiority of our method.
2.1 AUC Optimization
AUC is a widely-used performance metric. AUC optimization has garnered significant interest in recent years, and numerous research efforts have been devoted to the field. The researches include different formulations of objective functions, such as pairwise AUC optimization [8], instance-wise AUC optimization [49, 26, 50], AUC in the interested range (partial AUC [48], two-way partial AUC [47]), and area under different metrics (AUPRC [35, 44, 45], AUTKC [43], OpenAUC [42]. For more information, readers may refer to a review on AUC [46].
Some prior work investigates the robustness of AUC. For instance, [52] improves the robustness on noisy data and [15] studies the robustness under adversarial scenarios. In this paper, we further explore robustness under the local worst distribution.
2.2 Distributionally Robust Optimization
DRO aims to enhance the robustness and generalization of models by guaranteeing optimal performance even under the worst-case local distribution. To achieve this objective, an ambiguity set is defined as the worst-case scenario closest to the training set. A model is trained by minimizing the empirical risk on the ambiguity set. To quantify the distance between distributions, prior research primarily considers  $\phi -\text{divergence}$  [2, 16, 4, 31] and the Wasserstein distance [39, 28, 19, 3, 7] as distance metrics. For more details, readers may refer to recent reviews on DRO [28, 23].
DRO has applications in various fields, including adversarial training [39], long-tailed learning [37], label shift [55], etc. However, directly optimizing the AUC on the ambiguity set remains an open problem.
In this subsection, we provide a brief review of the AUC optimization techniques and DRO techniques employed in this paper. First, we introduce some essential notations used throughout the paper.
We use  $z \in \mathcal{Z}$  to denote the example-label pair, and  $f_{\theta}: \mathcal{Z} \to [0,1]$  to represent a model with parameters  $\theta \in \Theta$ . This is typical when connecting a Sigmoid function after the model output. For datasets,  $\widehat{P}$  denotes the nominal training distribution with  $n$  examples, while  $P$  represents the testing distribution. We use  $\widehat{P}_{+} = \{x_{1}^{+}, \dots, x_{n + }^{+}\}$  and  $\widehat{P}_{-} = \{x_{1}^{-}, \dots, x_{n - }^{-}\}$  to denote positive/negative training set, respectively. To describe the degree of imbalance of the dataset, we define  $\widehat{p} = \frac{n^{+}}{n^{+} + n^{-}}$  as the imbalance ratio of training set, and  $p = \operatorname*{Pr}(y = 1)$  as the imbalance ratio of testing distribution. The notation  $\mathbb{E}_P$  signifies the expectation on distribution  $P$ . We use  $c(z,z') = ||z - z'||_2^2$  to denote the cost of perturbing example  $z$  to  $z'$ .
3.1 AUC Optimization
Statistically, AUC is equivalent to the Wilcoxon-Mann-Whitney test [11], representing the probability of a model predicting a higher score for positive examples than negative ones
$$
A U C \left(f _ {\boldsymbol {\theta}}\right) = \underset {P _ {+}, P _ {-}} {\mathbb {E}} \left[ \ell_ {0, 1} \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {+}\right) - f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {-}\right)\right) \right] \tag {1}
$$
where  $\ell_{0,1}(\cdot)$  denotes the 0-1 loss, i.e.,  $\ell_{0,1}(x) = 1$  if  $x < 0$  and otherwise  $\ell_{0,1}(x) = 0$ . Based on this formulation, maximizing AUC is equivalent to the following minimization problem
$$
\min  _ {\boldsymbol {\theta}} \mathbb {E} _ {P _ {+}, P _ {-}} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {+}\right) - f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {-}\right)\right) \right] \tag {2}
$$
where  $\ell$  is a differentiable, consistent surrogate loss of  $\ell_{0,1}$ . However, the pairwise formulation of the above loss function is not applicable in an online setting. Fortunately, [49] demonstrates that using the square loss as a surrogate loss, the optimization problem (2) can be reformulated as presented in the following theorem.
Theorem 1 ([26]). When using square loss as the surrogate loss, the AUC maximization is equivalent to
$$
\min  _ {\boldsymbol {\theta}} \underset {P _ {+}, P _ {-}} {\mathbb {E}} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {+}\right) - f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {-}\right)\right) \right] = \min  _ {\boldsymbol {\theta}, a, b} \max  _ {\alpha} \quad \underset {P} {\mathbb {E}} [ g (a, b, \alpha , \boldsymbol {\theta}, \boldsymbol {z}) ] \tag {3}
$$
$$
\begin{array}{l} g (a, b, \alpha , \boldsymbol {\theta}, z) = (1 - p) \cdot \left(f _ {\boldsymbol {\theta}} (\boldsymbol {x}) - a\right) ^ {2} \cdot \mathbb {I} _ {[ y = 1 ]} + p \cdot \left(f _ {\boldsymbol {\theta}} (\boldsymbol {x}) - b\right) ^ {2} \cdot \mathbb {I} _ {[ y = 0 ]} \\ + 2 \cdot (1 + \alpha) \cdot \left(p \cdot f _ {\boldsymbol {\theta}} (\boldsymbol {x}) \cdot \mathbb {I} _ {[ y = 0 ]} - (1 - p) \cdot f _ {\boldsymbol {\theta}} (\boldsymbol {x}) \cdot \mathbb {I} _ {[ y = 1 ]} - p (1 - p) \cdot \alpha^ {2}\right). \tag {4} \\ \end{array}
$$
Moreover, with the parameter  $\theta$  fixed, the optimal solution of  $a, b, \alpha$ , denoted as  $a^{\star}, b^{\star}, \alpha^{\star}$ , can be expressed as:
$$
a ^ {\star} = \underset {P _ {+}} {\mathbb {E}} \left[ f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {+}\right) \right], b ^ {\star} = \underset {P _ {-}} {\mathbb {E}} \left[ f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {-}\right) \right], \alpha^ {\star} = b ^ {\star} - a ^ {\star}. \tag {5}
$$
Similar results hold if the true distribution  $P_{+}, P_{-}$  in the expressions are replaced with  $\hat{P}_{+}, \hat{P}_{-}$ .
Remark 1 (The constraints on  $a, b, \alpha$ ). Given that the output of the model  $f_{\theta}$  is restricted to  $[0, 1]$ ,  $a, b, \alpha$  can be confined to the following domains:
$$
\begin{array}{l} \Omega_ {a, b} = \{a, b \in \mathbb {R} | 0 \leq a, b, \leq 1 \}, \\ \Omega_ {\alpha} = \{\alpha \in \mathbb {R} | - 1 \leq \alpha \leq 1 \}. \tag {6} \\ \end{array}
$$
So that the minimax problem can be reformulated as:
$$
\min  _ {\boldsymbol {\theta}, (a, b) \in \Omega_ {a, b}} \max  _ {\alpha \in \Omega_ {\alpha}} \mathbb {E} _ {P} [ g (a, b, \alpha , \boldsymbol {\theta}, \boldsymbol {z}) ]. \tag {7}
$$
3.2 Distributionally Robust Optimization
Distributionally Robust Optimization (DRO) aims to minimize the learning risk under the local worst-case distribution. Practically, since we can only observe empirical data points, our discussion is primarily focused on empirical distributions. Their extension to population-level is straightforward
$$
\min  _ {\boldsymbol {\theta}} \sup  _ {\widehat {Q}: d (\widehat {Q}, \widehat {P}) \leq \epsilon} \mathbb {E} [ \ell (f _ {\boldsymbol {\theta}}, z) ] \tag {8}
$$
where  $\widehat{P}$  is the original empirical distribution,  $\widehat{Q}$  is the perturbed distribution and  $d$  is the metric of distributional distance. The constraint  $d(\widehat{Q},\widehat{P})\leq \epsilon$  naturally expresses that the perturbation induced  $\widehat{Q}$  should be small enough to be imperceptible.
As demonstrated in [7], when employing the Wasserstein distance  $\mathcal{W}_c$  as the metric, a Lagrangian relaxation can be utilized to reformulate DRO into the subsequent minimax problem.
Theorem 2 ([7]). With  $\phi_{\lambda}(z,\pmb {\theta}) = \sup_{z^{\prime}\in \mathcal{Z}}\{\ell (f_{\pmb{\theta}},z^{\prime}) - \lambda c(z,z^{\prime})\}$ , for all distribution  $\widehat{P}$  and  $\epsilon >0$  we have
$$
\sup  _ {\widehat {Q}: \mathcal {W} _ {c} (\widehat {Q}, \widehat {P}) \leq \epsilon} \mathbb {E} [ \ell (f (z)) ] = \inf  _ {\lambda \geq 0} \left\{\lambda \epsilon + \mathbb {E} [ \phi_ {\lambda} (z, \boldsymbol {\theta}) ] \right\}. \tag {9}
$$
With the theorem above, one can directly get rid of the annoying Wasserstein constraint in the optimization algorithms. We will use this technique to derive an AUC-oriented DRO framework in this paper.
4.1 Warm Up: A Naive Formulation for DRAUC
As a technical warm up, we first start with a straightforward approach to optimize AUC metric directly under the worst-case distribution. By simply incorporating the concept of the Wasserstein ambiguity set, we obtain the following definition of DRAUC in a pairwise style.
Definition 1 (Pairwise Formulation of DRAUC). Let  $\ell$  be a consistent loss of  $\ell_{0,1}$ , for any nominal distribution  $\widehat{P}$  and  $\epsilon > 0$ , we have
$$
D R A U C _ {\epsilon} \left(f _ {\boldsymbol {\theta}}, \widehat {P}\right) = 1 - \max  _ {\widehat {Q}: \mathcal {W} _ {c} \left(\widehat {Q}, \widehat {P}\right) \leq \epsilon} \mathbb {E} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {+}\right) - f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} ^ {-}\right)\right) \right]. \tag {10}
$$
However, generating local-worst Wasserstein distribution  $\widehat{Q}$  is loss-dependent, implying that we need to know all the training details to deliver a malicious attack. In our endeavor to secure a performance guarantee for our model, we cannot limit the scope of information accessible to an attacker. This pairwise formulation elevates the computational complexity from  $O(n)$  to  $O(n^{+}n^{-})$ , significantly increasing the computational burden. By a simple reuse of the trick in (7), one can immediately reach the following reformulation of the minimization of (10).
Proposition 1 (A Naive Reformulation). When using square loss as the surrogate loss, The DRAUC minimization problem:  $\min_{\theta}DRAUC_{\epsilon}(f_{\theta},\hat{P})$  , is equivalent to
$$
\left(\boldsymbol {O r i}\right) \quad \min  _ {\boldsymbol {\theta}} \max  _ {\widehat {Q}: \mathcal {W} _ {c} (\widehat {Q}, \widehat {P}) \leq \epsilon} \min  _ {(a, b) \in \Omega_ {a, b}} \max  _ {\alpha \in \Omega_ {\alpha}} \mathbb {E} _ {\widehat {Q}} \left[ g (a, b, \alpha , \boldsymbol {\theta}, z _ {i}) \right]. \tag {11}
$$
Unfortunately, the optimization operators adhere to a min-max-min-max fashion. There is no known optimization algorithm can deal with this kind of problems so far. Hence, in the rest of this section, we will present two tractable formulations as proper approximations of the problem.
4.2 DRAUC-Df: Distribution-free DRAUC
Let us take a closer look at the minimax problem  $(\mathbf{Ori})$ . It is straightforward to verify that, fix all the other variables,  $g$  is convex with respect to  $a, b$  and concave with respect to  $\alpha$  within  $\Omega_{a,b}, \Omega_{\alpha}$ . We
Algorithm 1 Algorithm for optimizing DRAUC-Df:
1: Input: the training data  $\mathcal{Z}$ , step number  $K$ , step size for inner  $K$ -step gradient ascent  $\eta_z$ , learning rates  $\eta_{\lambda}, \eta_w, \eta_\alpha$  and maximal corrupt distance  $\epsilon$ .  
2: Initialize: initialize  $a^0, b^0, \alpha^0 = 0, \lambda^0 = \lambda_0$ .  
3: for  $t = 1$  to  $T$  do  
4: Sample a batch of example  $z$  from  $\mathcal{Z}$ .  
5: Generate Local Worst-Case Examples:  
6: Initialize  $z' = z$ .  
7: for  $k = 1$  to  $K$  do  
8:  $z' = \Pi_{\mathcal{Z}}(z' + \eta_z \cdot \nabla_z \phi_{\lambda^t, a, b, \alpha}(\theta, z'))$ .  
9: end for  
10: Update Parameters:  
11: Update  $\alpha^{t+1} = \Pi_{\Omega_\alpha}(\alpha^t + \eta_\alpha \cdot \nabla_\alpha g^t(z'))$ .  
12: Update  $\lambda^{t+1} = \Pi_{\Omega_\lambda}(\lambda^t - \eta_l \cdot \nabla_\lambda [\lambda \epsilon + \phi_{\lambda^t, a, b, \alpha}(\theta, z'])$ .  
13: Update  $\mathbf{w}^{t+1} = \Pi_{\Omega_{\mathbf{w}}}(\mathbf{w}^t - \eta_{\mathbf{w}} \cdot \nabla_{\mathbf{w}} g^t(z'))$ .  
14: end for
are able to interchange the inner  $\min_{(a,b)\in \Omega_{a,b}}$  and  $\max_{\alpha \in \Omega_{\alpha}}$  by invoking von Neumann's Minimax theorem [41], which results in
$$
\min  _ {\boldsymbol {\theta}} \max  _ {\widehat {Q}: \mathcal {W} _ {c} (\widehat {Q}, \widehat {P}) \leq \epsilon} \max  _ {\alpha \in \Omega_ {\alpha}} \min  _ {(a, b) \in \Omega_ {a, b}} \mathbb {E} [ g (a, b, \alpha , \boldsymbol {\theta}, z) ]. \tag {12}
$$
Moreover, based on the simple property that  $\max_x\min_yf(x,y)\leq \min_y\max_xf(x,y)$ , we reach an upper bound of the objective function:
$$
\underbrace {\underset {\widehat {Q}: \mathcal {W} _ {c} (\widehat {Q} , \widehat {P}) \leq \epsilon} {\max } \underset {\alpha \in \Omega_ {\alpha}} {\max } \underset {(a , b) \in \Omega_ {a , b}} {\min } \underset {\widehat {Q}} {\mathbb {E}} [ g (a , b , \alpha , \theta , z) ]} _ {D R A U C _ {\epsilon} (f _ {\theta}, \widehat {P})} \leq \underbrace {\underset {(a , b) \in \Omega_ {a , b}} {\min } \underset {\alpha \in \Omega_ {\alpha}} {\max } \underset {\widehat {Q}: \mathcal {W} _ {c} (\widehat {Q} , \widehat {P}) \leq \epsilon} {\max } \underset {\widehat {Q}} {\mathbb {E}} [ g (a , b , \alpha , \theta , z) ]} _ {D R A U C _ {\epsilon} (f _ {\theta}, \widehat {P})} \tag {13}
$$
From this perspective, if we minimize  $\widetilde{DRAUC}_{\epsilon}(f_{\theta}, \widehat{P})$  in turn, we can at least minimize an upper bound of  $DRAUC_{\epsilon}(f_{\theta}, \widehat{P})$ . In light of this, we will employ the following optimization problem as a surrogate for (Ori):
$$
\left(\boldsymbol {D} \boldsymbol {f}\right) \quad \min  _ {\mathbf {w}} \max  _ {\alpha \in \Omega_ {\alpha}} \max  _ {\widehat {Q}: \mathcal {W} _ {c} (\widehat {Q}, \widehat {P}) \leq \epsilon} \mathbb {E} [ g (\mathbf {w}, \alpha , z) ] \tag {14}
$$
where  $\mathbf{w} = \pmb{\theta}, (a, b) \in \Omega_{a,b}$ . Now, by applying the strong duality to the inner maximization problem
$$
\max  _ {\widehat {Q}: \mathcal {W} _ {c} (\widehat {Q}, \widehat {P}) \leq \epsilon} \mathbb {E} \left[ g \left(\mathbf {w}, \alpha , z\right) \right]
$$
$$
\left(\boldsymbol {D} \boldsymbol {f}\right) \min  _ {\mathbf {w}} \max  _ {\alpha \in \Omega_ {\alpha}} \min  _ {\lambda \geq 0} \left\{\lambda \epsilon + \underbrace {\mathbb {E} [ \phi_ {\mathbf {w} , \lambda , \alpha} (z) ]} _ {\hat {P}} \right\} \tag {15}
$$
where  $\phi_{\mathbf{w},\lambda,\alpha}(z) = \max_{z' \in \mathcal{Z}}[g(\mathbf{w},\alpha,z) - \lambda c(z,z')]$ . This min-max-min formulation remains difficult to optimize, so we take a step similar to (13) that interchange the inner  $\min_{\lambda \geq 0}$  and outer  $\max_{\alpha \in \Omega_{\alpha}}$ , resulting in a tractable upper bound
$$
\left(\boldsymbol {D} \boldsymbol {f} \star\right) \min  _ {\mathbf {w}} \min  _ {\lambda \geq 0} \max  _ {\alpha \in \Omega_ {\alpha}} \left\{\lambda \epsilon + \underset {\widehat {P}} {\mathbb {E}} \left[ \phi_ {\mathbf {w}, \lambda , \alpha} (z) \right] \right\}. \tag {16}
$$
In this sense, we will use the  $(Df\star)$  as the final optimization problem for DRAUC-Df.
4.3 DRAUC-Da: Distribution-aware DRAUC
Though AUC itself is inherently robust toward long-tailed distributions, we also need to examine whether DRAUC shares this resilience. We now present an analysis within a simplified feature space on the real line, where positive and negative examples are collapsed to their corresponding clusters. The choice of the feature space is simple yet reasonable since it is a 1-d special case of the well-accepted neural collapse phenomenon [32, 10, 17, 57, 27].
Specifically, the following proposition states that the distributional attacker in DRAUC can ruin the AUC performance easily by merely attacking the tail-class examples.
Proposition 2 (Powerful and Small-Cost Attack on Neural Collapse Feature Space). Let the training set comprises  $n^+$  positive examples and  $n^-$  negative examples in  $\mathbb{R}^1$ , i.e.,  $\mathcal{D} = \left\{x_1^+, \dots, x_{n^+}^+, x_{n^+ + 1}^-, \dots, x_n^-\right\}$ , with the empirical distribution  $\widehat{P} = \frac{1}{n} \sum_{i=1}^{n} \delta_{x_i}$  ( $\delta_z$  represents the Dirac point mass at point  $z$ ). According to the neural collapse assume, we have:  $x_i^+ = x^+$ ,  $x_j^- = x^-$ . Given a classifier  $f(x) = x$ , we assume that the maximization of perturb distribution  $\widehat{Q}$  is further constrained on the subset:
$$
\mathcal {Q} = \left\{\widehat {Q}: \widehat {Q} = \frac {1}{n} \sum_ {i = 1} ^ {n} \delta_ {x _ {i} ^ {\prime}} \right\}
$$
where  $x_{i} \to x_{i}'$  forms a discrete Monge map. Then, we have:
$$
\inf_ {\widehat {Q} \in \mathcal {Q}, A U C (f, \widehat {Q}) = 0} \mathcal {W} _ {c} (\widehat {P}, \widehat {Q}) \leq \widehat {p} \cdot (1 - \widehat {p}) \cdot (x ^ {+} - x ^ {-}) ^ {2}
$$
where  $\widehat{p} = \frac{n^{+}}{n}$  is the ratio of the positive examples in the dataset. Moreover, the cost  $\widehat{p} \cdot (1 - \widehat{p}) \cdot (x^{+} - x^{-})^{2}$  is realized by setting:
$$
x ^ {+ \prime} = x ^ {- \prime} = \widehat {p} \cdot x ^ {+} + (1 - \widehat {p}) \cdot x ^ {-}
$$
the barycenter of the two-bodies system  $(x^{+}, x^{-})$ .
It is noteworthy that  $\widehat{p} \cdot (1 - \widehat{p})$  reflects the degree-of-imbalanceness, which is relatively small for long-tailed datasets. Moreover, the barycenter tends to be pretty close to the head-class examples. Therefore, only the tail-class examples are required to be revised heavily during the attack. In this sense, the attacker can always exploit the tail class examples as a backdoor to ruin the AUC performance with small Wasserstein cost. This is similar to the overly-pessimistic phenomena [6, 16] in DRO. The following example shows how small such cost could be in a numerical sense.
Example 1. Consider a simplified setting in which the training set is comprised of only one positive example and 99 negative examples, i.e.,  $\widehat{P} = \{x_1^+, x_2^-, \ldots, x_{100}^-\}$  with  $x^{+} = 0.99$  and  $x^{-} = 0.01$ . The minimum distance required to perturb the AUC metric from 1 to 0 is 0.009702. This result is achieved by perturbing the positive example from 0.99 to 0.0198 and the negative examples from 0.01 to 0.0198, respectively.
This perturbation strategy indicates a preference towards strong attack on tail-class examples. The resulting distribution  $\widehat{Q}$  is always highly biased toward the original distribution, despite the small Wasserstein cost. In the subsequent training process, one has to minimize the expected loss over  $\widehat{Q}$ , resulting in label noises.
Therefore, it is natural to consider perturbations on the positive and negative distributions separately to avoid such a problem. Accordingly, we propose here a distribution-aware DRAUC formulation:
Definition 2 (Distribution-aware DRAUC). Let  $\ell$  be a consistent loss of  $\ell_{0,1}$ , for any nominal distribution  $\hat{P}$  and  $\epsilon_{+},\epsilon_{-} > 0$ , we have