Title: DRAUC: An Instance-wise Distributionally Robust AUC Optimization Framework
Venue: NeurIPS 2023
Accepted: yes
Keywords: Robust Learning AUC
Abstract: The Area Under the ROC Curve (AUC) is a widely employed metric in long-tailed classification scenarios. Nevertheless, most existing methods primarily assume that training and testing examples are drawn i.i.d. from the same distribution, which is often unachievable in practice. Distributionally Robust Optimization (DRO) enhances model performance by optimizing it for the local worst-case scenario, but directly integrating AUC optimization with DRO results in an intractable optimization problem. To tackle this challenge, methodically we propose an instance-wise surrogate loss of Distributionally Robust AUC (DRAUC) and build our optimization framework on top of it. Moreover, we highlight that conventional DRAUC may induce label bias, hence introducing distribution-aware DRAUC as a more suitable metric for robust AUC learning. Theoretically, we affirm that the generalization gap between the training loss and testing error diminishes if the training set is sufficiently large. Empirically, experiments on corrupted benchmark datasets demonstrate the effectiveness of our proposed method. Code is available at: https://github.com/EldercatSAM/DRAUC.

DRAUC: An Instance-wise Distributionally Robust AUC Optimization Framework

Abstract

The Area Under the ROC Curve (AUC) is a widely employed metric in long-tailed classification scenarios. Nevertheless, most existing methods primarily assume that training and testing examples are drawn i.i.d. from the same distribution, which is often unachievable in practice. Distributionally Robust Optimization (DRO) enhances model performance by optimizing it for the local worst-case scenario, but directly integrating AUC optimization with DRO results in an intractable optimization problem. To tackle this challenge, methodically we propose an instance-wise surrogate loss of Distributionally Robust AUC (DRAUC) and build our optimization framework on top of it. Moreover, we highlight that conventional DRAUC may induce label bias, hence introducing distribution-aware DRAUC as a more suitable metric for robust AUC learning. Theoretically, we affirm that the generalization gap between the training loss and testing error diminishes if the training set is sufficiently large. Empirically, experiments on corrupted benchmark datasets demonstrate the effectiveness of our proposed method. Code is available at: https://github.com/EldercatSAM/DRAUC .

1 Introduction

The Area Under the ROC Curve (AUC) is an essential metric in machine learning. Owing to its interpretation equivalent to the probability of correctly ranking a random pair of positive and negative examples [11], AUC serves as a more suitable metric than accuracy for imbalanced classification problems. Research on AUC applications has expanded rapidly across various scenarios, including medical image classification [40, 51], abnormal behavior detection [5] and more.

However, current research on AUC optimization assumes that the training and testing sets share the same distribution [46], a challenging condition to satisfy when the testing environment presents a high degree of uncertainty. This situation is common in real-world applications.

Distributionally Robust Optimization (DRO) as a technique designed to handle distributional uncertainty, has emerged as a popular solution [38] in various applications, including machine learning [19], energy systems [1] and transportation [25]. This technique aims to develop a model that performs well, even under the most adversarial distribution within a specified distance from the original training distribution. However, existing DRO methods primarily focus on accuracy as a

metric, making it difficult to directly apply current DRO approaches to AUC optimization due to its pairwise formulation. Consequently, it prompts the following question:

Can we optimize the Distributionally Robust AUC (DRAUC) using an end-to-end framework?

This task presents three progressive challenges: 1) : The pairwise formulation of AUC necessitates simultaneous access to both positive and negative examples, which is computationally intensive and infeasible in online settings. 2) : The naive integration of AUC optimization and DRO leads to an intractable solution. 3) : Based on a specific observation, we find that the ordinary setting of DRAUC might lead to severe label bias in the adversarial dataset.

In this paper, we address the aforementioned challenges through the following techniques: For 1) , we employ the minimax reformulation of AUC and present an early trail to explore DRO under the context of AUC optimization. For 2) , we propose a tractable surrogate loss that is proved to be an upper bound of the original formulation, building our distribution-free DRAUC optimization framework atop it. For 3) , we further devise distribution-aware DRAUC, to perform class-wise distributional perturbation. This decoupled formulation mitigates the label noise issue. This metric can be perceived as a class-wise variant of the distribution-free DRAUC.

It is worth noting that [56] also discusses the combination of DRO techniques with AUC optimization. However, the scope of their discussion greatly differs from this paper. Their approach focuses on using DROto construct estimators for partial AUC and two-way partial AUC optimization with convergence guarantees, whereas this paper primarily aims to enhance the robustness of AUC optimization.

The main contributions of this paper include the following:

Methodologically : We propose an approximate reformulation of DRAUC, constructing an instance-wise, distribution-free optimization framework based on it. Subsequently, we introduce the distribution-aware DRAUC, which serves as a more appropriate metric for long-tailed problems.

Theoretically : We conduct a theoretical analysis of our framework and provide a generalization bound derived from the Rademacher complexity applied to our minimax formulation.

Empirically : We assess the effectiveness of our proposed framework on multiple corrupted long-tailed benchmark datasets. The results demonstrate the superiority of our method.

2 Related Works

2.1 AUC Optimization

AUC is a widely-used performance metric. AUC optimization has garnered significant interest in recent years, and numerous research efforts have been devoted to the field. The researches include different formulations of objective functions, such as pairwise AUC optimization [8], instance-wise AUC optimization [49, 26, 50], AUC in the interested range (partial AUC [48], two-way partial AUC [47]), and area under different metrics (AUPRC [35, 44, 45], AUTKC [43], OpenAUC [42]. For more information, readers may refer to a review on AUC [46].

Some prior work investigates the robustness of AUC. For instance, [52] improves the robustness on noisy data and [15] studies the robustness under adversarial scenarios. In this paper, we further explore robustness under the local worst distribution.

2.2 Distributionally Robust Optimization

DRO aims to enhance the robustness and generalization of models by guaranteeing optimal performance even under the worst-case local distribution. To achieve this objective, an ambiguity set is defined as the worst-case scenario closest to the training set. A model is trained by minimizing the empirical risk on the ambiguity set. To quantify the distance between distributions, prior research primarily considers ϕ -divergence [2, 16, 4, 31] and the Wasserstein distance [39, 28, 19, 3, 7] as distance metrics. For more details, readers may refer to recent reviews on DRO [28, 23].

DRO has applications in various fields, including adversarial training [39], long-tailed learning [37], label shift [55], etc. However, directly optimizing the AUC on the ambiguity set remains an open problem.

3 Preliminaries

In this subsection, we provide a brief review of the AUC optimization techniques and DRO techniques employed in this paper. First, we introduce some essential notations used throughout the paper.

We use z ∈ Z to denote the example-label pair, and f θ : Z → [0 , 1] to represent a model with parameters θ ∈ Θ . This is typical when connecting a Sigmoid function after the model output. For datasets, ̂ P denotes the nominal training distribution with n examples, while P represents the testing distribution. We use ̂ P + = { x + 1 , ..., x + n + } and ̂ P -= { x -1 , ..., x -n -} to denote positive/negative training set, respectively. To describe the degree of imbalance of the dataset, we define ̂ p = n + n + + n -as the imbalance ratio of training set, and p = Pr( y = 1) as the imbalance ratio of testing distribution. The notation E P signifies the expectation on distribution P . We use c ( z, z ′ ) = || z -z ′ || 2 2 to denote the cost of perturbing example z to z ′ .

3.1 AUC Optimization

Statistically, AUC is equivalent to the Wilcoxon-Mann-Whitney test [11], representing the probability of a model predicting a higher score for positive examples than negative ones

where ℓ 0 , 1 ( · ) denotes the 0-1 loss, i.e., ℓ 0 , 1 ( x ) = 1 if x < 0 and otherwise ℓ 0 , 1 ( x ) = 0 . Based on this formulation, maximizing AUC is equivalent to the following minimization problem

where ℓ is a differentiable, consistent surrogate loss of ℓ 0 , 1 . However, the pairwise formulation of the above loss function is not applicable in an online setting. Fortunately, [49] demonstrates that using the square loss as a surrogate loss, the optimization problem (2) can be reformulated as presented in the following theorem.

Theorem 1 ([26]) . When using square loss as the surrogate loss, the AUC maximization is equivalent to

where

Moreover, with the parameter θ fixed, the optimal solution of a, b, α , denoted as a ⋆ , b ⋆ , α ⋆ , can be expressed as:

Similar results hold if the true distribution P + , P -in the expressions are replaced with ̂ P + , ̂ P -. Remark 1 ( The constraints on a, b, α ) . Given that the output of the model f θ is restricted to [0 , 1] , a, b, α can be confined to the following domains:

So that the minimax problem can be reformulated as:

3.2 Distributionally Robust Optimization

Distributionally Robust Optimization (DRO) aims to minimize the learning risk under the local worst-case distribution. Practically, since we can only observe empirical data points, our discussion is primarily focused on empirical distributions. Their extension to population-level is straightforward

where ̂ P is the original empirical distribution, ̂ Q is the perturbed distribution and d is the metric of distributional distance. The constraint d ( ̂ Q, ̂ P ) ≤ ϵ naturally expresses that the perturbation induced ̂ Q should be small enough to be imperceptible.

As demonstrated in [7], when employing the Wasserstein distance W c as the metric, a Lagrangian relaxation can be utilized to reformulate DRO into the subsequent minimax problem.

Theorem 2 ([7]) . With ϕ λ ( z, θ ) = sup z ′ ∈Z { ℓ ( f θ , z ′ ) -λc ( z, z ′ ) } , for all distribution ̂ P and ϵ > 0 , we have

With the theorem above, one can directly get rid of the annoying Wasserstein constraint in the optimization algorithms. We will use this technique to derive an AUC-oriented DRO framework in this paper.

4 Method

4.1 Warm Up: A Naive Formulation for DRAUC

As a technical warm up, we first start with a straightforward approach to optimize AUC metric directly under the worst-case distribution. By simply incorporating the concept of the Wasserstein ambiguity set, we obtain the following definition of DRAUC in a pairwise style.

Definition 1 ( Pairwise Formulation of DRAUC ) . Let ℓ be a consistent loss of ℓ 0 , 1 , for any nominal distribution ̂ P and ϵ > 0 , we have

However, generating local-worst Wasserstein distribution ̂ Q is loss-dependent, implying that we need to know all the training details to deliver a malicious attack. In our endeavor to secure a performance guarantee for our model, we cannot limit the scope of information accessible to an attacker. This pairwise formulation elevates the computational complexity from O ( n ) to O ( n + n -) , significantly increasing the computational burden. By a simple reuse of the trick in (7), one can immediately reach the following reformulation of the minimization of (10).

Proposition 1 ( A Naive Reformulation ) . When using square loss as the surrogate loss, The DRAUC minimization problem: min θ DRAUC ϵ ( f θ , ̂ P ) , is equivalent to

Unfortunately, the optimization operators adhere to a min-max-min-max fashion. There is no known optimization algorithm can deal with this kind of problems so far. Hence, in the rest of this section, we will present two tractable formulations as proper approximations of the problem.

4.2 DRAUC-Df: Distribution-free DRAUC

Let us take a closer look at the minimax problem ( Ori ) . It is straightforward to verify that, fix all the other variables, g is convex with respect to a, b and concave with respect to α within Ω a,b , Ω α . We

Algorithm 1 Algorithm for optimizing DRAUC-Df:

1: Input: the training data Z , step number K , step size for inner K -step gradient ascent η z , learning rates η λ , η w , η α and maximal corrupt distance ϵ .

2: Initialize: initialize a 0 , b 0 , α 0 = 0 , λ 0 = λ 0 .

3: for t = 1 to T do

4: Sample a batch of example z from Z .

5: Generate Local Worst-Case Examples :

6: Initialize z ′ = z .

7: for k = 1 to K do

8: z ′ = Π Z ( z ′ + η z · ∇ z ϕ λ t ,a,b,α ( θ , z ′ )) .

9: end for

10: Update Parameters :

11: Update α t +1 = Π Ω α ( α t + η α · ∇ α g t ( z ′ )) .

12: Update λ t +1 = Π Ω λ ( λ t -η l · ∇ λ [ λϵ + ϕ λ t ,a,b,α ( θ , z ′ )]) .

13: Update w t +1 = Π Ω w ( w t -η w · ∇ w g t ( z ′ )) .

14: end for

are able to interchange the inner min ( a,b ) ∈ Ω a,b and max α ∈ Ω α by invoking von Neumann's Minimax theorem [41], which results in

Moreover, based on the simple property that max x min y f ( x, y ) ≤ min y max x f ( x, y ) , we reach an upper bound of the objective function:

From this perspective, if we minimize ˜ DRAUC ϵ ( f θ , ̂ P ) in turn, we can at least minimize an upper bound of DRAUC ϵ ( f θ , ̂ P ) . In light of this, we will employ the following optimization problem as a surrogate for (Ori) :

where w = θ , ( a, b ) ∈ Ω a,b . Now, by applying the strong duality to the inner maximization problem max

W

̂

Q

:

c

(

̂

Q,

̂

P

)

ϵ

E

Q

̂

where ϕ w ,λ,α ( z ) = max z ′ ∈Z [ g ( w , α, z ) -λc ( z, z ′ )] . This min-max-min formulation remains difficult to optimize, so we take a step similar to (13) that interchange the inner min λ ≥ 0 and outer max α ∈ Ω α , resulting in a tractable upper bound

In this sense, we will use the ( Df⋆ ) as the final optimization problem for DRAUC-Df .

4.3 DRAUC-Da: Distribution-aware DRAUC

Though AUC itself is inherently robust toward long-tailed distributions, we also need to examine whether DRAUC shares this resilience. We now present an analysis within a simplified feature space on the real line, where positive and negative examples are collapsed to their corresponding clusters. The choice of the feature space is simple yet reasonable since it is a 1-d special case of the well-accepted neural collapse phenomenon [32, 10, 17, 57, 27].

Specifically, the following proposition states that the distributional attacker in DRAUC can ruin the AUC performance easily by merely attacking the tail-class examples.

[

g

(

w

, α, z

)]

we have

Proposition 2 ( Powerful and Small-Cost Attack on Neural Collapse Feature Space ) . Let the training set comprises n + positive examples and n -negative examples in R 1 , i.e., D = { x + 1 , ..., x + n + , x -n + +1 , ..., x -n } , with the empirical distribution ̂ P = 1 n ∑ n i =1 δ x i ( δ z represents the Dirac point mass at point z .). According to the neural collapse assume, we have: x + i = x + , x -j = x -. Given a classifier f ( x ) = x , we assume that the maximization of perturb distribution ̂ Q is further constrained on the subset:

where x i → x ′ i forms a discrete Monge map. Then, we have:

where ̂ p = n + n is the ratio of the positive examples in the dataset. Moreover, the cost ̂ p · (1 -̂ p ) · ( x + -x -) 2 is realized by setting:

the barycenter of the two-bodies system ( x + , x -) .

It is noteworthy that ̂ p · (1 -̂ p ) reflects the degree-of-imbalanceness, which is relatively small for long-tailed datasets. Moreover, the barycenter tends to be pretty close to the head-class examples. Therefore, only the tail-class examples are required to be revised heavily during the attack. In this sense, the attacker can always exploit the tail class examples as a backdoor to ruin the AUC performance with small Wasserstein cost. This is similar to the overly-pessimistic phenomena [6, 16] in DRO. The following example shows how small such cost could be in a numerical sense.

Example 1. Consider a simplified setting in which the training set is comprised of only one positive example and 99 negative examples, i.e., ̂ P = { x + 1 , x -2 , ..., x -100 } with x + = 0 . 99 and x -= 0 . 01 . The minimum distance required to perturb the AUC metric from 1 to 0 is 0 . 009702 . This result is achieved by perturbing the positive example from 0 . 99 to 0 . 0198 and the negative examples from 0 . 01 to 0 . 0198 , respectively.

This perturbation strategy indicates a preference towards strong attack on tail-class examples. The resulting distribution ̂ Q is always highly biased toward the original distribution, despite the small Wasserstein cost. In the subsequent training process, one has to minimize the expected loss over ̂ Q , resulting to label noises.

Therefore, it is natural to consider perturbations on the positive and negative distributions separately to avoid such a problem. Accordingly, we propose here a distribution-aware DRAUC formulation:

Definition 2 ( Distribution-aware DRAUC ) . Let ℓ be a consistent loss of ℓ 0 , 1 , for any nominal distribution ̂ P and ϵ + , ϵ -> 0 , we have

For simplicity, let us denote

Similar to DRAUC-Df , we construct our reformulation as follows:

Moreover, we conduct a similar derivation as DRAUC-Df , to construct a tractable upper bound:

where ϕ w ,λ + ,α ( z ) = max z ′ ∈Z [ g ( w , α, z ) -λ + c ( z, z ′ )] and ϕ w ,λ -,α ( z ) = max z ′ ∈Z [ g ( w , α, z ) -λ -c ( z, z ′ )] . Please see Appendix A for the details.

Algorithm 2 Algorithm for optimizing DRAUC-Da:

Input: the training data Z , step number K , step size for inner K -step gradient ascent η z , learning rates η , η , η and maximal corrupt distance ϵ , ϵ .

1: λ w α + -2: Initialize: initialize a 0 , b 0 , α 0 = 0 , λ 0 + = λ 0 -= λ 0 . 3: for t = 1 to T do 4: Sample a batch of example z from Z . 5: Generate Local Worst-Case Examples : 6: Initialize z ′ + = z + , z ′ -= z -. 7: for k = 1 to K do 8: z ′ + = Π Z ( z ′ + + η z · ∇ z ϕ λ t + ,a,b,α ( θ , z ′ + )) . 9: z ′ -= Π Z ( z ′ -+ η z · ∇ z ϕ λ t -,a,b,α ( θ , z ′ -)) . 10: end for 11: Update Parameters : 12: Update α t +1 = Π Ω α ( α t + η α · ( p ∇ a g t ( z ′ + ) + (1 -p ) ∇ a g t ( z ′ -))) . 13: Update λ t +1 + = Π Ω λ ( λ t + -η l · ∇ λ + [ λ + ϵ + + ϕ λ t + ,a,b,α ( θ , z ′ + )]) . 14: Update λ t +1 -= Π Ω λ ( λ t --η l · ∇ λ -[ λ -ϵ -+ ϕ λ t -,a,b,α ( θ , z ′ -)]) . 15: Update w t +1 = Π Ω w ( w t -η w · ( ̂ p ∇ w g t ( z ′ + ) + (1 -̂ p ) ∇ w g t ( z ′ -))) . 16: end for

4.4 Algorithm

4.4.1 DRAUC Optimization

Motivated by the above reformulation, we propose our DRAUC optimization framework, where we solve this optimization problem alternatively.

Inner maximization problem : K -step Gradient Ascent : Following [39], we consider accessing K -step gradient ascent with learning rate η z to solve the inner maximization problem, which is widely used in DRO and can be considered as a variance of PGM. For α , we use SGA with a step size η α .

Outer minimization problem: Stochastic Gradient Descent : On each iteration, we apply stochastic gradient descent over w with learning rate η w and over λ with learning rate η λ .

See Algorithms 1,2 for more details.

4.5 Generalization Bounds

In this section, we theoretically show that the proposed algorithm demonstrates robust generalization in terms of DRAUC-Da metric, even under local worst-case distributions. That is, we show that a model sufficiently trained under our approximate optimization ( Da⋆ ) enjoys a reasonable performance guarantee in DRAUC-Da metric. Our analysis based on the standard assumption that the model parameters θ are chosen from the hypothesis set Θ (such as neural networks of a specific structure). To derive the subsequent theorem, we utilize the results analyzed in Section 4.3 and perform a Rademacher complexity analysis of DRAUC-Da. The proof for DRAUC-Df follows a similar proof and is much simpler, thus we omit the result here. For additional details, please refer to Appendix A.

Theorem 3 ( Informal Version ) . For all θ ∈ Θ , λ + , λ -≥ 0 , ( a, b ) ∈ Ω a,b , α ∈ Ω α and ϵ + , ϵ -> 0 , the following inequality holds with a high probability

where ˜ n is some normalized sample size and ̂ L = min w min λ + ,λ -≥ 0 max α ∈ Ω α { λ + ϵ + + λ -ϵ -+ ̂ p E ̂ P + [ ϕ w ,λ + ,α ( z )] + (1 -̂ p ) E ̂ P -[ ϕ w ,λ -,α ( z )] } .

In Thm.3, ( a ) represents the robust AUC loss in terms of expectation, ( b ) denotes the training loss that we use to optimize our model parameters, and ( c ) is an error term that turns to zero when the

(a) Imratio

0.01

(b) Imratio

0.05

(c) Imratio

0.1

(d) Imratio

0.2

70

85

92

95

method

AUC

Test

67

61

58

64

Q

+

AUC

Test

81

77

73

69

+

Test AUC

76

88

84

80

OD

AUC

Test

91

87

83

79

D

C

Q

CE

ADVShift

WDRO

DROLT

GLOT

AUCMLoSS

AUCDRO

L1

L2

L3

L5

L1

L2

L3

L1

L2

L3

L-4

L5

L1

L2

L3

L5

DRAUC-Df

PerturbationLevel

PerturbationLevel

PerturbationLevel

PerturbationLevel

DRAUC-Da

sample size turns to infinity. In this sense, if we train our model sufficiently within a large enough training set, we can achieve a minimal generalization error.

5 Experiments

In this section, we demonstrate the effectiveness of our proposed framework on three benchmark datasets with varying imbalance ratios.

5.1 Experiment Settings

We evaluate our framework using the following approach. First, we conduct a binary, long-tailed training set. Then, we proceed to train the model on the long-tailed training set with varying imbalance ratios, tune hyperparameters on the validation set, and evaluate the model exhibiting the highest validation AUC on the corrupted testing set. For instance, we train our model on binary long-tailed MNIST [22], CIFAR10, CIFAR100 [18], and Tiny-ImageNet [21], and evaluate our proposed method on the corrupted version of corresponding datasets [30, 13, 14]. Furthermore, we compare our results with multiple competitors including the baseline (CE), typical methods for long-tailed problems [24, 52, 56] and DRO methods [55, 20, 37, 34]. Please see Appendix B for more details.

5.2 Results and Analysis

5.2.1 Overall Performance

The overall performances on CIFAR10 and Tiny-ImageNet are presented in Table 1 and Table 2, respectively. We further compare model performances by altering the level of perturbation, with results displayed in Figure 1. Due to the space limitation, we attach results on MNIST and CIFAR100 in Appendix B. Based on these findings, we make the following observations:

Effectiveness. Our proposed method outperforms all competing approaches across Corrupted MNIST, CIFAR10, CIFAR100 and Tiny-ImageNet datasets for all imbalance ratios, thereby substantiating its effectiveness. Additionally, our approach exhibits enhanced performance as the level of perturbation intensifies, indicating its robustness in challenging testing scenarios.

Ablation results. Given that our method is modified on AUCMLoss [52], the results presented in Figure 1 can be treated as ablation results. Under the same hyperparameters of AUCMLoss, our method exhibits significant improvement over the baseline, indicating enhanced model robustness.

Advantage of Distribution-awareness. As presented in Table 1, DRAUC-Da attains higher scores than DRAUC-Df across almost all corrupted scenarios. This supports our hypothesis that a strong attack on tail-class examples can potentially compromise model robustness.

Performances on non-corrupted data. Within non-corrupted datasets, our approach continues to exhibit competitive performance under conditions of extreme data imbalance, specifically when the imbalance ratio equals to 0 . 01 . However, with less imbalanced training data, our method may suffer performance degradation, attributable to the potential trade-off between model robustness and clean performance, which is an unavoidable phenomenon in Adversarial Training [54].

0.55

0.60

0.65

Test AUC

8

32

64

128

Epsilon

(a) Effect of

ϵ

on

0.01

0.76

0.78

0.80

Test AUC

8

32

64

128

Epsilon

(b) Effect of

ϵ

on

0.05

0.82

0.84

0.86

Test AUC

8

32

64

128

Epsilon

(c) Effect of

ϵ

on

0.1

0.86

0.88

0.90

Test AUC

8

32

64

128

Epsilon

(d) Effect of

ϵ

on

0.01

0.55

0.60

0.65

Test AUC

0.01

0.02

0.1

0.2

Lambda lr

(e) Effect of

η

λ

on

0.01

0.76

0.78

0.80

Test AUC

0.01

0.02

0.1

0.2

Lambda lr

(f) Effect of

η

λ

on

0.05

0.82

0.84

0.86

Test AUC

0.01

0.02

0.1

0.2

Lambda lr

(g) Effect of

η

λ

on

0.1

0.86

0.88

0.90

Test AUC

0.01

0.02

0.1

0.2

Lambda lr

(h) Effect of

η

λ

on

0.2

5.2.2 Sensitivity Analysis

The Effect of ϵ . In Figure 2-(a)-(d), we present the sensitivity of ϵ . The results demonstrate that when the training set is relatively balanced (i.e., the imbalance ratio p ≥ 0 . 1 ), the average robust performance improves as ϵ increases. Nonetheless, when the training set is highly imbalanced, the trend is less discernible due to the instability of the training process in these long-tailed settings.

The Effect of η λ . In Figure 2-(e)-(h), we present the sensitivity of η λ . η λ governs the rate of change of λ and serves as a similar function to the warm-up epochs in AT. When η λ is small, λ remains large for an extended period, so the adversarial example is regularized to be less offensive. In cases where the training set is extremely imbalanced, a large η λ introduces strong examples to the model while it struggles to learn, increasing the instability of the training process and explaining why the smallest η λ performs best with an imbalance ratio of 0 . 01 . Conversely, when the model does not face difficulty fitting the training data, an appropriately chosen η λ around 0.1 enhances the model's robustness.

6 Conclusion and Future Works

This paper presents an instance-wise, end-to-end framework for DRAUC optimization. Due to the pairwise formulation of AUC optimization, a direct combination with DRO is intractable. To address this issue, we propose a tractable surrogate reformulation on top of the instance-wise formulation of AUC risk. Furthermore, through a theoretical investigation on the neural collapse feature space, we find that the distribution-free perturbation is a scheme that might induce heavy label noise into the dataset. In this sense, we propose a distribution-aware framework to handle class-wise perturbation separately. Theoretically, we show that the robust generalization error is small if both the training error and (1 / √ ˜ n ) is small. Finally, we conduct experiments on three benchmark datasets employing diverse model structures, and the results substantiate the superiority of our approach.

Owing to space constraints, not all potential intersections between AUC optimization and distributionally robustness can be exhaustively explored in this paper. Numerous compelling aspects warrant further investigation. We offer a detailed, instance-wise reformulation of DRAUC, primarily evolving from an AUC optimization standpoint. Future discussions could benefit from initiating dialogue from the angle of DRO. Additionally, integrating various formulations of AUC such as partial AUC and AUPRC with distributional robustness presents a fertile ground for exploration. The existence of a potentially overly-pessimistic phenomenon is yet to be conclusively determined, which paves the way for future inquiries and discoveries.

References

K. Baker, E. Dall'Anese, and T. Summers. Distribution-agnostic stochastic optimal power flow for distribution grids. In 2016 North American Power Symposium (NAPS) , pages 1-6. IEEE, 2016.

A. Ben-Tal, D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science , 59(2):341357, 2013.

J. Blanchet, Y. Kang, and K. Murthy. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability , 56(3):830-857, 2019.

J. C. Duchi, T. Hashimoto, and H. Namkoong. Distributionally robust losses against mixture covariate shifts. Under review , 2:1, 2019.

A. Feizi. Hierarchical detection of abnormal behaviors in video surveillance through modeling normal behaviors based on auc maximization. Soft Computing , 24(14):10401-10413, 2020.

C. Frogner, S. Claici, E. Chien, and J. Solomon. Incorporating unlabeled data into distributionally robust learning. arXiv preprint arXiv:1912.07729 , 2019.

R. Gao and A. Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research , 2022.

W. Gao and Z.-H. Zhou. On the consistency of auc pairwise optimization. arXiv preprint arXiv:1208.0645 , 2012.

N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299. PMLR, 2018.

X. Han, V. Papyan, and D. L. Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.02073 , 2021.

J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology , 143(1):29-36, 1982.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.

D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.

W. Hou, Q. Xu, Z. Yang, S. Bao, Y. He, and Q. Huang. Adauc: End-to-end adversarial auc optimization against long-tail problems. In International Conference on Machine Learning , pages 8903-8925. PMLR, 2022.

W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning , pages 2029-2037. PMLR, 2018.

V. Kothapalli, E. Rasromani, and V. Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

D. Kuhn, P. M. Esfahani, V. A. Nguyen, and S. Shafieezadeh-Abadeh. Wasserstein distributionally robust optimization: Theory and applications in machine learning. In Operations research &management science in the age of analytics , pages 130-166. Informs, 2019.

Y. Kwon, W. Kim, J.-H. Won, and M. C. Paik. Principled learning method for wasserstein distributionally robust optimization with local perturbations. In International Conference on Machine Learning , pages 5567-5576. PMLR, 2020.

Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

F. Lin, X. Fang, and Z. Gao. Distributionally robust optimization: A review on theory and applications. Numerical Algebra, Control and Optimization , 12(1):159-212, 2022.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

H. Liu, K. Han, V. V. Gayah, T. L. Friesz, and T. Yao. Data-driven linear decision rule approach for distributionally robust optimization of on-line signal control. Transportation Research Part C: Emerging Technologies , 59:260-277, 2015.

M. Liu, Z. Yuan, Y. Ying, and T. Yang. Stochastic auc maximization with deep neural networks. arXiv preprint arXiv:1908.10831 , 2019.

J. Lu and S. Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.08465 , 2020.

P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming , 171(1-2):115-166, 2018.

M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning . MIT press, 2018.

N. Mu and J. Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv:1906.02337 , 2019.

H. Namkoong and J. C. Duchi. Variance-based regularization with convex objectives. Advances in neural information processing systems , 30, 2017.

V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

H. Phan, T. Le, T. Phung, A. T. Bui, N. Ho, and D. Phung. Global-local regularization via distributional robustness. In International Conference on Artificial Intelligence and Statistics , pages 7644-7664. PMLR, 2023.

Q. Qi, Y. Luo, Z. Xu, S. Ji, and T. Yang. Stochastic optimization of areas under precisionrecall curves with provable convergence. Advances in Neural Information Processing Systems , 34:1752-1765, 2021.

L. Rice, E. Wong, and Z. Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning , pages 8093-8104. PMLR, 2020.

D. Samuel and G. Chechik. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9495-9504, 2021.

A. Shapiro. Tutorial on risk neutral, distributionally robust and risk averse multistage stochastic programming. European Journal of Operational Research , 288(1):1-13, 2021.

A. Sinha, H. Namkoong, R. Volpi, and J. Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571 , 2017.

J. Sulam, R. Ben-Ari, and P. Kisilev. Maximizing auc with deep learning for classification of imbalanced mammogram datasets. In VCBM , pages 131-135, 2017.

J. v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen , 100(1):295-320, 1928.

Z. Wang, Q. Xu, Z. Yang, Y. He, X. Cao, and Q. Huang. Openauc: Towards auc-oriented open-set recognition. Advances in Neural Information Processing Systems , 35:25033-25045, 2022.

Z. Wang, Q. Xu, Z. Yang, Y. He, X. Cao, and Q. Huang. Optimizing partial area under the top-k curve: Theory and practice. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

P. Wen, Q. Xu, Z. Yang, Y. He, and Q. Huang. When false positive is intolerant: End-to-end optimization with low fpr for multipartite ranking. Advances in Neural Information Processing Systems , 34:5025-5037, 2021.

P. Wen, Q. Xu, Z. Yang, Y. He, and Q. Huang. Exploring the algorithm-dependent generalization of auprc optimization with list stability. Advances in Neural Information Processing Systems , 35:28335-28349, 2022.

T. Yang and Y. Ying. Auc maximization in the era of big data and ai: A survey. ACM Computing Surveys , 55(8):1-37, 2022.

Z. Yang, Q. Xu, S. Bao, Y. He, X. Cao, and Q. Huang. When all we need is a piece of the pie: A generic framework for optimizing two-way partial auc. In International Conference on Machine Learning , pages 11820-11829. PMLR, 2021.

Y. Yao, Q. Lin, and T. Yang. Large-scale optimization of partial auc in a range of false positive rates. arXiv preprint arXiv:2203.01505 , 2022.

Y. Ying, L. Wen, and S. Lyu. Stochastic online auc maximization. Advances in neural information processing systems , 29, 2016.

Z. Yuan, Z. Guo, N. Chawla, and T. Yang. Compositional training for end-to-end deep auc maximization. In International Conference on Learning Representations , 2021.

Z. Yuan, Y. Yan, M. Sonka, and T. Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3040-3049, 2021.

Z. Yuan, Y. Yan, M. Sonka, and T. Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3040-3049, 2021.

Z. Yuan, D. Zhu, Z.-H. Qiu, G. Li, X. Wang, and T. Yang. Libauc: A deep learning library for x-risk optimization. In 29th SIGKDD Conference on Knowledge Discovery and Data Mining , 2023.

H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning , pages 7472-7482. PMLR, 2019.

J. Zhang, A. Menon, A. Veit, S. Bhojanapalli, S. Kumar, and S. Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint arXiv:2010.12230 , 2020.

D. Zhu, G. Li, B. Wang, X. Wu, and T. Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. In International Conference on Machine Learning , pages 27548-27573. PMLR, 2022.

Z. Zhu, T. Ding, J. Zhou, X. Li, C. You, J. Sulam, and Q. Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems , 34:29820-29834, 2021.

Peer reviews:
comments: SUMMARY:
This paper considers AUC optimization in the distribution shift setting, which is of great value to the machine learning community. The author formulates the problem as a min-max-min-max problem and relaxes it as an upper bound minimization problem. The experimental results also confirm the effectiveness of this proposal.

STRENGTHS:
The theoretical results seem reasonable. However, as I am not an expert on distributionally robust optimization, I cannot justify the theoretical significance of this paper.

WEAKNESSES:
Table 1 shows the results of CIFAR10-LT, but there is no description of the experimental settings in the context. The proposed method performs poorly on CIFAR10-LT, which seems to contradict the claim of the article. How can this phenomenon be explained?

Miscellaneous:

The colorful curve in Figure 1 may not be friendly for black-and-white print versions. Providing a legend directly may be a better choice.

The definition of $b^*$ in Eq (5) seems to be incorrect.

QUESTIONS:
Is there more discussion on the results of CIFAR10-LT, why is the performance of the proposal similar to AUCMLoss, and sometimes even weaker than AUCMLoss? 

---  
Thanks for the detailed clarifications, which have addressed my concerns. I would like to keep my score.

LIMITATIONS:
The authors do not provide a discussion about the limitations and potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nArea under the ROC curve (AUC) has been widely used for imbalanced classification, but optimizing the metric requires the assumption that training and test data come from the same distribution, which is not realistic. The authors tackle to explore distributionally robust optimization under the context of AUC optimization. They first integrate DRO and AUC and reformulate it to a tractable optimization problem using surrogate loss functions. Finally, they proposed distribution-free DRAUC and distribution-aware DRAUC optimization frameworks. The proposed frameworks are validated theoretically and empirically via experiments on multiple datasets.\n\nSTRENGTHS:\n- The authors propose DRAUC-df by combining DRO and AUC optimization in a tractable way and further extend it to DRAUC-da by considering perturbations on the positive and negative distributions separately.\n- Two proposed frameworks are validated on several standard datasets, which achieved meaningful performance improvement comparing to the baseline (AUCML).\n\nWEAKNESSES:\n- Experiments are only conducted on small-scale datasets, like as CIFAR or MNIST with ResNet and CNN architectures. It may need to validate on more realistic large-scale datasets, like as AUCML on large-scale medical image datasets with more recent network architectures (EfficientNet, DenseNet).\n- This paper addresses the robustness under worst local distribution from DRO perspective, but it restricts to binary classification problem (potentially due to the use of AUC metric) and validated only on binary version of datasets, so I have reservations about the suitability of this problem setting.\n\nQUESTIONS:\n- In L97, why the Wasserstein constraint in the optimization algorithms can be removed with Theorem 2?\n- Typos in the definition of \\ell_{0,1} and Eq. (2)\n\nLIMITATIONS:\nLimitations and potential societal impacts are not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The paper proposes an end-to-end framework to optimize the distributional robust AUC.
The authors address several key challenges of this task and robust generalization can be achieved with theoretical guarantee.
Experiments on several corrupted long-tailed benchmarks demonstrate the superiority of the proposed method.

STRENGTHS:
1. The paper addresses several key challenges to integrating AUC optimization with DRO, including computational infeasibility, intractable solution, and label bias. The proposed method is well-motivated and technically sound. And the generalization risk is bounded with theoretical guarantee.

2. Experiments on several corrupted long-tailed benchmarks demonstrate the superiority of the proposed method.

3. The paper is well-written and clearly organized. Sufficient preliminaries on AUC and DRO are provided.

WEAKNESSES:
1. The proposed method is only evaluated on small-scale datasets such as MNIST and CIFAR-10/100. It remains unclear whether the proposed method can be scaled to large datasets.

2. It is well known that DRO suffers from the problem of yielding overly-pessimistic models with low confidence [1]. This weakness might be inherited in the proposed method as Wasserstein DRO is adopted.

[1] Hu, Weihua, et al. "Does distributionally robust supervised learning give robust classifiers?." International Conference on Machine Learning. PMLR, 2018.

QUESTIONS:
Please check the weakness.

LIMITATIONS:
The limitation of the proposed method may lie in scalability and the issue of overly-pessimistic models.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Distributionally Robust AUC (DRAUC) optimization method that combines Wasserstein distance based DRO with AUC optimization. The combination is designed to be robust for data perturbations. The authors provide theoretical analysis from perturbation and generalization perspective, and justify the practical performance on the perturbed data version for MNIST and CIFAR10.\n\nSTRENGTHS:\n1) The motivation is clear: combining DRO (perturbation robust) with AUC.\n2) Authors provide both theoretical and practical justifications.\n\nWEAKNESSES:\n1) The idea is straightforward and the contribution is limited. Perturbation robustness idea has been explored for adversarial defense (e.g. Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019; Yin, Dong, Ramchandran Kannan, and Peter Bartlett. \"Rademacher complexity for adversarially robust generalization.\" International conference on machine learning. PMLR, 2019). From my view point, this paper directly extend the perturbation robustness to AUC learning.\n\n2) There are some issues for presentation: i) the $\\phi_\\lambda(z,\\mathbf\\theta)$ in line 94 doesn't make sense because the $z'$ will be simply optimized as the value $z$. The problem exists through the whole main content. ii) The '=' should be removed for E.q. 15 to be consistent with other presentations. iii) The $\\mathcal W$ below line 144 should be $\\mathcal W_c$. iv) The notations for $\\hat Q_+$ and $\\hat Q_-$ are abused for E.q. 18 and 19, which are confusing. v) Algorithm 1 and Algorithm 2 are similar, which can be further simplified or merged.\n\n3) It might be better to include (D. Zhu, G. Li, B. Wang, X. Wu, and T. Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. In International Conference on Machine Learning, pages 27548–27573. PMLR, 2022) as another baseline for experiments because it also combines DRO with AUC, although it is not designed for perturbation robustness.\n\n4) The Figure 3 is not very informative. I can only observe CE loss is worse than AUC-type losses.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}