DRAUC: An Instance-wise Distributionally Robust AUC Optimization Framework
Authors: Siran Dai
Qianqian Xu
Zhiyong Yang
Xiaochun Cao
Qingming Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Robust Learning AUC
TL;DR: This paper constructs a Distributionally Robust AUC optimization framework and investigates its characteristics.

Abstract: The Area Under the ROC Curve (AUC) is a widely employed metric in long-tailed classification scenarios. Nevertheless, most existing methods primarily assume that training and testing examples are drawn i.i.d. from the same distribution, which is often unachievable in practice. Distributionally Robust Optimization (DRO) enhances model performance by optimizing it for the local worst-case scenario, but directly integrating AUC optimization with DRO results in an intractable optimization problem. To tackle this challenge, methodically we propose an instance-wise surrogate loss of Distributionally Robust AUC (DRAUC) and build our optimization framework on top of it. Moreover, we highlight that conventional DRAUC may induce label bias, hence introducing distribution-aware DRAUC as a more suitable metric for robust AUC learning. Theoretically, we affirm that the generalization gap between the training loss and testing error diminishes if the training set is sufficiently large. Empirically, experiments on corrupted benchmark datasets demonstrate the effectiveness of our proposed method. Code is available at: https://github.com/EldercatSAM/DRAUC.

DRAUC: An wisse Distributionally Robust
AUC Optimization Framework
Siran Dai
Qianqian Xu
Zhiyong Yang*
Xiaochun Cao
Qingming Huang 4,3,6+
SKLOIS, Institute of Information Engineering. CAS
School of Cyber Security, University of Chinese Academy of Sciences
Key Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS
School of Computer Science and Tech., University of Chinese Academy of Sciences
School Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University 6 BDKM, University of Chinese Academy of Sciences daisiran@iie an yangzhi ac. en mexiiacchu@mmal... sye..euu.u... edu.cn guhuangducas. an.cn
xuqi eriiannianniier ac. cn
Abstract
The Area Under the ROC Curve (AUC) is widely employed metric in long-tailed classification scenarios. Nevertheless, most existing methods primarily assume that training and testing examples drawn i.i.d. from the same distribution, which is often unachievable in practice. Distributionally Robust Optimization (DRO) enhances model performance by optimizing it for the local worst-case scenario, but directly integrating AUC optimization with DRO results in an intractable optimization problem. To tackle this challenge, methodically we propose an instance- wise surrogate loss of Distributionally Robust AUC (DRAUC) and build our optimization framework on top of it. Moreover, we highlight that conventional DRAUC may induce label bias, hence introducing distribution-awe aware DRAUC as a more suitable metric for robust AUC learning. Theoretically, we affirm that the generalization gap between the training loss and testing error diminishes if the training set is sufficiently large. Empirically, experiments on corrupted benchmark datasets demonstrate the effectiveness of our proposed method. Code is available at: https:tt////tttt /attps://githbb witgub.thhho..o com htt..///eeeeeeettt///a.....
Introduction
The Area Under the ROC Curve (AUC) is essential metric in machine learning. Owing to its interpretation equivalent to the probability of correctly ranking a random pair of positive and negative examples []]], AUC serves as a more suitable metric than accuracy for imbalanced classification problems. Research on AUC applications has expanded rapidly across various scenarios, including medical image classification [40, 51], abnormal behavior detection [5] and more.
However, current research on AUC optimization assumes that the training and testing sets share the same distribution [46], challenging condition to satisfy when the testing environment presents high degree of uncertainty. This situation is common in real-world applications.
Distributionally Robust Optimization (DRO) as a technique designed to handle distributional uncer- tainty, has emerged as a popular solution [38] various applications, including machine learning [19], energy systems [1] and transportation [25]. This technique aims to develop model that performs well, even under the most adversarial distribution within a specified distance from the original training distribution. However, existing DRO methods primarily focus on accuracy
"Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

metric, making it difficult to directly apply current DRO approaches to AUC optimization due to its pairwise formulation. Consequently, it prompts the following question:
Can we optimize the Distributionally Robust AUC (DRAUC) using an end-to-end framework?
This task presents three progressive challenges: 1): The pairwise formulation of AUC necessitates simultaneous access both positive and negative examples, which is computationally intensive and infeasible in online settings. 2): The naive integration of AUC optimization and DRO leads to an intractable solution. 3): Based on specific observation, we find that the ordinary setting of DRAUC might lead to severe label bias in the adversarial dataset.
In this paper, address the aforementioned challenges through the following techniques: For 1), we employ the minimax reformulation of AUC and present an early trail to explore DRO under the context of AUC optimization. For 2), propose tractable surrogate loss that is proved to be an upper bound of the original formulation, building our distribution-free DRAUC optimization framework atop it. For 3), we further devise distribution-aware DRAUC, to perform class-wise distributional perturbation. This decoupled formulation mitigates the label noise issue. This metric can be perceived as a class-wise variant of the distribution- free DRAUC.
is worth noting that [56] also discusses the combination of DRO techniques with AUC optimization. However, the scope of their discussion greatly differs from this paper. Their approach focuses using DRO to construct estimators for partial AUC and two-way partial AUC optimization with convergence guarantees, whereas this paper primarily aims enhance the robustness of AUC optimization.
The main contributions of this paper include the following:
Methodologically:
We propose an approximate reformulation of DRAUC, constructing wist,, distribution-free optimization framework based it. Subsequently, introduce the distributionn-wwrr DRAUC, which serves as a more appropriate metric for long-tailed problems.
Theoretically:
We conduct a theoretical analysis of our framework and provide generaliza- tion bound derived from the Rademacher complexity applied to our minimax formulation.
Empirically:
We assess the effectiveness of our proposed framework on multiple corrupted long-tailed benchmark datasets. The results demonstrate the superiority of our method.
2 Related Works
2.1 AUC Optimization
AUC is widely-used performance metric. AUC optimization has garnered significant interest in recent years, and numerous research efforts have been devoted to the field. The researches include different formulations of objective functions, such as pairwise AUC optimization [8], instance-wist AUC optimization [49, 26, 50], AUC the interested range (partial AUC [48], two-way partial AUC [47]), and area under different metrics (AUPRC [35, 44, 45], AUTKC [43], OpenAUC [42]. For more information, readers may refer to a review on AUC [46].
Some prior work investigates the robustness of AUC. For instance, [52] improves the robustness on noisy data and [15] studies the robustness under adversarial scenarios. this paper, we further explore robustness under the local worst distribution.
2.2 Distributionally Robust Optimization
DRO aims to enhance the robustness and generalization of models by guaranteeing optimal perfor- mance even under the worst-case local distribution. To achieve this objective, an ambiguity set is defined as the worst-case scenario closest to the training set. A model is trained by minimizing the empirical risk on the ambiguity set. To quantify the distance between distributions, prior research primarily considers á divergence [2, 16, 4, 31] and the Wasserstein distance [39, 28, 19, 3, 7] as distance metrics. For more details, readers may refer to recent reviews DRO [28, 23].

---

DRO has applications in various fields, including adversarial training [39], long tailed learning [37], label shift [55], etc. However, directly optimizing the AUC on the ambiguity set remains an open problem.
3 Preliminaries
In this subsection, we provide brief review of the AUC optimization techniques and DRO techniques employed in this paper. First, we introduce some essential notations used throughout the paper.
We E 2 denote the example-label pair, and fo 2 [0, 1] represent model with parameters E @. This is typical when connecting a Sigmoid function after the model output. For datasets, denotes the nominal training distribution with n examples, while P represents the testing distribution. We use P+ = {z ... " } and P- { ..... T } to denote positive/negative training set, respectively. To describe the degree of imbalance of the dataset, we define ý 1 as the imbalance ratio of training set, and Pr(y 1) as the imbalance ratio of testing distribution. The notation Ep signifies the expectation on distribution P. We use c(z,z') = ||| - = =111 to denote the cost of perturbing example to z'.
12"
3.1 AUC Optimization
Statistically, iUC equivalent the Filcoxon-Mann-Whinne test [11], representing the probability of model predicting a higher score for positive examples than negative ones
AUC(Jo) P_P (f,1(fo(xx)) fo( fo (f(x)))) ))
EE
where (-) denotes the 0-1 loss, i.c., (x) = if æ < 0 and otherwise (o,( = 0. Based on this formulation, maximizing AUC is equivalent to the following minimization problem
min E Pt.P_
[((fo(x)) fo
where is differentiable, consistent surrogate loss of fo However, the pairwise formulation of the above loss function applicable in online setting. Fortunately, [49] demonstrates that using the square loss as a surrogate loss, the optimization problem (2) can be reformulated as presented in the following theorem.
Theorem 1 ([26)). When using square loss as the surrogate loss, the AUC maximization is equivalent to
min E 0 P.P
(fo(x - So(x ))} min max 0.0.b
Blo(a, b, a, 0,2)]
(3)
where
bla,b, 22 2) = (1 - p) (fo(x) a)² [= + (fo(z) b)² [[==
+2. (1 a) (p fo(x) [1=0 (1 p) fo(x)) I[v= p(1 p) a²).
Moreover, with the parameter 0 fixed, the optimal solution of a, b, a, denoted as a", a², ,,", can be expressed as:
a' E. [Jo(xx , [fo(==]),oo) ,a* b² aa
(5)
Similar results hold if the true distribution Pt, in the expressions replaced with P,, P_. Remark (The constraints on a, b, a). Given that the oulput of the model Se is restricted to [0, 1). a, can be confined the following domains:
Pa,b = {a. b E R < a, $1,
(6)
Ωr -
-
So that the minimax problem be reformulated as:
min max (a aEft
0,,
(7)
3

---

3.2 Distributionally Robust Optimization
Distributionally Robust Optimization (DRO) aims to minimize the learning risk under the local worst-case distribution. Practically, since we can only observe empirical data points, our discussion primarily focused on empirical distributions. Their extension to population-level is straightforward
min 0
sup Qss
where P the original empirical distribution, Q is the perturbed distribution and d is the metric of distributional distance. The constraint d(Q. P) S naturally expresses that the perturbation induced Q should be small enough to be imperceptible.
As demonstrated [7], when employing the Wasserstein distance We as the metric, a Lagrangian relaxation can be utilized to reformulate DRO into the subsequent minimax problem.
Theorem 2 ([7)) With (2, 0) su (e(fo, =²) Ac(z, f. all distribution P and
have
sup @.w.(Q.P)Se
[[(((()))] = inf (Xe + [2, A20
With the theorem above, one can directly get rid of the annoying Wasserstein constraint in the optimization algorithms. will use this technique derive an AUC-oriented DRO framework in this paper.
4 Method
4.1
Warm Up: A Naive Formulation for DRAUC
As a technical warm up, we first start with a straightforward approach to optimize AUC metric directly under the worst-case distribution. By simply incorporating the concept of the Wasserstein ambiguity set, we obtain the following definition DRAUC in pairwise style.
Definition (Pairwise Formulation of DRAUC). Let be a consistent loss of for any nominal distribution P and O, we have
DRAUC.(fo, P) P1=1- max E (fo(xx) fo(x Q.
(10)
However, generating local-worst Wasserstein distribution Q is loss-dependent, implying that we need to know all the training details to deliver malicious attack. In our endeavor secure a performance guarantee for our model, we cannot limit the scope of information accessible to an attacker. This pairwise formulation elevates the computational complexity from O(n) O(n n=),, significantly increasing the computational burden. By a simple reuse of the trick in (7), one can immediately reach the following reformulation of the minimization of (10).
Proposition (A Naive Reformulation). When using square loss the surrogate loss, The DRAUC minimization problem: ming DRAU C(fe, P), equivalent to
(Ori) min max @:W.
min (j.D), ae
max
E Elg(a,b.a,0,.) Q
(11)
Unfortunately, the optimization operators adhere to a min-max-min--inn fashion. There is no known optimization algorithm deal with this kind of problems so far. Hence, in the rest of this section, we will present two tractable formulations as proper approximations of the problem.
4.2 DRAUC-Df: Distribution-free DRAUC
Let us take a closer look at the minimax problem (Ori). It is straightforward to verify that, fix all the other variables, g is convex with respect to a, and concave with respect to within Pu Qa. We

---

Algorithm Algorithm for optimizing DRAUC-DÉ:
1: Input: the training data 2, step number K, step size for inner Ksttep gradient ascent 7/s, learning rates TA, The Tla and maximal corrupt distance c.
2: Initialize: initialize ,",b%, = 0, = Ag-
3: for 1 to I do
4: Sample batch of example z from Z.
5: Generate Local Worst-Case Examples:
6: Initialize 2'
7:
for
K do z' 1lz(2' + + V v=0xx.ahh(0,222))
8:
9:
end for
10:
Update Parameters:
Update a 1m (at + fla Vag)(='))
12:
Update A Il (A* m Vx(de + ax 2))).
13:
Update w = -Il (w' Tww V (z').
14:
end for
are able to interchange the inner min (a,) and maxaen, by invoking von Neumann's 's Minimax theorem [41], which results in
min Q.w.Q. P)se acR. (a.))ER.
max
max min Elg(a, b, 0,
(12)
Moreover, based on the simple property that max, min, fzz y) miny max, fzz y), we reach an upper bound of the objective function:
max max min Elg(a b, a, 0]] min max max (a.b)eR,, alRo QQ.. bl
@:W.(Q.P)se ael, (a.b)ER...
(FA.C.(fa.P)
DRAUC.(fo.P)
(13)
From this perspective, if we minimize DRAUC, C. P) in turn, we can at least minimize an upper bound of DRAUC. P). In light of this, will employ the following optimization problem as a surrogate for (Ori):
(Df) min max max w aett @.W. P)S
Elg(w,a,
(14)
where w = 0, (a. E Da.b. Now, by applying the strong duality to the inner maximization problem max [lg(w,a,,) @.w.(Q.P).e
we have
(Df) min max min Flo (2)] w agf, X20
(15)
where (2) max a, Ac(2, 2/)]. This min-max- max-min formulation remains difficult to optimize, so we take step similar to (13) that interchange the inner minA20 and outer max.en., resulting tractable upper bound
(Df+) min min max (de 1 X20 aEf,
(16)
In this sense, we will use the (Df*) as the final optimization problem for DRAUC-DE.
4.3 DRAUC-Da: Distribution-aware DRAUC
Though AUC itself is inherently robust toward long-tailed distributions, also need to examine whether DRAUC shares this resilience. We now present an analysis within a simplified feature space the real line, where positive and negative examples are collapsed their corresponding clusters. The choice the feature space is simple yet reasonable since it is a 1-d special case of the well-accepted neural collapse phenomenon [32, 10, 17, 57, 27].
Specifically, the following proposition states that the distributional attacker in DRAUC can ruin the AUC performance easily by merely attacking the tail-class examples.
5

---

Proposition 2 (Powerful and Small-Cost Attack on Neural Collapse Feature Space). Let the training set comprises n² positive examples and n negative examples R', i.e. D { Fut at +11 Fn with the empirical distribution P = ; = dz, (Ōz represents the Dirac point mass point 2.). According the neural collapse assume, we have: xx =xt, 2j x
Given classifier f(z) z, assume that the maximization of perturb distribution Q is further constrained on the subset:
o
82
n2
where 2; + x4 forms a discrete Monge map. Then, we have:
inf G AUCUL
W.(P. ②) S 0- (1 (1- (x x=
where P = a² the ratio of the positive examples in the dataset. Moreover, the cost p (1 P)
(x+ x )² is realized by setting:
x x = xpxx +11 (1-1)) D
the barycenter of the two-bodies system (x+,,=").
is noteworthy that p- (1-p) reflects the degree-of-imbalt of-iaalaanceesss, which relatively small for long-tailed datasets. Moreover, the barycenter tends to be pretty close to the head-class examples. Therefore, only the tail-class examples are required this sense, the attacker can always exploit the tail class examples as a backdoor to ruin the AUC performance with small Wasserstein cost. This is similar to the wverly-pessimine in DRO. The following example shows how small such cost could be in a numerical sense.
be revised heavily during the attack. In
phenomena [6, 16]
Example Consider simplified setting in which the training set comprised of only one positive example and 99 negative examples, i.e. P {Ø I2 ...., Foo) with x = 0.99 and x = 0.01. The minimum distance required to perturb the AUC metric from to 0 is 0.009702. This result is achieved by perturbing the positive example from 0.99 to 0.0198 and the negative examples from 0.01 to 0.0198, respectively.
This perturbation strategy indicates a preference towards strong attack on tail-class examples. The resulting distribution Q is always highly biased toward the original distribution, despite the small Wasserstein cost. In the subsequent training process, one has to minimize the expected loss over Q resulting label noises.
Therefore, is natural to consider perturbations on the positive and negative distributions separately to avoid such a problem. Accordingly, propose here distribution-awan DRAUC formulation: Definition 2 (Distribution-aware DRAUC). Let & be a consistent loss of fa,1, for any nominal distribution P and e+, 0, we have
DRAUCDs (fo, =1
max W...QQQ.PPP... Q Ps)se
E
[e(fo(+))
(00((=t=)((x)))) fo(27)))
(17)
Q+.Q_
For simplicity, us denote
2 - - W. M.
(18)
Similar to DRAUC-DE, we construct our reformulation as follows:
(Da) min max max E [g(a, b, a, 0, 2)].
(19)
w aeft, QE @_
Moreover, we conduct a similar derivation as DRAUC-DE, to construct tractable upper bound:
(Da*) min min max {444 + + + P E. w 2
(ew.)t.a(=))
P) (o))) (20)
P.
where oh,A, ia (z) = max:/ezlg(w. ar, c(2 z²)] and ow,x max (esseeeggw..,,
X_ c(22 2²)]. Please see Appendix A for the details.

---

Algorithm 2 Algorithm for optimizing DRAUC-Da:
1: Input: the training data 2, step number K, step size for inner K -step gradient ascent 7/2, learning rates TA, The Tla and maximal corrupt distance e-, 5: Generate Local Worst-Case Examples:
2: Initialize: initialize %.,b%, a = 0, x = X_ Ao-
3: for 1 to I do
4: Sample batch of example 2 from Z.
6: Initialize 24 24,
2'_ 24,2'_ =--
7: for
K do
8:
= =11 + 1/22 V=Φx a.b.a (0, (0, 2)).
9:
x__ 1z + 7== V=oa (0,
10:
end for
11:
Update Parameters:
12:
Update a+1 In (a' + aa (pV ag (24) (1 Vog
13:
Update A+1 = 1l (1+
VA, [444+ + +xx. (0, x4])).
14:
Update A =I (1_ Tp VALAA c- + ..b.a (0, x- }]).
15:
Update w = -Il (i' (w 1/w (FV wo (x4)
1) V w (z_))).
16:
end for
4.4
Algorithm
4.4.1 DRAUC Optimization
Motivated by the above reformulation, we propose our DRAUC optimization framework, where we solve this optimization problem alternatively.
Inner maximization problem K K-step Gradient Ascent: Following [39], consider accessing K- K-step gradient ascent with learning rate 7) to solve the inner maximization problem, which is widely used in DRO and can be considered
variance of PGM. For a. we use SGA with a step size Ma-
Outer minimization problem: Stochastic Gradient Descent: On each iteration, we apply stochastic gradient descent over w with learning rate nu and over À with learning rate f/a-
See Algorithms 1,2 for more details.
4.5 Generalization Bounds
In this section, we theoretically show that the proposed algorithm demonstrates robust generaliza- tion in terms of DRAUC-Da metric, even under local worst-case distributions. That is, we show that a model sufficiently trained under our approximate optimization (Da*) enjoys reasonable performance guarantee DRAUC-Da metric. Our analysis based on the standard assumption that the model parameters 0 are chosen from the hypothesis set O(such as neural networks of a specific structure). To derive the subsequent theorem, we utilize the results analyzed in Section 4.3 and perform Rademacher complexity analysis of DRAUC-Da. The proof for DRAUC-Df follows similar proof and is much simpler, thus we omit the result here. For additional details, please refer to Appendix A.
Theorem 3 (Informal Version). For all O θ, A 2 0, (a. E Elasa E Qa and C e_ > 0, the following inequality holds with a high probability
DRAUCDs (fo, P) < E +0(l1/A)
(21)
(a)
where ñ is some normalized sample size and 2 = min min 22 max P444 + +hc++
p EEp__ ( (z)] + (1 Ep 1dw, (z)]}.
In Thm.3, (a) represents the robust AUC loss in terms of expectation, (b) denotes the training loss that we use to optimize our model parameters, and (c) is error term that turns to zero when the
7

---

48-
method
ar-
68
Tesk AUD
Test AUG
AUCMLos
Tesh
Tooa
13
Focall oas
ADV
68-
DROLT GLOT AUCORO
Persurbation
Perfursation Level
Perturbation
Perfurbation Level
DRAUC-Da
Imratio 0.01
(h) Imratio 0.05
Imratio 0.1
(d) Imratio 0.2
Figure 1: Overall Performance of ResNet32 Across Perturbation Levels on CIFARIO. This graph illustrates the performance of various methods at different corruption levels, with Level 0 indicating no corruption and Level 5 representing the most severe corruption. In each figure, the seven lines depict the test AUC for CE, AUCMLOS, Loss, FocalLLsss, ADV WDRO, DROLT, GLOT, AUCDRO, DRAUC-Da and DRAUC-DÉ, respectively. Best viewed in colors.
sample size turns to infinity, this sense, if we train our model sufficiently within a large enough training set, can achieve minimal generalization error.
5
Experiments
In this section, we demonstrate the effectiveness of our proposed framework on three benchmark datasets with varying imbalance ratios.
5.1 Experiment Settings
We evaluate our framework using the following approach. First, we conduct a binary, long-tailes training set. Then, we proceed to train the model on the long-tailed training set with varying imbalance ratios, tune hyperparameters on the validation set, and evaluate the model cxhibiting the highest validation AUC on the corrupted testing set. For instance, we train model on binary long-tailed [21], and evaluate our proposed method on the corrupted version of corresponding datasets [30, 13, 14]. Furthermore, we compare results with multiple competitors including the baseline (CE), typical methods for long-tailed problems
MNIST [22], CIFARIO. CIFAR 100 [18], and Tiny-
[24, 52, 56] and DRO methods [55, 20, 37, 34]. Please Appendix B for more details.
5.2 Results and Analysis
5.2.1
Overall Performance
The overall performances on CIFARIO and Tiny-ImageNet are presented Table and Table respectively. We further compare model performances by altering the level of perturbation, with results displayed Figure 1. Due to the space limitation, we attach results on MNIST and CIFAR100 in Appendix B. Based these findings, we make the following observations:
Effectiveness. Our proposed method outperforms all competing approaches across Corrupted MNIST, CIFAR10, CIFAR 100 and Tiny-ImageNe datasets for all imbalance ratios, thereby substantiating its effectiveness. Additionally, our approach exhibits enhanced performance as the level of perturbation intensifies, indicating its robustness in challenging testing scenarios.
Ablation results. Given that our method is modified on AUCMLoss [52], the results presented in Figure can be treated as ablation results. Under the same hyperparameters of AUCMLoss, our method exhibits significant improvement over the bascline, indicating enhanced model robustness.
Advantage of Distributionn-wwarenss. As presented in Table 1, DRAUC-DD Da attains higher scores than DRAUC-DÉ across almost all corrupted scenarios. This supports our hypothesis that strong attack on tail-class examples can potentially compromise model robustness.
Performances on non-corrupted data. Within non-corrupted datasets, our approach continues to exhibit competitive performance under conditions of extreme data imbalance, specifically when the imbalance ratio equals to 0.01. However, with less imbalanced training data, our method may suffer performance degradation, attributable to the potential trade-off between model robustness and clean performance, which is unavoidable phenomenon in Adversarial Training [54].

---

Table 1: Overall Performance on CIFARIOS 10-C and CIFARIO-LT with different imbalance ratios and different models. The highest score each column is shown with bold, and darker color to represent higher performance.
CIFARIO-C
CIFAR IO-LT
Model
Methods
0.01
0.05
0.10
0.20
0.01
0.05
0.10
0.20
CE
62.48
75.87
83.13
86.20
65,43
84.12
92.32
95.68
AUCMLoss
63.93
76.77
81.75
85.26
68.88
84.74
90.97
94.40
FocalL
56.56
74.44
81.81
84.97
57.63
81.62
91.33
94.62
ADVShift
61.36
75.97
83.78
87.35
64.97
82.91
87.87
95.46
WDRO
63.19
78.90
80.59
86.02
68.80
88.54
91.04
94.04
ResNet20
DROLT
59.92
77.51
81.09
86.46
60.99
85.76
91.35
95.17
GLOT
63.98
77.19
83.33
87.57
65.95
88.37
90.51
94.62
AUCDRO
63.35
76.19
81.82
85.96
67.14
84.00
90.92
94.88
DRAUC-DF
65.58
80.18
85.71
88.83
68. 12.12
86.47
90.57
94.17
DRAUC-Da
66.06
80.13
85.91
89.51
68.71
84.43
90.30
93.76
CE
64.43
78.79
83.12
86.89
66.05
84.40
90.44
95.61
AUCML
64.00
76.98
81.87
85.66
68.90
84.94
91.52
95.16
FocalLoss
56.96
76.53
83.82
87.42
58.04
82.99
91.02
95.16
ADVShift
55.74
72.42
83.47
88.32
56.73
79.36
87.88
94.95
WDRO
64.51
78.45
83.87
88.03
68.16
86.48
90.11
95.23
ResNet32
DROLT
63.66
76.71
83.93
88.42
65.40
84.68
90.11
95.51
GLOT
62.59
77.21
83.67
87.30
64.53
82.62
89.59
94.62
AUCDRO
65.10
71.23
81.45
86.23
68.69
78.51
90.67
95.07
DRAUC-DE 65.44
80.27
85.70
90.62
67.11
85.03
90.63
94.86
DRAUC-Da
65.50
80.57
86.25
90.15
68.51
85.03
90.98
94.27
Table 2: Overall Performance on Niny--mageNet-C and Tiny-ImageNet-LTT with different imbalance ratios and different models. The highest score each column is shown with bold, and darker color to represent higher performance.
Iny-ImmaeNNr--C
Tiny- ImagemmageNet-L.T
Model
Methods
Dogs
Birds
Vehicles
Dogs
Birds
Vehicles
CE
78.46
85.19
87.53
93.72
94.49
97.72
AUCMLoss
77.35
85.98
82.37
93.35
94.11
97.34
FocalLoss
78.34
81.48
86.55
93.25
92.87
97.66
ADVShift
81.20
80.94
86.65
93.70
93.53
97.66
WDRO
82.20
85.23
85.92
94.46
95.50
98.19
ResNet20
DROLT
80.44
86.91
86.76
93.89
96.40
97.86
GLOT
81,96
85.89
86.80
94.67
96.14
98.05
AUCDRO
75.97
83.26
79.46
92.58
93.04
96.29
DRAUC-DÉ
84.11
87.30
88.67
93.39
95.58
97.50
DRAUC-Da
83.96
87.61
89.06
93.76
95.94
97.25
CE
82.55
84.64
86.26
94.31
94.49
97.76
AUCMLoSE
77.25
85.20
81.12
93.19
95.19
97.57
FocalLoss
77.96
79.80
85.33
93.41
92.85
97.78
ADVShift
84.30
84.56
86.43
92.92
94.71
97.59
WDRO
80.08
85.58
86.94
94.39
95.51
97.67
ResNet32
DROLT
79.25
85.75
86.77 79
91.68
96.06
97.82
GLOT
81.70
83.09
88.24
94.08
95.16
97.92
AUCDRO
78.21
80.55
85.26
91.56
93.15
96.33
DRAUC-DF
85.79
88.00
88.32
94.43
95.29
97.37
DRAUC-Da
84.56
87.60
88.46
94.03
95.96
97.65

---

Epsiion
Epsilon
Epsilon
Epsilan
0.55
0.80
5.65
0.78 AUC
0.82
D.B5
0,00
0.92
Test AUC
Test AUC
Test AUC
(a) Effect of on 0.01
(b) Effect of e on 0.05
(c) Effect of e on 0.1
(d) Effect of e on 0.01
0.2
0.2
Lambda Ir
Lambda Ir
Lambda Ir
0.1
0.3
Lambda
8.1
0.03
0.01
0.01
0.55
Test AUC
Test AUC
Test
Test AUC
(e) Effect of 7A 0.01
(I) Effect of Mл 0.05
(g) Effect of πa on 0.1
(h) Effect of Vaa 0.2
Figure 2: Sensitivity analysis of e and da on different imbalance ratios.
5.2.2 Sensitivity Analysis
The Effect of e. In Figure 2-(a)-(d), present the sensitivity of e. The results demonstrate that when the training set is relatively balanced (i.e., the imbalance ratio 2 0.1), the average robust performance improves as inereases. Nonetheless, when the training set highly imbalanced, the trend is less discernible due to the instability of the training process in these long-tailed settings.
The Effect of 77x- In Figure 2-(e)-(h), we present the sensitivity of 7/x- 7a governs the rate of change of À and serves as a similar function to the warm-up epochs in AT. When 7Jx is small, remains large for extended period, the adversarial example is regularized to be less offensive. In cases where the training set is extremely imbalanced, a large 7/x introduces strong examples to the model while struggles to learn, increasing the instability of the training process and explaining why the smallest 7/x performs best with an imbalance ratio of 0.01. Conversely, when the model does not face difficulty fitting the training data, an appropriately chosen na around 0.1 enhances the model's robustness.
Conclusion and Future Works
This paper presents an instance-wise, end-to-end framework for DRAUC optimization. Due to the pairwise formulation of AUC optimization, direct combination with DRO is intractable. To address this issue, we propose a tractable surrogate reformulation on top of the instance-wisg formulation of AUC risk. Furthermore, through theoretical investigation on the neural collapse feature space, we find that the distribution-fee perturbation scheme that might induce heavy label noise into the dataset. In this sense, we propose a distribution framework to handle class-wise perturbation separately. Theoretically, we show that robust generalization error is small if both the training error and (1/ (1/vñ)) vn) is small. Finally, we conduct experiments on three benchmark datasets employing diverse model structures, and the results substantiate the superiority of our approach.
Owing space constraints, not all potential intersections between AUC optimization and distribu- tionally robustness can be exhaustively explored in this paper. Numerous compelling aspects warrant further investigation. We offer a detailed, instance- wise reformulation of DRAUC, primarily evolving from an AUC optimization standpoint. Future discussions could benefit from initiating dialogue from the angle of DRO. Additionally, integrating various formulations of AUC such as partial AUC and AUPRC with distributional robustness presents fertile ground for exploration. The existence of potentially peesl-ppessimistic phenomenon is yet to be conclusively determined, which paves the way for future inquiries and discoveries.
10

---

Acknowledgements
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, U21B2033 U2001202,, 61931008, 62122075, 61976202, and 62206264, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDB28000000) and in part by the Innovation Funding of ICT, CAS under Grant No. E000000.
References
[1] K. Baker, E. Dall' Anese, and Summers. Distribution -agiiosii stochastic optimal power flow for distribution grids. In 2016 North American Power Symposium (NAPS), pages 1-6. IEEE, 2016.
[2] A. Ben-Tal, D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341 357, 2013.
[3] J. Blanchet, Y. Kang, and K. Murthy. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56(3):830-857, 2019.
[4] C. Duchi, T. Hashimoto, and H. Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2:1, 2019.
[5] A. Feizi. Hierarchical detection of abnormal behaviors in video surveillance through modeling normal behaviors based on auc maximization. Soft Computing, 24(14):0001111041,, 2020.
[6] C. Frogner, S. Claici, E. Chien, and J. Solomon. Incorporating unlabeled data into distribution- ally robust learning. arXiv preprint arXiv: /tt2::/9/2.7722 2019.
[7] R. Gao and Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 2022.
[8] W. Gao and 7.-H. Zhou. On the consistency of auc pairwise optimization. arXiv preprint arXiv: 1208 0645, 2012.
[9] N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theooy, pages 297-299. PMLR, 2018.
[10] X. Han, V. Papyan, and D. L. Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. arXiv preprint arXiv:2106 02073, 2021.
[11] J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29-36, 1982.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[13] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. arXiv preprint arXiv: /903. /2261, 2019.
[14] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. arXiv preprint arXiv: 9999999226600 2019.
[15] W. Hou, Q Xu, Z. Yang, S. Bao, Y. He, and Q. Huang. Adauc: End-to-end adversarial auc optimization against long-tail problems. In International Conference on Machine Learning, pages 8903-8925. PMLR, 2022.
[16] W. Hu, G. NIu, Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pages 2029-2037. PMLR, 2018.
[17] V. Kothapalli, E. Rasromani, and V. Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206. 04041, 2022.
[18] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

---

[19] D. Kuhn, P. M. Esfahani, V. A. Nguyen, and S. Shafieezadeh-Abadeh. Wasserstein distribution- ally robust optimization: Theory and upplications in machine learning. In Operations research & management science in the age of analyties, pages 130-166. Informs, 2019.
[20] Y. Kwon, W. Kim, J.-H. Won, and M. C. Paik. Principled learning method for wasserstein distributionally robust optimization with local perturbations. In International Conference on Machine Learning, pages 5567-5576. PMLR, 2020.
[21] Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N. 7(7):3, 2015.
[22] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-23)) 1998.
[23] Lin, X. Fang, and Z. Gao. Distributionally robust optimization: A review on theory and applications. Numerical Algebra, Control and Optimization, 12(1):159-213 2022.
[24] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[25] H. Liu, K. Han, V. V. Gayah, T. L. Friesz, and T. Yao. Data-driven linear decision rule approach for distributionally robust optimization of on-line signal control. Transportation Research Part C: Emerging Technologies, 59:260-277. 2015.
[26] M. Liu, Z. Yuan, Y. Ying, and T. Yang. Stochastic auc maximization with deep neural networks. 2019.
arXiv preprint 1908,
[27] J. Lu and S. Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint
arXiv:2012.08465, 2020.
[28] P. Mohajerin Esfahani and D. Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Progrumming, 171(1-2):115-666, 2018
[29] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning. MIT press, 2018.
[30] N. Mu and J. Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv: 1906.090000233/ 2019.
[31] H. Namkoong and J. C. Duchi. Variance- based regularization with convex objectives. Advances in neural information processing systems, 30, 2017.
[32] V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, (17440):24652-246) 2020.
[33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killcen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[34] H. Phan, T. Le, Т. Phung, A. T. Bui, N. Ho, and D. Phung. Global-local regularization via distributional robustness. In International Conference on Artificial Intelligence and Statistics, pages 7644-7664. PMLR, 2023.
[35] Q. Qi, Y. Luo, Z. Xu, S. Ji, and T. Yang. Stochastic optimization of areas under precision- recall curves with provable convergence. Advances in Neural Information Processing Systems, 34:1752-1765, 2021.
[36] L. Rice. E. Wong, and Z. Kolter. Overfitting in adversarially robust deep learning. In Interna- tional Conference on Machine Learning, pages 8093- 8104. PMLR, 2020.
[37] D. Samuel and G. Chechik. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9495-9504, 2021.
[38] A. Shapiro. Tutorial on risk neutral, distributionally robust and risk averse multistage stochastic programming. European Journal of Operational Research, 288(1):1-13, 2021.
[39] A. Sinha, H. Namkoong, R. Volpi, and J. Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10577 2017.
[40] Sulam, R. Ben-Ari, and P. Kisilev. Maximizing auc with deep learning for classification of imbalanced mammogram datasets. In VCBM, pages 131-135, 2017.
12

---

[41] J. v. Neumann. Zur theorie der gesellschaftspiele.. Mathematische annalen, 100(1):295-3200 1928.
[42] Z. Wang, Q. Xu, Z. Yang, Y. He, X. Cao, and Q. Huang. Openauc: Towards auc-oriented
open-set recognition. Advances in Neural Information Processing Systems, 35:25033-22045, 2022.
[43] Z. Wang, Q. Xu, Z. Yang, Y. He, X. Cao, and Q. Huang. Optimizing partial area under the top-k curve: Theory and practice. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[44] P. Wen, Q. Xu, Z. Yang. Y. He, and Q. Huang. When false positive is intolerant: End-to-end optimization with low for for multipartite ranking. Advances in Neural Information Processing Systems, 34:5025-5033 50375 2021.
[45] P. Wen, Q. Xu, Z. Yang. Y. He, and Q. Huang. Exploring the of aupre optimization with list stability, Advances in Neural Information Processing Systems, 35:28335-28349, 2022.
generalization
[46] T. Yang and Y. Ying. Auc maximization in the era of big data and ai: A survey. ACM Computing Surveys, 55(8):1-37, 2022.
[47] 7. Yang. Q. Xu, S. Bao, Y. He, X. Cao, and Q Huang. When all we need is piece of the pie: A generic framework for optimizing two-way partial auc. In International Conference on Machine Learning, pages I 1920-118299 PMLR, 2021.
[48] Y. Yao, Q. Lin, and T. Yang. Large-scale optimization of partial auc in a range of false positive rates. arXiv preprint arXiv:2203.01500: 2022.
[49] Y. Ying, L. Wen, and S. Lyu. Stochastic online auc maximization. Advances in neural information processing systems, 29, 2016.
[50] Z. Yuan, Z. Guo, N. Chawla, and T. Yang. Compositional training for end-to-end deep auc maximization. In International Conference Learning Representations, 2021.
[51] Z. Yuan, Y. Yan, M. Sonka, and T. Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3040-3049, 2021.
[52] Z. Yuan, Y. Yan, M. Sonka, and T. Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3040-3049, 2021.
[53] Z. Yuan, D. Zhu, Z.-H. Qiu, G. Li, X. Wang, and T. Yang. Libauc: A deep learning library for x-risk optimization. In 29th SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
[54] H. Zhang, Yu, Jiao, Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference machine learning, pages 7472-7482. PMLR, 2019.
[55] Zhang. A. Menon, A. Veit, S. Bhojanapalli,, S. Kumar, and S. Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint arXiv:2010. 12230, 2020.
[56] D. Zhu, G. Li, B. Wang. X. Wu, and T. Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. In International Conference on Machine Learning, pages 27548-27573. PMLR, 2022.
[57] Z. Zhu, T. Ding, Zhou, X. Li, C. You, J. Sulam, and Q Qu. geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820- 299334 2021.
13


Reviews:
[{"review_id": "HPLatexWAQ", "reviewer": "Reviewer_2tJ9", "summary": "The authors propose a Distributionally Robust AUC (DRAUC) optimization method that combines Wasserstein distance based DRO with AUC optimization. The combination is designed to be robust for data perturbations. The authors provide theoretical analysis from perturbation and generalization perspective, and justify the practical performance on the perturbed data version for MNIST and CIFAR10. ", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The motivation is clear: combining DRO (perturbation robust) with AUC.\n2) Authors provide both theoretical and practical justifications.", "weaknesses": "1) The idea is straightforward and the contribution is limited. Perturbation robustness idea has been explored for adversarial defense (e.g. Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019; Yin, Dong, Ramchandran Kannan, and Peter Bartlett. \"Rademacher complexity for adversarially robust generalization.\" International conference on machine learning. PMLR, 2019). From my view point, this paper directly extend the perturbation robustness to AUC learning.\n\n2) There are some issues for presentation: i) the $\\phi_\\lambda(z,\\mathbf\\theta)$ in line 94 doesn't make sense because the $z'$ will be simply optimized as the value $z$. The problem exists through the whole main content. ii) The '=' should be removed for E.q. 15 to be consistent with other presentations. iii) The $\\mathcal W$ below line 144 should be $\\mathcal W_c$. iv) The notations for $\\hat Q_+$ and $\\hat Q_-$ are abused for E.q. 18 and 19, which are confusing. v) Algorithm 1 and Algorithm 2 are similar, which can be further simplified or merged.\n\n3) It might be better to include (D. Zhu, G. Li, B. Wang, X. Wu, and T. Yang. When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee. In International Conference on Machine Learning, pages 27548–27573. PMLR, 2022) as another baseline for experiments because it also combines DRO with AUC, although it is not designed for perturbation robustness.\n\n4) The Figure 3 is not very informative. I can only observe CE loss is worse than AUC-type losses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable questions! To address your concern, we restate our contributions, fix the typos, and add AUCDRO as a new baseline. The details are as follows:\n\n>**Q1:** The idea is straightforward and the contribution is limited. Perturbation robustness idea has been explored for adversarial defense. From my view point, this paper directly extend the perturbation robustness to AUC learning. \n\n**A1:** Thank you for your inquiry. We want to clarify that the integration of DRO and AUC optimization is not trivial for several reasons:\n\n- **Identifying the strong duality of Distributionally Robust AUC is challenging.** To get rid of the intractable Wasserstein perturbation, one must employ strong duality, as in [a], to reformulate the original optimization problem. Nonetheless, the strong duality becomes much more complicated when assessing performance using AUC, which necessitates a pair-wise loss.\n\n- **The direct integration of DRO with AUC optimization imposes unaffordable computational costs.** Wasserstein DRO utilizes an adversarial training framework to perturb examples. The inner K-step PGD is computationally expensive. The pair-wise formulation requires a time complexity of $O(N^2dK)$ and a spatial complexity of $O(Nd)$, where $N$ is the size of the training set, and $d$ represents the input dimensions. Such complexities make this problem impossible to solve for large datasets. To this end, we conduct an instance-wise reformulation, successfully reducing the time complexity to $O(Ndk)$ and the spatial complexity to $O(Bd)$, where $B$ is the batch size.\n- **The reformulation introduces an intractable optimization problem.** Unfortunately, the original min-max-min-max formulation proves to be intractable. To address this, we propose a surrogate loss which serves as the upper bound of the original objective. This is also specifically designed for our algorithm.\n- **The collision of DRO and long-tailed optimization algorithm results in intriguing outcomes.** Under the assumptions of a neural collapsed feature space, the distributional attacker tends to target the tail classes. In defense of this, we propose distribution-aware DRAUC and evaluate its effectiveness empirically and theoretically.\n- **Typical Rademacher complexity theorem in DRO and adversarial training does not suit our setting.** The vital challenge is that the symmetrization scheme is not available in the pair-wise formulation of AUC. For instance, $\\ell(f_{{\\theta}}(x_1) -f_{{\\theta}}(x_2))$ is interdependent with $\\ell(f_{{\\theta}}(x_1') -f_{{\\theta}}(x_2'))$, if $x_1 = x_1'$ or $x_2 = x_2'$, which makes prior works not suitable in our setting. To address this issue, we propose the Rademacher complexity based on the instance-wise reformulation and successfully solve this problem entirely under the instance-wise setting. Besides, we build control of the excess risk of DRAUC by this Rademacher complexity, which is simpler and easier to use than prior analysis on the Rademacher complexity of AUC.\n---\n\n>**Q2:** The $\\phi_\\lambda(z,\\theta)$ in line 94 doesn't make sense because the $z'$ will be simply optimized as the value. The problem exists through the whole main content.\n\n**A2:** Thanks for your careful review! The term $\\ell(f_\\theta,z)$ on the right-hand side should actually be $\\ell(f_\\theta,z')$, making the corrected equation  $\\phi_\\lambda(z, \\theta) = \\sup_{z' \\in \\mathcal Z}\\{\\ell(f_\\theta,z') - \\lambda c(z,z')\\}$. We also correct the typos in Line 127 and Line 170.\n\n---\n\n>**Q3:** The '=' should be removed for E.q. 15 to be consistent with other presentations. The $\\mathcal W$ below line 144 should be $\\mathcal W_c$.\n\n**A3:** We have corrected the typos. Thanks a lot!\n\n---\n\n>**Q4:** The notations for and are abused for E.q. 18 and 19, which are confusing.\n\n**A4:** Thank you for pointing out the problem. We will modify our description to define the domain of our perturbation:\n\n$\\mathcal Q = \\\\{Q|~\\mathcal W_c(\\widehat Q_+,\\widehat P_+)\\le \\epsilon_+, \\mathcal W_c(\\widehat Q_-,\\widehat P_-)\\le \\epsilon_-\\\\}$\n\nand denotes as $\\max_{Q\\in\\mathcal Q}$.\n\n---\n\n>**Q5:** Algorithm 1 and Algorithm 2 are similar, which can be further simplified or merged.\n\n**A5:** Thank you for your valuable suggestion! We have integrated Algorithm 1 and Algorithm 2 by vectorizing the parameters. \n\n---\n\n>**Q6:** It might be better to include AUCDRO as another baseline for experiments because it also combines DRO with AUC.\n\n**A6:** Thank you for your suggestion! We add this method as our baseline (AUCDRO), and the results are listed as follows:\n\n|Datasets|Models|0.01-C|0.05-C|0.10-C|0.20-C|0.01|0.05|0.10|0.20|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR10|resnet20|63.35|76.19|81.82|85.96|67.14|84.00|90.92|94.88|\n|CIFAR10|resnet32|65.10|71.23|81.45|86.23|68.69|78.51|90.67|95.07|\n|CIFAR100|resnet20|55.96|61.65|62.67|65.72|57.14|64.74|66.59|70.66|\n|CIFAR100|resnet32|56.93|61.41|64.08|68.93|58.33|64.02|68.71|73.86|\n|MNIST|small_cnn|94.00|97.80|97.76|98.54|99.12|99.82|99.92|99.95|\n|MNIST|resnet20|89.11|94.40|95.71|96.73|98.65|99.87|99.89|99.95|\n\nThough AUCDRO is a state-of-the-art method in partial AUC optimization, it is not designed to improve model robustness; thus, its results on corrupted data are lower than ours.\n\n\n---\n\n>**Q7:** The Figure 3 is not very informative. I can only observe CE loss is worse than AUC-type losses.\n\n**A7:** Thank you for pointing out this problem! In constructing the long-tailed version datasets, we assign the first half of classes as positive and the rest half as negative. This results in a large intra-class variance, making it challenging for even a well-trained classifier to generate collapsed features. To prevent any misunderstanding, we decide to move this figure to the supplementary materials.\n\n---\n\n[a]Gao, Rui, and Anton Kleywegt. \"Distributionally robust stochastic optimization with Wasserstein distance.\" Mathematics of Operations Research 48.2 (2023): 603-655.\n"}, {"review_id": "5hgtpeg1VL", "reviewer": "Reviewer_9FMm", "summary": "The paper proposes an end-to-end framework to optimize the distributional robust AUC.\nThe authors address several key challenges of this task and robust generalization can be achieved with theoretical guarantee.\nExperiments on several corrupted long-tailed benchmarks demonstrate the superiority of the proposed method.", "questions": "Please check the weakness.", "limitations": "The limitation of the proposed method may lie in scalability and the issue of overly-pessimistic models.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses several key challenges to integrating AUC optimization with DRO, including computational infeasibility, intractable solution, and label bias. The proposed method is well-motivated and technically sound. And the generalization risk is bounded with theoretical guarantee.\n\n2. Experiments on several corrupted long-tailed benchmarks demonstrate the superiority of the proposed method.\n\n3. The paper is well-written and clearly organized. Sufficient preliminaries on AUC and DRO are provided.\n", "weaknesses": "1. The proposed method is only evaluated on small-scale datasets such as MNIST and CIFAR-10/100. It remains unclear whether the proposed method can be scaled to large datasets.\n\n2. It is well known that DRO suffers from the problem of yielding overly-pessimistic models with low confidence [1]. This weakness might be inherited in the proposed method as Wasserstein DRO is adopted.\n\n[1] Hu, Weihua, et al. \"Does distributionally robust supervised learning give robust classifiers?.\" International Conference on Machine Learning. PMLR, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive concerns! We would like to clarify the following questions:\n\n>**Q1:** The proposed method is only evaluated on small-scale datasets such as MNIST and CIFAR-10/100. It remains unclear whether the proposed method can be scaled to large datasets.\n\n**R1:** Thank you for your constructive question! To address your concern, we carried out several experiments on large-scale datasets, including **Melanoma** and **Tiny-ImageNet-200** (due to the limitation of length, kindly refer to the global response for the experiment settings). Additionally, to validate the adaptability of our proposed method across various network architectures, we incorporate the new backbones **EfficientNet** and **DenseNet**. Our results surpass all the competitors on large-scale datasets and network architectures, proving the effectiveness of our proposed method.\nFor clarity in our presentation, we employ **bold** for the highest scores. Results appended with '-C' depict testing performance on corrupted dataset versions. You may also refer to the global response for experiment settings and better represented results(**Table 1 and 2 in the pdf of global response**).\n\n**Empirical results on Melanoma:**\n|Methods|Mela-C(Enet-b0)|Mela-C(Dense-121)|Mela(Enet-b0)|Mela(Dense-121)|\n|:-:|:-:|:-:|:-:|:-:|\n|CE|69.65|72.05|83.27|83.37|\n|AUCMLoss|73.95|75.67|85.98|84.33|\n|FocalLoss|65.64|72.97|78.81|84.64|\n|ADVShift|70.98|71.91|80.34|81.60|\n|WDRO|75.88|74.71|86.29|**85.79**|\n|DROLT|71.50|76.04|82.89|83.30|\n|GLOT|71.52|69.83|83.95|81.59|\n|AUCDRO|74.57|76.09|**86.77**|85.51|\n|DRAUC-Df|77.82|**77.89**|85.06|81.44|\n|DRAUC-Da|**78.54**|76.82|82.98|81.61|\n\n**Empirical results on Tiny-ImageNet-200(splited by hyper-class):**\n|Models|Methods|Dog-C|Bird-C|Vehicle-C|Dog|Bird|Vehicle|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|resnet20|CE|78.46|85.19|87.53|93.72|94.49|97.72|\n|resnet20|AUCMLoss|77.35|85.98|82.37|93.35|94.11|97.34|\n|resnet20|FocalLoss|78.34|81.48|86.55|93.25|92.87|97.66|\n|resnet20|ADVShift|81.20|80.94|86.65|93.70|93.53|97.66|\n|resnet20|WDRO|82.20|85.23|85.92|94.46|95.50|**98.19**|\n|resnet20|DROLT|80.44|86.91|86.76|93.89|**96.40**|97.86|\n|resnet20|GLOT|81.96|85.89|86.80|**94.67**|96.14|98.05|\n|resnet20|DROAUC|75.97|83.26|79.46|92.58|93.04|96.29|\n|resnet20|DRAUC-Df|**84.11**|87.30|88.67|93.39|95.58|97.50|\n|resnet20|DRAUC-Da|83.96|**87.61**|**89.06**|93.76|95.94|97.25|\n|resnet32|CE|82.55|84.64|86.26|94.31|94.49|97.76|\n|resnet32|AUCMLoss|77.25|85.20|81.12|93.19|95.19|97.57|\n|resnet32|FocalLoss|77.96|79.80|85.33|93.41|92.85|97.78|\n|resnet32|ADVShift|84.30|84.56|86.43|92.92|94.71|97.59|\n|resnet32|WDRO|80.08|85.58|86.94|94.39|95.51|97.67|\n|resnet32|DROLT|79.25|85.75|86.79|91.68|**96.06**|97.82|\n|resnet32|GLOT|81.70|83.09|88.24|94.08|95.16|**97.92**|\n|resnet32|DROAUC|78.21|80.55|85.26|91.56|93.15|96.33|\n|resnet32|DRAUC-Df|**85.79**|**88.00**|88.32|**94.43**|95.29|97.37|\n|resnet32|DRAUC-Da|84.56|87.60|**88.46**|94.03|95.96|97.65|\n\n---\n\n>**Q2:** It is well known that DRO suffers from the problem of yielding overly-pessimistic models with low confidence [a]. This weakness might be inherited in the proposed method as Wasserstein DRO is adopted.\n[a] Hu, Weihua, et al. \"Does distributionally robust supervised learning give robust classifiers?.\" International Conference on Machine Learning. PMLR, 2018.\n\n**R2: Thank you for your valuable question! We believe such an overly pessimistic problem will not appear in our setting for the following reasons:**\n\nWhile both methodologies employ DRO techniques to augment model robustness, the distinction between approaches leveraging $f$-divergence and those using Wasserstein DRO is essential. Specifically, the DRO based on $f$-divergence considers generating a local most adversarial reweight to maximize the training loss, represented as:\n\n$ \\min\\_{{\\theta}}\\sup\\_{r\\in\\mathcal U\\_f} \\mathbb E\\_{P}[r(z)\\ell(g\\_{{\\theta}},z)] $\n\n$\\mathcal U\\_f \\equiv \\\\{r(z) |\\mathbb E\\_{P}[f(r(z))]\\le\\epsilon, \\mathbb E\\_{P}[r(z)]=1, r(z)\\ge 0 ,\\forall z \\in \\mathcal Z \\\\}$\n\nIn contrast, the Wasserstein DRO is designed to produce a local adversarial perturbation on the input to maximize the training loss:\n\n$\\min\\_{{\\theta}}\\max\\_{\\widehat Q:\\mathcal W\\_c(\\widehat Q,\\widehat P)\\le \\epsilon}  ~\\mathbb E\\_{\\widehat Q}\\left[\\ell(g\\_{{\\theta}},z)\\right]$\n\nFor a clearer comparison, akin to the approach in [a], we first present the empirical approximated formulation for both $f$-divergence DRO and Wasserstein DRO as:\n\n${(ERM)}\\quad\\min\\_{{\\theta}}\\frac 1 N\\sum\\_{i=1}^N \\ell(g\\_{{\\theta}},z)$\n\n${(f-divergence \\ DRO)}\\quad\\min\\_{{\\theta}}\\sup\\_{r\\in\\widehat{\\mathcal U}\\_f} \\frac 1 N {\\sum\\_{i=1}^N}r\\_i\\cdot\\ell(g\\_{{\\theta}},z)$\n\n$ {(Wasserstein~DRO)}\\quad \\min\\_{{\\theta}}\\sup\\_{z'\\in\\widehat{Q}} \\frac 1 N\\sum\\_{i=1}^N \\ell(g\\_{{\\theta}},z')$\n\nwhere\n\n$\\widehat{\\mathcal U}\\_f = \\left\\\\{ r |\\frac 1 N\\sum\\_{i=1}\\^N f(r_i) \\le \\epsilon \\frac 1 N\\sum\\_{i=1}\\^N r\\_i = 1,  r \\ge 0 \\right\\\\},\\widehat Q  =\\\\{Q:\\mathcal W\\_c(Q,\\widehat P)\\le \\epsilon\\\\}$\n\nBeing overly pessimistic implies that a model trained by $f$-divergence DRO may end up at a stationary point of the original ERM. This gives us an intuition that the adversarial reweight may be useless if structural assumptions are not introduced. As our approach does not engage in example reweighting, we avoid this overly pessimistic issue. Additionally, from an empirical perspective, we compare our results with AUCMLoss, which can be considered the ERM version of our method. The experimental results prove the enhanced robustness of our approach, which contradicts the proposition of over-pessimism.\nNonetheless, whether Wasserstein DRO suffers from a similar degradation problem is still an open question, and we want to discover that in future work.\n\n---\n\n[a]Hu, Weihua, et al. \"Does distributionally robust supervised learning give robust classifiers?.\" International Conference on Machine Learning. PMLR, 2018.\n"}, {"review_id": "Pshak1f04s", "reviewer": "Reviewer_u8fo", "summary": "Area under the ROC curve (AUC) has been widely used for imbalanced classification, but optimizing the metric requires the assumption that training and test data come from the same distribution, which is not realistic. The authors tackle to explore distributionally robust optimization under the context of AUC optimization. They first integrate DRO and AUC and reformulate it to a tractable optimization problem using surrogate loss functions. Finally, they proposed distribution-free DRAUC and distribution-aware DRAUC optimization frameworks. The proposed frameworks are validated theoretically and empirically via experiments on multiple datasets.", "questions": "- In L97, why the Wasserstein constraint in the optimization algorithms can be removed with Theorem 2?\n- Typos in the definition of \\ell_{0,1} and Eq. (2)", "limitations": "Limitations and potential societal impacts are not addressed.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose DRAUC-df by combining DRO and AUC optimization in a tractable way and further extend it to DRAUC-da by considering perturbations on the positive and negative distributions separately.\n- Two proposed frameworks are validated on several standard datasets, which achieved meaningful performance improvement comparing to the baseline (AUCML).", "weaknesses": "- Experiments are only conducted on small-scale datasets, like as CIFAR or MNIST with ResNet and CNN architectures. It may need to validate on more realistic large-scale datasets, like as AUCML on large-scale medical image datasets with more recent network architectures (EfficientNet, DenseNet).\n- This paper addresses the robustness under worst local distribution from DRO perspective, but it restricts to binary classification problem (potentially due to the use of AUC metric) and validated only on binary version of datasets, so I have reservations about the suitability of this problem setting.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive reviews! To address your concern, we conduct several more experiments on larger datasets, and expand our method toward multi-class settings. \n\n>**Q1:** Experiments are only conducted on small-scale datasets. It may need to validate on more realistic large-scale datasets, like as AUCML on large-scale medical image datasets with more recent network architectures (EfficientNet, DenseNet).\n\n**R1:** Thank you for your valuable question. To address your concerns, we conduct additional experiments on larger datasets, including **Tiny-ImageNet-200** and a large-scale medical image dataset **Melanoma**, utilizing **EfficientNet** and **DenseNet**. The outcomes validate our results' applicability to extensive datasets and modern network architectures.\nHere, we use **bold** to mark the highest scores, and the results with '-C' represent the testing performance in corrupted versions of datasets. **Due to the limitation of reply, please refer to Table 1 and 2 in the pdf of the global response for full results!**\n\n**Empirical results on Melanoma：**\n|Methods|Mela-C(Enet-b0)|Mela-C(Dense-121)|Mela(Enet-b0)|Mela(Dense-121)|\n|:-:|:-:|:-:|:-:|:-:|\n|CE|69.65|72.05|83.27|83.37|\n|AUCMLoss|73.95|75.67|85.98|84.33|\n|FocalLoss|65.64|72.97|78.81|84.64|\n|ADVShift|70.98|71.91|80.34|81.60|\n|WDRO|75.88|74.71|86.29|**85.79**|\n|DROLT|71.50|76.04|82.89|83.30|\n|GLOT|71.52|69.83|83.95|81.59|\n|AUCDRO|74.57|76.09|**86.77**|85.51|\n|DRAUC-Df|77.82|**77.89**|85.06|81.44|\n|DRAUC-Da|**78.54**|76.82|82.98|81.61|\n\n---\n\n>**Q2:** This paper addresses the robustness under worst local distribution from DRO perspective, but it restricts to binary classification problem (potentially due to the use of AUC metric) and validated only on binary version of datasets, so I have reservations about the suitability of this problem setting.\n\n**R2: Thanks for this heuristic question! We take a trial on expanding our method toward multi-class setting, and empirically prove that our proposed method also works well in multi-class cases.**\nAUC is often used in binary classification problems. However, we can generalize our method to accommodate multi-class settings by interpreting a multi-class classification problem as a series of binary decision-making processes. First, we define multi-class AUC:\n\n$AUCM(f\\_{{\\theta}}) = \\frac{1}{N\\_C}\\sum\\_{i=1}^{N\\_C}AUC\\_i(f\\_{{\\theta}}) = \\frac{1}{N\\_C}\\sum\\_{i=1}^{N\\_C} \\mathbb E\\_{\\widehat P\\_i, {\\widehat P\\_i'}}[\\ell(f\\_{{\\theta}}(x\\_i) -f\\_{{\\theta}}(x\\_i'))]$\n\nwhere $N\\_C$ is the number of classes, $AUC\\_i$ denotes the AUC score when assigning $i^{th}$ class as positive, and the rest classes as negative, $\\widehat P\\_i$ denotes the example distribution of $i^{th}$ class and $\\widehat P\\_i'$ denotes the distribution or the rest classes. Consistent with the notation in our paper, we use $\\widehat Q$ to describe the perturbed distribution. Following the idea of optimizing the multi-class AUC under local worst distribution, we now present the objective of Multi-class DRAUC(MDRAUC)\n\n\n$\\min\\_{{\\theta}}\\max\\_{\\widehat Q:\\mathcal W\\_c(\\widehat Q,\\widehat P)\\le \\epsilon}  ~\\mathbb E\\_{\\widehat Q}\\left[\\frac 1{N\\_C}\\sum\\_{i=1}^{N\\_C}AUC\\_i(f\\_{{\\theta}})\\right]$\n\nOur formulation of MDRAUC-Df is as follows:\n\n${(MDf\\star)}\\quad\\min\\_{{{\\theta}},\\vec a,\\vec b} \\min\\_{\\lambda > 0}\\max\\_{\\vec \\alpha} \\left\\\\{\\lambda\\epsilon + \\mathbb E\\_{\\widehat{ P}}[\\widehat \\varphi\\_{{{\\theta}}, \\vec a, \\vec b, \\vec \\alpha, \\lambda}(z)] \\right\\\\}$\n\nwhere \n\n$\\widehat \\varphi\\_{{{\\theta}}, \\vec a, \\vec b, \\vec \\alpha, \\lambda}(z) = \\max\\_{z' \\in \\mathcal Z}\\left[\\frac 1{N\\_C}\\sum\\_{i=1}^{N\\_C}g\\_i(a\\_i,b\\_i,\\alpha\\_i,{{\\theta}},z) - \\lambda c(z,z')\\right].$\n\nLimited by the length of reply, we place the **detailed derivation, experiment setting, and full results (Table 3 in the pdf file)** in the **global response**. We kindly refer you to get more details from it.\nWe further validate the effectiveness of the above multi-class DRAUC algorithm on CIFAR10-LT, and the experiment results are as follows:\n\n**Empirical results of ResNet20 on Multi-class-CIFAR10-LT**\nMethods|0.01-C|0.05-C|0.10-C|0.20-C|0.01|0.02|0.10|0.20|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CE|89.48|90.95|91.58|92.07|95.95|97.98|**98.58**|98.92|\n|AUCMLoss|86.77|88.66|90.88|91.29|95.31|97.69|98.21|98.56|\n|FocalLoss|89.82|91.27|91.97|91.67|**96.42**|**98.15**|98.56|98.95|\n|ADVShift|70.42|72.48|76.54|74.48|72.15|80.87|82.47|83.96|\n|WDRO|90.01|91.30|91.22|91.68|96.35|98.04|98.57|**98.96**|\n|DROLT|89.01|88.72|91.18|90.42|95.44|97.19|98.20|98.52|\n|GLOT|90.44|91.88|91.63|92.49|96.29|97.85|98.57|98.88|\n|DRAUC|**91.28**|**93.15**|**94.98**|**95.74**|94.83|96.47|97.87|98.12|\n|CDRAUC|90.92|92.81|94.20|94.75|95.12|97.06|98.26|98.63|\n\nOur methods outperform all competitors in the corrupted dataset in a multi-class setting. We acknowledge that some previous works have studied advanced algorithms of multi-class AUC learning. However, current derivations and empirical results are enough to prove that our method is capable in multi-class settings, and we would like to leave further explorations of multi-class DRAUC as future work.\n\n---\n\n>**Q3:** In L97, why the Wasserstein constraint in the optimization algorithms can be removed with Theorem 2?\n\n**R3:** Thanks for your valuable question! The Wasserstein constraint is not removed in Theorem 2; it is reformulated using the strong duality of Wasserstein DRO. Detailed proof can be found in [a].\n\n---\n\n>**Q4:** Typos in the definition of $\\ell\\_{0,1}$ and Eq. (2)\n\n**R4:** Thank you for your detailed review. We will adjust our Eq. (1) to ensure consistency with the definition of $\\ell\\_{0,1}$, as follows:\n\n$AUC(f\\_{{\\theta}}) = {\\mathbb E\\_{P\\_+,P\\_+}\\left[{\\ell\\_{0,1}(f\\_{{\\theta}}({{x}^+}) - f\\_{{\\theta}}({{x}^-}))}\\right]}$\n\n---\n\n[a]Gao, Rui, and Anton Kleywegt. \"Distributionally robust stochastic optimization with Wasserstein distance.\" Mathematics of Operations Research 48.2 (2023): 603-655."}, {"review_id": "aoQHmX9huX", "reviewer": "Reviewer_uBXL", "summary": "This paper considers AUC optimization in the distribution shift setting, which is of great value to the machine learning community. The author formulates the problem as a min-max-min-max problem and relaxes it as an upper bound minimization problem. The experimental results also confirm the effectiveness of this proposal.", "questions": "Is there more discussion on the results of CIFAR10-LT, why is the performance of the proposal similar to AUCMLoss, and sometimes even weaker than AUCMLoss? \n\n---  \nThanks for the detailed clarifications, which have addressed my concerns. I would like to keep my score.", "limitations": "The authors do not provide a discussion about the limitations and potential negative societal impact.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theoretical results seem reasonable. However, as I am not an expert on distributionally robust optimization, I cannot justify the theoretical significance of this paper.", "weaknesses": "Table 1 shows the results of CIFAR10-LT, but there is no description of the experimental settings in the context. The proposed method performs poorly on CIFAR10-LT, which seems to contradict the claim of the article. How can this phenomenon be explained?\n\nMiscellaneous:\n\nThe colorful curve in Figure 1 may not be friendly for black-and-white print versions. Providing a legend directly may be a better choice.\n\nThe definition of $b^*$ in Eq (5) seems to be incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments! Our response is as follows:\n\n>**Q1:** Table 1 shows the results of CIFAR10-LT, but there is no description of the experimental settings in the context. The proposed method performs poorly on CIFAR10-LT, which seems to contradict the claim of the article. How can this phenomenon be explained?\n\n**A1:** Thank you for your valuable question! We appreciate your feedback regarding the ambiguity in our paper and apologize for any confusion. To construct the binary, long-tailed version of the dataset, we followed the methodology proposed in [25]. The first half of the classes are assigned as positive, while the remainder are designated as negative. Subsequently, some examples are excluded to establish a desired imbalance ratio. We plan to relocate this section from the Appendix to the main body of the paper for better comprehension.\n\nResponding to your query, we would like to address it from two perspectives. First, we do not directly optimize performance on the clean CIFAR10-LT. Our primary focus is to enhance performance on the local worst-case distribution surrounding the original, uncorrupted distribution. In our experiments, this corresponds to the **corrupted versions** of datasets. \n\nHere, we directly list the empirical result on corrupted datasets of our proposed methods and the baseline (AUCMLoss), so you can check our progress directly.\n\n|Datasets|Models|Methods|0.01|0.05|0.10|0.20|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR10|resnet20|AUCMLoss|63.93|76.77|81.75|85.26|\n|CIFAR10|resnet20|DRAUC-Df|65.3(+1.37)|79.87(+3.1)|84.33(+2.58)|89.0(+3.74)|\n|CIFAR10|resnet20|DRAUC-Da|65.69(+1.76)|80.05(+3.28)|86.17(+4.42)|89.7(+4.44)|\n|CIFAR10|resnet32|AUCMLoss|64.00|76.98|81.87|85.66|\n|CIFAR10|resnet32|DRAUC-Df|64.66(+0.66)|80.25(+3.27)|85.88(+4.01)|90.63(+4.97)|\n|CIFAR10|resnet32|DRAUC-Da|65.38(+1.38)|80.84(+3.86)|86.28(+4.41)|89.44(+3.78)|\n|CIFAR100|resnet20|AUCMLoss|55.70|61.91|65.73|69.58|\n|CIFAR100|resnet20|DRAUC|57.27(+1.57)|62.43(+0.52)|66.25(+0.52)|71.34(+1.76)|\n|CIFAR100|resnet20|CDRAUC|57.5(+1.8)|62.28(+0.37)|66.11(+0.38)|71.06(+1.48)|\n|CIFAR100|resnet32|AUCMLoss|56.19|61.87|63.64|69.81|\n|CIFAR100|resnet32|DRAUC|57.04(+0.85)|61.9(+0.03)|65.96(+2.32)|71.22(+1.41)|\n|CIFAR100|resnet32|CDRAUC|57.38(+1.19)|62.12(+0.25)|66.13(+2.49)|71.31(+1.5)|\n|MNIST|small_cnn|AUCMLoss|95.52|98.16|98.04|98.60|\n|MNIST|small_cnn|DRAUC|96.06(+0.54)|98.38(+0.22)|98.53(+0.49)|98.84(+0.24)|\n|MNIST|small_cnn|CDRAUC|96.35(+0.83)|98.04(+-0.12)|98.56(+0.52)|98.92(+0.32)|\n|MNIST|resnet20|AUCMLoss|89.09|97.82|96.26|97.74|\n|MNIST|resnet20|DRAUC|94.72(+5.63)|98.21(+0.39)|98.32(+2.06)|98.67(+0.93)|\n|MNIST|resnet20|CDRAUC|94.52(+5.43)|98.01(+0.19)|98.53(+2.27)|98.79(+1.05)|\n\nHere are the experiment results we add in the rebuttal phase:\n|Datasets|Models|Methods|Dog|Bird|Vehicles|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|TINYIMAGENET|resnet20|AUCMLoss|77.35|85.98|82.37|\n|TINYIMAGENET|resnet20|DRAUC-Df|84.11(+6.76)|87.3(+1.32)|88.67(+6.3)|\n|TINYIMAGENET|resnet20|DRAUC-Da|83.96(+6.61)|87.61(+1.63)|89.06(+6.69)|\n|TINYIMAGENET|resnet32|AUCMLoss|77.25|85.20|81.12|\n|TINYIMAGENET|resnet32|DRAUC-Df|85.79(+8.54)|88.0(+2.8)|88.32(+7.2)|\n|TINYIMAGENET|resnet32|DRAUC-Da|84.56(+7.31)|87.6(+2.4)|88.46(+7.34)|\n\n|Datasets|Models|Methods|Results|\n|:-:|:-:|:-:|:-:|\n|MELANOMA|efficientnetb0|AUCMLoss|73.95|\n|MELANOMA|efficientnetb0|DRAUC-Df|77.82(+3.87)|\n|MELANOMA|efficientnetb0|DRAUC-Da|78.54(+4.59)|\n|MELANOMA|densenet121|AUCMLoss|75.67|\n|MELANOMA|densenet121|DRAUC-Df|77.89(+2.22)|\n|MELANOMA|densenet121|DRAUC-Da|76.82(+1.15)|\n\n\n\nSecondly, due to the phenomenon of the **trade-off between clean accuracy and robust accuracy** in adversarial training [a], in many cases, achieving optimal performance on both clean and adversarial examples simultaneously is unattainable. This phenomenon can explain the performance drop in the clean data.\n\n\n---\n\n>**Q2:** The colorful curve in Figure 1 may not be friendly for black-and-white print versions. Providing a legend directly may be a better choice.\n\n**A2:** Thank you for your kind suggestion. We will add a legend.\n\n---\n\n>**Q3:** The definition of $b^*$ in Eq (5) seems to be incorrect.\n\n**A3:** Thanks for the detailed review, the corrected version is $b^* = \\mathbb E_{P^-}[f_{{\\theta}}({x}^-)]$.\n\n---\n\n>**Q4:** Is there more discussion on the results of CIFAR10-LT, why is the performance of the proposal similar to AUCMLoss, and sometimes even weaker than AUCMLoss?\n\n**A4:** Thanks again for your valuable question! As discussed in **A1**, we attribute this phenomenon to the inherent trade-off between clean AUC and robust AUC. AUCMLoss can be perceived as an optimal classifier for clean data. However, it is vulnerable to distributional attacks. In contrast, our DRAUC might not perform as well as AUCMLoss on clean data, but it demonstrates superior performance on robust data. \n\n---\n\n[a]Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" International conference on machine learning. PMLR, 2019."}]

Author rebuttal:
Dear reviewers,
we sincerely appreciate your time and valuable comments. Thanks to your detailed, insightful reviews, we introduce the following changes:

- **Additional Empirical Support:** We add AUCDRO[a] as a new baseline. We conduct experiments on **large-scale datasets** including Melanoma and Tiny-ImageNet-200. We also evaluate our proposed method on **modern network architectures** including EfficientNet and DenseNet. 
- **Method Expansion:** We extend our algorithm to multi-class classification scenarios.
- **Improvement on readability:** We combine the DRAUC-Df/Da algorithm together for a more streamlined presentation, and fix several typos in our paper due to your careful reviews. We appreciate the thorough review.

**Empirical results:**
- **New baseline:**
  - **AUCDRO([a]):** As mentioned by Reviewer 2tJ9, it is better to compare with AUCDRO due to its integration of DRO and AUC optimization (though it is not designed for perturbed defense).
- **New datasets:**
  - **Tiny-ImageNet-200**: This dataset comprises 110,000 images in 200 classes, with a resolution of 64x64x3. We assign the binary long-tailed version by utilizing the hyper-class information. As detailed, we construct three subsets, Tiny-ImageNet-Dogs, Birds and Vehicles. Model robustness is then assessed on Tiny-ImageNet-C([b]).
  - **Melanoma:** Melanoma is a large-scale medical image dataset that contains 32,542 positive examples and 584 negative examples. We subdivided the dataset into training, validation, and testing sets in a 7:1.5:1.5 ratio. Due to our use of K-PGD to generate perturbed examples, running experiments on a large-scale dataset is really computationally expensive. We downsampling all images to a resolution of 224x224, and train our model with a batch size 32.
    To our knowledge, no corrupted version of the Melanoma dataset exists. In assessing the robustness of our proposed algorithm, we created a corrupted version like that presented in [b]. Additionally, we excluded weather-based corruptions such as snow, frost, and fog, which are implausible in the context of medical images. As a result, we incorporated 12 corruptions across five perturbation levels in the testing split of the Melanoma dataset to generate Melanoma-C.
  - **Multi-class CIFAR10-LT:** Multi-class CIFAR10-LT is a long-tailed version sampled from the original CIFAR10. The degree of class imbalance is indicated by the imratio, defined as the ratio of the smallest class's size to the largest. We conduct similar reformulation to AUCMLoss to compute multi-class AUC.

- **New backbones:**
  - **Efficientnet, Densenet121**.

**Methodlogically results:** We expand our DRAUC-Df and DRAUC-Da to multi-class settings, the derivation is as follows:

First, we define multi-class AUC:

$AUCM(f\_{{\theta}}) = \frac{1}{N\_C}\sum\_{i=1}^{N\_C}AUC\_i(f\_{{\theta}}) = \frac{1}{N\_C}\sum\_{i=1}^{N\_C} \mathbb E\_{\widehat P\_i, {\widehat P\_i'}}[\ell(f\_{{\theta}}(x\_i) -f\_{{\theta}}(x\_i'))]$

where $N\_C$ is the number of classes, $AUC\_i$ denotes the AUC score when assigning $i^{th}$ class as positive, and the rest classes as negative, $\widehat P\_i$ denotes the example distribution of $i^{th}$ class and $\widehat P\_i'$ denotes the distribution or the rest classes. Consistent with the notation in our paper, we use $\widehat Q$ to describe the perturbed distribution. Following the idea of optimizing the multi-class AUC under local worst distribution, we now present the objective of Multi-class DRAUC(MDRAUC)

$\min\_{{\theta}}\max\_{\widehat Q:\mathcal W\_c(\widehat Q,\widehat P)\le \epsilon}  ~\mathbb E\_{\widehat Q}\left[\frac 1{N\_C}\sum\_{i=1}^{N\_C}AUC\_i(f\_{{\theta}})\right]$

Upon applying the instance-wise reformulation, we obtain

$\min\_{{\theta}}\max\_{\widehat Q:\mathcal W\_c(\widehat Q,\widehat P)\le \epsilon} \frac 1{N\_C}\sum\_{i=1}^{N\_C}\min\_{{(a\_i,b\_i)\in \Omega\_{a,b}}}\max\_{{\alpha\_i \in \Omega\_{\alpha} }} ~\mathbb E\_{\widehat Q}\left[g\_i(a\_i,b\_i,\alpha\_i,{{\theta}},z)\right]$

where 

$g\_i(a\_i,b\_i,\alpha\_i,\theta,{z})= (1-p\_i)\cdot (f\_\theta({x}) - a\_i) ^ 2 \cdot \mathbb I\_{[y = i]} + p\_i \cdot (f\_\theta( x) - b\_i) ^ 2 \cdot  \mathbb I\_{[y \ne i]} $

$\qquad\qquad\qquad\qquad\qquad+ 2\cdot(1+\alpha\_i)\cdot(p\_i \cdot f\_\theta( x) \cdot \mathbb I\_{[y \ne i]} - (1-p\_i)\cdot f\_\theta( x) \cdot \mathbb I\_{[y = i]} - p\_i(1-p\_i)\cdot\alpha\_i^2).$

Here, $p\_i = \frac{n\_i} N$ denotes the proportion of examples in the $i^{th}$ class. Moreover, with ${\theta}$ and $\widehat Q$ fixed, $a\_i,a\_j$ are irrelevant $\forall i,j$, as are $b$ and $\alpha$. Thus, we can safely interchange the summation and inner min-max, resulting in

$\min\_{{\theta}}\max\_{\widehat Q:\mathcal W\_c(\widehat Q,\widehat P)\le \epsilon} \min\_{{(\vec a,\vec b)}}\max\_{{\vec\alpha }} ~\mathbb E\_{\widehat Q}\left[\frac 1{N\_C}\sum\_{i=1}^{N\_C}g\_i(a\_i,b\_i,\alpha\_i,{{\theta}},z)\right]$

where $\vec a = \\{a\_1,...a\_{N\_C}\\},\vec b = \\{b\_1,...b\_{N\_C}\\},\vec \alpha = \\{\alpha\_1,...\alpha\_{N\_C}\\}$. Using a technical similar to Sec 3.2, we obtain  our formulation of MDRAUC-Df as follows

${(MDf\star)}\quad\min\_{{{\theta}},\vec a,\vec b} \min\_{\lambda > 0}\max\_{\vec \alpha} \left\\{\lambda\epsilon + \mathbb E\_{\widehat{ P}}[\widehat \varphi\_{{{\theta}}, \vec a, \vec b, \vec \alpha, \lambda}(z)] \right\\}$

and 

$\widehat \varphi\_{{{\theta}}, \vec a, \vec b, \vec \alpha, \lambda}(z) = \max\_{z' \in \mathcal Z}\left[\frac 1{N\_C}\sum\_{i=1}^{N\_C}g\_i(a\_i,b\_i,\alpha\_i,{{\theta}},z) - \lambda c(z,z')\right].$

---

[a]Zhu, Dixian, et al. "When auc meets dro: Optimizing partial auc for deep learning with non-convex convergence guarantee." International Conference on Machine Learning. PMLR, 2022.

[b]Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." arXiv preprint arXiv:1903.12261 (2019).



Meta-review: This paper studies AUC maximization for distributional shift. The paper incorporates Wasserstein distance based DRO into min-max formulation of AUC maximization and derives upper bound of the objective for better optimization.

No critical issues have been raised in the reviews and during the discussion phase. The authors are encouraged to make the following changes in their revision.

The claim about training costs of pairwise loss formulation is not correct. Similar DRO technique could be easily injected into pairwise loss formulation and the optimization could be much easier. More discussion and comparison should be done.

Related work should be moved to the main text given there are tremendous literature on AUC maximization and DRO.

The distributional shift should be made clear in the experiments. Other strong baselines about AUC maximization that should be compared, e.g., SOPA-s, Compositional AUC [1].

[1] Compositional Training for End-to-End Deep AUC Maximization. Yuan et al.
Decision: Accept (poster)