Compositional Training for End-to-End Deep AUC Maximization
Authors: Zhuoning Yuan
Zhishuai Guo
Nitesh Chawla
Tianbao Yang
Venue: ICLR 2022 Spotlight (2022)
Keywords: Compositional Training
Imbalanced Losses
AUC optimization
Deep Learning

Abstract: Recently, deep AUC maximization (DAM) has achieved great success in different domains (e.g., medical image classification). However, the end-to-end training for deep AUC maximization still remains a challenging problem. Previous studies employ an ad-hoc  two-stage approach that first trains the network by optimizing a traditional  loss (e.g., cross-entropy loss) and then finetunes the network by optimizing an AUC loss. This is because that training a deep neural network from scratch by maximizing an AUC loss usually does not yield a satisfactory performance. This phenomenon can be attributed to the degraded feature representations learned by maximizing the AUC loss from scratch. To address this issue, we propose a novel compositional training framework for end-to-end DAM, namely compositional DAM. The key idea of compositional training is to minimize a compositional objective function, where the outer function corresponds to an AUC loss and the inner function represents  a gradient descent step for minimizing a traditional loss, e.g., the cross-entropy (CE) loss. To optimize the non-standard compositional objective, we propose an efficient and provable stochastic optimization algorithm. The proposed algorithm enhances the capabilities  of  both robust feature learning and robust classifier learning  by alternatively taking a gradient descent step for the CE loss and for the AUC loss in a systematic way.  We conduct extensive empirical studies on imbalanced benchmark and medical image datasets, which unanimously verify the effectiveness of the proposed method.  Our results show that the compositional training approach dramatically improves both the feature representations and the testing AUC score compared with traditional deep learning approaches, and yields better performance than the two-stage approaches for DAM as well. The proposed method is implemented in our open-sourced library LibAUC (https://www.libauc.org) and code is available at https://github.com/Optimization-AI/LibAUC.

Published as a conference paper at ICLR 2022
COMPOSITIONAL TRAINING FOR END-TO-END DEEP AUC MAXIMIZATION
Zhuoning Yuan', Zhishuai Nitesh V. Chawla², Tianbao Yang' Department of Computer Science, University of lowa 2 *Comuteer Science Engineering, University of Notre Dame
(zhuoning-yuan, zhishuai -guo, tianbao-yang) euiowa. nchawla@nd edu
ABSTRACT
Recently, deep AUC maximization (DAM) has achieved great success in differ- ent domains (e.g., medical image classification). However, the end-to-end train- ing for deep AUC maximization still remains a challenging problem. Previous studies employ an ad-hoc two-stage approach that first trains the network by op- timizing a traditional loss (e.g., cross-entropy loss) and then finetunes the net- work by optimizing an AUC loss. This is because that training deep neural network from scratch by maximizing an AUC loss usually does not yield a satis- factory performance. This phenomenon can be attributed to the degraded feature representations learned by maximizing the AUC loss from scratch. To address this issue, we propose novel compositional training framework for end-to-end DAM, namely compositional DAM. The key idea of compositional training is minimize compositional objective function, where the outer function corre- sponds to an AUC loss and the inner function represents a gradient descent step for minimizing traditional loss, c.g., the cross-entropy (CE) loss. To optimize the non-standard compositional objective, we propose an efficient and provable stochastic optimization algorithm. The proposed algorithm enhances the capabil- ities of both robust feature learning and robust classifier learning by alternatively taking gradient descent step for the CE loss and for the AUC loss in a systematic way. We conduct extensive empirical studies on imbalanced benchmark and med- ical image datasets, which unanimously verify the effectiveness of the proposed method. Our results show that the compositional training approach dramatically improves both the feature representations and the testing AUC score compared with traditional deep learning approaches, and yields better performance than the two-stage approaches for DAM as well. The proposed method is implemented our open-sourced library LibAUC (www. libaucc.... org) and code is available https: http://ggtttuu.... com wmwp://tttiiiiiiiiiiii AI LiDUCC
INTRODUCTION
Deep AUC maximization (DAM) represents a new learning paradigm for deep learning, which max- imizes the under ROC curve (AUC) training dataset for learning deep neural network. has received increasing attention recently due to the advancement in large-scale non-convex opti- mization algorithms for AUC maximization (Liu et al. 2019a Yuan et al. 2021 Guo et al. 2020a b).
Recently, DAM has been successfully applied to different domains with imbalanced data (Yuan al. 2020; Wang al., 2021b). For example, Yuan et al. (2020) has employed DAM for vari- cty of medical image classification tasks, e.g., classification of X-ray images, skin lesion images, mammograms, and microscopic images, and they observed great improvements with about 1%~55% AUC increase over traditional deep learning approaches by optimizing a standard loss function, e.g., the cross-entropy (CE) loss. These pioneering studies on DAM open a new direction for deep learn- ing in the presence of imbalanced data but also raise many questions yet to be solved. A particular question relates to how the network is trained by DAM. Existing studies employ a two-stage ap- proach for DAM, in which the first stage pretrains the network on the training data by optimizing traditional loss function (e.g. the CE loss) and the second stage finetunes the network by optimiz- ing an AUC loss. It was conjectured in (Yuan et al. 2020) the feature extraction layers learned by directly optimizing AUC from scratch are not as good as optimizing the standard CE loss, similarly optimizing wligh-weighted loss for deep learning with inbalanced data (Cao et al. 2019)

---

Published as a conference paper at ICLR 2022
CE (scratch)
AUC (scrwtch)
AUC-CE
Figure I: t-SNE visualization of feature representations of an imbalanced training set for the Catvs- Dog visualized by ISNE learned by different methods (from left to right); optimizing CE loss, an AUC loss, a linear combination of CE and AUC loss, and compositional objective by our method.
Although the ad-hoc two-stage method of DAM has achieved some success, this approach leads to several undesirable consequences increasing the engineering costs in practice: (i) which layers should we finetune in the second stage? Fine-tuning all layers increases the training costs but not necessarily improves the final performance (Jamal et al. 2020 Qi al. 2020); (ii) when do we stop the training for the first-stage? A long training time for the first-stage increases the overall training costs but not necessarily increases the final prediction performance, while a short training time for the first-stage could harm the prediction performance (Kang et al.) 2019). Hence, the literature has suggested different tricks for the two-stage approach, including the decoupling method that simply optimizes the classifier layer in the second-stage (Kang et al., (019) and deferred re- beiehgtting that only dedicates the iterations with the largest step size to the CE loss (Cao et al. 2019). which could be borrowed to DAM as well. However, an important question remains open regarding DAM:
How can we conduct end-to-ent training for deep AUC maximization?
To answer this question, we have examined the learned feature representations by optimizing an AUC loss directly from scratch and confirmed the conjecture in (Yuan et al. (020) that the learned feature representations exhibit no advantage over optimizing the CE loss directly. In Figure [ we visualize the feature representations on an imbalanced training set for the CatvsDog classification learned by different methods and visualized by t-SNE (van der Maaten & Hinton (2008). We can see that optimizing an AUC loss from scratch (2nd column) does not yield a cleaner feature representations for the two classes of data than optimizing the CE loss. What makes end-to-end deep learning successful is its superb feature learning capability, i.e., the lower layers capture the low-level features and higher layers capture the high-level features. In terms of feature learning. different examples roughly have equal weights regardless which classes they belong to. From this perspective, we could understand why optimizing AUC loss alone bears worse feature learning capability. The AUC loss assigns different weights to different examples from different classes for more robust classifier learning. In particular, the data the positive class has higher weight than data in the negative class. These non-equal weights are important for learning a robust classifier given that feature representations have been learned well but are not readily helpful for learning feature representations in an end-to-end fashion. Can we achieve both effects in unified and end-to-end learning framework, i.e., optimizing the CE loss with equal weights for robust feature learning and optimizing an AUC loss with uneven weights for robust classifier learning? A naive approach is to simply optimize a linear combination of the CE loss and an AUC loss. However, this approach has trade-off, meaning that AUC is not necessarily maximized due to the presence of the CE loss in the objective and the learned feature representations could be degraded by the AUC loss (Figure I 3rd column).
In this paper, we propose better and novel end-to-end training method that not only achieves both benefits of minimizing the CE loss for robust feature learning and minimizing an AUC loss for robust classifier learning, but also achieves the effect of "1+1>2", i.e., achieves better performance than the naive linear combination approach. The novel synthesis lies in how we composite the two training steps corresponding to the CE loss and the AUC loss. The central idea is to minimize a two-level compositional objective, where the outer function is an AUC loss, and the inner function is a gradient descent step towards minimizing the CE loss, which represents a quick adaptation to the solution to minimizing the CE loss. We propose novel efficient stochastic algorithm with provable convergence for minimizing the compositional objective, which performs alternating gradient-based updates that are first based on the gradient of CE loss and then based on the gradient of an AUC loss at the point obtained in the first step. We summarize
contributions below.
We propose a novel training framework for end -nd deep AUC maximization, namely compo- sitional DAM. The novel compositional objective enables not only the robust feature learning of
2

---

Published as a conference paper at ICLR 2022
lower layers from minimizing the standard loss function but also the robust learning of classifier from minimizing an AUC loss.
Theoretically, we propose efficient stochastic optimization algorithm for solving compositional DAM, and establish the same convergence rate as standard SGD for optimizing a standard aver- aged loss. Empirically, we conduct extensive studies on benchmark and medical image datasets, and observe that the proposed method not only improves the baseline methods including the naive linear combination approach but also improves ad-hoc two-stage approaches for DAM. The learned feature representations of compositional DAM (c.g., Figure rightt) are much better than those learned by minimizing the CE loss or an AUC loss alone and their combination.
2 RELATED WORK
Deep AUC Maximization, AUC maximization has a history of two decades. Most of the existing studies revolve around the design of efficient optimization algorithms. Earlier papers have focused
on full batch methods (Herbrich et al.
1999.
Yan et al. 2003) Ferri et al., 2002 Freund et al.. 2003]
Joachims 2005 Herschtal& Raskutti 2004 Rakotomamoniy
2004 2004|ZZang et al. 2012) and online
optimization methods (Zhao al. 201 Kar et al. 2014 2013 Gao 2013). Recently, stochas-
tic optimization for AUC maximization has become the dominating approach (Ying et al. 2016 Liu
et al. 2018, Natole et al. 2018 2019). Ying et al. (2016) propose milestone work for stochastic optimization of AUC. They consider optimizing the pairwise square loss and propose an equivalent min-max formulation that transforms the original ton-decomposable objective into decomposable one, which enables the design of efficient stochastic methods based on mini-batch of data without explicitly constructing the pairs. The min-max formulation also serves as the basis for recent works on DAM (Liu et al. 2019a Yuan et al. 2021] Guo et al. 2020a b). (Liu et al. 2019a) is the first work that explicitly considers DAM and develops the first practical and provable stochastic algorithms for DAM based on the min-max formulation of the pairwise square loss function. However, this work only focuses on optimization and experiments are done on simple benchmark datasets. Later, Yuan et al. (2020) propose a new robust loss in the min-max form for DAM and evaluates the performance of DAM on various medical image classification tasks, which demonstrates great success of DAM. However, none of these works have addressed the problem of end-to-end training for DAM.
Deep Learning with Imbalanced data. Deep learning in the presence of imbalanced data has recently attracted tremendous attention (Cui et al. 2019 Johnsson & Khoshgoftaar] 2209)) Masko & Hensman 2015 Lee et al. 2016 Khan et al.] 201 7, Dablain et al.] [021 Ren et al. 2018| Jamal
et al.] 20200 Qi et al. 2020; Lin et al. 2017: Cao al. [019 Kang et al. 2019: Liu al. 2019b
Zhu & Yang 2020 Zhou et al. 2020; Xiang et al. 2020; Wang et al. 2021a| [220]] Menon et al.
2021). Among these studies that are closely related to our work include (Lin et al. 2017] Cui
et al. 2019: Cao et al. 2019: Qi et al.) 2020: Kang et al. 2019 Jamal et al. (020)), which focus on optimizing different objectives from the standard CE loss, including class weighted loss, focal loss, individually weighted loss functions, etc. Nevertheless, these works are not directly comparable to our method for maximizing AUC.
Two-stage Approaches. However, directly optimizing a weighted loss for training deep neural network from scratch does not work well (Cao et al. 2019 Kang et al. 2019 Jamal et al. 2020 Qi
et al. 2020 Yuan et al. 2020). This phenomenon was first observed in (Cao et al.] 2019), which attributed to the degraded feature representations. To tackle this issue, [Co et al. 2019) propose a deferred
trick. It minimizes the standard average loss for the first stage and switches to minimizing the Eass-weighted loss or re-sampling method in the second stage. In their paper, the first stage is defined as the training period from the beginning to the iteration that the step size was reduced for the first time in SGD or momentum methods. Kang et al.] (2019) investigate a decoupling approach, where the first stage learns the feature representations (i.e., feature extraction network) by optimizing a standard loss with a large number of iterations, and the second stage learns robust classifier (i.e. the classifier layer). The authors show that the decoupling approach can achieve better performance than the deferred re- re-weighting trick in (Cao et al.] 2019). However, the decoupling approaches do not necessarily yield the best performance. In particular, some studies have found that fine-tuning some higher layers besides the classifier layer in the second stage is beneficial (Jamal et al., 2020, Qi et al. 2020). Recently, Yang & Xu (2020) propose to use relf-supervised learning for learning the feature representations for the first stage and to switch to re- weighting method in the second stage. Different from these studies, our work is design an elegant end-to-end training framework for deep learning with an AUC loss.
weeigghinngre-sapllin
3

---

Published as a conference paper at ICLR 2022
3 COMPOSITIONAL TRAINING FOR DEEP AUC MAXIMIZATION
Notations, We use (x. y) to denote an example, where X E Id denotes the input and y denotes its corresponding label. Let || || denote the Euclidean norm of a vector, and let w E Rd denote the weight parameters of a deep neural network. A function F(w) is called L-smooth if its gradient L-Lipschitz continuous, i.e., [VF(w) - VF((') I L|[
w'll Let f(w; x) denote the
prediction scores of deep neural network parameterized by w on an input where f'w x) E I for binary classification with y {1, -1]. Denote by D = (X11)))).... (xn= 9nh) a set of training examples. Let l f(w; x, y) denote loss function
individual data, e.g., cross-entropy
loss. We let L(w; S) denote an aggregate loss function defined on a set of samples S S D. When S = DD simply use the notation T.(w) (w) T(w:: D). Let (0) denotes an Fuclidean projection on the set f. Denote by n, (n-) the number of positive (negative) examples.
A standard approach of deep learning is to minimize averaged loss training examples, i.e.,
min LAva(W) ERE
AUC losses. AUC (area under the ROC curve) is commonly used measure for evaluating classi- fiers for binary classification with imbalanced data. Recently, there emerge voluminous studies on optimizing AUC score for learning predictive model (e.g., deep neural network). The idea is to optimize a surrogate loss for the AUC score. A special surrogate loss is the AUC square loss (Gao & Zhou 2015), which is defined as:
Σ (c-(((() ((w;)))))²
mil
Mi 1-1
where e is margin parameter (e.g., 1). Since directly optimizing the above pairwise loss is com- putationally expensive, existing works transform the above problem into equivalent min-max optimization (Liu et al., 2019a). which is decomposable over individual examples:
min max & (w, a, b,0) : Φ (w, a, 3;) where w.o,b BER
<( w. 2,,,,,,0xx,1100 Mi) = p) (f(w; ( a)² ||-= + P[s - b) 1)
(3)
p(1 p)0² + 20 (p(1 ppcc
- (1 (w; (w):)))
= and p = /n. From the above objective function 6, we can see that each example x; also has class-level weight for their contributed loss, i.e., the data from the positive class is weighted by - p and the data from the negative class is weighted by p.
It was recently shown that the AUC square loss is sensitive to noisy data and also has adverse effect when trained with easy data. Hence, Yuan et al. (2020) proposed the min-max AUC margin (AUCM) loss, whose optimization problem is (2) with f==10 to I u} for some Let us define LAuc(w) = min,,6 maxgen + (w, a, b, f) as the AUC loss function.
3.1 COMPOSITIONAL DAM: COMPOSITIONAL TRAINING METHOD FOR DAM
In this section, we present the proposed compositional training for DAM. Our proposed objective for end-to-end deep learning is given by
min LAUc(w - a o. LVVG))))) weRd
(4)
where is typerrpprameee Different from (w)) the above objective is a compositional func- tion, where the inner component w - a LAVG (w) is another function of w. We refer to the above objective the compositional objective and method for minimizing the above compositional ob- jective as compositional training.
To understand the compositional objective 4. we take the second-order Taylor expansion of the compositional objective, which include three terms:
LAuc(w- - a Lvva (w)) ~ Lwuc(w) a Luu(() VLAvG(W) + Ca² Ca2/22 / /2/ Lwwa(w)) (5) where C represents the Lipchitz continuity constant of V L.uc()). In order to understand how the three terms play their roles and evolve in the optimization process by our proposed algorithm pre- sented in next subsection (Algorithm []. conduct some empirical studies on several benchmark
4

---

Published as a conference paper at ICLR 2022
Leuc(w)
Pleue (w) (Lans(
Layz(
0.3
0.00
0.3
0.2
0.04
0.2
0.02
0.2
0.1
0.00
100
0.0
80
20
40
1.00
20
09
100
Epoch
Ezoch
Epoch
Epoch
Figure 2: Evolution of different terms in (5) computed in the process of our optimization algorithm (Algorithm I on the CatvsDog data. Lave is the averaged CE loss. Please refer to Appendix A.6 for more details of the calculations.
datasets reported in Appendix A.6 Here, we explain the result of the Cat classification shown in Figure 2 Initially, the first term LaUc(w) dominates the objective and the algorithm will fo- cus on pushing this term to be smaller (Ist column), once it reaches the same level of the third term the algorithm will shift its focus to push LAV (w)) smaller (2nd column) while keeping V VLAuc(w) (w) LwvG (w) to be positive (3rd column). This process will continue by alternating be- tween the efforts of pushing (wu smaller and of pushing |/ Lava (w)l(
be smaller while keeping V VLAUC(WW (w) V LAvG (w)) to be non-negative. We also notice that the final AUC loss is close to zero. The same phenomena are also observed on other datasets.
To further understand the compositional training intuitively, let us take a thought experiment by using gradient descent method to optimize the compositional objective. First, we evaluate the inner function by w a Lavo(w). We can see that u is computed by a gradient descent step for minimizing the averaged loss LAvG (w). which facilitates the learning of lower layers for feature extraction due to equal weights of all examples. Then, we take gradient descent step to update w for minimizing the outer function (-uc(-) by using the gradient VLAuc( instead of V VLAuc(w) Because u is better than in terms of feature extraction layers, taking gradient descent step using V LAUC(U)) would be better than using VLAUC(W). In addition, taking a gradient descent step for the outer function LAue(+) will make the classifier more robust the minority class due to the higher weights of examples from the minority class. Overall, we have two alternating conceptual steps, i.e., the inner gradient descent step step, and the outer gradient descent step w - n) - a Lova ()) V Luuc(u) acts as a classifier robustification step, where 17 is a step size.
V LAvG (w) acts feature purification
VS. linear combination approach. It is notable that minimizing the compositional objective Lauc(w a a LAVG(W) (w)) is different from minimizing linear combination of an AUC loss and the averaged loss, i.e., LAUc (w) + cLLvGG (w), where > is a combination weight. First, minimiz- ing the latter objective will push VLAuC(w) an opposite direction from V Lww at the optimal solution, which is different from minimizing the compositional objective in light of the three terms in ( Second, if we take a gradient descent method for minimizing this objective, the update of w is given by w w-7/ (VVVAu +cV T.vG(()) This update fundamentally different from the two alternating steps of compositional training that use gradients of the AUC loss and the average loss at different points. Third, minimizing the linear combination has a trade-off, meaning that the AUC score is not necessarily maximized due to the presence of the CE loss in the objective and the learned feature representations are degraded due to the presence of AUC loss.
LV = O. This makes V LAuC (w) to have
More discussions. Finally, we note that the inner gradient descent step w ilar to the idea used in model-agnostic meta learning (MAML) (Finn et al. compositional objective works fundamentally different from that for MAML... In MAMI., the outer loss function and the loss function for the inner gradient descent step is the same. In contrast, the two loss functions in our objective are different. But similar MAML approaches, we can also run multiple gradient descent steps for the inner function, i.e, the inner function w a a LAvG can be replaced by multiple gradient descent steps. In our experiments, we also found this trick to be helpful for improving the performance.
V LAvG (w) is sim- 2017). However, our
3.2 STOCHASTIC OPTIMIZATION ALGORITHMS
In this subsection, we develop efficient stochastic optimization algorithms for optimizing the compo- sitional objective (4) for DAM. First, we argue the necessity for such development. (i) the problem is a min-max form and the objective is a compositional function, which makes computing an unbiased stochastic gradient of the objective TAue (w- a LVVwa (w)) impossible. Existing algorithms of
5

---

Published as a conference paper at ICLR 2022
Algorithm Primal-Dual Stochastic Compositional Adaptive (PDSCA) method for solving (6)
Require Parameters: So. B1, a, Go. 77.7%% 2: Initialization: Wo (Wo: ao: bu) E IR 12 fo, Ug E Rd+2
3: for 0, ..... do
4: Sample two sets of examples denoted by S1. (nh(W;;:1) 5: U++1 = - (1-3o) +
6: O = (wh((w;;Si)
Vugi (4++1; + 0.V uu
7:
Zt-1 = (1 B1)Z, )z, ; +,00,
8:
22,1+1 ho([O), 0.....t)
ohe can be implemented by that in Appendix B owith the simplest form he =
9:
Wi+1 = W; 721.11+00
Ze+1
8,41 Il ( S US) Vgs((.)))
11: end for
non-convex min-max optimization for DAM that focus on minimizing LAUc(w) LLu et al. 2019a
Yuan al. 2021 Guo al. 2020a b) are not applicable due to the presence of inner function
w LLvG
(ii) Our objective is also different from that of MAML due to that LAUC (-) is
min-max form, which renders existing algorithms for MAML (Finn al. 2017; Fallah al. 2020)
not applicable. Hence, below we propose an efficient stochastic algorithm for solving the compo- sitional training for DAM whose objective of the min-max compositional form, and establish its convergence rate similar to that of standard SGD for minimizing the standard averaged loss.
In particular, for the considered AUC loss, the compositional objective becomes:
min max & (w a LAvG (w), b, w.a,b AEn
Φ (w a LAvG
y1).
(6)
We denote by a tuple w = (w; a: 6). For simplicity of presentation, we write d(w, a, b, 0, X,, Us) as (ww,0; vi = g x;; v1) + 0g2 x;, M.) go (0),
where
91 (w; x, 14) =(1 p) (f (f:(w: a) T + p(f( [=-
(7)
+ 2pf ( (W: - 2(1 (w;
and g2 (W; y:) 2 2
[p (1- x,, (vx and g3) p0(1 p)00
Σ Eies 91 (W;X,, g2 (w: S) [ST Eies 92 (w;; W:). Let h(w) = (w- a (VLavo(W):a b), Vwh(w) = -I- a LAva(W) 1;1), and h(W:S) =
Denote by gi (W:
[S
(w a LVVG S); b).
We propose primal-dual stochastic algorithm shown in Algorithm I which is referred to as PDSCA. We provide some explanations of our algorithmic design. First, the step of updating 4,+1 corresponds to feature purification step. We use moving average technique to update u,41 that takes all historical updates into account, which is inspired by existing stochastic algorithms for optimizing compositional functions (Wang et al. 2017). This is important for us to prove the con- vergence rate of O(1/VT) without using large batch size at each iteration. If we simply using u,+1 = ( S1) (i.e., setting 3o = 1) to estimate h( ((w,), there will be large error in estimating the gradient V u.gh (1++1:82) and Vug2(ut+1 (1++1:82) 82) in step 6. Second, the step 6 is estimate the gradient of the outer function. We use two independent mini-batches S1, S2 to ensure that O₂ is an unbiased estimator of Vh(w,) V @w 0). Using two independent mini-batches is also help- ful for improving generalization as demonstrated in experiments. Third, the steps 7 9 are similar the momentum and adaptive methods for updating the model parameter. The step 8 is used for computing the adaptive step size /22,1-1 + cu. which is similar to adaptive methods used for deep learning, such as Adam, AMSGrad, AdaBound (Kingma & Bal 2015 Reddi et al. 2018 Luo et al. 2019). We use general function he in the algorithm, which can be implemented by different methods corresponding to different adaptive step size choices. We present different he in the Ap- pendix B The simplest one hr = 1 corresponds to that we do not use adaptive step size and only use the momentum update. Fourth, the step 10 is for updating the dual variable 0 using a stochastic gradient ascent method. Finally, point out that PDSCA is similar to some existing non-convex strongly-concat min-max optimization algorithms (Guo et al., 2021) but with additional care on the inner gradient descent step w LVvG We present an informal convergence of PDSCA below.

---

Published as a conference paper at ICLR 2022
Theorem 1. (Informal) Under appropriate conditions on the functions LAve, 91. and boundness
conditioo on 0G, Wt, Vl(w;; x. y), (WEX y), with Bo, B1 = O(L/VT), 71 7/2 = O(L/VT)
and a small constant T, Algorithm ensures that E T+ E 0 VF(()))l - 0 where
F(w) maxsgn & (w- - aV LAvG aw),,,,
Remark: We will present the detailed conditions in the supplement when proving the above the- orem due to limit of space. The above theorem indicates that can optimize the compositional objective (6) with the same convergence rate as optimizing the averaged loss (1) for deep learning.
Practical Implementations. It is notable that Vh(w, S1) (1 - a involves the Hessian matrix VZW(W;S,) Indeed, we only need to compute the Hessian vector product involving in step 6. Similar computation occurs in the meta learning algorithms (Finn al., 2017 Fallah et al.) (020). Inspired by practical implementations of MAML (Finn et al.) 2017) that simply ignore the second-order term, use the same trick in our experiments, An additional useful trick inspired by MAML is that we can take k 2 1 gradient descent steps for the inner function, correspondingly we maintain and update several u variables similar to step 5, i.c., using h(,,,Si (1)
(step 6)
for updating the first u and using (uu, +++1 1 S2) for updating second + and so on so forth. In our experiments, we found that tuning k E [1, 2 is useful.
Finally, it is notable that although we focus on optimizing AUC loss for binary classification in this work, our compositional training method can be also extended to optimize other weighted losses in an and-to-end fashion, and we include some discussion and results in the Appendix D
EXPERIMENTS
In this section, we present some experimental results. We choose five baselines: optimizing the AUC loss from scratch (AUC*), optimizing the CE loss (CE), optimizing linear combination of the AUC loss and the CE loss with a tuned weight (AUC-CE), the two-stage method with deferred re-weighting trick (Cao et al. 2019) (TS-DRW), the two-stage method by decoupling the learning of feature network by minimizing CE loss and the learning of a classifier by minimizing the AUC loss (TS-DEC) (Kang et al. 2019). We denote our method by CT (AUC). For AUC loss, we use AUCM loss with the margin parameter fixed to be (Yuan et al., 2020). We conduct experiments on four benchmark datasetes and four medical image datasets. The statistics of these datasets are included in the Appendix A. More training configurations can be found in Appendix A.2
Benchmark datasets. We choose four benchmark image classification datasets, namely CatvsDog, CIFARIO (C10), CIFAR100 (C100), and STL10 (S10). For AUC maximization, we construct im- balanced binary versions of these datasets by varying the imbalanced ratios (the ratio of positive examples to the total number of training examples) similar to (Yuan et al. 2020). We use ResNet20 as the prediction network. The weight decay set le-4 for all experiments. For algorithms to maximize AUC, we use a batch size = 128 and train total of 100 epochs, and we use step size 0.1 and decrease it by 10 times at 50% and 75% of total training time. We tune the beta parameters of method in range [0.1, 0.99] with a grid search and find that good values are around 0.9. For linear combination methods, we tune the weight c of two losses in (0.25, O.5, 0.75). We tune the 0.1. For all benchmark data, we
number of inner gradient steps for CT in k E (1,2,3) with run three times for different random seeds and compute the mean and standard deviations.
Medical image datasets. We also conduct experiments naturally imbalanced medical datasets. We choose four medical image datasets, namely Melanoma data, Che DDSM+, and PatchCam data. The Melanoma dataset is from the Kaggle 2020 competition (Rotemberg al. (021), I). which contains 33,126 126 labeled images in training set, including 584 positive samples and 32,542 negative samples. We manually construct training, validation and testing datasets following 70/ split. For this dataset, we use the images with 256x256 resolution in the experiments. Che is large- scale chest X-ray dataset (Irvin et al. 2019). which has 224,316 images with 224 x 224 resolution. The dataset contains 5 binary classification tasks corresponding to 5 diseases, i.e., Cardiomegaly (CO), Edema.. Consolidation, Atelectasis, Pleural Effusion. We evaluate the performance on the official validation set consisting of 200 patient studies and report the averaged AUC scores of all 5 diseases. The DDSM+ data is a combination of two datasets namely DDSM and CBIS-DI DDDS (Lee et al.) 2017: Bowyer et al.] 1996] Heath et al. 1998), which consists of 55,890 mammographic training images (224x224) with an imratio of 13 % and 15,364 images for testing with an imra- tio of 13%. The PatchCamelyon dataset consists of 294.912 color images (96x96) extracted from
7

---

Published as a conference paper at ICLR 2022
Table I: Testing performance on benchmark datasets and medical datasets. The percentage number
is the second row denotes the imbalanced ratio (cf the text).
For AUC Maximization
For AUC Maximization
Datasets
imratio
1%
10%
30%
Dutasets
Method
AUC
CE
0.742=0.003
0.917=0.006
0.957=0.001
CE
0.879=0.008
A
0,753+0.003
0,915=0.002
0.964+0.003
AUC
0,868=0.006
AUC-CE
0.770=0.007
0.939=0.004
0.974+0.003
AUC-CE
0,880=0.005
CATvsDOG
Melanoma
TS-DRW
0.750+0.009
0.914=0.003
0.961+0.001
TS-DRW
0.878=0.007
TS-DEC
0.75420.010
0.918a0.003
0.963100001
TS-DEC
0.877=0.005
CT (AUC)
0.789=0.008
0.946+0.002
0.977:0.001
CT (AUC)
0.900=0.002
CE
0.68920.003
0,901+0:002
0.944400.001
CE
0.892=0.001
ALC
0.728:0.002
0.905=0.002
0.946:60.001
AUC
0.899=0.002
AUC-CE
0.735+0.003
0.928+0.001
0.957=0.001
AUC-CE
0,902=0.000
CIFARIO
CheXpert
TS-DRW
0.708#0.002
0.896#0.002
0.94620.00 003
TS-DRW
0.900=0.002
TS-DEC
0.707=0.002
0.897+0.002
0. 0.944+0.001
TS-DEC
0.897=0.001
(AUC)
0.739=0.004
0.935+0.001
0.964:0.001
CT (AUC)
0.909=0.003
CE 0.81950.004 0.655
0.83520.004
CE
0.949=0.001
AUC
1665+00005
0.805=0.011
0.887=0.007
AUC
0.929=0.00)
AUC-CE
0.66810.007
0.83610.006
0 0.901000001
AUC-CE
0,957
STLI0
TS-DRW
0.655#0.004 004
0.803=0.013
88720.002
DDSM+
TS-DRW
0.942=0.00
TS-DEC
0.661=0.002
0.816=0.007
0,882+0.000 007
TS-DEC
0.941=0.00
(AUCCC)
0.673+0.010
0.837+0.006
0.906200001
CT (AUC)
0.981
CE 0.58020:001
0.691
0.758:0.004
CE
0.869=0.007
A 1666=00004
0.705=0.003
0003
AUC
0.868=0.006
CIFAR100
AUC-CE
0.605+0.004
0.716+0.003
0.795+0.001
PatchCam
AUC-CE
0.868=0.00%
TS-DRW
0.58810.003
0.69110.004
0.76210.001
TS-DRW
0.867=0.006
TS-DEC
0,587=0.001
0.692=0.003
0,762+0.002
TS-DEC
0.869=0.009
CT (AUC)
0.609#0.002
0.725+0.001
0.80910.002
CT (AUC)
0.891=0.003
histopathologic scans of lymph node section for training and 32,768 images for testing with bal-
anced class ratio (Velling et al.) 2018 Bejnordi et al. (017). For PathCamelyon, we manually construct an imbalanced training dataset with an imratio of 1% and keep the testing set balanced. For Melanoma data, we adopt a EfficientNetV2-S. (Tan & Le 1021 the network structure, and for CheXpert, DDSM+, PatchCam, we use DenseNet121 (Huang et al. 2017). We tune the number of inner gradient steps for CT in k E 21 and also tune
10.1, 0.05, 0.01} for the inner steps.
We can see that the proposed composition training method outperforms all baselines all datasets for maximizing AUC. In addition, we have the following observations: (i) optimizing an AUC loss from scratch does not necessarily yield a better performance than minimizing the standard CE loss; (ii) the CT (AUC) method dramatically improves the performance of optimizing the AUC loss from scratch, with about 2%%~~5 improve- ment on difficult medical classification tasks; (iii) the CT (AUC) method is generally better than the linear combination approach (AU (AUC-CE), especially on the more difficult medical image datasets. It notable that the reported results on some medical datasets are not comparable with that in (Yuan et al. 2020) because (i) we report the performance on official CheXpert validation data instead of the official testing data; (ii) we use a smaller resolution on Melanoma data; (ii) we do not tune the margin parameter in the AUCM loss. We also plot the learned feature representations of training data of different datasets visualized by t-SNE in Figure 3 We can see that the proposed CT (AUC) method obtains better feature representations than the baseline approaches of optimizing CE or AUC alone and the naive lincar combination approach.
Results.
The testing AUC results are reported in Table I
4.1 ABLATION STUDY
We conduct some ablation study including (i) the comparison of convergence curves and the running time analysis of different methods; (ii) the verification of our algorithmic design.
Convergence Curve. Below, we compare the convergence speed of our CT (AUC) approach with other end-to-end learning baselines, i.e., CE, AUC, AUC-CE, with results on four benchmark datasets plotted in Figure 4 The results indicate that our algorithm enjoys even faster convergence in terms of number of epochs. The convergence curves of testing AUC are included in Appendix [A.4]
Runtime analysis. We notice our method has larger running time per-iteration than minimizing the CE and the AUC loss alone because of several backpropagations, but with a reward of faster con- vergence in epochs and better testing performance. For fair comparison, we have run the baselines with the same amount of time as our method and observed they are still worse than our method (cf Appendix A.5). For example, on CIFAR 10 (10%), CT (AUC) can achieve an AUC of 0.944 with running time of 1000s, in contrast, the baselines CE, AUC, AUC-CE, TS-DRW, and TS-DEC use same running time and achieve AUC scores of 0.925, 0.915, 0.943, 0.917, 0.917, respectively.
8

---

Published as a conference paper at ICLR 2022
AUC-
C100 AIC
110 AIC
Rearema AUC
DDGN AUC
PotchCas AUC
EIDET
Netrronaa CT
DSSN cT
Pattican
Figure 3: t-SNE visualization of training data (+ is positive and is negative) by (from top bottom)
optimizing the CE loss, an AUC loss from scratch, a linear combined loss, and our CT method.
CAT DOG (10%)
CIFAR10 (10%)
CIFAR100 (10%)
STL10 (10%)
1.0
1.0
1.0
0.9
Train AUG
Train AUC
0.8
3.0
Train
02
22
D2
AUD-CE
AUC 22
0.8
ALC-CE
AUC-CE
25
50
75
100
25
50
75
100
25
50
75
100
25
50
75
100
Spach
Eprics
Bzachh
Figure 4: Convergence curves on four benchmark datasets with an imbalance ratio of 10%.
Verification of Algorithmic Design. We validate three algorithmic choices. (i) Using two inde- pendent mini-batches S * S2 is generally better than using the same mini-batch. (ii) Using the momentum update for u (i.c., So < 1) is better than without using momentum update (6o = 1). (ii) tuning the number of inner gradient steps A E 21,2,3} 3} is helpful for improving the perfor- mance. The results are demonstrated in Table 2 where all results are averaged over three trials.
Table 2: Left: S S2 vs S S2, right: 8g = 1 8o in Algorithm I Note that we tune E (1,2,3) for for the left table and fix l = for the right table. left (S) * S2) vs right (gp S 1) verifies that tuning & is helpful.
Dataset
Method
1%
Imbalance Ratio
10%
30%
Method
1%
Imbalance Ratio
10%
30%
CATvSDOG
S
0.78420.007
0.94120.002
0.97520.001
Ag
0.76520.005
0.93720.004
0.97120.00
S # Sz
0.78920.008
0.94610.002
0.97749.001
0.76920.007
0.93920.004
0.97540.006
CIFAR10
S, = Sp
0.738=0.00S
00002
0.95950.0000
76 =511
0.724=0.008
0.91850.004
0.957.0.001
0.740=0.004
0.935#0.001
0.964+0.001
80
0.725=0.011
0.929=0.002
0.960:400001
S Sg
1.67180.007
0.82010.025
0.90240.008 003
===1
0.663=0.012
0.81920.018
0.895
STL.0
0.681
0.90740.001
86
0.666=0.0066
0.832=0.008
CIFARI00
S) ===8
0.608
0.70910.002
0.78840.00
=1
0.590:0.014
0.71430.004
0.791 20.004
0.60920.002
0.725e0.000
0,80910.002
0.598=0.002
0.714:0.003
0.80110.001
CONCLUSIONS
In this paper, we have proposed a novel end-to-end compositional training framework for deep AUC maximization by optimizing compositional objective. We also proposed efficient stochastic optimization method for compositional training of deep AUC maximization. We demonstrated the effectiveness of compositional training multiple benchmark datasets and medical datasets for maximizing AUC. future work, we will investigate compositional training for other imbalanced loss functions more extensively.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank anonymous reviewers for their valuable comments. This work was partially supported by NSF Career Award # 1844403, NSF Award #2110545, NSF Award # #1333212.
REFERENCES
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases women with breast cancer. Jama, 318(22):21 199922200.. 2017.
K Bowyer, D Kopans, WP Kegelmeyer, R Moore, M Sallam, K Chang, and K Woods. The digital database for screening mammography. In Third international workshop on digital mamnography, volume 58, pp. 27, 1996.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aaaaa margin loss. In Advances in Neural Information Processing Systems, pp. 1567-1578, 2019.
Yin Cui, Menglin Jia, Tsung-Yt Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268 9277, 2019.
Damien Dablain, Bartosz Krawczyk, and Nitesh V. Chawla. Deepsmote: Fusing deep learning and SMOTE for imbalanced data. CoRR, abs/2105.02340, 2021. URL
abs/2105. 02340
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model- -agossic meta-learning algorithms. In International Conference on Artificial Intelli gence and Statistics, pp. 1082-1092. PMLR, 2020.
C Ferri, PA Flach, and J Bernández-Orallo Learning decision trees using the area under the curve. In Claude Sammut and Achim Hoffmann (eds.), Proceedings of the 19th International Conference on Machine Learning, pp. 139 146. Morgan Kaufmann, 2002. ISBN 1558608737.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017. URL http::/procceednnge wmw...as.........mm......o...eae mlr. /ttesss/77//Einn7a..ion html
Yoav Freund, Raj Iyer, Robert E. Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. J. Mach. Learn. Res., 4(null):33-969, December 2003. ISSN 1532-
4435.
Lawrence Fulton, Alex McLeod, Diane Dolezel, Nathaniel Bastian, and Christopher P Fulton. Deep vision for breast cancer classification and segmentation. Cancers, 13(21):5384, 2021.
Wei Gao and Zhi-Hua Zhou. On the consistency of AUC pairwise optimization. In Qiang Yang and Michael J. Wooldridge (eds.), Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 935- AAAI Press, 2015. URL httl:: org/ Abstttttttttt1..33
Wei Gao, Rong Jin, Shenghuo Zhu, and Zhi-Hua Zhou. One-pass auc optimization. In International conference on machine learning, pp. 906-9144 2013.
Saced Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint (00.:0/800002244 2018.
Saeed Ghadimi, Andrzej Ruszczynski, and Mengdi Wang. A single timescale stochastic approxi- mation method for nested stochastic optimization. SIAM J. Optim., 30(1):960-99) 979, 2020. doi:
10.1 1205542. htt.::ttss////mm // org/10...00o httg//1..111000 1137/1 8M1 230542
10

---

Published as a conference paper at ICLR 2022
Zhishuai Guo, Mingrui Liu, Zhuoning Yuan, Li Shen, Wei Liu, and Tianbao Yang. Communication- efficient distributed stochastic AUC maximization with deep neural networks. In Proceedings of the 37th International Conference Event, volume 119 of Proceedings of Machine Learning Research, pp. 00..886433 3874. PMLR, 2020a. URL htk./http://proceeete wtt.:mm........aa...a... www.s/rss/vvvv w/ggo//g......o html
Machine Learning, ICML 2020, 13-18 July 2020, Virtual
ngs m
Zhishuai Guo, Zhuoning Yuan, Yan Yan, and Tianbao Yang. Fast objective and duality gap conver- gence for non-convex strongly-concave min-max problems. CoRR, abs/2006.06889,, 2020b. URL
htttps://arii http:://axxiv. wtg.://ivrrrrro.oo org/ http:/bs//20000
06889
Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. On stochastic moving-average
estimators for non-convex optimization. CoRR, abs/2104. 14840, 2021. URL htRppstt:://axiv
org/ aataabbb221000 abb/21144.1440
Michael Heath, Kevin Bowyer, Daniel Kopans, Philip Kegelmeyer, Richard Moore, Kyong Chang, and S Munishkumaran. Current status of the digital database for screening mammography. In Digital manmography, pp. 457-460. Springer, 1998.
Ralf Herbrich, Thore Graepel, and Klause Obermayer. Large Margin Rank Boundaries for Ordinal Regression. In Advances in Large Margin Classifiers, chapter 7, pp. 132. The MIT Press, 1999. [tRttttp:///// //wp::://wwww hwrerierrrrr.rr wwteebrrcch.eeepppeeeeeee wtt..me/paagee oorp:///mmmsssodddnnnnooo oordinrrin .ddf
Alan Herschtal and Bhavani Raskutti. Optimising area under the ROC curve using gradient de- scent. In Carla E. Brodley (ed.), Machine Learning, Proceedings of the Twenty-first International Conference (ICML 2004), Banff, Alberta, Canada, July 4-8, 2004, volume 69 of ACM Inter- national Conference Proceeding Series. ACM, 2004. doi: 10.1145/01153001115366. URL
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 590-597, 2019.
Muhammad Abdullah Jamal, Matthew Brown, Ming- Hsuan Yang. Liqiang Wang. and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 7610-7619, 2020.
Thorsten Joachims. A support vector method for multivariate performance measures. In Pro- ceedings of the 22nd International Conference on Machine Learning, ICML '05, pp. 377-384, New York, NY, USA. 2005. Association for Computing Machinery. ISBN 1595931805 doi: 1 102351. 102399. URL httpt:tt/////tt /ooi 1ttpg//...11111112333333 1102399
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):27, 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.0921) 09111 2019.
Purushottam Kar, Bharath Sriperumbudur, Prateek Jain, and Harish Karnick. On the generalization ability of online learning algorithms for pairwise loss functions. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, vol- ume 28 of Proceedings Machine Learning Research, pp. 449, Atlanta, Georgia, USA, 17- html
19 Jun 2013. PMLR. URL BRtLhtts::// /tt.tttppss/pppcccdnen.... m.r http://pwweeeennn........ereee//
Purushottam Kar, Harikrishna Narasimhan, and Prateek Jain. Online and stochastic gradient meth- ods for noomdecoeeo loss functions. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 1, NIPS" 14, pp. 694-702, Cambridge, MA, USA, 2014. MIT Press.
11

---

Published as a conference paper at ICLR 2022
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions neural networks and learning systems, 29(8):3573-35877 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
Hansang Lee, Minseok Park, and Junmo Kim. Plankton classification on imbalanced large scale
database via convolutional neural networks with transfer learning. In 2016 IEEE international conference on image processing (ICIP), pp. 3713-3717. IEEE, 2016.
Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel Rubin. A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific data, 4(1):1 2017.
Tianyi Lin, Chi Jin, and Michael I Jordan. On gradient descent ascent for nonconvex-concave minimax problems. arXiv preprint arXiv:1906.0033) 2019.
Tsung- Yi Lin. Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision. pp. 2980-2988, 2017.
Mingrui Liu, Xiaoxuan Zhang, Zaiyi Chen, Xiaoyu Wang, and Tianbao Yang. Fast stochastic auc maximization with o (1/n)-convergence rate. In International Conference on Machine Learning. pp. 3195-3203, 2018.
Mingrui Liu, Zhuoning Yuan, Yiming Ying, and Tianbao Yang. Stochastic auc maximization with deep neural networks. arXiv preprini arXiv: 1908. 10831, 2019a.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X. Yu. Large- open world. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2537-2546.
scale long-tailed recognition
Computer Vision Foundation IEEE, 2019b. doi: 10.1109/CVPRR22019990
URL http:
//openaccess. thecvf com http///oonttttttceccceeeeeess 201120019900 9/htm /Liu Large Scale_ haag--gaaaaee__ecoonnottttte
Saan__Ope___aaee Woo d CL_CcVv___00 9 __ppp__r____ .tt.l.
Liangchnn Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In 7th International Conference on Learning Representations (ICLR), 2019.
David Masko and Paulina Hensman. The impact of imbalanced training data for convolutional neural networks, 2015.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021. URL hutthtts::. /ttp:/::////eeeeeeeeeeee
net/ fow.mmidddnnnnveeeeeeoo
Michael Natole, Yiming Ying, and Siwei Lyu. Stochastic proximal algorithms for auc maximization. In International Conference on Machine Learning, pp. 3707-3716, 2018.
Michael Natole, Yiming Ying, and Siwei Lyu. Stochastic auc optimization algorithms with linear convergence. Frontiers in Applied Mathematics and Statistics, 5, 2019.
Qi Qi, Yi Xu, Rong Jin, Wotao Yin, and Tianbao Yang. Attentional biased stochastic gradient for imbalanced classification. CoRR, abs/2012.0699 06951, 2020. URL httts:://aaxi...rrrt/ abs/
Alain Rakotomamonjy. Support vector machines and area under roc curves. Technical report, 2004.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In 6th International Conference on Learning Representations (ICLR), 2018.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. arXiv preprint arXiv: 100..8003.0000 2018.
12

---

Published as a conference paper at ICLR 2022
Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz-Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Mare Combalia, Stephen Dusza, Pascale Guitera, David Gutman, et al. A petient-centrie dataset of images and metadata for identifying melanomas using clinical context. Scientific data, 8(1):1-8, 2021.
Eric Scuccimarra. Ddsm mammography: tfrecords files of scans from the ddsm dataset. Online, 2021. //ttps:/wwee www. http://wwwwwww....//e/oe/oe com/ /hooc//koch/////ooooooooo http://ooooc/ccmmmmmmmo.proooooe
ddsn-mammoogrpphy
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. arXiv preprint
urXiv:2104.00200 2021.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing high-dimensional data using t-sne. 2008.
Journal of Machine Learning Research,
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant cnns for digital pathology. In International Conference on Medical image computing and computer-ass assssstee- intervention, pp. 210-218. Springer, 2018.
Mengdi Wang,
Ethan X Fang, and Han Liu. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming, 161(1-2): 2017.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X. Yu. Long-tailed recognition by routing diverse distribution-aware experts. CoRR, abs/2010.01800 2020. URL https:
7/ //axiv.org/abs/20111 01809
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by routing diverse distribution-aware experts. In International Conference on Leurning Representa- lions, 2021a. Utthttps:: http:////ppeeevveiee net feet/forr forum?idddD 3drBz4UC,
Zhengyang Wang, Meng Liu, Youzhi Luo, Zhao Xu, Yaochen Xie, Limei Wang. Lei Cai, Qi Qi, Zhuoning Yuan, Tianbao Yang, and Shuiwang Ji. Advanced graph and sequence neural networks for molecular property prediction and drug discovery, 2021b.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for loig-tailed classification. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision ECCV 2020 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part V. volume 12350 of Lecture Notes in Computer Science, pp. 247 263. Springer, 2020. doi: 10. 0777797783-000-5855- 15. URL
https::/doi wrggl0gg....o.oo..oooo.oooo..... 1007 7/978 3 030 38558-7_11
Lian Yan, Robert Dodier, Michael C. Mozer, and Richard Wolniewicz. Optimizing classifier perfor- mance via an approximation to the wilcoxon mann- whitney statistic. In Proceedings of the Twen- fieth International Conference on International Conference on Machine Learning, ICML'03 00, pp-
848-855. AAAI Press, 2003. ISBN 1577351894.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class- laas-imbalanced learn- ing. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 19290-19301. Curran As- sociates, Inc., 2020. htRL/hhttpss//prreeeeiii wwwpeeedddiiinn.n.nineeeeeeeeeee e025b627 wwtee27791118833eeeeeeeeee (BB8:eee0ecaafccbe633333333
pddf
Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online auc maximization. In Advances in neural information processing systems, pp. 451-459, 2016.
Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Robust deep AUC maximization: A new surrogate loss and empirical studies on medical image classification. CoRR, abs/2012.00173 2020. URL htRps: arxiv. org/ abs / 22122.012222 03173
Zhuoning Yuan, Zhishuai Guo, Yi Xu, Yiming Ying, and Tianbao Yang. Federated deep AUC max- imization for hetergeneous data with constant communication complexity. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 1021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pp. 12219 12219-12229, PMLR, 2021. BRRhhtp: BttL:ttttp/pppreeeeeee http:///ooceeee....eeep mlr press/v139/ yuan211 html]
13

---

Published as a conference paper at ICLR 2022
Xinhua Zhang, Ankan Saha, and S. V. Vishwanathan. Smoothing multivariate performance mea- sures. J. Mach. Learn. Res., 13(1):6633333333 3680, December 2012. ISSN 1532-4435.
Peilin Zhao, Steven C. H. Hoi, Rong Jin, and Tianbao Yang. Online auc maximization. In ICML, pp. 233-240, 2011.
Yan Zheng, Yuchen Zheng, Daiki Suehiro, and Seiichi Uchida. Top-rank convolutional neural net- work and its application to medical image-based diagnosis. Pattern Recognition, 120: 2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: bilateral- network with cumulative learning for long-tailed visual recognition. In 2020 IEEE/CVF Conference Com- puter Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9716-9725 IEEE, 2020. doi: 10. 100.CCCV242000.220000000 10.1109/ 220.0///VE22220.22222222
httpstts:: /doi. orrg/
00974
Linchao Zhu and Yi Yang. Inflated episodic memory with region self-attention for long-tailed visual recognition. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 4343 4352. IEEE, 2020. doi: 10.
(DD0CC/PR42200.220000044 URL https: /tss:://dooo /ww. www.doio.oogoooooo 2ttp:///o...00////222222000
00440
14

---

# of classes Published as a conference paper at ICLR 2022
A SUPPLEMENT OF EXPERIMENTAL SECTION
A.1 DATASET DESCRIPTIONS
The detailed statistics of different datasetse are reported in Table Note that "# of images" refers to the number of samples for the original training set. "LT" denote long-tailed version of the datasets for multi-class tasks. Imbalance ratio (imratio) of XXX (Binary) data means the ratio of number of positive examples to number of all examples. Imbalance ratio (imratio) of XXX (LT) data means ration of the size of smallest class to the size of largest class. The DDSM+ is slightly different from the standard version of CBIS-DDSM or DDSM (Lee et al. (2017): et al. (1996); Heeth al.
(1998). ). We actually use the dataset from (Scuccimarra (2021)) constructed by Eric A. Seuccimarra, which consists of 55k images for training and 15k images for testing. The details about the dataset construction can be found here (Scuccimarra (021); Zheng et al. 20 Fulton et al. (222 ). For constructing DDSM+ the positive samples (cancer cases) are from CBIS-DDSM and negative samples (normal cases) are from DDSM. To increase the size of the training data, the author applies offline data augmentation and adds multiple augmented copies to the dataset. In particular, each image (ROI) is randomly cropped three times into 598x598 images, with random flips and rotations, and then the images are resized down to 299x299. For the testing set, the same augmentation is also applied and thus the imhalance ratios remain the same the training set. The imbalance ratio is about 13% in both training and testing sets. The train/test split in terms of patient ID follows the CBIS-DDSM split, which do not have any overlap. We will add these references in the revision and also correct the citation for CBIS-DDSM. Please also note that this dataset (DDSM+) has also been used for some recent works (Zheng et al. (2021): Fulton et al. (202I)).
Table 3: Description of datasets for classification tasks.
Dataset
# of images
Imbalance Ratio
CATvsDOG (binary)
20,000
2
1%, 10%, 30%
CIFARIO (binary)
50,000
2
1%, 10%, 30%
CIFAR 100 (hinary)
50,000
2
1%, 10%, 30%
STLIO (binary)
5,000
2
1%, 10%, 30%
CheXpert
223,416
2
12.2%. 32.2%. 6.8%. 31.2%, 40.3%
Melanoma
33.126
2
1.76%
DDSM+
55,000
2
13%
PatchCam
294.912
2
1%
CIFARIO (LT)
50,000
10
1%. 10%
CIFARIO0 (LT)
50,000
100
1%. 10%
STLIO (LT)
5,000
10
1%, 10%
ImageNet (LT)
115,800
1000
0.39%
A.2 TRAINING CONFIGURATIONS
All benchmark datasets are experimented by NVIDIA GTX-2080Ti and four medical datasets, i... CheXpert, Melanoma, DDSM+ and PatchCam, are experimented by NVIDIA V100. For the datasets of binary classification in Table we use the dataloaders from (Yuan et al. 2020). For the long-tailed datasets of multi-class classification as in Section D we uses the dataloaders from (Cui eet al. 2019). For TS-DRW, we train the models using cross-entropy loss at the first stage and then switch to imbalanced losses at the later stages. For TS-DEC, we first conduct the regular training using cross-entropy loss and same settings and then we discard the trained classifiers and finetune the new classifier for additional 10 epochs using imbalanced loss with the learning rate of 0.01. For the proposed compositional training methods, we consider the non-adaptive version with he(-) in all experiments. For all datasets, we use the train/val split to do cross-validation for parameter tuning, except CheXpert as explained below. For the benchmark datasets, we use 19k/1k, 45k/5k, 45k/5k. 4k/1k training/validation split on CatvsDog, CIFAR10. 10. CIFAR100, STL10, respectively. For melanoma dataset, we use 70/10/20 split for train/val/test. For PatchCam, we use their official validation set for tuning parameters, which includes about 37k images with balanced positive and negative samples. For DDSM+, we tune the parameters on 10% data sampled from the training set. For CheXpert, since the official testing set is not released and it will take a long time to evaluate all methods on the official testing data, hence, we evaluate different methods only based on the official validation set with parameters tuned according to this set. To make the experiment on Chexpert con-
15

---

Published as a conference paper at ICLR 2022
sistent with other datasets, experiments for all methods CheXpert by following the same cross- coossvvaliatioon procedure, i.c., by sampling 10% training data based on patient ID as the validation set to tune parameters and then report the average scores of five diseases the testing set (i.e., the official validation set as a testing set).
A.3 EVALUATIONS ON MEDICAL DATASET WITH MULTIPLE RUNS
For medical datasets, rum all experiments and report the average performance over three runs. We use batch size of 32 except for PatchCam that is 64, initial learning rate of 0.1 and weight decay of le-5. We train Melanoma for 12 epochs, CheXpert for epochs, DDSM+ for 5 epochs and PatchCam for 5 epochs. The learning rate is decayed at 50%. 75% of total training iterations by 10 times. For compositional training, we tune the number of inner gradient steps in & E (1, and also tune a e {0.1, 0.05, 0.01} for the inner steps. The results are summarized in the following table.
Table 4: Testing performance medical datasets.
Melanoma
CheXpert
DDSM+
PatchCam
Method
AUC
CE
0.87920.008
0.892+0.001
0,949=0.00)
0.869 000
AUC
0.868=0.008 006
0.899+0.002 002
1.99=00.001
0.86840.006
AUC-CE
0.880=0.005
0.902+0.00 002
0.957=0.001
0.868+0.005
TS-DRW
0.878+0.007
0.900 002
0.942+0.003
0.86740.008
TS-DEC
0.877+0.005
0.897+0.001
0,941#0.001
0.86940.00 1099
CT (AUC)
0.900=0.002
0.909=0.003
0.981=0.001
0.891+0.003
A.4 TESTING CONVERGENCE CURVES.
The convergence curves of testing AUC on the benchmark datasets of different methods are plotted in Figure
CAT DOG
CIFAR1S (10%)
CIFAR102 (10%)
STL,10 (10%)
0.9
0.9
0.70
0.8
AUC
Toun AUG
0.8
Tea AUC
0.8
0.7
05
CE AUD-CE CTUNUR
0F
0.7
AUC-CE CHIAAR
C (AUG
0.5
AUC-CE
01/00))
100
25
50 Epeach
100
25
50
100
Epoch
Spach
Esaadh
Figure 5: Convergence curves of testing AUC on four benchmark datasets with imbalance ratio of 10%.
A.5 COMPARISON WITH THE SAME RUNTIME
To compare the performance with the same running time, we train ResNet20 on four benchmark datasets with imbalance ratio of 10% GTX-208011 For each method, train 1000 seconds and report the best achieved testing AUC. The results are summarized in Table 5
Table 5: Achieved testing AUC for each method after training ResNet20 for 1000 seconds.
Dataset
CT (AUC)
AUC
AUC-CE
TS-DRW
TS-DEC
CATvsDOG (10%)
0.944
0.925
0.915
0.943
0.917
0.917
CIFARIO (10%)
0.936
0.900
0.905
0.928
0.897
0.898
STLIO (10%)
0.837
0.815
0.783
0.829
0.816
0.815
CIFARTOD (10%)
0.724
0.691
0.701
0.718
0.696
0.696
A.6 EVOLUTION OF DIFFERENT TERMS OF THE COMPOSITIONAL OBJECTIVE.
To better understand the proposed compositional objective, we plot the evolution curves of each term in the decomposition equation (5) and compare them with that of naive linear combination approach
16

---

Published as a conference paper at ICLR 2022
(Linear Comb.). We conduct experiments CATvsDOG, CIFARIO, CIFARI00 and STL10 with imbalance ratio of 10% using ResNet20. We start with initial learning rate 0.1 and decay it at 50th, 75th epoch by 0.1. In F we plot the values of Lauc(w). LAvG (w), V (w V (Lwa(W) LVv w) w)]? v.s. the number of epochs on training set. For the calculations of (w)) we compute its values based the optimal values of a, b,b, according epoch. Regarding the calculations of w, we compute the mean values of all layers of models. We defer results on other datasets appendix. From the results, we observe that initially Lwuc(w) dominate the objective and keep decreasing at earlier iterations. When it reaches similar level of the third term [VV the objective shifts its focus on third term and pushes it smaller while maintaining second term positive. Eventually, we can see that both AUC and CE losses of CT method reach level close to zero. addition, comparing CT with the linear combination method, it is amazing to see that CT drives both the CE loss and the AUC loss decrease faster and to a smaller level than the linear combination method.
(Yuan al., 2020) for each
Lauele)
Luv
Playo
0.02-
2.0
Liwer Combs
0.3
Liwar
0.2
1.5
0.1
0.5
0.0
25
0.1
100
0.0
40
1.0
20
Epoch
Lace
Layg/w)
0.08
Lnnon Oonto
0.3
Larcer Gomo
0:25
0.5
Cf
0.20
0.0
0.2
0.15
0.4
0.10
0.00
0.0
20
60
100
40
20
170
75
100
Epech
Epoch
Epoch
Epoch
Louclar
Laus(M)
Liwer Corn.
CT
1.5
0.25
0.4
1.0
0.20
0.15
0.2
0.5
0.10
25
75
100
20
50
100
0.05
0
23
50
150
20
40
60
60
100
Epoch
Epoch
Lage(w)
Layg(M) (w)
Wlacele WL (Launiw)
0.08
Our
0.6
0.30
Duna
0.100
0.08
0.4
0.20
0.0:0
0.075
0.2
0.15
0.025
0.10
0.000
0.0
20
0.05
Epoch
Epoch
Esoch
Epoch
Figure 6: Evolution of each term of compositional objective function CATvsDog, CIFARIO,
STL10 and CIFARI00 datasets (from top to bottom).
B IMPLEMENTATION OF 72,111 ho(Oo, O1,..., O.)
The h, is usually implemented by recursion, where examples are given in Table Similar to Guo ct al. 202T). we make the following assumption for our analysis.
Assumption 1. For the Adan-style algorithmms in Table 6 we assume that s; 1/(222,041 + Go) upper bounded and lower bounded, i.e., there exists a Cu such that Vi, a S ||sc, S C.. where $ denotes the i-th element of St-
Remark: Different implementations of by that satisfy this assumption is presented in Table
We need the following lemma to tackle the variance recursion.
Lemma (Lemma 2, Ghadimi & Wang 2018)). Consider a moving average sequence Ze+1 = (1- 8)z, (,,O((x (x:) for tracking h(x:), where E. (O)(x:)) = h(x,)) and h is T.-Lipschitz continuous
17

---

Published as a conference paper at ICLR 2022
Table 6: Different Adam-style methods and their satisfactions of Assumption
method
update for fr
Additional assumption
Cy and Cu
SHB
hx(-) 1,G=
q L,
Adam
22,011 = (1 81)22, + 18,0;
10,1 SG
q G466 Cu
Go
12,41 = (1 8,)22, + B8,0;
AMSGrad
22,++1 = max 122,111)
10,1100 S
q 0+66 Cu
Go
AdaFom
(AdaGrad)
22,1+1 [+1
_ 2i= Of
1001100 S G
C G406
Cu
Go
2 211 = (1 22 + 3,0,
Adabound
72,111 (1/e2/11// 22,1111
Go =
4 = Cr = Co
mapping.
Then we have
L L?]|x Xx-1111 111
Ee Ee|lz(+1
( 8.)lzz h(xx-1)11 (3/ - h(x;)11
(1-
Bg.
(8)
where E, denotes the expectation conditioned on all randonness before Ox(xx).
C PROOF OF THBOREM 1
Denote 7/ = S/, where s; = 1/(122,14 + Go). We make the following assumptions regarding
problem 6
Assumption 2.
Lwvo(W)) is CLac Lipschit continuous, 91 (w) is Cy, -Lipschit continuous and g2(w) is Caz- Lipschitz continuous.
42 VLLG is - continuous, Vgi (w) Ty -Lipschitz continuous, and V (W) Tuga -Lipschitz continuous.
V LAVG (w) - aV w: S)Il2 a², 2l (w) LVVLV a².
E V (w) Vai (w; S)
2 a² Bllgi(w) 9i (w; 5o B. V22(w) V92(w; I o².
a².
g2 -
|)
S2 a bounded convex set with radius D.
notable that the last assumption can be replaced by a condition that the dual variables 0, are bounded.
We also know that 93 (B) is = 2p(1 pt-strnygly convex and also Lgg smsothhh. De- note ("(w) arg max ((w, 0). Based on Assumption 2 we have that h (w) is (Cp. = 1 + AENt aCtwa a Liipschitz continuous, Vh(w) is (Ln, 1 aL L J Vh(w: < (1+ D2)a,, and IE ([() - h(w: S)1l < (1+ DD) We also know that F F(w) is
Lipsshhit continuous, E/|Vh( W)
L--soohhh, where LF = (Lm, + DL + A) + (Lo, +D +A)" (Lemma 4.3 of (Lin et al. 2019
x
We present Theorem T formally in the following Theorem.
Theorem 2. Assume F(Wo) F. I AF where F. min F(w). Suppose Assumptions
and hold. With Go = O(1/VT). B1 = 0(1/VT). So = O(1/VT), &r = O(1/VT), 81 2XC, +1200 2C,, Lc and 1/2 0(1/V)) where Ci and C3
m min! C.C.
BoA
proper constants specified in the proof, Algorithm 7 can ensure that
E
IVF(
00
T+1
To prove this theorem, first need couple of lemmas.
18

---

Published as a conference paper at ICLR 2022
Lemma 2. Suppose Assumption 1 and Assumption 2 hold. Considering the update in Algorithm 7 we have
F(we+1) & F(W.) + Ca
2 19 (h(w,)) + ( zt+1l
IVF((.))
71 1141112
2
Proof of Lemma 2 Due the smoothness of F, prove that under mn S (2(()
LF
F(W++1) - F F(w,) VF((.)
(w++1 w.)
W:+1 w.ll?
2
LF
=F(w,) VF (w:) (7h o Ze+1) +
|1 1c+1/l2
2
lI o V + (_ ( 11 o 2e+1 - - - 2c+1l13)
F(w.)
V
Th (VF(W,) 2+11)
2
2
2
S F(ww))
11 Cu
n&l (h( + (W 2++1l?
2
1/1 c LF 1/
|VF((,) Wl ||
+
|
2
=((w))
Vngi O1 21-111-- 710
VF
VF
(W:()))l2
1)
Z 11
2
4
Lemma 3. Let 01+1 = 11, + 7/2 72 US2) gg))(.))), we have
1/2
110++1 - -0((((w++))) 5(1
Bl|0 (h(w(w)))) 21/00²
2
(10)
167/2
4L3
C Ellue+1 - h(w.)l1
Bl((w,)) - h(We+1)]1
where La = Coz X x
is the Lipschitz continuous constant of (.).
Proof of Lemma 3
Since 0 (h(w.)) Th TT((r(((w))) + =a(02(((w)))) - (Ms((())))))) we
have
El - 8" (h(w.))
= =B||Iaa + (
[0] - 1 US) [ (ooe(((()))))T
-
[] - ( S1 US2) - V9a(O.)) - -1292(u441) +292(
[0*(h(w.)) (2)
Vga(0" (M(((((()))))l
- (1 (g)( - [0* (h(w:)) ( (20())(((w))))
+ ( U - 92 (1
< [0 - (2 (4/11) (o,().)) - [0 (h(w:)) + (22((((().)
+ 72200
(11)
where
El ||0p::100hhhaooooo00000000000000 (M
= =E 8* (h(((w))) -| Vgs (Kww)))
+ 21z - (1 (0 (0(((())))))
Vgs
0F ('(h(w.)))
Bl - R 00" (1 (w:)))
(12)
+ 72d El10, 0" ('(WWw)))1
41p 2 1 - 21 - - (h(w2))
812 2 E u E A(
$ (1 - 0 - 8 (1(WW)))
+
x
19

---

Published as a conference paper at ICLR 2022
where the first inequality uses strong monotone inequality as 9s(-) is X-strongly convex and the second incquality uses 72 min 813 A.. Then,
Ell0(41 - 8 (h(Ww++))11
2 $ (1 + 12> [B|00++1 - 0* (-((ww)))l +11 + mal (=I(0 - 0" (h(we++))11
2
S (1 + 12
(1 122)E110 - ('(wwW)) + (1 +
ned (1220² + 872 Co Ellect h(w.)ll)
2
2
+ (1 + 1/2
|B|10 (l(w.)) - (1(ww(1))]
(1 + 1/2 (1 7-AlEl1o. ("(((w:))ll' 22 1672 Cg Elluett h(w:)ll²
2
4L3
+ y
Ellb( (.) - h(w++1)11
112> El|0, - 0" (k(wa)))1 2n20² + 167
4L3
$ (1
C El 1/luu11-10 - l(w.))] + 1/21
E Ellk(we)) - h(We+1))??
2
x
where the third inequality is because that (.) is L.g = Cp2 -Lisschtt Lin et al. (019))
x
Proof of Theorem 2. Denote by 91 91(h(w))) = Ex, (9 xx. U:)] and g2(h(we))
Exc.r 192 (h(w,)) x,, v:)l- Note that V (h,,) = Vngi (h(w,) + 0g V2 ()).)(((w..)..
De-
note by Aud || 112, Az, |z++1 Vwh((w)) (h 02) |z0+1
<)(...) Nagi (h (w:) 0, ( and 6g 10. 0r ()((w)))
Applying Lemma to Uy, we have
B[Au,r+1] S (1 20) + 2880² +
h
Wi+1 - well?
(13)
Bo
Hence we have
Du.t Au,tt1
c2 Z++1 1
E
E
+ +28002 (T+
(14)
Bo
8
and
T+1
Au,t41 +28002 + 1) +
Ch 2 v
2++1 12
E
u,
E
u,t
(15)
(=0
t=0
Define
e: = (1- - (ww((w.)) Vv (((ww))) - Vw/i T (h( (Ww-)()))
We have
1 2(1 - (1), +f
-w. 100 (b(w))) - 8" ((h((w-1))11))
and
El|z:11 - Vwh(w.) V. (M(m)))))))
< [1 - 31)l2. - Voh(w..1) (h
+ 31 9;
04:
+ 3: (wh((w)) Vol /
TV, (H
< [1 - 8 - /t
+ 9, +
+ - V
(16)
< (1 + 8
- 111 11
2
2
+ + 31
2
+ +1+ ( V ) (b(()))))
81
+ V (V - -wh(w.) (
20

---

Published as a conference paper at ICLR 2022
where the last three terms can be bounded below. First,
B V Vwh(w (w)) V (u O²) - Vwh(w:) (V, (V,)(((w))))))
(17)
20 m + 1 EB| u1+1 h(w.)ll?.
Second,
0 V (w)) T V Vn 02) V (V,((((w)), (h(w:)))
(18)
1 CR 92 B| 0* (h.
Third,
w/ S1) V (u , 02) Vwh(ww.) V 0t)1
(19)
I 80 (Ch D² 0²) + 40 D2o²) 40 (302 + =3D²Ch + 202))
It follows that
[c+1 Vwh(ww)) V1. (W)II)
31 |E| |||11111 - Vwl((w.)) V. exll + (1+ +
2
$ (1 +
3 lexll
2
S (1 + $1 ( - 11))(112 - V.
2
+ 333 (L,, + L Elllet+ - 165 0 (1
(20)
+
+ 3D'C + - le:1
S1 - El|
)] + + h(wa)l
C3 15 W. wr-1
+ 11 0" (I(())))1 880
81
where C. = 320 (12) 01
+ +L2 ) 80 (3C +20
and C3 == 8I(C2, + +D'C2 1R
CL 91
L2 g2 )}.
Thus.
A..
T+1
T+1 Σ t=1
Cavie 2
A
<E
+
C1
Du,t + 16
de +
31T + 1)+
| 1
31
3
Lt=0
t=1
Using Lemma
we have
Σ
T
6,
2
1o
o" (A(wo)))1
4920 ( +
32
E m. E[A,..]
8LECRni c 1/22
T
|22++11l 11:
1/1
A
X
t=
and
T+1
8
S
2 yen
F.100 - 8" (h(wo)))l + 4120 ( + 1)
+
32
2
Σ
+
8L 2 x2
2
11
x2
92
1=0
2,11 E, Autt E, 8z and Lemma Combining the upper bound of E. A2,1
we have
T
VF(()))
B(F(w::) - F(We+1)) +
Cu
El 124+111 11
2
1=0
1=
2B (woo) - -Fa)
1
+ Sg
A..0
+ 0
Au. + 16
Σ
6: + 31 +
Canic 87
2141 1|
C/1
2 1=
C
31
2R(F(ww) F.)
+
Cu g
42,0
32
du + 31 C2(T + 1)
64no 2 (T 1)
+ C1
T+1
Au, +
512
c2
Σ Au,i
B1
deN
X2
1=0
B
Cu
Can
Cu
2
||z++1|| 2
87
1=0
2B(F(wo) - F.)
+ CR
A
3260
+ (C1 +
512C2
52
An.
7/
B1
72d
X²
3o
c. (T+
E 3102+ + 64m207
51202
+
+((11 +
1000²
x²
T
E
On
Car c
(C1
+
512C x²
Chn 2
)
1
||a0+z112
B
2
21

---

Published as a conference paper at ICLR 2022
Due to the setting
81
B0x
M1 S minf V C3 2 c2(C 512C
(21)
20h²
92
we have
Cu
Canice Co
(C1+
+
5120
92
1
1 0.
(22)
39
X
$
2
Hence, we have
E
(x()))
V
2T((F(W))
F.)
Cu
E
A
+
3200 A2r
+ (C1 +
5120
92
Au,0"
T
neT
aT
81
X²
9u
+
Cu
81O2 + 64m20²
+ +2(C +
5120
02
)00
a
A2
With 8 = = =1000/==0V00 and 1/2 O(L/VT), we have
|V s 00 1
E
7+1
VT
Table 7: Testing performance on benchmark datasets and ImageNet-LT. The percentage number is the second row denotes the imbalanced ratio (cf the text). All experiments on benehmark datasets are averaged over three runs with different random seeds. The network structure used in all experiments ResNet32.
For Accuracy Maximization
Datasets
Method
1%
10%
0.713220001
0.87620:002
CIFARIO (LT)
LDAM
Cao al.
2019
0.744x0.000
0.872=0.002
TS-DRW TS-DEC Kang
Cao et m..
2019
0.780+0.00%
0.879+0.000
2019
0.758+0.016
0,842=0:004
CT (CH-LDAM)
0.787*0.001
0.883+0.001
CE
0,396
0.572=0.000
LDAM
Can
2019
0.407
0.559
CIFARTOO (L.T)
TS-DRW TS-DEC
Coo et m.
2019
0.427=0.000
0.579=0.00
Kang cl al 2019
0.40340.003
0.536a0.001
CT
0,43020.005
0.585=0.002
CE
0.411=000117
0.63920.009
STLIO
LDAM
Cao ct al.
2019
0.440=0.010
0.641 008
TS-DRW
Cao
2019
0.458+0.006
0.651
TS-DEC
Kang et 2019
0,457=0.01
0.629=0.009
0.48840.012
0.662#0.005
Jamal 2020
0.2526
ImageNet (LTT
CB CE Cut et al 2019
0,2659
CT CCBC-CC
0.2661
D COMPOSITIONAL TRAINING WITH CLASS WEIGHTED LOSS
In this section, we extend the compositional training method to deep learning with class weighted loss. Let (ew(w) denote class weighted loss written as:
1 Σ Pv. ((w:x,,)) 1=1
Lew (w)
(23)
where Pv: (0, 1) denotes a weight assigned to the i-th example that depends on the class the data belongs to. There are different methods for determining the class-level weight. A simple method to set p; according to the reciprocal of its corresponding class size, i.e., Py. 1/ny.- Recently, Cui et al. (2019) proposed an improved variant of class- waishhee loss by using an effective number of samples per-class instead of the class size to compute the individual weight, i.c., Pv. = 1,7,
where ? (0. 1) is hyper- parameter. We refer the class weighted loss using these individual weights Ten (w) 1/n Ei 1-7,, ((w:x; X;, y:).
22

---

Published as a conference paper at ICLR 2022
With Class- Weighted Losses. The corresponding compositional objective is a standard two-level compositional function, i.e.,
min F(w) Lew(w a (VV.vV
(24)
WERd
Assuming that the stochastic gradient of Lew can be easily computed, the optimization of the above problem is easier than that for AUC loss.
We present a simplified stochastic adaptive algorithm with an Adam-style adaptive step size in Algo- rithm 2reffrrred to as SCA, where hz O₂) denotes an appropriate mapping function, which can be implemented by using different methods, including Adam, AMSGrad, Adabound, etc. Notice that when (-) Algorithm becomes the NASA algorithm (Ghadimi et al. 2020) to stochastic compositional optimization. We present an informal convergence of Algorithm 2 beloo.
Theorem 3. (Informal) Under appropriate conditions on the loss functions LAvG. Lew and Ei= [IFF()))) s
y(); with Bo, 31.7 ( Algorithm 2 ensures that TE
0 O
Remark: The appropriate conditions include the Lipschitz continuous conditions on LAva and Lew and bounded variance conditions V X;,;;;;;)) and V²((w; x,, We will present the detailed conditions below when proving the above theorem. The above theorem indicates that we can optimize the compositional objective (24) with the same convergence rate as optimizing the averaged loss ( for deep learning.
D.1 EXPERIMENTS WITH MULTI-CLAST DATASETS
We conduct experiments on three benchmark multi-class image classification datasets, namely CI- FARIO, CIFAR100 and STL10. We construct imbalanced versions of these datasets by keep their classes but making the class sizes follow long-tailed (LT) distribution with two imbalanced ratios (the ratio of the size of minority class to the size of majority class) similar (Cui et al.) 2019). We use ResNet32 as the network stucture. The weight decay is set to 2c-4 for all experiments. For algorithms, we train total of 200 epochs with a batch size 128 and we use step size 0.1 and de- crease it by 10 times at at 80% and 90% of total training time. We tune the beta parameters of our methods range [0.1, 0.99] with grid search and find that good values are around 0.9. For the class- weighted loss, we choose the class-weighted version of the LDAM loss (Cao et al. (219)). For baselines, we compare with optimizing the CE loss (CE), optimizing the LDAM loss (LDAM), the two-stage method with the deferred re-weighting that optimizes the class balanced LDAM loss in the second stage (TS-DRW) (Cao et al. 2019), the two-stage method with the decoupling trick that optimizes the class balanced LDAM loss in the second stage (TS-DEC) (Kang al.) 2019). The results are shown in Table / on all datasets. In addition, we conduct a large-scale experiment by following Jamal et al.) (2020) on ImageNet-LT with ResNet32. We use class-balande of our compositional objective. We use an initial learning rate of 0.1 and run a total of 90 epochs decaying learning rate every 35 epochs by factor of 10. Eventually, we achieve the topl accuracy of 26.61 %, which is better than two baselines, i.e., CE(25.26%) and CBCE((6.59%).
We can see that the proposed CT method achieves the best accuracy
loss (Cui et al., 2019) as outer loss function
D.2 ANALYSIS OF THEOREM
In the section we analyze Algorithm An algorithm utilizing the adaptive step size and moving average for nonconvex optimization has been studied in (Guo et al.] 2021). But here we have to tailor the algorithm and analysis to the considered formulation with composition. Denote s, 1 1/(22,41 + Go), 7h = 7s,, We make the following assumptions regarding the problem 24 Assumption 3.
* Lcw(u) is Cie continnuus, V Lww(ww) is - continuous.
Lcw((u) is LLcw Lipschiitz continuous, 2 Lwve(w) is L Lipsscitzz continuous.
The stochastic oracle satisfies V L (w; S)1? 1 0², Blla a LAvo(w; S)1l 2 o², E (Lww(u) - (u: S)1 < o²
We formally present Theorem 3 as follows
23

---

Published as a conference paper at ICLR 2022
Algorithm 2 Stochastic Compositional Adaptive (SCA) method for solving (24)
Require Parameters: 5o. B1, a, Go. 17
2: Initialization: Wo E IR" Ug
3: for 0, ..... do
4: Sample three sets of examples denoted by S,,S,
5: U = (1 86) + 3o( W. -v LavG. (w; Si))
6: O, = (I - a Lava (Wr; V Lew((u
7:
Zt-1 = (1 31) ) 3,0,
8:
22,1+1 ho (0 ................................ Oi)
ohe can be implemented by that in Appendix B
9: Wi+1 = W. 1l 72.
2+-1
owith the simplest form h4
10: end for
Theorem 4. Assume F(xo) F. S AF where F. min F(x). Suppose Assumptions 7 and
hold. With v { Vais Vais
Ve,30
rr fo Bo 0 (7) 81 $((7T) and constants
/ 4(1+002m0 Cant 20,Lo
LF L. (1 + aC Lova) acl Cs (AL Clew +2 aoL Algorithm
can ensure that
T
E
/VF(w))1
00( {0( [T)
T+1
we know that F(w) is smooth with coefficient LF == 2L (1 + aCtwa) 2 Lw a We can prove that under 7Le I c/(2)),
Proof of Theorem
4 By Assumption
LE
F(w++1) - F(w.) + VF(w.) (Wi+1 - w.) + 2
||e+1 w.ll?
=F(w.) VF(w:)) (me z.+1) + 1. I17e o zet1ll
LF
2
=F(w:) + 21lvmn (VF(w.) 2111) _ 2llv'e VF(w)) + bx Il 2 2 zllvm 2241113)
- F(::) n2uu /V((::) 2+1
1701 [VF(W))]]
rL ne
|2++1 /|?
2
< F(w:) + T/cu [VF(w:) 2011 1761
77 [IFF(::)]
||
2
4
(25)
Denote Az,t = |z41 FF 2:41 (1 a 2 L -aa-aaVVL LAVVVG(W;)
and A,, = |u441 (w; V LAvG(W:)))l
Applying Lemmal to Ug have
Bo) +280022 + (1 + aCLmc)
T T[A,, < (1
1| - will. WWe11-W112
(26)
9o
Hence we have
T An,t A..41 2T
(1+ aCowvo
) cZ 22
Σ u.t
S R
Σ
(27)
80
83
Defining
=(1 3:)(VF(w.)) -VFFF()))VV(..-I)))
=(1 8.)((T - (ow
Vi (we))
(28)
- I1 - LAvG (We-1 Vian (Wa-1) })],
we get
| < (1 -31 - -W
(29)
24

---

Published as a conference paper at ICLR 2022
It holds that
E||z++1 (..(w)) + e.ll - -V
+ (i av2 ( F(w)))1?
=El((1 B:)(z: VF((.-1)
+1(1(T - (.w (u V LV (w WWw.:w/WWLWWLW.... [L.NG(W.)]]l
+ (1 av
= 81) 112 VF(W
+ (1 [|
a
-2(1 - ()) VF((:-1))
-2(1 - ()) ||
[[[(1 8 vF(
+ + EI(T a 3
+ 2 EI[T aV LAVe ( - -V Lavv((
31
+1-
E Ez - V/
2
+2 aV²
LAva Lova W.w.))/)LLwwuu. (I - - L a LAva(ww))
(1--3.)
$1
|B||z VF( + +4((1 + aLLwa)* + + CZem)
2
+111(1 + =.I.wa )2 |1++1 (w: - -wecw(
81
- (1-)3)((1
}Ble - WF(( + f. + C...D....
2
where C2 1((1 + aLLww)) )2 + 0² +cecm) C3 4(1 ab ) 22
It follows that
| VF(w:))1 (1
- - (FFFF(()+s))) + +e:
+ (1+
3/2 |ee
2
3r
<1 21 )1 B1)(1
31
EBle VF(W (1+ Cr,
81
(1+ 31 10, u.t
(1-133) 1-311)
ww - 1/
(30)
31
< (1- - BI) - VF(ww- (0oo-)))l++220000 20"
+ A + 127l w.-1
31
Thus,
A A.,41 (C, +)+
4L2
E
3 q c 1211l 2CB Σ Au,!
31
T
Az,41
4L2
S
E
+ 20,33000 (T +1)
" c
|141 2
B1
B
1=0
T
Au,t Autt1 - 2800 2 (T + 1) +
(1 + aCtam
1 22
74
+2C₁E
30
88
t=0
(31)
25

---

Published as a conference paper at ICLR 2022
Combining this with
25 obtain
c
E
s F(wo) (W(w))l
F Σ
n1
ze+1
11
+
Σ
A
2.1
4
2
1=0
1=0
< F +
17
Az.t11
+ +20g
Au,t
uu,t11
2
B1
30
(32)
+
ne
[204 + (T (T++))
2
[4 2
21 + Lwn 2212c
T
+
?Ca
170
Σ
2
B7
4
t=
Due to the setting
(33)
ALFV T
4(1 + aCt.wa )v Coa
have
122 c2
2C + aCT.wo ) )21122 c2
1jCu
1/0
S.
(34)
2
4
Thus,
E
T
VF w. )
2DF
+
Cu
| .
+ 20s
18 E u,
+
Gu
120 4Cs
T +
naT
a
3,T
30T
C1
t=0
(35)
With Bo = 0 )1 = 0
and f = 0 }
T
50 1
E
VF w )
(36)
T
t=
26


Reviews:
[{"review_id": "s0na1MTkoYu", "reviewer": "Reviewer_LxEz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper provides good explanation on their newly proposed compositional loss and training method. \n2. The paper performs extensive experiments to demonstrate the effectiveness of their method.\n3. This paper is well written and organized. \n\nWeaknesses:\nOverall, this paper is a good work. But I do have several questions and concerns:\n1. The author hypotheses that using CE loss, different examples roughly have equal weights regardless which classes they belong to. And it is one of the key reasons that using CE loss has advantage over raw AUC loss. It would be great to show that the compositional loss can help achieve the equal weights property empirically to validate this hypothesis in the paper.\n2.  In the experiments section, the authors list AUC for multi-class classification tasks (e.g. CIFAR10 and CIFAR100), I wonder how the AUC is computed for these multi-class tasks. The details are missing in the paper.\n3. For medical image datasets, since the authors only report the results from a single run, it may not be very convincing to show the improvement. I would suggest the authors either report results from multiple runs or report the evaluation noise level (i.e train same model several times and perform evaluation on the test data, AA test to understand the eval noise), so that we can make sure the improvement comes from the proposed method.\n4. In figure 4, there are several bump ups in the convergence curves. What are the reasons of these bumps? Are they because a learning rate scheduling or something else? The detailed explanations are missing.\n5. In table 2, the authors mention that they tune the inner gradient steps k \\in {1,2,3} for the left table while keep k = 1 for the right table. However, the detailed tuning strategy is missing.  Does the algorithm tries out 1,2,3 and pick up the optimal one automatically during each iteration?\n6. In the runtime analysis, it would also be useful to report what hardware is used for testing, since different hardware may results different running time.\n", "summary_of_the_paper": "This work proposes a new compositional loss function that aims to help deep AUC maximization. Specifically, the outer function of the newly proposed loss function is a surrogate loss for AUC, i.e. AUC square loss; while the inner component can be understood as to facilitate the convergence of CE loss. The authors explain their compositional loss through Taylor expansion theoretically and the lens of feature purification and classifier robustification intuitively. Furthermore, they demonstrate the difference between their proposal and the linear combination of AUC and CE loss from both mathematical perspective and empirical perspective. Experimental wise, this paper has done extensive studies on different dataset ranging from popular natural image benchmarks to several medical image datasets. Though the improvement is rather limited on some cases, CT(AUC) outperforms all the other methods in all cases. ", "summary_of_the_review": "Overall, this paper is well written and explained. I have listed several questions and concerns in the Main Review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_lcOuB7Uu_R", "reviewer": "Reviewer_qWty", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper includes a good summary of the related work and the contributions. The proposed compositional approach to the deep AUC maximisation problem seems interesting and novel. It uses a reformulation of the optimised loss function that contains three different terms which jointly optimize the feature representation and the classification performance. A primal-dual stochastic algorithm is proposed that optimises the proposed cost function and an informal argument for its convergence is given. The reported experimental results are quite convincing, both for the computer vision tasks and for the medical imaging tasks, however a proper comparison of the experimental results with other state-of-art methods is missing.\n", "summary_of_the_paper": "The paper addresses the problem of how to perform end-to-end training for deep AUC (\"area under ROC curve\") maximisation. This is a challenging problem as this yields a harder optimisation problem than the standard CE minimisation. Existing approaches use two stage optimisation approaches where the first stage optimises a traditional loss function (e.g. the CE loss) and the second stage fine-tunes the network by optimising the AUC loss. The proposed approach is termed compositional deep AUC maximisation which can be trained end-to-end. The paper furthermore proposes an efficient stochastic optimisation algorithm solving compositional deep AUC maximisation problem. It has been evaluated on standard benchmark computer vision and medical imaging classification datasets.", "summary_of_the_review": "This is an interesting paper that proposed a number of technical advances (formulation of the compositional loss function and its optimization) with reasonable results, albeit there are weaknesses in the experimental evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XSMsw3GPPX2", "reviewer": "Reviewer_TF22", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I cannot speak to the proposed optimization approach, I feel the following are the major strengths and weaknesses of the paper:\n\nStrengths:\n\n- The authors present the first end-to-end training framework for AUC maximization (previous approaches required pretraining e.g. using standard CE loss first for good results)\n\n- The authors present an optimization method for the new loss that has similar rate of convergence as standard SGD\n\n- Experiments are presented using a large number of natural and medical image public datasets.\n\n- Generally the results show consistent improvement over AUC / CE - only loss, a standard linear combination of AUC + CE losses, and 2 other methods for maximizing AUC loss.\n\nWeaknesses:\n\n- My primary concern is some missing details in the experiments, which makes it difficult to accept the results at face value. While some of the datasets do mention a separation of training/validation/test, several datasets only discuss having either only validation or test set (eg. 3 out of the 4 medical datasets). Thus, it is unclear how the parameters for the different methods are tuned - even though all the methods are tuned on the same dataset partitions, if tuned on the set used for reporting results, this will not capture the generalization capability of the methods and thus it is difficult to assess which is really best. Please clarify the parameter tuning process.\n\n- I am confused about the DDSM+ dataset, where the authors state they have combined the DDSM with CBIS-DDSM datasets. This doesn't make sense to me, as my understanding is that CBIS-DDSM is an updated/standardized version of the DDSM, i.e. CBIS-DDSM is derived from the DDSM and contains the same patient data (also, the authors do not have the correct citation for CBIS-DDSM, which should be: Lee et al., A curated mammography data set for use in computer-aided detection and diagnosis research, 2017). Thus depending on how the data is partitioned, the same patient data will be in both training/testing sets, which may partially explain the extremely high AUC results. Furthermore, I am not sure how there are so many images (55K training and ~14K testing), as each dataset has approximately 2600 patients (again, the same patients), where each should have ~4 associated mammogram images (left/right and MLO/CC views). Please clarify the use of the mammography datasets.\n\n- The authors test on all the same datasets as those presented in the paper by Yuan et al., 2020, which introduced the AUCM loss. However, there are no comparisons made to the results from this paper, which seem to report higher AUC results for many of the datasets (eg. CheXpert, Melonoma, PatchCam). While the authors compared to AUCM loss, it was for training from scratch, not using the same training framework as proposed in Yuan et al. It would make sense that the authors should be comparing to the results in this paper if directly comparable, or running their own comparison to this method.\n\n- Regarding the ablation studies on algorithmic design choices for beta and k in Sec. 4.1, I feel like we do not learn much new from this, as it simply verifies that it is better to tune hyperparameters than to not tune them. \n\nMinor comments:\n\n- Although understandable, would be good to define what \"imratio\" means \n- The convergence curve results don't really seem to fall under \"ablation\" study, as nothing is being ablated...consider moving to other section or renaming 4.1 for clarity of presentation\n- p.2: \"presentations\" --> representations\n", "summary_of_the_paper": "This paper presents a new loss function for end-to-end training of deep networks for AUC maximization.  The proposed compositional loss combines the standard cross-entropy (CE) loss within the AUC loss such that the AUC is maximized for good classification while the CE portion of the loss helps to learn robust features. An optimization method is presented and is shown to have the same convergence rate as standard SGD. The proposed method is tested on 4 benchmark and 4 medical image datasets and compared against related losses, showing consistent improvement in AUC. \n", "summary_of_the_review": "My recommendation is based on the introduction of the novel end-to-end learning approach for AUC maximization and the presented empirical results on 8 different datasets that show general improvement compared to other AUC maximization / CE approaches. However, my enthusiasm was dampened by the unclear experimental setup for parameter tuning/use of validation sets and lack of comparison to highly related work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend this paper to be accepted. All reviewers are in agreement that this paper is above the bar.
Decision: Accept (Spotlight)