Federated Compositional Deep AUC Maximization
Authors: Xinwen Zhang
Yihan Zhang
Tianbao Yang
Richard Souvenir
Hongchang Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: federated learning
compositional optimization
minimax optimization
AUC maximization

Abstract: Federated learning has attracted increasing attention due to the promise of balancing privacy and large-scale learning; numerous approaches have been proposed. However, most existing approaches focus on problems with balanced data, and prediction performance is far from satisfactory for many real-world applications where the number of samples in different classes is highly imbalanced. To address this challenging problem, we developed a novel federated learning method for imbalanced data by directly optimizing the area under curve (AUC) score. In particular, we formulate the AUC maximization problem as a federated compositional minimax optimization problem, develop a local stochastic compositional gradient descent ascent with momentum algorithm, and provide bounds on the computational and communication complexities of our algorithm. To the best of our knowledge, this is the first work to achieve such favorable theoretical results. Finally, extensive experimental results confirm the efficacy of our method.

Federated Compositional Deep AUC Maximization
Xinwen Zhang Temple University Philadelphia, PA, USA ellenz@temple. edu
Yihan Zhang Temple University Philadelphia. PA, USA
Tianbao Yang Texas A&M University College Station, TX, USA
y 2lalmm.haag0000ttemmee....c. edu
munbao-yanggtamin
Richard Souvenir
Hongchang Gao
Temple University Philadelphia, PA, USA
Temple University Philadelphia, PA, USA
souvenir@temple.. edu
hongchang. gwo@temple edu
Abstract
Federated learning has attracted increasing attention due the promise of balanc- ing privacy and large-scale learning; numerous approaches have been proposed. However, most existing approaches focus problems with balanced data, and prediction performance far from satisfactory for many real-world applications where the number of samples in different classes is highly imbalanced. To address this challenging problem, we developed a novel federated learning method for imbalanced data by directly optimizing the under curve (AUC) score. In particular, we formulate the AUC maximization problem federated compo- sitional minimax optimization problem, develop a local stochastic compositional gradient descent ascent with momentum algorithm, and provide bounds on the computational and communication complexities of our algorithm. To the best of our knowledge, this is the first work to achieve such favorable theoretical results. Finally, extensive experimental results confirm the efficacy of our method.
Introduction
paradigm for training machine learning model across multiple devices without sharing the raw data from each device. Practically, models are trained on each device, and, periodically, model parameters are exchanged between these devices. By not sharing the data itself, federated learning allows private information in the raw data to be preserved to some extent. This property has allowed federated learning to be proposed for numerous real-world computer vision and machine learning tasks.
Federated learning
19 34
Currently, one main drawback of existing federated learning methodologies is the assumption of balanced data, where the number of samples across classes is essentially the same. Most real-world data is imbalanced, even highly imbalanced. For example, in the healthcare domain, it is common to encounter problems where the amount of data from one class (c.g., patients with rare disease) significantly lower than the other class(es), leading to distribution that highly imbalanced. Traditional federated learning methods do not handle such imbalanced data scenarios very well. Specifically, training the classifier typically requires minimizing classification-eres induced loss function (e.g., cross-entree cooss-entropy....ommm As result, the resulting classifier may excel at classifying the majority, while failing to classify the minority.
To handle imbalanced data classification, the most common approach is to train the classifier by optimizing metrics designed for imbalanced data distributions. For instance, under the single-machine
"Equal contributions
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

setting, Ying et al. [35] proposed to train the classifier by maximizing the Area under the ROC curve (AUC) score. Since the AUC score can be affected by performance on both the majority and minority classes, the classifier is less prone to favoring one class above the rest. Later, 8 40 extended this approach to federated learning. However, optimizing the AUC score introduces some new challenges, since maximizing the AUC score requires solving minimax optimization problem, which is more challenging to optimize than conventional minimization problems. More specifically, when the classifier is a deep neural network, recent work [39 has demonstrated empirically that training deep classifier from scratch with the AUC objective function cannot learn discriminative features; the resulting classifier sometimes fails to achieve satisfactory performance. To address this issue, [39] developed a compositional deep AUC maximization model under the single-maching setting, which combines the AUC loss function and the traditional cross- pross-entrppy loss function, leading to stochastic compositional minimax optimization problem. This compositional deep AUC maximization model can learn discriminative features, achieving superior performance over traditional models consistently.
Considering its remarkable performance under the single-machine setting, a natural question is: How can compositional deep AUC maximization model be applied to federated learning? The challenge is that the loss function of the compositional model involves two levels of distributed functions. Moreover, the stochastic compositional gradient is biased estimation of the full gradient. Therefore, on the algorithmic design side, it is unclear what variables should be communicated when estimating the stochastic compositional gradient. On the theoretical analysis side, it is unclear if the convergence rate can achieve the linear speedup with respect to the number of devices in the presence of biased stochastic compositional gradient, two levels of distributed functions, and the minimax structure of the loss function.
To address the aforementioned challenges, in this paper, we developed a novel local stochastic compositional gradient descent ascent with momentum (LocalSCGDAM) algorithm for federated compositional deep AUC maximization. In particular, we demonstrated which variables should be communicated to address the issue of two levels of distributed functions. Moreover, for this nonconvex-strongly-ooo
problem, we established the convergence rate of our algorithm, dis- closing how the communication period and the number of devices affect the computation and communication complexities. Specifically, with theoretical guarantees, the communication period can be large O /K3/4) ) so that our algorithm can achieve (1/VKTT) convergence rate and ( ////4//3/// K3/4) communication complexity, where K is the number of devices and T is the number of iterations. To the best of our knowledge, this is the first work to achieve such favorable theoretical results for the federated compositional minimax problem. Finally, we conduct extensive experiments on multiple image classification benchmark datasets, and the experimental results confirm the efficacy of our algorithm.
In summary, we made the following important contributions in our work.
We developed novel federated optimization algorithm, which enables compositional deep
AUC maximization for federated learning.
We established the theoretical convergence rate of our algorithm, demonstrating how is affected by the communication period and the number of devices.
We conducted extensive experiments on multiple imbalanced benchmark datasets, confirm- ing the efficacy of our algorithm.
2 Related Work
Imbalanced Data Classification. the field of machine learning, there has been a fair amount of work addressing imbalanced data classification. Instead of using conventional cross-entropy loss functions, which are not suitable for imbalanced datasets, optimizing the AUC score has been proposed. For instance, Ying et al. (35) have proposed the minimax loss function to optimize the AUC score for learning linear classifiers. Liu et al. [8] extended this minimax method to deep neural networks and developed the manconvex-stronglyycoo have proposed a compositional training framework for end-to-end deep AUC maximization, which minimizes a compositional loss function, where the outer-level function is AUC loss, and the inner-level function substitutes a gradient descent step for minimizing a traditional loss. Based on empirical results, this approach improved the classification performance by large degree.
loss function. Yuan et al. [39]
To address stochastic minimax optimization problems, there have been a number of diverse efforts launched in recent years. In particular, numerous stochastic gradient descent ascent (SGDA) algo-

---

have been proposed. However, most of them focus on non- conpossiiiii optimization problems. On the other hand, to solve compositional optimization problems. existing 38 7 tends to only focus on the minimization problem. Only two recent works [5] 39 studied how to optimize the compositional minimax optimization problem, but they focused on the single-machine setting.
rithms -22 17
21 33
work 31
Federated Learning. In recent years, federated learning has shown promise with several empirical studies in the field of large-scale deep learning [99 20 26). The FedAvg [19] algorithm has spawned [ (36] designed to address the minimization problem. For instance, by maintaining a local momentum, Yu et al. [36] have provided rigorous theoretical studies for the convergence of the local stochastic gradient descent with momentum (LocalSGDM) algorithm. These algorithms are often applied balanced datasets, and their performance in the imbalanced imbalanced
a number of variants [25] (3
regime is lacking.
To address minimax optimization for federated learning, Deng et al. proposed local stochastic gradient descent ascent (LocalSGDA) algorithms to provably optimize federated minimax problems. However, their theoretical convergence rate was suboptimal and later improved by [27]. However, neither method could achieve a linear speedup with respect to the number of devices. Recently, Sharma al. (23 developed the local stochastic gradient descent ascent with momentum (LocalSGDAM) algorithm, whose convergence rate is able to achieve linear speedup for nonconvex-strongg -ornceo optimization problems. Guo et al. [ proposed and analyzed a communication-effiiin optimization algorithm (CoDA) for the minimax AUC loss function under the assumption of PL- condition, which can also achieve linear speedup, in theory. Yuan al. 40) extended CoDA to hetereogneous data distributions and established its convergence rate. Shen et al. [24] proposed to handle the imbalance issue via introducing a constrained optimization problem and then formulated it as an unconstrained minimax problem. While these algorithms are designed for federated minimax problems, none can deal with the federated compositional minimax problems.
distributed
To handle the compositional optimization problem under the distributed setting, Gao et al. 3 developed the first parallel stochastic compositional gradient descent algorithm and established its convergence rate for nonconvex problems, inspiring many federated learning methods 12, 28 27
in the past few years. For instance, [12 directly used a biased stochastic gradient to do local updates, suffering from large sample and communication complexities. 4 employed the stochastic compositional gradient and the momentum technique, which can achieve much better sample and communication complexities than [12] 7 considered the setting where the inner-level function is distributed on different devices, which shares similar sample and communication complexities as However, all these works restrict their focus on the compositional minimization problem.
It is worth noting that our method is significantly different from the heterogeneous federated learning approaches [4] 29). Specifically, most existing heterogencous federated learning approaches consider a setting where the local distribution is imbalanced but the global distribution balanced. For example. Scaffold [14) method uses the global gradient to correct the local gradient because it assumes the global gradient is computed on a balanced distribution. On the contrary, our work considers setting where both the local and global distributions are imbalanced, which is much more challenging than existing heterogeneous federated learning methods.
3 Preliminaries
In this section, we first introduce the compositional deep AUC maximization model under the single-machine setting and then provide the problem setup in federated learning.
3.1 Compositional Deep AUC Maximization
Training classifiers by optimizing AUC (9, is an effective way to handle highly imbalanced datasets. However. traditional AUC maximization models typically depend on pairwise sample input, limiting the application to large-scale data. Recently, Ying et al. (35) formulated AUC maximization
model as minimax optimization problem, defined as follows:
min max CAUC (w, W1, W2 Tें a, b) & (1 (h((w;)) - w1)² [1 - -p(1 - p) w,01, Wog
- -))((w;a) (1)
+ p(h((;;)) a) 222) 2I + 2(1 +
3

---

where h denotes the classifier parameterized by E R", ūn E R2 E R. di E R are the parameters for measuring AUC score, (a, b) substitutes the sample's feature and label, p the prior probability of the positive class, and IT is an indicator function that takes value if the argument is true and O otherwise. Such minimax objective function decouples the dependence of pairwise samples so that it can be applied to large-scale data.
Since training a deep classifier from scratch with LAUC loss function did not yield satisfactory performance, Yuan et al. [39] developed the compositional deep AUC maximization model, which is defined as follows:
min max LAuc(W, 181, tiz W3; a, b) W,001,022 1ig
s.L.
w -= WWWW-WWCEW(;;
(2)
Here, LCE denotes the cross-entropy loss function, w -V indicates using the gradient descent method to minimize the cross-entropy loss function, where > 0 is the learning rate. Then, for the obtained model parameter w, one can optimize it through optimizing the AUC loss function.
By denoting g(x) = x pA(x) and y W3 where x [w "2]"" A(x) [VwCor( W; a, b),)" 0, 0J and ʃ LauC, Eq. (2)
can be represented as generic compositional
minimax optimization problem as follows:
min max f(g(x),)) XER yER".
(3)
where g is the inner-level function and is the outer-level function. It is worth noting that when f is a nonlinear function, the stochastic gradient regarding x is biased estimation of the full gradient. As such, the stochastic compositional gradient [31] is typically used optimize this kind of problem. We will demonstrate how to adapt this compositional minimax optimization problem to federated learning and address the unique challenges.
3.2 Problem Setup
In this paper, optimize the deep compositional AUC maximization problem under the cross-silo federated learning setting, we will concentrate on developing an efficient optimization algorithm to solve the following generie federated stochastic compositional minimax optimization problem:
K
K
min max XER-1 yER
(x),))
(4)
K
k=1
where K' the number of devices, ()) = E E 1g (8) (+:E) Rdd denotes the inner-level
function for the data distribution D of the k-th device, ((x) : BwD ((*)) (,, (C] represents
the outer-level function for the data distribution D()) of the k-th device. It is worth noting that both the inner-level function and the outer-level function distributed different devices, which is significantly different from traditional federated learning models. Therefore, we need to design a new federated optimization algorithm to address this unique challenge.
Here, we introduce the commonly used assumptions from existing work 6 the convergence rate of our algorithm.
for investigating
Assumption 1. The gradient of the outer-level function funnriioo Lj i.e.,
(..-) -) is -ippschhtt continuous where IV o (11, V [22, 12) < L7 - (zz. N2))?
(0)
(5)
[Vy (21. Vy (2 v2)1 < )3)((z).11) 82)11
hold for V(zI. Y2) Rd d2 The gradient of the inner-level function g (x) is Lg Lipschitz continuous where Lg > 0, i.e.,
[Vg (k) (x1) (x) 11 < L3|x1 x211?
(6)
holds for Vx1, X2 E ddd..
Assumption 2. The second moment of V Vyf(k) (z. y: () and V ( (x: E) satisfies:
EIIVV (z.y: (2.
C El (x;
(7)
for V(2,y) y) E Rdd x dR and Vx E RRd, where C > 0 and C. > Meanwhile, the second moment of the full gradient is assumed to have the same upper bound.

---

Assumption 3.
The variance of the stochastic gradient of the fumecttt (c.-) satisfies:
EIIV (2.Yi)) - ( for y) IRd, x Red, where of > 0. Additionally, the variance of the stochastic gradient and the stochastic function value of g(8) satisfies:
y)1l - (l, (z. V
V ( y) o (8)
( (x:E) (x)]1) S oz) [xIlg(( - - g0h S og
(9)
for Vx E Rd, where og 0 and O > 0.
Assumption 4. The outer-level function ((k y) is y-strongly-concave with respect to y for any fixed z E RR,,, where μ 0, i.e.
(2. S / (2, + vV (2. - -y²) - 2 ly - y2ll
(10)
Notation: Throughout this paper, a, denotes the variable of the k-th device in the t-th iteration and (k) denotes the averaged variable across all devices, where a denotes any variables used in this paper. x, denotes the optimal solution.
a, K Σk=1 a
4 Methodology
In this section, present the details of our algorithm for the federated compositional deep AUC maximization problem defined in Eq.
Algorithm 1 LocalSCGDAM
Input: Xo. Yo. 17 E (0, Th > 0, 72y > 0. Br > 0, By > 0, a > 0, an E (0,1), Bun E (0, 1),
By7 E (0, 1).
All workers conduct the steps below to update x, y. x) = Xo Yo = yo.
(M)
(R) g*)) (xD
(k). (A))
hg
50
ug (gg(h) (xg 50
(k) (k)
a
(bb
(k) (k), (k))
No
vo
v, (k) (k) (K)),
No (600)
1: for = =0, -..... do
2: Update x and y:
((x) (k)
qu R3)
y+1 = yt (k)
(k)
+ t
3: Estimate the level function: (k) (k)
b (1- annnn) an) (K)
ang (k)
(x(+1i ((+1)
4: Update momentum:
u (1 u (1)
(k)
Ban (b) (x 1441 S1+1) JTV gf (+) y
(k) (1- 11--8 v + By y (h (k) (k) (k))
(k)
Vitl
(11: 9+1 11+1)
5: if mod(t + +,p))
= then
6:
h (K)
*+1 -ht11
k
1
Σ
h (k')
H+1
u (k)
ilt+1
A K
>x==
K
(K))
v Vt+i A
(k)
K H1
k
(k)
(K)
K (K)
ye ye+1 - E (K)
(k)
x [+1
xx+1
K
=== X
7: end if
8: end for
To optimize Eq. 4 we developed a novel local stochastic compositional gradient descent ascent with momentum algorithm, shown in Algorithm [ Generally speaking, in the t-th iteration, employ the local stochastic (compositional) gradient with momentum to update the local model parameters xx(*) and ( on the k-th device. There exists an unique challenge when computing the local stochastic compositional gradient compared traditional federated learning models. Specifically, as shown in Eq. (4), the objective function depends on the global inner-level function. However, it is not feasible to communicate the inner-level function every iteration. To address this challenge, we propose to employ the local inner-level function to compute the stochastic compositional gradient at each iteration and then communicate the estimation of this function periodically to obtain the global inner-level function.
5

---

In detail, since the objective function in Eq. 4 is compositional function whose stochas- (k), :
tic gradient regarding x, i.e., (g (k), (k) . of (g()
). ). (()),, is bi- ased estimation for the full gradient, we employ the stochastic compositional gradient (g(h) (k), Et f( (R) (M), ((()) to update the model parameter x, where h,d) is the
Ss
:Yt moving-average estimation of the inner-level function y (*)) (x{", ( on the k-th device, which is defined as follows:
h (1 on) b + ang (8) (xg
(k),
(11)
where a > 0 and r > 0 are two hyperparameters, and on E (0, 1). The objective function in Eq. (4) not compositional regarding y. thus we can directly leverage its stochastic gradient to perform an update. Then, based on the obtained stochastic (compositional) gradient, we compute the momentum as follows:
(A) = - Bon) (k) V
(k) "V gf (k) (h,
)
(12)
(k) (1 Bym) v- B (he Y
(88
(1) (8) (k)
where Bg 0 and By
two hyperparameters, B²n E (0, 1). and Byn E (0, Based on the obtained momentum, each device updates its local model parameters as follows:
(k) 2-1 X
yit1 = Yt
(k) + t
(13)
where Y and Ty 0.
As we mentioned before, to obtain the global inner-level function, our algorithm periodically commu- (k) nicates the aoving-average estimation of the inner-level function, i.e., h In particular, at every iterations, i.e., (t + p) == O where >> 1 is the communication period, each device uploads
(k) to the central server and the central server computes the average of all received variables, which will be further broadcast to all devices as follows:
h (2)
h
(14)
In this way, each device is able obtain the estimate of the global inner-level function periodically. As for the model parameters and momentum, we employ the same strategy as traditional federated learning methods 23 36 communicate them periodically with the central server, which is shown
in Step 6 in Algorithm 1
In summary, developed novel local stochastic compositional gradient descent ascent with momentum algorithm for the compositional minimax problem, which shows how to deal with two distributed functions in federated learning. With our algorithm, we enable federated learning for the compositional deep AUC maximization model, benefiting imbalanced data classification tasks.
5 Theoretical Analysis
In this section, provide the convergence rate of our algorithm to show how it is affected by the number devices and communication period.
To investigate the convergence rate of our algorithm, we introduce the following auxiliary functions:
K
K Σ k=1
(x) arg max yER
Σ (x)))))) M
f(x) K
(())
(2)
(x).
K
K
K
k=
W=1
(15)
i we can obtain that (b() is I---mmotth, where T. = 20919 + CC
Then, based on Assumptions
#
The proof can be found in Lemma 2 of Appendix A. In terms of these auxiliary functions, we establish the convergence rate of our algorithm.
Theorem
1. Given Assumption
by setting a
>
0,
Br
0.
By
7
min 2% by 10p L. a
1
Ba
17.
S min
6l.p"
100L"
3
3.9a
16p
S.
and Yo
S
min
ax
8.
By 1113
Vapu
(10021/110013
32 /CD0)/+0
21C 1000 +20
2003 L3

---

Algorithm 7 has the following convergence rate
Σ [x)) 2(0(xx)) (xx.)
L4
Iyo y'o)ll + +x +
+
Va7/l'
n/
0pp²n)² + 0(p"1'()))))))))))))))r())))00 + + 0 + +0('0111))))
(16)
Remark 1. In terms of Theorem for sufficiently large T, by setting the learning rate O(K pp / Algorithm I can achieve O(1/ ((1/v KT) convergence rate, which indicates linear speedup with respect to the number of devices K. In addition, it straightforward to show that the communication complexity of our algorithm is T/p = O(K3/=T3/4). Moreover, achieve the c-accuracy solution, i.e., Et= 11 (x)] S é, by setting 7j O()²) and
p ((1((Ke) (Ke)/ then the sample complexity on each device is (11 ()) and the communication complexity is (()/e).
Challenges. The compositional structure in the loss function, especially when the inner-level functions are distributed on different devices, makes convergence analysis challenging. In fact, the existing federated compositional minimization algorithms 4 27
12 28 only consider much simpler case where the inner-level functions are not distributed across devices. Therefore, our setting is much more challenging than existing works. On the other hand, all existing federated compositional minimization algorithms [ 27 12 28] fail to achieve linear speedup with respect to the numbe of devices. Thus, it is still unclear whether the linear speedup is achievable for federated compositional optimization algorithm. In this paper, we successfully addressed these challenges with novel theoretical analysis strategies and achieved the linear speedup for the first time for federated compositional minimax optimization algorithms. We believe our approaches, e.g., that for bounding consensus errors, be applied to the minimization algorithms achieve linear speedup.
6 Experiments
In this section, we present the experimental results to demonstrate the performance our algorithm.
Sorr 3 Shicea Po
0.75
LocaSGDM Leca5GDM, RL
LOCa/SGDM
LOCD/GGOM
LOCaISGDM JRL
LOCKISGOM JRL
CoDA
0.80
CoDA
CoDA
035
LoCaSCEDAM
LOCaSOCDDAM
LOCNSCGDAM
side
13:00
30000
13:000
Iteratton
Ite matton
fremattoo
(a) CATvsDOG
(b) CIFARIO
(c) CIFAR100
0.58
0.3
6.79
AUC
0.18
AVCC
2.00
0:59
LocalSciOM_R RL
LocalSGDM RI.
LocaiSCOM RL
6.50
CoDA
CoDA
CoDA
LocalSGDAM
LocalSGDAM
Local (SCDAM
LoCaISCGDAM
LocalSCEDAM
6.0
2000
40:0
590
2000
40:00
19903
120
1399
2000
5000
5099
7000
(d) STLIO
(e) FashionMNIST
(f) Melanoma
Figure 1: Testing performance with AUC score versus the number of iterations when the communica-
tion period p = 4.

---

AUC
679
AUC
Loc455G0M
LacWSGDM
CoDA
285
CoDA
CoDA
IeratSCDAM
InCAlSGDAM
lacwSGDAM
LecalSCODAM
LoCAIS
LocASCCDDAM
(a) CATvsDOG
(b) CIFARIO
(c) CIFARIO0
04.
044
6.79
LocalSCIM
LocalSGDM
LeraiSCDMM
LocaISGOM_AL
042
LocalS5DM,N RL
LocalSEDM,AN
1.5
LocalSGDAM
LocalSGDAM
Loca Loca/SCDAM
LocaISCGDAM
LOCAISCEDAM
LoraISCG DAM
39.0
Derratt ion
Ieeraion
Reration
(d) STLIO
(e) FashionMNIST
(f) Melanoma
Figure 2: Testing performance with AUC score versus the number of iterations when the communica-
tion period p 8.
Alay Audane Aw
6.00
6.79
LocalSriOM LocalSGiMM RL
LocalSGDM
LecalSGIM LocalSGDM RL
LocaISGDM RL
6.0
LocalSGibAM LocaISCUDAM
LocalSGDAM LocalSCUDAM
LocalSGDAM
0.0
LocalScG
SCN
20000
2040
Toet Bleration
Earation
(a) CATv&DOG
(b) CIFARIO
(c) CIFAR100
2.30
0.33
08
2.79
0.08
LocatScioM
0.05
LocatSCiOM
LocalScioM
LocalSGDDM
Local5CDM
CoDA
CoDA
CoDA
LocatSCiDAM
LocalSGDAM
LecaiSCDAM
LocaISCGDAM
2000
(d) STL10
(c) FashionMNIST
(f) Melanoma
Figure 3: Testing performance with AUC score versus the number of iterations when the communica- tion period p = 16.
6.1 Experimental Setup
Datasets. In our experiments, we employ six image classification datasets, including CIFAR10
15). CIFAR 100 [15), STLIO [1. FashionMNIST (32), CATvsDOG P and Melanoma [22]. For the first four datasets, following [39]. we consider the first half of classes to be the positive class, and the second half as the negative class. Then, in order to construct highly imbalanced data, we randomly drop some samples of the positive class in the training set. Specifically, the ratio between positive samples and all samples is set to 0.1. For the two-class dataset, CATvsDOG, we employ the same
httpp::/// cttp://l..co/////e///eeeeee

---

Table I: The comparison between the test AUC score of different methods on all datasets. Here, denotes the communication period.
Datasets
Methods
AUC
= 4
8
p = 16
CATvsDOG
LocalSCGDAM
0.000
0.000
0.928+10000
CoDA
0.805:10.000
0.892-10.000
0.883:10.001
LocalSGDAM
0.899:10.000
0.88410.0000
0.884:10.001
LocalSGDM
0.888:10.001
0.889=00.000
0.887110.000
LocalSGDM RL
0.909=000000
0.901=0.001
0.917=00001
CIFARIO
LocaISCGDAM
0.914
0.000
0.906
CoDA
0.890+000000
0.886+00.000
0.883++00000
LocalSGDAM
0.893+0.000
0.880=10.000
0,880=10.000
LocalSGDM
0.88310.001
0.0000
0.87410.000
LocalSGDM_RL RL
0,890100000
0.90410.001
0,883110.001
CIFARIO0
LocAlSCGDAM
0.702
0.70410.001
0.703
CoDA
0.6041
0.681+0.001
0.685
LocalSGDAM
0.600
0.694+0..000
0.689-10.001
LocalSGDM
0.675#000001
0.669=0.000
0.669+0.000
LocalSGDM RL
0.676110.000
0.690=00.001
0.682:10.001
STL1O
LocalSCGDAM
0.000
0.807
0.801
CoDA
0,801
0.784
0.783110.000
LocalSGDAM
0.7921000000
0.790-L0.000
0.780+0.0000
LocalSGDM
0.760+10.001
0.0000
0.757
LocalSGDM RL
0.773
0.000
0.752
MNNST
LocaISCGDAM
L.,000
1.000
1.00000000
CoDA
0.976+00000
0.976+00000
0.976+
LocaISGDAM
0.977=00.000
0.977
0,976110.000
LocalSGDM
0.963
0.956
0,955100000
LocalSGDM RL
0.980
0.957
0.962-10.000
Melanoma
LocaISCGDAM
1.0000
0,880 100000
0.870-10.000
CoDA
0.734
0.721
0.725+10.003
LocalSGDAM
0.730+00.00M
0.700
0.721
LocalSGDM
0.774
0.766
0.750+10.000
LocalSGDM RL
0.737#0.001
0.7000
0.773+0.001
strategy to construct the imbalanced training data. For these synthetic imbalanced datasets, the testing set is balanced. Melanoma is an intrinsically imbalanced medical image classification dataset, which we do not modify. The details about these benchmark datasets are summarized in Table
Experimental Settings, For Melanoma, we use DenseNet121 1 where the dimensionality of the last layer is set for binary classification. The details for the classifier for FashionMNIST can be found in Appendix B. For the other datasets, we use ResNet20 [0]. where the last layer is also set to 1. To demonstrate the performance of our algorithm,
compare it with three state-of-the-ar methods: 23 Specifically, LocalSGDM uses momentum SGD to optimize the standard cross-entropy loss function. LocalSGDM_RL employ momentum SGD to optimize a Ratio Loss function, which is to add a regularization term to the standard cross-entropy loss function address the imbalance distribution issue. CoDA leverages SGDA to optimize AUC loss, while LocalSGDAM exploits momentum SGDA to optimize AUC loss. For fair comparison, we use similar learning rates for all algorithms. The details can be found in Appendix B. We use 4 devices (i.c., GPUs) in our experiment. The batch size on each device is set to for STL10, 16 for Melanoma, and 32 for the others.
LocalSGDM [36,, LocalSGDM _R 30). CoDA 3 LocalSGDAM
6.2 Experimental Results
In T we report the AUC score of the test set for all methods, where we show the average and variance computed across all devices. Here, the communication period set to 4, 8, and 16, respec- tively. It can be observed that our LocalSCGDAM algorithm outperforms all competing methods

---

for all cases. For instance. our LocalSCGDAM can beat baseline methods on CATvsDOG dataset with a large margin for all communication periods. These observations confirm the effectiveness of our algorithm. In addition, we plot the average AUC score of the test set versus the number of can also be observed that our algorithm outperforms baseline methods consistently, which further confirms the efficacy of our algorithm.
iterations Figures 2 g
To further demonstrate the performance of our algorithm, we apply these algorithms the dataset with different imbalance ratios. Using the CATvsDOG dataset, we set the imbalance ratio to 0.01, 0.05, and 0.2 construct three imbalanced training sets. The averaged testing AUC score of these three datasets versus the number of iterations is shown in Figure 4 It can be observed that our algorithm outperforms competing methods consistently and is robust to large imbalances in the training data. Especially when the training set is highly imbalanced, e.g., the imbalance ratio is 0.01, all AUC based methods outperform the cross-entropy loss based method significantly, and our LocalSCGDAM beats other AUC based methods with large margin.
275
646-
LocalSCDM
LocalSGDM
0.9
CoDA
LocalSEDM RL
CAN
CoDA
LoCaISGOM RL
639
LocalSGOM RL
CoDA
2.5
LocalSEDAN
LocaI5GDAM
LocalSGDAM
LoCaISCGDAM
LocalSCGDAM
caR-
3000
2010
Johnn
2010
Ibcration
Iseration
(a) Imbalance Ratio:0.01
(b) Imbalance Ratio:0.05
(c) Imbalance Ratio:0.2
Figure 4: The test AUC score versus the number of iterations when using different imbalance ratios for CATvsDOG.
Finally, compare our algorithm with two additional baseline methods: SCAFFOLD [1]] and FedProx [16]. These methods assume the local data distribution imbalanced but the global one is balanced. Then,
0.82
they the global gradient correct the local one. However, this kind of methods do not work when the global data distribution imbalanced. Specifically, when the global gradient itself is computed on the im- balanced data, rather than the balanced one, it cannot alleviate the imbalance issue in the local gradient. In Figure we show the test AUC score of the STLI0 dataset, where we use the same experimental setting as that of Figure [ i.e., both the local and global data distributions are imbalanced. It can be observed that our algorithm outperforms those two baselines with large margin, which confirms the effectiveness of our algorithm in handling the global imbalanced data distribution.
0.78
0.78
0.72
SCAFFOLD
Fed
LocaISCGDAM
Figure 5: The test AUROC score for STL10.
Conclusion
In this paper, we developed a novel local stochastic compositional gradient descent ascent algorithm solve the federated compositional deep AUC maximization problem. On the theoretical side, established the convergence rate of our algorithm, which enjoys a linear speedup with respect to the number devices. On the empirical side, extensive experimental results multiple imbalanced image classification tasks confirm the effectiveness of our algorithm.
Acknowledcements
We thank anonymous reviewers for constructive comments. Yang was partially supported by NSF Career Award 2246753, NSF Grant 2246757.
10

---

References
[1] A. Coates, A. Ng. and H. Lec. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 215-223, JMLR Workshop and Conference Proceedings, 2011.
[2] Y. Deng and M. Mahdavi. Local stochastic gradient descent ascent: Convergence analysis and communication effiiciecy. In International Conference Artificial Intelligence and Statistics, pages 1387-1395. PMLR, 2021.
[3] H. Gao and H. Huang. Fast training method for stochastic compositional optimization problems. Advances in Neural Information Processing Systems, 0-25333--23455 2021.
[4] H. Gao, J. Li, and H. Huang. On the convergence of local stochastic compositional gradient descent with momentum. In International Conference on Machine Learning, pages 7017-7035. PMLR, 2022.
[5] H. Gao, X. Wang, L. Luo, and X. Shi. On the convergence of stochastic compositional gradient descent ascent method. In Thirtieth International Joint Conference on Artificial Intelligence, 2021.
[6] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and hamnsssing adversarial examples. arxiv preprint 141v::412..6777 2 2014.
[7] Z. Guo, R. Jin, J. Luo, and T. Yang. Fedxl: Provable federated learning for deep x-risk optimization. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 11934-11966. PMLR, 2023.
[8] Z. Guo, M. Liu, Z. Yuan, L. Shen, W. Liu, and T. Yang. Communication-effciienn distributed stochastic auc maximization with deep neural networks. In International Conference on Machine Learning, pages 3864-3874. PMLR, 2020.
[9] J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(():29-36. 1982.
[10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[11] A. Herschtal and B. Raskutti. Optimising area under the roc curve using gradient descent. In Proceedings of the twenty-first international conference on Machine learning, page 49, 2004.
[12] F. Huang, J. Li, and H. Huang. Compositional federated learning: Applications in distributionally robust averaging and meta learning. arXiv preprint arXiv:2106.1 2021.
[13] G. Huang, Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[14] S. P. Karimireddy, Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning. pages 5132-5143. PMLR, 2020.
[15] A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
[16] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. Proceedings of Muchine learning and systems, 2:429 450, 2020.
[17] T. Lin, C. Jin, and M. Jordan. On gradient descent ascent for nonconvex-concav minimax problems. In International Conference on Machine Learning, pages 6083-6093. PMLR, 2020.
[18] M. Liu, Z. Yuan, Y. Ying, and T. Yang. Stochastic auc maximization with deep neural networks.
arXiv preprint arXiv: 1003, 2019.

---

[19] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication--ffiienn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMIR, 2017.
[20] D. Povey, X. Zhang, and S. Khudanpur. Parallel training of dnns with natural gradient and parameter averaging. arXiv preprint arXiv: 2014.
[21] S. Qiu, Z. Yang, X. Wei, J. Ye, and Z. Wang- Single-timescal stochastic cnnconvex--oonare optimization for smooth nonlinear td learning. arXiv preprint arXiv:2008. 10103, 2020.
[22] V. Rotemberg, N. Kurtansky, B. Betz-Stablein, L. Caffery, E. Chousakos, N. Codella, M. Com- balia, Dusza, P. Guitera, D. Gutman, et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Scientific data, 8(1):188, 2021.
[23] P. Sharma, R. Panda, G. Joshi, and P. Varshney. Federated minimax optimization: Improved convergence analyses and algorithms. International Conference on Machine Learning, pages 19683-19730. PMLR, 2022.
[24] Z. Shen, J. Cervino, H. Hassani, and A. Ribeiro. An agnostic approach to federated learning with class imbalance. In International Conference on Learning Representations, 2021.
[25] U. Stich. Local sgd converges fast and communicates little. arXiv preprint arXiv: arXiv:1805.000 180.099767 2018.
[26] H. Su and H. Chen. Experiments on parallel training of deep neural network using model averaging. arXiv preprint arXiv: 557.0//39 2015.
[27] D. A. Tarzanagh, M. Li, C. Thrampoulidis, and S. Oymak. Fednest: Federated bilevel, minimax, and compositional optimization. arXiv preprint arXiv:2205.02211 2022.
[28] B. Wang, Z. Yuan, Y. Ying, and T. Yang. Memory-based optimization methods for model- agnostic meta-learning. arXiv preprint arXiv:2106.044) 0,9/0, 2021.
[29] Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611 1-7623, 2020.
[30] L. Wang, Xu, X. Wang, and Q. Zhu. Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 10165-10173, 2021.
[31] M. Wang, X. Fang, and H. Liu. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming, (61(1):419- 449, 2017
[32] H. Xiao, K. Rasul, and R. Vollgraf. Pashion-mnist novel image dataset for benchmarking machine learning algorithms, 2017.
[33] Y. Yan, Y. Xu, Q. Lin, W. Liu, and T. Yang. Optimal epoch stochastic gradient descent ascent methods for min-max optimization. Advances Neural Information Processing Systems, 33:5789-5800, 2020.
[34] T. Yang. Trading computation for communication: Distributed stochastic dual coordinate ascent. Advances in neural information processing systems, 26, 2013.
[35] Y. Ying, L. Wen, and Lyu. Stochastic online auc maximization. Advances in neural information processing systems, 29, 2016.
[36] H. Yu, R. Jin, and S. Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning, pages 7184-7193. PMLR, 2019.
[37] H. Yu, S. Yang, and S. Zhu. Parallel restarted sgd with faster convergence and less communica- tion: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 5693-5700, 2019.
12

---

[38] H. Yuan, X. Lian, and J. Liu. Stochastic recursive variance reduction for efficient smooth non-convex compositional optimization. arXiv preprint arXiv:1912.13515, 2019.
[39] Z. Yuan, Z. Guo, Chawla, and T. Yang. Compositional training for end-to-end deep auc maximization. In International Conference Learning Representations, 2021.
[40] Z. Yuan, Z. Guo, Y. Xu, Y. Ying, and Yang. Federated deep maximization for hetergeneous data with constant communication complexity. In International Conference on Machine Learning, pages 1 PMLR, 2021.
[41] Zhang and L. Xiao. composite randomized incremental gradient method. In International Conference on Machine Learning, pages 7454-7462, 2019.
[42] J. J.Zanng P. Xiao, R. Sun, and Z. Luo. A single-loop smoothed gradient descent- ascent algorithm for nonconvex-coneet conceve min-max problems. Advances Neural Information Processing Systems, 33:7377-7389, 2020.
13


Reviews:
[{"review_id": "ECg6e4ed2R", "reviewer": "Reviewer_jnCq", "summary": "The paper proposes a new federated learning algorithm to address the class imbalance problem. Instead of using cross-entropy loss functions, the proposed algorithm directly optimizes the AUC score by solving a federated stochastic compositional minimux optimization problem. Specifically, the paper proposes to employ the local stochastic gradient with momentum to update the local model parameters. Experiments show that the proposed algorithm achieves better model performance than the other baselines.", "questions": "1. Is existing FL approaches applicable with LocalSCGDAM? Existing studies usually optimize the standard cross-entropy loss. I’m curious whether they are applicable to the studied loss function and how they compare with LocalSCGDAM. \n\n2. Is the experimental setting satisfies the assumptions in Section 3.2?\n\n3. Is the algorithm applicable in the client sampling setting? Also, how many clients are used in the experiments? I suggest the authors to add experiments to study the scalability.\n\n4. How does the convergence rate of LocalSCGDAM compare with other related studies?\n\n5. Is directly optimizing AUC loss a mainstream method in class-imbalanced learning?\n", "limitations": "One potential limitation is that the algorithm may not work well in the client sampling setting.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper considers a setting where the global distribution is also class-imbalanced, which is interesting.\n\n2. The paper provides a theoretical analysis of the convergence of the proposed algorithm.\n", "weaknesses": "1. There are many FL studies that try to address non-IID challenge in FL. However, the baselines seem to focus on different optimization methods and lack SOTA FL studies that address the non-IID data (e.g, [1][2]). \n\n[1] Addressing class imbalance in federated learning\n[2] No fear of heterogeneity: Classifier calibration for federated learning with non-iid data\n\n2. The theoretical analysis requires assumptions on the outer-level and inner-level functions. I’m not sure how realistic these assumptions are.\n\n3. The algorithm estimates the inner-level function in local training. It may not be applicable in the client sampling setting, where a client may be selected after many rounds.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's comments and suggestions. We address the reviewer’s comments below:\n\nAnswer for weakness. \n\n**1**. Firstly, our method is significantly different from the heterogeneous federated learning methods. Specifically, our paper aims to address a more challenging issue than the traditional FL methods designed for non-IID data. \nMore specifically, even though some traditional FL methods try to address the non-IID issue, they typically assume **the global data distribution (i.e., combining the data of all clients) is balanced**. For instance, the reference [2] that the reviewer provided uses Dir distribution to simulate the non-iid data, where different clients have different distributions but the global distribution is balanced. On the contrary, our method and the reference [1] that the reviewer provided assume **the global distribution is imbalanced**, which is much more challenging. To address this challenging task, we developed a federated compositional AUC maximization model to handle the global imbalance issue, which has shown superior performance according to our experimental results.\n\nSecondly, we have compared with [1] in our experiments, which is denoted by LocalSGDM-RL in Line 257. All experimental results have shown that our algorithm can outperform [1] by a large margin. \n\n**2**. Those assumptions are common in compositional optimization problems. Especially, the single-machine counterpart algorithms [4, 35] use the same assumptions. We do not have any strong assumptions compared with them. In fact, these assumptions are mild and the model used in our experiment, which is also used in [35], satisfies those assumptions. \n\n\n**3**. It is trivial to extend our algorithm to the client sampling setting. Each selected client updates the inner-level function estimator and then computes stochastic compositional gradient to update model parameters, which does not require special operations under the client sampling setting. We will provide discussions about this point in the final version. \n\n\nAnswer for questions.\n\n**1**. Existing FL approaches learn the classifier via minimizing the cross-entropy loss function, which is a **minimization** problem. On the contrary, the AUC loss function used in this paper is a **minimax** loss function. For minimization problems, one should use the **stochastic gradient descent** algorithm, while the **stochastic gradient descent ascent** algorithm should be used for minimax problems. Since LocalSCGDAM is designed for a compositional minimax problem, not a minimization problem, the standard FL approaches cannot be applied to the compositional minimax problems. \n\n\n**2**. Please see the answers for weakness 2.\n\n**3**. Please see the answers for weakness 3. We use four clients currently. We will use more clients to verify the performance in the final version. \n\n**4**. As shown in Remark 1, the convergence rate is $O(1/\\sqrt{KT})$ where $K$ is the number of clients, $T$ is the number of iterations. Obviously, this convergence rate matches that of standard FedAvg for nonconvex problems. \n\n**5**. Directly optimizing AUC loss is a powerful approach to address the imbalanced data classification issue. It has attracted a lot of attention in the past few years, e.g., [6,35]. \n"}, {"review_id": "QcMTfw2Pdj", "reviewer": "Reviewer_BTwE", "summary": "This paper firstly studies the federated compositional AUC maximization problem, which includes both the local and global imbalanced distributions, and proposes the momentum-based algorithm LocalSCGDAM to solve this problem. The SOTA convergence rates are established and various experiments are used to evaluate the proposed algorithms. ", "questions": "1. In the math, $\\mathcal{D}^k_{g}$ and $\\mathcal{D}^k_{n}$ are clear and they are the datasets in the inner layer and out layer. But in the AUC maximization, how to define these datasets? \n\n2. This work solves the binary imbalanced data distribution problem well. I am curious whether it is possible to extend it to the multiple-class imbalanced data distribution problem. \n\n3. In the nonconvex optimization, we consider the convergence rate, including sample complexity and communication complexity, to reach an $\\epsilon$-stationary point. In this case, what is the convergence rate of your algorithm?  ", "limitations": "No", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novelty: This paper is the first work to consider the federated compositional AUC maximization problem. Data heterogeneity is a key problem in federated learning and many works focus on it.  The Federated compositional AUC maximization problem is more challenging because it considers both the local and global imbalanced distributions.\n\nQuality: It proposes a new algorithm to solve the problem. The structure of the algorithm is clear.  Both theoretical analysis and experimental verification are provided. The theoretical analysis is very complete.\n\nClarity: The paper is organized well and it is easy to follow. \n\n", "weaknesses": "1. More motivation should be introduced. It includes why 1) AUC is significant and 2) why FL AUC should be considered.\n\n2. In the experiments, all samples are set to 0.1. Discussion about different imbalanced data settings is welcomed. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's comments and suggestions. We address the reviewer’s comments below.\n\nAnswers for weakness.\n\n**1**. First, when the data distribution is imbalanced, directly minimizing the cross-entropy loss function cannot learn a good classifier since it may ignore the minority class. On the contrary, the AUC loss function focuses on both majority and minority classes, which can alleviate the limitation of traditional cross-entropy loss function [31]. Thus, we leverage the AUC maximization to address the imbalance issue, which has been confirmed by our experiments, i.e., LocalSCGDAM outperforms LocalSGD. \n\nSecond, the AUC loss function is a minimax function, which is more difficult to optimize than the traditional convex cross-entropy loss function. Thus, we developed the LocalSCGDAM algorithm to optimize the compositional AUC loss function, which incorporates the pertaining process as shown in Eq.(2). As such, the prediction performance is better than directly optimizing the AUC loss function, which is confirmed by our experiments, i.e., LocalSCGDAM outperforms CoDA and LocalSGDAM. \n\nThird, FL is an effective tool for real-world data analysis tasks and data distributions in those tasks are typically imbalanced, e.g., the electronic health record (EHR) data. Thus, enabling federated learning for AUC optimization is necessary and important to address real-world learning challenges. \n\n**2**. Thanks for the reviewer's suggestion. In fact, we have already conducted this experiment in Appendix. The results in Figure 4 has confirmed the superior performance of our method for different imbalanced ratios. \n\nAnswers for questions. \n\n**1**. As shown in Eq. (2), the inner-level function is to pre-train the classifier by minimizing the cross-entropy loss function, and the outer-level function is to learn the classifier via optimizing the AUC loss function. Both of them use the training samples and their labels. In other words, $\\mathcal{D}_f$ and $\\mathcal{D}_g$ in compositional AUC maximization denote the two copies of the training dataset. \n\n**2**. It is easy to extend the binary classification case to the multi-class case. Please see Eq.(4) in [16].\n\n**3**. Based on Theorem 1, to reach the $\\epsilon$-stationary point, i.e., $\\frac{1}{T} \\sum_{t=0}^{T-1} \\mathbb{E}\\left[\\left\\|\\nabla \\Phi\\left(\\overline{\\mathbf{x}}_t\\right)\\right\\|^2\\right]\\leq \\epsilon^2$ the convergence rate is $O(\\frac{1}{K\\epsilon^4})$, the sample complexity on each client is  $O(\\frac{1}{K\\epsilon^4})$, the communication complexity is $O(\\frac{1}{\\epsilon^3})$.\n\n"}, {"review_id": "pWiyhJIuzX", "reviewer": "Reviewer_5nau", "summary": "This work aims to address the challenges of imbalanced data in FL. To this end, the authors propose to optimize AUC score. Some experiments are conducted to verify the effectiveness of the proposed method.", "questions": "cf. Weaknesses", "limitations": "cf. Weaknesses", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is easy to follow. The notations are well-defined. The studied problem is promising.", "weaknesses": "This work confuses me a lot.\n\nThe authors believe that data imbalance is a crucial issue under the FL scenario, which is consistent with my understanding. The problem has motivated the data-split fashion, i.e., Latent Dirichlet Sampling [1]. Moreover, many excellent works have demonstrated the effectiveness of their efforts in mitigating data heterogeneity [2, 3]. However, I cannot find these works in this work. This indicates that the authors may overlook some advanced methods in this field.\n\n[1] Measuring the effects of non-identical data distribution for federated visual classification\n[2] Federated optimization in heterogeneous networks\n[3] SCAFFOLD: Stochastic controlled averaging for federated learning\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's comments and suggestions. We address the reviewer’s comments below.\n\nFirst, our method is significantly different from the heterogeneous federated learning approaches. Specifically, most existing heterogeneous federated learning approaches consider a setting where **the local distribution is imbalanced but the global distribution is balanced**, i.e., different clients have different data distributions but the combination of all clients' data is balanced.  On the contrary, our work considers a setting where **both the local and global distributions are imbalanced**, which is much more challenging than existing heterogeneous federated learning methods. \n\nSecond, we have cited [2] and [3] in Lines 112-118. We will cite and discuss [1] as the reviewer suggested. \n\nThird, we have compared the SOTA method that is designed to address the global imbalance issue, i.e., LocalSGDM-RL [27]. Our method outperforms this strong baseline algorithm with a large margin for all datasets. \n\n**Hope we have answered all your questions. We sincerely hope the reviewer can consider our response and re-evaluate the contributions of our work. In fact, we considered a more challenging setting and provided a novel solution to address this challenge. Both theoretical and empirical results confirm the effectiveness of our method. We believe our contributions are important to this area and our work can inspire more follow-up works to handle more challenging federated learning settings.**"}]

Author rebuttal:

Meta-review: The paper considers federated supervised learning (classification) in the case where the data belonging to different classes is (highly) imbalanced. The main idea behind the proposed method is to directly optimize the area under curve (AUC) score. This is performed by casting the AUC maximization problem as a federated compositional minimax optimization problem, for which the authors develop a local stochastic compositional gradient descent ascent with momentum algorithm, and provide bounds on the computational and communication complexities of our algorithm.

The reviewers had several concerns and the authors made a great effort to address those concerns. After going through the paper and all the responses, my assessment is that the major concerns have been resolved and the contributions are sufficiently novel to warrantee acceptance. 

I recommend to the authors to incorporate all the reviewers' comments in the updated version (also, please add all the simulations results you did during the rebuttal/discussion period to the main paper). As a side note, there are approaches to federated learning with imbalanced data which are based on constrained optimization; in my view, those approaches can also viewed as some kind of a minimax method (once viewed from the angle of duality). I would recommend the authors to discuss those works as related works too (e.g. see Shen et al: "An agnostic approach to federated learning with imbalanced data")
Decision: Accept (poster)