Decentralized Decoupled Training for Federated Long-Tailed Learning
Authors: Wenkai Yang
Deli Chen
Hao Zhou
Fandong Meng
Jie Zhou
Xu Sun
Venue: Accepted by TMLR ()

Abstract: In the real world, the data samples often follow a long-tailed distribution, which poses a great challenge for Federated Learning (FL). That is, when the data is decentralized and long-tailed, FL may produce a poorly-behaved global model that is severely biased towards the head classes with the majority of the training samples. To settle this issue, decoupled training has recently been introduced to FL. Decoupled training aims to re-balance the biased classifier after the normal instance-balanced training, and has achieved promising results in centralized long-tailed learning. The current study directly adopts the decoupled training idea on the server side by re-training the classifier on a set of pseudo features, due to the unavailability of a global balanced dataset in FL. Unfortunately, this practice restricts the capacity of decoupled training in federated long-tailed learning as the low-quality pseudo features lead to a sub-optimal classifier. In this work, motivated by the distributed characteristic of FL, we propose a decentralized decoupled training mechanism by leveraging the abundant real data stored in the local. Specifically, we integrate the local real data with the global gradient prototypes to form the local balanced datasets, and thus re-balance the classifier during the local training. Furthermore, we introduce a supplementary classifier in the training phase to help model the global data distribution, which addresses the problem of contradictory optimization goals caused by performing classifier re-balancing locally. Extensive experiments show that our method consistently outperforms the existing state-of-the-art methods in various settings. Our code is available at https://github.com/keven980716/Federated_Learning_Experiments.

Published in Transactions on Machine Learning Research (04/2024)
Decentralized Decoupled Training for Federated Long- Tailed Learning
Wenkai Yang
wenkaiyang ruc. ruc.edu.cn
Gaoling School of Artificial Intelligence Renmin University of China
Deli Chen DeepSeek
victorchen derbsseke@ddeeeeee..comm
Hao Zhou
tuxzhou tencent..onm
Pattern Recognition Center, WeChat Tencent Inc.
Fandong Meng
fandongmeng tencent.com
Pattern Recognition Center, WeChat Al Tencent Inc.
Jie Zhou
withtomzhou @ tencent.com
Pattern Recognition Center, WeChat Tencent Inc.
Xu Sun
xusui @ pku..pkuuuuu
National Key Laboratory for Multimedia Information Processing, School of Computer Science Peking University
Reviewed OpenReview: http:////oeeeeeeeeeee/e///////ea
Abstract
In the real world, the data samples often follow a long-tailed distribution, which poses a great challenge for Federated Learning (FL). That is, when the data is decentralized and long-tailed, FL may produce poorly-hehaved global model that severely biased towards the head classes with the majority of the training samples. To settle this issue, decoupled training has recently been introduced to FL. Decoupled training aims re- re-balance the biased classifier after the normal instance instance-balanced training, and has achieved promising results in centralized long-tailed learning. The current study directly adopts the decoupled training idea on the server side by re-training the classifier on a set of pseudo features, due to the unavailability of global balanced dataset in FL. Unfortunately, this practice restricts the capacity of decoupled training in federated long-tailed learning as the low- quality pseudo features lead sub-optimal classifier. In this work, motivated by the distributed characteristic of FL, we propose decentralized decoupled training mechanism by leveraging the abundant real data stored in the local. Specifically, integrate the local real data with the global gradient prototypes form the local balanced datasets, and thus re-balance the classifier during the local training. Furthermore, we introduce a supplementary classifier in the training phase to help model the global data distribution, which addresses the problem of contradictory optimization goals caused by performing classifier re-balancing locally. Extensive experiments show that our method consistently outperforms the existing of methods in various settings. Our code available at atttpps: ht//ttttus////// com/ wttaa//eveee9801110 F/ Learning Exper iimentmmtsss

---

Published in Transactions on Machine Learning Research (04/2024)
1 Introduction
Federated Learning (FL) McMMaan et al. 2017) is proposed as an effective distributed learning framework to enable local clients to train a global model collaboratively without exposing their local private data to each other. However, the performance of FL is heavily hindered by the lingttaile/cclass lall--aablalneeedd data distribution phenomenon in the real world. That is, the global data distribution (i.e., the data distribution of the training samples merged from all clients' local data) usually shows long-tailed pattern (Zhang al. 2021; Wang et al. 2021). where head classes occupy a much larger proportion of the training samples than tail classes. Applying FL directly on such long-tain tailed data will produce a poorly performing global model that is severely biased to the head classes (Wang al. 2021
Dealing with FL on the non-i.id. and long-tailed data is challenging in two aspects: First, as the data samples are not identically and independently distributed (non-i.i.d.) across different clients in FL (McMahan et al. 2017). the local data distributions (i.c., local imbalance) show inconsistent long-tailed patterns compared to that of the global data distribution (i.e., global imbalance) (Wang et al., (021). Thus, tackling the local imbalance problem only (e.g.. Fed-Focal Loss (Sarkar al. 2020) will not help address the global imbalance problem in FL. Second, considering the data privacy concern, it is infeasible to explicitly obtain the imbalance pattern of the global data distribution from
the local data information. This further limits the application of the global class-level re-weighting strategy (Cui al.
2019
Decoupled training is first proposed in centralized long-tailed learning (Kang et al., 2019 Zhou et al. 2020), which disentangles the long-tailed learning into the representation learning phase and the classifier learning phase. These works (Kang et al. [2199] Zhou et al., 2020) find that the eettace-ballanee training (i.e., uniform sampling on the entire training set to make the contribution of each sample the same) leads to well-learned representations but biased classifier. Thus, they propose to re-train the classifier on a small balanced dataset after the instance-balanced training and have achieved promising results. We believe the decoupled training idea is also suitable for tackling the class-imbalanced problem in FL. as it can adjust the biased classifier at the global level without the necessity of obtaining the global imbalance pattern.
Nevertheless, applying decoupled training into FL faces a great challenge that there lacks a public balanced dataset for re-training classifier due to the data privacy concern. A recent study CReFF (Shang et al., 2022b) directly performs decoupled training on the server side, by re-training the classifier a set of pseudo features created the server. However, we argue that adjusting the classifier in such centralized manner neglects the decentralized characteristic of FL that the high-quality training data is actually stored in the local, and only produces sub-optimal classifier caused by the poor-quality pseudo features with high similarity per class.
In this paper, we propose the decentralized decoupled training mechanism to realize the full potential of decoupled training in federated long-tailed learning. Our main idea to fully utilize the abundant real data that only stored in the clients. Therefore, we are encouraged to allow clients to re-balance the classifier during local training. Specifically, we make each client re-balance the classifier a local balanced dataset that is mixed with the local real data and the global gradient prototypes of the classifier sent by the server, while the latter is supposed to address the issue of missing classes in the local datasets caused by the non-i.i.d. data partition issue. In this case, the classifier re- re-balancing is performed on the client side and in a decentralized way. Moreover, we add a supplementary classifier in the training phase to jointly model the global data distribution. This practice helps to overcome the optimization problem brought by the practice of local classifier re-balancing- Compared with CReFF, our fully-decentralizzed paradigm allows the clients to collaboratively train a balanced classifier with their sufficient local real data during local training, which needs extra requirements on the server and produces an optimal classifier with better generalization ability. We conduct extensive experiments three popular long-tailed image classification tasks, and the results show that our method can significantly outperform all existing federated long-tailed tailed learning methods in various settings.
All in all, our contributions can be summarized as:
We analyze the challenges of directly applying decoupled training, which has achieved great success in centralized long-tailed learning, into tackling long-tailed global data problem in FL setting. Motivated by
I Also, class-level weightghting practice itself an improper solution according to the fact that will do harm the representation learning phase Kang al. 2019) Zhou al. (200).
2

---

Published in Transactions on Machine Learning Research (04/2024)
the distributed characteristic of FL, we propose novel decentralized decoupled training idea to effectively re-halanced the biased classifier during local training stage by making fully use of the abundant local real data.
Specifically, we first formulate the optimization target into constrained optimization problem, in which we introduce supplementary classifier in the training phase to address the optimization difficulty of directly fitting one classifier on two different data distributions.
We apply the method of Lagrange Multipliers to solve the above optimization problem, in which we re-balance the original classifier on local balanced dataset mixed with local real data and global gradient prototypes, while the latter is introduced to address the issue of missing classes on client's local data due the non-i.i.d. data partition issue.
We conduct extensive experiments in various settings to validate the effectiveness of our proposed method, showing our improvement on fully unleashing the potential of decoupled training on tackling the federated long-tailed learning problem.
2 Related work
2.1 Federated learning
Federated Averaging (FedAvg) (McMahan et al. 2017) is the most widely-used FL algorithm, but it has been shown
that the performance of FedAvg drops greatly when the data is non-i.idd (Karimireddy al. 2020b). Therefore, plenty
of existing FL studies (Li et al. 2018 Acaar et al. 2020 Karimireddy et al., 2020b; Hsu et al. 2019 Reddi al. 2020
Karimireddy et al. 2020aa] Yang et al. 2023) target dealing with the non-ii.d. data partitions in FL. However, these
studies neglect another realistic and important data distribution phenomenon that the data samples usually show a
long-tailed pattern.
2.2 Long-talled/imbalaneed learning
In the real world, the data points usually show long-tailed distribution pattern. Therefore, learning good models on the mang-tailedlllss-mbblla. imbalance data has been widely studied (Zhang al. 2021 in the traditional centralized learning. and attracts more and more attention in the FL setting.
Centralized Long-Tailed Learning The methods in the centralized long-tailed learning can be mainly divided into three categories: (1) Class-level re-balancing methods that includes over-sampling training samples from tail classes (Chawla et al., 2002). under-sampling data points from head classes (Liu et al. 2008), or re-weighting the loss values or the gradients of different training samples based on the label frequencies (Cui et al. 2019 Cao et al., 2019)
or the predicted probabilities of the model (Lin al. 2017 (2) Augmentation-
methods aim create more data samples for tail classes either from the perspective of the feature space (Chu al. 2020 Zang et al.) 2021) or the sample space (Chou et al.) 2020). (3) Classifier re-balancing mechanisms are based on the finding that the uniform sampling on the whole dataset during training benefits the representation learning but leads to biased classifier, so they design specific algorithms to adjust the classifier during or after the representation learning phase (Zhou et al. 2020
Kang et al. 2019).
Federated Long-Tailed Learning Recently, some studies begin to focus on the class imbalance problem in FL. Fed-Focal Loss (Sarkar et al.] (2020) directly applies Focal Loss (Lin et al. 2017) in the clients' local training, but it neglects the fact that the local imbalance pattern is inconsistent with the global imbalance pattern. Ratio Loss (Wang et al. 2021) utilizes an auxiliary dataset on the server (which is usually impractical in real cases) to estimate the global data distribution, and send the estimated information to clients to perform class-level re- re-weighting during local training. CLIMB (Shen et al. (02T) is proposed as client-level re-whighting method to give more aggregation weights to the clients with larger local training losses. However, both Ratio Loss and CLIMB bring negative effects to the representation learning due to the re- ce-wiiihting practice (Kang et al.) 2019). thus the improvement brought by them is limited. FEDIC (Shang et al., 2022a) also needs the impractical assumption to own an auxiliary balanced dataset and amount of unlabeled data for fine-tuning and performing knowledge distillation on the global model on the server. Most recently, CReFF (Shang et al. 2022b) adopts the decoupled training idea to re-train the classifier the server by
3

---

Published in Transactions on Machine Learning Research (04/2024)
creating number of federated features for each class, and achieves previously state-of-the-an performance. However, the low quality and the limited quantity of federated features restrict its potential.
Some personalized federated learning (PFL) studies (Marfoq et al. 2021 Wu et al. 2023) can also mitigate the global long-tailed data issue to some extent from another perspective. However, we point out the optimization target and evaluation protocol of PFL are different from that our paper. PFI. usually makes each client hold personalized parameters in the local, leading different local models instead of unified global model. Then, each local model will be evaluated on testing samples from each client's local data distribution. While in setting (McMahan al. (001)) all clients participate learn well-behaved model on the global long-tailed data and this global model will be evaluated on all testing samples. Then, in this setting, the global long-tailed data distribution issue has a severe impact on the generalization ability of the global model, which is the major problem we aim to solve in this paper.
3 Methodology
3.1 Problem definition
In FL, each client & (k = ...................... good global model. Then, the optimization goal FL. can be formulated as
... N) has its own local dataset Dx. and all clients form a federation to jointly train a
D Σ1=1 D
0 arg min T(0)- arg min
T(00D*))
where [Da] represents the sample quantity of Da. Dx) is the local training objective in client k.
Federated Averaging (FedAvg) (McMahan et al. 2017) is the most popular FL framework to solve the above optimization problem. Specifically, each round, the server sends the latest global model 0' to sampled clients k E C², and each client & performs multiple updates on 0t with its local dataset Dj and gets the new model 0 Then only sends the accumulated gradients g% 0 0f back to the server, which aggregates the collected gradients and updates the global model as:
D2
0--1 7
D, gi.
(2)
C
Ziec
KEC
where 7/s is the server learning rate, |C*| is the number of clients participating in the current round. In this paper, we study the optimization problem of FL in which the global data distribution Uk Dx is long-tailed.
3.2 Motivation
Previous studies in centralized long-tailed learning (Kang et al. 2019) propose to decouple the training on the long-tailed classification tasks into representation learning and classifier learning phases, and point out that performing class-level re-weighting rather than instance-balanced training brings negative impact on the representation learning, and the imbalanced data distribution mainly affects the classifier learning. Then they achieve significant improvement by re- balancing the classifier learning (Shang et al., 2022b). Thus, our main idea is to effectively re-balance the classifier when dealing with the long-tailes tailed global data in FL. However, different from the centralized training, there is a lack of the global balanced dataset in FL for re-balancing classifier. Then, being aware of the data-decentralized property of FL, we are motivated to make clients re-balance the classifier locally during training by taking great advantage their abundant local real data.
balanced dataset after training. This finding is also verified in federated long-tailed
3.3 Optimization target
We split the original model architecture/paramettes 0 = (P, W) into two parts: the representation encoder and the classifier W. and aim to re-balance W during the local training to make it behave well on the blass-balanced data distribution Dool This creates the optimization target Teon as:
(PP.W:U. U, Dz). s.t. L(W; P, Dhal) min L(Wp; P, Dbub)) P.Dbut)
Too
min L(P, W: P.W.
(3)
Wr
4

---

Published in Transactions on Machine Learning Research (04/2024)
Representation Learning
Forward Propagation
Classifier Re-Balancing
Forward Propagation
Client Uploads
Propagation
Back Propagation
Server Broadcasts
Local Training on Client
Server Aggregation
& Ic Lx}
Bird
Lion
Cat:
Bird:
Absent
Absent
9 Swe. 9Wr.
[gw.ke
Dog:
Re-Balanced Classifier We
9 Oiiiird 9 Wilio
pro
Local Balanced Dataset bul
L(Px wx: )
Average and Update
P.W. W. W
L(P WR. Dw)
Inference
Local Dataset Dx with Label Set fg
Encoder Px
W
Cat
Global Classifier W
Figure 1: Full illustration of our method. We add a new global/supplementary classifier W, and perform instance- balanced training on both the encoder and two classifiers. Additionally, we only re-balance the original classifier during local training on a local balanced dataset mixed with local real data and global gradient prototypes {gw.ch In inference, we only keep the encoder P and the re-balanced classifier W.
where the first component before semi-colon in L(::-) is the variable while the latter is the constant condition The first part of Eq. (3) represents that based on FedAvg, the whole model trained under the global data distribution. As for the second part, if we want to further re-balance the classifier W, the re-balanced classifier should also behave well on the balanced data distribution given the encoder fixed. However, when the global data distribution = Uk Dx is long-tailed. Eq. 3 faces severe optimization difficulty, as naively using D and Dhal alternately optimize the classifier produces contradictory optimization directions and leads to poor solution of W
To address the problem of distinct optimization directions for W brought by D and Dial, we design an architecture of the two-stream classifiers by adding supplementary classifier W in the training phase (refer to Figure I in order to help model the global data distribution D and make re- re-balancing W feasible. We explore the necessity of introducing We then re-formulate our global optimization target as:
this supplementary classifier in Section 5.1
U. T= min L(P,W, W. Dx), s.t. L,W;P, P.W.W
= min Li: Dbal Wr
(4)
By making the combination of two classifiers model the global data distribution in the first part Eq. 4 the imbalanced pattern in D can be largely fitted by W and the original W is expected to be mainly fit the balanced data distribution Dbal, thus the optimization problem W is addressed. Furthermore, it makes sure that the representation encoder is trained under the instance-balanddd learning most. In the following, we introduce our proposed algorithm to solve Eq. (4) from three aspects, including the local training, the server aggregation, and the inference stages. The full illustrations of model architecture and training process are displayed in Figure I
training paradigm. which benefits the representation
For example, L(A; B) contains the variable A and the given condition B.
There indeed exists perfect classifier that achieves 100% classification accuracy all classes, in which case behave perfectly both D and Dial However, obtaining this solution is non-trivial from Eq. (3) because and Doat provide distinct optimization directions for W. We also consider another alternatiee two-stage classifier re-halancing paradigm Section but the results show that it much ineffective than following introduced joint optimization paradigm.

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 1 Local Training Process of RedGrape
Input: Round number local data Dj: with local label CR. local model (Pk. Wk. Ws) initialized as received global model (P(-1, W Wi-1), global gradient prototypes ( E L}.
1: Calculate local gradient prototypes (ow',,cc E L based on Eq.
2: for Local step i 1.2,
do
3: Update P and W based on Eq. 7
4: Compute batch gradients for W based on Eq.
5: Compute re- balancing gradients for W based on Eq. 9 Eq. 10 and Eq.
Update W based on Eq. (3).
end for
return Local gradients (P) P
W
W
W W ) local gradient prototypes (g k. L+).
3.4
Classifier re-balancing by integrating local real data with global gradient prototypes
3.4.1
Local training stage
In the local training, according to the basic idea of FedAvg framework, each client aims solve the sub-problem of Eq.
Th
min W. (.PWW.WDD s.t. L(W; P, D min L(Wp; P, Dbubl) P.W.W
(5)
It is a constrained optimization problem that is non-trivial to solve, therefore, we manage to address it by applying the method Lagrange Multipliers. That is, we turn it into the following unconstrained optimization problem by adding a penalty term on solving W:
The min Ca(P,W,W W) P.W.W
(6)
(:: + ((W: min L(Wp; P. )bbbl)) }
min P.W.W
WP
After choosing a proper A, all parameters can be optimized by taking the derivative of Lı over them and performing the normal gradient decent algorithm.
An overall look Before introducing technique details, first give overall look of the whole process of local training in our method Algorithm To be brief, the encoder parameters P and the supplementary classifier W will be trained under instance-balanced manner (Line 3). When updating the original classifier W. besides the gradients of the batch samples from D (Line 4), method creates local balanced dataset Diat to help re-balancing W (Line 5-6) following the second target of Eq. (6. The detailed steps include the following parts:
Updating and W, and calculating local batch gradients for W the t-th round, we perform the batch stochastic gradient decent mechanism" to update and W That is, for the local step i = =1,2 2.... I, a random batch of examples Bj is sampled from D; to perform that:
P P
mp
V P L(P W W - (,))) T V L(P w W :))))
W W
!h the local learning rate. One important thing is, when calculating the above loss on each sample (æ, y), the representation vector h = f(x; P) will be first fed into both two classifiers and get two logits WTh and W h. Then
'erre, SGD an example, do have the assumption the type local optimizer. For simplicity, omit hias term here, while method still applicable when hias exists.

---

Published in Transactions on Machine Learning Research (04/2024)
perform the feement-wise addition to get the final logits W + W h, and use z for the loss calculation. Also, in the same forward and backward propagations, we can obtain the first part of gradients for W in Eq. (6 as
9 Local W V L:- W :
Calculating re-balancing gradients for W For the second part of Eq. 6 it needs calculate the gradients of Wi-1 on a small balanced set Diat, which is supposed to be created in the local. However, there exists difficulty in constructing Dial from Dx. since it is very likely that there are some classes missing in the local label set La of D) due to the non-i.i.d data partitions. Thus, we propose a mixed gradient re-balancing mechanism to overcome this challenge by Ratign local rEal Data with Global gRAdient prototy PES (RedGrape as our method). Specifically, for each class c:
(1) If the sample quantity of class in Dj reaches a threshold T, we think client i has sufficient samples of class in its local dataset, and randomly samples T training samples of class e (which can be different across rounds) form D Then, the re-balancing gradients contributed by class are
9
bal 9
V W
LWW k P )...).
(2) If client & does not have enough samples in class c, we choose estimate the gradient contribution of class e with the global gradient prototype - of class e in the (t - 1)-th round which is the averaged gradient by training samples of class c w.r.t. W across selected clients in the last round (Shang et al. 2022b):
pro 9
C²-1
-
(10)
kEC
pro kw, = Vw T. p'-2 Dk.c),
(11)
where Ch-1 represents the set of clients that are sampled in the (t 1)-th round and have the training samples of class c, and Dx.e denotes all training samples of class c in Dr. Thus, it requires each client sampled in the previous round to first calculate the local gradient prototype of each class E Ls on the same model (Pt-2, (P-)) then return ( lc E } back to the server along with other local gradients. The server will average and update the global gradient prototypes, and broadcast them in the current round i re-balancing W are
Based on Eq. and Eq. (10). the gradients for
Σ
ww:-
C
-we-2,0 20)
(12)
Cbul
where Chal C Lb is the label set in which each class contains at least T samples and C is the entire label set.
Updating W According to the form of Eq. (6). the final updating rule for wii- i
W W m [ow!! tocel >g (lllggg tocal
Il/llo
bast |1)]
(13)
In Eq. (3), we normalize the scale of - at each step, by making its scale consistent with the decreasing trend of the scale of real gradients during training. The reason is, - is constant used to calculate g and update wil! while the scale of local gradients g/ decreases during the training, it will do harm to the training when pro
gw 2,c becomes the dominant part on updating the model for consecutive steps.
After training, the new model is (Pr, Wt. k: W and client k sends the local gradients (gp: 1.:-1..::W---.. Sww- -
(P - P' W
W
W along with the local gradient prototypes (o u. E Lx) to the server.
SWe discuss about the impact of utilizing previous gradients information in current round in Appendix A TW discuss about the limitation of extra communication caused by transmitting global gradient prototypes in Appendix B 5 Sminw L(Wp; Dbol) constant when taking derivatives over W.
7

---

Published in Transactions on Machine Learning Research (04/2024)
3.4.2 Server aggregation stage
The server first aggregates the gradients and updates the global model as
[Dxl
P W W - (P w W(-1) 7)s Σ
(14)
kEC [Dj]
Eiec
Also, the server needs to update the global gradient prototypes as
-wi-1 hho
C. 0,
CC
pro -
RECS
(15)
pro Iw-
=
in which the second case represents that all clients in in the current round do not contain samples of class c. this case, we re-use the global gradient prototype of class e from the last round. The updated global model and global gradient prototypes broadcast the sampled clients in the next round.
3.4.3 Inference stage
After the federated training, we only keep the re-balanced classifier W and abandon W during inference:
3/pred = arg max T f(x; P)].
(16)
3.4.4 Transmitting local/global gradient prototypes in privacy-preserving manner
As discussed above, our method requires the clients to uploading local gradient prototypes to the server and also requires the server to broadcast the updated global gradient prototypes to clients along with new model parameters in the next round. We point out that this transmission can achieved in srivacy-preservinn manner and will not expose the local data privacy. Specifically, when uploading global gradient prototypes, we can use as Homomorphic Eneryption to allow the server only get the encrypted average of global gradient prototype of each class instead of the single local gradient prototype from each client, which further enhances the protection of the local privacy. The steps are the following:
methods such
Create a secret key that is only known by clients.
Clients encrypt the gradients and the gradient prototypes with the secrete key, then upload them the server.
The server only calculate the homomorphic average of the local gradients and gradient prototypes, but can not obtain the original values of gradient information as it does not know the secrete key. Then, the server broadcasts the updated information to the clients in the next rounds.
After receiving the encrypted global information from the server, the clients decrypt the information in the local with the key, and use it to perform the local training.
4 Experiments and analysis
4.1 Experimental settings
Datasets and Models We conduct experiments on three popular image classification benchmarks: MNIST (LeCun) et al. 1998), CIFAR-10 and CIFAR-100 (Krizhevsky et al. 2009). We also conduct extra experiments large-scale realistic federated dataset FEMNIST (Cohen et al.) 2017) and put the detailed results in Appendix H We follow existing studies (Cao et al, 2019 Shang al. 2022b to create the long-tailed versions of training sets of above three datasets (i.e., MNIST-LT, CIFAR- 10R1-1///00-LY and keep the test sets as balanced. We first define the term Imbalance
max. (nc), which is the ratio between the maximum sample number across all classes and the minimum Ratio: IR min, (R.).
8

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Results under the full client participation setting. We report the overall test accuracy and standard deviation the balanced testing set of each dataset.
MNIST-LT
CIFAR-I0-LT
CI0AR-100-LT
Method
IR
10
100
10
IR
50
IR
100
IR
10
IR
50
IR
100
FedAvg+CE
97.99.1 0.02)
95.98(1 0.03)
92.71(1 0.50)
76.21(4 0.76)
68.41c=
0.54)
59.83 0.52)
49.08L1 0.22)
36,47(1 0.56)
33.28 0.28)
Fed- Fed-Focal Loss
97.90tt 0.03)
96.
0.08)
92.97t 064)
77.92( 0.60)
61.21c=
191
59.86 0.49)
48.14 0.26)
35.51c4 0.70)
30.05 0.81)
Ratio Loss
97.96c1 0.04)
96.20ct
0.04)
92.99(1
036)
08))
68.01c=
048)
59.27c2 0.47)
48.30 0.16)
37.62(1
31,9 $1,92:1 0.38)
CLIMB
97.89c1 0.03)
35.87(t 0.06)
92.71(1 0.47)
78.95(2
66, 26.25kc
57.67ct 1.06)
49.27(1
36.13(10.24)
32. 18:1:
CReFF 97.68c1 0.03)
96.49(1
0.03)
93.85(1 0.35)
83.18(2 0.27)
73,46
69.36c1 0.32)
46. 16.58 0.42)
35.82(1 0.67)
33.46:1
Ours 98.34.1 0.02)
97.06 0.03)
95.73(L 0.46)
83,74 dL 0.20)
74.01 0.31)
71.04 0,46)
51.09(1 0.13)
38.49tL 0.24)
34.63 d 0.29)
sample number across all classes, to reflect the imbalance degree of the global data distribution. Then, the training sample quantity of each class follows an exponential decay. We choose IR 10, 50, 100 in our main experiments. We also conduct experiments in another binary class imbalance setting (Shen et al. 2021), the details and results are in Section the Dirichlet distribution Dir(a:) for the non-i.i.d. data partitioning, in which a controls the non-i.i.d. degree. We set a = 1.0 in our main experiments, and put the results on other as in Appendix E We use the convolutional neural network (CNN) (McMahan et al., 2017) for MNIST-LT, and use ResNet-56 (He ct al. 2016) for CIFAR-10/100-LT. More details are in Appendix
Furthermore, we follow the existing studies (Reddi et al.) 2020 Shang al. 2022b) to adopt
Baseline Methods We mainly compare method with the existing federated long-tailed learning algorithms, including FedAvg with the CrossEntropy Loss (FedAvg+CE) applied in the local training (McMahan et al. (211)) Fed-Focal Loss (Sarkar et al. 2020), Ratio Loss (Wang et al., (02T), CLIMB (Shen et al. (021). and the state-of-the-ar method CReFF (Shang et al. 20226). Also, we conduct extra experiments to compare with other FL methods that are only designed for tackling the non-i.i.d. data issue (i.e., FedProx (Li et al. 2018), FedAvgM (Hsu et al. (219)) and FedAdam (Reddi et al. 2020)), the results are in Appendix
Training Details We conduct experiments in two popular FL settings based on the ratio of clients participating in each round: (1) Full client participation setting: all clients participate in updating the global model in each round, and the total number of clients is 10; (2) Partial client participation setting: the total number of clients is 50 but only 10 clients are randomly sampled in each round. We also conduct experiments with a larger number of total clients (i.e., 100), the results are in Appendix G We adopt SGDM as the optimizer for local training. The local learning rate is 0.01 for MNIST-LT and 0.1 for CIFAR-10/100-11 10/100-LT. The number of local epochs is 5 for all datasets. We set different total communication rounds in different setting considering the different convergence speeds of the global models: (1) In the full client participation setting, the number of communication rounds is 200 for MNIST-LT, and 500 for CIFAR-10/ 100-LT (2) In the partial client participation setting, the number of communication rounds is 500 and 1000 for MNIST-LT and
separately. As for our method, the re-balance factor is fixed as 0.1 in all experiments, and we explore the effect of different values of in Section 5.2 The quantity threshold T for each class to create the local balanced dataset is set as 8 for MNIST-LT and CIFAR- and for CIFAR-100-LL, and we put further discussion about this hyper-parameter in Section 5.3 Each experiment is run on random seeds. Complete training details (c.g., detailed Mhyerr-pramedte of baselines) in Appendix C.2
4.2
Main results
In the main paper, we report the averaged accuracy over the last 10 rounds (with standard deviation) on the balanced testing set in each experiment following Reddi et al.] (2020). We also display the averaged test accuracy on tail classes in each setting in Appendix tj show that our method can significantly bring improvement to the model's performance on tail classes. The overall results on the balanced testing sets under full client participation setting are in Table I and Table 2 displays the results under partial client participation setting. We can draw the main conclusion from these tables as: our method consistently outperforms the existing algorithms in all settings.
As we can see, Fed-Focal Loss underperforms FedAvg with CE loss in some settings, which validates the claim that directly applying the centralized long-tailed learning methods can not help address the global class imbalance
9

---

Published in Transactions on Machine Learning Research (04/2024)
Table 2: Results under the partial client participation setting.
We report the overall test accuracy and standard
deviation on the balanced testing set of each dataset.
MNIST-LT
CIFAR-10-LT
CIFAR-
Method
IR 10
100
IR
50
IR
FedAvg+CE
95.51 0.20)
91.8214 0.17)
89.92(1 0.0)
60.38(4
0.36)
45.15k= 0.39)
40.06,1 0.96)
40.81(4 0.19)
24.62 063)
22.08(a: 0.83)
Fed- Fed-- Loss
96.79;1 0.11)
92.59 0.14)
90,45ca 042)
61.16(# 0.44
46.20x= 0.34)
41.10:1 0.82)
40.85 0.19)
24.73c4 046)
20.17(# 133)
Ratio Loss
95.17c1
91.10(2
0.36)
05))4(2051)
03997
44.22c
42.11c
40.96(1
24.12:1
23.06::
CLIMB
95.67c1
92.24(1
89.75(1 0.36
61.75(20.31)
46.91c2
42.02ct
40.64
23.99:1
21 4,.4
CReFF 96.29c1
94,16(1
92.16(1 0.17)
69.38(1 0.21)
60.52
55.63(1 0.60)
39.38
25,42 0.24)
24.77(1 0.50)
Ours
97.54 0.06)
95.17(ca 0.11)
93.61 0.15)
71.68 0.35)
61.42 = 0.30)
$7.11 0.52)
42,97 0.11)
27.73 0.29)
25.64 0.43)
problem FI., as it ignores the mismatch between the global and the local imbalance patterns. Ratio Loss and CLIMB apply the class-level re-weighting and client-level re-weighting ideas separately, and gain slight improvement over FedAvg. We analyze that the reason for the limited improvement lies in that though the re-weighting practice helps the model to focus data of head classes according to Kang et al. (2019). Moreover, the assumption of obtaining global auxiliary dataset makes Ratio Loss impractical in real cases.
the learning of tail classes, it not conducive to the representation learning the abundant
The great performance of CReFF helps to validates the effectiveness of the idea of classifier re-balancing. How- ever, the optimization of the federated features requires massive computations on the server (especially when the number of classes is large), and the federated features from the same class may converge to be similar. Thus, the re-trained classifier faces the problem that it may over- fit on the highly similar and small amount of the federated features, which is reflected in the poorer performance un- der smaller IR. Our method instead takes full advantage of the local real data supplemented by the global gra- dient prototypes to locally re-balance the classifier, and consistently outperforms all baselines by a large margin. Compared with CReFF and Ratio Loss, we do not have extra requirements except for the normal aggregations on the server, and produce a re-balanced classifier that has better generalization ability with the help of abundant real data.
Evaluction Accuaacy (K6
12
FodAvg CE
Fed-Foeei Loss
Ratio
CLIMB
CRaFF
Ours
250
330
Figure 2: The evaluation accuracy curve of each method in CIFAR-10-LT Our method achieves faster convergence speed and better performance than all existing baselines.
We further display the evaluation accuracy curve after each round in CIFAR-10-LT (IR 100) under the full client participation setting in Figure 2 As we can our method not only has the best converged performance, but also achieves much faster convergence speed than all baseline methods. That is because our method re- re-balances the classifier at each local training step, and this makes it converge faster to the optimal balanced classifier.
4.3 Results in another class imbalance setting
We also conduct experiments in binary class imbalance setting in FL Shen al. (2021). in which three classes randomly chosen the tail classes, and they assigned with 1/IR number of sampled compared with other normal/head classes. The experiments are conducted on MNIST-LT and CIFAR-10-LE datasets with IR =100, and other experimental settings are kept as the same as that in our main experiments. The results are in Table 3 The conclusion remains the same that, our method achieves the best performance in all cases.
5 Further explorations
In this section, conduct ablation studies and make further explorations of our method.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Table 3: The overall evaluation accuracy in the binary class imbalance setting. IR 100 and the randomly chosen tail classes are 0, 7 and 8.
Full Participation
Partial Participation
Method
MNIST
CIFAR-I0
MNIST
CIFAR-10
FedAvg+CE
93.91
70.91
91.97
61.27
Fed-Focal Loss
93.53
70.77
91.77
59.18
Ratio Loss
94.23
73.22
92.32
62.86
CLIMB
93.89
72.04
92.11
63.18
CReFF
96.70
78.98
96.01
71.41
Ours
96.86
79.88
96.43
73.31
80
40
31-
15-
-2
Evaluaton -2
TedAng CB
TedAsg CB
Ours wio Extr Classifier
Curs wis Extra Classifer
(a) Results CIFAR-IO-IA
(b) Results CIFAR-100-LT.
Figure 3: The positive effect of adding supplementary classifier in the training phase to help model the global data distribution when locally re-balancing the classifier by our method.
5.1 Supplementary classifier is crucial for effective classifier re-balancing
Here, we conduct experiments on CIFAR-10/100-1 to explore the necessity of introducing supplementary classifier to address the optimization difficulty caused by performing classifier re-balancing locally. The results are displayed in Figure "FedAvg+CE" the naive baseline, which can also considered as an ablation situation in which we add the supplementary classifier W but without further re-balancing on W. "Ours w/o Extra Classifier" represents the situation where do not add W but still re-balance W based on Eq. (33 We observe that if we do not add the supplementary classifier in the training phase, the model will converge than that if we adopt the two-stream classifier architecture. This helps to validate our motivation and the great effectiveness of introducing a new global classifier to address the optimization difficulty brought by the local classifier re-balancing practice.
bad local optimum and behave much worse
5.2 Re-balancing strength decides on the convergence trade-off
In order to solve the optimization target of Eq. (5), we consider to turn Eq. 5 into an unconstrained optimization problem by adding a penalty term on the local objective function as Eq. (6) and update W as Eq. (I3). where re-balance factor À is used to control the re-balancing strength. Here, conduct experiments to explore the effect of different Às the model's performance, and the results are shown in Figure We find that the smaller À results in slower convergence speed but obtains relatively better performance of the converged model. We analyze the reason lies in that, the - contains part of global gradient prototypes calculated in the previous round and is a constant when updating W. It will adversely affect the model's convergence in the late stage of the training when we are still using a large À to re-balance the classifier. An interesting direction to improve our method is designing an adaptive that decays along with the training, which we leave future work. When 0.0. the addition of classifiers equals normal classifier used in FedAvg, FedAvg is special case of method in this case.
bal
11

---

Published in Transactions on Machine Learning Research (04/2024)
72
Evaluation Accuracy (%)
70.91
71.03
71.04
70.66
70.39
1-1.00
0R
2-0.10
A-0.01
2-0.00
8
16
150
200
250
300
330
430
400
Sample Quantity Threshold Eaah Class
Communictionn Round
Figure 4: The test accuracy of using different re-balance factor À in CIFAR-IO-LT. Smaller leads to slower con- vergence speed but relatively better generalization ability,
Figure 5: The test accuracy of using different sample quantity thresholds for each class when creating the local balanced datasets in CIFAR-IO-LT with full client partic- ipation.
Table 4: The comparison between the joint optimization paradigm adopted in our main paper and an alternative two-stage optimization paradigm. 100.
Full Participation
Partial Participation
Method
CIFAR-IO-LT
CIFAR-100-LT
CIFAR-IO-IT
CIFAR-100-LT
Two-Stage Optimization
65.74
32.38
52.28
21.85 25.64
Joint Optimization (in our method)
71.04
34.63
57.11
5.3 Local real data plays an important role in re-balancing the classifier
During creating the local balanced datasets, we set a threshold to decide whether each client owns the enough data of specific class. Larger T decreases the number of classes which the real data can be used calculate local gradient prototypes in Eq. (9) while smaller T leads to the relatively unreliable gradients of class c. We then explore the effect of different Tss CIFAR-1O- and put the results in Figure 5 We indeed observe a trade-off pattern expected and find that 7 = 4, 8 are generally proper choices. T = means we remove the role of local real data on the classifier re-balancing and only use the global gradient prototypes instead, and we find the performance degrades greatly, which verifies the large benefits of using local real data to adjust the classifier.
5.4 The joint optimization paradigm outperforms the two-stage optimization paradigm
In proposed method, choose re- re-balance the classifier during the local training. That is, the representation learn- ing and the classifier re-balancing are performed jointly, which is different from the decoupled training method (Kang et al. 2019) in the centralized setting. The reason is, unlike the centralized setting, the data is only kept in the local clients in FL. Thus, in practical, there is no global balanced dataset on the server to perform the two-stage training by re-training the classifier after representation learning. CReFF (Shang et al.] 2022b) then creates some pseudo features in the server for re-training the classifier after each round's aggregation. However, we point out that the low quality and limited number of pseudo features only produce a sub-optimal classifier. Therefore, we are encouraged put the classifier re-balancing along with the local training by fully utilizing the local real data, and propose such joint optimization paradigm.
Also, there exists another solution to locally re-balance the classifier: train the entire model using standard FedAvg for sufficient rounds, then freeze the encoder and only re-train the classifier under FedAvg framework. Therefore, we conduct extra experiments with this choice by freezing the encoder trained by the bascline FedAvg and further re-training a new balanced classifier following our RedGrape mechanism with extra 200 rounds until converged. The results Table Isggest that this two-stage training choice with more communication costs still underperforms the joint optimization paradigm adopted in our main paper, validating the effectiveness of our original motivation.
12

---

Published in Transactions on Machine Learning Research (04/2024)
We think the reason is, Eq. 6 formulates integrated optimization target that requires synchronous updates for all parameters, while this separate two-stage training practice will lead to sub-optimal solution. Moreover, above practice will take more communication rounds for extra classifier re-training, which will cause an unfair comparison with other baselines.
6 Conclusion
In this paper, we propose decentralized decoupling mechanism to effectively re-balance the classifier for tackling federated long-tailed learning. Motivated by the distributed characteristic of FL, we propose to re-balance the classifier during local training with the help of abundant local real data supplemented by global gradient prototypes. Furthermore, in order to address the problem of contradictory optimization goals brought by performing local classifier re-balancing, introduce two-stream classifiers architecture to help model the global data distribution. Thorough experiments verify the great effectiveness of our method over strong basclines without extra data requirements.
Broader impact statement
Our purpose is address the optimization problem of FL on the non-i.idd and long-tailed data and help to learn better global model that has good performance on all classes. The datasets used in our experiments are all publicly available. Also, our method only requires the normal gradients transmission between the server and the clients as other FL methods do, which will not expose the local data privacy (refer to Section 3.4.4).
Acknowledgments
We sincerely thank all the anonymous reviewers and the Action Editors for their great reviews, constructive comments and helpful suggestions. This work was supported by Tencent Research Grant. Xu Sun is the corresponding author of this paper.
References
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label- distribution-awanl margin loss. Advances in neural information processing systems, 32, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over- sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Hsin-Ping Chou, Shih-Chieh Chang, Jia- Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. In European Conference on Computer Vision, pp. 95-110. Springer, 2020.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In European Conference on Computer Vision, pp. 694-710. Springer, 2020.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (ICNN), pp. 2921-2926. IEEE, 2017.
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arxiv:2007.13518 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778. 2016.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for 2019.
federated visual classification. arXiv preprint
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.
Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2019.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mchryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint wrXiv:2008:03666 2020a.
Sai Pranceth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient based learning applied to document recognition. Proceedings of the IEEE, 86(11 (1:21)22272222 -2224,4, 1998.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv:/8/2.06/12 /8/2.06/27, 2018.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Xu- Ying Liu, Jianxin Wu, and Zhi-Hus Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2):539 -5.5,9955,,000 2008
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34:15434-15444 2021.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eficieent learning of deep networks from decentralized data, In Artificial intelligence and statistics, pp. 1273-1282, PMLR, 2017.
Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2020.
Dipankar Sarkar, Ankur Narang, and Sumit Rai. Fed-focal loss for imbalanced data classification in federated learning. arXiv preprint arXiv:2011.06688 2020.
Xinyi Shang, Yang Lu, Yiu-ming Cheung, and Hanzi Wang. Fedic: Federated learning on non-iid and long-tailed data via calibrated distillation. arXiv preprint arXiv:2205.0017 00172, 2022a.
Xinyi Shang, Yang Lu, Gang Huang, and Hanzi Wang. Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. arXiv preprint htt.:2204./3399 2022b.
Zebang Shen, Juan Cervino, Hamed Hassani, and Alejandro Ribeiro. An agnostic approach to federated learning with class imbalance. In International Conference on Learning Representations, 2021.
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10165-10173, 2021.
Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. In International Conference on Machine Learning, pp. 77860--77873 PMLR, 2023.
Wenkai Yang. Yankai Lin, Guangxiang Zhao, Peng Li, Jie Zhou, and Xu Sun. When to trust aggregated gradients: Addressing negative client sampling in federated learning. Transactions on Machine Learning Research, 2023.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long- tailed instance segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3457-3466, 2021.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596 2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbu: Bilateral-branch network with cumulative learning for long-tailed visual recognition. Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 9719-9728, 2020.
15

---

Published in Transactions on Machine Learning Research (04/2024)
A
The impact of utilizing previous gradient information on re-balancing current classifier and the corresponding convergence analysis
According to Eq. 6 and Eq. (9) when calculating the re- re-balancing gradients for W the ideal solution is to calculate the gradient w.r.t. W on real samples of each class in Diai. If so, the convergence can be well guaranteed by the property of the SGD algorithm. However, as explained in the main paper, due to the facts that there will be some classes missing in each client's local data and each client can only get the information from last round at the beginning of current round, it is unavoidable to use some previous gradient prototypes Iw: 2,e w.I.t. W 2 in the current round as an alternative of gool gw.r. re-balance W Fortunately, the similar form of using gradient information from previous rounds to help current round's model training has also been adopted in previous studies such as SCAFFOLD (Karimireddy et al., 2020b) with good theoretical guarantees. Specifically, the usage format of local bal control variate in SCAFFOLD is similar to that of re-balancing gradient -w in our method. Thus, the analysis on the convergence of SCAFFOLD can also be applied to show the convergence property of our method. Moreover, in our method, the re-balancing gradient -w contains a part of real gradients bul gw if current client contains sufficient real samples of class c, which means the eellaannning gradient in our method even more accurate and reliable than the local control variate in SCAFFOLD.
B The extra computational and communication costs of our method
At each optimization step in the local training. our method not only requires the complete forward and backward propagations through the entire model on the local batch (Eq. and Eq. (8), but also needs a complete forward propagation and an incomplete backward propagation w.r.t. the classifier W on the balanced batch (Eq. (9). Thus, our method will cause additional local computational cost compared with other baselines. However, notice that though our method requires additional computations on balanced batches, the corresponding backward propagation process is only performed on the final classification head and does not propagate to all model parameters, which reduces a significant portion of computational cost. Thus, the extra computational cost of our method is acceptable.
In our method, besides uploading the normal gradient of the local model, each selected client also needs to upload the additional local gradient prototypes w.r.t. the classifier W and the gradient w.r.t. the additional classifier W to the server. Also, the server will then broadcast the aggregated global gradient prototypes to the clients in the following rounds. This will cause extra communication costs between the server and clients compared with some baselines such as Ratio Loss and CLIMB, and the additional parameter transmission will increase linearly with the number of classes. We admit this is the potential limitation of our method. However, we think the extra communication cost and storage memory of the supplementary classifier and global gradient prototypes are the necessary costs to effectively improve global model's performance on long-tailed global data. Previous FL studies (Karimireddy et al. [2200b [Shang et al.] 20226; Karimireddy et al., 2020a) have proven that introducing extra gradient information in the local training can bring significant improvement, thus the achievement gained by these advanced methods is also caused by adding extra gradients and parameters. For instance, SCAFFOLD (Karimireddy et al. (0220b) requires an additional transmission of the local control variate for each client that has the same number of parameters of the original model. Furthermore, compared with previous state-of-the-art method CReFF (Shang et al. 2022b), our method does not introduce extra parameters/gradients, while consistently achieves better performance.
Finally, we make a numeric analysis on the maximum parameter transmission cost of each method on each dataset in each round, and put the results in Table The total parameter transmission cost of each method includes both the amount of parameters uploaded from each client to the server and the amount of parameters broadcast from the server to each client. We should point out that the above results are under the worst scenario in which each client needs to transmit the local gradient prototypes of all class to the server. In real cases, due to the non-i.i.d. issue, there will be large proportion of classes missing in the clients' local datasets, thus the newly introduced communication costs of our method will be much smaller than that in Table 5] As we can see, when the number of classes is relatively small (c.g., MNIST-LT and CIFAR- IO-LT,, the transmission cost of the newly introduced parameters is very small compared with the original FedAvg. For CIFAR- 100-LT, the maximum number of newly introduced gradient parameters in CReFF and our method is about 2.5 10% while the original base model ResNet-56 contains 6 x 10 parameters.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Table 5: The maximum number of gradient parameters that need to be transmitted between each client and the server by each method in each round on all datasets.
Method
MNIST-LT
CIFAR-10-LT
CIFAR-I00-LT
FedAvg+CE
2.400 X 106
1.183 X 106
1.229
106
Fed-Focal Loss
2.400 x 106
1.183 X 106
1.229 x 106
Ratio Loss
2.400 x 106
1.183 x 106
1.229 x 106
CLIMB
2.400 x 106
1.183 x 106
1.229
106
CReFF
2.426 x 106
1.234 x 106
6.368
106
Ours
2.428 106
1.239
106
6.420
10c
Detailed experimental settings
C.1 Datasets and models
Here, we introduce the datasets and the backbone models we used in our experiments. We choose three classical image
C classification tasks, including MNIST [eCun al., 1998). CIFAR-10 and CIFAR- 100 "oroo""oKKireeeeeeaaa
(Krizhevsky al. 2009).
We then follow existing centralized and federated long tailed learning studies (Cao et al. 2019 Shang et al. 2022b to create the long-tailed versions of the training sets of above datasets (i.e., MNIST-LT, CIFAR-10/100-LI7 Specifically, the long-tailed degree is controlled by a ratio called the Imbalance Ratio: IR= max=(ne). where n. represents the sample quantity of class e (0-indexed). Then, manage make the sample quantity of each class follow exponential decay trend:
mine [
The = no
c= c= 0, c......
(17)
As for the non-i.i.d. data partitioning, we follow existing studies (Reddi et al.) 2020 Shang et al.) (022b) to adopt the Dirichlet distribution Dir(a). Smaller a means the heavier non-i.i.d. degree. We choose a = 1.0 in our main paper, and we also put the results on different os in Appendix E
We use the same convolutional neural network (CNN) used McMahan et al. (017)) for experiments on MNIST-LT, and adopt ResNet-56 (He al. 2016 as the backbone model for -0/10--
C.2 Complete training details
Local training settings
We utilize SGDM the local optimizer in all experiments. The search grid for server learning rate is [0.5, 1.0, 2.0, 3.0] and the search grid for local learning rate is [0.01, 0.05, 0.1, 0.5). The final local learning rate is 0.01 for MNIST-LT, and 0.1 for CIFAR-10/100. local training is 64, and the number of local training epochs is 5. As mentioned in main paper, we perform experiments in both full client participation and partial client participation settings. We set different total communication rounds in different setting considering the different convergence speeds of the global models: (1) the full client participation setting. the number of communication rounds is 200 for MNIST-LT, and 500 for CIFAR- 1AAR-100100-L11 (2) In the partial client participation setting, the number of communication rounds is 500 and 1000 for MNIST-LT and CIFAR-10/1004 separately.
The tuned server learning rate is 1.0 for all settings. For all three datasets, the batch size for
Server aggregation settings
During server aggregation, we follow the same procedure as that in FedAvg to aggregate the collected local gradients in the current round, and update the global model with the averaged gradients. The server learning rate is tuned as 1.0 for all experiments. Furthermore, as for CReFF and method, the server needs update the global gradient prototypes
9Can be downloaded from http:://ttpnn...tttta locun. cow/exdt/miis// Can downloaded from https://wwww...c. wt.ooootoooddu krizleffarhtml

---

Published in Transactions on Machine Learning Research (04/2024)
(refer to Section 3.4 in main paper) by averaging local gradient prototypes. However, different from CReFF, we do not have extra requirements on the server to make it perform further optimization and training process.
Hyper-parameters of each method
Here, introduce the choices of syper-parameters used in each method in detail.
Fed-Focal Loss: Fed-Focal Loss (Sarkar et al. 2020) directly applies Focal Loss (Lin ct al. 2017) to the local training. The form of Focal Loss is
focol = -11
(1 - pr>" log(pr).
(18)
where pr the predicted probability of the sample corresponding to the ground truth class. We set r = our experiments.
Ratio Loss: Ratio Loss (Wang et al. 2021) applies the class-level re-weighting practice by first estimating the global imbalance pattern on the server with an auxiliary balanced dataset. Its form can be written as
Lratio = (a + (R)
(19)
where Loz is the traditional CrossEntropy Loss, R is the ratio vector that contains the relatively estimated sample quantity of each class the server, and / are two syper-parameters Thus, follow the original study (Wang al. 202T) to set the sample number of each class on the auxiliary balanced dataset to be 32, 1.0, B 0.1.
CLIMB: CLIMB (Shen et al. 2021) aims to perform the client-level re- re-weighting to up-weight the aggregation weights for the local gradients with larger local training losses, as the global model behaves poorly on these clients' local data. The hyper-paramenn follow the original setting to set e 0.01 for MNIST-LT and 0.1 for CIFAR-10/100-L11 set v) 2.0 and 0.1 for MNIST-LT and CIFAR-10/100-LT separately.
in CLIMB include a tolerance constant and dual step size np. In our experiments,
CReFF: CReFF (Shang et al. 2022b) needs to create a set of federated features on the server, of which the number per class is 100. Following the original setting, the optimization steps on the federated features is 100, the classifier re-training steps is 300. Further, the learning rate of optimizing the federated features is 1.1 for all datasets, and the learning rate of classifier re-training is kept the same as that used in the local training on that dataset.
RedGrape (Ours): Our method introduces two dyper-parameters: À for the classifier re- re-balancing strength, and ' for the sample quantity threshold of each class on creating local balanced datasets. We put the detailed discussions about these two hyper-parameters in our main paper. The recommended search grids for À are (1.0, 0.1, 0.01), and (2,4,8) for T.
Code and infrastructure
Our code is implemented based on the open- sourced FL platform FedML (He et al. 2020). We will release our code upon acceptance. Our experiments are conducted on 8 GeForce RTX 2080 Ti.
D Results on the tail classes in main experiments
In our main paper, we put the results of the overall accuracy on the balanced testing sets of each method, and we have that our method consistently outperform all other methods in all settings. Here, we put the averaged accuracy on the tail classes of each method. Specifically, define the tail classes the last 30% classes with the minimum sample quantity.
The results on the tail classes are in Table 6 and Table 7 As we can see, our method brings significant improvement on the tail classes in most cases. Also, we find that CReFF tends to achieve better performance on the tail classes when the imbalance degree larger. However, the performance of CRcFF on the overall testing sets is worse than our method according to the results in our main paper. This validates our analysis that, re-training the classifier on a set number of federated features on the server can indeed re-balance the classifier some extent, but it is likely to produce sub-optimal classifier that overfits on these limited number of pseudo features.
18

---

Published in Transactions on Machine Learning Research (04/2024)
Table Averaged evaluation accuracy on the tail classes under the full client participation setting in main experiments.
MNIST-LT
CIFAR-10-LT
CIFAR-100-L3
Method
IR
10
IR = 50
IR 100
10
IR = 50
IR
100
IR
10
IR
IR
100
FedAvg+CE
96.05
90.23
82.21
67.87
55.26
29.24
36.25
13.37
8.50
Fed-Focal Loss
95.84
90.47
82.62
74.94
47.37
33.86
34.40
12.53
6.49
Ratio Loss
96.04
90.76
83.01
71.27
55.79
33.28
34.55
15.35
7.98
CLIMB
95.70
89.93
82.01
73.68
55,86
30.95
35.81
13.31
8.65
CReFF
95.98
91.98
86.62
82.87
66.10
57.03
38.18
22.77
18.98
Ours
96.78
92.97
89.59
83.86
69.74
60.41
40.11
20.19
15.58
Table 7: Averaged evaluation accuracy on the tail classes under the partial client participation setting in main experiments.
MNIST-LT
CIFAR-10-LT
CIFAR-100-LT
Method
IR-
10
IR = 50
IR = 100
IR
10
IR = 50
IR = 100
IR = 10
IR = 50
IR
100
FodAvg+CE
89.75
80.05
74.35
51.99
24.03
2.64
29.65
8.55
5.74
Fed-Focal Loss
92.65
82.43
75.46
49.78
27.43
5.68
29.51
8.89
4.08
Ratio Loss
89.83
77.93
73.35
55.54
23.81
4.87
29.28
8.36
5.86
CLIMB
90.09
81.49
74.58
54.45
23.90
4.14
29.85
8.37
4.97
CReFF
92.87
88.94
83.75
74.81
62.67
45.30
34.38
17.46
16.58
Ours
94.82
90.06
86.38
75.71
63.58
43.92
35.34
17.91
13.39
Table 8: The overall accuracy on the balanced testing sets under different non-i.i.d. degrees. Our method consistently outperforms all baselines.
MNIST-LT
CIFAR-IO-LT
CIFAR-100-L.T
Method
0.1
1.0
10.0
0.1
1.0
10.0
0.1
1.0
10.0
FedAvg+CE
93.42
92.71
94.09
59.46
59.83
63.01
31.18
33.28
31.74
Fed-Focal Loss
94.16
92.97
94.21
53.47
59.86
61.06
31.72
30.05
30.18
Ratio Loss
93.50
92.99
94.34
59.30
59.27
61.82
32.60
31.92
31.93
CLIMB
93.80
92.71
94.23
61.10
57.67
61.46
31.64
32.18
32.45
CReFF
93.94
93.85
94.76
63.25
69.36
70.30
32.20
33.46
31.60
Ours
95.34
95.73
95.93
64.30
71.04
71.82
32.86
34.63
34.42
E Experiments under different non-i.i.d. degrees
In our main experiments, we fix the non-i.i.d. degree 1.0, in order to mainly explore the effects of different imbalance degrees. Here, we conduct extra experiments with = 0.1 and a 10.0, and fix the imbalance ratio IR = 100. We conduct experiment on MNIST-LT and CIFAR- 10 under the full client participation setting, and other experimental settings are kept as the same as that in our main experiments. We put the results the overall evaluation accuracy on the balanced testing sets in Table We can draw the main conclusion from the table that, our method can achieve the best performance under different non-i.i.d degrees.
19

---

Published in Transactions on Machine Learning Research (04/2024)
Table 9: The overall evaluation accuracy of more FL baselines MNIST-LT and CIFAR-IO-LT with IR 100 under full client participation setting.
Method
MNIST-LT
CIFAR-IO-LT
FedAvg+CE
92.71
59.83
FedProx
92.43
60.79
FedAvgM
94.51
57.81
FedAdam
95.06
58.45
Fed-Focal Loss
92.97
59.86
Ratio Loss
92.99
59.27
CLIMB
92.71
57.67
CReFF
93.85
69.36
Ours
95.73
71.04
Table 10: The overall evaluation accuracy MNIST-LT and CIFAR-IO-LT with IR- 100 under full client participation setting.
Table 11: The results on the bal- anced FEMNIST test set.
Method
MNIST-LT
CIFAR-IO-LT
Method
FEMNIST
FedAvg+CE
87.06
32.68
FedAvg+CB
70.34
Fed-Focal Loss
87.56
33.04
Fed-Focal Loss
70.79
Ratio Loss
87.70
33.60
Ratio Loss
70.84
CLIMB
87.46
34.38
CLIMB
70.36
CReFF
90.56
42.25
CReFF
71.13
Ours
92.03
43.78
Ours
74.28
F Results on FL methods only designed for non-i.i.d. data issue
In our main experiments, we choose the methods that are specifically designed for tackling long-tailed global data distribution as baselines. Here, we make further comparison between our method with popular FI. methods that are designed only for the non-i.i.d. data issue, including FedProx (Li et al. 2018), FedAvgM (Hsu et al. 2019) and MNIST-LT and -EFAR-IO-LT with IR 100
FedAdam (FedOpt) (Reddi al., 2020). We perform experiments under the full client participation setting. The results in Table g As
see, these heterogeneous data-oriented methods consistently underperform our method, due to the reason that they do not take the long-tailed data issue into consideration and will also produce bad global models that are biased to the head classes.
G Experimental results with 100 clients
In the partial client participation setting in the main paper, the total number of clients set as 50. this section, we perform experiments with larger number of total clients. Specifically, we set the total number of clients as 100 and the number of clients sampled in each round is 10. The total number of communication rounds is 500 for MNIST-LT and 1000 for CIFAR-IO-LT. The results displayed in Table 10 show that our method can still achieve significant improvement when the number of clients is larger.
H Experiments on FEMNIST
Here, we conduct extra experiments on a large-scale realistic federated dataset FEMNIST (Cohen et al., 2017). FEMNIST collected from 3400 clients, and has 62 classes. contains total of about 671K training samples, and 77K testing samples. The original test data of FEMNIST also long-tailed, thus, we follow previous long-tailed learning studies (Kang al. 2019 Shang al. 2022b) to create balanced test set for our evaluation. Specifically,
20

---

Published in Transactions on Machine Learning Research (04/2024)
randomly sample 200 testing samples for each class, and evaluate each method these 12,400 testing samples. We adopt the same neural network used in MNIST-LT for FEMNIST. In each round, we randomly sample 20 clients. The local learning rate is 0.01, the local batch size is 20, the server learning rate is 1.0. The number of local epochs is 5, and the umber of total communication rounds is 500. The pppee-parameters of each method on FEMNIST are kept as the same as that on MNIST-LT. We put the results Table [TT As we can see, our method still achieves the significantly best performance on FEMNIST, indicating the great generalizability and scalability of our method in large-client settings.
21


Reviews:
[{"review_id": "IoWusg9N67", "reviewer": "Reviewer_kjEm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- this paper shows algorithmic novelties that are very useful to solve decentralized long-tailed learning issues, such as the use of local decoupled training to perform class re-balancing on a local scale (thus taking into account the fact that local datasets may be long-tailed) and the communication of global gradients to address the issue of missing classes in local datasets (which takes inspiration from popular FL methods used in heterogeneous data settings)\n- the authors pay attention to bring intuition on the design of their method and the impact of algorithmic blocks in practice through a very detailed ablation study \n- the comparison with previous existing works is well conducted\n\nWeaknesses:\n- in my opinion, a rigorous mathematical framework is missing, which hurts the readability of the paper\n- following my first point, theoretical results (even simple ones) are missing to certify the well-posedness of their approach\n- I do not think that the extra cost incurred by the communication of global gradient prototypes is a drawback of the method, but a numerical study on this specific aspect is missing to certify that the algorithm may be used in practice\n- the topic of privacy preserving is mentioned without being developed in the numerical experiments\n- I think that experiments should be conducted on more realistic datasets (in my opinion, MNIST data is not complex enough to validate the use of a FL algorithm in real-world settings)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents novel contributions to tackle classification tasks in a Federated Learning (FL) setting where it is assumed that the data distribution is long-tailed, on local and global scales, in the sense that some classes occupy a much larger proportion in the training samples than other classes. This assumption is realistic, since in practice, data may be distributed in an heterogeneous manner among the clients. Elaborating on the success of decoupled training to perform class re-balancing in long-tailed centralized learning, the authors propose a decentralized learning algorithm, where decoupled training is used on the client side, in the local updates. This approach is original compared to previous existing works. In their method, the local training phase is performed on an additional classifier, which is rebalanced via local data and via global gradient prototypes sent by the server, that overcome the local issue of missing classes. They conduct several experiments with MNIST, CIFAR-10 and CIFAR-100, under various heterogeneity settings, and show improved results compared to literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I strongly suggest the authors to adopt a more rigorous style when introducing their mathematical framework in Section 3, and to keep the current 'informal' structure as a way to provide intuition on the mathematical objects.\n- I also suggest the authors to provide theoretical results (existence of solutions to the optimization problem, convergence results, communication cost results), even if they rely on strong assumptions, that would bring more rigor to their approach.\n- I recommend the authors to remove systematic long sentence boldening (which is not conventional as far as I know) and rather make shorter sentences with high impact (that would also enable to develop the mathematical perspective). \n- I recommend the authors to provide numerical results on more realistic datasets, see [1] for example.\n\n[1] FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings, 2022", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "garr5vuSls", "reviewer": "Reviewer_taLT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) The authors proposed to use gradient prototypes from the server to work as data which is not in the local client. The gradient prototypes can help in the re-balancing training. \n2) The authors compared the proposed method on different image classification data with variable imbalance ratio. Results show that the proposed method work marginally better than compared method. \n3) The authors conducted many ablation studies to analyze the effectiveness of the proposed component. It is helpful to understand the agorithm.\n\nWeakness\n1) In the paper, the authors claimed that the gradients can be transmitted in a privacy-preserving manner, is it really used in the experiments? it is not clear. does it cause any extra computation time? \n2) Regarding the privacy-preserving method, is the privacy guaranteed in theory? \n3) All test data are simulated not the number of clients is small. It would be great if the authors can compare on real federated data e.g. listed in https://www.tensorflow.org/federated/api_docs/python/tff/simulation/datasets", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors extend the decoupled training for imbalanced classification task in the setting of federated learning, and proposed a method 'RedGrape' to integrate the local data with global gradient prototypes. Experimental results show that the proposed method outperforms related compared methods in different simulated datasets with different Imbalance Ratios.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer the following questions or discuss them in the paper:\n\n1) In the paper, the authors claimed that the gradients can be transmitted in a privacy-preserving manner, is it really used in the experiments? it is not clear. does it cause any extra computation time? \n2) Regarding the privacy-preserving method, is the privacy guaranteed in theory? \n3) All test data are simulated not the number of clients is small. It would be great if the authors can compare on real federated data e.g. listed in https://www.tensorflow.org/federated/api_docs/python/tff/simulation/datasets", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v3LGgG5ltK", "reviewer": "Reviewer_H635", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very verbose while at the same time hard to read. I am not sure I can follow a line of argumentation properly. The authors seem to try and use overly verbose mathematical notation to describe their method. \nI am trying to understand EQ (3) but I'm not sure what the individual elements are supposed to represent, i.e. What is W_P? How do we motivate this constraint? If I understand correctly, we're trying to \"rebalance\" the classifier such that at its optimum it is as if it were trained on a balanced dataset? How does this change when moving to EQ (4)? Figure 1 is way to busy to help me understand what is going on. \nMaybe my best attempt at summarizing is that $\\hat{W}_k$ is trained on the local clients' dataset whereas the actual classifier of interest is trained on both, the balanced dataset as well as the local dataset. I do not, however understand the motivation for the constraint and subsequently the lagrangian formulation\n\nIf I understand the concept of gradient prototypes correctly, then this method will scale extremely poorly as a for $C$ classes, $C|W|$ need to be communicated between client and server at every round. This should be compared to the baseline methods in the paper. A plot with GB-communicated on the x-axis and test-accuracy on the y-axis seems appropriate. How does this compare to using SCAFFOLD for the classifier-weights?\n\nIn terms of privacy analysis, I believe that this method at the least shares information about local clients' label availability as well potentially the label distribution in the shape of the per-class gradients. \n\nIn terms of experimental evaluation, I appreciate the ablations, but MNIST in 2024 is not really appropriate, I believe - and C10 & C100 are not great evidence for general applicability of the method. I might suggest e.g. FeMNIST to study the large-client setting with data-amount imbalance, as well as other modalities such as Shakespear or the Reddit dataset, all of which are frequently used in FL research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method for federated learning that addresses specifically the setting of long-tailed distributions at the clients, i.e. a client has an imbalanced number of examples per class. The method consists of two elements: Training an additional classifier and enhancing the local dataset with global gradient prototypes. \nThe authors discuss their method and show empirical results on some image-classification datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I believe this paper could benefit from less verbosity and clean-up of notation. Conciseness would greatly help in communicating the core ideas, which I believe should be possible on a lot less pages and make it more accessible to the reader. I am not sure what the reasoning behind __boldening__ so many sentences is, but I do not feel it helped me in parsing the paper. \nI would ask the authors to do another pass of the paper, strengthen the variety in experimental datasets and improve clarity.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DvL96ADAr6", "reviewer": "Reviewer_qVgK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1). The paper introduces a novel decentralized decoupled training mechanism to address the challenges of long-tailed data distribution in Federated Learning, offering a fresh perspective on improving classifier performance.\n2). By leveraging local real data and global gradient prototypes, the proposed method effectively re-balances the biased classifier during local training, leading to improved model performance.\nExperimental Validation: Extensive experiments conducted in various settings demonstrate that the proposed decentralized decoupled training mechanism consistently outperforms existing state-of-the-art methods, showcasing its effectiveness.\n3). The approach maximizes the use of abundant real data stored locally in FL settings, highlighting the importance of leveraging local resources for improving model performance.\nPrivacy Preservation: The method only requires normal gradients transmission between the server and clients, ensuring data privacy is maintained in line with FL principles.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel approach to addressing the challenges posed by long-tailed data distribution in Federated Learning (FL). The paper introduces the concept of decoupled training, which aims to re-balance the biased classifier in FL after normal instance-balanced training. This approach has shown promising results in centralized long-tailed learning scenarios.\nOne of the key contributions of the paper is the proposal of a decentralized decoupled training mechanism that leverages the abundant real data stored locally in FL settings. By integrating local real data with global gradient prototypes to form local balanced datasets, the proposed method effectively re-balances the classifier during local training. Additionally, the introduction of a supplementary classifier in the training phase helps model the global data distribution, addressing the issue of contradictory optimization goals that arise from local classifier re-balancing.\nThe experimental results presented in the paper demonstrate that the proposed decentralized decoupled training mechanism consistently outperforms existing state-of-the-art methods in various settings. This highlights the effectiveness of the approach in improving classifier performance in FL on long-tailed data distributions.\nOverall, the paper provides valuable insights into addressing the challenges of long-tailed data distribution in FL through decentralized decoupled training. The proposed mechanism offers a practical solution that leverages local real data and global gradient prototypes to enhance classifier performance without requiring additional data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Cons:\n1). The decentralized decoupled training mechanism may introduce additional complexity to the FL system, potentially requiring more computational resources and coordination among clients. The need for transmitting global gradient prototypes to clients for re-balancing the classifier during local training may lead to increased communication overhead, especially in large-scale FL systems. Thus, it is strongly suggested to add additional theoretical analysis and experimental studies on the time complexity and communication costs. The acc comparison should also list what the communication costs are.\n\n2). Limited Scalability: The method's performance may be impacted in scenarios where clients have limited local data or when dealing with a large number of clients, potentially affecting the scalability of the approach. The experiments are suggested to include much more clients and more data points. For example, FEMNIST can be used in the validation.\n\n3). More thorough literature review is suggested. Some recent FL works are not included in the discussion or comparison, such as the personalized FL papers:\nFederated Multi-Task Learning under a Mixture of Distributions. NeurIPS'22.\nPersonalized Federated Learning under Mixture of Distributions. ICML'23.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jZA6nzWNuF", "reviewer": "Reviewer_Pmr2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper is overall well-written. The approach proposed in this paper is innovative and enlightening to mitigate the impact of long-tailed data distribution in FL. The literature review is broad and sufficient. The comparison with existing literature (either centralized or federated) is thorough. The illustration of the algorithm is clear. \n\nWeakness: It would be helpful if the authors could shed some light on the theoretical perspective of the proposed algorithms --- understood this is a empirical-faced paper, but a theoretical justification of the convergence would be greatly appreciated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel approach to federated long-tailed learning by introducing a decentralized decoupled training mechanism. This method leverages local real data and global gradient prototypes to address the challenges of class imbalance and privacy concerns in federated learning (FL). It involves a supplementary classifier to model the global data distribution effectively and uses the method of Lagrange Multipliers for optimization. Extensive experiments demonstrate its superiority over existing state-of-the-art methods in various settings, showing significant improvements in handling federated long-tailed learning problems. The approach promises better model generalization and privacy preservation without additional server requirements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness paragraph of the section above.", "broader_impact_concerns": "No concern on broad impact", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to address the problem of long-tailed decentralized training, and proposes a decentralized learning algorithm, where decoupled training is used on the client side, in the local updates. The experimental results show that the proposed method is effective.
The reviewers agree that the paper provides interesting algorithms and sufficient results to support the findings. It would be great to further enhance the mathematical logic of the proposed approach in the final version.
Decision: Accept (poster)