Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer
Authors: Zikai Xiao
Zihan Chen
Songshang Liu
Hualiang Wang
YANG FENG
Jin Hao
Joey Tianyi Zhou
Jian Wu
Howard Hao Yang
Zuozhu Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated learning
Long-tailed learning
Data heterogeneity
TL;DR: This paper presents Fed-GraB, a novel fl approach for long-tailed scenarios, which effectively addresses data imbalance and heterogeneity through a Self-adjusting Gradient Balancer (SGB) and Direct Prior Analyzer (DPA), achieving SOTA performance.

Abstract: Data privacy and long-tailed distribution are the norms rather than the exception in many real-world tasks. This paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit a long-tailed distribution. Under such a setting, existing federated optimization and/or centralized long-tailed learning methods hardly apply due to challenges in (a) characterizing the global long-tailed distribution under privacy constraints and (b) adjusting the local learning strategy to cope with the head-tail imbalance. In response, we propose a method termed $\texttt{Fed-GraB}$, comprised of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients' gradients in a closed-loop manner, based on the feedback of global long-tailed distribution evaluated by a Direct Prior Analyzer (DPA) module. Using $\texttt{Fed-GraB}$, clients can effectively alleviate the distribution drift caused by data heterogeneity during the model training process and obtain a global model with better performance on the minority classes while maintaining the performance of the majority classes. Extensive experiments demonstrate that $\texttt{Fed-GraB}$ achieves state-of-the-art performance on representative datasets such as CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist.

Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer
Zikai Xiao",
Zihan Chen
Songshang Liu'
Hualiang Wang
Yang Feng*
Jin Hao
Joey Tianyi Zhou Jian Wu
Howard Hao Yang'
Zuozhu Liu"'
ZZeejiang University,
Singapore University of Technology and Design,
The Hong Kong University of Science and Technology,
Angelalign Technology Inc, 5 Sttate Key Laboratory of Oral Diseases, Sichuan University,
"Centre for Frontier Al Research (CFAR), A*STAR, Singapore zikai@z) eikaiezju.. edu. cn
Abstract
Data privacy and long-tailed distribution are the norms rather than the exceptions in many real-world tasks. This paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit long-tailed distribution. Under such setting, existing federated optimization and/or centralized long- tailed learning methods hardly apply due challenges in (a) characterizing the global long tailed distribution under privacy constraints and (b) adjusting the local learning strategy to cope with the head-tail imbalance. In response, we propose a method termed Fed-GraB, comprised of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients' gradients on the feedback of global long-tailed prior derived from a Direct Prior Analyzer (DPA) module. Using Fed-GraB, clients can effectively alleviate the distribution drift caused by data heterogeneity during the model training process and obtain a global model with better performance on the minority classes while maintaining the performance of the majority classes. Extensive experiments demonstrate that Fed-GraB achieves state-of-the-art performance on representative datasets such CIFAR-1O-LT, CIFAR-100-12 ImageNet-LT, and iNaturalist. Our codes are available at
closed-loop manner based
Introduction
Federated learning (FI.) is approach for massively distributed clients to train global machine learning model collaboratively without exposing their private data, which has garnered ever-increasing attention in academia and industry alike [1, 2]. Unlike conventional machine learning methods, FL brings the learning objective directly onto the end-user devices for local training, where only the intermediate parameters (c.g., gradients) need to be sent to server for model aggregation and update. This approach not only substantially reduces the communication overheads but, more importantly, facilitates the clients in obtaining generic global model without sharing their private data, thereby contributing to the development of trustworthy intelligent systems [3]. Despite its great potential,
"Co-first author.
fCo (Co-Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

a
FedGrraB Framework
Classifiers Global Modelmn,
Direet Prior Analyzer (DPA)
(b)
Softmax
Eqlv2-FL
Waiglts
Head&Mildle
Tuil
A,00
Ourss
Client
Cliom X
Dierations
Poior Vector P.
Training
Positine
Hedd
Middle
Classifiers
SGB
Re-weighted
Gradients
A (1)
Negative
Gradient Difference
Tail
goag
Bonch images
Legis la Padds
Figure 1: (a): An illustration the Fed-GraB framework, which SGB mounted on all classes of each client based on the prior vector Pe derived by the DPA module, ensuring federated balanced local training; (b): Comparison for the difference of positive and negative gradients.
realizing FL also requires one to overcome new hurdles in real-world implementations, among which particular challenge stems from data heterogeneity, i.e., the non-IlI) distribution and imbalanced dataset sizes across clients [4-7]. These factors often give rise to a pernicious issue where the data set of different clients jointly follows global long-tailed distribution, which is commonly known as the Federated Long-Taiee Tailed learning (Fed-LT) problem [8, 9]. For instance, patients' diagnosis varies substantially across medical centers but collaboratively form long-tailed distributions for certain diseases [10, 11]. broad range applications, performance on the minority classes (i.e., tail classes) [12], such as rare diseases, dangerous behaviors in autonomous driving [13, 14], and abnormal breath or heart rates in wearable devices [15], play a pivotal role in developing reliable and robust solutions to FL system.
The difficulty in addressing the Fed-LT problem primarily stems from the fact that although clients' datasets can be locally balanced, globally, i.e., if the datasets were aggregated, the distribution may be long-tailed As clients would not disclose their data information due privacy concerns, identifying the global long tailedness in data distribution could be onerous. To that end, a natural question arises:
Question 1: How leverage the global long- tailed statistics, especially for the minorities that need to be carefully tailored, without conflicting with privacy concerns?
.ilent
On the other hand, even if the globally long-tailed data information is available, existing methods do not directly apply to address such problem. Specifically, typical FL methods cope with data heterogeneity via generic techniques such dy- namic regularizations, client selections, data aug- mentation, distillation, and personalization train- ing [16-19, 4, 20, 21]. These approaches ignored the diverged/imbalancee class levels that can re- sult in the global long-tailed issue [22, 23]. Hence, they cannot guarantee decent performance for the tail classes. A key ingredient to improve long- tailed learning is to boost the performance of tail classes without degrading the performance of the head classes [24, 25]. While several learning meth- ods dedicated to long-tailed data exist, they usually assume global class priors for re-balancing, which are not available in the context of FL due to privacy concerns [8]. Moreover, these methods may not be applicable local training as local distributions could present diverse long-tailed characteristics and/or not even long-tailed [9]. As such, it begs another question as follows:
Class
Figure 2: An example of the global and local distributions of CIFAR-IO-LT (Left) and CIFAR- 100-LT (Right) under non-IID setting. The bar on the top demonstrates the sorted global long- tailed distribution. Each row represents the di- verse statisties of class imbalance of each client.

---

Question 2: How to perform local training to synergistieally aggregate a global model that excels on both majority and minority classes under the FL setting?
A possible choice is to re-balance the clients' local gradients via approaches such as Seasaw loss [26] or Equalization Loss v2 [27]. However one needs customize these methods on different clients with heuristically selected hyper-parameters, resulting in complicated models with limited generalization capability. Moreover, the trained local models may have huge variance, as illustrated in Fig. (a),
and hardly formulate a synergistic model to handle tail class.
In light of the above challenges, this paper proposes Fed-Grab comprised of a long-tailed statistic analyzer and a gradient rebalancer, for the Fed-LT problem, as illustrated in Fig. 1. More specifically, develop federated weight-norm- based method, coined as the Direct Prior Analyzer (DPA), to derive a prior vector for global data statistics, utilizing the weight parameters from the global classifier. To overcome the degradation caused by the distribution discrepancy between the server and clients (or amongst the clients) in Fed-LT, we establish a self-adjusting gradient balancer (SGB) based on the estimated global head-tail characteristics, which is integrated at every client to re- balance the positive and negative gradients in class-wise and privacy-preserving manner. A marked advantage of Fed-GraB is that trains all heterogeneous clients in a feedback-based closed-loop manner, encouraging all clients to contribute to the global long-tailed recognition task collaboratively. We also provide comprehensive theoretical and empirical analyses verify the effectiveness of Fed- GraB. We conduct excessive experiments on multiple benchmark datasets (including the CIFAR- 10/100-LT, ImageNet-LT, and iNaturalist) with both long-tailed and non-IID data distributions. The results demonstrate that Fed-Gra BeB outperforms state-of-the-ar (SOTA) baselines, including federated optimization for heterogeneous data, FL methods for LT classification, and centralized LTT learning methods in FL settings.
2 Related Work
2.1 Federated learning with data heterogeneity
Numerous FL approaches have been developed to cope with the heterogeneous or imbalanced data distribution. For instance, FedProx [16], FedDyn [17], and MOON [28] modify the local loss function by adding regularizers at the local side. SCAFFOLD [18] proposes a control variates-based method to reduce the clients' distribution drift brought by the discrepancies. FedIR [4] dynamically adjusts the sampling ratio of each client based on each client's contribution and data distribution situation. And FedNova [20] is proposed to tackle the induced objective inconsistency problem. On the server side, FedAvgM [19] and FedAdaGrad [29] are proposed to mitigate the performance degradation. Moreover, FL-oriented client selection [30, 31] and data augmentation strategies [32] have been investigated to enhance the performance. To address the poor generalization of single generic model, personalization training methods have been recently explored to maintain multiple personalized local models [33, 34, 21]. However, most of these methods pay more attention to the discrepancies in inter-client distributions while ignoring the inconsistency among different classes, failing to achieve satisfactory performance on the tail classes. In lieu of this, alternate paradigm [35] solely addresses the personalized long-tail conundrum, disregarding the variances within inter-client distributions. Notable exceptions could be found in [8, 36, 6] to tackle class imbalance. Still, they require the exchange of local features [8], private local data distribution [36], and auxiliary data [6], which would lead to potential privacy issues.
2.2 Long-tailed learning
Long-tailed data widely exists in real-world machine learning tasks, where head classes dominate the training [10, 37]. To tackle the poor performance of tail classes in such scenarios, many approaches have been explored to boost the performance of tail classes via class re-balancing, information augmentation, and module improvements [38-41, 25]. Re-balancing strategies, as the mainstream approaches in long-taiil
learning, can be categorized into class re- rammpiinn re- reieegghtnng, and logit adjustments [10, 42]. Re-sampling approaches [43-46] address the class imbalance via war-sanmling tail classes or under-sampling head classes. On the other hand, the re- weightittn approaches aim to balance the loss( [47-50]) or gradients (Seesaw loss [26], Eglv2 [27]). Specifically, Focal loss [39] increases the prediction probabilities to achieve better prediction performance for tail classes. And a
seee---ittrbution-awee aware margin loss LDAM is introduced by [49] using label frequency. Moreover,
3

---

multi-expert framework (TADE [51]), prototype learning (OLTR [24]), and head-to-tail knowledge transfer (LEAP [52]) have been proposed for improved long-taile data performance. However, such methods" effectiveness may be constrained in Fed-I.T due to discrepancies in local and global statistics.
3 Method
3.1 Preliminaries
Consider FL system with N clients and an M-class visual recognition dataset D = [D&)k=n for classification tasks. Dx. denotes the local dataset of client with size [Dx]- Usually, the FL. training process iterates multiple communication rounds until convergence. In a typical communication round I, the central server randomly selects a subset of clients Si and distributes the latest global model w them. For client
Federated Learning:
St, performs local update to w based on Dr. where the locally computed model is denoted as w At the end of the round, the global model could be updated by the aggregation criterion (e.g., FedAvg) for the next round computation.
Long-tailed distribution FL:
To characterize the long-tailed data distribution in Fed-LT, we measure the imbalance factors in global perspectives, which is denoted by IFu and computed over the global dataset D. We could employ a predefined imbalance IFG to sample from balanced datasets, constructing long-tailed datasets; see Fig. for visualization.
Positive and Negative Gradients: Notably, positive and negative cumulative gradients have an important effect long-tailed tasks such as visual recognition [53], object detection [27]. and instance segmentation [26]. Suppose we have a neural network-bass classification model with Cross Entropy loss as L(z) = Ei log(o;). with
a²t 2n=1 e where is the ground truth label, [21. 2MI] are the logits, and o = [01. ....... J. denotes the probabilities of the classifier. Given sample with label j, the positive gradients are defined as the derivative of the loss with respect to 2j. while the rest shall be the negative gradients, i.e.,
pos (C) = =jj
Viny Zigh (C) = -
3.2 Direct Prior Analyzer (DPA)
This section details the proposed DPA module which calculates prior vector for balanced train- ing. It utilizes the same information required by FedAvg, without the need for extra data distribu- tion details. Essentially, DPA derives a prior prob- ability vector that discerns potential tail classes by examining the L2-norm of the weight param- eters within the classifiers. The reason for this design arises from existing studies [25, 54], reveal- ing that model recognizes head or tail classes via classifier and tends to give higher scores for head classes when trained with imbalanced data. Thus, the weight norm serves as an effective in- dicator of the imbalance degree since the norm of the active weights could empirically reflect the behavior and frequency of different classes. In particular, let w denote the weights of classifier j in the global model classifier with weights Wg. The estimated global distribution would be given by P. = {pj / |12)}, where h())) is a linear transformation for scaling the norm to share of the total. This estimation can be executed after global model aggregation with full client participation.
Training Loop
Compete Gradicato
Updat: Parancers A Food-frood
V (C
CiN (C
V
(29)
gooe
A,10
Gradicut
Collector
Balancer
Re-weighting Target =(t)
Re-weighting Feedback
Figure 3: The closed-loop in SGB. updated according to
( (t), 10,,8hen (t) the controller, and the re-weighted gradients are stored in the collector for future cumulative computation of (t).
In the Fed-GraB implementation, the prior vector P. derives from DPA at the onset of local training. It is also worth mentioning that P. does not need to be perfect and only requires information that reflects the model prediction inclinations at the class level, rather than complete correspondence with the distribution itself. See more detailed analysis in experiments.

---

3.3 Self-adjusting Gradient Balancer for Fed-LT
This part establishes the adjfsanju Gradient Balancer (SGB), an effective tool that can be integrated into biased classifiers to re-balance local training and improve the overall performance. Particularly, we consider the difference between cumulative positive and negative gradients A(t). with (t) for class j defined as (t = 9; (t) gj neg (t), where gt(t) = Ei 2, (C*) is the cumulative gradients of class j. capturing the overall imbalance degree throughout the training process. In a toy balanced scenario with M samples each with equal probability to the M classes, the expected target 2t(t) of (t) is:
M-1
E(3((t)) = Σ 1 -(),))))))) 0.
indicating that (t) approaches zero when the distributions become identical. The core idea of SGB is to consistently align (t) to O for tail classes in a closed-loop and self- adjusting manner. Specifically, we carry out proportional-inteeral-eeriieee
controller [55] to continually update the re-weighting coefficients of gradients with regard to the logits in a class-wise manner, as demonstrated in Fig. The re-weighting process is based on adaptive self-adjustment
with eπтоr feedback until A(t) reaches the pre- rre-determined balanced target, rather than following
heuristic methods as [26, 27]. The error feedback et At( (t) for class represents the
distance between the current status (t) and a target (t) during training. Given e(t), the output of
the controller in SGB is
uy(()= = Kpe(()) + K₁ Ee(0) kDle() e, 1)),
3=0
where Kp, K2, and KD controlling factors. The proportional item with Kp shows the con- troller would give bigger re-weighting coeffi- cients 33 to compensate the imbalance for cur- rent larger errors. Meanwhile, the integral item with K2 could look back on the past errors to reduce the steady-state offset which could not be handled by the proportional item. Further- more, computing 83 only with the propor- tional and integral loop could hardly achieve steady and smooth re- re-weighting, we introduce
Algorithm 1 Local training process of Fed-GraB
Input: we, local model
client
Output: Updated local model u
I (ENTUPPATT((())
1: Update prior vector Pe by DPA with vof
2:
for batch E do
3: for each classifier 2j da
#initlalize A = 0
4:
Compute V. (C;b) [VPg (C; 6), V). (C;6)
5:
Sample a random value r ~ Uniform(U,
8 Grudient Balancer Inpui Compute output SGB uu() by
6:
e(t) t- (,(
the third differential item with Ki to control the fluctuations based on future expectations. For instance, when (t) is close to O, the differ- ential item would prohibit u(t) from diverging re-weighting and vise versa. Overall, the three items work together to achieve quick, precise, and stable adjustments.
7:
8:
Compute # Gradient Collectov
cocfficients
Eq. (4)
9:
Update (t(() with the re-weighted gradients by Eq. (5)
10:
we (+1 4- w/ "MM B,,oy(C;))
11:
return u0.
We first map (t) through a simple activation function o(-) to calculate 35 = a (t-u,(t)) Sgas ((u;())) where (xx) -- Then, we combine the global information obtained by DPA to calculate the final coefficients for positive and negative gradients per iteration as:
B, = 4>ppbb 10700 33 + l.g..b) [1,
(4)
where II(-) is the indicator function. Consequently, the re-weighted gradients are computed as:
pos' (C') = (t) ( (C'),
(5)
(C) 1 (C))
The working mechanism of SGB at a typical client summarized in Fig. 3. As depicted in Fig. 1 (a), SGB is implemented across all the classes in each client, with parameters adjusted in accordance with the prior vector derived by DPA. By utilizing global information, this approach encourages all clients to contribute coherently to the global model while simultaneously rectifying prediction biases arising from long-tailed datasets. As shown in Fig. (b), SGB improves the balance between cumulative positive and negative gradients. See experiments and Supplementary for further analysis.
5

---

3,4 Algorithm Summary for Fed-GraB
The training pipeline of our Fed-GraB framework is presented in Algorithm 1, which consists of two stages. First. DPA calculates the prior vector Pe after receiving aggregated weights wy. Then, SGB is applicd to all the classes. The local classifier would be dynamically adjusted with weighted positive and negative gradients via A(t)-based closed-loop controller in accordance with prior vector Pa derived from DPA. The combination with SGB and DPA is detailed in Eq. (4).
3.5 Privacy discussions of DPA
As the local prior computation is performed at the client side, there would be a few privacy concerns with the DPA method. It is, however, noteworthy that the potential privacy issue exists in the general FL frameworks rather than specific to our proposed DPA method. For instance, gradient inversion [56] can pose a threat to almost all gradient transmission- FL methods without any privacy- prwccy-pressrvationn techniques. As the privacy issue of the FL framework is beyond the scope of this work, we briefly include the discussion in this subsection.
4 Experiments
4.1 Experimental Setup
Baselines: We consider two types of SOTA baselines: (1) FL-oriented methods to tackle data
heterogeneity (FedProx [16], FedNova [20]) or federated long-tailed data-oriented (FedIR [4]
and CReFF [8]), and FedAvg is also included for reference; (2) Long-tailed learning (LT)-oriented
methods (r-norm [25], Eqlv2 [27]. LDAM [49], Focal-loss [39] and GCL-loss [42]) applied at
local training of each client. We also provide the results of the -ailed methods in centralized learning (CL) settings as the oracle performance upper bound, including SGB.
Datasets:
We conduct the experiments in three benchmark datasets for long-tailed classification, i... S0TAR-10/10011 [57]. ImageNet-LT [58]. CIFAR-10/100-1) are sampled into long-taile tailed distribution by exponential distribution controlled by IF [49], and we use the same configurations as in [24] for ImageNet-II with the number of images per class ranging from 5 to 1280. To evaluate INaturalist-Uset 160k, with
the performance real-world data, we also conduct experiments 160k examples of 1,023 species classes and partitioned on the basis of iNaturalit-2011 [59].
Federated settings: We use non-IID data partitions for all experiments, implemented via symmetric Dirichlet distributions with concentration parameter a control the identicalness of local data distributions among all the clients. We train ResNet-18 18 over N = 40 clients on CIFAR-IO-LT. ResNet-34 and ResNet-50 are used on CIFAR-I00-LT and ImageNet-LT respectively with N 20 clients. For iNaturalist- 1iiiisstt-6000 600, we use the same settings as ImageNet-LT.
For the sake of faimess, we keep consistent settings external for experiments. We conduct experiments using a starting model from FedAvg. For CReFF, the number of federated features 100, rate respectively on CIFAR-10/100 10.LL..TT We report the official result [8] on ImageNet-LT, As r-norm one-shot method, we provide a pre-trained model by FedAvg and adjust the classifier weights on the server post-process. Focal -Loss and EQL v2 do not require distribution prior. We directly replace the local cross-entropy loss with their proposed re-balancing methods. For LDAM, we use the local data distribution for its local training.
Implementation of baselines:
1. 0.01 federated feature learning rate and main learning
4.2 Comparison with State-of-the-art Methods
Evaluation on CIFAR-1O-LT:
The performance of Fed--edaaaaa -rdd-rrB/SGBB on diverse settings are reported in Tab. 1. Notably, Fed-GraB achieves the best overall accuracies on all settings, with a significant improvement on the tail classes while ensuring good performance of the head classes. The performance gain becomes more evident under extremely imbalanced data. Moreover, the SGB implemented in the CL setting (i.e., performance upper bound testing) surpasses existing SOTA long-tail methods, further demonstrating the universal effectiveness of Fed-Grab under different scenarios.

---

1Pg-10
1F
IFo-100
Seting
Method
Many
Med
Few
All
Many
Med
Few
All
Many
Med
Few
All
Solbmaxx
0.501
0.879
0.886
0.895
0,908
0.776
0.742
0.81720.00
0.920
0.745
0.675
0.707.0.001
Pqlv2
0,902
0.880
0.874
0000
0.901
0.774
0.774
0.810
0.112
0.751
0.679
0.776
LDAM
0.957
0.799
0.854
0.964
0.739
0.685
0. 0.700400011
0.936
0.729
0.610
0.698..0..00
SOB-CL
0.897
0.901
0.901
0. 0.898
0.891
0.817
0.833
0.8.6620.004
0.901
0.738
0.814
0.81820.00
FodAvg
0.896
0.858
0.846
0,877
0.858
0.771
0.693
0. 1o.00s
0.922
0.716
0.616
1..57
FodProx
0.898
0.859
0.854
0
0.891
0.773
0.691
1.79
0.921
0.725
0.582
0.729
FedNova
0.912
0.853
0.848
0.802
0,903
0.757
0.702
a.797
0.934
0.734
0.599
0.70920.005
FodR
0.966
0.862
0.808 10.001
0.972
0.781
0.969
0.755
1.728
6ml
CReFF
0.911
0,850
0.887
0.884 i0
0.896
0.769
0.664
0.791
0.935
0,223
0.574
0.726 g0.26
T-corm
0.877
0.871
0.90K
1.881.0.000
0.378
0.790
0.725
0.
0.922
'd.76
0.668
20700
Eqlv2-FL
0.896
0.852
0.857
0.878 .0.000
0,886
0.786
0.690
0.790
0.919
0.704
0.597
0.72910.000
LDAM-FL
0.901
0.825
0.863
0.881
0.708
0.638
0.495
0.679
Focal-FL
0.877
0.839
0.834
0.809
0.877
0.744
0.665
a.770
0.916
0,701
0.558
0.788
GCL-FL
0,923
0.747
0.781
0.796 00.000
0.949
0.748
0.689
0.761
0.963
0.726
0.608
0.726
Fed-GraB GraB
0.886
0.882
0.893
0.885++.002
0.875
0.784
0.775
0.818+0.003
0.910
0.698
0.713
0.766+0.00s
FodAvg
0.890
0.864
0.861
0.876..0.00
0,865
0,772
0.685
0.781
0.906
0.720
0.585
0.719
FedProx
0.883
0.864
0.863
0.874.0.000
0.857
0.776
0.688
0.78220.00
0.892
0.712
0.564
0.715 o11
FedNova
0.903
0.8%6
0.834
Lo.00e
0,888
0.773
0.788 1oc0o
0.924
0.739
0.609
0.739
FedRR
0.955
0.816
0.570
0.866_20.01
0.961
0.771
0.698
0.781
0.976
0.726
0.562
0.715 00s
a=0.5
CReFF
0.900
0.838
0.880
0.877
0,878
0,778
0.664
0.796
0.932
0.699
0,580
1.718
r-porm
0.885
0.863
0.350
0.877
10.00
0.367
0.759
0.728
0.706
0.962
0.726
0.611
0.73110..0
Eglv2-FL
0.895
0.855
0.877 000
0.872
0.771
2.775
0.898
0.717
0.586
0.714go.011
LDAM-FI.
0,885
0.860
0.848
0.850.m.00w
0.863
0.742
0.673
0.709
0.877
0.583
0.490
0.657
Focal-FL
0.568
0.853
0.858
0.865_.0.00
0.842
0,780
0.661
0.707
0.883
0.703
0,581
0.728
GCL.FL
0.902
0.731
0.813
0.708
0,938
0.725
0.725
0. 0.788
0.954
0.730
0.623
0.713
Fed-GraB
0.864
0.891
0.897
0.881
L..000
0.836
0.790
0.783
0.806
0.910
0.698
0.713
0.761 Lo.00s
Table 1: Test accuracies of Fed-GraB/SGi and SOTA methods on CIFAR-IO-LT with diverse imbalanced and heterogeneous data settings. More results (e.g.. DPA with CL weighttht methods and IID settings) are in the Supplementary.
CIFAR- (00-LT
ImageNet-LT
iNoturalist-
Method
Many
Med
Few
All
Many
Med
Few
All
Many
Med
Few
All
FedAvg
0.643
0.410
0.182
0.305
0.428
0.258
0.127
0.287cm000
0.596
0.425
0.242
0.434a0.000
FedProx
0.639
0.416
0.181
g.300 600
0.439
0.268
0.128
0.202_.002
0,582
0.444
0.241
0.42520001
FedNova
0.661
0:129
0.195
0,378 po.00
0.415
0.231
0,114
0.265
0.561
0.403
0.226
0.40410.000
FoaIR
0.644
0.410
0.182
0. 0.36420.000
0,388
0,206
0.074
0.236-00011
0.579
0.387
0.191
0.306540.000
CReFP
0.684
0.440
0.146
0,263
7-norm
0.439
0.362"
0.325
0.308
0.347
702265
0.260
0.268
0.337
0.435
0.287
Eqlv2-FL
0.652
0.434
0.198
0.381 50
0.433
0,262
0,118
0.28120000
0.570
0.397
0.376
0. 0.440g0004
I.AM-FI.
0.699
0.409
0.168
0.365
0.216
0.112
0.242__000
0.00
(1560
0449
0.344
0.414
Focal-FL,
0.655
0.418
0.179
0. 0.367
0.424
0.766
0.136
0.283-n.00
0.596
0.430
0.247
0.430g0.000
GCL-FL.
0.567
0.346
0.156
0.301
20.012
0.317
0.209
0.124
0.224
0.565
0.375
0.196
4.388
Fod-GimB
0.683
0.553
0.221
00.02
0.407
0.294
0.215
0.31
1..110
0.527
0.449
0.372
10.00
Table 2: Test accuracies of various methods on CIFAR-IO0-LT, ImageNet-LT and iNaturalist-160k with non-IID data settings.
Evaluation on CIFAR-100-LT,, ImageNet-LT, and Naturalist-160k: Tab. 2 shows that Fed-GraB consistently outperforms all baselines on test overall accuracies for CIFAR-100-LT and ImageNet-LTT with remarkable improvements on middle and tail classes. While LT-orientated methods obtain better results than the FL method in the extremely imbalanced ImageNet-LT, red-GraB can still surpass it'. The reason boils down to that for classification tasks with large number of classes, local data samples for each class would be extremely small. In this case, the SGB with global prior for class-wise re-balancing is more effective in learning a better- aggregated model.
As for iNaturalist- 160k, BraB significantly boosts the classification accuracy on tail classes while maintaining the best overall performance compared to FL methods. In comparison to LT- oriented methods, Fed-GraB's SGB provides robust mechanism against data heterogeneity, leading to an outstanding performance on both 3-shot and overall metrics.
A salient advantage of Fed-GraB is that it achieves universal outperformance on three large-scale or real-world scenarios, demonstrating the robustness over other methods such as CReFF, T norm or Eqlv2-FL which attain good performance on specific dataset.
Evaluation of Communication Efficiency: Tab. 3 presents the communication efficiency of different methods trained on CIFAR- under two non-IID settings. We compare the required communica-
'The result of CReFF on ImageNet-LT LT are from [8] and the result on iNaturalist is not included due to memory issues in large-scale datasets.
7

---

tion rounds for the tail classes to reach 55% test accuracy. We can see from this table that Fed-Gral attains the best convergence rate under both settings, confirming its effectiveness on the tail classes.
4.3 Effectiveness of DPA and SGB
SGB for consistent gradient re-weighting across clients: When various clients employ a gradient re-weighting method on specific class, the strength of compensating divergence is because of the diverse local distributions and local dataset sizes. Therefore, although traditional re- re-weighting methods can mitigate the imbalance, discrepancies still exist. We demonstrate the advantage of SGB for addressing this issue. Specifically, we compute the mean and standard deviation (std) of (t) across all clients in CIFAR- 10-LT for both Eqlv2 and Fed-GraB, illustrated in Fig. 4 (a). We make two observations here. First, the mean of (t) (t) in Eqlv2 keeps decreasing alongside the training process while F Fed- oraBrra can always align a near-zero value, suggesting that Fed-GraB can better re- re-weight the positive and negative gradients. Second, Eqlv2 exhibits large vari- ance among clients during training. In contrast, Fed-GraB maintains very small deviation, ev- idently demonstrating the Fed-GraB can consis- tently motivate the heterogeneous clients to con- verge towards a better aggregated global model.
1FG=50
IFG=100
Method
a
0.5
Ct
1
a
0.5
FedAve
96
122
192
CReFF
285
286
309
348
EqIv2-FL
301
210
191
205
LDAM-FL
483
427
Focal-FI.
276
174
347
257
Fed-GBA
59
122
Table A comparison of communication effi- ciency on SIFAR-10, in terms of the number of required rounds, where "_" indicate the method cannot obtain the target accuracy.
Effectiveness on global distribution and tail class estimation: We evaluate DPA under three IF settings with two metries: (1) the L2-distance between the prior vector Pe and ground truth distribution; (2) the tail identification accuracy (per- centage of tail classes that are identified correctly by DPA). We plot the two metries on global model with FedAvg for 200 rounds in Fig. 4 (b). We can see that DPA converges after 70 rounds of training and captures more than 90% of the tail classes under different IFs. Moreover, the tail identi fication accuracy becomes more accurate with higher IF, which is attributed to the extreme imbalance among classes (i.e., easier to distinguish).
IFg = 10
Text. Aucuracy
1.2 Di-Dimnno
IF-100
IF 10
IF-50
Local-SGB Global SGB
Distance
Accurec
100
150
200
300 Rounds
40o
(a)
(b)
(c)
Figure (a): Visualization of mean and std of A1()) during training: (b): Tail identification performance with DPA; (c): Comparison of accuracies with global/local-SGB mounting strategy.
DPA
Partially Mounted
Salmac
Iniiaizzation
Training
0%
t,
A, zo
0.72
0.70
0-9
1-9
29
34
29
69
16
$.0
9.5
Phase
Pleaee
Mounting Strategy
Figure 5: (a): An illustration of effectiveness of P. with class-wise performance; (b): Mounting strategies with SGB; (c): An illustration of the trend of A()) with different values of =tt). Effectiveness of global P.: We investigate local version (Local-SGB) of Fed-GraB (Global-SGB), where each client implements SGB based on their local distribution instead of the global prior vector derived by DPA. The results on CIFAR- 10-LT in Fig. 4 (c) show that Global-S -oobal-GGBBBB achieves higher

---

test accuracy. Furthermore, we visualize the accuracies of FedAvg and on the received global model Fig. (a), which demonstrate that Fed-Grall can achieve better and more balanced performance across classes, especially for tails (e.g., class 2, 3, 8, 9).
after local updates
Mounting strategies with SGB: In Fed-GraB, SGB is universally applied to all classifiers through the use of DPA, rather than selectively targeting specific classes without DPA. We demonstrate that full mounting approach with DPA can more readily attain optimal performance compared custom-tailored tailed class (i.e., mounting on 7 to 9 classes). As depicted in Fig. 5 (b), the model excels when SGB is employed on tail classes, while accuracy diminishes when mounting all classes or solely the exceedingly biased classes. Notably, DPA significantly bolsters overall performance.
4.4 Ablation Study and Model Analysis
The target of (()): A critical step of SGB to compute re-weighting coefficients ,3 by e(t) A(t) 10-22 z()). Here we argue that for a static z(t), different values of z(t) would not lead to significant differences in training, as long as those values are not extremely biased. We visualize the training process with two different =(t) in Fig. 5 (c) by assuming continuous scenario. The training would consist of two phases during the closed-loop controlling. The Phase I could be regarded as a noisy initialization stage which is usually very short, resulting in different initial values based e(t). Afterward, the Phase 2 is started, and similar adjusting behaviors of e(t) could be observed, i.e., A(t) fluctuates around z(t), leading to similar 8 that stands for re- weighting strength. As the duration 8 of Phase / is negligibly short and would not impose significant influence for training. The experimental results with different target values in Fig. 6 (a) further demonstrate our argument. More analysis is presented in the Supplementary.
FedAvg
Failoy
FedAvg
Tedt Accuracy
Fed-GraBB
0.90
FedNova
Test Accuray
Fod-GraB
0.85_
0,80-
0.80
-10
10
20
40
20
A())
IF
(4)
(b)
(c)
Figure 6: (a): Performance ablation with different A(t); (b): Ablation on different Kp, K2 and KD in Fed-GraB; (c): Performance evaluation with diverse IFG.
Ablation on SGB sypameparameters: To investigate the effects of pareeeppraaeters Kr. Kr, and K D. we observe the real-time (1) during the training process with various parameter groups, as depicted in Fig. 6 (b) using the parameter sets: KP 1, 10, 10, 10, K, = O, O, O, 0.01, KD = 0, 0.1, 0.1. We notice that Kp 10 (green) more effectively constrains (t)) around compared to Kp (orange). To reduce fluctuations, we incorporate KD = 0.1 (red). In this case, A(t) remains above 0, necessitating the introduction of K2 (purple) to correct the static deviation. Notably, SGB demonstrates substantial robustness to the parameter group, as further evidenced in the Supplementary.
Fed-Grab different imbalance levels: To demonstrate Fed-GraB is versatile across different global data distributions, we conduct experiments with global imbalance factors ranging from 5 to 50. As shown in Fig. 6 (c), Fed-GraB achieves better performance than others a broad range of IFG- The results indicate that Fed-Gral could alleviate the performance degradation in moderate imbalanced cases while yielding more significant improvements on highly imbalanced data.
Computational and Storage Cost of SGB:
The additional cost from SGB are mainly attributed to the computation and storage of A(t) and u(t). As shown in Eq. (3), the main computational steps are the differential and summation of e(t). which should be linearly proportional to number of the gradients. Such extra computational cost analogous to the additional manipulations of gradients advanced stochastic gradient decent methods such as Momentum or Adam [60]. Overall, the extra computation which is implemented with several lines of code could be done very quickly. Besides, Fed-GraB needs some extra storage cost to store the weighted gradients which is quite cheap well.

---

Conclusions and Limitations
We proposed Fed-GraB, a self-adjusting and closed-loop gradient re-balancing framework for Fed-LT tasks. Fed-GraB comprises DPA, federated global long-tailedness analyzer, and SGB, a local gradient balancer, addressing the discrepancies in inter-client and inter-class statistics. We carried out extensive experiments to analyze the functionality of different components in Fed-GraB and demonstrate the efficacy of Fed-GraB various Fed-LT configurations.
Limitations. In this work, we mainly focus on aggregating a global Fed-LT model, while extensions to include clients of high interest such as training superior heterogeneous clients with with personalized strategies. Besides, experiments with real-world medical or autonomous vehicle datasets could be examined to further demonstrate the effectiveness Fed-GraB
Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant No. 62106222, No. 62201504), the Natural Science Foundation of Zhejiang Province, China(Grant No. LZ23F020008, No. LGI22F010001), Zhejiang Lab Open Research Project (No. K2022PD0AB05) and the Zhejiang UniversttyAAngelliin Inc. R&D Center for Intelligent Healthcare.
References
[1] Q Yang, Y. Liu, T. Chen, and Y. Tong, "Federated machine learning: Concept and applications," ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1-19, 2019.
[2] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, "Federated learning: Challenges, methods, and future directions," TEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50-60, 2020.
[3] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, "Communication--ffiiaee learning of deep networks from decentralized data," in Artificial intelligence and statistics, pp. 1273-1282, PMLR, 2017. I
[4] T.M. H. Hsu, H. Qi, and M. Brown, "Federated visual classification with real-world data distribution."" in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part X, pp. 76-92, 2020. 2,
[5] Q. Li, Y. Diao, Q. Chen, and B. He, "Federated learning on non-iid data silos: An experimental study," in 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 965-978, IEEE, 2022.
[6] L. Wang. S. Xu, X. Wang, and Q. Zhu, "Addressing class imbalance in federated learning," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 10165-10173, 2021.
3
[7] J. Xu, Z. Chen, T. Q. Quek, and K. F. E. Chong, "Fedcorr: Multi-stage federated learning for label noise correction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022. 2
[8] X. Shang, Y. Lu, G. Huang, and H. Wang, "Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features," arXiv preprint arXiv:2204.13399, 2022.
2, 2,3.0 6,
[9] Z. Chen, S. Liu, H. Wang, H. H. Yang, T. Q. Quek, and Z. Liu, "Towards federated tailed learning," " in International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2022 (2))),AA122)) 2022.
[10] Y. Zhang, B. Kang. B. Hooi, S. Yan, and J. Feng, "Deep long-tailed learning: A survey," arXiv preprint 2rXiv:2110.04596 2021. 2.3
[11] I.. Yang, H. Jiang, Q. Song, and Guo, "A survey on long-tailed visual recognition," Interna- tional Journal of Computer Vision, pp. 1-36, 2022. 2
10

---

[12] H. Tang, Z. Li, Z. Peng, and J. Tang. "Blockmix: Meta regularization and self-calibrated inference for metric-based meta-learning"" in ACM Multimedia, pp. 610-618, ACM, 2020. 2
[13] O. Makansi, Ö. Çigek, Y. Marrakchi, and T. Brox, "On exposing the challenging long tail in future prediction of traffic actors," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13 13147-13157, 2021. 2
[14] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuscenes: multimodal dataset for autonomous driving," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11621 1663-, 2020.
2
[15] A. Haque, A. Milstein, and L. Fei-Fei, "Illuminating the dark spaces of healthcare with ambient intelligence." Nature, vol. 585, no. 7824, pp. 193-202 2020.
[16] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," Proceedings of Machine Learning and Systems, vol. 2, pp. 429-450, 2020. 2, 3,6
[17] D. A. E. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama, "Federated learning based on dynamic regularization," International Conference on Learning Represen- lations, 2021. 3
[18] P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, "Scaffold: Stochastic controlled averaging for federated learning," in International Conference on Machine Learning. pp. 5132-5143, PMLR, 2020. 3
[19] T.-M. H. Hsu, H. Qi, and M. Brown, "Measuring the effects of non-identical data distribution for federated visual classification," arXiv preprint arXiv: drXiv:1909.06335, 2019. 2, 3
[20] Wang, Q. Liu, H. Liang, G. Joshi, and Poor, "Tackling the objective inconsistency problem in heterogeneous federated optimization," Advances in neural information processing systems, vol. 33, pp. 761 1-762,,,, 2020. 2,
[21] A. Z. Tan, H. Yu, L. Cui, and Q. Yang, "Towards personalized federated learning," IEEE Transactions on Neural Networks and Learning Systems, 2022. 2,
[22] M. Luo, F. Chen, D. Hu, Y. Zhang. J. Liang, and Feng, "No fear of heterogeneity: Classifier calibration for federated learning with non-iid data," Advances in Neural Information Processing Systems, vol. 34, pp. 5972-5984, 2021. 2
[23] J. Zhang, Z. Li, B. Li. J. Xu, S. Wu, S. Ding, and C. Wu, "Federated learning with label distribution skew via logits calibration," in International Conference on Machine Learning. pp. 26311-26329, PMLR, 2022. 2
[24] 7. Liu, 7. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, "Large-scale long-tailed recognition in an open world," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019. 2, 4,6 6
[25] B. Kang, S. Xic, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, "Decoupling rep- resentation and classifier for long- long-tailed recognition," in International Conference Learning Representations, 2019. 2, 3, 4,
[26] J. Wang, W. Zhang, Y. Zang. Y. Cao. J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin, "Seesaw loss for long-tailed instance segmentation," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9695-9704, 2021. 3, 4,5
[27] J. Tan, X. Lu, G. Zhang, C. Yin, and Q. Li, "Equalization loss v2: A new gradient balance approach for long-tailed object detection," in Proceedings of the IEFF/VVF conference on computer vision and pattern recognition, pp. 1685-1694, 2021. 3, 5,
[28] Q. Li, B. He, and D. Song, "Model-contrastive federated learning." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10713-10722, 2021.

---

[29] Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan, "Adaptive federated optimization," arXiv preprint WiXiv:2003.00295, 2020. 3
[30] R. Balakrishnan, T. Li, T. Zhou, N. Himayat, V. Smith, and J. Bilmes, "Diverse client selection for federated learning: Submodularity and convergence analysis," in ICML 2021 International Workshop on Federated Learning for User Privacy and Data Confidentiality 2021. 3
[31] Z. Chen, K. F. E. Chong, and T. Q Quek, "Dynamic iitention-based commmunication-efficcne federated learning," " in International Workshop on Federated and Transfer Learning for Data Sparsity and Confidentiality in Conjunction with IJCAI (FTL-IJCAI22211) 2021.
[32] T. Yoon, S. Shin, S. J. Hwang, and E. Yang, "Fedmix: Approximation of mixup under mean augmented federated learning," in International Conference on Learning Representations, 2020.
3
[33] A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li, "Lotteryfl: Personalized and communication-effitie federated learning with lottery ticket hypothesis on non-iid datasets," arXiv preprini arXiv:2008.03:: 03377, 2020. 3
[34] C. T Dinh, N. Tran, and J. Nguyen, "Personalized federated learning with moreau envelopes." Advances in Neural Information Processing Systems, vol. 33, pp. 21394-21405, 2020. 3
[35] Y. Lu, P. Qian, G. Huang, and H. Wang, "Personalized federated learning on long-tailed data via adversarial feature augmentation," arXiv preprint A.Kiv:2303.11666 2023.
[36] M. Duan, D. Liu, X. Chen, R. Liu, Y. Tan, and L. Liang, "Self-balancing federated learning with global imbalanced data in mobile systems," IEEE Transactions Parallel and Distributed Systems, vol. 32, no. 1, pp. 59-71, 2020.
[37] H. Tang, C. Yuan, Z. Li, and J. Tang, "Learning attention-guided pyramidal features for few-shot fine- ume-grrined recognition," " Pattern Recognit., vol. 130, p. 108792, 2022. 3
[38] T. Wang, Y. Li, B. Kang. J. Li, J. Liew, S. Tang, S. Hoi, and J. Feng, "The devil is in classification: A simple framework for long-tail instance segmentation," in European conference on computer vision, pp. 728-744, Springer, 2020. 3
[39] T.Y.. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017. 3,
6
[40] S. Li, K. Gong, C. H. Liu, Y. Wang, F. Qiao, and X. Cheng, "Metasaug: Meta semantic augmentation for long-tailed visual recognition." " in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5212-5221, 2021.
[41] J. Wang, T. Lukasiewicz, X. Hu, J. Cai, and Z. Xu, "Rsg: A simple but effective module for learning imbalanced datasets," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3784-3793, 2021.
[42] M. Li, Y.-m. Cheung, and Y. Lu, "Long-tailed visual recognition via gaussian clouded logit adjustment," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9929-6938, 2022. 3,6
[43] Y. Zang, C. Huang, and C. C. Loy, "Fasa: Feature augmentation and sampling adaptation for loig-tailed instance segmentation." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3457-3466, 2021. 3
[44] B. Liu, H. Li, H. Kang, G. Hua, and N. Vasconcelos, "Gistnet: a geometric structure transfer network for long-tailed recognition," in Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 9209-8218, 2021.
[45] J. Cai, Y. Wang, and J.-N. Hwang, "Ace: Ally complementary experts for solving long-tailed recognition in one-shot," in Proceedings of the IEFF//VV International Conference on Com- puter Vision, pp. 112-121, 2021.
12

---

[46] H. Guo and S. Wang, "Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings." " in Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 15089-15098, 2021. 3
[47] B. Li, Y. Yao, J. Tan, G. Zhang, F. Yu, J. Lu, and Y. Luo, "Equalized focal loss for dense long-tailed object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6990-6999 2022.
[48] T. Wu, Q. Huang, Z. Liu, Y. Wang, and D. Lin, "Distribution-bllaneed loss for multi-label classification in long-tailed datasets," in European Conference on Computer Vision, pp. 162-178, Springer, 2020.
[49] K. Cao, Wei, A. Gaidon, N. Arechiga, and T. Ma, "Learning imbalanced datasets with label- distribution-awen awarre margin loss," Advances in neural information processing systems, vol. 32, 2019. 6,6
[50] Wang. Y. Zhu, Y. Chen, C. Zhao, B. Yu, J. Wang. and M. Tang, "C2am loss: Chasing a better decision boundary for long-tail object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6980-6989, 2022. 3
[51] Y. Zhang, B. Hooi, L. Hong, and J. Feng, "Test-agnostic long-taile tailed recognition by test-time aggregating diverse experts with self-supervision, arXiv preprint
2021. 4
[52] J. Liu, Y. Sun, C. Han, Z. Dou, and W. Li, "Deep representation learning on long-tailed data: A learnable embedding augmentation perspective," in Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 2970-2979, 2020. 4
[53] Tan, C. Wang, Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, "Equalization loss for long- stig-t object recognition," in Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 11662-111 1671, 2020. 4
[54] Y. Li, T. Wang, B. Kang. S. Tang, C. Wang, J. Li, and J. Feng, "Overcoming classifier imbalance for cong-tail object detection with balanced group softmax," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10991-11000, 2020. 4
[55] S. Bennett, "Development of the pid controller," IEEE Control Systems Magazine, vol. 13, no. 6, pp. 58-62, 1993. 5
[56] Y. Huang, Gupta, Z. Song, K. Li, and Arora, "Evaluating gradient inversion attacks and defenses in federated learning," Advances in Neural Information Processing Systems, vol. 34, pp.
7232-7241, 2021.
[57] A. Krizhevsky, G. Hinton, et al., "Learning multiple layers of features from tiny images," " 2009.
6
[58] J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierar- chical image database." " in 2009 IEEE conference on computer vision and pattern recognition, pp. 2448-55,, Icce. 2009. 6
[59] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, "The inaturalist species classification and detection dataset," in Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 8769-8778, 2018.
[60] D. Kingma and Ba, "Adam: method for stochastic optimization," in ICLR (Poster), 2015.
13


Reviews:
[{"review_id": "FOJaIFlsy6", "reviewer": "Reviewer_XuVs", "summary": "This paper considers the globally long-tailed distribution of data and its impact on federated learning (FL). To address this issue, the authors propose the Fed-GraB algorithm that re-balances gradients. The problem is timely and the proposed algorithm is novel. Overall, the paper is well-written, and simulations are extensive. However, the limitations of the proposed framework are not rigorously addressed via for instance mathematical or quantitative analysis. Furthermore, the applicability of the proposed framework to various applications and other state-of-the-art FL algorithms are questionable.", "questions": "1. According to (1), the proposed algorithm relies on classification and cross entropy loss, as opposed to the original FL that has no restriction on its task and loss function. Is the proposed algorithm applicable to 1) non-classification tasks as well as 2) non-cross entropy loss or cross-entropy with other regularizers?\n\n2. There are various advanced FL algorithms that intentionally distort gradients via quantization, sampling, and noise injection for compression, privacy protection, and so forth. Is the proposed algorithm still applicable to these algorithms?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The potential impact of the proposed framework would be significant, particularly on two emerging areas of research: fairness guarantee and rare-event detection. The proposed idea of re-balancing gradients is not entirely original, yet still sufficiently novel as it has yet been considered in FL for this purpose. Notably, the proposed idea does not incur significant costs in terms of computation and memory, which is a big plus for practical implementation. The literature of the considered problem and relevant frameworks is well reviewed, making the paper easily readable.", "weaknesses": "Some potential weakness of the proposed framework has been proactively described, yet the claims are not very convincing. In particular, in the Privacy Discussions, it claims that \"potential privacy issue exists in the general FL frameworks rather than specific to our proposed DPA method...As the privacy issue of FL framework is beyond the scope of this we briefly include the discussion in this subsection.\" This justification hardly responds to the issues when applying the proposed framework to other privacy-preserving FL frameworks, for instance, that applies quantization and noise injection under which the proposed framework's performance may be degraded. Furthermore, in Computational and Storage Cost of SGB, it claims that \"the extra computation which is implemented with several lines of code could be done very quickly. Besides, Fed-GraB needs some extra storage cost to store the weighted gradients which is quite cheap as well.\" This is not very convincing as it does not compare, for example, the resultant convergence speed and guarantee as well as FLOPS and bytes with the cases without the proposed solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing thorough and insightful comments on our paper.\n\n> **1. Is the proposed algorithm applicable to non-cross entropy loss or cross-entropy with other regularizers as well as non-classification tasks?**\n> \n\nWe thank the reviewer for this comment. Our proposed algorithm can be used in conjunction with different loss functions and regularization methods, as it acts on the gradients of the logits during backpropagation. During the rebuttal period, we have conducted experiment on the combination of cross entropy with L2 regularization and the focal loss. The experiments are based on the CIFAR-10 dataset, with results in the table below. \n\nAs we can see, the Fed-GraB algorithm could work seamlessly with the Focal loss or the cross entropy loss with L2-regularizer, where the overall performance is even slightly improved. In contrast, the performance of original FedAvg degraded a bit when combined with the L2 regularizer. The results demonstrate that effectiveness of Fed-GraB for different loss and regularizers. \n\n| Method | Many | Med | Few | All |\n| --- | --- | --- | --- | --- |\n| FedAvg | 0.906 | 0.720 | 0.585 | 0.719 |\n| FedAvg + L2 regularizer | 0.981 | 0.778 | 0.503 | 0.709 |\n| Focal loss | 0.935 | 0.729 | 0.620 | 0.727 |\n| Fed-GraB+Focal loss | 0.958 | 0.718 | 0.640 | 0.735 |\n| Fed-GraB | 0.942 | 0.692 | 0.720 | 0.753 |\n| Fed-GraB+L2 regularizer | 0.962 | 0.667 | 0.743 | 0.757 |\n\nRegarding adaptability to non-classification tasks, theoretically, Fed-GraB can be applied to instance segmentation and detection tasks, such as Cascade Mask R-CNN on the LVIS v1 dataset. This is because Fed-GraB, similar to seesaw loss [1], EQL [2], Droploss [3] (which can be used for instance segmentation), and EQLv2 [4] (further extended to object detection), all employ the method of negative gradient over-suppression. They function by applying weighted positive and negative gradients to the classification heads in instance segmentation and object detection tasks.\n\nHowever, in the context of federated long-tail learning, for instance segmentation and object detection tasks, given the presence of long-tail class distribution within a single image, our dataset partitioning would require more complex and delicate design. Correspondingly, the formulation and metrics would also need to be appropriately designed. This goes beyond the scope of our current work. We believe that federated long-tail non-classification learning presents a challenging yet fruitful area with numerous application scenarios, making it a worthwhile focus for future research.\n\n> **2. How about the resultant convergence speed and guarantee as well as FLOPS and bytes with the cases without the proposed solution.**\n> \n\nThanks for the comments. We conducted tests on CIFAR-10 with a heterogeneity of 0.5 and an imbalance factor of 100, comparing the in-process memory overhead of different baselines and our method. The number of clients was set to 40, with local epochs fixed at 5.\n\nWe reported the convergence speed (how many rounds it would take to achieve a classification accuracy of 70%), the computational cost per round (how much time it would take to train the model for 1 epoch) and memory cost (expressed as a multiple of the memory consumed by the FedAvg, which is 3467MB) in the table below. Our method shows a computational cost the computation and memory nearly identical to the EQLv2, along with a much faster convergence rate. In terms of memory cost, the PyTorch gradient capture hook function does occupy some storage space. This overhead can be improved through official code refactoring.\n\n| Method | convergence speed (70%) | computational cost/round | memory cost |\n| --- | --- | --- | --- |\n| Fedavg | 280rounds | 2m 1s | 1.000x |\n| EqlV2 | 218 | 2m 34.2s | 2.023x |\n| Fed-GraB | 133 | 2m 35s | 2.024x |\n\n> **3. There are various advanced FL algorithms that intentionally distort gradients via quantization, sampling, and noise injection for compression, privacy protection, and so forth. Is the proposed algorithm still applicable to these algorithms?**\n> \n\nWe have conducted experiments on CIFAR-10 with a heterogeneity of 0.5 and an imbalance factor of 100 about Fed-GraB with  the bucket_quantile algorithm of quantization[1] and the federated differential privacy algorithm (noise injection)[2]. The  performance is presented in the table below. It can be observed that distorting the gradients or inject noise tends to decrease model performance in the Fed-LT scenario. However, when compared to the FedAvg baseline, our Fed-GraB can  obtain significant performance improvements in both cases, especially in the tail classes. \n\nThe results are reasonable as the current Fed-GraB frameworks are not yet tailored for gradient distortion or noise injection. We thank the reviewer for this valuable comment, and we’d like to expect future versions of Fed-GraB to better solve these issues.  \n\n| Method | Many | Med | Few | All |\n| --- | --- | --- | --- | --- |\n| FedAvg | 0.906 | 0.720 | 0.585 | 0.719 |\n| Fed-GraB | 0.910 | 0.698 | 0.713 | 0.761 |\n| FedAvg + quantization | 0.969 | 0.707 | 0.472 | 0.665 |\n| Fed-GraB + quantization | 0.949 | 0.560 | 0.686 | 0.689 |\n| FedAvg + injected noise | 0.977 | 0.751 | 0.538 | 0.711 |\n| Fed-GraB + injected noise (DP) | 0.956 | 0.705 | 0.696 | 0.752 |\n\n[1] Seesaw Loss for Long-Tailed Instance Segmentation.\n\n[2] Equalization loss for long-tailed object recognition.\n\n[3] Droploss for long-tail instance segmentation.\n\n[4] Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection.\n\n[5]Sketchml: Accelerating distributed machine learning with data sketches.\n\n[6] Communication-Efficient Learning of Deep Networks from Decentralized Data.\n\n[7] LDP-Fed: Federated learning with local differential privacy."}, {"review_id": "7pqpK7tIU9", "reviewer": "Reviewer_nXLC", "summary": "In this paper, a self-adjusting and closed-loop gradient re-balancing framework Fed-GraB was proposed and long-tailed learning tasks processing performance was improved.", "questions": "Please see Weaknesses", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a methodology named Fed-GraB, which incorporates a Self-adjusting Gradient Balancer (SGB) module. This module dynamically adjusts the weightage of gradients from individual clients in a closed-loop manner, guided by the feedback obtained from a Direct Prior Analyzer (DPA) module. By employing Fed-GraB, clients can effectively mitigate the challenges posed by disparate data distributions during the model training process. They achieve a global model that exhibits enhanced performance on underrepresented classes while maintaining the performance levels of the majority classes.", "weaknesses": "1. Equation 4 lacks ending punctuation.\n2. Figure 4(b) adds markers to make the curve more legible.\n3. The definition of IFG is not clear. Please give a formula that says how to define IFG.\n4. How to divide the dataset to get the Cifar-10/100-LT, there is no specific explanation.\n5. The insight of the algorithm should be explained more clearly.\n6. The baselines selected in this paper are not fully appropriate, because only one of them is designed for federated learning. The solutions to unbalanced data in FL are necessary to be compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful feedback. For weakness 1 and 2, we will ensure the equation contains proper ending punctuation and markers in the revised manuscript.\n\n> **1. The definition of IFG is not clear. Please give a formula that says how to define IFG. And how to divide the dataset to get the Cifar-10/100-LT?**\n> \n\nThank you for the comment. We apologize for the lack of clarity in the definition of IFG in the main body of the paper. The imbalance factor $IF_G$ represents the ratio of the number of samples in the head (most populous) category to the number in the tail (least populous) category within the dataset. $n_k^{(i)}$ is the number of data samples of class $i$ in client $k$. The $IF_G$ is define as $IF_G = \\frac{\\max_i{n^{(i)}}}{\\min_i{n^{(i)}}}$, where $\\begin{aligned}n^{(i)}=\\sum_{k=1}^{N}n_k^{(i)}\\end{aligned}$ and $N$ is the number of clients. And also, there are more details regarding $IF_G$ in our supplementary in section 2.1.\n\nIn regards to dataset partition, we have provided details on how the dataset was divided to obtain Cifar-10/100-LT in the supplementary material, specifically in section 2.3. First, we truncated the dataset followed by an exponential distribution class-wisely, and controlled the imbalanced level by the global imbalanced factor. Second, we conducted sampling to divide the dataset using Dirichlet distribution-based approach for the non-IID data partition. Please refer to the supplementary material for more information.\n\n> **2. The insight of the algorithm should be explained more clearly.**\n> \n\nWe appreciate your feedback and agree that the insight of the algorithm can be further explained. Fed-GraB consists of two main components: the Direct Prior Analyzer (DPA) and the Self-adjusting Gradient Balancer (SGB).\n\n1. **Direct Prior Analyzer (DPA)**\n    \n    The purpose of DPA is to ***analyze global long-tailed statistics***. It infers a prior vector of global data statistics by leveraging the weight parameters of the global classifiers. This prior vector allows us to understand the long-tailed distribution with imbalances between the head and tail.\n    \n2. **Self-adjusting Gradient Balancer (SGB)**\n    \n    The SGB primarily functions to rebalance gradients on a per-class basis at each client, informed by the global head-tail properties estimated by the DPA. Unlike heuristic methods that can lead to client divergence, SGB harmonizes all clients towards a common rebalancing direction. This approach not only reduces statistical variance across clients (as illustrated in Fig. 4 (a) of the main paper) but also mitigates divergence. Therefore, SGB effectively ***achieves data rebalancing and divergence alleviation simultaneously***.\n    \n\nWe hope that this explanation provides a clearer understanding of the Fed-GraB algorithm. We stand prepared to answer any additional queries you might have on this subject matter.\n\n> **3. The baselines selected in this paper are not fully appropriate, because only one of them is designed for federated learning. The solutions to unbalanced data in FL are necessary to be compared.**\n> \n\nWe categorize the baselines into two major classes. The first class encompasses imbalance federated learning methods, including CReFF[1], Fed-Focal Loss[2], and FedIR[3], as well as traditional heterogeneity methods such as FedProx and FedNova. The second class assesses the federated effects of existing long-tail methods. We have also incorporated a newly proposed method, ETF[4], which addresses federated global imbalance bias. We conducted experiments on the CIFAR-10 dataset, with an imbalance factor of 100 and a heterogeneity parameter alpha set to 0.5. The results are compiled and presented in the corresponding table.\n\n| method | Many | Med | Few | All |\n| --- | --- | --- | --- | --- |\n| CReff | 0.962 | 0.726 | 0.611 | 0.731 |\n| Fed-Focal Loss | 0.883 | 0.703 | 0.581 | 0.728 |\n| FedIR | 0.976 | 0.726 | 0.562 | 0.715 |\n| ETF | 0.499 | 0.624 | 0.703 | 0.631 |\n| Fed-GraB | 0.910 | 0.698 | 0.713 | 0.761 |\n\n[1] Federated Learning on Heterogeneous and Long-Tailed Data via Classiﬁer Re-Training with Federated Features.\n\n[2] Fed-focal loss for imbalanced data classification in federated learning.\n\n[3] Federated Visual Classiﬁcation with Real-World Data Distribution.\n\n[4] No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier."}, {"review_id": "25hEDUdnXb", "reviewer": "Reviewer_vVFc", "summary": "This paper presents an approach, Fed-GraB, for addressing the challenges of Federated Long-Tailed Learning, an issue characterized by data heterogeneity and privacy concerns. The authors tested their method on several benchmark datasets, which significantly outperforms state-of-the-art baselines. The paper is mostly an experimental work.", "questions": "Please see the above comments", "limitations": "more technical analysis or discussions are needed", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed Fed-GraB model is based on an interesting technique called Self-adjusting Gradient Balancer to rebalance gradients. The experiments are comprehensive. ", "weaknesses": "The authors propose to tackle the challenges posed by data heterogeneity and privacy concerns in the FL setting. However, the paper does not sufficiently justify the uniqueness of their approach. A clearer and more compelling argument is needed on how the proposed Fed-GraB model uniquely and effectively addresses these challenges.\n\nThe proposed model needs to be corroborated with theoretical analyses or at least insights, especially regarding the function and operation of the DPA modules.\n\nThe paper often assumes that global class priors are available for re-balancing, which may not always be the case in real-world applications due to privacy constraints. More importantly, it's unclear how the proposed method would handle issues where local distributions are not long-tailed or present diverse long-tailed characteristics.\n\nA more comprehensive description of the interplay between the SGB and DPA modules, and the technical novelty there, would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **1. How the proposed Fed-GraB model *uniquely and effectively* addresses these challenges?**\n> \n\nFed-GraB comprises two components, Self-adjusting Gradient Balancer (SGB) and Direct Prior Analyzer (DPA), each addressing distinct challenges.\n\nChallenge1: In the context of Federated long-tail learning, addressing how to estimate the global long-tailed statistics infringing on privacy concerns is a significant challenge.  ***Please note that global statistics are not available and need to be estimated through DPA***.\n\nThe Direct Prior Analyzer (DPA) in the Fed-GraB model uniquely and effectively addresses the challenges in federated long-tailed learning by estimating global data statistics using the weight parameters from the global classifier. This allows the model to understand the global data distribution without transmitting additional information beyond the gradients [1].\n\nChallenge2: In addition, figuring out how to perform local training that synergistically aggregates a global model that excels on both majority and minority classes under the Federated Learning (FL) setting, poses another challenge.\n\nSGB establishes an expected target for the cumulative difference of positive and negative gradients across all clients, aiming to achieve synergistic aggregation. Instead of relying on predetermined heuristic algorithms [2-4], SGB incorporates a feedback loop for explicit reweighting and mitigates the long-tail bias of the model and enhances overall performance through the adaptive adjustment of the weighting coefficients of positive and negative gradients, mediated by a feedback loop.\n\n> **2. The paper's assumption of available global class priors for rebalancing, which may not always be applicable in real-world scenarios due to privacy constraints.**\n> \n\nThanks for the comments. We would like to clarify that in Fed-GraB, ***we did not assume that the global prior distribution is available***, neither to the clients nor the server. Instead, we only assume that the local distributions (heterogeneous or iid) from clients would aggregate a global long-tailed distribution, while the detailed characteristics of the global LT distribution is unknown. As for the local distributions, they were obtained by Dirichlet distribution-based sampling approaches, which might be balanced or long-tailed with different numbers of data samples. The detailed partition process are in the Supplementary Section 2.3. \n\n***As the global class prior is not available, we try to estimate the characteristics of the global distribution using the DPA module in a privacy-preserving manner.*** \n\nFor comprehensive contents of estimation accuracy under various imbalance factors and degrees of heterogeneity, please refer to lines 288-298 in the main paper. In real-world scenarios, we conducted experiments on real-world datasets including ImageNet-LT and iNaturalist, as demonstrated in Table 2 of the main paper.\n\nTo further illustrate the accuracy of DPA estimations, we performed additional evaluations of DPA's proficiency in identifying global $50\\%$ tail categories on CIFAR-100-LT. These results are recorded in the table of Question 2. The findings, as outlined in the table below, suggest that DPA offers accurate estimations across a broad spectrum of distributions.\n\n> **3. The ambiguity regarding how the proposed method would address situations where local distributions are either not long-tailed or exhibit a variety of long-tailed characteristics.**\n> \n\nBenefiting from the Dirichlet heterogeneous divisions in our work, ***the local distribution naturally encapsulates a variety of scenarios, including those that are not long-tailed and those that exhibit a range of long-tailed characteristics***. We have provided a heatmap (see Fig. 2 in the main paper), which allows for a convenient visual representation of the status of the local distribution. \n\nIn the main paper, we conducted numerous experiments with various imbalance factors and alpha values (non-IID), which generate a vast array of local distribution scenarios. Please refer to Table 1 in the main paper for further details.\n\nTo further demonstrate the performance of Fed-GraB across various local distributions, we also evaluated its performance under the IID condition. Specifically, we se CIFAR-10 with $IF_G=100$ and the results underscored the exceptional performance of our method under the IID condition, surpassing the current state-of-the-art by more than 1.9%.\n\n| Models | Many | Med | Few | All |\n| --- | --- | --- | --- | --- |\n| FedAvg | 0.929 | 0.720 | 0.595 | 0.733 |\n| CReFF | 0.938 | 0.734 | 0.592 | 0.738 |\n| FedNova | 0.927 | 0.736 | 0.625 | 0.749 |\n| EQLv2-FL | 0.932 | 0.734 | 0.595 | 0.738 |\n| Focal-FL | 0.929 | 0.710 | 0.573 | 0.721 |\n| Fed-GraB | 0.921 | 0.714 | 0.695 | 0.768 |\n\n> **4. A more comprehensive description of the interplay between the SGB and DPA modules is needed.**\n> \n\nSGB calculates $\\beta_j^{\\mathbf{pos}}(t), \\beta_j^{\\mathbf{neg}}(t)$. We combine global prior with SGB outputs:\n\n$\\beta_j = [\\beta_j^{\\mathbf{pos}}(t), \\beta_j^{\\mathbf{neg}}(t)] = \\mathbb{I}_{r > \\mathcal{P}_c[j]} \\cdot [\\beta^{pos}_j, \\beta^{neg}j] + $\n\n$\\mathbb{I}_{r \\leq \\mathcal{P}_c[j]} \\cdot [1, 1]$.\n\n$\\mathbb{I}$ is indicator function and the random value $r$ is sampled from a uniform distribution ranging from 0 to 1, denoted as $r ∼ Uniform(0, 1)$, suggesting that tail classes $j$ with smaller priors $\\mathcal{P}_c[j]$, have a higher probability of undergoing re-weighting.\n\n\n\n> **5. The proposed model needs to be corroborated with theoretical analyses or at least insights.**\n>\nDue to space limitation, this issue has been moved to the global response. Sorry for the inconvenience.\n\n[1] Addressing Class Imbalance in Federated Learning.\n\n[2] Seesaw Loss for Long-Tailed Instance Segmentation.\n\n[3] Fed-focal loss for imbalanced data classification in federated learning.\n\n[4] Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection."}, {"review_id": "MbfARBXtyT", "reviewer": "Reviewer_TCDG", "summary": "This paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit a long-tailed distribution. The authors propose the Fed-GraB to coordinate the global long-tailed distribution and the local learning strategy. Specifically, it consists of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients’ gradients in a closed-loop manner based on the feedback of global long-tailed prior derived from a Direct Prior Analyzer (DPA) module. Experiments are conducted on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist.", "questions": "1. What's the correlation between uj in Eq.3  and beta_j in Eq.4?  And what's the correlation between beta_j and (beta_j^pos, beta_j^neg)? This is essential for understanding the re-weighting process of SGB.\n2. In line 196, the author gives the definition of φ, but I could not find where the φ be applied.\n3. In Eq.4, a random number r is adopted as a threshold for Pcj, what's the role of it in the SGB?\n4. What's the definition of the expected target zj(t)? And why define the error feedback ej(t) as (gpos - gneg -  zj(t))?\n5. The EQLv2 strives to keep the cumulative positive and negative gradients to be equal for each category which seems to share similar ideas with the proposed method. Compared with EQLv2-FL, what are the novelty and advantages of this approach?", "limitations": "The authors have declared the limitations in their paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Investigating the federated long-tailed learning problem from the view of combining global and local sounds interesting.\n2. The design of the proposed method is in line with the motivation of the paper, which seems technically feasible.\n3. Experiments are conducted on many long-tailed benchmarks with various settings, which demonstrate the effectiveness of the proposed method.", "weaknesses": "Although the technical contribution of the manuscript sounds qualified, the presentation of the manuscript is not clear enough. There are some confusing issues that I will explain in the questions section.  Please resubmit a revised version. I will update my final score according to your revision. Meanwhile, too many hyperparameter settings will affect the generalizability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **1. What's the correlation between uj in Eq.3 and beta_j in Eq.4? And what's the correlation between beta_j and (beta_j^pos, beta_j^neg)?Where is the φ in line 196 be applied?**\n> \n\nWe apologize for any confusion caused due to the missing of certain details in the paper. We will reiterate the details here and update the revised version accordingly.\n\nThe error feedback $e_j(t)=\\Delta_j(t) - z_j(t)=\\Delta_j(t)$ for class $j$ represents the distance between the current status $\\Delta_j(t)$ and a target $z_j(t)$ during training. Hence, the input of the controller $e_j(t)$ is the cumulative difference of positive and negative gradients. Given $e(t)$, the output of the controller in SGB is $u_j(t) = K_{P}e_j(t)+K_{I} \\sum_{j=0}^{t}e_j(t) +K_D(e_j(t)-e_j(t-1))$. The output $u_{j}(t)$ indicates the strength to adjust the reweighting coefficients $\\beta_j$.\n\nWe first map $u_j(t)$ through a simple activation function $\\phi(\\cdot)$ to calculate $\\beta_j^{neg}=\\phi(-u_j(t)),~\\beta_j^{pos}=\\phi(u_j(t))$, where $\\phi(x)=\\frac\\gamma{1+\\delta e^{-\\zeta x}}$. Then, we combine the global information obtained by DPA to calculate the final re-weighting coefficients:\n\n$\\beta_j  = [\\beta_j^{\\mathbf{pos}}(t), \\beta_j^{\\mathbf{neg}}(t)] = \\mathbb{I}_{r > \\mathcal{P}_c[j]} \\cdot [\\beta^{pos}_j, \\beta^{neg}j] + $\n\n$\\mathbb{I}_{r \\leq \\mathcal{P}_c[j]} \\cdot [1, 1]$.\n\n$\\mathbb{I}$ is indicator function and the random value $r$ is sampled from a uniform distribution ranging from 0 to 1, denoted as $r ∼ Uniform(0, 1)$, suggesting that tail classes $j$ with smaller priors $\\mathcal{P}_c[j]$, have a higher probability of undergoing re-weighting.\n\nAfterwards, $\\beta_j$ is employed to reweight the positive and negative gradients.\n\nWe will update the missing part about $\\phi(\\cdot)$ in the revised version. \n\n> **2. In Eq.4, a random number r is adopted as a threshold for Pcj, what's the role of it in the SGB?**\n> \n\nThanks for the comments. The value of $P_c(j)$ serves as a threshold, determining whether the SGB should operate during each local iteration. In essence, SGB is a class-wise balancer that can enhance the performance of designated classifiers. $P_c(j)$ is seen as the degree of imbalance, approximating the global distribution of class j. We've designed three deployment strategies:\n\n**S1**: SGB is applied to all classes and operates continually.\n\n**S2**: SGB is only applied to estimated tail classes and operates continually.\n\n**S3**: SGB is applied to all classes and operates based on priors derived from DPA.\n\nWe discovered that **S1** can critically undermine the model due to overcorrection of the head classes, resulting in a catastrophic representation learning. As for **S2**, we observed that it facilitates model learning. Furthermore, applying SGB to different proportions of tail-end classes, as estimated by DPA, results in varying model performances (indicated by black dots, please refer to Fig. 5 (b) in the main paper). Therefore, **S2** requires the selection of an appropriate tail-end proportion. Thus, in step **S3**, there is no longer a need for the manual selection of specific-tailed classes. We utilize DPA to globally regulate the activation of SGB based on the comparison between the sampled value $r$ and $P_c(j)$, which readily achieves the optimal effect as seen in step **S2** (refer to the dashed line in Fig. 5 (b)).\n\n> **3. What's the definition of the expected target zj(t)? And why define the error feedback ej(t) as (gpos - gneg - zj(t))?**\n> \n\nThe term $z_j(t)$ represents the desired target of $g_j^{pos}(t)-g_j^{neg}(t)$. During the training process, $g_j^{pos}(t)-g_j^{neg}(t)$ is adaptively constrained to stay close to $z_j(t)$. As for the desired target $z_j(t)$, in a simplified balanced scenario where there are $M$ samples each having an equal probability for the $M$ classes, the expectation of the target $z_j(t)$ for $\\Delta_j(t)$ is given as: $E(z_j(t))=\\sum_{i=1}^{M-1}(E(\\sigma_i(t))-1)-E(\\sigma_j(t))=0.$\n\nThis signifies that $\\Delta(t)$ approaches zero when the distributions become identical. Hence, we define $z_j(t)$ as 0, which implies the ideal balance point from the perspective of positive and negative gradients.\n\nThe term $e_j(t)$ is the input to the PID controller, typically defined as the distance between the set ideal value $z_j(t)$ and the actual $g_j^{pos}(t)-g_j^{neg}(t)$. The controller adapts the positive and negative gradient weighting coefficients based on this error distance, to ensure that $g_j^{pos}(t)-g_j^{neg}(t)$ is better constrained by $z_j(t)$.\n\n> **4. Meanwhile, too many hyperparameter settings will affect the generalizability of the proposed method.**\n> \n\nWe thank the reviewer for pointing out this. We agree that the presence of certain hyperparameters in our algorithm, including the internal parameters of the PID controller ($K_P, K_I, K_D$) and the parameters of the mapping function ( $\\gamma,\\delta,\\zeta$). \n\n***We have provided a comprehensive ablation study to discuss the impact of these six parameters in Sec. 4.2, 4.3, and 4.4 of the Supplementary Material and Sec. 4.4 of the main paper***. The conclusion drawn is that $K_P$ has a substantial impact on model performance, with a total accuracy increase of $3.4\\%$ observed when $K_P$ is adjusted from $3$ to $10$. Meanwhile, other parameters demonstrate significant robustness. In addition, we carried out experiments on the CIFAR-100/10, ImageNet-LT, and iNaturalist datasets. We welcome further discussions on issues related to generalizability.\n\n> **5. The EQLv2 strives to keep the cumulative positive and negative gradients to be equal for each category which seems to share similar ideas with the proposed method. Compared with EQLv2-FL, what are the novelty and advantages of this approach?**\n>\n\nDue to space limitation, this issue has been moved to the global response. Sorry for the inconvenience."}]

Author rebuttal:
# For Reviewer 3

> **5.The proposed model needs to be corroborated with theoretical analyses or at least insights, especially regarding the function and operation of the DPA modules.**
> 

Regarding SGB, we provided the expected target $z_j(t)$ under ideal equilibrium conditions in Equation 2 of the paper. As for the error feedback regulator, our insights hail from the theory of automatic PID control within control theory, which is underpinned by a robust theoretical foundation. This includes various analyses such as Bode plots, Nyquist plots, and evaluations of phase margin and gain margin [1-5].

In the context of DPA, we explain the relationship between the Euclidean Norm of the classifier and the real distribution from two perspectives: the neural collapse and forward propagation.

From the perspective of neural collapse, four deeply interconnected phenomena occur during the training and gradual convergence of the neural network [6]. Among these, the third phenomenon (Neural Collapse 3) reveals that, up to rescaling, the last-layer classifiers, which implicitly represent the classifier decision, collapse to the class means. For balanced datasets, class means are uniformly distributed and form an equiangular tight frame simplex. However, for imbalanced datasets, class means naturally exhibit bias, and the classifier decision converges to these biased class means [7,8]. Therefore, when the model tends to predict the head classes, the class means of these classes become larger, leading to greater decision boundaries and a larger Euclidean Norm of the class vector. Numerous theories and experiments detailing this phenomenon are presented in [6,7].

From the forward propagation standpoint, the class with the maximum logit is selected as the predicted class. The logit $s_{ij}=f_iw_j^\top$ can be written as $\Vert f_i \Vert_2 \Vert w_j \Vert_2\cos\theta_{ij} \propto \Vert w_j \Vert_2\cos\theta_{ij}$, suggesting a large Euclidean Norm of $w_j$ biases the model towards predicting the corresponding class.

To validate the effectiveness of DPA, we conducted various experiments on CIFAR100-LT with different imbalance factors and heterogeneity, to demonstrate the estimation accuracy of DPA under various circumstances. The accuracy of global $50\%$ tail categories was determined as the ratio of correct predictions to the total number of predictions, with imbalance factors of 2, 50, and 100 and alpha values of 15, 0.5, and 0.05. The results are recorded in the following table. As can be seen, DPA demonstrates excellent generalizability and achieves higher accuracy under severe long-tail conditions. Additionally, we conducted experiments to illustrate the relationship between the Euclidean Norm of the classifier and the real distribution, as shown in ***Fig. 1 of the rebuttal material.***

|  | Alpha=15 | Alpha=0.5 | Alpha=0.05 |
| --- | --- | --- | --- |
| IF=2 | 0.780 | 0.740 | 0.680 |
| IF=50 | 0.960 | 0.980 | 0.860 |
| IF=100 | 0.980 | 0.980 | 0.880 |

[1] PID control system analysis, design, and technology.

[2] Tuning of PID controllers based on Bode's ideal transfer function.

[3] The direct Nyquist array design of PID controllers.

[4] Tuning of PID controllers based on gain and phase margin specifications.

[5] Performance and gain and phase margins of well-known PID tuning formulas.

[6] Prevalence of neural collapse during the terminal phase of deep learning training.

[7] Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training.

[8] Decoupling representation and classifier for long-tailed recognition.

# For Reviewer 4:
> **5.The EQLv2 strives to keep the cumulative positive and negative gradients to be equal for each category which seems to share similar ideas with the proposed method. Compared with EQLv2-FL, what are the novelty and advantages of this approach?**
> 

We thank the reviewer for pointing out this. As one of our baselines, EQLv2, developed under a centralized learning, is an effective method to alleviate the long-tail problem and has provided significant insights for the design of our methods. We will expound on the limitations of EQLv2 in the context of federated learning (FL) via two implementation methods: global EQLv2 and local EQLv2.

1.Applying local EQLv2 in FL, we noted a considerable variance in $g_j^{pos}(t)-g_j^{neg}(t)$ across different clients during the model aggregation, indicating a substantial divergence among them, which is not favourable for federated learning (please refer to Fig. 4: (a) in the main paper). SGB, on the other hand, constrains all clients' $g_j^{pos}(t)-g_j^{neg}(t)$ with the feedback loop to satisfy $E(z_j(t))=0$, thereby ensuring coordination among different clients, without the individually designing re-weighting parameters  ($\alpha, \gamma, \mu$ in EQLv2) that are suitable for each client's local distribution.

2.In the case of global EQLv2, it is necessary to upload each client's accumulated positive and negative gradients. As these accumulations closely mirror the local distributions (***refer to Fig. 2 of the rebuttal material***), this can potentially lead to privacy leakage concerns and associated risks. Conversely, SGB does not require the transmission of any extra information, thus mitigating the risks associated with privacy leakage.

In summary, while EQLv2 is an effective solution in a centralized learning environment, FedGraB has been designed for federated long-tailed learning, taking into account factors of distributed training such as self-adjusting gradients synchronization among clients and privacy preservation.

Meta-review: This paper identifies a federated long-tailed learning scenario and proposes a novel method called Fed-GraB to address the corresponding challenges. It designs a Self-adjusting Gradient Balancer (SGB) module and a Direct Prior Analyzer (DPA) module that work collaboratively to resolve the Fed-LT task in a close-loop and privacy-preserving manner. The paper conducted comprehensive experiments on multiple datasets, exhibiting superior performance over extensive baselines. After the rebuttal and discussion, all reviewers  recommended accepting this paper.
Decision: Accept (poster)