Title: Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated learning

Long-tailed learning

Data heterogeneity
Abstract: Data privacy and long-tailed distribution are the norms rather than the exception in many real-world tasks. This paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit a long-tailed distribution. Under such a setting, existing federated optimization and/or centralized long-tailed learning methods hardly apply due to challenges in (a) characterizing the global long-tailed distribution under privacy constraints and (b) adjusting the local learning strategy to cope with the head-tail imbalance. In response, we propose a method termed $\texttt{Fed-GraB}$, comprised of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients' gradients in a closed-loop manner, based on the feedback of global long-tailed distribution evaluated by a Direct Prior Analyzer (DPA) module. Using $\texttt{Fed-GraB}$, clients can effectively alleviate the distribution drift caused by data heterogeneity during the model training process and obtain a global model with better performance on the minority classes while maintaining the performance of the majority classes. Extensive experiments demonstrate that $\texttt{Fed-GraB}$ achieves state-of-the-art performance on representative datasets such as CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist.

Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer

Abstract

Data privacy and long-tailed distribution are the norms rather than the exceptions in many real-world tasks. This paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit a long-tailed distribution. Under such a setting, existing federated optimization and/or centralized longtailed learning methods hardly apply due to challenges in (a) characterizing the global long-tailed distribution under privacy constraints and (b) adjusting the local learning strategy to cope with the head-tail imbalance. In response, we propose a method termed Fed-GraB , comprised of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients' gradients in a closed-loop manner based on the feedback of global long-tailed prior derived from a Direct Prior Analyzer (DPA) module. Using Fed-GraB , clients can effectively alleviate the distribution drift caused by data heterogeneity during the model training process and obtain a global model with better performance on the minority classes while maintaining the performance of the majority classes. Extensive experiments demonstrate that Fed-GraB achieves state-of-the-art performance on representative datasets such as CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist. Our codes are available at https://github.com/ZackZikaiXiao/FedGraB.

1 Introduction

Federated learning (FL) is an approach for massively distributed clients to train a global machine learning model collaboratively without exposing their private data, which has garnered ever-increasing attention in academia and industry alike [1, 2]. Unlike conventional machine learning methods, FL brings the learning objective directly onto the end-user devices for local training, where only the intermediate parameters (e.g., gradients) need to be sent to a server for model aggregation and update. This approach not only substantially reduces the communication overheads but, more importantly, facilitates the clients in obtaining a generic global model without sharing their private data, thereby contributing to the development of trustworthy intelligent systems [3]. Despite its great potential,

†

(a)

Server

Client 1

Balanced

Weights

Client

N

Tail

8.95

4.47

3.73

3.93

Logits for Panda

Batch of images

SGB

Gradient

Difference

Re-weighted

Gradients

Middle

FedGraB Framework

Positive

Negative

DPA

Classifiers

Head

Direct Prior Analyzer (DPA)

8.95

8.95

4.47

3.93

3.93

3.73

3.73

4.47

Head&Middle

Tail

Classifiers of Global Model

Prior Vector

Local Training with SGB

0

500

1000

1500

-2

0

2

Iterations

Eqlv2-FL

Softmax

Ours

(b)

△;(t)

Wg

△;(t)

qneg

C

realizing FL also requires one to overcome new hurdles in real-world implementations, among which a particular challenge stems from data heterogeneity, i.e., the non-IID distribution and imbalanced dataset sizes across clients [4-7]. These factors often give rise to a pernicious issue where the data set of different clients jointly follows a global long-tailed distribution, which is commonly known as the Federated Long-Tailed learning (Fed-LT) problem [8, 9]. For instance, patients' diagnosis varies substantially across medical centers but collaboratively form long-tailed distributions for certain diseases [10, 11]. In a broad range of applications, performance on the minority classes (i.e., tail classes) [12], such as rare diseases, dangerous behaviors in autonomous driving [13, 14], and abnormal breath or heart rates in wearable devices [15], play a pivotal role in developing reliable and robust solutions to an FL system.

The difficulty in addressing the Fed-LT problem primarily stems from the fact that although clients' datasets can be locally balanced, globally, i.e., if the datasets were aggregated, the distribution may be long-tailed. As clients would not disclose their data information due to privacy concerns, identifying the global long-tailedness in data distribution could be onerous. To that end, a natural question arises:

Question 1: How to leverage the global longtailed statistics, especially for the minorities that need to be carefully tailored, without conflicting with privacy concerns?

On the other hand, even if the globally long-tailed data information is available, existing methods do not directly apply to address such a problem. Specifically, typical FL methods cope with data heterogeneity via generic techniques such as dynamic regularizations, client selections, data augmentation, distillation, and personalization training [16-19, 4, 20, 21]. These approaches ignored the diverged/imbalanced class levels that can result in the global long-tailed issue [22, 23]. Hence, they cannot guarantee decent performance for the tail classes. A key ingredient to improve longtailed learning is to boost the performance of tail

Class

Client

Class

1

6

8

6

06080L090500E0Z0

classes without degrading the performance of the head classes [24, 25]. While several learning methods dedicated to long-tailed data exist, they usually assume global class priors for re-balancing, which are not available in the context of FL due to privacy concerns [8]. Moreover, these methods may not be applicable in local training as local distributions could present diverse long-tailed characteristics and/or are not even long-tailed [9]. As such, it begs another question as follows:

Question 2: How to perform local training to synergistically aggregate a global model that excels on both majority and minority classes under the FL setting?

A possible choice is to re-balance the clients' local gradients via approaches such as Seasaw loss [26] or Equalization Loss v2 [27]. However one needs to customize these methods on different clients with heuristically selected hyper-parameters, resulting in complicated models with limited generalization capability. Moreover, the trained local models may have huge variance, as illustrated in Fig. 4 (a), and hardly formulate a synergistic model to handle tail class.

In light of the above challenges, this paper proposes Fed-GraB , comprised of a long-tailed statistic analyzer and a gradient rebalancer, for the Fed-LT problem, as illustrated in Fig. 1. More specifically, we develop a federated weight-norm-based method, coined as the Direct Prior Analyzer (DPA), to derive a prior vector for global data statistics, utilizing the weight parameters from the global classifier. To overcome the degradation caused by the distribution discrepancy between the server and clients (or amongst the clients) in Fed-LT, we establish a self-adjusting gradient balancer (SGB) based on the estimated global head-tail characteristics, which is integrated at every client to rebalance the positive and negative gradients in a class-wise and privacy-preserving manner. A marked advantage of Fed-GraB is that it trains all heterogeneous clients in a feedback-based closed-loop manner, encouraging all clients to contribute to the global long-tailed recognition task collaboratively. We also provide comprehensive theoretical and empirical analyses to verify the effectiveness of Fed-GraB . We conduct excessive experiments on multiple benchmark datasets (including the CIFAR10/100-LT, ImageNet-LT, and iNaturalist) with both long-tailed and non-IID data distributions. The results demonstrate that Fed-GraB outperforms state-of-the-art (SOTA) baselines, including federated optimization for heterogeneous data, FL methods for LT classification, and centralized LT learning methods in FL settings.

2 Related Work

2.1 Federated learning with data heterogeneity

Numerous FL approaches have been developed to cope with the heterogeneous or imbalanced data distribution. For instance, FedProx [16], FedDyn [17], and MOON [28] modify the local loss function by adding regularizers at the local side. SCAFFOLD [18] proposes a control variates-based method to reduce the clients' distribution drift brought by the discrepancies. FedIR [4] dynamically adjusts the sampling ratio of each client based on each client's contribution and data distribution situation. And FedNova [20] is proposed to tackle the induced objective inconsistency problem. On the server side, FedAvgM [19] and FedAdaGrad [29] are proposed to mitigate the performance degradation. Moreover, FL-oriented client selection [30, 31] and data augmentation strategies [32] have been investigated to enhance the performance. To address the poor generalization of a single generic model, personalization training methods have been recently explored to maintain multiple personalized local models [33, 34, 21]. However, most of these methods pay more attention to the discrepancies in inter-client distributions while ignoring the inconsistency among different classes, failing to achieve satisfactory performance on the tail classes. In lieu of this, an alternate paradigm [35] solely addresses the personalized long-tail conundrum, disregarding the variances within inter-client distributions. Notable exceptions could be found in [8, 36, 6] to tackle class imbalance. Still, they require the exchange of local features [8], private local data distribution [36], and auxiliary data [6], which would lead to potential privacy issues.

2.2 Long-tailed learning

Long-tailed data widely exists in real-world machine learning tasks, where head classes dominate the training [10, 37]. To tackle the poor performance of tail classes in such scenarios, many approaches have been explored to boost the performance of tail classes via class re-balancing, information augmentation, and module improvements [38-41, 25]. Re-balancing strategies, as the mainstream approaches in long-tailed learning, can be categorized into class re-sampling, re-weighting, and logit adjustments [10, 42]. Re-sampling approaches [43-46] address the class imbalance via over-sampling tail classes or under-sampling head classes. On the other hand, the re-weighting approaches aim to balance the loss( [47-50]) or gradients (Seesaw loss [26], Eqlv2 [27]). Specifically, Focal loss [39] increases the prediction probabilities to achieve better prediction performance for tail classes. And a label-distribution-aware margin loss LDAM is introduced by [49] using label frequency. Moreover, a

multi-expert framework (TADE [51]), prototype learning (OLTR [24]), and head-to-tail knowledge transfer (LEAP [52]) have been proposed for improved long-tailed data performance. However, such methods' effectiveness may be constrained in Fed-LT due to discrepancies in local and global statistics.

3 Method

3.1 Preliminaries

Federated Learning: Consider an FL system with N clients and an M -class visual recognition dataset D = {D k } N k =1 for classification tasks. D k denotes the local dataset of client k with size |D k | . Usually, the FL training process iterates multiple communication rounds until convergence. In a typical communication round l , the central server randomly selects a subset of clients S l and distributes the latest global model w l to them. For client k ∈ S l , it performs local update to w l based on D k , where the locally computed model is denoted as w l k . At the end of the round, the global model could be updated by the aggregation criterion (e.g., FedAvg ) for the next round computation.

Long-tailed distribution in FL: To characterize the long-tailed data distribution in Fed-LT, we measure the imbalance factors in global perspectives, which is denoted by IFG and computed over the global dataset D . We could employ a predefined imbalance IFG to sample from balanced datasets, constructing long-tailed datasets; see Fig. 2 for visualization.

Positive and Negative Gradients: Notably, positive and negative cumulative gradients have an important effect on long-tailed tasks such as visual recognition [53], object detection [27], and instance segmentation [26]. Suppose we have a neural network-based classification model with Cross Entropy loss as L ( z ) = -∑ M i =1 y i log( σ i ) , with σ i = e z i ∑ j M =1 e z j , where y i is the ground truth label, z = [ z 1 , z 2 , . . . , z M ] are the logits, and σ = [ σ 1 , σ 2 , . . . , σ M ] denotes the probabilities of the classifier. Given a sample with label j , the positive gradients are defined as the derivative of the loss with respect to z j , while the rest shall be the negative gradients, i.e.,

̸

Gradient

Collector

Re-weighting

Re-weighting Target

Compute

Gradients

Update Parameters

& Feed-forward

Training Loop

Re-weighting Feedback

Figure 3: The closed-loop in SGB. β pos j ( t ) , β neg j ( t ) are updated according to the controller, and the re-weighted gradients are stored in the collector for future cumulative computation of ∆ j ( t ) .

3.2 Direct Prior Analyzer (DPA)

This section details the proposed DPA module which calculates a prior vector for balanced training. It utilizes the same information required by FedAvg, without the need for extra data or distribution details. Essentially, DPA derives a prior probability vector that discerns potential tail classes by examining the L2-norm of the weight parameters within the classifiers. The reason for this design arises from existing studies [25, 54], revealing that a model recognizes head or tail classes via classifier and tends to give higher scores for head classes when trained with imbalanced data. Thus, the weight norm serves as an effective indicator of the imbalance degree since the norm of the active weights could empirically reflect the behavior and frequency of different classes. In particular, let w j g denote the weights of classifier j in the global model classifier with weights w g . The estimated global distribution would be given by P c = { p j | h ( ∥ w j g ∥ 2 ) } , where h ( · ) is a linear transformation for scaling the norm to share of the total. This estimation can be executed after global model aggregation with full client participation.

In the Fed-GraB implementation, the prior vector P c derives from DPA at the onset of local training. It is also worth mentioning that P c does not need to be perfect and only requires information that reflects the model prediction inclinations at the class level, rather than a complete correspondence with the distribution itself. See more detailed analysis in experiments.

3.3 Self-adjusting Gradient Balancer for Fed-LT

This part establishes the Self-adjusting Gradient Balancer (SGB), an effective tool that can be integrated into biased classifiers to re-balance local training and improve the overall performance. Particularly, we consider the difference between cumulative positive and negative gradients ∆( t ) , with ∆ j ( t ) for class j defined as ∆ j ( t ) = g pos j ( t ) -g neg j ( t ) , where g j ( t ) = ∑ t i =0 ∇ z j ( L t ) is the cumulative gradients of class j , capturing the overall imbalance degree throughout the training process. In a toy balanced scenario with M samples each with equal probability to the M classes, the expected target z j ( t ) of ∆ j ( t ) is:

indicating that ∆( t ) approaches zero when the distributions become identical.

The core idea of SGB is to consistently align ∆ j ( t ) to 0 for tail classes in a closed-loop and selfadjusting manner. Specifically, we carry out a proportional-integral-derivative controller [55] to continually update the re-weighting coefficients of gradients with regard to the logits in a class-wise manner, as demonstrated in Fig. 3. The re-weighting process is based on adaptive self-adjustment with error feedback until ∆( t ) reaches the pre-determined balanced target, rather than following heuristic methods as [26, 27]. The error feedback e j ( t ) = ∆ j ( t ) -z j ( t ) for class j represents the distance between the current status ∆ j ( t ) and a target z j ( t ) during training. Given e ( t ) , the output of the controller in SGB is

where K P , K I , and K D are controlling factors. The proportional item with K P shows the controller would give bigger re-weighting coefficients β j to compensate the imbalance for current larger errors. Meanwhile, the integral item with K I could look back on the past errors to reduce the steady-state offset which could not be handled by the proportional item. Furthermore, as computing β j only with the proportional and integral loop could hardly achieve steady and smooth re-weighting, we introduce the third differential item with K D to control the fluctuations based on future expectations. For instance, when ∆ j ( t ) is close to 0 , the differential item would prohibit u ( t ) from diverging re-weighting and vise versa. Overall, the three items work together to achieve quick, precise, and stable adjustments.

We first map u j ( t ) through a simple activation function ϕ ( · ) to calculate β neg j = ϕ ( -u j ( t )) , β pos j = ϕ ( u j ( t )) , where ϕ ( x ) = γ 1+ δe -ζx . Then, we combine the global information obtained by DPA to calculate the final coefficients for positive and negative gradients per iteration as:

where I ( · ) is the indicator function. Consequently, the re-weighted gradients are computed as:

The working mechanism of SGB at a typical client is summarized in Fig. 3. As depicted in Fig. 1 (a), SGB is implemented across all the classes in each client, with parameters adjusted in accordance with the prior vector derived by DPA. By utilizing global information, this approach encourages all clients to contribute coherently to the global model while simultaneously rectifying prediction biases arising from long-tailed datasets. As shown in Fig. 1 (b), SGB improves the balance between cumulative positive and negative gradients. See experiments and Supplementary for further analysis.

3.4 Algorithm Summary for Fed-GraB

The training pipeline of our Fed-GraB framework is presented in Algorithm 1, which consists of two stages. First, DPA calculates the prior vector P c after receiving aggregated weights w l k . Then, SGB is applied to all the classes. The local classifier would be dynamically adjusted with weighted positive and negative gradients via a ∆( t ) -based closed-loop controller in accordance with prior vector P c derived from DPA. The combination with SGB and DPA is detailed in Eq. (4).

3.5 Privacy discussions of DPA

As the local prior computation is performed at the client side, there would be a few privacy concerns with the DPA method. It is, however, noteworthy that the potential privacy issue exists in the general FL frameworks rather than specific to our proposed DPA method. For instance, gradient inversion [56] can pose a threat to almost all gradient transmission-based FL methods without any privacy-preservation techniques. As the privacy issue of the FL framework is beyond the scope of this work, we briefly include the discussion in this subsection.

4 Experiments

4.1 Experimental Setup

Baselines: We consider two types of SOTA baselines: (1) FL-oriented methods to tackle data heterogeneity ( FedProx [16], FedNova [20]) or federated long-tailed data-oriented ( FedIR [4] and CReFF [8]), and FedAvg is also included for reference; (2) Long-tailed learning (LT)-oriented methods ( τ -norm [25], Eqlv2 [27], LDAM [49], Focal-loss [39] and GCL-loss [42]) applied at local training of each client. We also provide the results of the long-tailed methods in centralized learning (CL) settings as the oracle performance upper bound, including SGB.

Datasets: We conduct the experiments in three benchmark datasets for long-tailed classification, i.e., CIFAR-10/100-LT [57], ImageNet-LT [58]. CIFAR-10/100-LT are sampled into long-tailed distribution by exponential distribution controlled by IF [49], and we use the same configurations as in [24] for ImageNet-LT with the number of images per class ranging from 5 to 1280. To evaluate the performance on real-world data, we also conduct experiments on iNaturalist-User-160k, with 160k examples of 1,023 species classes and partitioned on the basis of iNaturalist-2017 [59].

Federated settings: We use non-IID data partitions for all experiments, implemented via symmetric Dirichlet distributions with concentration parameter α to control the identicalness of local data distributions among all the clients. We train a ResNet-18 over N = 40 clients on CIFAR-10-LT. ResNet-34 and ResNet-50 are used on CIFAR-100-LT and ImageNet-LT respectively with N = 20 clients. For iNaturalist-160k, we use the same settings as ImageNet-LT.

Implementation of baselines: For the sake of fairness, we keep consistent settings external for experiments. We conduct experiments using a starting model from FedAvg . For CReFF , the number of federated features is 100, we use 0.1, 0.01 as federated feature learning rate and main net learning rate respectively on CIFAR-10/100-LT. We report the official result [8] on ImageNet-LT. As τ -norm is a one-shot method, we provide a pre-trained model by FedAvg and adjust the classifier weights on the server as post-process. Focal-Loss and EQL v2 do not require distribution prior. We directly replace the local cross-entropy loss with their proposed re-balancing methods. For LDAM , we use the local data distribution for its local training.

4.2 Comparison with State-of-the-art Methods

Evaluation on CIFAR-10-LT:

The performance of Fed-GraB /SGB on diverse settings are reported in Tab. 1. Notably, Fed-GraB achieves the best overall accuracies on all settings, with a significant improvement on the tail classes while ensuring good performance of the head classes. The performance gain becomes more evident under extremely imbalanced data. Moreover, the SGB implemented in the CL setting (i.e., performance upper bound testing) surpasses existing SOTA long-tail methods, further demonstrating the universal effectiveness of Fed-GraB under different scenarios.

Evaluation on CIFAR-100-LT, ImageNet-LT, and iNaturalist-160k: Tab. 2 shows that Fed-GraB consistently outperforms all baselines on test overall accuracies for CIFAR-100-LT and ImageNet-LT, with remarkable improvements on middle and tail classes. While LT-orientated methods obtain better results than the FL method in the extremely imbalanced ImageNet-LT, Fed-GraB can still surpass it 3 . The reason boils down to that for classification tasks with a large number of classes, local data samples for each class would be extremely small. In this case, the SGB with global prior for class-wise re-balancing is more effective in learning a better-aggregated model.

As for iNaturalist-160k, Fed-GraB significantly boosts the classification accuracy on tail classes while maintaining the best overall performance compared to FL methods. In comparison to LToriented methods, Fed-GraB 's SGB provides a robust mechanism against data heterogeneity, leading to an outstanding performance on both 3-shot and overall metrics.

A salient advantage of Fed-GraB is that it achieves universal outperformance on three large-scale or real-world scenarios, demonstrating the robustness over other methods such as CReFF , τ -norm or Eqlv2-FL which attain good performance on a specific dataset.

Evaluation of Communication Efficiency: Tab. 3 presents the communication efficiency of different methods trained on CIFAR-10-LT under two non-IID settings. We compare the required communica-

tion rounds for the tail classes to reach 55% test accuracy. We can see from this table that Fed-GraB attains the best convergence rate under both settings, confirming its effectiveness on the tail classes.

4.3 Effectiveness of DPA and SGB

SGB for consistent gradient re-weighting across clients: When various clients employ a gradient re-weighting method on a specific class, the strength of compensating divergence is because of the diverse local distributions and local dataset sizes. Therefore, although traditional re-weighting methods can mitigate the imbalance, discrepancies still exist. We demonstrate the advantage of SGB for addressing this issue. Specifically, we compute the mean and standard deviation (std) of ∆ j ( t ) across all clients in CIFAR-10-LT for both Eqlv2 and Fed-GraB , as illustrated in Fig. 4 (a).

We make two observations here. First, the mean of ∆ j ( t ) in Eqlv2 keeps decreasing alongside the training process while Fed-GraB can always align it to a near-zero value, suggesting that Fed-GraB can better re-weight the positive and negative gradients. Second, Eqlv2 exhibits a large variance among clients during training. In contrast, Fed-GraB maintains a very small deviation, evidently demonstrating the Fed-GraB can consistently motivate the heterogeneous clients to converge towards a better aggregated global model.

Effectiveness on global distribution and tail class estimation: We evaluate DPA under three IF settings with two metrics: (1) the L2-distance

between the prior vector P c and ground truth distribution; (2) the tail identification accuracy (percentage of tail classes that are identified correctly by DPA). We plot the two metrics on a global model with FedAvg for 200 rounds in Fig. 4 (b). We can see that DPA converges after 70 rounds of training and captures more than 90% of the tail classes under different IFs. Moreover, the tail identification accuracy becomes more accurate with a higher IF, which is attributed to the extreme imbalance among classes (i.e., easier to distinguish).

0

500

1000

1500

-3

-2

-1

0

1

Iterations

Average of

Eqlv2

Ours

0

50

100

150

200

1

2

3

4

5

0.2

0.4

0.6

0.8

1.0

Rounds

L2-Distance

IF=10

IF=50

IF=100

Distance

Tail Estimation Accuracy

0

100

200

300

400

500

0.2

0.4

0.6

0.8

Rounds

Test Accuracy

Local-SGB

Global-SGB

(a)

(b)

(c)

Numbers

0

1

2

3

4

5

6

7

8

9

0

50

100

150

200

250

0.0

0.2

0.4

0.6

0.8

1.0

Class

Softmax

Ours

0-9

1-9

2-9

3-9

4-9

5-9

6-9

7-9

8-9

9-9

0.70

0.72

0.74

0.76

0.78

Mounting Strategy

DPA

Partially Mounted

Test Accuracy

Iterations

Phase 1

Phase 2

Training

Initialization

(a)

(b)

(c)

Effectiveness of global P c : We investigate a local version (Local-SGB) of Fed-GraB (Global-SGB), where each client implements SGB based on their local distribution instead of the global prior vector derived by DPA. The results on CIFAR-10-LT in Fig. 4 (c) show that Global-SGB achieves a higher

test accuracy. Furthermore, we visualize the accuracies of FedAvg and Fed-GraB after local updates on the received global model in Fig. 5 (a), which demonstrate that Fed-GraB can achieve better and more balanced performance across classes, especially for tails (e.g., class 2, 3, 8, 9).

Mounting strategies with SGB: In Fed-GraB , SGB is universally applied to all classifiers through the use of DPA, rather than selectively targeting specific classes without DPA. We demonstrate that a full mounting approach with DPA can more readily attain optimal performance compared to a custom-tailored tailed class (i.e., mounting on 7 to 9 classes). As depicted in Fig. 5 (b), the model excels when SGB is employed on tail classes, while accuracy diminishes when mounting all classes or solely the exceedingly biased classes. Notably, DPA significantly bolsters overall performance.

4.4 Ablation Study and Model Analysis

The target of ∆( t ) : A critical step of SGB is to compute re-weighting coefficients β by e ( t ) = ∆( t ) -z ( t ) . Here we argue that for a static z ( t ) , different values of z ( t ) would not lead to significant differences in training, as long as those values are not extremely biased. We visualize the training process with two different z ( t ) in Fig. 5 (c) by assuming a continuous scenario. The training would consist of two phases during the closed-loop controlling. The Phase 1 could be regarded as a noisy initialization stage which is usually very short, resulting in different initial values based on e ( t ) . Afterward, the Phase 2 is started, and similar adjusting behaviors of e ( t ) could be observed, i.e., ∆( t ) fluctuates around z ( t ) , leading to similar β that stands for re-weighting strength. As the duration δ of Phase 1 is negligibly short and would not impose significant influence for training. The experimental results with different target values in Fig. 6 (a) further demonstrate our argument. More analysis is presented in the Supplementary.

-20

-10

0

10

20

0.80

0.90

Test Accuracy

Fed-GraB

FedAvg

0

200

400

600

800

1000

Iterations

-2

0

2

-2

0

2

-2

0

2

-2

0

2

-2

0

2

(t)

FedAvg

[  1,      0,    0]

[10,      0,    0]

[10,      0, 0.1]

[10, 0.01, 0.1]

10

20

30

40

50

0.80

0.85

0.90

Test Accuracy

FedAvg

FedNova

Fed-GraB

(a)

(b)

(c)

Ablation on SGB hyper-parameters: To investigate the effects of hyper-parameters K P , K I , and K D , we observe the real-time ∆( t ) during the training process with various parameter groups, as depicted in Fig. 6 (b) using the parameter sets: K P = 1 , 10 , 10 , 10 , K I = 0 , 0 , 0 , 0 . 01 , K D = 0 , 0 , 0 . 1 , 0 . 1 . We notice that K P = 10 (green) more effectively constrains ∆( t ) around 0 compared to K P = 1 (orange). To reduce fluctuations, we incorporate K D = 0 . 1 (red). In this case, ∆( t ) remains above 0 , necessitating the introduction of K I (purple) to correct the static deviation. Notably, SGB demonstrates substantial robustness to the parameter group, as further evidenced in the Supplementary.

Fed-GraB on different imbalance levels: To demonstrate Fed-GraB is versatile across different global data distributions, we conduct experiments with global imbalance factors ranging from 5 to 50. As shown in Fig. 6 (c), Fed-GraB achieves better performance than others on a broad range of IFG. The results indicate that Fed-GraB could alleviate the performance degradation in moderate imbalanced cases while yielding more significant improvements on highly imbalanced data.

Computational and Storage Cost of SGB:

The additional cost from SGB are mainly attributed to the computation and storage of ∆( t ) and u ( t ) . As shown in Eq. (3), the main computational steps are the differential and summation of e ( t ) , which should be linearly proportional to number of the gradients. Such extra computational cost is analogous to the additional manipulations of gradients in advanced stochastic gradient decent methods such as Momentum or Adam [60]. Overall, the extra computation which is implemented with several lines of code could be done very quickly. Besides, Fed-GraB needs some extra storage cost to store the weighted gradients which is quite cheap as well.

5 Conclusions and Limitations

We proposed Fed-GraB , a self-adjusting and closed-loop gradient re-balancing framework for Fed-LT tasks. Fed-GraB comprises DPA, a federated global long-tailedness analyzer, and SGB, a local gradient balancer, addressing the discrepancies in inter-client and inter-class statistics. We carried out extensive experiments to analyze the functionality of different components in Fed-GraB and demonstrate the efficacy of Fed-GraB in various Fed-LT configurations.

Limitations. In this work, we mainly focus on aggregating a global Fed-LT model, while extensions to include clients are of high interest such as training superior heterogeneous clients with with personalized strategies. Besides, experiments with real-world medical or autonomous vehicle datasets could be examined to further demonstrate the effectiveness of Fed-GraB .

References

Q. Yang, Y. Liu, T. Chen, and Y. Tong, 'Federated machine learning: Concept and applications,' ACM Transactions on Intelligent Systems and Technology (TIST) , vol. 10, no. 2, pp. 1-19, 2019. 1

T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, 'Federated learning: Challenges, methods, and future directions,' IEEE Signal Processing Magazine , vol. 37, no. 3, pp. 50-60, 2020. 1

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, 'Communication-efficient learning of deep networks from decentralized data,' in Artificial intelligence and statistics , pp. 1273-1282, PMLR, 2017. 1

T.-M. H. Hsu, H. Qi, and M. Brown, 'Federated visual classification with real-world data distribution,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X , pp. 76-92, 2020. 2, 3, 6

Q. Li, Y. Diao, Q. Chen, and B. He, 'Federated learning on non-iid data silos: An experimental study,' in 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pp. 965-978, IEEE, 2022.

L. Wang, S. Xu, X. Wang, and Q. Zhu, 'Addressing class imbalance in federated learning,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, pp. 10165-10173, 2021. 3

J. Xu, Z. Chen, T. Q. Quek, and K. F. E. Chong, 'Fedcorr: Multi-stage federated learning for label noise correction,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10184-10193, 2022. 2

X. Shang, Y. Lu, G. Huang, and H. Wang, 'Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features,' arXiv preprint arXiv:2204.13399 , 2022. 2, 3, 6, 7

Z. Chen, S. Liu, H. Wang, H. H. Yang, T. Q. Quek, and Z. Liu, 'Towards federated long-tailed learning,' in International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2022 (FL-IJCAI'22) , 2022. 2

Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, 'Deep long-tailed learning: A survey,' arXiv preprint arXiv:2110.04596 , 2021. 2, 3

L. Yang, H. Jiang, Q. Song, and J. Guo, 'A survey on long-tailed visual recognition,' International Journal of Computer Vision , pp. 1-36, 2022. 2

H. Tang, Z. Li, Z. Peng, and J. Tang, 'Blockmix: Meta regularization and self-calibrated inference for metric-based meta-learning,' in ACM Multimedia , pp. 610-618, ACM, 2020. 2

O. Makansi, Ö. Çiçek, Y. Marrakchi, and T. Brox, 'On exposing the challenging long tail in future prediction of traffic actors,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 13147-13157, 2021. 2

H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, 'nuscenes: A multimodal dataset for autonomous driving,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 11621-11631, 2020. 2

A. Haque, A. Milstein, and L. Fei-Fei, 'Illuminating the dark spaces of healthcare with ambient intelligence,' Nature , vol. 585, no. 7824, pp. 193-202, 2020. 2

T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, 'Federated optimization in heterogeneous networks,' Proceedings of Machine Learning and Systems , vol. 2, pp. 429-450, 2020. 2, 3, 6

D. A. E. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama, 'Federated learning based on dynamic regularization,' in International Conference on Learning Representations , 2021. 3

S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, 'Scaffold: Stochastic controlled averaging for federated learning,' in International Conference on Machine Learning , pp. 5132-5143, PMLR, 2020. 3

T.-M. H. Hsu, H. Qi, and M. Brown, 'Measuring the effects of non-identical data distribution for federated visual classification,' arXiv preprint arXiv:1909.06335 , 2019. 2, 3

J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, 'Tackling the objective inconsistency problem in heterogeneous federated optimization,' Advances in neural information processing systems , vol. 33, pp. 7611-7623, 2020. 2, 3, 6

A. Z. Tan, H. Yu, L. Cui, and Q. Yang, 'Towards personalized federated learning,' IEEE Transactions on Neural Networks and Learning Systems , 2022. 2, 3

M. Luo, F. Chen, D. Hu, Y. Zhang, J. Liang, and J. Feng, 'No fear of heterogeneity: Classifier calibration for federated learning with non-iid data,' Advances in Neural Information Processing Systems , vol. 34, pp. 5972-5984, 2021. 2

J. Zhang, Z. Li, B. Li, J. Xu, S. Wu, S. Ding, and C. Wu, 'Federated learning with label distribution skew via logits calibration,' in International Conference on Machine Learning , pp. 26311-26329, PMLR, 2022. 2

Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, 'Large-scale long-tailed recognition in an open world,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2537-2546, 2019. 2, 4, 6

B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, 'Decoupling representation and classifier for long-tailed recognition,' in International Conference on Learning Representations , 2019. 2, 3, 4, 6

J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin, 'Seesaw loss for long-tailed instance segmentation,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 9695-9704, 2021. 3, 4, 5

J. Tan, X. Lu, G. Zhang, C. Yin, and Q. Li, 'Equalization loss v2: A new gradient balance approach for long-tailed object detection,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 1685-1694, 2021. 3, 4, 5, 6

Q. Li, B. He, and D. Song, 'Model-contrastive federated learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10713-10722, 2021. 3

S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Koneˇ cn` y, S. Kumar, and H. B. McMahan, 'Adaptive federated optimization,' arXiv preprint arXiv:2003.00295 , 2020. 3

R. Balakrishnan, T. Li, T. Zhou, N. Himayat, V. Smith, and J. Bilmes, 'Diverse client selection for federated learning: Submodularity and convergence analysis,' in ICML 2021 International Workshop on Federated Learning for User Privacy and Data Confidentiality , 2021. 3

Z. Chen, K. F. E. Chong, and T. Q. Quek, 'Dynamic attention-based communication-efficient federated learning,' in International Workshop on Federated and Transfer Learning for Data Sparsity and Confidentiality in Conjunction with IJCAI (FTL-IJCAI'2021) , 2021. 3

T. Yoon, S. Shin, S. J. Hwang, and E. Yang, 'Fedmix: Approximation of mixup under mean augmented federated learning,' in International Conference on Learning Representations , 2020. 3

A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li, 'Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets,' arXiv preprint arXiv:2008.03371 , 2020. 3

C. T Dinh, N. Tran, and J. Nguyen, 'Personalized federated learning with moreau envelopes,' Advances in Neural Information Processing Systems , vol. 33, pp. 21394-21405, 2020. 3

Y. Lu, P. Qian, G. Huang, and H. Wang, 'Personalized federated learning on long-tailed data via adversarial feature augmentation,' arXiv preprint arXiv:2303.15168 , 2023. 3

M. Duan, D. Liu, X. Chen, R. Liu, Y. Tan, and L. Liang, 'Self-balancing federated learning with global imbalanced data in mobile systems,' IEEE Transactions on Parallel and Distributed Systems , vol. 32, no. 1, pp. 59-71, 2020. 3

H. Tang, C. Yuan, Z. Li, and J. Tang, 'Learning attention-guided pyramidal features for few-shot fine-grained recognition,' Pattern Recognit. , vol. 130, p. 108792, 2022. 3

T. Wang, Y. Li, B. Kang, J. Li, J. Liew, S. Tang, S. Hoi, and J. Feng, 'The devil is in classification: A simple framework for long-tail instance segmentation,' in European conference on computer vision , pp. 728-744, Springer, 2020. 3

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, 'Focal loss for dense object detection,' in Proceedings of the IEEE international conference on computer vision , pp. 2980-2988, 2017. 3, 6

S. Li, K. Gong, C. H. Liu, Y. Wang, F. Qiao, and X. Cheng, 'Metasaug: Meta semantic augmentation for long-tailed visual recognition,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 5212-5221, 2021.

J. Wang, T. Lukasiewicz, X. Hu, J. Cai, and Z. Xu, 'Rsg: A simple but effective module for learning imbalanced datasets,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3784-3793, 2021. 3

M. Li, Y.-m. Cheung, and Y. Lu, 'Long-tailed visual recognition via gaussian clouded logit adjustment,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6929-6938, 2022. 3, 6

Y. Zang, C. Huang, and C. C. Loy, 'Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 3457-3466, 2021. 3

B. Liu, H. Li, H. Kang, G. Hua, and N. Vasconcelos, 'Gistnet: a geometric structure transfer network for long-tailed recognition,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 8209-8218, 2021.

J. Cai, Y. Wang, and J.-N. Hwang, 'Ace: Ally complementary experts for solving long-tailed recognition in one-shot,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 112-121, 2021.

H. Guo and S. Wang, 'Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 15089-15098, 2021. 3

B. Li, Y. Yao, J. Tan, G. Zhang, F. Yu, J. Lu, and Y. Luo, 'Equalized focal loss for dense long-tailed object detection,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6990-6999, 2022. 3

T. Wu, Q. Huang, Z. Liu, Y. Wang, and D. Lin, 'Distribution-balanced loss for multi-label classification in long-tailed datasets,' in European Conference on Computer Vision , pp. 162-178, Springer, 2020.

K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma, 'Learning imbalanced datasets with labeldistribution-aware margin loss,' Advances in neural information processing systems , vol. 32, 2019. 3, 6

T. Wang, Y. Zhu, Y. Chen, C. Zhao, B. Yu, J. Wang, and M. Tang, 'C2am loss: Chasing a better decision boundary for long-tail object detection,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 6980-6989, 2022. 3

Y. Zhang, B. Hooi, L. Hong, and J. Feng, 'Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision,' arXiv preprint arXiv:2107.09249 , 2021. 4

J. Liu, Y . Sun, C. Han, Z. Dou, and W. Li, 'Deep representation learning on long-tailed data: A learnable embedding augmentation perspective,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 2970-2979, 2020. 4

J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, 'Equalization loss for long-tailed object recognition,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 11662-11671, 2020. 4

Y. Li, T. Wang, B. Kang, S. Tang, C. Wang, J. Li, and J. Feng, 'Overcoming classifier imbalance for long-tail object detection with balanced group softmax,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 10991-11000, 2020. 4

S. Bennett, 'Development of the pid controller,' IEEE Control Systems Magazine , vol. 13, no. 6, pp. 58-62, 1993. 5

Y. Huang, S. Gupta, Z. Song, K. Li, and S. Arora, 'Evaluating gradient inversion attacks and defenses in federated learning,' Advances in Neural Information Processing Systems , vol. 34, pp. 7232-7241, 2021. 6

A. Krizhevsky, G. Hinton, et al. , 'Learning multiple layers of features from tiny images,' 2009. 6

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in 2009 IEEE conference on computer vision and pattern recognition , pp. 248-255, Ieee, 2009. 6

G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, 'The inaturalist species classification and detection dataset,' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 8769-8778, 2018. 6

D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' in ICLR (Poster) , 2015. 9

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit a long-tailed distribution. The authors propose the Fed-GraB to coordinate the global long-tailed distribution and the local learning strategy. Specifically, it consists of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients’ gradients in a closed-loop manner based on the feedback of global long-tailed prior derived from a Direct Prior Analyzer (DPA) module. Experiments are conducted on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist.\n\nSTRENGTHS:\n1. Investigating the federated long-tailed learning problem from the view of combining global and local sounds interesting.\n2. The design of the proposed method is in line with the motivation of the paper, which seems technically feasible.\n3. Experiments are conducted on many long-tailed benchmarks with various settings, which demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\nAlthough the technical contribution of the manuscript sounds qualified, the presentation of the manuscript is not clear enough. There are some confusing issues that I will explain in the questions section.  Please resubmit a revised version. I will update my final score according to your revision. Meanwhile, too many hyperparameter settings will affect the generalizability of the proposed method.\n\nQUESTIONS:\n1. What's the correlation between uj in Eq.3  and beta_j in Eq.4?  And what's the correlation between beta_j and (beta_j^pos, beta_j^neg)? This is essential for understanding the re-weighting process of SGB.\n2. In line 196, the author gives the definition of φ, but I could not find where the φ be applied.\n3. In Eq.4, a random number r is adopted as a threshold for Pcj, what's the role of it in the SGB?\n4. What's the definition of the expected target zj(t)? And why define the error feedback ej(t) as (gpos - gneg -  zj(t))?\n5. The EQLv2 strives to keep the cumulative positive and negative gradients to be equal for each category which seems to share similar ideas with the proposed method. Compared with EQLv2-FL, what are the novelty and advantages of this approach?\n\nLIMITATIONS:\nThe authors have declared the limitations in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper presents an approach, Fed-GraB, for addressing the challenges of Federated Long-Tailed Learning, an issue characterized by data heterogeneity and privacy concerns. The authors tested their method on several benchmark datasets, which significantly outperforms state-of-the-art baselines. The paper is mostly an experimental work.

STRENGTHS:
The proposed Fed-GraB model is based on an interesting technique called Self-adjusting Gradient Balancer to rebalance gradients. The experiments are comprehensive.

WEAKNESSES:
The authors propose to tackle the challenges posed by data heterogeneity and privacy concerns in the FL setting. However, the paper does not sufficiently justify the uniqueness of their approach. A clearer and more compelling argument is needed on how the proposed Fed-GraB model uniquely and effectively addresses these challenges.

The proposed model needs to be corroborated with theoretical analyses or at least insights, especially regarding the function and operation of the DPA modules.

The paper often assumes that global class priors are available for re-balancing, which may not always be the case in real-world applications due to privacy constraints. More importantly, it's unclear how the proposed method would handle issues where local distributions are not long-tailed or present diverse long-tailed characteristics.

A more comprehensive description of the interplay between the SGB and DPA modules, and the technical novelty there, would be helpful.

QUESTIONS:
Please see the above comments

LIMITATIONS:
more technical analysis or discussions are needed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

comments: SUMMARY:
In this paper, a self-adjusting and closed-loop gradient re-balancing framework Fed-GraB was proposed and long-tailed learning tasks processing performance was improved.

STRENGTHS:
This paper presents a methodology named Fed-GraB, which incorporates a Self-adjusting Gradient Balancer (SGB) module. This module dynamically adjusts the weightage of gradients from individual clients in a closed-loop manner, guided by the feedback obtained from a Direct Prior Analyzer (DPA) module. By employing Fed-GraB, clients can effectively mitigate the challenges posed by disparate data distributions during the model training process. They achieve a global model that exhibits enhanced performance on underrepresented classes while maintaining the performance levels of the majority classes.

WEAKNESSES:
1. Equation 4 lacks ending punctuation.
2. Figure 4(b) adds markers to make the curve more legible.
3. The definition of IFG is not clear. Please give a formula that says how to define IFG.
4. How to divide the dataset to get the Cifar-10/100-LT, there is no specific explanation.
5. The insight of the algorithm should be explained more clearly.
6. The baselines selected in this paper are not fully appropriate, because only one of them is designed for federated learning. The solutions to unbalanced data in FL are necessary to be compared.

QUESTIONS:
Please see Weaknesses

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the globally long-tailed distribution of data and its impact on federated learning (FL). To address this issue, the authors propose the Fed-GraB algorithm that re-balances gradients. The problem is timely and the proposed algorithm is novel. Overall, the paper is well-written, and simulations are extensive. However, the limitations of the proposed framework are not rigorously addressed via for instance mathematical or quantitative analysis. Furthermore, the applicability of the proposed framework to various applications and other state-of-the-art FL algorithms are questionable.\n\nSTRENGTHS:\nThe potential impact of the proposed framework would be significant, particularly on two emerging areas of research: fairness guarantee and rare-event detection. The proposed idea of re-balancing gradients is not entirely original, yet still sufficiently novel as it has yet been considered in FL for this purpose. Notably, the proposed idea does not incur significant costs in terms of computation and memory, which is a big plus for practical implementation. The literature of the considered problem and relevant frameworks is well reviewed, making the paper easily readable.\n\nWEAKNESSES:\nSome potential weakness of the proposed framework has been proactively described, yet the claims are not very convincing. In particular, in the Privacy Discussions, it claims that \"potential privacy issue exists in the general FL frameworks rather than specific to our proposed DPA method...As the privacy issue of FL framework is beyond the scope of this we briefly include the discussion in this subsection.\" This justification hardly responds to the issues when applying the proposed framework to other privacy-preserving FL frameworks, for instance, that applies quantization and noise injection under which the proposed framework's performance may be degraded. Furthermore, in Computational and Storage Cost of SGB, it claims that \"the extra computation which is implemented with several lines of code could be done very quickly. Besides, Fed-GraB needs some extra storage cost to store the weighted gradients which is quite cheap as well.\" This is not very convincing as it does not compare, for example, the resultant convergence speed and guarantee as well as FLOPS and bytes with the cases without the proposed solution.\n\nQUESTIONS:\n1. According to (1), the proposed algorithm relies on classification and cross entropy loss, as opposed to the original FL that has no restriction on its task and loss function. Is the proposed algorithm applicable to 1) non-classification tasks as well as 2) non-cross entropy loss or cross-entropy with other regularizers?\n\n2. There are various advanced FL algorithms that intentionally distort gradients via quantization, sampling, and noise injection for compression, privacy protection, and so forth. Is the proposed algorithm still applicable to these algorithms?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}