Tackling Imbalanced Class in Federated Learning via Class Distribution Estimation
Authors: You-Ru Lu
Xiaoqian Wang
Dengfeng Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
class imbalance
class distribution estimation

Abstract: Federated Learning (FL) has become an upsurging machine learning method due to its applicability in large-scale distributed system and its privacy-preserving property. However, in real-world applications, the presence of class imbalance issue, especially the mismatch between local and global class distribution, greatly degrades the performance of FL. Moreover, due to the privacy constrain, the class distribution information of clients can not be accessed directly. To tackle class imbalance issue under FL setting, a novel algorithm, FedRE, is proposed in this paper. We propose a new class distribution estimation method for the FedRE algorithm, which requires no extra client data information and thus has no privacy concern. Both experimental results and theoretical analysis are provided to support the validity of our distribution estimation method. The proposed algorithm is verified with several experiment, including different datasets with the presence of class imbalance and local-global distribution mismatch. The experimental results show that FedRE is effective and it outperforms other related methods in terms of both overall and minority class classification accuracy.

Under review as a conference paper at ICLR 2023
TACKLING IMBALANCED CLASS IN FEDERATED LEARNING VIA CLASS DISTRIBUTION ESTIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) has become upsurging machine learning method due its applicability in large-scale distributed system and its arivacy-preserving property. However, in real-world applications, the presence of class imbalance issue, especially the mismatch between local and global class distribution, greatly degrades the performance of FL. Moreover, due the privacy constrain, the class distribution information of clients can not be accessed directly. To tackle class imbalance issue under FL setting, novel algorithm, FedRE, is proposed in this paper. We propose a new class distribution estimation method for the FedRE al- gorithm, which requires extra client data information and thus has privacy concern. Both experimental results and theoretical analysis are provided to sup- port the validity of our distribution estimation method. The proposed algorithm is verified with several experiment, including different datasets with the presence of class imbalance and local-global distribution mismatch. The experimental results show that FedRE is effective and it outperforms other related methods in terms of both overall and minority class classification accuracy.
INTRODUCTION
Federated Learning (FL) was first proposed (McMahan et al. 2017) when they were developing the application of next-word prediction on mobile keyboard. It enables multiple clients to collab- oratively learn a machine learning model, without sharing their locally stored raw data (Li et al.] 2020a). This property greatly reduce the communication cost and preserve the privacy of clients, which makes FL. become an upsurging research direction, not only machine learning commu- nity but also in variety of engineering applications, including communication (Wang et al. 2022
Niknam et al. 2020 Mills et al. 2019), edge computing (Zhang et al. 202 Wang et al. 2019a
and energy engineering (Saputra et al. 2019 Hamdi et al. 2 et al.) 2022).
Standard FL consists of four major steps, which are client selection, broadcast, client computation, and aggregation (Kairouz et al. (021). In each iteration, the central server will select subset of clients in each global iteration at first, and then broadcast the global model to them. After receiving the global model, the selected clients will perform model update based on their local dataset given the global model as initial condition, and then upload updates to the server. As the final step, the server will aggregate the collected information update the global model, and then start new iteration.
In FL framework, one of the most difficult challenges is class imbalance issue. Class imbalance means that the data distribution among all classes are not uniform. In other words, the majority of data samples may belong to certain classes, and other minority classes may only have small amount of data. Class imbalance results in low classification accuracy on minority classes, and also slow down the training speed. In the literature, several methods have been proposed to re- solve class imbalance issue in centralized machine learning scheme. In general, the methods can be categorized as data-level methods, algorithm level methods, and hybrid methods (Johnson & Khosh- goftaar 2019). For data-level method, Jo & Japkowicz (2004) proposed a cluster-based sampling scheme to tackle the class imbalance issue. For algorithm level method, [ing & Sheng (2008) pro- posed the cost-sensitive learning to improve the classification performance
minority class. For hybrid method, [Sun et al. (2007) integrated both sampling techniques and cost-sensitive learning and showed significant performance boost in most of the cases. However, in FL., since all training data

---

Under review as a conference paper at ICLR 2023
distributed and stored locally and not exchangeable, it infeasible to apply data-level methods. Besides, due to the inconsistency of local and global class imbalance, algorithm-lewe methods such cost-sensitive learning
not effective and may impose negative effect on the performance of global model (Wang et al. (02T). Thus, those methods that have shown great achievements on class imbalance issue in centralized machine learning cannot be applied directly in Fl., and new algorithm has to be considered under the constrains of FL scheme.
In this work, the class imbalance issue in FL addressed, especially the issues of global class imbal- ance and mismatch between local and global class distribution. To tackle the challenges mentioned previously while preserving the privacy condition, a new FL algorithm is proposed. Due to the pri vacy condition, the local dataset and local class distribution of each clients are not accessible. Thus, an estimation method to estimate the class distribution developed, requiring no extra client dataset information. Based on the estimated class distribution, the loss re-weight method can be applied to handle the global class imbalance. The proposed method requires no additional client informa- tion, the privacy safety can be guaranteed. Besides, from the experimental results, the proposed method has shown a significant improvement on the classification accuracy of minority class under the class imbalance scenario.
Contribution. In summary, the contributions of this work is as follows.
1. The proposed estimation method can estimate the class distribution without additional client information, so the privacy safety can still be guaranteed. Moreover, the estimation method doesn't require much extra computation, so the overall efficiency of FL. algorithm is not degraded. The theoretical analysis is also provided to support the validity of the proposed distribution estimation method.
2. The proposed new FL algorithm based on class distribution estimation achieves significant improvement on handling the class imbalance issue in FL., and the proposed algorithm is verified by experiments with different heterogeneity level and different dataset.
RELATED WORKS
2.1
FEDERATED LEARNING
Due to the development of edge devices and the increasing popularity of mobile devices, the dis- tributed machine learning has become one of the most popular direction of machine learning re- search. However, common mobile devices such smart phone and wearable electronics have lim- ited computation and communication power. Moreover, those mobile devices contains personal in- formation which is private and not exchangeable. Thus, new distributed machine learning problem, also known as Federated Learning (FL), has to be considered to overcome the challenges of commu- nication and computation cost, data heterogeneity, and privacy. To overcome the novel challenges, FedAvg (McMahan et al., 2017) was first proposed and addressing the communication efficiency problem. Since then, many researches have been conducted resolve challenges in FL. [i et al.) (2020b) proposed the algorithm FedProx to tackle the data heterogeneity issue by introducing proximal term in local objective function, and provided theoretical analysis on convergence. To improve the convergence speed and reduce the communication cost, in Reddi et al. (2020) different adaptive learning techniques were applied in the aggregation steps. Waan et al. (2020b) investigated the fundamental cause of heterogeneity, and proposed normalized averaging algorithm FedNova to improve the FL performance. et al. (2021) applied local batch normalization to tackle the feature- shift non-iid issue, which is another type of data heterogeneity. Some recent works in literature also provided improvement on FL algorithm by proposing different client selection scheme (Nishio & Yonetani 2019 Ribero Vikalo 2020 Balakrishnan et al. 202T).
2.2 CLASS IMBALANCE IN FEDERATED LEARNING
One of the most difficult yet important challenges in FL is class imbalance. The class imbalance issue can be categorized as local imbalance and global imbalance (Wang et al. 2021). Works men- tioned previously were addressing ever, as stated in section[] due to the mismatch between local and global distribution, only handling local data imbalance is not enough. To deal with class imbalance issue, the algorithm Astraea
data heterogeneity, which belongs local imbalance. How-
2

---

Under review as a conference paper at ICLR 2023
from Duan al. (2019) introduced mediators between local clients and central server to facili- tate scheduling and aggregation. The proposed method requires real client data distribution, which violates the privacy condition. Furthermore, if the distribution information are leaked during the transmission, malicious attackers who intercept such information could analyze client preference and launch targeted attacks (Wang et al., 202T). To preserve client privacy, some previous meth- ods in literature build proxies to estimate the class distribution. Yaagg et al. (202I) suggested that the gradient magnitude can be proxy estimate the class distributions of clients, and utilized it to design new client selection method. [Wang et al.] (2021) suggested that there exists proportional relation between gradient magnitudes and sample quantities. Based on this argument, Waang et al. (202T) developed a monitoring scheme that can estimate the class distribution, and introduced the algorithm, Loss, to perform class-wise re-weight. However, the suggested proportional rela- tion between gradient magnitude and sample quantity is based on strong assumptions, which may not be feasible in practice. As another branch of research direction, there are works utilize active learning and reinforcement learning to eliminate the effects of class imbalance (Goetz et al., [019] Zhang et al. 2021b Wang et al.) 2020a). However, these methods usually rely on different client selection schemes. In practical setting of FL, available clients in each communication round are different and uncontrollable by server. Therefore, methods applying client selection have limited applicability (Shen et al.| 2021). Compared with others works, we propose a new FL algorithm and a new class distribution estimation method that can facilitate the proposed algorithm without viola- tion of privacy condition. Besides, the proposed algorithm does not rely on active client selection schemes. Thus, it has less limitation on applicability in practice.
3 METHODOLOGY
3.1 DEFINITION AND BACKGROUND
In this paper, the multi-class classification problem in FL. is considered. Let a Rd be the d dimensional input and t E R& be the target one-hot vector of Q classes. Under the FL framework, each client & has a local objective (w(w)) = + -/= 4 (w, Zt tz), where ny is the number of samples of client k, w is the model parameters, and is the loss function. In FL, the goal is to minimize the global average of loss functions, which be stated as follows:
min F(w)
Σ pkFF((w)
where K the total number of clients, Ph: nh and n = Σ ni is the total number of samples. The challenge of class imbalance in FI. is that for each client ki their local class distribution Dd are different. Besides, there may be huge mismatch between local and global class distribution, that is, D₂ and global class distribution D, can be very different. Moreover, due to the privacy assumption in FL, local data and data distribution can not be accessed directly. To tackle these challenges, a new FL algorithm based on class distribution estimation is developed, and will be introduced in detail in the following sections
3.2 DISTRIBUTION ESTIMATION
As discussed in previous section, the biggest challenge that the distribution information is un- known. If those information can be estimated without violation of privacy, one can utilize those information to handle the class imbalance problem. Therefore, the problem lies in how to estimate the class distribution under the framework of FL. The class distribution estimation method developed in this work is based on the following observation:
When updating the model parameters by Gradient Descent (GD) Stochastic Gradient Descent (SGD), in the parameter space w will converge to a flat region in the early stage of training, such that the prediction output of all input from the model is approximately equals to the class distribution ratios.
Observation:
The observation can be demonstrated in the following experiment synthetic data. Consider binary classification problem, where the sample points from two classes C₁ and Co are distributed on the 2-1) plane as shown in [ Without loss of generality, assume that the minority class Co
3

---

Under review as a conference paper at ICLR 2023
3.0
25
25
100
100
(a) Imbalance training samples
(b) Balance testing samples
Figure/ 1: 2-D synthetic data samples for the binary classification problem. Blue dots represent
samples from majority class C₁ and orange dots represent samples from minority class Co- (a) Imbalance training samples with nig 10 and nt 90. (b) Balance testing samples with ni 12
50. Co is generated by uniformly random sampling such that æ = {[x1, x2) x1, x2 E (1,2)), and C1 is generated by uniformly random sampling such that {[<1, x2] x1, (0.3) (1,2)).
iteration
Figure 2: Training loss with respect to training iteration for the binary classification problem imbalance synthetic data. The plateau region can be observe at the early stage of training (before 2000 iteration). In this region, the output values approximately equals to 0.9
(labeled as target 0) and majority class C1 (labeled as target 1) have samples quantities co and e respectively, and the imbalance level C1 c1/co 9. To solve this classification problem, simple one-layer fully connected neural network with 4 hidden Sigmoid neurons is used and the square loss function is applied. The prediction output is the probability of input belonging to C1. The plot of training loss with respect to iterations on the imbalanced dataset shown Fig One can observe that the training loss drops quickly the beginning and then reaches a plateau. If look the prediction output at this region as shown in Table I one can observe that the prediction output from both C1 and Co are approximately equals to 0.9 nı ni/N, which the portion of C1 among total number of samples N. This phenomenon can also be observed using Binary Cross Entropy (BCE) loss. or by different imbalance level p. Moreover, the phenomenon can be observed in the multi- class classification problem as shown in Fig(] The following Theorem explains the cause of this phenomenon from theoretical point of view.
2
3
4
5
6
8
9
10
0.9066
0.8865
0.9075
0.8999
0.8865
0.8981
0.8926
0.8921
0.8953
0.8909
Table 1: Ten outputs of samples from synthetic testing dataset in the binary classification problem. Outputs to 5 are from samples in majority class C₁ and outputs to 10 are from samples in minority class Co
4

---

Under review as a conference paper at ICLR 2023
300
150
-2-
1000
1500
2000
4000
(a) Imbalance training samples
(b) Training loss to iteration
Figure 3: Experiment on synthetic data for the multi-class classification problem. (a) Imbalance training samples from classes with number of samples [M1, f2,, 7,,, nz] [100, 50, 250] and class distribution D [0.25, 0.125, 0.625], respectively. (b) The plot of training loss with respect to iteration. At the plateau region, the average output from testing set is (0.2545, 0.1316, 0.6199) ~ D with error percentage less than 6%.
cl
2.00
c2 error
175
20
error
150
15
125
100
-0.5
0.0
2.0
255
Number hidden layers
(a) Imbalance training samples of 4 classes
(b) Class distribution estimation error percentage with respect number of hidden layers
Figure 4: Experiment on synthetic data for the multi-class classification problem with nonequal class sample averages. (a) Imbalance training samples from classes with number of samples [R1. 122, N3. n24 = [50, 100, 150, 200] and class distribution D [0.1, 0.2, 0.3, 0.4]. respectively. (b) the distribution estimation error percentages of 4 classes with respect number hidden layers. One can observer that the estimation error decays with layers number increased and errors of c2 c4 reach () at m 3.
Theorem For m-layer fully connected neural network with d-dimensional input and Q- dimensional output, assume the Sigmoid activation functions and square loss are used, and the training data distribution D = [ce//n, cq] In the parameter space, if at some point such that all the model prediction outputs y D, that point a critical point by first order optimality.
The proof of Theorem is provided in Appendix, and the proof can also be directly extended to BCE loss case. Theorem explains why such point in the parameter space is preferred during the model updating process, which also provides an explanation of the cause of such phenomenon in theoretical point of view. Note that even though in the proof it utilizes the assumption that the average inputs from all classes are the same, in practice the error caused by the violation this assumption can be eliminated by increasing the number of layers m. which is demonstrated in the experiment on synthetic data shown Fig This result supports the validity of the estimation in practice.
Based on this observation, the class distribution estimation method is designed and summarized in Algorithm [
At line 2 the modelU pdate function is commom machine learning process, given

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Class Distribution Estimation
Input: minibatch size B, number of epochs E, learning rate imbalance dataset D, balance aux- iliary dataset A, loss function 1, optimizer opt SGD
Output: estimated class distribution D
I: initialize w
2: w t- modelU modellipdate(ww) B, E, 17, D,,, opt)
3: D A Eial (w.B.)))))
loss function I and the optimizer. In default, the SGD is used because it is more feasible than GD in practice. The pypee-parameters B, E and 1j for SGD are chosen such that the model parameters can reach the plateau region. Note that the range of plateau region is in general large enough as shown previously in experiments, so in practice the hyper-parameters can be chosen without care- fully fine-tuning. Some other diagnostic approach (Chee & Toulis 2018 Bottou al., 2018) can also be adopted to setup the stopping criteria of modellppatte which is beyond the scope of the discussion here. In line of Algorithm [ after obtaining the updated model, balance auxiliary dataset A is used to estimate the class distribution. As discussed previously, the prediction outputs from any inputs are approximately equal the class distribution during the early stage of training. By this property we calculate the average output values from A as the estimated class distribution D. f(w, æ) here denotes the prediction output of the neural network given model parameters w and input æ. and operator denotes the size of the set. In practice, A can be just small por- tion of data from public dataset and stored in the central server, so it step will not incur much extra computational cost.
3.3 OUR METHOD
The flow chart of the proposed new FL method addressing on class imbalance is presented in Fig and the detail of the algorithm summarized in Algorithm [ In order to incorporate the class distribution estimation stated in Algorithm into FL. framework, we have to train the model locally, and utilize the locally updated model to estimate class distribution in the server. In the first global iteration, server distributes initial model wD all clients, and each client trains the model on their local dataset Da and obtains updated model Wk. similar to line 2 in Algorithm [ Afterward, updated local models are sent back to the server, and the server can use an auxiliary dataset A to estimate D for all clients k. Note that the hyper-parameters in modelU pdate line can be tuned separately from those at line 15. In this work, the square loss function 1 = (y - t) is used, whose validity is verified experimentally and theoretically in previous section.
With the estimated class distributions, the global class imbalance issue can be tackled my applying loss re-weight method, where the loss weight vector R = [R1, .... R2] E IRQ calculated at line 9. The function get.oss Reweight( is defined in Algorithm 2 The idea is that the global
class distribution Dy be estimated based the estimated client class distribution DR, client sample amount ru, and total sample amount = =>m ng. The loss weight for each class is assigned inverse proportionally to the corresponding global class ratio, so the minority class will receive higher weight. In the function Loss Rew.osshseWwehhh(). tuned. In this work,
and / are and 8 0.01 are used achieve the best overall performance.
that can be
The second half of Algorithm is the standard FL process. In each global iteration, CK clients are randomly selected, where C E (0.1] the portion of client selection. The selected clients will perform local update, and the sever will then aggregate updated local models to obtain new global model. Note that the loss function used in local model update weighted loss function depending on the weight vector R. In this work, the weighted Cross Entropy Loss is used as stated in the following:
Q
exp (ya) 2 exp (y:)
L Lyy, (y - - R. log
tc:
where y [y1,-.., [Q] is the prediction probability vector and t is the target one-hot vector.

---

Under review as a conference paper at ICLR 2023
Estimate class distribution
Get loss reweight
Server
All clients
All clients
yes
Dx
no
Local model update
Aggregate local model (g
CK clients
Server
e
Figure 5: Flow chart of proposed method in each iteration. Note that in = 1 only the class distribution estimation is performed. The global model is not updated in first iteration.
Algorithm 2 FedRE: Fedrated learning with Ratio Estimation
Input: maximum global iteration T', local minibatch size B, number of local epochs E, learning rate 7), clients index set K, weighted loss function balance auxiliary dataset A
Output: model w
initialize w
2: for t = 2.. do
3: i then
4:
for all k E K do in parallel w t modelU pdate(w" B, E, 17, 1x,1)
5:
line Algorithm
6:
upload w to server
7:
end for estimate D2 for all k by D₃ 6 1 E l(w xi)) R Loss getLossReweighte ID.,... Dx] [m1...., mx))
8:
[A]
line Algorithm
9:
10:
Send R to all clients continue
11:
12:
else
13:
randomly sample CK clients and create selection set S₁
for ls St do parallel. 1+1 +- modelU pdatelw'' B, B, Dx, L, R) end for w-+1 +
15:
> Clients local update by SGD
17:
15 15 k= #t
b> Aggregation step
end
19: end for
Function Rewessll Dxc, (RI,..,,N)))
1: D - Σk= nu D
2: R - a D
3: return R
EXPERIMENTAL RESULTS
The effectiveness of the proposed method are verified by several experiments, and the performance are compared with other FL algorithms. The experimental results highlight the advantages of the proposed method the presence of severe heterogeneity on class distribution and mismatch between local and global imbalance. Three other algorithms. FedAvg (McMahan et al., (017). FedNova (Wang et al., 2020b), and Ratio-Loss (Wang et al. 2021) are used in the comparison. FedAvg is the most widely-used base-line algorithm in FL. FedNova is one of the most common variation of FedAvg, which applies normalized averaging method to eliminate the effect of data heterogeneity. Ratio-Loss utilizes monitoring scheme to estimate the class distribution and detect the presence of global class imbalance, and then applies a novel loss re- re-weight method to mitigate the effect of class imbalance.

---

Under review as a conference paper at ICLR 2023
4.1 EXPERIMENT SETUP
All the algorithms are implemented in PyTorch, and methods from other works are reimplemented in the experiments. Two different datasets: MNIST and CIFARIO are chosen, which are very common benchmark dataset and frequently used for verification of FL algorithms. Different models are used for the tests on different datasets. To have a fair comparison, for each dataset the same model will be used for comparing different FL algorithms. The model used in MNIST dataset consists of one convolutional layer having 128 3 x 3 filters with ReLU activation followed by 2 fully connected layers having 1000 and 100 neurons with Sigmoid activation. The model used in CIFARIO 10 dataset consists of 2 convolutional layers having 128 5 x filters with ReLU activation followed by 2 fully connected layers having 394 and 192 neurons with Sigmoid activation. For all clients, in estimation step, the Square Error Loss is used and the learning rate ij = 0.01. In local update, the weighted Cross Entropy Loss stated in () is used and the learning rate n 0.05. The minibatch size B = 32, the local training epoch E = 5, and the local optimizer is SGD in both estimation and local update.
To simulate the presence of globally imbalance class distribution, we simulate two kinds of scenarios that there are and 2 minority classes within all classes, respectively. Each minority class has in total 8 samples, and for the reset of the classes, each of them has ps samples, where is the level of imbalance. In the experiment, to test the validity of the algorithm, two levels of imbalance = 5 and 10 are chosen, and s 300 and s = 150, respectively. In other words, there are 1500 samples belongs to each non-minority classes for both two imbalance level. These globally imbalanced samples are randomly distributed into each client by Dirichlet distribution to simulate the beterogeneity of local client data distribution, which also simulate the presence of mismatch between local and global class distribution.
For the Federated Learning environmental setting, two kinds of selection scenarios are built, which are the picking 5 clients out of 5 (C 1) and picking 5 clients out of 16 (C = 0.3). respectively. For experiments on MNIST and CIFAR10. the global iteration numbers are 100 and 150, respectively. In the server, there is a balanced testing dataset consists of 32 samples for each class. The balanced auxiliary dataset for distribution estimation also consists of 32 samples for each class, in total 320 samples. Note that compared to the size of training set, the auxiliary dataset is much smaller, which incurs not much computational cost.
4.2 RESULTS SUMMARY
The experimental results of classification accuracy for 4 different methods with I and 2 minority classes are summarized and compared in Table 2 and Table 3 The numbers shown in the table without parentheses are overall classification accuracy, and the numbers with parentheses are clas- sification accuracy of minority class. Note that for the case with minority classes, the worst classification accuracy on 2 minority classes is reported in order to provide fair comparison. As one can see the tables, in both sets of experiments with different number of minority classes and different level of heterogeneity, the proposed method FedRE. outperforms all other methods in terms of both overall and minority class classification accuracy. All other methods fail in correctly classi- fying the minority class, even if they still have acceptable overall accuracy. The overall performance of FedNova is very similar to the baseline method FedAAg in the experiments. This is because even thought FedNova can handle client heterogeneity, it is focusing on the data heterogeneity, that is, the difference in quantities of samples for each clients. Thus, it has minor effect on resolving the heterogeneity of imbalance class distribution. Ratio-Loss is shown have a better performance than FedAvg and FedNova in general. However, it still can't tackle the issue with highly imbal- ance distribution in the experiments. On the other hand, the proposed method successfully achieves the bast performance in overall accuracy and classification accuracy on minority class in different level of heterogeneity. In the experiments, especially in tests on MNIST dataset, one can observe that the overall accuracy is boosted significantly. One can also observe that the overall accuracy is significantly improved in the experiments with minority classes. These improvements are mainly because of the success in tackling imbalance class distribution issue. Other methods which fail in tackling the issue are affected by majority classes and end up with not learning the minority class. According to these experimental results, can conclude that the proposed algorithm works ef- fectively and provide a significant improvement on minority class classification accuracy without compromising the overall accuracy.
8

---

Under review as a conference paper at ICLR 2023
P
C
FedAvg
Nvva 88.13%
Ratio-Loss 88.75%
FedRE
88.13%
98.44%
1
(0.0%)
(0.0%)
(0.0%)
(93.75% %)
MNIST
80.31%
80.94%
88. 13%
96.88%
0.3
(0.0%)
(0.0%)
(0.0%)
(93.75 %)
88.44%
88.44%
89.38%
98.13%
10
(0.0%)
(0.0%)
(0.0%)
(93.75 %
81.25%
80.36%
88.44%
97.5%
0.3
(0.0%)
(0.0%)
(0.0%)
(93.75 %
51.25%
51.88%
54.69%
55.00
(0.0%)
(0.0%)
(0.0%)
(40.63 %)
CIFARIO
37.81%
37.5%
40.63%
48.44%
0.3
(0.0%)
(0.0%)
(0.0%)
(62.5%)
50.31%
50.94%
53.13%
54.06%
10
(0.0%)
(0.0%)
(0.0%)
(37.5%)
33.13%
34.06%
42.81%
44.69 %
0.3
(0.0%)
(0.0%)
(0.0%)
(40.63%)
Table Performance comparison between different algorithms with minority class. Results with respect to different datasets, level of imbalance p. and client selection ratio C' are reported. The numbers without parentheses are overall classification accuracy and numbers with parentheses are minority class classification accuracy.
p
FedAvg
FedNova
Ratio-Loss
FedRE
79.06%
79.38%
79.38%
98.13% %
1
5
(0.0%)
(0.0%)
(0.0%)
(93.75% %
MNIST
71.88%
72.19%
78.44%
96.56 %
0.3
(0.0%)
(0.0%)
(0.0%)
(90.63 %)
78.44%
78.44%
79.69%
97.81% %
10
(0.0%)
(0.0%)
(0.0%)
(87.5%)
70.94%
70.94
76.88%
96.25%
0.3
(0.0%)
(0.0%)
(0.0%)
(87.5%)
49.06%
48.75%
50.94%
53.44%
(0.0%)
(0.0%)
(0.0%)
(9.38%)
CIFARIO
26.25%
30.31%
39.38%
45.63%
0.3
(0.0%
(0.0%)
(0.0%)
(34.38%)
50.63%
50.63%
52.5%
55.13%
10
(0.0%)
(0.0%)
(0.0%)
(3.13%)
25.62%
30.0%
39.06%
45.31%
0.3
(0.0%)
(0.0%)
(0.0%)
(6.25%)
Table 3: Performance comparison between different algorithms with 2 minority classes. Results with respect to different datasets, level of imbalance p. and client selection ratio C' are reported. The numbers without parentheses are overall classification accuracy and numbers with parentheses are the worst minority class classification accuracy.
CONCLUSION
In this paper, a new algorithm based on a novel class distribution estimation method is proposed to tackle the class imbalance issue in Federated Learning with the presence of global class imbalance and distribution mismatch between local and global dataset. class distribution estimation method is developed based on the experimental observation, in which the model parameters converge to a local minimum in the early stage of training where the prediction outputs are equal to the class distribution ratios. The class distribution estimation method is verified with both experiments and theoretical analysis. With the developed estimation method, the proposed algorithm utilizes the estimated information perform loss re-weight to tackle the class imbalance issue. The validity and the effectiveness the proposed algorithm are verified in extensive experiments with a variety of practical settings in FL., which also shows the superiority of the proposed method over other arts.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. Diverse client selection for federated learning via submodular maximization. In International Conference Learning Representations, 2021.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-31) 1113--11,,21 2018.
Jerry Chee and Panos Toulis. Convergence diagnostics for stochastic gradient descent with constant learning rate. In International Conference Artificial Intelligence and Statistics, pp. 1476-1485.
PMLR, 2018.
Xu Cheng, Fan Shi, Yongping Liu, Xiufeng Liu, Lizhen Huang, et al. Wind turbine blade icing detection: a federated learning approach. Energy, 254(PC), 2022.
Moming Duan, Duo Liu, Xianzhang Chen, Yujuan Tan, Jinting Ren, Lei Qiao, and Liang Liang. Astraea: Self ballncalnn federated learning for improving classification accuracy of mobile deep learning applications. In 2019 IEEE 37th international conference on computer design (ICCD), pp. 246-254. IEEE, 2019.
Jack Goetz, Kshitiz Malik, Due Bui, Seungwhan Moon, Honglei Liu, and Anuj Kumar. Active federated learning. arXiv preprint arXiv: 12641, 2019.
Rami Hamdi, Mingzhe Chen, Ahmed Ben Said, Marwa Qarage, and Vincent Poor. Federated learning over energy harvesting wireless networks. IEEE Internet of Things Journal, 9(1):92- 103, 2021.
Taeho Jo and Nathalie Japkowicz. Class imbalances versus small disjuncts. ACM Sigkdd Explo- rations Newsletter, 6(1):40-49, 2004.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1-54, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. Foundations and Trends in Machine Learning. 14(1-2):1-210, 2021.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- lems, 2:429-450, 2020b.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.
Charles X Ling and Victor S Sheng. Cost-sensitive learning and the class imbalance problem. Encyclopedia of machine learning, 0011:231-221 2008.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eficientt learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017.
Jed Mills, Jia Hu, and Geyong Min. Communication-effiieeee federated learning for wireless edge intelligence in iot. IEEE Internet of Things Journal, 7(7):598-5537 5994, 2019.
Solmaz Niknam, Harpreet S Dhillon, and Jeffrey H Reed. Federated learning for wireless commu- nications: Motivation, opportunities, and challenges. IEEE Communications Magazine, 58(6): 46-51, 2020.
10

---

Under review as a conference paper at ICLR 2023
Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. ICC 2019-2019 IEEE international conference on communications (ICC), pp. 1-7. IEEE, 2019.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00299 2020.
Monica Ribero and Haris Vikalo. Communication effficii federated learning via optimal client sampling. arXiv preprint urXiv:2007.15197, 2020.
Yuris Mulya Saputra, Dinh Thai Hoang, Diep N Nguyen, Eryk Dutkiewicz, Markus Dominik Mueck, and Srikathyayani Srikanteswara. Energy demand prediction with federated learning for electric vehicle networks. In 2019 IEEE Global Communications Conference (GLOBECOM), pp. 1-6. IEEE, 2019.
Zebang Shen, Juan Cervino, Hamed Hassani, and Alejandro Ribeiro. An agnostic approach to fed- erated learning with class imbalance. In International Conference on Learning Representations, 2021.
Yanmin Sun, Mohamed S Kamel, Andrew KC Wong, and Yang Wang. Cost-sensitive boosting for classification of imbalanced data. Pattern recognition, 40(12):3358-3 3378, 2007.
Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning on non-iid data with reinforcement learning. In IEFE INFOCOM 2020-IEFE Conference on Computer Commu- nications, pp. 1698-1707. IEEE, 2020a.
Jinny Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7623 2020b.
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10165-10173, 2021.
Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas Communications, 3206)):205-122 1221, 2019a.
Xiaofei Wang, Yiwen Han, Chenyang Wang, Qiyang Zhao, Xu Chen, and Min Chen. In-edge ai: Intelligentizing mobile edge computing, caching and communication by federated learning. IEEE Network, (3(5):1566116 165, 2019b.
Yu Wang, Guan Gui, Haris Gacanin, Bamidele Adebisi, Hikmet Sari, and Fumiyuki Adachi. Fed- erated learning for automatic modulation classification under class imbalance and varying noise condition. IEEE Transactions on Cognitive Communications and Networking, 8(1):86-96, 2022.
doi:
Miao Yang, Ximin Wang, Hongbin Zhu, Haifeng Wang, and Hua Qian. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 2174-2178. IEEE, 2021.
Daniel Yue Zhang, Ziyi Kou, and Dong Wang. Fedsens: A federated learning approach for smart health sensing with class imbalance in resource constrained edge computing. In IEEE INFOCOM 2021 IEEE Conference on Computer Communications, pp. 1-10, 2021a. doi: 111199INNOOCOM4221122211111111
Hangjia Zhang, Zhijun Xie, Roozbeh Zarei, Tao Wu, and Kewei Chen. Adaptive client selection in resource constrained federated learning systems: A deep reinforcement learning approach. IEEE Access, 9:98423 98432,, 2021b.
11


Reviews:
[{"review_id": "NJTAcYm4GGK", "reviewer": "Reviewer_8crs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a privacy-preserving class distribution estimation method for dealing with class-imbalanced FL scenarios.", "summary_of_the_review": "While the class distribution estimation method is interesting, it needs more theoretical or empirical justifications. Moreover, it needs to carefully find a scenario where high-quality data A can be obtained; otherwise, the usefulness of the mechanism is very limited.", "strength_and_weaknesses": "S1. The paper is easy to follow.\n\nS2. Some theoretical analysis is provided.\n\nW1. The assumption of theorem 1 is unrealistic (i.e., the input distribution of all classes is equivalent). While the authors provide an example of Fig. 4 to discuss that this can be addressed with the increase of network layers, it is insufficient. More experiments on different types of real data are needed for empirical analysis, and also a theoretical foundation should be established.\n\nW2. The authors’ method needs balanced auxiliary data A for the server to estimate class distribution for each client. In practice, a high-quality A is very hard to obtain (how about the feature distribution of A being different from clients’ data?). Getting high-quality A is itself a very challenging problem in FL. Not to say, for many scenarios, no data can be collected on the server (this is why FL is needed). Then, the usefulness of the authors’ mechanism is doubtful.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4Jb2ac6zFbe", "reviewer": "Reviewer_oRX1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a class-distribution estimation based framework for federated learning. The method incorporates loss reweighting scheme for handling global class imbalance. The paper is completely based on the assumption that after certain rounds of training with imbalanced class, the class probability returned by any sample will be equal to the global distribution of the classes. E.g. if global distribution of a 3-class problem is 5%, 10% and 85%, the returned softmax for each sample will be close to 0.05, 0.10, and 0.85. This is shown with a 2-class example experiment (page 4) and stated as Theorem 1 (page 5).", "summary_of_the_review": "The paper addresses an important issue in federated machine learning. However, the current version is far from being matched to the high standards expected for a conference like ICLR.", "strength_and_weaknesses": "Unfortunately, I noted a number of issues regarding the technical quality of the paper and these are elaborated below.\n\n1) Paragraph below Theorem 1 (page 5) states that the noise arising out of any violation of above assumption can be eliminated by adding more number of layers. This is demonstrated by some results in fig 4, but no proof or theoretical discussion has been provided.\n\n2) In results, accuracy for minority class in all SOTA algorithms are shown to be ABSOLUTELY zero, whereas the proposed algorithm reaches considerable accuracy (except for a few rows for CIFAR10 dataset where it is single digit). It should be investigated if they have implemented the SOTA algorithms correctly.\n\n3) The brief survey on methods for handling class imbalance in the Introduction section is a bit outdated. Given the problem is quite popular there are many majors developments that took place in recent years. Can you please explain a bit more on why data level imbalance handling techniques cannot be applied locally? If possible can you please cite a reference as you did for the case of cost sensitive learning. \n\n4) While it is somewhat clear to me what the paper is aiming to propose the motivation remains a bit vague. A bit more clarity on motivation and a glimpse of the algorithm preferably with an illustrative example will only help to highlight the contributions and novelty. \n\n5) The simulation study I felt is too simple. A more complex example may be using an imbalanced version of MNIST or Fashion-MNIST using a deeper CNN and a high imbalance ratio may be more beneficial. \n\n6) Section 3.2 introduces three new hyper-parameters. Unfortunately,  we do not know how the plateau region behaves with the three hyper-parameters or if its behaviour changes in any way in practice. Also I did not understand how the other cited stopping criteria can aid us or why they are mentioned if not in the scope of the current article.\n\n7) The getLossReweight is a key ingredient of the proposed algorithm and deserves its own place in the paper. Also no intuitive explanation behind the heuristic of converting the estimated priors to corresponding weights is given. Is there any ablation study on this using different choices of functions?\n\n8) Why Accuracy is used in an imbalanced setting given we know it fails to properly evaluate the performance is such cases? Especially, for Table 3 we have only the worst performing minority class performance to compare on. That Minority class may change over algorithms and if that happens the comparison may become unfair to an extent. Also what happens to the second minority class remains a mystery. Can you use some indices that provides an unbiased evaluation of the classifier in presence of class imbalance (see for example, https://doi.org/10.1016/j.patcog.2020.107197)? I also felt the experimental setup is somewhat simple. May be you can order the classes and select different number of samples from each class, so that 1. Pairwise imbalance ratios will vary and 2. The overall imbalance ratio can be made really high like in the range of 50-100.You can also check how the estimated class priors match to that of the real one. \n\n9) One problem of cost sensitive learning is that it can overcompensate. In effect this may bring down the TPRs for relatively majority classes. Can you please confirm if such a case is happening? \n\n10) The algorithm assumes we can generate synthetic data that is balanced, or some real-life balanced dataset is publicly available to server. Balanced auxiliary dataset is mentioned as input to Algorithm 2 and used in line 8 of that algorithm. How much practically relevant is that?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2jJfFta3grV", "reviewer": "Reviewer_wyG4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the global class imbalance problem in Federated Learning and proposes a novel class distribution estimation method. The estimation method is motivated by the observation that the averaged output probability vector from a classifier at the beginning of training is numerically close to the class ratio. The averaged class ratio is then integrated into the local model training in the form of weighted loss. \n", "summary_of_the_review": "While the reviewer appreciates the insights and novelty of the class estimation method, it is poorly supported empirically and theoretically. More experiments on real datasets and larger neural networks, and better description and discussion of the theorem should help improve clarity and make the motivation more convincing. ", "strength_and_weaknesses": "Pros:\n\nThe method of class distribution estimation is novel, and the observation is very interesting. \n\nCons:\n\n**1, Lack of empirical support for the observation.** While the class distribution method is novel, it is only motivated by toy examples. It would be great to see this trend on real datasets and large neural networks.\n\n**2, The theorem is not clear.** The paper provides a theoretical result to support the estimation method. However, it is hard to directly see why this is the case from the description of the theorem alone. \n\nQuestions: \n\n1, It is not clear why local loss functions are reweighted with global class ratios. Shouldn’t each client have its own imbalance ratios? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject