Label-distribution-agnostic Ensemble Learning on Federated Long-tailed Data
Authors: Yaopei Zeng
Lei Liu
Baoyuan Wu
ShaoGuo Liu
Li Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: fedearted learning
long-tailed learning

Abstract: Federated Learning (FL) is a distributed machine learning paradigm that enables devices to collaboratively train a shared model. However, the long-tailed distribution in nature deteriorates the performance of the global model, which is difficult to address due to data heterogeneity, e.g., local clients may exhibit diverse imbalanced class distributions. Moreover, existing re-balance strategies generally utilize label distribution as the class prior, which may conflict with the privacy requirement of FL. To this end, we propose a Label-Distribution-Agnostic Ensemble (LDAE) learning framework to integrate heterogeneous data distributions using multiple experts, which targets to optimize a balanced global objective under privacy protection. In particular, we derive a privacy-preserving proxy from the model updates of clients to guide the grouping and updating of multiple experts. Knowledge from clients can be aggregated via implicit interactions among different expert groups. We theoretically and experimentally demonstrate that (1) there is a global objective gap between global and local re-balance strategies\footnote{The local re-balance strategy means that each client utilizes re-balance methods based on the local label distribution, while the global re-balance strategy applies re-balance methods using global label distribution as the class-wise prior.} and (2) with protecting data privacy, the proxy can be used as an alternative to label distribution for existing class prior based re-balance strategies. Extensive experiments on long-tailed decentralized datasets demonstrate the effectiveness of our method, showing superior performance to state-of-the-art methods.

Under review as a conference paper at ICLR 2023
LABBL-DISTRIBUTIINNAANNSTTT ENSEMBLE LEARN- ING ON FEDERATED LONGED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) is a distributed machine learning paradigm that enables devices to collaboratively train a shared model. However, long-tailed distribu- tion in nature deteriorates the performance of the global model, which is difficult to address due to data heterogencity, e.g., local clients may exhibit diverse im- balanced class distributions. Moreover, existing re-balance strategies generally utilize label distribution as the class prior, which may conflict with the privacy requirement of FL. To this end, propose Labell-iitribution-Agoooo semble (LDAE) learning framework to integrate heterogeneous data distributions using multiple experts, which targets to optimize a balanced global objective un- proxy from
En-
der privacy protection. In particular, we derive a privacy-preserte the model updates of clients to guide the grouping and updating of multiple ex- perts. Knowledge from clients can be aggregated via implicit interactions among different expert groups. We theoretically and experimentally demonstrate that (1) there is global objective gap between global and local re-balance strategies" and (2) with protecting data privacy, the proxy can be used as an alternative to label distribution for existing class prior based re- re-balance strategies. Extensive exper- iments on aangttailed decentralized datasets demonstrate the effectiveness of our method, showing superior performance state-of-the-art methods.
INTRODUCTION
Federated Learning (FL) aims to collaboratively learn from data dominated by number of remote clients and produce highly accurate global model the server with aggregated knowledge. The most important issues in practical FL applications mainly involve data heterogeneity and privacy protection during collaboration of disparate data sources. Such issues are even more significant in the setting of long-tailed data distribution for some real-world scenarios (Cui et al., 2019 Liu
al., 2019), such as medical applications (Li et al. 2019 Malekzadeh al., 2021 and autonomous vehicles (Samarakoon et al. 2019, Pokhrel & Choi 2020).
Under the long-tailes global data distribution, is extremely challenging learn an effective global model by leveraging knowledge from local clients. From the local perspective, there can be large divergence among the imbalanced label distributions of different clients, resulting in the heteroge- neous imbalance shown in Figure ((a) i.e., local datasets different clients may have different imbalance ratios or minority classes. From the global perspective, one should handle the imbalance issue with privacy preservation (Li et al. 2021a), i.e., the server should not require clients upload label distributions for re-balance strategies.
Several techniques have been proposed to tackle the class imbalance problem in FL, such as loss
re- ee-weighting (Wang et al.) [021] Shen et al.) 2021). client clustering (Duan et al.) (020) and the client selection scheme (Yang et 2021). Most of them focus on datasets with only few classes (e.g., s.. ten or twenty classes), suffering from significant performance drops on large-scale imbalanced datasets with more classes (Liu al., 2019: Zhang et al., (2021b). Simultancously, existing solutions generally assume that some sensitive information is accessible to the global server, e.g., balanced
'The local re- be-balacce strategy means that each client utilizes re- re-balance methods based on the local label distribution, while the global re-balance strategy applies re-halance methods using global label distribution as the class-wise prior.

---

Under review as a conference paper at ICLR 2023
CTAB-100 JT. B1 220
magelee L7
Servec
Loca Re calarac balance
balerce
&
Tacal
Clabed Daiset
LDAN
15M
LAZt
R1EE
1600
LDNN
6:M
LA02
1120
PoOo
(a) Heterogeneous Imbalance in FL
(b) Global Re-balance vs. Local Re-balance
Illustration of the long-tailed FL. problem. (a) Different local datasets exhibit diverse imbalanced label distributions. which even differ from the global dataset. (b) For existing class prior based re- re-balance algorithms, the global re-balance strategy outperforms the local one, motivating us to explore global prior information for the long-taile FL. problem.
Figure 1:
mini-dataset (Duan al. 2020 Wang et al. 202T) learnable hyper-parameters of local clients (Shen et al. 2021), ), which may pose data privacy concerns for the realistic FL applications. Further- more, we notice that the effectiveness of existing class prior based strategies (Cao al. 2019 Ren
et al. 2020 Hong et al. 202T for the imbalance issue in FL remains under-explored.
In this work, we investigate the effectiveness of existing class prior based re-balance algorithms from the global and local perspectives under the long-tailed distribution. Experimentally, as indi- cated by Figure [(b)p, combined with these algorithms, the global re-balance strategy yields higher recognition accuracy than the local re- re-balance strategy in the setting of FI.. We theoretically demon- strate that the main reason arises from the gap of objective functions between the global and local re-balance strategies in FL, where the former can yield the matched objective of the centralized training. However, obtaining global label distribution requires clients to upload their own label dis- tributions to the server, which may violate the privacy protection principle in FL (Wang al.. 2021 McMahan et al.] 2017). Thus, it is a critical issue to exploit privacy-pres seraaccpeeeevingggg priors for global re-balance strategy to maintain balanced global objective function.
To overcome the mbetiomentioned problem, we propose a Diis--ibutitt--Aaapaep learning framework (LDAE) to deal with the data heterogeneity and privacy in the long-tailed FL. set- ting. Specifically, we present the proxy information as the class prior of global re-balance strategies rather than label distribution. The proxy information derived from the model updates uploaded by local clients, which is agnostic about the local label distributions for privacy protection. To alleviate the heterogeneous issue, we propose multi-expert model architecture to aggregate the knowledge from different client groups, where clients in the same group have similar local data distribution and train a corresponding expert. The heterogeneity could mitigated through information interaction among different experts trained on different local data distributions.
Ensemble
In conclusion, the key contributions of this work are:
(1) We experimentally and theoretically explore the effectiveness of existing class prior based re- balance algorithms FL. demonstrated that there a mismatch of objectives between local and global re-balance strategies, which indicates that the global re-balance performs better than the local one on the imbalanced decentralized data.
(2) To address the imbalance issue with privacy protection, we propose novel FL framework called LDAE to utilize uploaded model updates to cluster clients into different groups, where a multi- expert architecture is used aggregate the knowledge from different groups with heterogeneous data distribution. Our method agnostic to the label distribution of the clients.
(3) The experimental results multiple benchmark datasets demonstrate that LDAE can signifi- cantly outperform previous ofte--of-theann (SOTA) methods under heterogencous data distribution, simultaneously protecting the data privacy of the clients.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Federated Learning. FL. (McMahan et al. 2017) is a learning framework to protect the data privacy of participants. There is a central server and multiple clients in FL. A global model is learned on distributed data from different clients. One of the most important challenges in FL is data hetero- geneity. Many previous studies concentrated on this problem (Smith et al. 2017 Zhao et al. 2018
Sattler et al. 2020, Karimireddy et al., 2020 [i et al. 2020b T Dinh et al. (020). However, most of them consider that the global dataset (the dataset where all local data is centralized together) is balanced. Recently, some works (Duan al.) 2020 Yang et al. 202 1 Wang et al. 2021] Shen et al.) 2021] focused on the impact of class imbalance in FL and propose different strategies. However, these methods all require extra information other than model parameters experteeeeeeeeee be satisfied with the data privacy concern. Moreover, they only conduct experiments on datasets with a few classes, and their algorithms may not be efficient enough to deal with the large-scale long-tailed imbalance problem. CReFF (Shang et al.) (022)) considers long-tailed data in FL in- spired by (Kang et al. 2019). However, while claiming it has no privacy concerns, their work needs feature gradients of clients' data, which can be used to recover original data with model updates.
the server and may not
Long-tailed Learning. In real-world scenarios, data often has long-tailed label distribution, where the majority classes have massive samples while the minority classes only have a few samples (Zhang et al., 2021b). There are many re-balance strategies proposed from different perspectives
in long-tailed learning. Class re-sampling (Chawla et al., 2002 He & Garcial 2009
Kang et al.,
2019 is common type, such as over-sampling the minority classes (Shen et al.
2016
Kang al.
2019 or under-sampling (He & Garcia (2009) the majority classes. Another scheme to learn a bal-
anced model is loss re-weighting (Cost-sensitive Learning) (Sun et al. 2007 Cui et al.,
2019 Lin
et al. 2014). Generally, these methods tend to give large training loss when the sample belongs to
the minority class. Recently, many works in long-tailed learning have focused on learning a good representation extractor to improve the generalization ability of the model. PaCo (Cui et al. 2021) introduces the contrastive learning method to the long-tailed dataset. Ensemble learning based meth- ods are also becoming more important in long-tailes learning (Zhou et al. 2020 Xiang al. 2020 Wang et al. 2020 Zhang et al. 202 la Cai et al. 2021]. Until now, many re- balance strategies have worked well solve the long-tail imbalance problem on centralized datasets. However, remains a question whether they are useful in FL. In this work, we answer this question through theoreti- cal analysis and experiments results. Then, we propose a novel algorithm to help these re-balance strategies maintain their effectiveness in FL in the following section.
3 METHOD
In this section, we first highlight the challenges when data heterogeneity encounters long-tailed distribution in FL. Then we systematically explore the local and global re-balance strategies as the motivation. Finally, we describe the proposed LDAE framework in detail.
Notations. We discuss a typical FL setting with K clients indexed by [K] and a central server. Each client k has a local training dataset & with a total of n samples. We call the dataset D = Uketa 2* as the global dataset. Considering C-class classification task on the global dataset D, each class is indexed by [C] and (x.8) E X x [C] denotes a sample in D, where x is an image in the input space X' and y is its corresponding label. Let 0 θ be the model parameters and solæ y) denote the loss of sample pair (æ. y). We set nj as the number of training sample for class j. and Σド= n2 is the total number of training samples. Suppose D follows long- long-tailed distribution, i.e., the sample size is exponentially distributed w.F.t. class index. Without loss of generality, assume that the classes are sorted by cardinality in decreasing order, i.e., if i < j. then n, 2 ny. The global imbalance ratio is defined T.max/ / Tmmin.
Typically, FL aims at learning a single shared model and optimizing the global objective, which is the aggregation of the local objective:
K
min
nh
(l(O),
where Fa(0)= = Tik (D.y)e
Σ
fo v).
(1)
VER
3

---

Under review as a conference paper at ICLR 2023
However, the above formulation suffers from long-tailed data distribution and performs poorly the minority classes. In a decentralized system with global imbalanced data distribution, the training data given client is typically based the particular data source. Hence, local datasets may exhibit varying imbalanced label distributions and even differ from the global distribution. We refer such circumstances the following heterogeneous imbalance.
Definition (Heterogeneous Imbalance). Given a global long-tailed training dataset for FL, we define heterogeneous imbalance by the following items: (1) Various imbalanced ratios majority (minority) classes among local datasets. (2) Different imbalanced ratios or majority (minority) classes between local and global datasets.
The definition reflects that the class imbalance issue aggravated by data heterogeneity due to var- ious local imbalanced distributions w.rf the global one. As a theoretical motivation, we consider the following configuration of heterogeneous imbalance to investigate the effectiveness prior based re-balance techniques for the class imbalance issue in FL. The theoretical results inspire us to op- timize a balanced objective from a global perspective instead of a local one. We also verify the theoretical results via extensive experiments.
3.1 THEORETICAL MOTIVATION
We consider a binary classification problem where the ground truth is either positive (y") or negative (yT). Under the imbalanced setting, we assume that clients Co and C1 have heterogeneously imbalanced data distributions, iee., client Co accesses no positives and no negatives, while client C1 accesses nn positives and n; negatives. Without loss of generality, we consider a simple loss re- re-weighting strategy that takes the inverse of the proportion of each class the weight of the loss for this class. To be specific, we set the weight of loss to be no/n _ for the positive class and no /mo for the negative class on client Co- Similarly, client C1 uses n1/n for the positive class and n1 /n for the negative class. For the global re-balance strategy, inverse of the global label distribution is taken as the weights of loss, (i.e., (no 1 +1 //(p +nī] ) for positive class and (mo + +t))) / (mg + + for negative class). We denote the global objective of the global re-balance strategy and the local re- re-balance strategy as (0) and G)(0), respectively. Then we can measure the difference of global object functions between local and global ba-aneence strategies.
Lemma
Using the global re-balance strategy, the global objective yields the same form as the objective the centralized dataset with re-balance methods:
Gy(0) no + n (any+)E
Σ
fo(x,y*)
Σ
(o(xv))
(2)
To + th (0..))
JED
Theorem Under the above setting, let & be the biased estimation of global label distribution. Then there exists group of re- re-balance weights e derived from &, whose global objective Ge satisfies:
(0) < Go( Gi(O), where the objective gap A Gi(()) (0) can be written as: (n no(n
mi (n,) + nn }
Σ fo(w, (
Σ (2.N (R.W EE
Sola,
no ni (No + n1 (n) + nt) (R.Ut)ED
No ni (No (n) n2
(4)
Interpretation. The above analysis illustrates the following points: (1) The global re-balance strat- egy for the federated long-tailes problem optimizes the same objective function the re-balance strategy on the centralized dataset. (2) The local re-balance strategy with a larger objective less (3) There exists re- re-balance strategy derived from the global perspective, which yields a smaller objective than the local re-balance strategy.
effective than the global one with a smaller objective.
3.2 GLOBAL PROXY INFORMATION
Our theoretical findings show that the global re-balance strategy works better than the local re- balance strategy in FL. However, it may violate the privacy protection requirements in FL. To over- come this drawback, we propose class prior called the Global Proxy Information (GPI) based on
4

---

Under review as a conference paper at ICLR 2023
the model updates of the clients, inspired by empirical observation; in the FC layer, the neuron weight of majority classes is updated more frequently than minority classes on the imbalance data. We further provide theoretical guarantee that the label distribution cannot be observed through the numerical LPI value from the global perspective. Thus, GPI can be combined with existing re- balance strategies under privacy protection, which requires additional information from clients.
Definition 2 (Local Proxy Information). We denote the gradient of whs after one local optimization loop on D as Da W and the dimension of the input of the FC layer as H. Then the local proxy information of class on D defined as the gradient magnitude which is associated with i-th neuron of the FC layer:
H
OF.(0) Owoi
2
DA W
where
V W
(5)
f=1
Intuitively, local proxy information (LPI) accumulates the updates of the corresponding neurons of the last FC layer over the local dataset. Note that LPI is calculated without requirements for label distributions of the clients and the central server collects no additional information compared to the vanilla FL algorithm. We consider that the local training uses cross-entropy loss. As a motivation, we claim the theoretical evidence of the privacy protection of LPI in the following.
Theorem For class i in local dataset DR: LPI implicitly connects with its label frequency (sam- ple number in this class) and the probability that the sample is predicted to be the current class:
S
1 n (n
Z.)
Σ (x.g)E j=1
Σ
Sjr
where
Z
Σ 24
(6)
(R.G))
where 8y is j-th input of the FC layer, is é-th term of the one-hot ground truth, and 2; is the probability of being predicted as class i. Theorem 2 indicates that the numerical value of LPI is implicitly relevant to the residual between the label frequency and probability from the local model, hence the local label distributions cannot be derived from the LPI values uploaded by the clients.
Definition 3 (Global Proxy Information). The global proxy information of class i defined as the weighted summation of the local proxy information:
K
Ω = Σ
n* T( > 0)
where I(() is the indicator function and the value is 1 if is true, otherwise.
I(()) is necessary since negative LPI may result in unstable behavior of GPI. In fact, GPI is the gradi- ent accumulation over the global dataset by collecting the LPI of the clients, which only involves the gradient information over local datasets. Importantly, we empirically find that GPI for re-balance strategies alleviate the class imbalance issue while maintaining the recognition performance for majority classes, as shown the experiment section 4
During standard training in the practical FL setting, mini-batch SGD is commonly used for lo- cal training, where GPI can be approximately measured by accumulating gradients over all local datasets after several epochs of gradient descent steps. Besides, in each communication round, is difficult to obtain an accurate GPI, since only fraction of clients are selected for local updates. Thus, the global server calculates GPI based on the uploads of all clients only in the beginning communication round, then keeps balanced training for subsequent rounds.
3.3 LABEL-DISTTIBUTION-AAAAN
ENSEMBLE LEARNING
Guided by our proxy information analysis, we propose a group-based ensemble learning framework to further handle the heterogeneous imbalance issue for FL. Our method separates clients into dif- ferent groups based on their LPI. Models trained on individual groups are integrated together with a multi-expert structure, where each expert focuses a group of clients with similar [.I. The overall
5

---

Under review as a conference paper at ICLR 2023
Step Client Selection
Step Local Training
Step Aggregation
Updined Expert
Proore Expert
Dewnload
Group
Selected
Uplana for and Backlonre
Group
Group 2
ajs
Clients
Shared Backbone
o =0 (random)
Group
Group M
Selected Clisn
Updates for and Bldlhone
Global Model
No Group
(a) Training pipeline of LDAE.
(b) Client selection with different a
Figure 2: An overview of group-based ensemble learning framework. The training pipeline with four steps in a communication round: 1) For each expert, aS clients are randomly selected from the corresponding group and (1 a clients are selected from the other clients. 2) The selected clients download the global model. 3) Each selected client updates the backbone and associated experts. 4) The server receives the updates and aggregates them into new global model. (b) Cases with different a values for client selection. When
O, clients in group only train the corresponding expert. When < a < some clients may be chosen to update multiple experts.
framework and the model structure are shown in Figure 2(a) The global and local models have the same structure, iee. multiple experts with a shared backbone, and each expert has individual learnable blocks.
Client Grouping. Based on the section 3.2 the similarity score between LPI and GPI for each client be calculated via the cosine distance, allowing monotonically rank the local clients. Then the ranked clients with close similarity scores are divided into the same group. Let Hyper-paramete M be the number of groups, then M experts allocated for these groups respectively, i.e., group P; corresponds to expert E.
Client Selection. In communication round, parts of the clients selected update the same expert: (1) aS clients are randomly selected in each group (Pt). and (2) (1 a)SS clients are randomly selected from all clients other than clients (1). S S 1 is hyper-parameter to control the client selection. Empirically, the difference between experts can be controlled by the 1, there is no interaction among the clients from different groups, where each group only updates the corresponding expert. If a 0, we adopt M times booststrap sampling from clients, where the clients selected at i-th time are used to update E; individually. As such, the experts are actually updated on the same global dataset and tend to be similar after sufficient rounds.
value of a. As shown in Figure 2bb).
Expert Ensembling, After the client selection step, the selected clients download the current global model parameters and perform local training on their own datasets. particular, for each client, the classification loss is calculated based on the average logits of all experts to interact the knowledge of different experts. Here, we adopt the balanced softmax loss (BSM) (Ren et al., (0220) for local training due the effectiveness of logit adjusted loss for dealing with the class imbalance issue [Menon et al. 2020). We denote the class prior information, e.g., global label distribution or GPI, as π,, then the BSM is:
M
LBSM
1
Σ (x.
-ylog
1
(a. log
(
M
where o.) is the softmax function and v.(.) is the output logits expert E,. Notice that the local training of expert, only the parameters of the backbone and corresponding expert updated,

---

Under review as a conference paper at ICLR 2023
while other experts are kept frozen. After receiving all the local updates, the server aggregates them into the global model with FedAvg McMahan et al. (2017).
Table 1:
Top-1 accuracy on CIFAR- CIFAR-I00- and ImageNet-LT. The GPI is combined
with various class prior based re- re-balance algorithms to address the imbalance problem in FL.
CIFAR-I0-1E
CIFAR
-O0-LT
Methods
ImageNet-LT
100
50
100
50
FedAvg
53.16
61.67
34.48
36.84
33,80
FedProx
61.43
71.78
34.16
38.77
32.98
Ratio Loss
53.31
62.26
33.06
34.94
33.15
CLIMB
60.28
72.29
34.66
40.22
35,29
Focal Loss
53.88
59.00
33.88
38.03
32.85
CRT-IB
53.80
63.52
32.48
37.61
31,77
CRT-CB
63.31
69.87
34.06
39.60
35,52
LDAM + GPI
66.73
72.58
35.36
39.17
34.88
BSM + GPI
67.44
74.36
37.19
41.91
37.64
LADE GPI
64.89
71.79
38.61
41.08
38.52
RIDE + GPI
58.64
68.63
35.88
39.39
40.30
PaCo GPI
68.84
75.35
37.76
43.24
39.04
LDAE + GPI (ours)
71.07
76.85
40.42
45.16
45.75
4 EXPERIMENT
Datasets and Setup. We conduct experiments three long-tailed classification datasets: CIFAR- 10-LT, CIFAR-100-LT (Cao et al. 2019) and ImageNet-LT (Liu et al., 2019). Following (Cao et al, 2019), construct CIFAR- IO-LT and CIFAR- 1RR-100-LTY by exponential decay with the controllable
IR (imbalance ratio). We show the experimental results with IR = 50 and IR = 100. We adopt Resnet- 18 et 2016) for CIFAR-1O-LT Resnet-32 for CIFAR- CIFAR-100-LT" and ResneXt for ImageNet-L.T.
To simulate the data heterogeneity in FL. we divide the samples of each class in the global dataset into different clients according to the Dirichlet distribution (He al.] 2020 Yurochkin al. 2019). Specifically, we first generate P. ~ Dirk (Orri) for class e and then allocating the Po. proportion of the samples in class client k. The degree of data heterogeneity is controlled by adir- the parameter of the Dirichlet distribution. A small Oddr indicates high data heterogeneity. We set adir as 0.5 for CIFAR-IO-LE 0. for CIFAR-100-L and 0.05 for IeageNet-LTT each communication round, 20 clients are selected, and each client trains 2 local epochs. More implementation details are reported in Appendix[A]
We evaluate the global model on the balanced dataset using top-1 accuracy. Following typical long-tail recognition tasks [iu et al. (2019), we categorize the classes into three groups: Many-shot class with > 100 samples, Medium-shot class with 2 20 and < 100 samples and Few-shot class with < 20 samples.
Evaluation Protocol.
4.1 RESULTS
Comparison with Previous Methods. To evaluate the effectiveness of our group-based multi- expert model, we conduct experiments comparing it with previous methods including the baseline algorithm FedAvg (McMahan et al. 2017). the method for dealing with data heterogeneity FedProx (Li et al.) 2020a), methods for dealing with class imbalance in FL Ratio Loss Wang et al.) (02T) and CLIMBBShen al. 2021), 1 and centralized long-tailed learning methods Focal Loss (L.in al, 2017). CRT-BB, CRT-CB (Kang et al. 2019). LDAM (Cao et al.) 2019), BSM (Ren et al. 2020), LADE (Hong et al., 2021), RIDE (Wang et al. 2020), and PaCo (Cui et al. 2021). The test ac- curacy three typical long-tailed datasets is summarized in Table [ Our approach significantly outperforms previous works on each dataset. Notice that the re- balancing methods requiring class prior information utilize our GPI. They have better results than other previous methods, especially on CIFAR- 100-L and ImageNet-LT, which have large number classes. The subsequent exper- iments show that the GPT makes considerable contribution to their good performance.

---

Under review as a conference paper at ICLR 2023
Table Comparison the performance of different re-balance strategies with local re-balance, global re- re-balance and GPI on CIFAR-IO-LT, CIFAR-100-LT and ImageNet-LT.
CIFAR
10-LT
CIFAR-IOS
1O0-T
Methods
ImageNet-LT
100
50
100
50
FedAvg
53.16
61.67
34.48
36.84
33.80
Re-balance
LDAM
63.55
69.50
32.54
37.01
30.70
Local
BSM
66.64
74.29
35.55
39.50
34.39
LADE
66.05
74.54
33.28
39.89
36.32
RIDE
59.95
69.05
34.94
39.09
40.00
PaCo
65.11
71.35
35,92
40.03
38.26
LDAM
67.69
72.41
36.61
40.01
38.66
Global Re-balance
BSM
65.81
74.15
38.56
42.45
39.63
LADE
66.03
74.06
39.82
42.95
39.37
RIDE
60.37
68.39
34.59
39.35
40.81
PaCo
69.60
72.36
37.63
41.26
39.74
Ours
69.76
75.12
41.77
45.32
45.57
LDAM
66.73
72.58
35.36
39.17
34.88
BSM
67.44
74.36
37.19
41.91
37.64
GPI
LADE
66.89
74.79
38.61
41.08
38.52
RIDE
58.64
68.63
35.88
39.39
40.30
PaCo
68.84
75.35
37.76
43.24
39.04
Ours
71.07
76.85
40.42
45.16
45.75
Effectiveness of GPI. To further verify the effectiveness of GPI, we use local label distribution (lo- cal re-balance), global label distribution (global re-balance), and GPI as the class prior information for several typical centralized long-tailed methods. As shown in Table 2 GPI exhibits better perfor- mance than local re-balance and comparable results to global re-balance, indicating that our method can handle the imbalance issue for FL without data privacy Icakage.
Table 3: Top-1 accuracy of re-balance strategies with local re-balance, global re-balance and GPI on Many/Medium/FFew classes in CIFAR-100-LLT with IR 100.
Methods
Local Re-balance
Global Re-balance
GPI
Many
Medium
Few
Many
Medium
Few
Many
Medium
Few
FedAvg
62.03
32.26
4.93
LDAM
56.23
29.91
7.97
52.03
37.63
17.43
56. 14.14
33.80
12.93
BSM
57.63
36.34
8.87
51.54
41.49
20.00
52.80
39.26
19.07
LADE
54.00
33.74
8.57
52.57
43.34
20.83
55.34
40.46
12.20
RIDE
64.97
31.11
4.37
63.29
32.34
3.73
63.66
34.91
4.60
PaCo 35.60
46.34
24.13
42.06
41.23
28.27
45,94
42.23
22.67
Ours
50.29
44.71
28.07
49.97
44.14
24.93
Furthermore, we report the accuracy of many/mcdium/ccwssooe classes on CIFAR- with IR 100 in 3 Compared with local re-balance using local label distribution, previous cost-sensitive re-balance methods (i.e., LDAM, Balanced SM and LADE) with GPI can achieve significant im- provements for medium-shot and few-shot classes. Similarly. benefiting from GPI, RIDE and PaCo also further increases the recognition accuracy for medium-shot and many classes. The slight performance improvement of few-shot classes may be attributed to the fact that RIDE and PaCo fo- cus on representation quality over all classes rather than merely minority classes. On the other hand, compared with global re- re-balance, GPI shows competitive results for all classes, although exhibits relatively lower accuracy for the medium-shor and few-shot classes. Notice that GPI performs better many-shot classes sometimes due to the subtraction of the probability term, section 3.2 Compared with the previous SOTA method RIDE, our method obtains higher accuracy since heterogeneous imbalance is alleviated via client grouping.
mentioned in
GPI Visualization. As shown in Figure 3 we visualize the class-wist wise GPI curves on CIFAR-100 LT with IR 10 and IR 100. Compared with the groundtruth label distribution the same IR,
8

---

Under review as a conference paper at ICLR 2023
GPI curves exhibit a similar tendency, which can be used for class prior based re-balance strategies. Besides, GPI can maintain the performance of majority classes by using smaller GPI values for re-balancing. It should be noticed that GPI is merely obtained from the model updates implicitly combining label frequency and prediction probability as mentioned in Theorem
Ablation Studies the Ensemble Framework. Ta- ble shows the component analysis of the proposed ensemble framework. We compare three different grouping methods: grouping clients randomly (Ran- dom Grouping), grouping clients by the cosine dis- tances between local and global label distributions (LD Grouping). and grouping clients by the cosine dis- tances between LPI and GPI (PI Grouping). The re-
0.30
0.445
Clocal Lasel Clarsiction Ih-
Precy Inoomation BB
0.040
Giocal
0.20-
Prpprttnn
sults show that Random Grouping contributes little to the performance, while LD Grouping and PI Grouping bring about 3% improvement in accuracy. Our group- ing method further improve the final performance of other re- re-balance strategies besides BSM (Ren et al.] 2020), such LDAM loss (Cao et al. 2019). More- over, we evaluate the influence of different values of a, the tapperparamettr to control the client selection scope of each expert. As shown in Figure 4 the performance can be enhanced via an appropriate value of a.
0310
0005
0.000
Figure 3: Comparison between the global label distribution and GPI on CIFAR-100 LT with IR 10 and IR 100.
Methods LDAM
Accuracy
40
10.4
40
LDAE
35.36
BSM
37.19
88 98.22
PI Grouping + LDAM
38.25
Top-1
Random Grouping + BSM
37.38
LD Grouping + BSM
40.51
36
35.19
0.0
0.2
0.4
0.6
0.8
1.0
PI Grouping + BSM (Ours)
40.42
valuu of
Table 4: Ablation studies for the grouping method and training loss of the ensemble framework CIFAR-100-LT with IR 100.
Figure 4: Ablation studies on different a for client selection in the ensemble framework on DIFAR-100-13 with IR = 100.
Data Heterogeneity. Existing re-balance strategies un-
der the FL setting generally aim to train a balanced local model with compromising performance due to the heterogeneous property of FL. As shown in Table Table and Table GPI can significantly improve the performance by alleviating the data heterogeneity issue. Moreover, our multi aulti-expertt framework could ensemble knowledge learned from different local data sources to further address the heterogeneous imbalance.
Privacy Protection. To train balanced global model, class prior based re-balance methods mainly rely on global label distribution without privacy protection, while the proposed GPI is merely related to the model updates without extra label information. Thus, our method can maintain the data privacy of clients and simultancously achieve better performance on all the benchmarks.
CONCLUSIONS
In this work, we focus on the long-tailed distribution problem in FL. Existing re-balance methods for FL suffer from the data heterogeneity issue and assume label distribution available without consid- ering privacy requirements. To this end, we propose nabel-distriuttion-ggn (LDAE) framework to re-balance the global objective with preserving privacy. In particular, we de- sign the proxy information guide the balanced training of multiple experts, which only involves model updates for preserving privacy. Extensive experimental results indicate the effectiveness of method compared with previous SOTA methods.
ensemble learning

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition one-shot. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 112-121, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distribution--ware margin loss. Advances in neural information processing systems, 32, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International Conference on Machine Learning, pp. 2089-2099. PMLR, 2021.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 715-724, 2021.
Yin Cui, Menglin Jia, Tsung-Y. Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pp. 9268 09779 2019.
Moming Duan, Duo Liu, Xianzhang Chen, Renping Liu, Yujuan Tan, and Liang Liang. Self- balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 32(1):59-71, 2020.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang. Xiaoyang Wang, Pra- neeth Vepakomma, Abhishek Singh, Hang Qiu, al. Fedml: research library and benchmark for federated machine learning. arXiv preprint 2007:2007..3518. 2020.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering,
2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattem recognition, pp. 770-778, 2016.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Dis- entangling label distribution for long-tailed visual recognition. In Proceedings of the IЕEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6626-6636, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint 1rXiv:1910.09221, 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143, 5143. PMLR, 2020.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: vision, hype and reality for data privacy and protection. IEEE Transactions Knowledge and Data Engineering, 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429-450. 2020a.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR, 2021b.
10

---

Under review as a conference paper at ICLR 2023
Wenqi Li, Fausto Milletari. Daguang Xu. Nicola Rieke, Jonny Hancox, Wentao Zhu, Maximilian Baust, Yan Cheng, Sébastien Ourselin, M Jorge Cardoso, et al. Privacy-preserving federated brain tumour segmentation. In International workshop machine learning in medical imaging, pp- 133-141. Springer, 2019.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learn- ing on non-iid features via local batch normalization. In International Conference on Learning Representations, 2020b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Mohammad Malekzadeh, Burak Hasircioglu, Nitish Mital, Kunal Katarya, Mehmet Emre Ozfatura, and Deniz Gündüzz. Dopamine: Differentially private federated learning on medical data. arXiv preprint arXiv:2101.11693 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiienn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8397-8406, 2022.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 2020.
Shiva Raj Pokhrel and Jinho Choi. Federated learning with blockchain for autonomous vehicles: Analysis and design challenges. IEEE Transactions Communications, 4744)4443444448 2020.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in Neural Information Processing Systems, 33:4175-4186, 2020.
Sumudu Samarakoon, Mehdi Bennis, Walid Saad, and Mérouane Debbah. Distributed federated learning for ultra-reliable low-latency vehicular communications. IEEE Transactions on Commu- nications, (8(2):1146-1159) 2019.
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. Robust and communication-efficie federated learning from non-i.id. data. IEEE Transactions Neural Networks and Learning Systems, 3109)3300-3413, 2020. doi: 10. 111199/NNNSS.2199.99999.9
Xinyi Shang, Yang Lu, Gang Huang. and Hanzi Wang. Federated learning heteroge- neous and long ttiged data via classifier re- re-training with federated features. arXiv preprint arXiv:2204. 13399, 2022.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference computer vision, pp. 467-482. Springer, 2016.
Zebang Shen, Juan Cervino, Hamed Hassani, and Alejandro Ribeiro. An agnostic approach to fed- crated learning with class imbalance. International Conference on Learning Representations, 2021.
11

---

Under review as a conference paper at ICLR 2023
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems. 30, 2017.
Yanmin Sun, Mohamed S Kamel, Andrew KC Wong, and Yang Wang. Cost-sensitive boosting for classification of imbalanced data. Pattern recognition, 40(12))3388-3378, 2007.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en-
velopes. Advances in Neural Information Processing Systems, 33:21 18:21394-222 21405, 2020.
Lixu Wang. Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10165-10173, 2021.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-aware experts. arXiv preprint arXiv:2010.0/809 2020.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self- Self-paced knowl- edge distillation for long-tailed classification. In European Conference on Computer Vision, pp- 247-263. Springer, 2020.
Miao Yang, Ximin Wang, Hongbin Zhu, Haifeng Wang, and Hua Qian. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 2178- IEEE, 2021.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Interna- tional Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv preprint arXiv:2107.09249, 2021a.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:21 10.04596, 2021b.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint 180.:1800600882 2018.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for tailed visual recognition. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 9719-9728, 2020.
12

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
The number of communication rounds is 2000 for most methods on CIFAR-I0/100-E1 and 1000
on ImageNet-LT. On the CIFAR-100-LA the learning rate is initialized 0.5 and decayed by 0.05
at round 1600 for all methods except PaCo (Cui et al.) 2021), where the initial learning rate 0.3.
On the CIFARR-O- 1--LT (Cao et al. 2019) with imbalance ratio 50 and 100, the initial learning rate is 0.1 for FedAvg (McMahan et al. 2017). Ratio Loss (Wang et al. (021). CLIMB (Shen et al.) 2021 Focal Loss (Lin et al. (017). CRT (Kang al. 2019). LDAM (Cao al., 2019). BSM (Ren et al. 2020 LADE (Hong et al. 2021), and LDAE. It is 0.2 for RIDE (Wang et al. 2020) and 0.3 for PaCo. The learning rate decayed by 0.1 for all methods. With RIDE, PaCo, and LDAE, the model is trained for 2500 communication rounds on EIFAR-10-LIT due to their complex model structures. For training on ImageNet-LT (Liu et al., 2019), we set the training round as 1000 and learning rate as 0.1, which is decayed at 800th round by 0.1. For CRT, retrain the classifier on the last 200 rounds. For LADE, we set the weight of LADER as 0.01 for CIFAR-IO-LT and 0.1 for LLFAR-100-LL and ImageNet-LT. We set 0.5 in LDAE on all datasets.
B PROOFS
B. PROOF TO THEOREM
I
We denote the global objective of FL model with global re-balance and local re-balance as (0) and (0),, respectively, then we have:
Gg(0)= = Σ
12k
Σ
no + n1
Σ
((x,5500))
no + Th
(ng + nfjme (0)
kEf0,1}
oE(+.-}
Σ
Σ
f(x. 0).
Bi + ni (2.
cE(+i-}
and
Gi(() =
Σ
Σ
(a 0)
no n1
kE(0,1)
oE(+.-)
(ED
(10)
Σ
n10
n1
Σ
f(x, B)
no + n1
n
ni
(a.
where e is the class index. Then, the objective gap is:
A Gi) G)(
Σ
(n6)) + no
f(a. v. 0). (m.yF)E
Σ
(11)
mini(mo m1) (mg n2))
-E(+.-)
For sample number ng > O, we have A > O. indicating that the objective function value of local the same dataset. Thus, let & be the biased
re-balance is always larger than global re- re-balance estimation of global label distribution, there exists a group of re-balance weights derived from E, where e close to the global re-balance strategy. Then re-balance weights e based global objective G. satisfies:
G)( S (0) (()..
(12)
B.2 PROOF TO THEOREM 2
Follow the definition 2 we have:
(F
Ω
(13)
Ow
Then assuming cross-entropy loss are used, we can derive the following equation:
H
2 n
(u
2,) 8j
Σ
(14)
(
1=1
13

---

Under review as a conference paper at ICLR 2023
where ş² is j-th input of the FC layer, y² is i-th term of the one-hot ground truth, and of is the probability of being predicted as class i. Since Σj=1 8y is usually independent of the class, write the above equation
1 nk ( 2,)
Σ
8jt
where
2,
Σ 24-
(15)
(e.
(æ.y)
C COMPARISON OF FEDERATED LEARNING METHODS
Table 5:
Top-1 accuracy of various FL algorithms for dealing with data heterogeneity CIFAR-
100-LT.
FedAvg
FedProx
SCAFFOLD
IR
w/o BSM
w/ BSM
w/o BSM
w/ BSM
w/o BSM
w/ BSM
50
36.84
41.91
38.77
42.33
40.44
42.27
100
34.48
37.19
34.16
37.78
34.70
38.55
FedAlign
Ditto
FedRep
IR
w/o BSM
w/ BSM
w/o BSM
w/ BSM
w/o BSM
w/ BSM
50
39.80
43.66
37.03
39.29
37.69
39.17
100
35,36
39.21
33.45
35.18
32.23
35.04
We further experimented with FedProx (Li et al. 2020a). SCAFFOLD (Karimireddy et al. 2020
FedAlign (Mendieta et al., 2022), Ditto (Li et al. 2021b) and FedRep (Collins et al. 2021) as baselines on the CIFAR 1O0- LT. They are recent FL algorithms for dealing with data heterogeneities.
When combined with BSM loss (Ren et al. 2020), for non-personalized FL methods (i.c.. FedProx
(Li et al. 2020a), SCAFFOLD (Karimireddy et al., (020). FedAlign (Mendieta et al. 2022)). we GPI the class prior, and for personalized FL methods (i.e., Ditto (Li et al. 202 FedRep (Collins et al. 2021) ) local re-balance is applied.
As shown in Table 5 the non-personalized FL methods effective in FL with relatively mild class imbalance (imbalance ratio = 50), while are less effective when class imbalance severe (imbalance ratio 100). More importantly, when combined with re-balance strategy, these methods can obtain a significant performance improvement for various imbalance ratios.
However, we test the personalized FL methods Ditto [4] and FedRep [5] on the federated long- tailed problem, and as the table above shows, they do not perform well on the federated long-tailed problem, even with more training rounds. We believe the main reason is the distribution shift. Under the general setting of personalized FL without the long-tailed problem, the distribution of test data and training data are identical. However, considering the long-tailes problem, the training data is imbalanced and the test data is balanced for each client, resulting in a distribution shift. The personalized model focuses more on fitting the local training data distribution and therefore generalizes poorly to different data distributions. Compared with the personalized FL methods, non- personalized FL methods with a well-trained global model have a stronger generalization ability, so they can achieve better performance the federated long-tailed problem.
Table 6: Top-1 accuracy CIFAR-100-LE and the model size of LDAE with various expert num- bers.
Expert Number
CIFAR- 100
CIFAR--00-LT1 50
Parameters(Million)
37.19
41.91
0.46
2
38.71
44.92
0.52
3
40.42
45.16
0.77
4
41.68
46.88
1.02
41.85
46.59
1.27
14

---

Under review as a conference paper at ICLR 2023
D ABLATION STUDY FOR EXPERT NUMBER OF LDAE
We show the effect of expert number M of our multi-expert method L.AFF. larger M can lead to more parameters. The results indicate that as M increases, the recognition accuracy improves. Considering the trade-off between performance and communication cost, 3 and 4 are both good values for M. When M is larger than 4, the improvement in accuracy is slight.
E VISUALIZATION OF DATA DISTRIBUTION
(a) a=
(b) a=5
(c) a=0.5
Nunder
Cherct
Clext fs
clos Lobe
(d) a=50
(c) a=5
(f) a=0.5
g Global label distribution
Figure 5: The data distributions of local datasets with different settings.
To simulate the data heterogeneity, for each class in the global dataset, partition the samples into different clients according to the Dirichlet distribution. In this section, we partition the CIFAR 10-LT into 10 clients with different values of Odir. i.e., the hyper-parameter of the Dirichlet distribution. In Figure []] (d, f) show the data distributions of local clients with different values of Cdir. and (a, c) show the corresponding probability density of the two-dimensional Dirichlet distribution. The point size indicates the sample number. Small Ourrr results in higher heterogeneity. The global label distribution the CIFARIO-LT with IR 100 is shown in (g).
n}
oe w!!
of
0 =
( > o)af
F For Wr
LPI Class Client
GPI Class
Client
Model Updates
Final PClayer
Figure 6: An example of LPI and GPI calculation process.
F EXAMPLE OF LPI AND GPI CALCULATION PROCESS
15

---

Under review as a conference paper at ICLR 2023
Figure Socure/@jinuuuuue intuiiiivly illustrates how LPI and GPI are calculated. In the first round, the server receives the model updates uploaded by each client. The LPI of class for client calculated by adding the updated values of the weights connected to neuron x² according to the reference local proxy. The GPI of class is then the weighted sum of all clients' LPI based on the Eq The GPI of other classes is calculated in the same way.
G VISUALIZATIONS OF Z VALUE, LPI, GPI, AND LABEL DISTRIBUTION
for Cheir-1
Client
90
99
20
ClowDD
Clas-D)
Clas-DD
(a) Z value for Client-1
(b) Z value for Client-2
(c) Z value for Client-3
of
0.11
32
603
Clns-U
(d) LPI for Client-1
(c) LPI for Client-2
(f) LPT for Client-3
GPT
Figure 7: Visualizations of Z (Bq.6), LPI with local label distribution, GPI with global label distri- bution on CIFAR-1O-LT with 0.5. Sub-figure (a) and (d) are for client 1, (b) and (e) are for client 2, (c) and (f) are for client 3.
To verify Theorem 2, we conduct an empirical study on CIFAR-IO-LT (a = 0.5 and IR=100), which include Z (Eq.6) value curve, LPI with local label distribution, and GPI with global label distribution. These values are obtained in the first communication round. The experimental setup is described in Appendix A We randomly select clients from 100 clients for visualizations.
Z Value (Eq.6). As shown in Sub-figure (a), (b), and (c) of Figure 7 we visualize the class-wise Z, values for the models of different clients at the initial communication round. observed that the model in the first communication round may make biased prediction instead of uniform predictions. Therefore, the Z, obtained at this stage may also be biased.
LPI with Local Label Distribution. As shown in Sub-figure (d), (e), and (f) of Figure we visualize the class-wise LPI estimated at the initial communication round and the groundtruth local label distribution for different clients. The LPI with negative value is set 0 since it will not be used in the calculation of GPI. As these Sub-figures show, even under the biased Z, values, the LPI curve for each client can exhibit similar tendency to the corresponding local label distribution.
16

---

Under review as a conference paper at ICLR 2023
GPI with Global Label Distribution. As shown in Sub-figure (g) of Figure class-wise GPI estimated at the initial communication round. GPI curves have a similar tendency when compared with the groundtruth global label distribution (black curve). Thus, although Z; may be biased at the beginning communication round, the estimated GPI can still reflect the trend of global label distribution. Therefore, GPI can be used the class prior for re-balance strategies under the biased prediction at the beginning communication round.
visualize the
17


Reviews:
[{"review_id": "gzDgfH8rVj", "reviewer": "Reviewer_prdG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies Federated Long-tailed Data, which is a highly challenging task. To address this task, this paper proposes to use a privacy-preserving proxy to guide model training and develops a label-distribution-agnostic ensemble learning framework. Both theoretical analysis and empirical verification are provided.", "summary_of_the_review": "I like this paper overall, but there are a few issues that need to be addressed before acceptance.\n\n**********Post rebuttal***************\nThanks a lot for the response. Most of my concerns have been clarified. Even so, I still have some concerns about the biased prediction at the early stage. However, it is just a minor concern, so I keep my original score of 6 but increase my confidence to 5.  \n\n", "strength_and_weaknesses": "Strengths:\n1. The studied task is practical and highly challenging.\n2. Exploring privacy-preserving proxy to guide model training is new to me.\n3. The empirical results seem promising.\n\nWeaknesses:\n1. Introduction is good-writing, but the method section is hard to follow. \n2. Please carefully check the proof of Theorem 2. It seems not correct from Eq.12 to Eq.13.\n3. please clarify why local label distributions cannot be derived from the LIP values if it is relevant to the residual.\n4. It is unclear to me why the estimated GPI based on the beginning communication round can keep balanced prior for guiding model training.\n5. For global proxy information, how can you know n^k, i.e., the class-level sample number in each client?\n6. Please ablate the expert number M.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7lpMeONnEy", "reviewer": "Reviewer_EiQm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to deal with long-tailed recognition in a federated setting. Different from previous works, they pay much attention to privacy, which is the core idea of federated learning. Since the global label distribution is inaccessible in this setting, they instead use the model updates of the clients as surrogates. In the experiments, they show the proposed method achieves comparable results, compared to methods with global priors. ", "summary_of_the_review": "The paper considers a more strict long-tailed recognition setting by retaining the constraint of privacy in federated learning. This paper is well presented and the results are also good. However, some points are worth further discussing. ", "strength_and_weaknesses": "Strength:\n\n1. Overall, the paper is clear and easy to follow. \n\n2. They especially consider the importance of privacy in federated learning, while alleviating the problem of the long tail. \n\n3. The analysis of the proposed method is comprehensive and the results of the experiment are also good.\n\nWeaknesses:\n\n1. The setting of this paper is not clear. In \"dataset and setup\" in Section 4, a brief operation of how to simulate FL is introduced. \"the global dataset is partitioned into 100 local datasets with the Dirichlet distribution\". For each local dataset, do they draw samples from each class by following Dirichlet distribution?\n\n2. The combination of federated learning and long-tailed datasets is good. However, in this paper, only a primitive federated method is considered. Most of the baselines focus on different reweighting methods in long-tailed recognition. I am curious about the results of directly applying the latest federate methods into long-tailed datasets, including the works discussed in the related works. \n\n3. Can you show some visualization examples of the data distribution of local clients?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0wzwL4NPeDA", "reviewer": "Reviewer_5sfY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address the issue of heterogeneous, imbalanced label ratio among clients of a federated learning system. That is, the setting is that the global label distribution has a long tail; and each local dataset has imbalanced label ratio.\n\nPreviously, this can be addressed by re-balancing strategies which can be applied to either each client separately or to the server directly. Local re-balancing appear to perform worse than global re-balancing but on the flip side, global re-balancing requires local clients to share their label distribution. This does not preserve the privacy of data.\n\nTo combine the best of both worlds, the paper proposes a concept of local proxy information (LPI) that (1) makes use of some aggregated form of local parameter gradient to (roughly speaking) summarize statistics of the local label distribution; and (2) can be combined via weighted averaging in a fashion almost similar to the direct combination of local label distributions albeit with the exclusion of those with negative local proxy. \n\nThe combined proxy is referred to as the global proxy information (GPI), which is core to the proposed label-distribution ensemble learning. First, cosine distances between LPI and GPI are used to partition clients into groups. Then, in each round, an expert component is created for each group: alpha x S clients will be selected for the intended group; while (1 - alpha) x S client will be selected for the rest. \n\nOnce created, selected clients will download its corresponding expert model, bake in local data using the balanced softmax loss parameterized by the GPI and send the updated expert back. Next, to complete the communication round, the server aggregates updates for each expert; the ensemble is the average of those experts.\n\nThe proposed algorithm is tested on long-tailed versions of CIFAR-10, CIFAR-100 and ImageNet. The comparison baselines include FedAvg and many other class-prior based re-balance algorithms. ", "summary_of_the_review": "The paper presents a new idea for handling the long-tailed and imbalanced label distributions in FL. However, key concerns arise regarding (1) the lack of comparison with other types of FL algorithms that could potentially handle the same issue; (2) ambiguities over what the theoretical analysis implies; as well as potential flawed claim on privacy preservation of LPI & GPI. If these issues are addressed properly, the paper would be acceptable but for now, my preliminary assessment is that it is marginally below bar. But, I do look forward to further discussion with the authors to (hopefully) recalibrate my rating.", "strength_and_weaknesses": "Strengths:\n\n+ The problem is well-motivated; the presentation of the proposed algorithm is well-structured.\n+ The algorithm is tested on multiple large-scale benchmark datasets for image classification (CIFAR-100, ImageNet)\n+ The proposed algorithm appears novel & reportedly performs better than the baselines\n\nWeaknesses:\n\n- Lacking coverage of other types of FL algorithms that deal with data heterogeneities, such as personalized federated learning\n- Despite the claim, the theoretical analysis in Section 3 does not shed insight on how global rebalancing is better than local rebalancing -- I will elaborate more on the next section\n- This is also questionable whether one can estimate local label distribution from LPI - will elaborate in the next section\n- It is also not clear which of the re-balance baseline corresponds to the analysis of Section 3; and more importantly, how these baselines are transplanted into the context of FL., i.e. are they applied locally or globally  -- do they all require sharing local label distribution etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject