Harnessing Client Drift with Decoupled Gradient Dissimilarity
Authors: Zhenheng TANG
Yonggang Zhang
Shaohuai Shi
Xinmei Tian
Tongliang Liu
Bo Han
Xiaowen Chu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Deep Learning

Abstract: The performance of Federated learning (FL) typically suffers from client drift caused by heterogeneous data, where data distributions vary with clients. Recent studies show that the gradient dissimilarity between clients induced by the data distribution discrepancy causes the client drift. Thus, existing methods mainly focus on correcting the gradients. However, it is challenging to identify which client should (or not) be corrected. This challenge raises a series of questions: will the local training, without gradient correction, contribute to the server model's generalization of other clients' distributions? when the generalization contribution holds? how to address the challenge when it fails? To answer these questions, we analyze the generalization contribution of local training and conclude that the generalization contribution of local training is bounded by the conditional Wasserstein distance between clients' distributions. Thus, the key to promote generalization contribution is to leverage similar conditional distributions for local training. As collecting data distribution can cause privacy leakage, we propose decoupling the deep models, i.e., splitting into high-level models and low-level models, for harnessing client drift. Namely, high-level models are trained on shared feature distributions, causing promoted generalization contribution and alleviated gradient dissimilarity. Experimental results demonstrate that FL with decoupled gradient dissimilarity is robust to data heterogeneity.

Under review as a conference paper at ICLR 2023
HARNESSING CLIENT DRIFT WITH DECOUPLED GRA- DIENT DISSIMILARITY
Anonymous authors Paper under double-bling review
ABSTRACT
The performance of Federated Learning (FL) typically suffers from client drift caused by heterogeneous data, where data distributions vary with clients. Recent studies show that the gradient dissimilarity between clients induced by the data distribution discrepancy causes the client drift. Thus, existing methods mainly fo- cus on correcting the gradients. However, it is challenging to identify which client should (or not) be corrected. This challenge raises series of questions: will the local training, without gradient correction, contribute to the server model's general- ization on other clients' distributions? when does the generalization contribution hold? how to address the challenge when it fails? To answer these questions, we analyze the generalization contribution of local training and conclude that the gen- eralization contribution of local training is bounded by the conditional Wasserstein distance between clients' distributions. Thus, the key to promote generalization contribution is to leverage similar conditional distributions for local training. As collecting data distribution can cause privacy leakage, we propose decoupling the deep models, i.e., splitting the model into high-level model and low-level one, for harnessing client drift. High-level models are trained on shared feature distributions, causing promoted generalization contribution and alleviated gradient dissimilarity. Experimental results demonstrate that FL with decoupled gradient dissimilarity is robust to data heterogeneity.
INTRODUCTION
To protect data privacy while cooperatively training machine learning models between personal users and organizations, Federated Learning (FL) (Brendan McMahan et al. (016) is widely exploited as powerful framework in recent years. In the FL framework, many clients train models without communicating private data. Federated Average (FedAvg) is proposed to make FL practical in low- bandwidth and low-computing resources environments. However, when data distributions between clients are severely heterogeneous Noon-Independent and Identically Distributed, Non-IID), the convergence and the generalization performance of FL much worse than centralized training which collects all the data (Li et al., 2020a, Karimireddy et al.) 2020 Kairouz et al. 2019
The FL community theoretically and empirically found that the "client drift" caused by the heteroge- neous data is the main bottleneck of FedAvg (Li et al. 2220aa] Karimiredyy et al.) 2020 Kairouz et al.) 2019 Wang al. 2020a). It means that, after several or more training epochs on private datasets, local models on clients become extremely far away from each other. Recent convergence analysis (Li al. 2020a Reddi al.) [202T] Woodworth al. 2020) of FedAvg shows that the degree of client drift is linearly upper bounded by gradient dissimilarity. Therefore, most existing works (Karimireddy
jet al. 2020 Wang et al., 2020a) focus on gradient correction techniques to accelerate the convergence
rate of local training.
However, how to correct the gradients during the local training is still an open problem (Kairouz et al. Woodworth et al. 2020 Karimireddy et al. 2020), especially for achieving better generalization ability. The challenge lies in the lack of criterion for identifying which client should (or not) be corrected. This challenge raises a fundamental question in FL systems: Can the local training specific client m contribute to the generalization performance of the server model when evaluted on other clients' distributions? Moreover, it is also unclear under which conditions the local training can

---

Under review as a conference paper at ICLR 2023
lead to generalization contribution. The in-depth question is how deal with the conditions where local training cannot contribute to the server models' generalizability to other clients.
To answer these questions, we formulate the objective of local training in FL systems as a generalization contri- bution problem. The generalization contribution means how much local training on one client can improve the generalization performance on other clients' distributions for server models. Specifically, we evaluate the generaliza- tion performance
fu
Server
High- level model
Update H
server model locally trained on one client using other clients' data distributions. Our theoret- ical analysis shows that the generalization contribution of local training is bounded by the conditional Wasserstein distance between clients' distributions. This implies that even if the marginal distributions on different clients are the same, is insufficient to achieve guaranteed gener- alization performance of local training. Therefore, the key to promoting generalization contribution to leverage the same or similar conditional distributions for local training.
Homogeneous
feature
Concat
Update H
Low-level model gite
-x
Download R from server
(x
Dm
Client m 5"
However, collecting data to construct identical distribu- tions shared across clients is forbidden due to privacy concerns. To avoid privacy leakage, we propose decou- pling deep neural network into a low-level model and high-level one, i.e., feature extractor network and a clas- sifier network. Consequently, we can construct a shared identical distribution in the feature space. Namely, on each client, we estimate the feature distribution obtained by the low-level network and send the estimated distribution to the server model. After aggregating the received distributions, the server sends the aggregated distribution and the server model to clients simultancously. Theoretically, we show that introducing such simple decoupling strategy promotes the generalization contribution and alleviates gradient dissimilarity. Our extensive experimental results demonstrate the effectiveness of method, where consider the global test accuracy of four datasets under various FL settings following previous works (He et al., 20206 Li al. 2020a] Wang et al.] 2020a).
Figure 1: Training process of our frame- work. The low-level model uses the raw data as inputs, and outputs hm. The high-level model uses ht, and samples from a shared distribution H" as inputs for forward and backward propagation.
Our main contributions include: (1) We theoretically show that the generalization contribution from clients during training is bounded by the conditional Wasserstein distance between clients' distributions, answering the question that when the local training on client contribute to the generalization performance of server models on other clients' distributions.
(2) We are the first theoretically propose that sharing similar features between clients can improve the generalization contribution from local training, and significantly reduce the gradient dissimilarity. (3) We experimentally validate the gradient dissimilarity reduction and benefits of our method on generalization performance.
2 RELATED WORKS
We review FL. algorithms aiming to address the Non-IID problem and introduce other works related to measuring client contribution and decoupled training. Due limited space, we leave detailed discussion of the literature review in Appendix
2.1 ADDRESSING NON-IID PROBLEM IN FL
Model Regularization focuses on calibrating the local models to restrict them not to be excessively far away from the server model. A number of works like FedProx (Li al.] 2020a). FedDyn (Acar)
al. 2021), SCAFFOLD (Karimiredyy et al.) 2020) and FedIR (Hsu et al. 2020) add a regularizer of local-global model difference. MOON (Li ct al. 2021bb adds the local-globe contrastive loss to leam similar representation between clients.
Reducing Gradient Variance tries to correct the directions of local updates at clients via other gradient information. This kind of method aims to accelerate and stabilize the convergence, like Fed- Nova (Wang et al. (2020a). FedAvgM (Hsu et al. 2019), FedAdaGrad, FedYogi, and FedAdam (Reddi) angle reduce gradient variance.
al. 202T) Our theorem 4.2 provides
2

---

Under review as a conference paper at ICLR 2023
Personalized Federated Learning aims make clients optimize different personal models to learn knowledge from other clients and adapt to their own datasets (Tan et al. 2022). The knowledge transfer of personalization is mainly implemented by introducing personalized parameters (Liang et al. 2020 Thapa et al. 2020; Li et al. 2021a), or knowledge distillation on shared local features or extra datasets (He al., 2020 Lin al. 2020 Li & Wang) 2019). Due to the preference for optimizing local objective functions, however, personalized federated models do not have a comparable generie performance (evaluated on global test dataset) to normal FL (Chen & Chao
2021). Some works (Collins et al. 2021; Arivazhagan et al. 2019) also propose to share feature
representations for personalized FL.
2.2 MEASURING CONTRIBUTION FROM CLIENTS
Clients are only willing to participate in an FL training when given enough rewards. Thus, it is important to measure their contributions to the model performance (Yu et al. 22000 Ng et al. 2020
al. 2022 Sim et al. 2020). Some works (Yuan et al. 2022) propose to measure the performance
gaps from the unseen client distributions experimentally. Data shapley (Ghorbani & Zou 2019 Sim et al. 2020 Liu et al. 2022) is proposed to measure the generalization performance gain of client participation. Precisely, these works measure the generalization performance gap with or without some clients that never join the whole process of FL. However, we hope to understand the contribution of clients at each communication round. Consequently, our theoretical conclusion guides modification on data distributions that cannot provide generalization contribution, so that they can improve the generalization performance of the trained model.
2.3 SPLIT TRAINING
Some works propose Split FL (SFL) to utilize split training to accelerate federated learning (Oh et al. 2022 Thapa et al., 2020). In SFL., the model is split into client-side and server-side parts. At each communication round, the client only downloads the client-side model from the server, and conducts forward propagation, and sends the hidden features to the server to compute the loss and conduct backward propagation. These methods aim to accelerate the training speed of FL on the client side and cannot support local updates. addition, sending all raw features could introduce a high risk of data leakage. Thus, we omit the comparisons to these methods.
2.4
PRIVACY CONCERNS
There many other works (Luo et al. 2021 Chang et al., 2019 Li & Wang 2019 Bistritz et al.)
2020 He et al. 2020a Liang et al. 2020: Thapa et al. 2020 Oh et al. 2022) that propose to share the hidden features the server or other clients. Different from them, our decoupling strategy shares the parameters of the estimated feature distributions instead of the raw features, avoiding privacy leakage. We demystify the differences between our method and others in Appendix
PRELIMINARIES
3.1 PROBLEM DEFINITION
Suppose we have a set of clients _ (1,2. 11, .M} with M being the total number of participating clients. FL aims make these clients with their own data distribution D,n, cooperatively learn a machine learning model parameterized as 0 E Rd. Suppose there are C classes in all datasets D, m indexed by [C). A sample in 'D,, is denoted by (x. y) E X' x |C]. where x is model input in the space X and is its corresponding label. The model is denoted by p(0; r) X + RC.
Formally, the global optimization problem of FL can be formulated (McMahan et 2017
et al. 2020a):
M Σ
min F(0)
Pmn Pm Fo. (0)
Σ Pm (e (0;
6EE4
where (0) B(e,,)DD. 1(); v. y) is the local objective function of client m with [(; x. v) CE((((:z),))), CE denotes the cross-entropy loss, Pm > 0 and ΣM Pm = 1. Usually, Pm is set as 2m. where nim denotes the number of samples on client and = >M 7mm The clients usually have a low communication bandwidth, causing extremely long training time. To address this issue. the classical FL algorithm FedAvg (McMahan et al.) 2017) proposes to utilize local updates. Specifically, at each round r, the server sends the global model to subset of clients S C M which are randomly chosen. Then, all selected clients conduct some iterations of updates to obtain
3

---

Under review as a conference paper at ICLR 2023
new client models {0% ), which sent back the server. Finally, the server averages local models according to the dataset size of clients to obtain a new global model 8".
3.2 GENERALIZATION QUANTIFICATION
Besides defining the metric for the training procedure, we also introduce metric for the testing phase. Specifically, we define criteria for measuring the generalization performance for a given deep model. Built upon the margin theory (Koltchinskii & Panchenko 2002 Elsayed et al. 2018), for given model p(0:-) parameterized with 0, we use the worst-case margin to measure the generalizability on the data distribution D:
Definition 1. (Worst-case margin.) Given a distribution D, the worst-case margin of model p(0): -) is defined as Wa(p(0), D) = (((ay))> inf p(Oxx) d(x', x)) with d being a specific distance, where the argmax, ((0; x'); # means the p(0; x') mis-classifies the x'
This definition measures the expected largest distance between the data x with label y and the data x' that is mis-classified by the model p. Thus, smaller margin means higher possibility to mis- classify the data a Thus, we can leverage the defined worst-case margin to quantify the generalization performance for a given model p and a data distribution D under specific distance. Moreover, the defined margin is always not less than zero. It is clear that the margin is equal to zero, the model mis-classifies almost all samples of the given distribution.
DECOUPLED TRAINING AGAINST DATA HETEROGENEITY
This section formulates the generalization contribution in FL systems and decoupling gradient dissimilarity.
4.1 GENERALIZATION CONTRIBUTION
Although Eq. [ quantifies the performance of model with parameter 0, it focuses more on the training distribution. In FL, we cooperatively train machine learning models because of a belief that introducing more clients seems contribute the performance of the server models. Given client m, we quantify the "belicf", i.c., the generalization contribution, in FL systems as follows:
EAI(D,,)) Wa(p(e + D\,m)),
(2)
where A is a pseudo gradient obtained by applying learning algorithm L(-) to a distribution Dm- Wa is the quantification of generalization, and D D'Dm means the data distribution of all clients except for client m. Eq, depicts the contribution of client m to generalization ability. Intuitively, we prefer the client where the generalization contribution can be lower bounded.
Definition 2. The Conditional Wasserstein distance Ca(D, D') between the distribution D and D':
1 F inf JeS(P)u. d((x,)) +
Ca(D, D') 2
2 (--))- - (P
inf
Built upon Definition I
and Eq. we are ready to state the following theorem (proof in
Appendix B.
Theorem 4.1. With the pseudo gradient A obtained by L(Dm)). the generalization contribution is lower bounded:
(aL( D).) (SLL A), D..) |BLL(D.)) ) Wale(0 A), + Dm)) (D,,
where Dm represents the dataset sampled Dm-
Remark Theorem 4.1 limlies that three terms are related the generalization contribution. The first and second terms are intuitive, showing that the generalization contribution of distribution D,,. is expected to be large and similar to a training dataset D,,- The last term is also intuitive, which implies that promoting the generalization performance requires constructing similar conditional distributions. Both the Definition 2 and Theorem 4. use distributions conditioned on the label so we write the feature distribution Hy y as H for brevity in rest of the paper
The similar definition is used in the literature (Franceschi et al. 2018
"hhe pseudo gradient round calculated as: A 07
0
with the maximum local iterations T.
4

---

Under review as a conference paper at ICLR 2023
Built upon the theoretical analysis, is straightforward to make all client models trained on similar distributions to obtain higher generalization performance. However, collecting data to construct such distribution is forbidden in FL due to privacy concerns. To address this challenge, we propose decoupling a deep neural network into a feature extractor network Pot parameterized by Stow E Rd4 and a classifier network parameterized by hhihh E IRd", and making the classifier network trained on the similar conditional distributions with less discrepancy, as shown in Figure T Here, d, and dn represent the dimensions of parameters flow and Bhigh. respectively
Specifically, client can estimate the its own hidden feature distribution as H. using the local hidden features h = PP (x) (x..(>D)) and send Hm to the server for the global distribution approximation. Then, the server aggregates the received distributions obtain the global feature distribution H and broadcasts it, being similar to the model average in the FedAvg. Finally, classifier networks of all clients thus performs local training on both the local hidden features (t)=)))>D., and the shared H during the local training. In what follows, we show that such decoupling strategy can reduce the gradient dissimilarity, besides the promoted generalization performance.
4.2
DECOUPLED GRADIENT DISSIMILARITY
The gradient dissimilarity in FL resulted from heterogeneous data, i.c.. the data distribution on client m, Dm is different from that on client k, D3 (Karimireddy et al. 2020 Li et al. 2020a). The commonly used quantitative measure of gradient dissimilarity is defined as inter-client gradient variance (CGV).
Definition 3. Inter-client Gradient Variance (CGV): (Kairouz et al. 2019 Karimireddy et al. 2020 Vm x. y) V
Woodworth et al. 2020 Koloskova et al. 2020) (GV(F,,)) E v
CGV IS usually assumed to be upper bounded (Kairouz et al. 2019 Woodworth et al. 2020
Lian
et al.
2017 ie., (GV(F, = (( ... fm (f:2,
V F(O)) So with a constant d.
Lower bounded gradient dissimilarity benefits the theoretical convergence rate (Woodworth et al. 2020). Specifically, lower gradient dissimilarity directly causes higher convergence rate (Karımireddy 2020: Li et al., 2020a Woodworth et al., 2020). This means that the decoupling strategy can also beneht the convergence rate if the gradient dissimilarity can be reduced. Now, we are ready to demonstrate how to reduce the gradient dissimilarity CGV with our decoupling strategy. With representing V fm (0; æ, y) as V Ve fm (0; x. y), V Venigh S (0; x, y) we propose that the CGV can be divided into two terms the different parts (see Appendix B.2 for details):
et al.
CGV(F, 0) Bm | fm (0; v, y) - [()))
E(a.y)-D.. [[[ Pooms Jm (0; xx 3) - V Vi F(O)11 + [[ Smihgh Vonig F[O)
(4)
According to the chain rule of the gradients of deep model, we can derive that the high-level part of gradients that are calculated with the raw data and labels (z, y) Dn, is equal to gradients with the hidden features and labels (h- Pirw (x), y) (proof in Appendix B.2
M
V Oragh fm (0; (0, v) = V Vorggn fm (0: h, y), Vongg F(A) = Σ Pm Eara))DD.. Vusgg (0; f h. v), (5)
m==1
in which fm (0; h, v/) is computed by forwarding the iPlwo (x) through the high-level model without the low-level part.
We propose let all clients share a global feature distribution H which approximates all features of clients. Client m will sample h ~ and hm = Po.... (x) to train their classifier network, then the objective function becomes as
M
M Σ m=
min F(O) BERd
Σ Pen ((.a)) f10 h. m=1
Pm ((a.))D hM7
+5005000 h. v) [[(0: (0 (x), + S(0;
haH
(6)
Here, Pm fum with R'm and âm being the sampling size of (2,y) ~ Dm and h N+N
H
respectively, and Ñ = MM Am Now, we are ready to state the following theorem of reducing gradient dissimilarity by sampling features from the same distribution (proof in Appendix B.3
We reuse here for brevity, the input of f can be the input x the hidden feature h P0 (z).
5

---

Under review as a conference paper at ICLR 2023
Theorem 4.2. Under the gradient variance measure CGV (Definition 3 with â. satisfying
Thyn
N N+N' the objective function F(0) causes a tighter bounded gradient dissimilarity, i.e.
Tim +ho
the CGV(F. 0) B(a.a))Dm..ll Vorno Jm (0: x. Vor.. F(0)112 + (N+NY) IIV Bragg fm (0: v)
N2a
V Vonggh F(0)11 CGV(F.)).
Remark 2. Theorem shows that the high-level gradient dissimilarity can be reduced as N²
(N+ Nya times by sampling the same features between clients. Hence, estimating and sharing feature distribu- tions is the key to promoting the generalization contribution and the reduction of gradient dissimilarity, Note that choosing N = oo can eliminate high-level dissimilarity. However, two reasons make it impractical to sample infinite features h. First, the distribution is estimated using limited samples, leading to biased estimations. Second, infinite sampling will dramatically increase the calculating cost. We set Ñ = in our experiments.
4.3 TRAINING PROCEDURE
The training procedure of the proposed decou- pling strategy is simple to implement. Specif- ically, it merely requires two extra steps com- pared with the vanilla FedAvg method: es- timating and broadcasting a global distribution H; b) performing local training with both the local data (a, 3) and the hidden features (h Hly. v).
Algorithm Framework of our method.
server input: initial 0", maximum communication
round R
client m's input: local iterations
Initialization: server distributes the initial model 6
all clients, and the initial global H
Moreover, sampling h Hlyy has two additional advantages as follows. First, directly sharing the raw hidden features may incur privacy concerns. The raw data may be reconstructed by feature inversion methods (Zhao al. 202 One can use different distribution approximation meth- ods to estimate (hm m E 3) avoid expos- ing the raw data. Second, the hidden features usually have much higher dimensions than the raw data (Lin et al. (021). Hence, communicat- ing and saving them between clients and servers may not be practical. We can use different dis- tribution approximation methods to obtain H. Transmitting the parameters of H. can consume less communication resource than hidden fea- tures {Am Im E M).
Server Executes:
for each round r = 0, R do server samples set of clients S, S {1, M). server communicates 8, and 7t" all clients
E S.
for each client m S in parallel do do 0 E..-1 H + ClientUpdate(mm H"")
end for 6"41
>MM Pm. 0
m.
Update 71+ using { ( Im E }
end for
ClientUpdate(mn P.) for cach local itcration it with t = 0,,... , do
Sample raw data (x. y) ~ Dm and h ~ Hly.
Following previous work (Kendall & Gal 2017). simply assume Gaussian distribution ap- proximate the feature distributions. Namely, on the client-side, use a Gaussian Distribution N (pm, J..) parameterized with pm and om to approximate the feature distribution on client m. On the server-side, another Gaussian Distribu- tion A (Pg. (g) estimate the global feature distributions. As shown Figure I and Algorithm I during the local training, clients update pmm and o. using the real feature hmm following a moving
Om +- Om,, Mon, V ((;;; x, h v). i... Eq.
Update Hm. using hr... Forow (z).
end for Return and Hmm server.
average strategy which is widely used in the literature (Toffe & Szegedy 2015 Wang et al. 202T):
((t+1) (1 + (1 3mm) mean(hm),
(t+1) 3mo
(t) m + (1 - Bm) x variance(htm).
a m
where is the iteration of the local training, 3m is the momentum coefficient. On the server side, Mg and og are aggregated as:
04
o.
(8)
Sr
iES
jESt

---

Under review as a conference paper at ICLR 2023
Table 1: Best test accuracy (%) of all experimental results.
Cent. Acc.
FL Setting
FL Test Accuracy
Dataset
E
M
FedAvg
FedProx
SCAFFOLD
FedNova
Ours
0.1
10
83.65
83.22
82.33
84.97
88.45
0.05
1
10
75.36
77.49
33,6
73.49
81.75
CIFAR-10
92.53
0.1
5
10
85.69
85,33
84.4
86.92
88.10
0.1
1
100
73,42
68.59
59.22
74.94
77.56
Average
79.53
78.66
64.89
80.08
83.97
0.1
10
88.67
88.92
87.81
87.97
90.83
0.05
1
10
82.73
83.66
76.16
81.89
86.42
FMNIST
93.7
0.1
5
10
87.6
88.41
88.44
87.66
89.87
0.1
1
100
90.12
90.39
88.24
90.40
90.98
Average
87.28
87.85
85.16
86.98
89.53
0.1
10
88.20
87.04
83.87
88,48
92.37
0.05
10
80.67
82.39
82.29
84.01
90.25
SVHN
95.27
0.1
5
10
86.32
86.05
83.14
88.10
91.58
0.1
1
100
92.42
92.29
92.06
92.44
93.42
Average
86.90
86.94
85.34
88.26
91.91
0.1
10
69.38
69.78
65.74
69.52
70.28
0.05
10
63.80
64.75
61.49
64.57
66.60
CIFAR-100
74.25
0.1
5
10
68.39
68.71
68.67
67.99
68.79
0.1
1
100
53.22
54.10
23.77
55,40
56.07
Average
63.70
64.34
54.92
64.37
65.44
"Cent." means centralized training. "Ace." means test accuracy.
where 7 stands for the maximum iteration of local training.
We perform the second step the proposed decoupling strategy by optimizing the designed objective function, i.e., Eq. [6] Built upon the above analysis, the decoupling strategy can benefit both the performance contribution, i.e., conclusion of Theorem 4.1 and the convergence rate, i.e., Theorem 4.2
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
Federated Datasets and Models. We verify our method with four datasets commonly used in the FL. community, i.e., CIFAR-10 (Krizhevsky & Hinton 2009). FMNIST (Xiao et al. 2017). SVHN (Netzer al., 201T), and CIFAR- 100 (Krizhevsky
Hinton 2009). We use the Latent Dirichlet Sampling (LDA) partition method to simulate the Non-IID data distribution, which is the most used partition method in FL (He al. 20206 et al.
202 lb
Luo et al.) (02T). We conduct
experiments with two different Non-IID degrees, 0.1 and
0.05. Some additional experiment
results are shown in Appendix D.3
Baselines and Metrics. We choose the classical FL algorithm, FedAvg (McMahan et al.
2017) and
recent effective FL algorithms proposed to address the client drift problem, including FedProx (Li
et al. 2020a), SCAFFOLD (Karimireddy et al., 2020). and FedNova (Wang al.) 2020a).
baselines. The detailed hyper- hyper-parameters of all experiments are reported in Appendix D We use two metrics, the best accuracy and the number of communication rounds to achieve target accuracy,
which is set to the best accuracy of FedAvg. We also measure the weight divergence (Karimireddy Eies ||0 0e |l. as reflects the effect on gradient dissimilarity reduction.
al. 2020
5.2
EXPERIMENTAL RESULTS
Basic FI. setting. shown Table using the classical FL training setting, i.e. a 0.1, and M 10, for CIFAR-10, 10, FMNIST and SVHN, method achieves much higher generalization performance than other methods. We also find that, for CIFAR-100, the performance our method similar FedProx. We conjecture that CIFAR- 100 dataset has classes than other datasets, leading to the results. Thus, a powerful feature estimation approach instead of a simple Gaussian assumption can be promising direction enhance the performance.
Impacts of Non-IID Degree. As shown in Table I for all datasets with high Non-IID degree (a 0.05), methods obtain more performance gains than the case of lower Non-IIID degree
7

---

Under review as a conference paper at ICLR 2023
Table 2: Communication Round to attain the target accuracy.
FL Setting
Target Acc.
Communication Round to attain the target accuracy
Dataset
E
M
FedAvg
FedProx
SCAFFOLD
FedNova
Ours
82.0
142
128
863
142
128
0.05
10
73.0
247
121
NaN
407
112
CIFAR-10
0.1
5
10
84.0
128
128
360
80
78
0.1
100
73.0
957
NaN
NaN
992
706
0.1
10
87.0
83
76
275
83
32
0.05
81.0
94
NaN
395
FMNIST
0.1
10
87.0
147
31
163
88
17
0.1
100
90.0
375
470
NaN
317
441
0.1
10
87.0
292
247
NaN
251
50
0.05
10
80.0
578
68
358
242
50
SVHN
86.0
350
NaN
NaN
0.1
100
92.0
356
669
356
346
0.1
10
69.0
712
857
NaN
733
614
0.05
10
61.0
386
386
755
366
313
CIFAR-100
0.1
5
10
68.0
335
307
182
282
300
0.1
100
53.0
992
939
NaN
910
854
"NaN." means that this algorithm does not achieve the target.
(a 0. For example, we obtain 92.37% test accuracy SVHN with 0.1, higher than the FedNova by 3.89%. Furthermore, when Non-IID degree increases to a = 0.05, we obtain 90.25% % test accuracy, higher than FedNova by 6. 14%. And for CIFAR-100, our method shows benefits when a = 0.05, demonstrating that our method can defend against more severe data heterogeneity.
Different Number of Clients. We also show the results of 100-client FL setting in Table method works well with all datasets, demonstrating excellent scalability with more clients.
[ Our
Different Local Epochs. More local training epochs E could reduce the communication rounds, saving communication cost in practical scenarios. Table the results of = on CIFAR- R-00, FMNIST, and SVHN verify that our method works well when increasing local training time.
Weight Divergence. We show the weight di- vergence of different methods in Figure 2 (b), where FedNova is excluded due to its significant weight divergence. The divergence calculated by [S Eies- |0-0,], which shows the dissim- ilarity of local client models after local training. At the initial training stage, the weight diver- gence is similar for different methods. During this stage, the low-level model still unstable and the feature estimation is not accurate. After about 500 communication rounds, our method begins to show lower weight divergence than others, indicating that it converges faster than other methods.
Raurd
(a) Test Accuracy
(b) Weight Divergence
Figure 2: CIFARIO with a 0.1, 1, M 10.
Convergence Speed. Figure shows that our method can accelerate the convergence of F And we compare the communication rounds that different algorithms need to attain the target accuracy in Table 2 The results show that our method can improve the convergence speed. The possible reasons for failure cases may be due to the too many categories in the dataset.
Table 3: Splitting at different layers.
Layer
5-th
9-th
13-th
17-th
Test Acc. (%)
87.64
88.45
87.86
84.08
'Due to the high instability of training with severe data heterogeneity, we show the actual test accuracy as semitransparent lines and the smoothed test accuracy opaque lines for better visualization. We also provide convergence figures in Appendix D
8

---

Under review as a conference paper at ICLR 2023
5.3 ABLATION STUDY
To verify the impacts of the depth of gradient decoupling, we conduct experiments by splitting at different layers, including the 5-th, 9-th, 13-th and 17-th layers. Table 3 demonstrates that our method can obtain benefits at low middle layers. Decoupling at the 17-th layer will decrease the performance, which is consistent with our conclusion in Sec. 44. Specifically, decoupling at a very high layer may not be enough resist gradient dissimilarity, leading to weak data heterogeneity mitigation. Interestingly, according to Theorem 4.2 decoupling at the 5-th layer should diminish more gradient dissimilarity than the 9-th and 13-th layers; but it does not show performance gains. We conjecture that it is due to the difficulty of distribution estimation, since biased estimation leads to poor generalization contribution. As other works (Lin et al., (02T) indicate, features at the lower level usually are richer larger than at the higher level. Thus, estimating the lower-level features is much more difficult than the higher-level.
5.4 DISCUSSION
In this section, provide some more exper- imental supports for our method. All experi- ment results of this section are conducted on CIFAR-10 with ResNet-18, a = 0.1, E' = and M 10. And further experiment result are shown in Appendix D.3 due to the limited space.
Our method only guarantees the reduction of high-level gradient dissimilarity without consid- ering the low-level part. We experimentally find
1000
Roard
(a) Some Chonsen layers.
(b) Different Parts.
Figure 3: Layer divergence of FedAvg.
that low-level weight divergence shrinks faster
We choose and show the divergence of 10 layers in Figure 3 (a), and the different stages of R8 in Figure 3 (b). As we hope to demonstrate the divergence trend, we normalize each line with its maximum value. The results show that the low-level divergence shrinks faster than the high-level divergence. This means that reducing the high-level gradient dissimilarity is more important than the low-level.
than high-level. Here, we show the layer-wise weight divergence Figure 3
We conduct FedAvg with many communication rounds with and without learning rate decay. We show the results of the first 5000 rounds in Figure M(a) in Appendix D FedAvg without learning rate decay can only achieve 86.96% accuracy, and FedAvg with learning rate decay only achieves 82.65% accuracy. The results show that the longer training time cannot fill the generalization performance gap between FedAvg and centralized training, encouraging us develop new optimization schemes to improve the performance of FL.
LIMITATIONS
Estimation of Feature Distribution. In this work, we only use the Gaussian Distribution to estimate the feature distribution. This significantly limits the performance of this framework, while it can work well in our experiments. Future works may exploit better feature estimators like generative models (Goodfellow et al.) 20144] Karras et al. 2019) sample higher-quality features.
Extra Communication and Calculation Cost. Our method only needs to communicate the pa- rameters of the estimated feature distribution, which are much less than all features of clients. Some quantization or sparsification methods can be used to further reduce the communication cost. urtherrrher,,,, our method doubles the calculation costs of the forward and backward process of the high-level model. Thus, more reducing gradient dissimilarity, more calculation costs. This plays as a trade-off and needs to be further studied in the future.
7 CONCLUSION
In this paper, we raise a series of fundamental questions related to measuring the generalization contribution of local training from the clients. Then, we theoretically show the relationship of this generalization contribution with the conditional Wasserstein distance between clients' distributions. The theoretical conclusion inspires to propose decoupling gradient dissimilarity, which greatly reduces the gradient dissimilarity by training with a shared feature distribution without privacy concerns. We theoretically verify the gradient dissimilarity reduction and experimentally validate our methods' benefits on generalization performance. Our work opens a new view promoting FL performance from generalization perspective.

---

Under review as a conference paper at ICLR 2023
ETHIC STATEMENT
This paper does not raise any ethical concerns. This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, diseriminationniiaa/aaaaee concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility, we have listed all hyper-parameterr hardware and software of all experiments in the Appendix [D] Due to the privacy concerns, we will upload the anonymous link of
codes and instructions during the rebuttal to make only visible to reviewers. All explanations of assumptions can found in Section and and the complete proof of Theorem and 4.2 can be found in Appendix
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regularization. In International Conference on Learning Representations, 2021. URL httppsss///ppeeee...eee net. wtt.:/e.ee..rrret////// forum?
Id=B7v40MR679W
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. CoRR, abs/1912.00818, 2019. URL http
wwtpxii.og//bs/////// 00818
Ilii Bistritz, Ariana Mann, and Nicholas Bambos. Distributed distillation for on-device learning. Advances in Neural Information Processing Systems, 33:22593- 2020.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Conmunication--Eficcce Learning of Deep Networks from Decentralized Data. arXiv e-prints, art. arXiv:1602.05629, February 2016.
Kuntai Cai, Xiaoyu Lei, Jianxin Wei, and Xiaokui Xiao. Data synthesis via differentially private markov random fields. Proc. VLDB Endow., (411))2190-2202,1 jul 2021. ISSN 2150-8097, doi: 11t.//10111777/3311
10.14779/3346224447777777.. URL httph:: 118
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv: 1812.01077 2018.
Hongyan Chang, Virat Shejwalkar, Reza Shokri, and Amir Houmansadr. Cronus: Robust and heteroge- neous collaborative learning with blxck-box knowledge transfer. arXiv preprint 1rXiv:1912.11277 2019.
Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large-cohort training for federated learning. Advances in Neural Information Processing Systems, 34, 2021.
Chatalic, V Schellekens, F Houssiau, de Montjoye, L Jacques, and Gribonval. Compressive learning with privacy guarantees. Information and Inference: A Journal of the IMA, 0S 2021. ISSN 2049-8772. doi: 10.1093//maii//raab0000. URL iaab005 iaab005.
org 10. 1093/ /maiai///
Hong-You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference Learning Representations, 2021.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.0/24. 2020.
10

---

Under review as a conference paper at ICLR 2023
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representa- tions for personalized federated learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2089-2099 PMLR, 18-24 Jul 2021.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In Advances in Neural Information Processing Systems, 2018.
Jean- Yves Franceschi, Alhussein Fawzi, and Omar Fawzi. Robustness of classifiers uniform lp and gaussian noise. In International Conference on Artificial Intelligence and Statistics, pp. 1280-1288. PMLR, 2018.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-2251. PMLR, 2019.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint
arXiv:2008. 04489, 2020.
Jack Goetz, Kshitiz Malik, Duc Bui, Seungwhan Moon, Honglei Liu, and Anuj Kumar. Active federated learning. arXiv preprint MxXiv:/909, 12641, 2019.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings the 27th International Conference on Neural Information Processing Systems- - Volume 2, NIPS' 14, pp. 2672-2680, Cambridge, MA, USA, 2014. MIT Press.
Weituo Hao, Mostafa El-Khamy, Jungwon Lee, Jianyi Zhang, Kevin J Liang, Changyou Chen, and Lawrence Carin Duke. Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
3310-3319, 1021.
Moritz Hardt and Guy N. Rothblum. A multiplicative weights mechanism for privacy-preservee data analysis. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pp. 61-70, 2010.
doi: 10. 10.1109/FOCS..000888
Moritz Hardt, Katrina Ligett, and Frank Mesherry. A simple and practical algorithm for dif- ferentially private data release. F. Pereira, C. C. Burges, L. Bottou, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- neurips pttpp/ccppppaa//22//11111 ciates, Inc., 2012. URL https: htttttt::/pprccceddiiiee .pdf Sofeeffffeee5cceccccccaa22222223
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. In Advances in Neural Information Processing Systems 34, 2020a.
Chaoyang He, Songze Li, Jinhyun So. Mi Zhang, Hongyi Wang. Xiaoyang Wang. Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang. Murali Annavaram, and Salman Avestimehr. Fedml: A research library and benchmark for federated machine learning. arXiv preprint vxXiv22007.33118 2020b.
Chaoyang He, Alay Dilipbhai Shah, Zhenheng Tang, Fan Adarshan Naiynar Sivashunmugam, Keerti Bhogaraju, Mita Shimpi, Li Shen, Xiaowen Chu, Mahdi Soltanolkotabi, and Salman Avestimehr. Fedev: A federated learning framework for diverse computer vision tasks. arXiv preprint arXiv:2 2 1066, 2021.
T. Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. ArXiv, 0tts/199..06335. 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part X 16, pp. 76-92. Springer, 2020.
Zhouyuan Huo, Bin Gu, and Heng Huang. Training neural networks using features replay. Advances in Neural Information Processing Systems, 31, 2018.

---

Under review as a conference paper at ICLR 2023
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pp. 448-454455 456.. PMLR. 2015.
Max Jaderberg. Wojciech Marian Czamecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In International conference on muchine learning, pp. 1627-1635. PMLR, 2017.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Communication-effiee effficient on-device machine learning: Federated distillation and augmentation under non-iid private data. NeurlPS, 2018.
Noah Johnson, Joseph P Near, and Dawn Song. Towards practical differential privacy for sql queries. Proceedings of the VLDB Endowment, 11(5):526 539, 2018.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint avXiv:1912.04977, 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In ICML, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 538 1-381, PMLR, 13-18 Jul 2020. URL mlr www.prrssssssse htt//v119//// http////////ooo/aaaaaaaaaaa/aa html
Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. The Annals of Statistics, 30(1):1-50, 2002.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Fan Lai, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. Oort: Efficient fed- erated learning via guided participant selection. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pp. 19-35. USENIX Association, July 2021. URL https: http:://wwwww..i...te http:///oo//cooeeeeee//////222//
ISBN 978-1-939133-22- presentation/la
Ang Li. Jingwei Sun. Binghui Wang, Lin Duan, Sicheng Li, Yiran Chen, and Hai Li. Lotteryfl: Empower edge intelligence with personalized and communication -ffficcinnn federated learning. In 2021 IEEE/ACM Symposium on Édge Computing (SEC), pp. 68-79, 2021a. doi: 10. 145/3453142. 3492909.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv 2019.
preprint arXiv:
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 2021b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Sys- tems, volume 2, pp. 429 450,
URL http://wwwwo/ttttt/////ooeogggnn
mlays. www./org//ooo ort/paperrppp /
12

---

Under review as a conference paper at ICLR 2023
Xiang Li. Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. International Conference on Learning Representations, 2020b. URL
/ttEps://ppenzevee... net http://wwwweet/ouum.mm.. Tot.e/ooorrm-H--HAAAY.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized study for decentralized parallel stochastic
algorithms outperform centralized algorithms? gradient descent. In Advances in Neural Information Processing Systems, pp. 5330-5340, 2017.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach. David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint (rXi::201..0552. 2020.
Lin, Wei-Ming Chen, Han Cai, Chuang Gan, and song han. Aleoory-efficieee patch- based inference for tiny deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL net Eww.ommdd=CCmeeemm
7uKNp
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In NeurlPS, 2020.
Zelei Liu, Yuanyuan Chen, Han Yu, Yang Liu, and Lizhen Cui. Gtg-shapley: Efficient and accurate participant contribution evaluation in federated learning. ACM Trans. Intell. Syst. Technol., 13(4), may 2022. ISSN 2157-6904.
Yunhui Long, Boxin Wang, Zhuolin Yang. Bhavya Kailkhura, Aston Zhang, Carl Gunter, and Bo Li. G-pate: Scalable differentially private data generator via private aggregation of teacher discriminators. Advances in Neural Information Processing Systems, 34, 2021.
Sindy Löwe, Peter O' O'Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient-isolated learning of representations. Advances neural information processing systems, 32, 2019.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-IID data. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. UtLpst www..eeee.....eccoooopppp. net/ forum?id=AAF iH CNnVhS
Enrique S Marquez, Jonathon S Hare, and Mahesan Niranjan. Deep cascade learning. IEEE transactions neural networks and learming systems, 2911))55475 1445475---855, 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication-effiche learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1288 1282, 2017.
Omar Montasser, Steve Hanneke, and Nathan Srebro. Ve classes adversarially robustly learnable, but only improperly. Conference on Learning Theory, pp. 2512-2530. PMLR, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http: stanford.edu. pdd
Kang Loon Ng, Zichen Chen, Zelei Liu, Han Yu, Yang Liu, and Qiang Yang. A multi-player game for studying federated learning incentive schemes. In IJCAI International Joint Conference on Artificial Intelligence, pp. 5279, 2020.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In International conference machine learning, pp. 4839 4850. PMLR, 2019.
Seungeun Oh, Jihong Park, Praneeth Vepakomma, Sihun Back, Ramesh Raskar, Mehdi Bennis, and Seong-Lyun Kim. Locfedmix-sl: Localize, federate, and mix for improved scalability, convergence, and latency in split learning. In Proceedings of the ACM Web Conference 2022, pp. 3347-3357, 2022.
13

---

Under review as a conference paper at ICLR 2023
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference Learning Representations, 2021.
1dd=LkFG31R13US
Monica Ribero and Haris Vikalo. Communicationn-efficinn federated learning via optimal client sampling. arXiv preprint arXiv:2007.15197, 2020.
MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. arXiv preprint ar htl::0006.0544., 2020.
Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low, Col- laborative machine learning with incentive-aware model rewards. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-17, 2022. doi: 10.1109/ 1NN.S..002.31606699.
Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. arXiv preprint arXiv:2004. 2020.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem heterogeneous federated optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 7611-7623, 2020a.
URL htttpa://proceddings htw.eeeedddinn.....iep..ee neurips wwtpaperppppppppppccccooee 2aaeer/222//FF0 file/ pdf
56412 33c1277cccca0002450eeeeeeefffee0
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. In International Conference on Learning Representations, 2020b.
Yulin Wang, Gao Huang, Shiji Song, Xuran Pan, Yitong Xia, and Cheng Wu. Regularizing deep networks with semantic data augmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch local sgd for heterogeneous distributed learning, Advances in Neural Information Processing Systems, 33:6281 016228166922 6292, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXhv:1708.07747 2017.
Haibo Yang, Minghong Fang, and Jia Liu, Achieving linear speedup with partial worker participation in non-iid federated learning. In International Conference on Learning Representations, 2020.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In International Conference on Learning Representations, 2021.
http//ppnnrvvieeeeee net. wtte://fooummm.d0aga2000a00
Han Yu, Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong, Xi Weng, Dusit Niyato, and Qiang Yang. A sustainable incentive scheme for federated learning. IEEE Intelligent Systems, 35(4): 58-69, 2020. doi: 10. I 109/MIS. 1000009.2.0000..9999... 2090.2987774.
Honglin Yuan, Warren Richard Morningstar, Lin Ning, and Karan Singhal. What do we mean by generalization in federated learning? In International Conference on Learning Representations, 2022. URL whtpsttpp.... /tttt::::///enrevviii
n wtwee//foormmmmdaimmmmmm
Yufeng Zhan, Peng Li, Zhihao Qu, Deze Zeng, and Song Guo. A learning- based incentive mechanism for federated learning. IEEE Internet of Things Journal, (0966600-366 2020. doi: 10.1 109/JIOT. 2020.2967773
14

---

Under review as a conference paper at ICLR 2023
Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimina- tion good for transfer learning? In International Conference on Learning Representations, 2021. URL /ttp:/ttpp///pereeieee.. net,
net/ Eorumn1dd=CCLL2WUUA http:///aaaaaaaaeeoooee.cccootta
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:/806..00 2018.
Huiping Zhuang, Zhenyu Weng, Fulin Luo, Toh Kar-Ann, Haizhou Li, and Zhiping Lin. Accumulated decoupled learning with gradient staleness mitigation for convolutional neural networks. In International Conference Machine Learning, pp. 12935-12944 PMLR, 2021.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX
A BROADER IMPACT
Measuring Client Contribution During Local Training. As discussed in the section current works mainly focus on measuring generalization contribution from clients from participating during the whole training process. We consider measuring this contribution during each communication round, which opens a new angle toward the convergence analysis of FL. Future works may fill the generalization gap between FL and centralized training with all datasets.
Relationship between Privacy and Performance. We analyze the relationship between the sharing features and the raw data in section 4.2 and Appendix. However, we do not deeply investigate how sharing features or parameters of estimated feature distribution threatens the privacy of private raw data. Sharing features at lower level may reduce gradient dissimilarity and high generalization performance of FL, leading to higher risks of data privacy. Future works may consider figuring out the trade-off' between data privacy and the generalization performance with sharing features.
Connections of our work to knowledge distillation and domain generalization. The approximation of features generated based on the client data and low-level models can be seen as kind of knowledge distillation of other clients. More in-depth analyses of this problem would be exciting direction, which will be added to our future works. The domain generalization is also an exciting connection to federated learning. It is interesting connect the measurements of client contribution to the domain generalization.
B PROOF
B. BOUNDED GENERALIZATION CONTRIBUTION
Given client m, quantify the generalization contribution, in FL systems follows:
E..L(P..) Wa(p(O + A),
where A is pseudo gradient obtained by applying learning algorithm L(D,.) distribution D,n- Wa is the quantification of generalization, and 2 ()) means the distribution of all clients except for client m.
Theorem B.1.
With the pseudo gradient obtained by L(Dm), the generalization contribution is lower bounded:
W
2 -Wa(p(0 D..) (D,, D),m),
( Wa A),
where Dm represents the dataset sampled from Dm
Proof. To derive the lower bound, we decompose the conditional quantification of generalization. i.e. Wa(p(0 + D)m.) Walp(O P),)) Wa(p(0 + D), Wa(p(8 + D,,
WW(p(0 D),)) Walp(O + A), Wa(p(O A),
(10)
where we denote p as p(0 + A) for brevity and D,m stands for the dataset sampled from Dm Built
upon the decomposition, we have:
2 (.L. A),
EAL(P. Wa(p(O A), Wa(p(0 A),
(FLL/p.) +), A), Walp(0 A), Pm))) ))
(11)
The first term Eq. represents the empirical generalization performance. The second term in Eq. 11 means that the performance gap between the model trained sampled dataset and that trained
16

---

Under review as a conference paper at ICLR 2023
on the distribution, rigorous analysis can be found in (Montasser et al. 2019). Note that, the first two terms are independent on the distribution D,D D)Dm), so the focus of generalization contribution is mainly on the last term, i.c., [B..L(D...) Wape...WWW((((( Wa(p(8 A), D)....) W. Wa(p(O A), D..)).
The proof is relatively straightforward, as long we derive the upper bound of Wa(p(0 + A), Dm)
and Wa(o(o + Dmm) For Wa(o(O + A), Dm), we have:
Wa(p(O A), Dm)
= (-))), E BowDml
inf argmax, ((0a)))
d(., x')
-Rul- B Ded argmax, ((0:)) ),y d(r,x²))
inf
inf <1B(, E ( argmax, ((0r))
d(x', x²)) d(x, x²)
-Bu E (, argnan p(0:=)) " }
inf
d(x' x²) + ((), ( d(x,,") x²)
inf -Il Eg DDmD p( (oyy
d(x', ,²)) B(ly)>D,, (a. dir, x"),
where J, stands for the optimal transport between the conditional distribution Dm and D D' m. Similarly, we have:
Walp(B AA) SB (1))--) Er
inf d(x',x") p(0:" #y
B w),D),, B (a, 1( xzx
Combining these two inequality, we have:
[Wa(p(0 + D..) Wa(p(E + D), $2Cd D)Dm))
+ max {5(Dm: D (Dm), ((D DVDm))
(12)
where
6(Dm, D(Dm). = =B(,)))-D. ED Dm
inf pl0 Ag inf
d(x', x²)
B( (r En ap Dey
d(x', x²)),
and
D(Dm)) B W) B.nly (00:)))
D)D,,) = =B(,)) By Dall inf d(x', x²))
inf d(x', x²))
The upper bound is straightforward. For example, if the label distributions the same, i.. y ~ Dmm is equal to y ~ Dm, we have:
Wa(p(E + A), Dm.) Wa(p(B A), P(Dmm) 2Ca D(.m.))
According to Eq. 12 the last term in Eq. is bounded:
EAL(RR.) W (p( + A), Dm)) -
(13)
<EALDD)) Wa(p(O + A), P..)) A)
which is further upper bounded by conditional Wasserstein distance when the label distributions are not the same:
ESLIP, Walp:)))W(p(00- + ()) P.)) Wa (.kr::/WW6000---))...)---00----0
(14)
< (D. D (...).) + max (6(Dm, D(D (D, D(Dm)
Thus, the label distribution will have additional impact on the bound. If the label distributions are the same, then we have
EALTP,, Ma.. A), - D..)) (Dm D(D,,),
(15)
which completes the proof.
n
17

---

Under review as a conference paper at ICLR 2023
B.2 JUSTIFICATION OF DECOUPLING GRADIENT VARAINCE
The derivation of Equation Because Vm = {V (Veoo V Venser Sm E Rd,, Viioe Sm E lRd and
V Onignnm E TR we have
(( ()
(16)
d
fm (0:2,0) -
V m (0; v) VF(0)()) + (VIm(0:2,1)))) vF()))))
i=d++1
-B( V fm (0: 2. y) V 8tow F(O) + V Vnigh (m 2, y) - V Vonigh F(0]
The derivation of Equation 5 Assuming multi-layers neural network consists L linear layers, each of which is followed by an activation function. And the loss function is C'E()). The forward function can be formulated as:
f(0, [00
(17)
Then the gradient on I-th weight should be:
0ʃ
0s
(z)) Izn θTn-1 0 Ozn-1 0741(2441)
Ozi
9i
00,
Ör (2n)
den (Tn-1(2n-1) dzn-1 0Tn-2(2n-2) 02411
08,
(18)
0f
(2(2n) (2n- 1.. ((22)))))))))))
(19)
(2n)
0s
II r((:)))) ((41(242)7()) 11-4+2
(20)
(zn)
in which I ni zt, is the weight, activation function, output of the 1-th layer, respectively. Thus, can see that the gradient of I-th layer is independent of the data, hidden features, and weights before I-th layer if we directly input a z (-th layer.
B.3 PROOF OF THEOREM
We restate the optimization goals of using the private raw data (z,z) 3) of clients and the shared hidden features h H y as following:
M
min F(0) = DeRd
Pr FL 10 h,
E Prr BeeDDDDD [M(;a,,) S(0; ((0;hh.0) b.)
ha7H
hw7t
(21)
Theorem B.2. Under the gradient variance measure CGV (Definition with i'm satisfying
N Rom +hem Ñ+ the objective function F(0) causes tighter bounded gradient dissimilarity. i.e.,
the CGV(F.0) B(z,))DD..1 | Ver.o Im (0; xz Von. F(O) + (N+NY= 11 Venngg I' (0; x 1)
Naa
Vorigh 2(0) < CG)(F,
Proof.
CGV(F,0) = -E (0; v ((0)112
hn7
-B Di [I] V Blo (0; Viw F3
+ B( hw7
|I| Onege, Jm (0; x. 3) + V Vonga smn (0; y) - Vonign F(O)12. (22)
(23)
18

---

Under review as a conference paper at ICLR 2023
On m-th client, the number of samples of (z, y) is nm and the hm is fim Then the high-level gradient variance becomes:
E (m.y)~Dm hm7
Thm
Venegn (0;
im
V Vesga Im (0: h v) - V Vonign F(0)]1
nm
im
nmm + fem
= =IE (a.y))) hitt M
Tym
Vonigh SmiinnSm(0:000 10
im
V Vosign Im (0: R y)
nm
nen + Pom
T.m +
flm
Rym
V Vessn (0; 2)
imm
V Bhigh Sm (0; h 1)
N
Mm
Nmm Thy
=10 (r.,)
7m
Whiigh fm 10;2,11
nm
Venio f (n,(0;,) y)
Nmm + flm
N + N
m=1
N (N N2)2
M
E (a.
V Pr 20;
Σ
Rym
V Vensgh fm (0; x, y)11
(24)
N
m=1
Combining Equation 24 and 22 we obtain
CGV(F, 0) E (e, V Dro Im (0; (2...))))) u)
V fro F(0)112
N
+ (N N2²
V Vnnggs Im (0((00 (0;2,1) Vinigh F(0)11
which completes the proof.
For the convergence analysis, there have been many convergence analyses of FedAvg from a gra- dient dissimilarity viewpoint (Woodworth et al. 2020: Lian et al. 2017 Karimireddy et al. (020)) Specifically, the convergence rate is upper bounded by many factors, among which the gradient dissimilarity plays crucial role in the bound. In this work, propose novel approach inspired by the generalization view to reduce the gradient dissimilarity, we thus provide a tighter bound regarding the convergence rate. This is consistent with our experiments, see Table 2
C MORE RELATED WORK
C.I ADDRESSING NON-IID PROBLEM IN FL.
The convergence and generalization performance of Federated Learning (FL) McMahan et al. 2017 suffers from the heterogeneous data distribution across all clients (Zhao et al. Kairouz et al., 2019). There exists a severe divergence between local objective functions of clients, making local models of FL diverge (Li al. 2020a Karimireddy al. 2020), which is called client
2018 Li et al. 2020b
drift.
Although researchers have designed many new optimization methods to address this problem, it is still an open problem. The performance of federated learning under severe Non-IID data distribution far behind the centralized training. The previous methods that address Non-IID data problems can be classified into the following directions.
Model Regularization focuses on calibrating the local models to restrict them not to be excessively far away from the server model. A number of works (Li et al. 2020a Acar et al. 2021: Karimireddy al. 2020) add regularizer of local-global model difference. FedProx (Li et al. (020a) adds
penalty of the 1.2 distance between local models to the server model. SCAFFOLD (Karimireddy et al. 2020) utilizes the history information to correct the local updates of clients. FedDyn (Acar et al. 2 proposes to dynamically update the risk objective to ensure the device optima is asymptotically consistent. FedIR (Hsu et al. 2020) applies important weight to the client's local objectives to obtain an unbiased estimator of loss. MOON (Li et al. 2021b) adds the local-global contrastive loss to learn a similar representation between clients. CCVR (Luo et al. (202)) transmits the statistics of logits and label information of data samples to calibrate the classifie.
Reducing Gradient Variance tries to correct the local updates directions of clients via other gradient information. This kind (Wang et al. 2020al Hsu et al. 2019 Reddi et al. 2021) of methods aims
19

---

Under review as a conference paper at ICLR 2023
to accelerate and stabilize the convergence. FedNova (Wang et al. 2020a) normalizes the local updates to eliminate the inconsistency between the local and global optimization objective functions. FedAvgM (Hsu et al. 2019) exploits the history updates of the server model to rectify clients' updates. FEDOPT (Reddi et al. 2021) proposes a unified framework of FL.. It considers the clients' updates as the gradients in centra lize training to generalize the optimization methods in centralized training into FL. FedAdaGrad and FedAdam are FL versions of AdaGrad and Adam.
Sharing Features. Personalized Federated Learning hopes make clients optimize different personal models to learn knowledge from other clients and adapt their own datasets (Tan et al. 2022). The knowledge transfer of personalization is mainly implemented by introducing personalized
parameters (Liang et al.
20200] Thapa et al. 2020 Li et al., (021a). or knowledge distillation (He)
ct al. 2020a, Lin et al., 2020 & Wang 2019] Bistritz et al.] 2020) on shared local features or
extra datasets. Due to the preference for optimizing local objective functions, however, personalized
federated models do not have a comparable generic performance (evaluated on global test dataset) to
normal FL (Chen & Chao 2021). Our main goal is to learn better generic model. Thus, we omit comparisons to personalized FL algorithms.
Except Personalized Federated Learning, some other works propose to share features improve federated learning. Cronus (Chang et al., 2019) proposes sharing the logits to defend the poisoning attack. CCVR (Luo et al. 2021) transmit the logits statistics of data samples to calibrate the last layer of Federated models. CCVR (Luo et al. 2021) also share the parameters of local feature distribution. However, do not need to share the number of different labels with the server, which protects the privacy of label distribution of clients. Moreover, our method acts as framework for exploiting the sharing features to reduce gradient dissimilarity. The feature estimator does not need be the Gaussian distribution of local features. One may utilize other estimators or even features of some extra datasets rather than the private ones.
The original cause of client drift is data heterogeneity. Some researchers find that sharing a part of private data can significantly improve the convergence speed and generalization performance (Zhao et al.) 2018), yet it sacrifices the privacy of clients' data.
Sharing Data.
Thus, to both reduce data heterogeneity and protect data privacy, series of works (Hardt Rothblum 2010; Hardt et al.) 2012 Chatalic et al. [202]] Johnson et al.) 20088] Cai et al.) (021) add noise on data to implement sharing data with privacy guarantee to some degree. Some other works focus on
sharing a part of synthetic data(Jeong et al. 2018 Long et al., 2021; Goetz & Tewari 2020 Hao jet al. 202I) or data statistics (Shin et al.) 2020 Yoon et al. 2021 to help reduce data heterogeneity rather than raw data.
FedDF in et al. 2020) utilizes other data and conducts knowledge distillation based on these data to transfer knowledge of models between server and clients. The core idea of FedDF is to conduct finetuning on the aggregated model via the knowledge distillation with the new shared data.
C.2 MEASURING CONTRIBUTION FROM CLIENTS
Generalization Contribution. Clients are only willing to participate FL training when given enough rewards. Thus, it is important to measure their contributions to the model performance Yu
al.
2020 Ng al. 2020 Liu al. 2022 Sim al. 2020).
There have been some works (uuan et al. 2022 Yu et al. 2020 Ng et al. 2020 Liu et al. 2022
Sim et al. 2020) proposed measure the seneralization contribution from clients in FL. Some
works Yuan et al. 2022) propose to experimentally measure the performance gaps from the unseen 2020) is proposed to measure the client distributions. Data shapley (Ghorbani & Zou| 2019 Yu et al.
generalization performance gain of client participation. (Liu et al. (0022) improves the calculation efficiency of Data Shapley. And there some other work that proposes measure the contribution by learning-based based methods (Zhan et al. 2020). Our proposed questions are different from these works. Precisely, these works measure the generalization performance gap with or without some clients that never join the collaborative training of clients. However, we hope to understand the contribution of clients at each communication round. Based on this understanding, we can further improve the FL training and obtain better generalization performance.
has been empirically verified that a large number of selected clients introduces new challenges to optimization and generalization of Fl. (Charles et al. 2021), although some theoretical works show
20

---

Under review as a conference paper at ICLR 2023
Table 4: Demystifying different FL. algorithms related the sharing data and features.
Shared Thing
Low-level Model
Objective
Chatalic al
202
Cai et
2021
Raw Data With Noise
Shared
Others
et
Params, of Data Generator
Shared
Global Model Performance
Yoon el m1
212T Shin el
STAT. of raw Data
Shared
Global Model Performance
STAT. of Logis, Lahel Distribution
Shared
Global Model Performance
Chang CE
2019)
Hiddon Features
Shared
Defend Poisoning Attack
2019
logits
Private
Personalized FL
Ic cl a.
21200
Lranng el
2020
Hiddeen Features
Private
Personalized FL
1 Iapps of
2020
Ul al
20
Hidden Features
Shared
Accelerate Training
Ours
Parain. Estimated Feat. Distribution
Shared
Global Model Performance
Note: "STAT." means statistic information, like mean or standard deviation, "Feat. means hidden features, "Params." mcans parametors.
the benefits from it (Yang et al., 2020). This encourages us to understand what happens during the local training and aggregation.
Client Selection. Several works (Cho et al., 2020; Goetz et al.) 2019 Ribero & Vikalo 2020 Lai et al. (021) propose new algorithms to strategically select clients rather than randomly. However, these methods only consider the hardware resources or local generalization ability. How local training affects the global generalization ability has not been explored.
C.3 SPLIT TRAINING
To efficiently train neural networks, split training instead of end-to-end training is proposed to break the forward, backward, or model updating dependency between layers of neural networks.
To break the backward dependency on subsequent layers, hidden features could be forwarded to another loss function to obtain the Local Error Signals (Marquez et al., 2018 Nøkland & Eidnes 2019; Löwe et al. 2019: Wang et al., 2020b Zhuang et al.] How to design suitable local error still remains as an open problem. Some works propose to utilize extra modules to synthesize gradients (Jaderberg et al. 2017). so that the backward and updates of different layers can be decoupled. Features Replay (Huo et al. 2018) is to reload the history features of the preceding layers into the next layers. By reusing the history features, the calculation different layers could be asynchronously conducted.
Some works propose Split FL (SFL) utilize split training to accelerate federated learning Oh et 2022; Thapa et al., 2020). In SFL, the model is split into client-side and server-side parts. At each communication round, the client only downloads the client-side model from the server, conducts forward propagation, and sends the hidden features to the server for computing loss and backward propagation. This method aims to accelerate FL's training speed on the client side and cannot support local updates. In addition, sending all raw features could introduce high data privacy risk. Thus, we omit the comparisons to these methods.
We demystify different FL. algorithms related to the shared features in Table
D DETAILS OF EXPERIMENT CONFIGURATION AND ADDITIONAL EXPERIMENTS
D.1 HARDWARE AND SOFTWARE CONFIGURATION
We conduct experiments using GPU GTX-2080 Ti, CPU Intel(R) Xcon(R) Gold 5115 CPU @ 2.40GHz. The operating system is Ubuntu 16.04.6 LTS. The Pytorch version is 1.8.1. The Cuda version is 10.2.
D.2 PAPE-EPRRAMESERS
The learning rate configuration has been listed in Table We report the best results and their learning rates (grid search in {0.0001, 0.001, 0.01, 0.1,).3))
And for all experiments, we use SGD as optimizer for all experiments, with batch size of 128 and weight decay of 0.0001. Note that set momentum as 0 for baselines, as we find the momentum
21

---

Under review as a conference paper at ICLR 2023
of 0.9 may harm the convergence and performance of FedAvg in severe Non-IID situations. We also report the best test accuracy baselines that are trained with momentum of 0.9 in Table 6 The client-side momentum in FL training does not always commit better convergence because the momentum introduces larger local updates, increasing the client drift, which is also observed in recent benchmark (He et al., (02T). And the server-side momentum (Hsu al. 2019) may improve the performance. The compared algorithms including FedAvg, FedProx, SCAFFOLD, FedNova do use the server-side momentum. For the fair comparisons did for all algorithms.
the server-side momentum
For K 10 and 100, the maximum communication round is 1000, For K 10 and E the maximum communication round is 400 (due to the E = 5 increase the calculation cost). The number of clients selected for calculation per round for K 10, and 10 for K 100.
Table 5: Learning rate of all experiments.
FL Setting
Dataset
FedAvg
FedProx
SCAFFOLD
FedNova
Ours
0.1
1
10
0.1
0.1
0.1
0.1
0.1
0.05
1
10
0.1
0.1
0.01
0.1
0.1
CIFAR-IE 10
0.1
5
10
0.1
0.1
0.1
0.1
0.1
100
0.1
0.01
0.1
0.1
0.1
0.1
0.05
10
0.1
0.1
0.001
0.1
0.1
FMNIST
0.1
5
10
0.1
0.1
0.1
0.1
0.1
0.1
100
0.1
0.1
0.01
0.1
0.1
0.1
10
0.1
0.1
0.01
0.1
0.05
0.1
0.1
SVHN
0.1
5
10
0.1
0.01
0.01
0.01
0.1
0.1
100
0.1
0.1
0.001
0.1
0.1
0.1
10
0.1
0.1
0.1
0.1
0.1
0.05
10
0.1
0.1
0.1
0.1
0.0
CIFAR-100
0.1
0.1
0.1
100
0.1
1
100
0.1
0.1
0.1
0.1
0.1
Except the Figure in the main paper, provide more convergence results Figures 4
6 and
These results show that our method can accelerate FL training and obtain higher generalization performance.
Mi a Mnn
382
150
259
264
2000
(a) a
0.1, K 10, (b)
0.1, K (0
0.1, K
100. (d)
0.05, K 10.
E
E=
==1
E=
Figure 4: Convergence comparison of CIFAR-10.
D.3 ADDITIONAL EXPERIMENTS
Training with Longer Time. To demonstrate the difficulty optimization of FedAvg in heterogeneous-data environment, we show the results of training 10000 rounds, shown in Figure (a). During this 10000 rounds, the highest test accuracy of FedAvg with fixed learning rate is 88.5%. and it of the FedAvg with decayed learnign rate is 82.65%. Note that set the learning rate decay exponentially decay at each communication round, wich rate 0.997. Even after 2000 rounds, the learning rate becomes the around 0.0026 times the original learning rate.
22

---

Under review as a conference paper at ICLR 2023
Table 6: Baselines with Momentum-SGD.
FL Setting
Dataset
FedAvg
FedProx
SCAFFOLD
FodNova
E
K
0.1
1
10
79.98
83.56
83.58
81.35
0.05
1
10
69.02
78.66
38.55
64.78
CIFAR-10
0.1
5
10
84.79
82,18
86.20
86,09
0.1
1
100
49.61
49.97
52.24
46.53
0.1
1
10
86.81
87.12
86.21
86.99
0.05
1
10
78.57
81.96
76.08
79.06
FMNIST
0.1
5
10
87.45
86.07
87.10
87.53
0.1
100
90.11
90.71
85.99
87.09
10
88.56
86.51
80.61
89.12
0.05
1
10
82.67
78.57
74.23
82.22
SVHN
0.1
5
10
87.92
78.43
81.07
88.17
0.1
1
100
89.44
89.51
89.55
82.08
0.1
10
67.95
65.29
67.14
68.26
0.05
1
10
62.07
61.52
59.04
60.35
CIFAR-100
0.1
5
10
69.81
62,62
70.68
70.05
0.1
1
100
48,33
48.14
51.63
48.12
200
425
2812
100 150 200 200 100 200 199
200 425
3956
800
1000
(a)
0.1, K (b)
0.1, K (c)
0.1, K
(d)
0.05, K 10.
E
E
=
E.
Figure 5: Convergence comparison of FMNIST.
(a) a 0.1, K
1b, E 5
0.1, K
(0) a 0.1, K
1d0,
0.05. K 10.
E ==1
E
Figure 6: Convergence comparison of SVHN.
(a)
0.1. K
(b)
0.1,
10,(c)
0.1.
100.(d)
0,05,
10.
5=
=
==1
Figure 7: Convergence comparison of CIFAR100.
Table 7: Test Accuracy of our method with different degrees of noise.
#4
0.0
0.001
0.005
0.01
0.05
0.1
0.5
Test Accuracy (%)
88.45
88.43
88.23
88.26
88.07
88.11
88.3
23

---

Under review as a conference paper at ICLR 2023
Accuracy [96]
Top-1 Test Accuracy [%)
80
80
70
6.
60
0.005
Av
50
-0.0000 0.050
Faxed Learning Rate
Pr 0.100
Decayed Learning
0.500
40
2000
4000
6000
8000
10000
200
600
800
1000
Round
Round
(a) Long Training Time
(b) Our method with different degrees noise
Figure 8: Additional experiments on CIFAR-IO with a = 0.1, K = 10, =
0.30
1.0
12
Werggt Diargence
0.28
En cie
Wekget Diaegeetee
2.0
0.0
Vilce
700 cae
0.04
0.4
0.4
0.02
8.2
0.2
200
200
800
1000
1000
(a) ResNet-18 with FMNIST
(b) ResNet--1 SVHN
(c) ResNet-50 with CIFAR-100
Figure 9: Layer divergence FedAvg.
1084 Text Acc Aocuuneyy (4)
80
70
60
Peetfee
Fechoo)
40
100
Figure 10: Convergence comparison of FEMNIST with 3400 clients.
Table 8: Test Accuracy of different algorithms with FEMNIST.
FedAvg
FedProx
FedNova
Ours
Test Accuracy (%) Comm. Round to attain the Target Acc.
80.83
79.70
68.96
82.77
82
NaN
NaN
45
Sharing Estimating Parameters with noise. To enhance the security of the sharing feature distribu- tion, we add the noise ~ Ao s.) on the am and pmm The privacy degree could be enhanced by the larger Me. We show the results of our method with different Pe in F (b) and Table 7 The results show that under the high perturbation of the estimated parameters, our method attains both high privacy and generalization gains.
More Experiments of the Real-world Datasets. To verify the effect of our methods on the real-world FL. datasets, we conduct experiments with Federated EMNIITTFEMNISTT (Caldas et al.) 2018 [ee et al.] 2020b). which has 3400 users, 67 151585 training samples and 77483 testing samples. We sample 20 clients per round, and conduct local training with 10 epochs. We search the learning rate for algorithms in {0.01, 0.05, 0.1} and find the 0.05 is the best for all algorithm. Figure 10 and Table g show that our method converges faster and attains better generalization performance than other methods. Note that the SCAFFOLD is not included the experiments, it has very high requirement (storing the control variates) simulating 3400 clients with few machines.
24

---

Under review as a conference paper at ICLR 2023
More Results of the Liyer-wise Divergence. We conduct more experiments of the layer divergence of FedAvg with different datasets including FMNIST, SVHN and CIFAR-100, training with ResNet- 18 and ResNet-50. As Figure and the divergence of the low-level model divergence shrinks faster than the high-level. Thus, reducing the high-level gradient dissimilarity more crucial than the low-level.
25


Reviews:
[{"review_id": "wwk3gUt2Rl", "reviewer": "Reviewer_UyvV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of data heterogeneity in federated learning. It suggests mathematical analysis based on the “worst-case margin” theory to measure the generalization contribution of clients’ FedAvg updates. Based on theoretical results authors propose decoupling the neural network into 2 parts: feature extractor and classifier in a desire to decrease gradient dissimilarity. Experimental evaluation shows the benefits of the proposed method on (synthetic) heterogeneous federated datasets.", "summary_of_the_review": "The main part of the paper lacks a lot of details which makes it very hard to understand clearly the theoretical contributions. That is why it is hard to judge their significance based only on text claims.\n\nThe empirical section lacks important baselines and relies on a very restrictive assumption of gaussian features distribution.\n\nAlthough, if my concerns are addressed and questions are clarified I am open to raising my recommendation.\n\n\n**Minor comments and typos**\n\n- Is the problem setting limited only to classification?\n- Typos: “varaince” -> variance on pages 6, 16\n    - “Learnign” -> learning on page 20\n- Looks like a missed word on page 7 in the sentence “We conjecture that CIFAR-100 datasets … , **leading to the results**”.\n- SCAFFOLD work was already published and could be referred to as an ICML paper.\n", "strength_and_weaknesses": "## Strengths\n\n1. The paper is trying to address a critical problem in federated learning.\n\n2. Experimental evaluation is quite extensive and done accurately.\n\n3. Theoretical concepts suggested for analysis seem novel in this setting.\n\n\n## Weaknesses\n\n1.  Problem origin: it is said that FedAvg suffers from client drift caused by heterogeneous data. The issue is that the so-called “client drift” is a consequence of the optimization method which relies on local training. In the distributed case without multiple local SGD steps, the variance can be eliminated via known control variate techniques, and even with local steps, it is not a problem from an optimization viewpoint. The current theory is not able to explain the bad performance of these methods in the setting of training non-convex deep neural networks. So, it is hard for me to agree with the original problem formulation.\n\n2. The authors make an assumption that feature distributions can be approximated by a Gaussian referring to some previous works. Why is it the case? Was this assumption tested in practice e.g. via a statistical test? Is it the case only for a particular dataset and model architecture? Can the proposed method be combined with standard federated learning techniques like differential privacy and secure aggregation?\n\n3. Experiments were conducted only on synthetically created small federated versions of centralized datasets. It is not clear whether the conclusions can be generalized to natural federated datasets like Federated Extended MNIST, Stackoverflow, etc.\n\n4. Experimental results are obtained for single runs which limits their credibility. In addition, the curves in Figure 2 (b) are almost indistinguishable and the benefits of the proposed approach are not clear.\n\n5. Important baselines like FedAdam/FedAdagrad are missing. They were shown to tackle well the natural heterogeneity in federated datasets. Can the method from this work be combined with adaptive optimizers?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ciEOp7bZ72", "reviewer": "Reviewer_3s3d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of client drift in Federated Learning, a problem well-known in the literature to diminish the efficacy of FL methods. The authors propose to use a decoupling of features into low and high-level ones, they show that by grouping and sharing the low-level features among clients, gradient dissimilarity can be reduced, and thus performance gains can be obtained.\nIn practice, the low-level distribution is approximated by a Gaussian distribution to avoid high communication costs and make the algorithm feasible.\n\nThe authors also proclaim that this process preserves privacy.", "summary_of_the_review": "The paper has some new and interesting ideas, but the way it was written makes it hard to follow, also the theory needs some adjustments.\n In general, the idea is interesting and I would encourage the authors to polish the paper, make it more clear, and resubmit. ", "strength_and_weaknesses": "Strengths: the idea of decoupling features is not 100% new, but sharing the low-level features in the fashion proposed in the paper in addition to approximating the low-level feature distribution is a nice idea. Also, the experiments (I haven't gone through all the details of them) seem thorough, and most importantly they show that this method works better than other already-known methods.\n\nWeaknesses: The writing of the paper is not clear sometimes, and some statements are just wrong, for example, the remark after Th 4.1 the authors say that Th4.1 implies \"the generalization contribution of the training distribution is\nexpected to be large on the training set\", this is not implied by Th4.1 you simply expect from any good algorithm to perform well on data similar to what it was trained on, the second implication is also not implied by Th4.1.  The notations are hard to follow (the Gaussian distribution mean is denoted by sigma and the variance by mu is an example, another example is $\\zeta,y$ I would say $\\zeta = x,y$ but you keep using both, so in the distance do you include the label or it is just the distance between inputs). Also in theorem 4.1, I  would expect the learning algorithm to depend on $\\tilde{D}_m$ not on $D_m$, because if I understood correctly, you will be training on the sampled distribution $\\tilde{D}_m$ (it would have been better to introduce this before the theorem and explain exactly what you are doing: we sample $\\tilde{D}_m$ from $D_m$ and use it for training and so on...). One last thing is in the definition of the worst-case margin, you should explain why this would be a good measure of generalization (I would say you can make a link with SVMs, but you should explain it).\n\nAlso, the theory, in my opinion, is not very strong (maybe because of the notation, but there is also $\\tilde{D}_m$ and $D_m$). you are also proving one thing and using a different thing in the experiments (the Gaussian approximation is not included in your theory). Also, you proclaim that your approach preserves privacy but you don't prove it (because there can still be information leakages)\n\nOne final point about the theory: there is domain adaptation that has similar goals to the generalization part in your paper (training on one dataset and testing on another), so I think it will be useful for you to take a look at it. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WfyspMtSZ_", "reviewer": "Reviewer_fFS7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the generalization contribution of local training in Federated learning. Their results show the key to promote generalization (conditional Wasserstein distance) and therefore this paper proposes decoupling the deep models for harnessing client drift and protecting privacy. Experimental results show FL with decoupled gradient dissimilarity is robust to data heterogeneity.\n\n", "summary_of_the_review": "Overall, I think that the studied local training generalization contribution in this paper is well-motivated. The idea behind the proposed algorithm is novel and effective.  Therefore, I recommend “accept”.", "strength_and_weaknesses": "\nStrength: This paper is well written and the topic of client drift is important and interesting. The theoretical result of local training generalization is sound and answers the question related to local training. \n\nWeakness: The communication cost should be reduced (e.g., quantizations) (although authors talked in the limitations.)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AtZNUQOFhRb", "reviewer": "Reviewer_eaTq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "One of the key challenges in learning from private data sources in a collaborative manner as in federated learning is the data heterogeneity (non-IIDness) among data shards. This causes drift among local models which slows down the convergences or entails a poor generalization on local distributions. This issue is further exacerbated in the presence of infrequent synchronization which is widely used in resource-limited scenarios such as low communication bandwidth to reduce the number of communication rounds. This led to an enormous amount of efforts such as clustered learning, personalization, or explicit drift reduction using control variates or regularization (often dynamic) techniques. \n\nThis paper focuses on improving the generalization by proposing an effective gradient correction method. To do so, the authors view local training as a generalization contribution problem, how local training of a specific client contributes to the generalization performance of the global model with respect to distribution of other clients. This can provide a mechanism to understand the contribution of each client at each communication round.  The authors quantify this by the conditional Wasserstein distance between clients’ distributions. The authors propose a decoupling idea to jointly learn  a low-level model (a feature extractor network) and a high-level one (head or classifier)to construct a shared identical distribution in the feature space. From a theoretical standpoint, the authors show that the effect of non-IID ness (gradient dissimilarity) on generalization and convergence can be decoupled based on two distributions: features and labels. Then it is shown that estimating and sharing feature distributions improves the generalization by  reducing gradient dissimilarity.\n", "summary_of_the_review": "Overall, the paper studies an interesting question, proposes a novel metric to quantify in-training generalization, but IMHO is poorly executed and  the quality of exposition seems to be low which makes me lean toward rejection.\n", "strength_and_weaknesses": "To understand the contribution of each client at each communication round, the basic idea is to  generalize the margin based discrepancy to incorporate the gradient (pseudo gradient between two consecutive communication rounds for each client to be precise) as defined in Eq. 2. Lower bounding this error can guarantee the update will benefit the other clients if the drift is properly mitigated. The analysis leads to the algorithmic contribution of the paper where authors propose to learn/iteratively update a common feature distribution (Guassian in their case) among all clients.\n\nThe paper studies with an interesting question, and proposes an interesting in-training metric that can help to quantify and guide the algorithm to mitigate data heterogeneity issues. However, in my opinion the proposed algorithm and execution of ideas suffers from serious issues that prevents me from giving it a high score:\n\n1. Data heterogeneity and client drift might exist due to numerous issues: covariate shift,  model shift, or both. It seems learning a shared model for features can handle covariate shift, but it might fail in the presence of model shift. For example, consider the case that data at each client comes from a Gaussian distribution with mean 0 and covariance \\Sigma (shared among all clients), but a different regressor \\theta_i. In this case I am not sure how drift among high level parts can be mitigated. Note that the original metric considers discrepancy in both covariates and labels. I might be mistaken mostly due to the fact that authors use the same notation \\mathcal{D} when referring to both data distribution (feature, label) pair and solely feature distribution and further clarification would be very helpful. \n\n2. It is not clear how the proposed algorithm can improve the generalization theoretically.  While the decoupling idea is inspired from definition of conditional Wasserstein and decomposition of inter-client gradient dissimilarity into low and high level parts, but it would be nice to see how the iterative adaptation of these parameters and heuristics used to make the algorithm practical (such as Guassian) can indeed entail a good generalization. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject