Acceleration of Federated Learning with Alleviated Forgetting in Local Training
Authors: Chencheng Xu
Zhiwei Hong
Minlie Huang
Tao Jiang
Venue: ICLR 2022 Poster (2022)
Keywords: Federated learning
non-i.i.d. data

Abstract: Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy by independently training local models on each client and then aggregating parameters on a central server, thereby producing an effective global model. Although a variety of FL algorithms have been proposed, their training efficiency remains low when the data are not independently and identically distributed (non-i.i.d.) across different clients. We observe that the slow convergence rates of the existing methods are (at least partially) caused by the catastrophic forgetting issue during the local training stage on each individual client, which leads to a large increase in the loss function concerning the previous training data provided at other clients. Here, we propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage by regularizing locally trained parameters with the loss on generated pseudo data, which encode the knowledge of previous training data learned by the global model. Our comprehensive experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep and the clients' data are extremely non-i.i.d., but is also able to protect privacy better in classification problems and more robust against gradient inversion attacks.

Published as a conference paper at ICLR 2022
ACCELERATION OF FEDERATED LEARNING WITH ALLEVIATED FORGETTING IN LOCAL TRAINING
Chencheng Xu, 1.2 Zhiwei Hong, 1.2 Minlie Huang, 1.2- Tao Jiang
BNRIST, Tsinghua University, Beijing 100084, China
DDepartment of Computer Science and Technology, Tsinghua University, Beijing 100084, China 'Department of Computer Science and Engineering, UCR, CA 92521, USA (xucc18, 18, hzw1 7 @ma ils s. tsinghua, edu. a aihuang@te nigua edu. cn, jiangê cs wur. edu
ABSTRACT
Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy by independently training local models on each client and then aggregating parameters on central server, thereby producing an effective global model. Although a variety of FL. algorithms have been pro- posed, their training efficiency remains low when the data are not independently and identically distributed (non-i.i.d.) across different clients. We observe that the slow convergence rates of the existing methods are (at least partially) caused by the catastrophic forgetting issue during the local training stage on each in- dividual client, which leads to large increase in the loss function concern- ing the previous training data at the other clients. Here, we propose FedReg, algorithm accelerate FL with alleviated knowledge forgetting in the lo- cal training stage by regularizing locally trained parameters with the loss on generated pseudo data, which encode the knowledge of previous training data Iearned by the global model. Our comprehensive experiments demonstrate that FedReg not only significantly improves the convergence rate of FI., especially when the neural network architecture is deep and the clients' data are extremely non-i.i.d., but is also able to protect privacy better in classification problems and more robust against gradient inversion attacks. The code is available at: com http:///tttttt.oooommooooottt//d
INTRODUCTION
Federated learning (FL) has emerged as a paradigm to train a global machine learning model in distributed manner while taking privacy concerns and data protection regulations into consideration by keeping data on clients (Voigt & Von dem Bussche 2017). The main challenge faced in FL is how to reduce the communication costs (in training) without degrading the performance of the final resultant model, especially when the data different clients are not independently and identically distributed (non-i.i.d.) (Yuan & Ma 2020 Wang et al., 2020b). The most popular FL algorithm FedAvg (McMahan et al., 2017a). In each training round of FedAvg, local training steps are sepa- rately performed at every client and the locally trained models are transferred to the server. Then, the server aggregates the local models into global model by simply averaging their parameters. Although FedAvg succeeds in many applications, its training processes often diverge when the data non-i.i.d., also known heterogeneous, across the clients (Li et al. 2019 Zhao et al.) 2018). Several FL algorithms have been designed to improve FedAvg and tackle the heterogeneity issue mainly by reducing the difference between locally trained parameters (Li et al.,
2018 Karimireddy et al. 2020) or aggregating these parameters into different groups (Wang et al. [2000a] Yurochkin
al. 2019). However, the performance of these methods is still far from satisfactory when deep
neural network architecture is employed (Rothchild et al. 2020; Li et al. (021). On the other hand,
recent work in the literature (Geiping et | 20200) shows that the transmission of trained model pa-
rameters does not ensure the protection of privacy. Privacy-sensitive information can be recovered
Minlie Huang and Tao Jiang are the co-corresponding authors.

---

Published as a conference paper at ICLR 2022
Server (glohal meseel)
Clert E (Doca model)
8P-11
Unde gitl Day (
gio gio glo
2x(0
Af right CI -aathr
e
girll
P
e
Figure 1: The catastrophic forgetting issues non-ii.d MNIST (a) and EMNIST (b) data, and an illustration of FedReg (c). Here, loss(t-1) denotes the loss data Dy which is the local data of client j sampled in round 1, computed with parameters ((+-1), and loss denotes the averaged loss on the same data computed with the locally trained parameters in round t. See Appendix B.1 for their detailed definitions. The values loss(t) are significantly larger than those of (oss(t-1), indicating that after the local training stage, the knowledge about the training data in the previous round at the other clients has been forgotten. To accelerate the convergence rate of FL by alleviating such forgetting issue in the local training stage, FedReg generates pseudo data TD and perturbed data DD and regularizes the local parameter (t+.s) by constraining the loss on DD and D. Therefore, instead of using the plain gradient with learning rate ne. FedReg updates ((t.3) with the gradient 9- (D²) computed with updated parameters a(5,), which prevents the gradient from converg- ing to zero in few steps, and then it regularizes ((t.i) with combination of the gradients go and go (DD?) computed from the pseudo and perturbed data.
with gradient inversion attacks (Zhu et al. 2019). Differential privacy (DP) (McMahan et al. 2017b] Abadi et al. 2016) is one of the most widely used strategies to prevent the leakage of private infor- mation. However, when DP is incorporated into FL., the performance of the resulting model decays
significantly (Jayaraman & Evans 2019).
We observe that when the data are non-i..d. across the clients, the locally trained models suffer from severe forgetting of the knowledge of previous training data at the other clients (i.e., the well-known catastrophic forgetting issue), perhaps due to the discrepancy between local data distributions and the global data distribution. As shown in Figure [ and supplementary Figure C.I] this forgetting issue Icads to a large increase in the loss concerning these training data under the non-i.i.d set- ting, thereby slowing down the convergence speed. In this work, we propose a novel algorithm, FedReg, that reduces the communication costs in training by alleviating the catastrophic forgetting issue in the local training stage. FedReg reduces knowledge forgetting by regularizing the locally trained parameters with generated pseudo data, which are obtained by using modified local data to encode the knowledge previous training data as learned by the global model. The potential con- flict with the knowledge in the local data introduced by the pseudo data is dampened by the use of perturbed data, which are generated by making small perturbations to the local data, whose predic- tive values they help ensure. The generation of the pseudo data and perturhed data only relies on the global model received from the server and the local data on the current client. Thus, compared with FedAvg, no extra communication costs concerning the data on the other clients are incurred. Our extensive experiments demonstrate the superiority of FedReg in accelerating the FL training process, especially when the neural network architecture is deep and the clients' data are extremely heterogeneous. Furthermore, the pseudo data can be further utilized in classification problems to defend against gradient inversion attacks. More specifically, we show that with similar degree of protection of private information, the degradation in the performance of the global model learned by our method is much less than that learned by using FedAvg combined with DP.
Our contributions. We demonstrate that when the data across clients are non-i.i.d., catastrophic forgetting in the local training stage is an important factor that slows down the FL training process. We therefore propose novel algorithm, FedReg, that accelerates FL by alleviating catastrophic forgetting with generated pseudo data. We also perform extensive experiments to establish the su- periority of FedReg. We further show that in classification problems, the pseudo data generated in FedReg can help protect private information from gradient inversion attacks with much less impact on the performance of the resultant model compared with DP.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
2.1 FEDERATED LEARNING
FL is proposed to address privacy concerns in a distributed learning environment. In the FL paradigm, a client i keeps its local data DD; on the client machine during the whole training process so that the server and other clients cannot directly access D, The objective to find the parameter 0" that minimizes the loss on the global data Uie where C is the set of clients, i.e.,
0" arg min
Σ La (d),
Eiec [T.
dEvieeD.
In this equation, Lg is the loss function with parameters O, which can be cross-entropy or in some other custom-defined form. FedAvg (McMahan et al.) (017a) and FedProx (Li et al, (018) are the most widely used FL algorithms. In FedAvg, in a training round t, the server sends the initial pa- rameters (((-1) to a set of sampled clients C(O. Then, the parameters are updated independently for S epochs on each of these clients to minimize the loss on the local data, and the locally trained parameters ((,,)) are then sent to the server. The server aggregates the parameters by simply averag- ing over them and obtains the parameters ((+,, i.e. O(1) k Eiegres Ott.s) where K is the number of sampled clients in round t. FedAvg is unstable and diverge when the data are non-i..dd across different clients (Li et al. 2018).. FedProx stabilizes FedAvg by including a proximal term in the loss function limit the distance between (1.33) and ((t-1), Although FedProx provides a theoretical proof of convergence, empirically it fails to achieve good performances when deep neural network architectures are used (Li et al. 202T). SCAFFOLD (Karimireddy et al. 2020) assumes that the heterogeneity of data leads to a client-drift in gradients and correlates the drift with gradient correc- tion terms. As clients need to send extra information to the server, SCAFFOLD increases the risk of privacy leakage and doubles the communication burden compared with FedAvg. Furthermore, the accuracy of the correction terms highly correlated with the training history of the client. Thus, the performance of SCAFFOLD decreases significantly when the number of clients is large, in which case each client is only sampled for few times and the estimation of the gradient correction terms is often inaccurate. FedCurv (Shoham et al. (219)) aims to tackle the forgetting issuc on non-i.idd data with elastic weight consolidation (Kirkpatrick et al. 2017). To achieve this, FedCurv needs to transfer Fisher information between the server and clients, which increases the communication costs to 2.5 times compared with FedAvg. Multiple methods (Luo et al. 2021 Hao et al. 202 Goetz & Tewari 2020) have also tried to introduce synthesized data to help reduce the effect of heterogeneity, but they either rely on some specific architectures of neural networks (Luo et al.
2021) such as batch
normalization (Toffe & Szegedy, 2015) or require the synthesized data to be shared with the server & Tewari 2020), which contradicts the privacy protection objectives of FL..
Hao et al.
2.2 CATASTROPHIC FORGETTING
Catastrophic forgetting occurs specifically when the neural network is trained sequentially on mul- tiple tasks. In this case, the optimal parameters for the current task might perform poorly the objectives of previous tasks. Many algorithms have been proposed to alleviate the forgetting issue. Memory-based methods (Parisi et al., 2019) have achieved excellent performances in accommodat- ing new knowledge while retaining previously learned experience. Such memory-based methods, such as gradient episodic memory (GEM) (Lopez-Paz & Ranzato 2017) and averaged gradient episodic memory (A-GEM) (Chaudhry et al. 2018), store a subset of data from previous tasks and replay the memorized data when training on the current task. For instance, A-GEM treats the losses on the episodic memories of previous tasks as inequality constraints in optimizing the objectives of current tasks and changes the model updates from the plain gradient g to g wgref, where gref is the gradient computed from the loss on the memorized data and w is a non-negative weight. Unfortunately, these memory-base -based techniques are not suitable for FL. due to privacy concerns.
2.3 GRADIENT INVERSION ATTACKS
FL provides privacy guarantee by keeping the users' data individual client machines locally and only sharing the model updates. However, recent research has shown that data information can be recovered from the parameter updates (Geiping et al. 2020 Zhu al. 2019] in the FedAvg
3

---

Published as a conference paper at ICLR 2022
framework by simply finding data with updates similar to the values returned from the client. DP, which avoids privacy leakage by adding noise to training data (Sun et al. 2019) or model updates (Abadi et al.] 2016; McMahan et all, 2017b), is the most widely used strategy to protect private information. When adding noise to model updates, such as differentially private SGD (DPSGD) (Abadi al. 2016), the noise level in DP is controlled by the gradient norm bound C and the noise scale c.. DP often causes a large performance degradation in the resultant model.
3
METHOD
Our main challenge is how to alleviate the forgetting of previously learned knowledge at each client without having to access data the other clients in the local training stage. For any data set ID, we de- note the set of data near D within the Euclidean distance of ó as (D, 8) = (m.)ee)) ( y 10 < || a²'| I 63. We assume that in round & on client i, (1) for all data d= E U/ec/(1) Dy lo- cal to the other clients, the global model with parameter (((-1) has better feature representa- tion than the local model with parameter ((t,3); and (2) for any C. 0. 3e,8 0 such that if vd' = y') E N (TD;, 6), Lou.o ((x', Sor. 1) (ax)) Logc ((x', fou (x"))) & 6, then vd (x= " (1 Dj. Ligt.a : 13 ()(x))))))))
Ligie 1) ((x foa 1) (a) S c, where fe is the prediction function with parameters O. The assumption (2) guaranteess that, in the local training stage, the range of changes in the predicted values of previous training data at the other clients can be controlled by constraining the changes in the predicted values of the data near D;. Based on these assumptions, we first generate pseudo data and then alleviate the catastrophic forgetting issue by regularizing the locally trained parameters with the loss on the pseudo data. In other words, we hope that the pseudo data would achieve the same effect as the previous training data in constraining the local model. Note that FedReg does not assume any particular form of the loss function, but the method of modifying gradients to enhance privacy protection (to be discussed below in Section 3.3) is currently only applicable to classification problems. The pseudo code of FedReg is provided in Appendix D
3.1 GENERATION OF PSEUDO DATA AND PERTURBED DATA
With the above assumptions, the catastrophic forgetting issue can be alleviated by constraining the prediction values for data in N(D, 8). However, such a constraint is too strong and would hinder the learning of new knowledge from the local data, Moreover, it is also computationally inefficient to enumerate all such data. Observe that for data point d' = (a', y') E N (D;, 8). after locally training g(t.i) with D. if fec... (x')= faa-1) 1) (x') then the constraint on d' will not contribute to addressing the forgetting issue. Therefore, to efficiently regularize the locally trained parameters, only data points with large changes in the predicted values should be used to alleviate knowledge forgetting. Such data points are usually within a small distance to the data points in D; but their predicted values given by ((t-1) could be very different from the labels in D, Data satisfying these conditions can be casily obtained with the fast gradient sign method (Goodfellow et al. 2014)) and we denote the data generated in this way as pscudo data D; More precisely, each data point (x= y²) Do is iteratively generated from data point (æ. v) E DD; as follows:
xx xe y² Socc-1 (x*),
x 3==x" 2 + sesign (V (o Lov-1) ((0)
1, E. x x.
where 7) is the step size and E the number of steps. Despite the relaxation in constraints achieved by the pseudo data generated above, some information conflicting with the knowledge embedded in the local data Dj could possibly be introduced in Dy due to the inaccuracy of the global model during the training process. To further eliminate such conflicting information, perturbed data DD/ = {(x", } with small perturbations on DD; are iteratively generated as follows:
x= = x 1 y. n..- 27 + signnn (Væj Loue-1 ((z)- j= 1,..,,
..... E, a. x,
(3)
where the step size Thp satisfies 7p < 7s in order to make sure that the perturbed data is much closer to the local data than the pseudo data. In our experiments, we take 7p = 0.011, and E = 10 to reduce the complexity of tuning hyper-parameters.
4

---

Published as a conference paper at ICLR 2022
3.2 ALLEVIATION OF CATASTROPHIC FORGETTING
With the pseudo data ! and perturbed data Dy generated above, we regularize ((+,8) by requiring
Σ d' EDDy
Teuta (d)) S
Σ
Lggi (d")
(4)
d'ED;
Louas (dP) v dPED?
Σ
Σ
Loga (dP)
(5)
dRED?
where constraint alleviates the catastrophic forgetting issue and constraint ( eliminates con- flicting information introduced in 4 Constraint (5) also helps improve the robustness of the resultant model (Madry et al.) 2018). Expanding both sides of (4) and of (5) respectively with 3 0.5 (o(t.)) + 0(t-1) that the second-order term can be eliminated and ignoring higher- order terms (see Appendix A. for the details), we obtain
(o (o(-11)-0(t))) - T
(D;) 2 O, gs (ID;) = V
L oafe (d))
(6)
D d'e
T (0l1-1) -((.)))
93 0, g8 (D))
Σ d' dPED?
L Lgggss (dP)
Due to the existence of potentially conflicting information in D and DDP directly finding ((.i)) to satisfy the above two inequalities is mathematically ill-defined. Besides, given the complex archi- tectures of deep neural networks, is computationally too challenging to solve either of the above inequalities. Therefore, we approximate the optimal parameters (t.,s)* by solving the following constrained optimization problems:
@it,0' arg ming 0 - o(c.0)1 s.l. (0(--1) -0) 0)* T
ga (D 2 0,
(8)
((e.1)* arg ming ||0 010.0112 s.l. (0(t-1) 0) g (D.) 2 0.
By solving 8 and (see Appendix A.2 for the detailed derivation), we obtain
(t,a)* = (t.i) wa9a (IDF) (Ppga(D?))
(10)
(0 @ ga(D)) go (DF) g (D:)
0) Wp = max (e -1:i)-ww.p w.. 8l 11)" ga 9a(2)))
W. = max
go (D.)) go (T?)
Based on the generation process of the pseudo data, it is easy to note that Ea (d = ming Edr Lo(d"), and thus the inequality sign in 4 can in fact be replaced by an equality sign. In practice, however, we have observed that the traimig process would be more stable and result in non-negative and wp with the inequality formulation, probably due to errors introduced in the above approximation. Finally, to prevent the gradient from converging to zero in a very small
number of steps, slowly-updated parameters (Zhang et al. 2019b) e((.1) (t.0) + (1 - (t-1) with 7 [0, 1] are used compute the gradient for D. i.e., g, Vogio p. Eden, Lo (d).
In summary, taking into acount the above improvements, the local parameters are updated in each training step as:
(t.i) (t.i)) Tlog> (ID;), (t.i +- (t.i) w.gs (ID;) wpga (D)))
(11)
3.3 MODIFICATION OF GRADIENTS TO PROTECT PRIVACY
We further notice that in classification problems, the pseudo data be used to modify the gradient g (I,) to enhance the protection of privacy. If we denote DY {(x, 1 Σ (Z))[(z) y) e IDF) for an r-claasiffcation problem, where eli) is the standard basis vector with a at position j. then the data points in IDy are similar those in D. but they may have totally different labels. Hence,
5

---

Published as a conference paper at ICLR 2022
it is reasonable to assume that 9, (D;) and 9,, (D,") contain similar semantic information but dif- ferent classification information. Since in the training of FL, only the classification information contributes to the model performance, removing the semantic information in 9, (D;) will enhance the privacy protection capability of FL. without severely degrading the performance of the resultant model. Based on this intuition, we presume that the components of g (D²) in the same (or roughly the same) direction of 9-, (II)") contain the semantic information and the other (orthogonal) compo- nents contain the classification information. Thus, we compute a modified gradient (MG) to enhance the protection of privacy as
9y (ID.)²
9y (Dz')
(D,) 9, (D:) vg, (D.')
(12)
9, (Dz') 9y (TD;')
We refer to the version of FedReg in which the g, (ID)) term with MG.
11 replaced by 9 (ID,) as FedReg
4 EXPERIMENTS
We compare the performance of FedFeg against the above-mentioned baseline FL algorithms in- cluding FedAvg, FedProx, FedCury and SCAFFOLD as well as the well-known SGD algorithm (Bottou 2012). Note that SGD corresponds to special case of FedAvg where the local training stage consists of only one step and all data on a client are used in large single batch.
4.1 DATA PREPARATION
FedReg is evaluated on MNIST Deng 2012). EMNIST (Cohen et al. 2017). CIFAR-10 10 and CIFAR-
100 (Krizhevsky al. 2009), and CT images of COVID-19 (He 2020p. To simulate scenario for FL, the training data in each dataset are split into multiple clients in different ways and the performance of the trained model is evaluated the test data. The data preparation steps for each
dataset are described below and more experimental details are provided in Appendix B
The training data are split into 5,000 clients. Each client has images either from only one class, named MNIST (one class), or from two classes, named MNIST (two classes). The number of images per client follows a power law distribution (Li al. 2018). A 5-layer neural network with three layers of convolutional neural networks (CNN) and two layers of full connections (FC) is used.
MNIST
EMNIST The training data of the digits in EMNIST are split into 10,000 clients. Each client owns 24 images from the same class. The same model architecture used on MNIST is used here.
CIFAR-10 The training data are split into 10,000 clients. Each client owns immggs from random classes, named CIFAR-10 (uniform). or from only one class, named CIFAR-10 (one class). A ResNet-9 network with the Fixup initialization (Zhang et al. (219a) is trained from scratch.
CIFAR-100 The training data are split into 50,000 clients and each client owns image. A ResNet- 9 network with the Fixup initialization trained from scratch, named CIFAR-100 (ResNet-9), and pre-trained transformer (Dosovitskiy et al. 2020) is fine-tuned, named CIFAR-100 (Transformer).
CT images related to C0VID-19 The dataset contains 349 CT images of COVID-19 patients with source information and 397 non-C Non-COVVD-19 CT images without source information. The data split into training and test data in the same way provided in (He) 2020). Then, the CT images of COVID-19 patients in the training data split to make sure that the images from the same source are assigned to the same client, and 32 clients are obtained in this way. The non-Co aon-COVID-19 images in the training data are distributed each client following the proportion of COVID-19 images. A DenseNet-12 121 network (Huang et al. 2017) and a 10-layer neural network with 9 layers of CNNs and one layer of FC are trained these CT images to distinguish the COVID-19 images from the non-COVID-19 ones.
4.2 COMPARISON OF CONVERGENCE RATES
The results on MNIST and EMNIST are shown in Table [ FedReg required fewer communication rounds to converge compared with the baseline methods and achieved a higher final accuracy (i.e.,

---

Published as a conference paper at ICLR 2022
Table 1: Comparison results MNIST and EMNIST. The ACC columns show the final accuracy on test data. The Rs columns show the minimum number of rounds required to reach 100% of the accuracy of SGD in the corresponding experiment.
MNIST (two classes)
MNIST (one class)
EMNIST
Method
Ro.5
Ro.9
R1.0
ACC
Ro.5
Ro.9
R1.0
ACC
Ro.5
Ro.9
R1.0
ACC
SGD
22
81
478
0.975
33
95
422
0.976
74
160
470
0.982
FedAvg
5
37
491
0.975
28
74
0.976
118
170
466
0.984
FedProx
7
0.974
21
74
0.975
126
178
446
0.984
SCAFFOLD
7
47
0.957
29
73
0.930
149
221
0.967
FedCurv
7
0.974
29
74
0.976
117
170
432
0.985
FedReg
367
0.977
32
384
0.978
299
0.987
Table 2: Comparison results on CIFAR-10.
CIFAR-I0 (uniform)
CIFAR-10 (one class)
Method
Ro.5
Ro.9
R1
ACC
Ro.s
Ru.9
R1.0
ACC
SGD
40
160
185
0.483
39
168
197
0.449
FedAvg
52
94
0.576
108
0.371
FedProx
94
0.576
108
0.373
SCAFFOLD
7
116
0.447
88
0.230
FedCurv
94
0.569
108
0.368
FedReg
2
40
0.715
9
58
84
0.616
accuracy of the respective models after finishing all training rounds) under all the experimental set- tings, especially on EMNIST, where the number of clients is twice compared MNIST, showing clearly the superiority of FedReg in dealing with non-i.i.d. data when shallow model is used and its robust scalability. FedCurv also aims to allieviate the forgetting issue, its performance not significantly better than FedProx. Note that due to its heavy reliance on the optimization his- tory when estimating the correction term, in our experiments, SCAFFOLD performed worse than FedAvg. Table 2 illustrates that, CIFAR-10. 10, when the data are randomly distributed, FedReg significantly outperformed munication rounds) and final model accuracy, demonstrating the superiority of FedReg in training deep neural networks. When data across clients become more heterogeneous and the images at each client from the same class, the multiple local training steps in the baseline algorithms (other than SGD) failed to improve the performance of the locally trained model, resulting in a worse perfor- mance than that of SGD, while FedReg was still able save about half of the communication costs compared with SGD, A similar conclusion can be drawn when training the ResNet-9 network from scratch and fine-tuning the transformer model on CIFAR-100, as shown in Table 3 indicating the robustness of FedReg various datasets and model architectures. The results in distinguishing COVID-19 CT images from mon-COVID-19 images on DenseNet- 121 network, as shown in Ta- ble 3 further exhibits the superiority of FedReg in medical image processing applications when the model architecture deep and complex. To assess the advantage of FedReg on real-world data, is popular real-world dataset Landmarks-User 160k (Hsu al.) 2020 Lai et al.) 2021). where the data at each client consists of many classes, and as shown in sup- plementary Table C.T FedReg still outperformed the baseline methods. The wall-clock time usages of the methods each local training round on MNIST and EMNIST are given in supplementary Table C.2 Note that since FedReg performs additional computation each round, its local training requires bit more time.
the baseline methods both convergence speed (measured com-
compared with the other methods
4.3 ALLEVIATION OF CATASTROPHIC FORGETTING
To prove that FedReg indeed alleviates the catastrophic forgetting, increases of the loss values con- cerning previous training data the other clients compared between FedReg and FedAvg. As

---

Published as a conference paper at ICLR 2022
Table 3: Comparison results CIFAR-100 and CT images related to COVID-19.
CIFAR-100 (ResNet-9)
CIFAR- 100 (Transformer)
COVID-19 CT Images
Method
Ro.5
Ro.9
Ri.0
ACC
Ro.5
Ra.3
R1.0
ACC
R1.0
ACC
SGD
510
1004
1200
0.434
4
16
74
0.888
4
0.511
FedAvg
793
0.323
3
0.883
0,614
FedProx
793
0.325
3
26
0.880
4
0.620
SCAFFOLD
6
0,598
FedCurv
774
0.316
3
18
0.882
0.614
FedReg
248
612
796
0.502
2
14
53
0.898
0.673
shown in Figure 2 the increases of the loss significantly lower in FedReg than those in FedAvg, suggesting that although FedAvg and FedReg both forget some of the learned knowledge, the for- getting issue is less severe in FedReg. To further explore the role of the generated pseudo data in the alleviation of forgetting, the values of the empirical Fisher information (Ly et al. 2017) in the pseudo data and previous training data are compared. As shown in Figure the values of the pseudo data and previous training data are significantly correlated. Following the Laplace approximation (MacKay 1992, Kirkpatrick al., 2017). if we approximate the posterior distribution of the opti- dataset by normal distribution centered at ((t-1) with diagonal precision matrix, then the precision matrix can be approximated by the empirical Fisher information. The similarity in the empirical Fisher information suggests that a model with high performance on the pseudo data has a relatively high probability to perform well on previous training data. Therefore, regularizing the parameters with the pseudo data can help reduce the performance degradation on previous training data and alleviate the knowledge forgetting issue.
mal parameters
MNISTT (two clarooo))
MNNST (one (uneellas)) class)
EMNIST
Figure 2: Increase of the loss (top row) concerning previous training data and distribution of the correlation (bottom row) between the empirical Fisher information in the pseudo data and that in the previous training data at the other clients.
4.4 THE IMPACT OF STEP SIZE IN GENERATING PSEUDO DATA
The impact of the hyper-parameter T) which determines the distance between the pseudo data and the local data, is explored here. Intuitively, when is too large, the distance between the generated pseudo data and the local data will be too large to effectively regularize the model parameters. On the other hand, when 7), is too small, the conflicting information from inaccurate global model will slow down the convergence speed, This intuition is empirically confirmed by the experiments presented in supplementary Figure C.2 On the EMNIST and CIFAR-10 10 (one class) datasets, when m, is too small, takes more communication rounds to achieve the same accuracy. As 7) increases, the number of communication rounds decreases minimum and then bounces back, indicating that a less effective regularization on the model parameters in the local training stage has been reached.
8

---

Published as a conference paper at ICLR 2022
4.5 PROTECTION OF PRIVACY
The protection of private information is one of the most important concerns in the FL paradigm, especially medical applications. To illustrate the privacy security of FedReg with MG, gradient inversion attacks used to recover information from updated parameters in classification prob- lems. Under a comparable degree of privacy protection, the resultant model performance is com- pared between FedReg with MG and the baseline methods with DPSGD (Abadi et al. 2016). Using gradient inversion attacks, the quality of images recovered from the updated parameters a local step is examined. Note that when only one local training step is performed in each round, the up- dated parameters in FedAvg, FedProx, FedCurv, and SGD are all the same, leading to the same images recovered from these methods. We do not include SCAFFOLD here since its performance was significantly inferior to the other methods in the above experiments. As shown in supplemen- tary Figures C.3]and [..] for some images sampled from the EMNIST, CIFAR-10 10 and COVID-19 CT datasets, the quality of the images recovered from FedReg with MG is significantly worse than those recovered from the baseline methods, exhibiting better privacy protection capability of Fe- dReg with MG. To compare the impact of privacy protection on model performance, we tune the noise level in DPSGD to generate images with similar quality as measured by PSNR (Hore Ziou, 2010) and compare the performance of the resultant models. The results are shown in Table It can be seen that the protection of private information with DP costs a large performance degradation in the baseline methods, while the performance of FedReg with MG is degraded much less when main- taining a similar privacy protection level. Note that the CT images used in the above experiments contain some privacy-sensitive information, including the date of birth of the patient and the date when the image was taken. In the basic FedAvg, when a 10-layer neural network with 9 layers of CNNs and one layer of FC is used, such information can be easily recovered with gradient inversion attacks. When DP is applied, although such time information could be protected, the resultant model suffers from severe performance degradation, as shown in Table In contrast, FedReg with MG is able to protect the sensitive time information with only mild model performance decay.
Table 4: Performance comparison between the resultant models when DPSGD is applied to the baseline methods and when MG is employed in FedReg. In each case, the final accuracy on the respective test data is reported.
Method
EMNIST
CIFAR-10 (uniform)
CIFAR-10 (one class)
CT images of COVID-19
DPSGD
0.932
0.285
0.233
0.533
FedAvg with DPSGD
0.882
0.391
0.157
0.570
FedProx with DPSGD
0,881
0.392
0.156
0.570
FedCurv with DPSGD
0.880
0.396
0.179
0.586
FedReg with MG
0.986
0.703
0.599
0.657
5 CONCLUSIONS AND FUTURE WORK
In this work, proposed novel algorithm, FedReg, to accelerate the convergence speed of FL by alleviating the catastrophic forgetting issue in the local training stage. Pseudo data are generated to carry knowledge about previous training data learned by the global model without incurring extra communication costs or accessing data provided at the other clients. Our extensive experiments show that the generated pseudo data contain similar Fisher information as the previous training data at the other clients, and thus the forgetting issue can be alleviated by regularizing the locally trained parameters with the pseudo data. The pseudo data can also be used to defend against gradi- ent inversion attacks in classification problems with only mild performance decay on the resultant model compared with DP. Although FedReg exhibits competitive performance in accelerating con- vergence speeds, its performance is influenced by hyper-parameters. Automatic tuning of the hyper- parameters could make FedReg easier to use. Meanwhile, the question of how to modify gradients in FedReg to effectively protect privacy in regression problems remains to be explored. Moreover, rigorous proof of the convergence rate of FedReg is of theoretical interest. We leave these questions to future work.

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENT
This work has been supported in part by the US National Institute of Health grant IROINS the National Key Research and Development Program of China grant 2018YFC0910404 and the Guoqiang Institute of Tsinghua University with grant no. 2019GQG1.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Advances in Neural Information Processing Systems, 34, 2021.
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pp. 421- 436. Springer, 2012.
Arslan Chaudhry, Marc' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (LJCNN), pp.
2921-2926. IEEE, 2017.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-143) 2012.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2XXV:20110.11929 2020.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller Inverting gradients- how easy it to break privacy in federated learning? arXiv preprint arXiv:2003. 10053, 2020.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint
arXiv:2008.04453 04489, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Weituo Hao, Mostafa El-Khamy. Jungwon Lee, Jianyi Zhang, Kevin Liang, Changyou Chen, and Lawrence Carin Duke. Towards fair federated learning with zero-shot data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3310-3319, 2021.
X He. Sample-efficient deep learning for covid-19 diagnosis based on scans. IEEE transactions on medical imaging, 2020.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2369 IEEE, 2010.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real-world data distribution. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pp. 76-92. Springer, 2020.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinherger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 PMLR, 2015.
10

---

Published as a conference paper at ICLR 2022
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th {USENIX} Security Symposium ((USENIX) Security 19). pp. 1895-1912, 1912, 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXivv:412.69900 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3511-3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical fisher approx- imation for natural gradient descent. Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 4156-4167, 2019.
Fan Lai, Yinwei Dai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Fedscale: Benchmarking model and system performance of federated learning. In Proceedings the First Workshop on Systems Challenges in Reliable and Secure Federated Learning, pp. 1-3,2021 2021.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. arXiv preprint arXiv:2103.16257, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv: 2018.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv: 2019.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6470-6479, 2017.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data, arXiv preprint arXiv:2106.05001, 2021.
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagemmak- ers. A tutorial on fisher information. Journal of Mathematical Psychology. 80:40-55, 2017.
David JC MacKay, practical bayesian framework for backpropagation networks. Neural compu- lution, 4(3):448-477 1992.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eeficcne learning deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282. PMLR. 2017a.
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint /7110:/710.0660 2017b.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.

---

Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Published as a conference paper at ICLR 2022
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication--fficieee federated learning with sketching. In International Conference Machine Learning, pp. 8253-8263 PMLR, 2020.
Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor- MoseY.sef, and Itai Zeitak. Overcoming forgetting in federated learning on non-iid data. arXiv preprint arXiv:1910.07720 19.0999....09..... 2019.
Zongkun Sun, Yinglong Wang, Minglei Shu, Ruixia Liu, and Huiqi Zhao. Differential privacy for data and model publishing of medical data. IEEE Access, 7:152103-152114, 2019.
Paul Voigt and Axel Von dem Bussche. The general data protection regulation (gdpr). Practical Guide, Ist Ed., Cham: Springer International Publishing, 10:3152676, 2017.
Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris Papailiopoolos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint 2XX1::2002.064400 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems, 33, 2020b.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Neural Information Processing Systems, 33, 2020.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. Interna- tional Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint /9Xiv: 00321,, 2019a
Michael R Zhang. James Lucas, Geoffrey Hinton, and Jimmy Ba. Lookahead optimizer: k steps forward, step back. arXiv preprint arXiv:1907.08611 2019b.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, 2019.
12

---

Published as a conference paper at ICLR 2022
A
SUPPLEMENTARY METHOD
A.1 TAYLOR EXPANSION
Expanding both sides of the inequality (4) with 0² -0(t)) obtain
Σ d'EDy
Loue (d*) = EddEDD Togg.n (d*) + (B(w) 08.01)
o Edeen T.og.00 (d))
+ (ol.s) -06i))-f(11 - x 72 0
(Idrep Logio (dr)) (0l1.) - -
+0(105 - 113.
Σ Liutn (d') = EareDy Logeo0 (d)) + (g(1-1) -09t)) T d' dD
V Vogi. Edied; Log.0) (d)
+ (0it-1) -1(-11)-0(.0))
V2 0g (Edreny (d))) (ol--1)-0f
+0 (10 (1))
113
Substituting both sides of (4). dividing them by DDd and ignoring all high order terms, then only the first order terms are left and the thus inequality O T
is obtained, i.e.
(g - g(t..)
Logg (d") 0.
d'ED;
The inequality 7 can be derived from in similar way.
A.2 SOLVING THE CONSTRAINT OPTIMIZATION
The constraint optimization problems in method. More specifically, for the problem
8 and can be solved by the Lagrange multiplier
8 i.e.,
@it.i) arg ming 1|0 g(t.0)
s.t. (o - 0) go 2 0,
its Lagrangian can be written as
(( a)) |00 12 + a 0 ( T g (ID)),
where a 2 0. The dual problem
D(a) min L(O a).
0
Setting the derivatives of C(O, a) w.r.t 0 to zero, the value of ((t.)) minimizing [(0, a) is obtained
2 (olt,y- - ((t.)) + ags (ID)) = 0.
@(t,)) = (t.i)) aggs (D;).
Thus, D(x)) can be simplified as
D(a) =
4 a g: (I))) ge( + ((((t.)) - ((c-1))TT ga (ID)).
Solving a" arg max Daa). the value of a* is
2 (Q(t.))
o (t 1)) T
ga (DF)
max
0
go (D?) T go (D);)
Thus, the value of ((t,)) is
(0th))) - ( T
g (12)) ga (D:))* go (D))
0 G.ay (t..))
0) ge (ID)).
max
The constraint optimization problem 9 can be solved similarly.
13

---

Published as a conference paper at ICLR 2022
B EXPERIMENTAL DETAILS
In all the experiments, the learning rates for different methods are optimized by grid search, the optimal weight for the proximal term in FedProx is searched among {1.0, 0.1, 0.01, 0.001 the optimal À for FedCurv is searched among (0.001, 0.0001, 0.00001), and the and n,) in FedReg are optimized by grid search as well. The number of epochs in the local training stage is optimized in FedAvg and applied to the other methods.
(?
B.1 COMPUTATION OF loss AND (tss(t)
Formally, the value (osg(t-1) 1) for a client i E ((+-1) is computed as loss
1
1) (t) |D&
Lo
deD,
and the value of loss(!) is computed as loss
(9 1
Ejecio (Lotess
|C01|
dED,
where (t.i) is the parameters sent from client j in round t.
B.2 COMPARISON OF CONVERGENCE RATES
The hyper-parameters in the experiments are listed below:
Ten clients are sampled in each training round and 500 rounds are run in total with batch size of 10 that all training data are utilized once average. The learning rate in all methods is 0.1. On MNIST (two classes), in the local training stage, the local data are processed in 40 epochs. The weight for the proximal term in FedProx is 0.001 and À in FedCurv is 10-4 In FedReg. = 0.4 and 1/, = 0.2. On MNIST (one classes), the local data are processed in 20 epochs. The weight for the proximal term in FedProx is 0.01 and À in FedCurv is 10 In FedReg, = 0.3 and = 0.2.
MNIST.
EMNIST. 20 clients are sampled each training round, and in the local training stage, the local data are processed in 20 epochs with batch size of 24. 500 rounds are run in total so that all the training data are utilized once on average. The learning rate in SCAFFOLD is 0.1 and in other methods, it is 0.2. The weight for the proximal term in FedProx is 0.001 and in FedCury is 10-4. In FedReg, y = 0.4 and 1/s- = 0.05.
CIFAR-10. Following the experimental settings in (Rothchild et al. 2020). 240 rounds are run in total with 100 clients sampled in each training round, and in the local training stage, the batch size is 5. On CIFAR-IÓ (uniform), the local data are processed in 30 epochs. The learning rate is 0.05 in FedAvg, FedCurv and FedProx, 0.1 in SGD and FedReg, and 0.01 in SCAFFOLD respectively. The weight for the proximal term in FedProx is 0.001 and in FedCurv is 10 50- In FedReg, data are processed in 20 epochs. The learning rate is 0.05 in FedAvg, FedCurv, FedProx. SCAFFOLD and FedReg, and 0.1 in SGD, respectively. The weight for the proximal term in FedProx is 0.01 and À in FedCury is 10 In FedReg, = 0.5 and 1/s- = 0.1.
0.5 and 7/- 0.1. On CIFAR-10 (one class), the local
C000 When ResNet-9 is adopted, 1,200 rounds are run in total with 100 clients sampled in each training round, and in the local training stage, the local data are processed in 10 epochs with batch size of 1. The learning rate 0.05 in FedAvg, FedCurv and FedProx, and 0.1 in SGD and FedReg, respectively. The weight for the proximal term in FedProx is 0.01 and = 10-3 in FedCurv. In FedReg, ? = 0.25 and 17+ = 0.1. When the pre-trained transformer is adopted, 100 rounds are run in total with 500 clients sampled in each training round, and in the local training stage, the local data are processed in epochs with a batch size of 1. The learning rate is 0.1 in all methods. The weight for the proximal term in FedProx is 0.001 and À = 10-5 in FedCury. In FedReg, ? = 0.02 and n) 10 ? Note that SCAFFOLD is too reemory-consmminn 100 given our computational resources, the results of SCAFFOLD on CIFAR- 100 are not reported.
to
CIFAR-
14

---

Published as a conference paper at ICLR 2022
CT images related to COVID-19. In our experiments, ten rounds run in total and in each round, 10 clients are sampled. In the local training stage, the local data are processed in 20 epochs with a batch-size of 10. The learning rate is x 10 in FedAvg, FedCury and FedProx, 5 x 10-1 SCAFFOLD, x 10-1 in SGD, and 5 x 10-3 (0-3 in FedReg, respectively. The weight for the proximal term in FedProx is 1.0 and in FedCurv is 10 In FedReg,
? 0.5 and 77- = 10-6. Note that grou-normalizatioo (Wu & He 20018) is used in the DenseNet-121 network as the batch size small in the local training stage.
The training data in this dataset has 164,172 images of 2,028 landmarks from 1,262 users, and the test data contains 19,526 images. Each user has images of 29 classes on average. A DenseNet-121 network trained this dataset. 2,000 rounds are run in total, and in each round 100 clients are sampled. In the local training stage, the local data processed 40 epochs. The learning rate FedReg, and pFedGP (Achituve et al.) (02T). As pFedGP is designed for personalized federated learning scenarios and trains an independent GP-tree for each client, to apply to this dataset, small global dataset is built by sampling 2,028 images from all training data, which covers all the classes contained in Landmarks-Use -esearrk---------ee 160k. In each round, the deep kernel, which is the DenseNet- 121 network without the classification layer, is trained on each client and aggregated the server. Then, a global GP-tree is trained on the sampled global sub-dataset classify the images in the test data.
as 0.1 in SGD, FedAvg,
B.3 ALLEVIATION OF CATASTROPHIC FORGETTING
The hyper-parameters used here, except y in FedReg, are the same the values used in the previous section.
Computation of loss increment. To fairly compare the forgetting issue in FedReg and FedAvg, is set to be 1.0 in FedReg, and in each round, the client parameters are initialized with the parameters obtained in FedReg to avoid a large discrepancy between the values of losg(t-1) computed in FedReg and FedAvg. For client i E C(t-1), the loss increase is computed as (t) loss (t-)()
The computation of Fisher information. Following (unstner al. 2019), in round t of client E C(), for the previous training data (t- Vject-> Dj. where C(--)
U C(+), the empirical Fisher information is computed as
fishergass) (D(t-))-
Σ
(Veu.y log Pau.o (yl=)))
(a, (D))))DD
Similarly for the pseudo data Dj the empirical Fisher information is computed as
fi
(D
Σ
V Vos.n " log Poi.o (y² ()))="))²
D
(ar".y")) JENG
Note that considering the generation process of the pseudo data, the parameter values (t.,)) used to compute Fisher information are the same local model parameters after a local train- ing step.
B.4 PROTECTION OF PRIVATE INFORMATION
The target of gradient inversion attacks in (Geiping et al. 2020) and (Zhu et al. 2019) is to find
d' = arg min Distance (Ao (ole), denue) A0 (l)) .d) + TTV (d)
where AO (0()).) denotes the amounts (updates) that the parameters have changed from (()) with the corresponding data, TV (d) the total variation of d. and dirue a data point on the client. The distance function could be cosine distance (Geiping al. 2020) L2-distance (Zhu et al.) 2019). Note that the method proposed in (Zhu et al., 2019) sets w attacking experiments, the parameters are only trained for step with each data point, in
To consider the worst case, in
15

---

Published as a conference paper at ICLR 2022
which case the values of A (0(+), d) obtained in FedAvg, FedCurv, FedProx and SGD exactly the same. 32,000 attacking iterations are used in all experiments and both distance functions (i.e., cosine and L2) are considered. Adam (Kingma & Ba 2014) optimizer is used, which performs better than L--BGGS empirically. 'he learning rate and the weight for the total variation are optimized in (the basic) FedAvg and applied to the other methods. The learning rate decaying scheme follows the method provided in (Geiping et al. 2020).
More details concerning each dataset are discussed below.
The images to be recovered are randomly sampled. The model architecture and hyper-parameters used FedAvg, FedProx, FedCurv, SGD, and FedReg are the same the ones used in section B.2 In the baseline methods with DPSGD, C 1.0 and о = 0.01, and other nyper-parametens parameters are the same as the ones used in the corresponding baseline methods (without DPSGD). In FedReg with MG, the learning rate and the value of 7 are the same as the ones used in FedReg, and 7), = 0.03. In the attacking algorithm, the initial learning rate 1.0, the distance function L2-distance, and the weight for the total variation term is 10-
EMNIST.
The images to be recovered are randomly sampled. The model architecture and ayper-parameters used in FedAvg, FedProx, FedCurv, SGD, and FedReg the same as the ones used in section [82 In the baseline methods with DPSGD, C = 1.0 and a = 0.05, and the other hyper-parameters are the same the ones used in the corresponding baseline methods (without DPSGD). In FedReg with MG, the learning rate and the value of are the same the one used in FedReg, and 77, 0.001. the attacking algorithm, the initial learning rate is 1.0, the distance function is cosine distance, and the weight for the total variation term is 10-6
CIFAR-10.
CT images. Only the images containing time information used here. As DenseNet-121 is too complex to be attacked, we use a ten-layer neural network with 9 layers of CNNs and one layer FC instead. 10 rounds are run in total and in each round, 10 clients are sampled. In the local training stage, the local data are processed in 20 epochs. The learning rate is FedCury, respectively. The weight for the proximal term in FedProx is 0.1 and À in FedCurv is 0.001. In FedReg, 7 = 0.5 and 1/4 10-6 In the baseline methods with DPSGD, C = 1.0 and a = 0.001, and other hyper-parameters are the same as the ones used in the corresponding methods (without DPSGD). In FedReg with MG, the learning rate and the values of y and 77 are the same those in FedReg. In the attacking algorithm. the initial learning rate is 1.0, the distance function is L2-distance, and the weight for the total variation term 10-8.
10-4 in SGD, FedAvg, FedProx and FedReg, and 1 10-3 3
16

---

Published as a conference paper at ICLR 2022
C
SUPPLEMENTARY RESULTS
a.
10-
10-
8-
10ss
lossiti
6-
4-
2-
0
0
2
4
6 loss - 11
8
10
10
losg 13
Figure C.I: The values of (oss!!) and (oss(t-1) on homogeneously distributed (a) EMNIST and (b) MNIST, where the whole dataset is uniformly partitioned into 10, 000 clients in EMNIST and 2, 000 clients in MNIST, respectively. It can be seen that the values of loss()) are comparable to those of (t-s(t-)), indicating not much previous training data has been forgotten.
Table C.I: Comparison results on Londmarks-Userrr 160k after 2,000 rounds.
Landmarks-Usee 160k
Method
Ro.5
Ro.9
R1.0
ACC
SGD
1178
1824
1953
0.045
FedAvg
84
169
190
0.145
pFedGP
0.005
FedReg
50
101
127
0.160
Table C.2: Average wall-clock time for running a local training stage on MNIST and EMNIST, The GPUs are 1080 Ti.
Method
MNIST (two classes)
EMNIST
FedAvg
0.7
0.3 s
FedProx
0.7
0.3 s
FedCury
0.7
0.3
FedReg
1.1
0.5 s
17

---

Published as a conference paper at ICLR 2022
a.
b.
126
Racc 0.5
175
0.
Ract a 0.8
R.ce20.3
Racc 0.9
Racc 0.4
Round
80-
Racc
Round
8.
50-
24
25
f5
Figure C.2: Impact of y) on the model performance on EMNIST (a) and CIFAR-10 (one class) (b).
The curve Raceza shows the minimum number of rounds required to reach the accuracy of a.
Ground Truth
4
18.67
15.76
18.25
14.90
16.57
16.78
41.23
11.92
13.96
44.23
34.75
14.98
25.70
22.77
23,80
23.17
27.45
23.26
Baseline
methods
9.35
9.85
9,88
8.64
12
9.29
9.53
16.17
10.77
9.93
10.54
11.08
12.70
12.49
Baseline
methods
FedRez with
Figure C.3: Images recovered from updated parameters computed on the EMNIST, CIFAR-10. and COVID-19 CT datasets, and their corresponding PSNR scores. Note that in the attacking process, when only one local training step is performed in each round, the images recovered from the baseline methods FedAvg, FedProx, FedCurv, and SGD, are all the same. High-resolution versions of the CT images are shown Figure C.4
D ALGORITHM
Algorithm D.1 Training Procedure of FedReg.
Input: K, T. [Dihic. (0),, 110, Ts. Upp S, E
Output: {0(*) 17'
=t=1
1: for 1,2, do
2: Randomly sample K clients (())
3: for E (1) do
4: g(t,))
(t) generate TD and D. with n/. 7p and E 5: for s = 2, 1.., do
6:
Compute 9, (ID+), (t,,)) +- (t..)) 1199; (T,,), 0 <- 0.5 (0((.)) + ((1-1))
7: Compute ga (D?), go (ID)), ) w. and wp, (+,,i) +- g(t,t) w.gs (D;) - Wp99 (Dz)
8: end for
9: Send ((c,)) to the server
10: end for
11: ((t) 4- K Σiecto ((,,))
end for
13: return {
18

---

Published as a conference paper at ICLR 2022
188-1
DOB:1
22-22-1 PM
0OB319774
DOB: OV-
2011 May 10 14, 15387
2x 51
B70
07
DOB:
Tm 17
Br
Figure C.4: High-resolution versions the CT images (rows and 2) and the images recovered from FedAvg (rows 3 and 4). Note that the time stamps located the upper right corners of the recovered images are visible.
19


Reviews:
[{"review_id": "t_nKpSmwzBe", "reviewer": "Reviewer_YGd2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem presented in the paper is well-motivated and the proposed solution for handling it looks promising.\n- Generating the pseudo-datasets at the client using the global model is a good idea and avoids potential private information sharing.\n- The paper is technically correct. I have gone through its derivations and did not find any flaws.\n- The results of the method are strong compared to the baseline methods examined.\n- The protection of privacy is a nice addition to the paper which distinguishes it from standard papers in the field.\n- For the most part, the paper is written clearly.\n- Full experimental details and code were provided.\n\nWeaknesses:\n- Some parts of the method, in my opinion, could be motivated and explained better. For example:\n     1. Why did you choose to generate the pseudo-datasets via adversarial training? Several alternatives for generating and using pseudo data were proposed in the literature (e.g., [1, 2, 3, 4]). I think the authors should address this line of research in the paper. \n     2. Why did you need to perform Taylor expansion to Eq. 4 & 5? Why can't they be used directly?\n     3. Given the approximations made, can you please clarify what is the effect of replacing the inequality sign in Eq. 4  with equality (two lines below Eq. 10)?\n\n- Having a shared model only is fine, but the paper misses an important line of work of personalized federated learning which combats the data heterogeneity using personalized components of the model. I believe that the issue of catastrophic forgetting is less severe in these models as the personalized components should make up for that. Did you check that? I think that a comparison to recent strong methods (such as [1]) is missing.\n\nGeneral Questions:\n- Regarding the training procedure, why did you choose to perform one step on $\\mathbb{D}_i$ followed by one step on $\\mathbb{D}_i^s$ and $\\mathbb{D}_i^p$?\n\n- Some experimental details are not clear. For example, how did you choose the hyper-parameters based on the grid search? Did you have a validation set?\n\n\n[1] Achituve, I., Shamsian, A., Navon, A., Chechik, G., & Fetaya, E. (2021). Personalized Federated Learning with Gaussian Processes. arXiv preprint arXiv:2106.15482.  \n[2] Luo, M., Chen, F., Hu, D., Zhang, Y., Liang, J., & Feng, J. (2021). No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data. arXiv preprint arXiv:2106.05001.  \n[3] Hao, W., El-Khamy, M., Lee, J., Zhang, J., Liang, K. J., Chen, C., & Duke, L. C. (2021). Towards Fair Federated Learning with Zero-Shot Data Augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3310-3319).  \n[4] Goetz, J., & Tewari, A. (2020). Federated learning via synthetic data. arXiv preprint arXiv:2008.04489.", "summary_of_the_paper": "The authors claim that in non-iid federated learning setups models experience catastrophic forgetting of previously encountered data. As a result, the performance of the model degrades and the convergence is slower. As a solution, the authors propose FedReg, a method to regularize the model with pseudo-data generated from the local data of the client. Two datasets are generated using the fast gradient sign method. The first dataset, termed pseudo-data, aims at simulating data of other clients in the system. The second dataset, termed perturbed data, aims at retaining the model power on the local dataset of the current client. These datasets are used along with the original data of the client to update the local model parameters. The authors further introduce a method to modify the gradients to enhance the protection of privacy. The authors compared their method against several baselines on MNIST, EMNIST, CIFAR-10, CIFAR-100, and CT images. The comparisons show that FedReg achieves significant improvements in accuracy and convergence, reduction in catastrophic forgetting, and better protection of privacy compared to baseline methods.", "summary_of_the_review": "Overall I think that this paper could be a nice contribution. I think that the experimental section can be improved, and some important references are missing. Upon addressing my concerns I am willing to reevaluate the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "r96d1AIlx_", "reviewer": "Reviewer_9rQY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n-Empirical evaluations show strong results, in several experiments\n- Idea is novel\n\nWeaknesses:\n- My main issue is with the writing of this paper. They claim the main issue is catastrophic forgetting but they don't really show it. They show fig. 1 but should compare to homogenous data where the forgetting shouldn't happen\n- Also, they show good results also on homogeneous data (CIFAR-10 uniform) which again points at something else as the reason why their method works\n- It is also unclear why the synthetic dataset helps with catastrophic forgetting. Should be explain better\n- Assumption 2 is unclear and also I am not sure where exactly do they use it\n- Just below eq. 10 they claim that L_{\\theta^{t-1}} is optimal but that doesn't seem possible considering eq. 4 (in convex case we would have a fixed point right away).\n", "summary_of_the_paper": "The paper addresses the problem of federated learning, they claim the main issue is catastrophic forgetting, which they solve by generating a synthetic dataset per client. ", "summary_of_the_review": "The algorithm seems to work well, but I am skeptical about the alleged problem it solves and that it really addresses it. Writing is the main issue.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YUQF5hbGyma", "reviewer": "Reviewer_UTZk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n(1) Very interesting and novel idea: it is the first paper I have read about showing adversarial examples of global model can help to prevent forgetting in local models. Previous works are mainly focused on directly regularizing the parameters or training on previous data samples.\n(2) good result comparing with other methods.\n\nWeakness:\n(1) writing is a little bit unclear in describing the method. Clearer definition of notations in section 3.1 and 3.2 will be really helpful. \n(2) experiments only focus on small scale dataset with small number of classes. It will be really helpful to include larger scale dataset with natural non-iid data distribution.   ", "summary_of_the_paper": "This paper aims to alleviate forgetting in federated learning on non iid data. The method proposed FedReg focuses on regularizing locally trained parameters with the loss on generated pseudo data, which are based on adversarial examples of the global model in the previous step and the adversarial examples of the local model at current step. This paper experiments on several small scale dataset with large number of clients. It shows improved performance and less forgetting during the training.", "summary_of_the_review": "Because I think the main idea of the paper is really interesting, I will give 6-7 grade. I am more than willing to increase my score if more large scale experiments are added and writing is improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "SRbleR8nD9w", "reviewer": "Reviewer_6gnX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper adopts a new perspective to improve federated learning, i.e., alleviating the catastrophic forgetting issue. This paper is clearly written and it is easy to follow. The proposed method is well explained. Experiments are solid with different datasets and baselines. \n\nI have the following concerns and questions. I would really appreciate it if  the authors could address them.\n1. Catastrophic forgetting issue is mainly studied in the scenario where the neural network is trained sequentially on multiple tasks, such as continual learning. However, in federated learning, we only consider one task as the joint objective for all clients. It is not clear why catastrophic forgetting could occur in federated learning as well. I expect more discussion on the motivation of this problem under FL settings, especially why your solution intuitively solves it. \n\n2. Related to my first concern, the authors mentioned that “the locally trained models suffer from severe forgetting of the knowledge of previous training data”. Given that each selected client needs to do local training for S epochs, I am not exactly sure why previous training data could be forgotten. Is it because of random selection of clients? It would be helpful to precisely describe what “previous training data” means in FL.\n\n3. In Figure 1, I assume that loss^(t-1) is computed using the averaged model from all selected clients. How about loss^t?  I am not sure if loss^t is only using the local model at client i or the averaged model from all selected clients. \n\n4. The assumption (2) in Section 3 is highly related to the assumption that different clients should have similar examples (considering the perturbation). I am wondering how practical this assumption (2) is when the data is non-i.i.d. across different clients. \n\n5. The experiments showed that the proposed method converges faster than the baseline methods in terms of the number of rounds. However, the additional cost incurred by pseudo data generation and regularization could result in longer wall-clock time per round. It would be great if the authors can compare the convergence rate in terms of wall-clock time.  \n\n6. The dataset and data split mechanism in the evaluation section is not the real-world FL dataset (see reference [1-3] as below), which is hard to empirically convince me. For example, FedReg may benefit more by alleviating the catastrophic forgetting when each client only has one or two classes. So I would like to see stronger experiments including the settings and dataset.\n\nReference:   \n[1] Kairouz, Peter, et al. \"Advances and open problems in federated learning.\" arXiv preprint arXiv:1912.04977 (2019).   \n[2] Lai, Fan, et al. \"FedScale: Benchmarking model and system performance of federated learning.\" arXiv preprint arXiv:2105.11367 (2021).    \n[3] Yuan, Honglin, et al. \"What Do We Mean by Generalization in Federated Learning?.\" arXiv preprint arXiv:2110.14216 (2021)", "summary_of_the_paper": "This paper considers the catastrophic forgetting issue in federated learning. The authors observe that this issue is (at least partially) responsible for slow convergence of existing FL methods when the data are not independently and identically distributed (non-i.i.d.) across different clients. This paper proposes FedReg, an algorithm to alleviate the catastrophic forgetting issue by regularizing the local model parameters on the generated pseudo data. ", "summary_of_the_review": "I have been working on related areas and I have read this paper carefully.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of distributed optimization in the Federated Learning setting in particular when the data in the clients is non-i.i.d. The paper points to the problem of catastrophic forgetting during the local update stages to be a cause for the bad training of models and proposes to fix it via introducing a pseudo loss, which are based on adversarial examples of the global model in the previous step and the adversarial examples of the local model at current step. 

The paper's core idea was generally appreciated by the reviewers as well as the favorable evaluation of the proposed method when compared with other existing algorithms. The authors also provided further additional information during the rebuttal period and addressed author comments adding enough justification to the paper to resolve issues in the submission. One lingering issue that remains is the hyper-parameter tuning on test set results that was performed. The authors have promised to redo experiments based on validation sets. 

Overall the paper is borderline but I am recommending accept based on novelty of the idea and strong experimental results.
Decision: Accept (Poster)