Preservation of the Global Knowledge by Not-True Distillation in Federated Learning
Authors: Gihun Lee
Minchan Jeong
Yongjin Shin
Sangmin Bae
Se-Young Yun
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
federated learning
continual learning
knowledge distillation

Abstract: In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that tackling down forgetting will relieve the data heterogeneity problem. To this end, we propose a novel and effective algorithm, Federated Not-True Distillation (FedNTD), which preserves the global perspective on locally available data only for the not-true classes. In the experiments, FedNTD shows state-of-the-art performance on various setups without compromising data privacy or incurring additional communication costs.

Preservation of the Global Knowledge by Not-True Distillation in Federated Learning
Gihun Lee", Minchan Jeong", Yongjin Shin, Sangmin Bae, Se-Young Yun
KAIST
{operisis, mcjeong, yj shin, bsmn0223, yunseyoung)00kale Okaist. ac.kr kr
Abstract
In federated learning, a strong global model is collaboratively Icarned by aggre- gating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data hetero- geneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that tackling down forgetting will relieve the data heterogeneity problem. To this end. we propose novel and effective algorithm, Federated Not-True Distillation (FedNTD), which preserves the global perspective on locally available data only for the not-true classes. In the experiments, FedNTD shows state-of-the -ett--tthe--tt performance on various setups without compromising data privacy or incurring additional communication costs!.
Introduction
At present, massive data is being collected from edge devices such as mobile phones, vehicles, and facilities. As the data may be distributed on numerous devices, decentralized training is often required to train deep network models. Federated learning [23, 24] is a distributed learning paradigm that enables the learning of global model while preserving clients' data privacy. In federated learning, clients independently train local models using their private data, and the server aggregates them into single global model. In this process, most of the computation is performed by client devices, while the global server only aggregates the model parameters and distributes them to clients [2. 50].
Most federated learning algorithms are based on FedAvg [37], which aggregates the locally trained model parameters by weighted averaging proportional to the amount of local data that each client had. While various federated learning algorithms have been proposed thus far, they each conduct parameter averaging certain manner [1. 9. 20, 30, 47, 52]. Although this aggregation scheme empirically works well and provides a conceptually ideal framework when all client devices are active and i.i.d. distributed (a.k.a. LocalSGD), the data heterogeneity problem [31, 56] a notorious challenge for federated learning applications and prevents their widespread applicability [19, 29].
As the clients generate their own data, the data is not identically distributed. More precisely, the local data across clients are drawn from heterogeneous underlying distributions; thereby, locally available data fail represent the overall global distribution, which is referred as data heterogeneity. Despite its inevitable occurrence in many real-world scenarios, data heterogeneity not only makes theoretical analysis difficult [31, 56] but also degrades many federated learning algorithms' performances [18, 27). By resolving the data heterogeneity problem, learning becomes more robust against partial participation [31, 37], and the communication cost also reduced by faster convergence [46, 52].
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Local
Local 3
Local
Local
Not-True Distillation
Forgetting
Knowledge Preservation
Global
Area
Global
Global
Forgetting
Forgetting,
(a) Continual Learning
(h) Federated Learning (FedAvg)
(c) Federated Learning (FedNTD)
Figure I: An overview of forgetting in learning scenarios. As catastrophic forgetting in (a) Continual Learning, (b) Federated Learning also experiences forgetting. However, (c) FedNTD prevents forgetting by preserving global knowledge during local training.
Interestingly, continual learning [42, 45] faces a similar challenge. continual learning, a learner model is continuously updated sequence of tasks, with objective perform well on whole tasks. Unfortunately, owing to the heterogeneous data distribution each task, learning on the task sequence often results in catastrophic forgettn [36, 40], whereby fitting with the parameters important for previous tasks. As consequence, the model parameters drift away from the area where the previous knowledge desirably preserved (Figure la).
task interferes
Our first conjecture such forgetting also exists in federated learning. While the server aggregates local models, the distribution where they are trained may be largely different from those of previous rounds. As a result, the global model faces the distributional shifts at each round, which may cause the forgetting in continual learning (Figure lb). To empirically verify this analogy, examine the global model's prediction consistency. More specifically, we measure its class-wise accuracy while the communication rounds proceed.
The observations verify our conjecture: the global model's prediction is highly inconsistent across communication rounds, significantly reducing the performance of predicting some classes that the previous model originally predicted well. We dig deeper to analyze how averaging the locally updated parameters induces such forgetting and confirm that it occurs in local training: the global knowledge, corresponding to the region outside of local distribution, is prone to be forgotten. As merely averaging the local models cannot recover it, the global model struggles to preserve previous knowledge.
Based on our findings, we hypothesize that mitigating the issue of forgetting can relieve data heterogeneity (Figure lc). To this end, we propose a novel algorithm Federated Not-True Distillation (FedNTD). FedNTD utilizes the global model's prediction locally available data, but only for the not-true classes. We demonstrate the effect of FedNTD preserving global knowledge outside of a local distribution and its benefits on federated learning. Experimental results show that FedNTD achieves sate--o-thheall performance in various setups.
To summarize, our contributions follows:
We present a systematic study on forgetting federated learning. The global knowledge outside of the local distribution is prone to be forgotten and is closely related to the data heterogeneity issue (Section 2).
We propose a simple yet effective algorithm, FedNTD, prevent forgetting. Unlike prior works, FedNTD neither compromises data privacy nor incurs additional communication burdens. We validate the efficacy of FedNTD on various setups and show that it consistently achieves ofte-oof-the-rr performance (Section 3, Section 4).
We analyze how FedNTD benefits federated learning. The knowledge preservation by FedNTD improves weight alignment and weight divergence after local training (Section 5).
1.1 Preliminaries
Federated Learning We aim to train image classification model in a federated learning system that consists K clients and central server. Each client & has local dataset D* where the whole dataset D Uke[k] D At each communication round the server distributes the current global

---

model parameters wi(-1) to sampled local clients K(O), Starting from w(t-1), each client & e K(9)
updates the model parameters w/ using its local datasets D with the following objective:
(f) argmin (x [C[ (w;; w(t-1),xx,,)) 3x]
(1)
w
where & is the loss function. At the end of round t, the sampled clients upload the locally updated parameters back to the server and aggregate by parameter averaging as w(() as follows:
ID4
(t(
Σ
K'EKI [DE] "k
kEK
Knowledge Distillation Given a teacher model T and a student model S, knowledge distillation [17] matches their softened probability 9f and o3 using temperature The e-th value of the 9, can be described as gr (c) Σ. exp(2./T) where 2e is the e-th value of logits vector z and is the number of classes. Given sample æ. the student model S is learned by a linear combination of cross-entropy loss LCe for one-hot label l and Kullback-Leiin Leibler divergence loss LKL using a hyperparameter (:
exp(x./7)
L (1 B) L) + BT² Lx (z)
(3)
c
e
q (c)
Cealg, 1,) = -- _ (c))
CxL = > q
(c) log
(4)
(c)
2 Forgetting in Federated Learning
To understand how the non-llI) data affects federated learning, we performed an experimental study on heterogeneous locals. We choose CIFAR-10 [25] and convolutional neural network with four layers as in [37]. We split the data to 100 clients using Latent Dirichlet Allocation (LDA), assigning the partition of class samples to clients by p Dir(a). The heterogeneity level increases as the a decreases. We train the model with FedAvg for 200 communication rounds, and randomly sampled clients are optimized for 5 local epochs at each round. More details are in Appendix B.
2.1 Global Model Prediction Consistency
To confirm our conjecture on forgetting, we first consider how the global model's prediction varies as the communication rounds proceed. If the data heterogeneity induces forgetting, the prediction after update (i.e.. parameter averaging) may be less consistent compared to the previous round. To examine it, we observe the model's class-wise test accuracy at each round, and measure its similarity the previous round. The results are provided in Figure 2a and Figure 2b.
(b) Server Consistency
(c) Measured Forgetting
(a) Class-wise Accuracy
IID Server
Cosine Similarity
ID Server
D Server
Forgeting (7)
GFF--10
CF10-100
hounds
100
NIID Server
Hounds
100
58
20
220
100
200
Heterogeneity Level (a)
0.05
Rounds
Forgetting analysis on the global server model. (a): Class- wise test accuracy on CIFAR-10 -00 and NIID (a=0.1) cases. (b): Cosine similarity of class-wise accuracy vector w.rt. previous round global model on IID and NIID (a = 0.1) cases. (c): Forgetting FF by different heterogeneity levels on CIFAR- 10 and CIFAR- 100.
Figure 2:
As expected, while the server model learned from i.i.d. locals (III Server) predicts each class evenly well at each round, prediction is highly inconsistent in the non-i.i.d. case (NIID Server). In the non-IID case, the test accuracy on some classes which originally predicted well by the previous global model often drops significantly. This implies that forgetting occurs in federated learning.
To measure how the forgetting is related to data heterogeneity, borrow the idea of Backward Transfer (BwT) [5], a prevalent forgetting measure in continual learning [4, 7, 8, 14], as follows:
F =
max tE II
(A A(D)
(5)
3

---

where ! is the accuracy on class c at round t. Note that the forgetting measure, F, captures the averaged gap between peak accuracy and the final accuracy for each class at the end of learning. The result on the varying heterogeneity levels is plotted in Figure 2c, showing that the global model suffers from forgetting more severely the heterogeneity level increases.
2.2 Knowledge Outside of Local Distribution
We take a closer look at local training to investigate why aggregating the local models induces forgetting. In the continual learning view, a straightforward approach is to observe how fitting on the new distribution degrades the performance of the old distribution. However, in our problem setting, the local clients can have any class. Given that only their portions the local distribution differ across clients, such strict comparison is intractable. Hence, we formulate in-local distribution p(D) and its out-local distribution ((D) to systematically analyze forgetting in local training.
Label Distribution
P(D)
((D)
630)
Clien
(000
33
000
12
004
31
002
Classes
Caasse
Classes
Figure example of in-local distribution p(D) and out-local distrihution P(D) on CIFAR- (a = 0.1).
Definition 1. Consider local dataset D consists of N data points 24 and its label yi in C-class classification problem. The in-local distribution vector & = p(D*) and its out-local distribution vector pt p(D*) are
N
p = [ ,c], where Pe ==
Ily = c)
(6)
1
p pcl, where Po = C [1 pe)
The underlying idea of out-local distribution ((D) is to assign higher proportion the classes with fewer samples in local datasets. Accordingly. it corresponds the region in the global distribution where the in-local distribution p(D) cannot represent. Note that if p(D) is uniform, p(D) also collapses uniform, which aligns well intuitively. An example of label distribution for 10 clients and their p(D) and ((D)s are provided in Figure 3.
We measure the change of global and local models' accuracy on p(D) and P(D) during each communication round, as Figure 4. After local training, the local models are well-fitted towards p(DD) (Figure 4a), and the aggregated global model also performs well on it. On the other hand, the accuracy on (DD significantly drops, and the global model accuracy on it also degrades (Figure 4b).
(a) P(D): Round100
(b) P(D): Round100
(c) Learning
1.0
0.8
0.20
Local
Accuracy
0.6
Accuracy
0.35
Accuracy
Global
0.4
0.10
Localgzo
02
0.05
Giobalz-11
Locala
Globali
Globalr-11 11
Localnn
Globale
Rounds
Figure 4: Accuracy of glohal model and sampled local models for p(D) and ((D) on CIFAR-10 (a=0.1). The emror har stands for the standard deviation of the 10 sampled local clients. In (a) and (b), the global model accuracies for p(D) and ((D) is measured on their joint distributions from 10 sampled clients.
To summarize, the knowledge on the out-local distribution ((D) is prone to be forgotten in local training, resulting the global model's 's forgetting. Based on our findings, we hypothesize that forgetting could be the stumbling block in federated learning.

---

2.3 Forgetting and Local Drift
First empirically observed by [56], the deviation local updates from the desirable global direction has been widely discussed as a major cause of slow and unstable convergence in heterogeneous federated learning [21, 30, 31J. Unfortunately, given the difficulty of analyzing such drift, a common approach is to assume bounded dissimilarity between the local function gradients [20, 31].
One intriguing property of knowledge preservation out-local distribution is that it corrects the local gradients towards the global direction. We define the gradient diversity A to measure the dissimilarity of local gradients and state the effect of knowledge preservation as follows:
Definition 2. For the uniformly weighted K clients, the gradient diversity of local functions f* 11 fh is defined as:
towards the global function f = K Σk=1
A K 12 |
(8)
Here, A measures the alignment of gradient direction of the local function f's w.r.t the global function f. Note that the A becomes smaller as the directions of local function gradients V f become aimilar-e.g. if the magnitudes of the || f* are fixed, the smallest obtained when the direction of V f are identical. To understand the effect of preserving knowledge the out-local distribution p DD,, we analyze how the local gradients and their diversity varies by adding gradient signal çi (D) with factor 3 and obtain the following proposition.
Proposition 1. Suppose uniformly weighted K clients with in- in-local distribution p² .pt.......l.. If we assume the class-wise gradients Je are orthogonal with uniform magnitude, increasing 3 & C/2-1 reduces the gradient diversity from the local gradient V (p" -3p²) .g with the ratio:
MKC.PP
2B
(1+3)
where 3 stands for the effect of knowledge preservation on the out-local distribution B.. The MK.C.² 0 is constant term consists of K, C, and (p²) k==1
The proof is given in Appendix P. Note that here we treat f as sum of class-wise losses Σ. Pee where L. [ w)] is the loss on the specific class c. When B there no regularization to preserve out-distribution knowledge, so the local model only needs to fit on the in-local distribution ph The above proposition suggests that the preserved knowledge on out-local distribution 5z (i.e., as 3 increases) guides the local gradient directions to be more aligned towards the global gradient, reducing the gradient diversity A. Such forgetting perspective provides opportunity to handle the data heterogeneity at the model's prediction level.
3 FedNTD: Federated Not-True Distillation
In this section, we propose Federated Not-True Distillation (FedNTD) and its key features. The core idea of FedNTD is to preserve the global view only for the not-true classes. More specif- ically, FedNTD conducts local-side distillation by the linearly combined loss function L be- tween the cross-entropy loss Lee and the not- true distillation loss LNTD:
N.T. logits
N.T. Sofimax
Global Model (g)
Sample
Dicard True-Closs
KL. divergence loss
Lypn
Closs:
Local Model (i)
M.T. logits
Sofasax
Figure 5: An overview of Not-True Distillation. The true class (Class 3) logits ignored in softmax.
L Lcelg', 1g) + B LNTD(H', (.). (10)
Here, the hyperparameter 8 stands for the strength of knowledge preservation the locll distribution. Then, the not-true distillation loss LNID is defined as the KL-Divergence loss between the not-true softmax prediction vector of and q follows:
(c)
exp (xp/1//)) Ee exp (=//7)
df (c)
LNTD (á., 27) - Σ
(c) log
where
(Yc v). (11)
(c)
(c) Dazy (+g/T)
(a2/7)
5

---

which take softmax with temperature only for the not-true class logits. Figure illustrates how the not-true distillation works given sample æ. Note that ignoring the true-class logits makes gradient signal of LNTD to the true-class as O. The detailed algorithm is provided in Algorithm 1.
Algorithm Federated Not-True Distillation (FedNTD)
Input: total rounds T, local epochs E, dataset sampled clients sets K(O) C K in round f, learning rate 7
Initialize w) for global server weight
for each communication round f = 1, do
Server samples clients (99) and broadcasts w(()) + w((t
for each client & (()) in parallel do for Local Steps 1 .... E. do
for Batches 1... do
10 w V. 03 1D* 1D*1)
Using [Equation 10]
end for end for end for Upload vig to
Server Aggregation (1+1)
Eiekto o
end for
Server output : uT
We now explain how learning minimize LNTD preserves global knowledge out-local distribution f(D). Suppose there are N number of data points in the local dataset D. The accumulated Kullback- Leibler divergence loss CxL. between o the probability vector for the data 24, and its reference qg to be matched for is:
1 CKI. = N
Σ ad (c) log
g (C)
(12)
(C)
By splitting the summands for the true and not-true classes, the term becomes:
N
g (v.) g (y6)
N
C
1 (HE) log
not-tree KL.
of (c
log
(13)
N
of (c)
Proposition 2. Consider the in-local distribution p(D) [PI pc] such that p.
PC]
15 and its
out-local distribution p(D) = [pr..... ... Pel where S. is the set of indices satisfying y: = c. Then the Ciree and Chee-mm each are equivalent the weighted sum on PDD) and p(D)
(c)
Cire
Pe Eies.
(c)
(c)
(14)
(c)
KL
Pe Eies.
(c) log
C -
(c)
With minor amount of calculation from Equation 13, we derive the above proposition. The derivation provided in Appendix Q. The proposition suggests that matching the true-class and the not-true class logits collapses to the loss on the in-local distribution p(D) and the out-local distribution 5 p(D).
In the loss function of our FedNTD (Equation 10), we attain the new knowledge the in-local distribution by following the true-class signals from the labeled data in local datasets using the Lcs. In the meanwhile, we preserve the previous knowledge on the out-local distribution by following the global model's 's perspective, corresponding to the not-true class signals, using the LNTD- Here, hyperparameter /3 controls the trade-off between learning on the new knowledge and preserving previous knowledge. This resembles to the stability-lassiiiiyy dilemma [39] in continual learning, where the learning methods must balance retaining knowledge from previous tasks while learning new knowledge for the current task [35].

---

Table I: Accuracy @1 (%) MNIST [11], CIFAR-10 [25], CIFAR-100 [25], and CINIC-10 [10].
The values in the parenthesis are forgetting measure F. The arrow (I. to shows the comparison to the FedAvg. The standard deviation of each experiment is provided in Appendix F.
NIID Partition Strategy
Sharding
CIFAR-10
Method
MNIST
CIFAR-100 CINIC-10
8
3
8
5
8
10
FedAvg [37]
(o.6)))
40. 40.14 (a..9)
(7.17 (o.40)
64.91 (0.26)
25.57 (0.49)
39.64(a.69)
FedCurv [43]
78, 78,56(0.21)
(4.52
L 47))
62.19 (o.a))
(2.89(u.49)
(0.45(0.37)
FedProx [30]
51.65(u.45) (1.44 (8.26().21) 1
1
64.66 (o.3)
I 25. (5,.90 L (1.47700.7)
FodNova [47]
(7.04)
(2.62.0..5)
(2.03 1
(2.30)) 66.97(0.20) (0.20) (6.96(o.41) (42.55(0.50)
SCAFFOLD [20]
(0.07)
(4.60(
54.26(0.39) 1 (65.4((.23)
68.97 (0.16) 1 (0.82 (2.66600554
MOON [28]
(6.56
(0.5)
50.47 47) 1 56.69 (6.69(0.39) (0.39)
65.30 (0.25) 1 25.29 (0.450 48) 37.07
FedNTD (Ours)
84.44 (o.1))
(o.63
(..18 1 (0.9) 68.56 (o. 15) 31.69 (0.32) 48.07(0.48)
NIID Partition Strategy: LDA
CIFAR-10
Method
MNIST
CIFAR-100 CINIC-10
a
0.05
0.1
0.3
0.5
FedAvg [37]
28.24 28.24(a.711 (9.73(
(0.49((o.s1)
57.24
30.69 (2.53
38. 38.14
FedCury [43]
33. (3.64 78.72(0.20)
44.26(u. | (4.93
1 29.1 (v.3)
29.16)(.32)
36.69(o.61) (0.61)
FedProx [30]
79.25 79.25(0.19) (0.19)
37.19 (o.62) 447.65(0.49)
57.335 (0.35)
300660go.3) (0.27)
39.47 (0.58)
FedNova [47]
60.37 (0.38)
10.00 (Failed) | 28.06(a.71) | (7..44
1 32.15 (4.65(0.23)
30.44 (0.08)
SCAFFOLD (20)
71.57 (0.26) 1
10.00 (Failed) | 23.12 || (2.01(0.29)
(6.16 1 33.668 25) 28.78
MOON [28]
78.95 (0.20)
28.35
(o.77 (8.35
63.10
/30.64 (3.20(
(0.27)
(0.32) (7.92(0.61))
FedNTD (Ours)
40.17:0.58) (o.
54.42 42) 1 (2.42 66.12 (0.19) 32.37 (0.26) (6.24(o.50)
Experiment
4.1 Experimental Setup
We test our algorithm on MNIST [11], CIFAR-10 [25], CIFAR- 100 [25], and CINIC-10 [10]. We distribute the data 100 clients and randomly sample clients with ratio of 0.1. For CINIC-10, use 200 clients, with a sampling ratio of 0.05. We use a momentum SGD with an initial learning rate of 0.1, and the momentum is set as 0.9. The learning rate is decayed with a factor of 0.99 each round, and a weight decay of le-5 is applied. We adopt two different NIID partition strategies:
(i) Sharding [37]: sort the data by label and divide the data into same-sized shards, and control the heterogeneity by 8, the number of shards per user. In this strategy only considers the statistical heterogeneity as the dataset size is identical for each client. We set s as MNIST
(s 2). CIFAR-IO (s (2,3, 5, 10)), CIFAR-100 (s = 10), and CINIC-10 (s = 2).
(ii) Latent Dirichlet Allocation (LDA) [34, 46]: assigns partition of class e by sampling Pe Dir(a.). In this strategy, both the distribution and dataset size are different for each client. We set a as MNIST (a = 0.1), CIFAR-10 (a E {0.05, 0.1, 0.3, 0.5)), CIFAR-100 0.1), and CINIC-10 (a- 0.1)
More details model, datasets, hyperparameters, and partition strategies are provided in Appendix B.
4.2 Performance on Data Heterogeneity
We compare our FedNTD with various existing works, with results shown in Table 1. As reported in [27], even the state-of-the-ar methods perform well only in specific setups, and their performance often deteriorates below FedAvg. However, our FedNTD consistently outperforms the baselines on all setups, achieving state-of-the-ar results in most cases.
For each experiment in Table 1, we also report the forgetting F in the parenthesis along with the accuracy. Note that the smaller F value indicates the global model less forgets the previous knowledge. We find that the performance in federated learning is closely related to forgetting, improving as the forgetting reduces. We believe that the gain from the prior works is actually from the forgetting prevention in their own ways.
7

---

We emphasize that the prior works to lear from heterogeneous locals often require statefulness (i.e.. clients should be repeatedly sampled with identification) [28, 47], additional communication cost [20], or auxiliary data [33]. However, our FedNTD neither compromise any potential privacy issue nor requires additional communication burden. A brief comparison is provided in Appendix C.
We further conduct experiments on the effect of local epochs, client sampling ratio, and model architecture is in Appendix G, the advantage of not-true distillation over knowledge distillation in Appendix H. and the effect of hyperparameters of FedNTD in Appendix K. In the next section, we analyze how the knowledge preservation of FedNTD benefits on the federated learning.
5 Knowledge preservation of FedNTD
In Figure 7, we present the test accuracy on diverse heterogeneity levels. Although both FedAvg and FedNTD show little change in local accuracy in-local dis- tribution p(r), FedNTD significantly im- proves the local accuracy out-local dis- tribution p(x), which implies it prevents forgetting. Along with it, the test accuracy of the global model also substantially im- proves. These gaps are enlarged when the number of local epochs increases, where the local models much deviate from the global model. The accuracy curves during local training are in Figure 6. shows that fitting on the local distribution rapidly leads to forgetting on out-local distribution,
Round 0-50
Round 50-100
Round 100-200
1.0
p.e
002
Local
0.0
0.1
0.0
Local Epochs
Local Epochs
Local Epochs
FedAug
Figure 6: Accuracy on CIFAR-I0 (s=2) Local Training. The error bars stand for the standard deviation on clients.
But FedNTD effectively relieves this tendency without hurting the learning ability towards the in-local distribution.
Sharding (s =s=2) Sharding (s = 5) Sharding (s 10)
LDA (a = 0.05)
LDA (.1)
LDA (a 0.5)
Server Out-Local In-Local
0.
0.6
0.2
0.0
0.3
03
Comm.
Comm.
Comm.
Comm.
Comm.
FedAvg (E 10)
FedNTD 10)
FedAvg
FedNTD =1
Figure 7: Learning curves of FedAvg [37] (blue line) and our FedNTD (red line) on CIFAR- 10 with various heterogeneity setups for local epochs E E (1, 103. (1st and 2nd row): Local test accuracy on in-local distribution p(x) and out-local distribution p(æ). (3th row): Global server test accuracy.
Our interest is how the FedNTD's knowledge preservation on out-local distribution benefits on the federated learning, despite little change of its performance on in-local distribution. To figure it out, we analyze the local models FedNTD after local training, and suggest two main reasons:
Weight Alignment: How much the semantics of each weight is preserved?
Weight Divergence: How far the local weights drift from the global model?
5.1 Weight Alignment
In recent studies, has been suggested that there is mismatch of semantic information encoded in the weight parameters across local models, even for the same coordinates (i.c.. same position)

---

[46, 53, 54]. As the current aggregation scheme averages weights of the identical coordinates, matching the semantic alignment across local models plays an important role in global convergence.
To analyze the semantic alignment of each parameter, we identify the individual neuron's 's class preference by which class has the largest activation output average. We then measure the alignment for a layer between two different models as the proportion of neurons that the class preference is matched for each other. The result provided in Table 2.
While FedAvg and FedNTD show little difference in the IID case, FedNTD significantly enhances the alignment in the NIID cases. The visualized results are in Appendix M, with more details on how the alignment is measured. We further analyze the change of feature after local training using a unit hypersphere in Appendix N and T-SNE in Appendix O.
Table 2: Alignment last two fe-layers for Distributed Global (DG), 10 Locals (L.) and Aggregated Global (AG) models on CIFAR-10 10 datasets for IID and NIID (Sharding 8 = 2, LDA a = 0.05) at round 200.
IID
NIDD (s
NII (a 0.05)
Layer
Alignment
FedAvg
FedNTD
FodAvg
FedNTD
FedAvg
FedNTD
Linear_ 1
w-lt1)
vs. w
0.679
0.668
0.635
0.703
0.597
0.756
(dim: 512)
W w-(t-1) vs. wolf)
G
0.850
0.830
0.787
0.871
0.670
0.856
Linear_ 1
W W-l-i)
vs. w.t
0.771
0.765
0.488
0.552
0,512
0.730
(dim: 128)
w-(t-)) vs. Wol
G
0.898
0.906
0.609
0.836
0,586
0.859
G
5.2 Weight Divergence
The knowledge preservation by FedNTD leads the global model to predict each class more evenly. Here we describe how the global model with even prediction performance stabilizes the weight divergence. Consider a model fitted on specific original distribution, and now it is trained on a new distribution. Then the weight distance between the original model and fitted model increases as the distance between the original distribution and new distribution grows.
2.5
2.5
I1W6 Wall
2.0
Fedkig
FooNTD
1.2
FedNTD
redlvg
|Wa Will
2.0
Fedkvg FedNTD
1.5
1.0
1.0
1.0
0.8
0.5
0.5
0.6
0.4
0.6
0.8
0.4
0
150 Comm. Rounds
50
100
200
50
100
150
200
[|4a - P11l
I|Aa
Pi1l
Comm.
Rounds
Figure 8: Distances for weights and distributions on CIFAR 10 (s=2). (a), (b): The relationship between two distances. The opacity is higher for later rounds. (c), (d): The measured distances for 200 rounds.
We argue that if the distance between the global model's s underlying distribution and local distributions is small, the moved distance between the global model and local models also becomes close. If we assume the local distributions are generated arbitrarily, the most robust choice for the global model's underlying distribution is uniform distribution. We formally rephrase argument the follows:
Proposition 3. Let P: Ac - R20 be the probability measure for the client's 's local distribution and II be the set of measure in the hypothesis. Assume that the class-wise loss Lo(w) Eg [ w)] is X-smooth and we be a optimum of L.. Then the loss L of the client with distribution p becomes:
[(w)= Epchcl "pocdlwe)+ 1
P well
(15)
Then for arbitrary P PE II, the uniform distribution attains the minimax value:
unif.dist e argmin sup Ey->Lll w w,ll,, where wo = Epow
(16)
PEAc Pell
The proof is provided in Appendix S. Although the global model's underlying distribution is unknown, 1 [ai,.... .ai.....g ac],
the normalized class-wise accuracy vector is a handy approximation for as: Ag A where A is the global model's test accuracy and a its class-wise accuracy on the class

---

The results in Figure 8 empirically validate our argument. There is strong correlation between weight divergence || - well (for global model and client k's local model wR) and distribution distance |Ag p& (for client k's distribution pt). By providing better starting point for local training, FedNTD effectively stabilizes the weight divergence.
6 Related Work
Federated Learning (FL) is proposed to update a global model while the local data is kept in clients' devices [23, 24]. The standard algorithm is FedAvg [37], which aggregates trained local models by averaging their parameters. Although its effectiveness has been largely discussed in i.i.d. settings [44, 48), many algorithms obtain the sub-optimal when the distributed data is heterogeneous [31, 56]. Until recently, a wide range of variants of FedAvg has been proposed to overcome such a problem. One line of work focuses on local-side modification by regularizing the deviation of local models from the global model [1, 20, 30, 47]. Another is the server-side modification, which improves the efficacy of aggregation of local models in the server [9, 33, 46, 55J. Our work aims to preserve global knowledge during local training, which belongs to the local-side approach.
Forgetting View in FL A pioneer work that considers forgetting in FL is FedCurv [43]. It regards each local client task, and [43] regulates the change of local parameters to prevent accuracy drop on all other clients. However, it needs to compute and communicate parameter-wise importance across clients, which severely burdens the learning process. On the other hand, we focus the class-wise forgetting and suggest that not-true logits from local data contain enough knowledge to prevent it. A concurrent work of our study [49], which also reports the forgetting issue in local clients by empirically showing the increasing loss of previously learned data after the local training. To prevent forgetting, [49] exploits generated pseudo data. Instead, we focus on the class-wise forgetting and suggest that true logits from local data contain enough knowledge to prevent it. The continual learning literature is further discussed in Appendix D.
Knowledge Distillation (KD) in FL In FL, typical approach is using KD to make the global model learn from the ensemble of local models [9, 26, 33, 57]. By leveraging the unlabeled auxiliary data, KD effectively tackles the local drifts by enriching the aggregation. However, such carefully engineered proxy data and may not always be available [30, 55, 58]. Although more recent works generate pseudo-data to extract knowledge by data-free KD [55, 58], they require additional heavy computation, and the quality of samples is sensitive to the many hyperparameters involved in the process. On the other hand, simple variant of KD, our proposed method surprisingly performs well on heterogeneity scenarios without any additional resource requirements.
7 Conclusion
This study begins from an analogy to continual learning and suggests that forgetting could be major concern in federated learning. Our observations show that the knowledge outside of local distribution prone to be forgotten in local training and is closely related to the unstable global convergence. To overcome this issue, we propose a simple yet effective algorithm, FedNTD, which conducts local-side distillation only for the not-true classes prevent forgetting. FedNTD does not have any additional requirements, unlike previous approaches. We analyze the effect of FedNTD from various perspectives and demonstrate its benefits in federated learning-
Broader Impact We believe that Federated Learning an important learning paradigm that enables preecy-presrriing.. ML. Our work suggests the forgetting issue and introduces the methods to relive it without compromising data privacy. The insight behind this work may inspire new researches. However, the proposed method maintains the knowledge outside of local distribution in the global model. This implies that if the global model is biased, the trained local model is more prone to have a similar tendency. This should be considered for MI participators.
Acknowledgments
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITPP) grant funded by Korea government (MSIT) [No. 2021-0-00907, Development of Adaptive and Lightweight Coll-CCollbboraivee Analysis Technology for Enabling Proactively Immediate Response and Rapid Learning, 90%] and [No. 2019-0-00075, Artificial Intelligence
Graduate School Program (KAIST), 10%].
10

---

References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2/11.0422
2021.
[2] Mohammed Aledhari, Rehma Razzak, Reza M Parizi, and Fahad Saced. Federated learning: A survey on enabling technologies, protocols, and applications. IEEE Access, 8:140699-140725, 2020.
[3] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139-154, 2018.
[4] Sungmin Cha, Hsiang Hsu, Taebaek Hwang, Flavio Calmon, and Taesup Moon. Cpr: Classifer--rojeetionn regularization for continual learning. arXiv preprint arXiv:2006.07322 2020.
[5] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference Computer Vision (ECCV), pages 532-547, 2018.
[6] Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip Torr, and David Lopez-Paz. Us- ing hindsight to anchor past knowledge in continual learning. arXiv preprint urXiv:2002.08165
2(7), 2020.
[7] Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip HS Torr. Continual learning in low-rank orthogonal subspaces. arXiv preprint arXiv:2010. / 1635, 2020.
[8] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv: 2018.
[9] Hong-You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint arXiv:2009.01974 2020.
[10] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- 10 is not imagenet or eifar-10. arXiv preprint arXiv:18100035505 2018.
[11] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-143 2012.
[12] Terrance Vrie and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: Doxiiv:708..000 04552, 2017.
[13] Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated class-incremental learning, arXiv preprint arXiv: 1473, 2022.
[14] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762-3773. PMLR, 2020.
[15] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 13211:131..6211, 2013.
[16] Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang. Xiaoyang Wang, Pranceth Vepakomma, Abhishek Singh, Hang Qiu. et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint vrXiv:2007.13511 2020.
[17) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 5505.0253311 2015.
[18] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:/909 1909006355. 2019.
[19] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings. et al. Advances and open problems in federated learning. arXiv preprint 1912.0991 2019.
[20] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pages 5132-5143, PMLR, 2020.
11

---

[21] Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 25$1945529 PMLR, 2020.
[22] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351--3526 2017.
et al.
[23] Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimiza- tion: Distributed machine learning for on-device intelligence. arXiv preprint 11110:661. 2016.
[24] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint 1r010:1610 2016.
[25] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar- 100 datasets. URI:
httpps///www. cs. toronto. edw/kriz/ifaar html, 6, 2009.
[26] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint 191000391, 2019.
[27] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. arXiv preprint arXiv:2102.02079, 2021.
[28] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrasse federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713-10722 2021.
[29] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3)550-60, 2020.
[30] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. P Proceedings of Machine Learning and Systems, 2:429 450, 2020.
[31] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint 17Xivv190..02189 2019.
[32] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
[33] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. arXiv preprini arXiv:200..07242, 2020.
[34] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of het- erogeneity: Classifier calibration for federated learning with non-iid data. arXiv preprint 0tti:21006.000011 2021.
[35] Mare Masana. Xialei Liu, Bartlomiej Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation on image classification. arXiv preprint arXiv:2010.1522 15277, 2020.
[36] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[37] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication- effficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273 1282. PMLR, 2017.
[38] Matias Mendieta, Taojiannan Yang, Pu Wang. Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 8397-8406,
2022.
[39] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plastiiity dilemma: In- vestigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
12

---

[40]
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[42] Mark B Ring. Child: A first step towards continual learning. In Learning to learn, pages 261-292. Springer, 1998.
[43] Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor- Yosef, and Itai Zeitak. Overcoming forgetting in federated learning on non-iid data. arXiv preprint 1000777,, 2019.
[44] Sebastian U Stich. Local sgd converges fast and communicates little. 1805:9780, 2018.
arXiv preprint
[45] Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pages 181-209. Springer, 1998.
[46] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint 2rXiv:2002.06440, 2020.
[47] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481, 2020.
[48] Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning, pages 10334-10343, PMLR, 2020.
[49] Chencheng Xu, Zhiwei Hong, Minlie Huang, and Tao Jiang. Acceleration of federated learning with alleviated forgetting in local training. arXiv preprint arXiv:2203.02645 2022.
[50] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
[51] Jachong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pages 12073-12086. PMLR, 2021.
[52] Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In International Conference on Learning Representations, 2021.
[53] Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. Fed2: Feature- aligned federated learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 2066-2074, 2021.
[54] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pages 7252-7261. PMLR, 2019.
[55] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and I Ling- Yu Duan. Fine-tuning glohal model via data-free knowledge distillation for non-id federated learning. arXiv preprint arXiv:2203.09244 2022.
[56] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint / 00582, 2018
[57] Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.0793 079999, 2020.
[58] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heteroge- neous federated learning. arXiv preprint 2rXiv.2105.10056, 2021.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] . You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? and the code is MIT license
[Yes] The datasets are public datasets
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We clarified the scope in both abstract and introduction.
The contributions are summarized at the end of Section 1.
(b) Did you describe the limitations of your work? [Yes] Before reference, we discussed about potential limitations broad impact
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Before reference, we discussed as broader impact
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In all proposi- tion parts. The detailed notations are in Appendix A.
(b) Did you include complete proofs of all theoretical results? [Yes] Included in the corresponding Appendix sections.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] Included in supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Included in the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We indicated the standard deviation each result for the main experiments in the corresponding Appendix section.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Included in the Appendix.
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A] We used the public benchmarks.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We only used the public dataset that has been sufficiently considered about the issues.
4. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "s-VK77J72hd", "reviewer": "Reviewer_eF2S", "summary": "This paper addresses the data heterogeneity problem in federated learning, by drawing an analogy with the forgetting problem in continual learning. The authors find that the global model forgets the knowledge from previous rounds, when aggregating clients' locally trained models that optimize toward the local data distribution. They propose a novel and simple algorithm to mitigate the forgetting of knowledge. Specifically, the authors add to the training loss a term that penalizes the discrepancy of the predicted class distribution between the local model and the global model, where the discrepancy considers all but the ground-truth class. The authors perform experiments to thoroughly compare with state-of-the-art federated learning methods and demonstrate the preservation of global knowledge in their method.", "questions": "The authors may respond to the comments raised in the \"Strengths And Weaknesses\" section.\n", "limitations": "The authors are aware that privacy is an important subject in federated learning and state that their approach does not compromise data privacy. The authors also alert that bias in the global model, if any, may cause a similar tendency in the local models.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe proposed method is simple and elegant. It appears to work quite well. The authors study the heterogeneity problem in federated learning by drawing inspiration from continual learning and knowledge distillation. They successfully develop a technique that can cope with data heterogeneity. The idea is intuitive and it could be easily used by practitioners.\n\nThe authors spend a significant amount of effort to study the problem of forgetting. They define a forgetting measure, illustrate the forgetting behavior, justify suppressing gradient diversity as a way to mitigate forgetting, propose a novel loss term for training, and demonstrate its effectiveness in preserving the global knowledge. All in all, the work is solid and well done.\n\nThe paper is generally well written, but it needs thorough proofreading. Many obvious grammatical errors are spotted all over the paper.\n\nA few comments for improvement:\n\n- Line 123. It is a bit unclear what \"... (the) accuracy varies on $p(D)$ and $\\tilde{p}(D)$\" means. The authors may want to elaborate.\n\n- Line 146. It is unclear, in the first pass of reading, why the local gradient $\\nabla f^k$ is computed in the said way. This may have something to do with the loss function (10) that is proposed only later. The authors may want to give some motivation before Proposition 1. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a4OJyqEqNs3", "reviewer": "Reviewer_sAmP", "summary": "This paper suggests that forgetting could be the bottleneck of global convergence and then proposes FedLSD to handle the forgetting problem by using Not-True Distillation in the local training stage. Unlike traditional distillation methods in Federated Learning, FedLSD does not require additional auxiliary datasets because the distillation works on the clients. To preserve global knowledge on out-local distribution, the distillation only considers the not-true classes. Some experiments are conducted to show its effectiveness.", "questions": "See Weaknesses 2 and 4.", "limitations": "1. The novelty of the paper is limited.\n2. Distillation on the client side may limit the model's learning from local data on the clients", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Knowledge forgetting in the global model is a key issue. This paper makes a lot of analysis on forgetting in federated learning.\n2. The main strength of the algorithm is that there is no need to use additional auxiliary data during distillation.\n3. This paper is well written and easy to read.\n\n**Weaknesses**\n1. The knowledge distillation used on the clients is straightforward, although the author expands it to not-true distillation, the novelty is still limited.\n2. In the experiment section, direct distillation on clients should also be used as a baseline; otherwise, one cannot see how much impact the proposed Not-True Distillation brings to the results.\n3. Figure 1 is less informative. Any method based on regularization can be expressed as Figure 1(c), so it is hard to see the characteristics of the proposed method.\n4. According to previous observations and experience, distillation on the client side is very likely to limit the model's learning from local data on the clients. Although this paper only uses the Not-True classes for distillation, such a problem may still exist to restrict the learning of new knowledge.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EAMGJnluX0B", "reviewer": "Reviewer_qrmZ", "summary": "The paper proposes a method to mitigate catastrophic forgetting in federated learning, which focuses on improving the performance of the local model by teaching the knowledge of the global model to the local model through an improved knowledge distillation method. According to the experimental results presented in the paper, the method possesses a good performance.", "questions": "1. For the MOON method, the results in the table seem to be not as good as the performance of the original paper. the MOON method is more sensitive to parameters, and it is suggested that more comparisons of different hyper parameters can be put in the appendix\n\n2. In fact, I am curious why discarding not-true has a positive impact, can you describe it simply?", "limitations": "See the weaknesses and questions", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The experimental setup of this paper is relatively adequate, taking into account scenarios with different federal settings\n2. The experimental approach is simple and achieves good results\n3. The method is well analyzed\n\n**Weaknesses**\n1. lack of comparison. The following paper is similar in approach to the method proposed in this paper, and also performs knowledge distillation locally\n    - Mendieta M, Yang T, Wang P, et al. Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 8397-8406.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept