Meta Knowledge Condensation for Federated Learning
Authors: Ping Liu
Xin Yu
Joey Tianyi Zhou
Venue: ICLR 2023 poster (2023)

Abstract: Existing federated learning paradigms usually extensively exchange distributed models, rather than original data, at a central solver to achieve a more powerful model. However, this would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous. As a result, current federated learning methods often require plenty of communication rounds in training. Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the federate learning communication cost without leaking original data. In this work, we first present a meta knowledge representation method that extracts meta knowledge from distributed clients.  The extracted meta knowledge encodes essential information that can be used to improve the current model. As the training progresses, the contributions of the same training samples to a federated model should also vary. Thus, we introduce a dynamic weight assignment mechanism that enables informative samples to contribute adaptively to the current model update. Then, informative meta knowledge from all active clients is sent to the server for model update. Training model on the combined meta knowledge that is regarded as a condense form of original data can significantly mitigate the heterogeneity issues. Moreover, to further ameliorate data heterogeneity, we also exchange meta knowledge among clients as conditional initialisation for meta knowledge extraction. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method. Remarkably, our method outperforms the state-of-the-art by a large margin (from $74.07\%$ to $92.95\%$) on MNIST with a restricted communication budget (\textit{i.e.}, 10 rounds).

Published as a conference paper at ICLR 2023
META KNOWLEDGE CONDENSATION FOR FEDERATED LEARNING
Ping Liu
Xin Yu
Center for Frontier Al Research
Information Technology and Electrical Engincering
A*STAR
The University of Queensland
138632, Singapore
Brisbane, Australia [xin yuu uuq. edu. au
(pino.pingliu] @gma com
Joey Tianyi Zhou
Center for Frontier Al Research
STAR
138632, Singapore
(joey_zhou] @cfar.a-star. -sstarrddu.sg
ABSTRACT
Existing federated learning paradigms usually extensively exchange distributed models a central solver achieve a more powerful model. However, this would incur severe communication burden between server and multiple clients espe- cially when data distributions are heterogeneous. As result, current federated learning methods often require a large number of communication rounds in training. Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning. In this work, we first intro- duce a meta knowledge representation method that extracts meta knowledge from distributed clients. The extracted meta knowledge encodes essential information that can be used to improve the current model. As the training progresses, the contributions of training samples federated model also vary. Thus, intro- duce dynamic weight assignment mechanism that enables samples to contribute adaptively the current model update. Then, informative meta knowledge from all active clients is sent to the server for model update. Training a model on the combined meta knowledge without exposing original data among different clients can significantly mitigate the heterogeneity issues. Moreover, to further ameliorate data heterogeneity, we also exchange meta knowledge among clients as condi- tional initialization for local meta knowledge extraction. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method. Remarkably, method outperforms the state-of-the-att by large margin (from 74.07% 92.95%) on MNIST with a restricted communication budget (i.e., 10 rounds).
INTRODUCTION
Most deep learning-bait based models are trained in a data-centralized manner. However, in some cases, data might be distributed among different clients and cannot be shared. To address this issue, Federated Learning (FL.) (Yang et al. [001b]]a] a Kairouz et al., (222)) has been proposed to learn a powerful model without sharing private original data among clients. In general, most prior FL works often require frequent model communications to exchange models between local clients and a global server, resulting in heavy communications burden (Wu & &Wann] 202T Chencheng et al. 2022) Therefore, it is highly desirable to obtain powerful federated model with only a few communication rounds.
In this work, we propose new meta knowledge-drive federated learning approach to achieve an effective yet communication--efiiiennn model, thus significantly reducing communication costs. Unlike
"Equal contributions corresponding author

---

Published as a conference paper at ICLR 2023
Deceutrulized Knowledge Coudensation
Centrul Model Training
Knowledge Cendensation Optinizzation
apops
Troinin CK
Leurnable Constraint
sGp
Node N
FL Modal
Dynamic Weights
Condensed
Knowledge
Active
Canditional
Clienta
Initialization
Figure 1: Illustration of our pipeline. in which only three active clients are shown. The local clients conduct meta knowledge condensation from local private data, and the server utilizes the uploaded meta knowledge for training a global model. The local meta knowledge condensation and central model training are conducted in an iterative manner. For meta knowledge extraction on clients, we design two mechanisms, Le., meta knowledge sharing, and dynamic weight assignment. For server-side central model training, we introduce a lcarnallc constraint.
prior works, we formulate federated learning in new perspective, where representative information will be distilled from original data and sent to the server for model training. On the client side, we extract representative information of original data and condense it into a tiny set of highly-compressee synthetic data, namely meta knowledge. Furthermore, we develop two mechanisms, i.e., dynamic weight assignment and meta knowledge sharing, in the condensation process to mitigate the data heterogeneity issue widely existing in decentralized data. On the server side, we train our global model with meta knowledge uploaded from clients rather than simply averaging client models.
Specifically, we firstly distill the task-specific knowledge from private data on local clients and condense it as meta knowledge. The meta knowledge condensation process is modeled as a bi-level optimization procedure under the federated learning setting: the inner-loop minimizes the training loss meta knowledge to update model; and the outer-loop minimizes the training loss original data to update meta knowledge based on the updated model. the optimization process, we assign dynamic weights to each sample based its training loss. By dynamically adjusting the weight of each sample in training, we empower each sample to contribute adaptively to the current model. Besides, further mitigate heterogeneous data distributions among different clients, we design meta knowledge sharing mechanism.
Our model can be trained with meta knowledge of various clients, which better describes the overall distribution. This is in contrast to previous methods that average local models on the server. To further improve the stability of the central model training, we incorporate a learnable conditional generator. The generator models the statistical distribution of the uploaded meta knowledge and generates synthetic samples, which provide historical information the model update. It is worth noting that meta knowledge, which contains the essential information of the original data and the corresponding class information, can be used as normal training data for model training. As a result, global model is trained with both the uploaded and generated meta knowledge the server side, effectively reducing the impact of data heterogeneity and reducing the number of communication rounds.
We have conducted extensive experiments on several benchmark datasets, including MNIST (LeCun let al. (2100), SVHN (Netzer et al.) 2011), CIFARIO (Krizhevsky & Hinton 2009), and CIFART00 (Krizhevsky & Hinton] 2009). The results demonstrate the efficacy and efficiency of our proposed approach. In particular, our method demonstrates a significant improvement over the competing works, particularly in scenarios with limited communication budgets (i.e., 10 communication rounds). Overall, our key contributions are summarized as follows:
We propose new meta knowledge driven federated learning approach, in which we present a novel approach for federated meta- meta-knowledge extraction. Our method can effectively encodes local data for global model training. Specifically, we formulate a dynamic weight assignment mechanism to enhance the informative content of the extracted neta-knooledde, and design a knowledge sharing strategy to facilitate the exchange of knowledo among clients without exchanging the original data.
We introduce a server-side conditional generator that models the statistical distribution of uploaded meta knowledge stabilize the training process. Benefiting from the extracted
2

---

Published as a conference paper at ICLR 2023
meta knowledge and learned statistical distribution, model requires fewer communication rounds compared to competing methods while achieving superior performance.
METHODOLOGG
2.1 PROBLEM DEFINITION
Federated Learning (FL) trains model across set of decentralized devices, i.e., set of clients and a server. Suppose there is data distributed on clients, each which has a local private training
dataset D { M. S S n and a weight value p². It is noted that = UD and Σ =
Without loss of generality, discuss multi-class classification problem under federated learning setting. The learning target is formulated as follows:
[in D) & (w,
c= where (-) is local objective function that is optimized on the c-th client. The loss function (*(-) formulated follows:
C (w, DC)
2 vf).
where ((-) user-defined loss function (e.g., cross-entropy loss), and w denotes model parameters. As most FL algorithms need to exchange locally trained models multiple times, communication burden is often non -oggegiiii.le. Though one-shot FL (Zhou al., (220aa) has been proposed to reduce communication cost, it suffers performance degradation.
2.2 FEDERATED LEARNING VIA META KNOWLEDGE
We propose new FL approach to solve the aforementioned limitations. Specially, as shown in Figure D our method conducts federated meta knowledge extraction (FMKE) on local clients and server-side central model training (CMT). To mitigate the data heterogeneity issue in FMKE, we design two mechanisms, i.e., dynamic weight assignment, and meta knowledge sharing. To stabilize the central model training, introduce learnable constraint modeled by conditional generator. The technical details are provided in the following subsections.
2.2.1 FEDERATED META KNOWLEDDGE EXTRACTION CLIENTS
We design Federated Meta Knowledge Extraction (FMKE) to extract key information from decen- tralized data. In the decentralized scenario, the original data D is distributed on set of clients. Fach client has its private data 'De (D UD=, I c < C, where C is the client number), and model downloaded from server. For simplifying the following discussion, we denote the model downloaded by client w
On each active local client c, FKME distills key information from corresponding local private data
D The distilled information is condensed as meta knowledge Dr, which will be used to replace the original data extracted meta knowledge from all active clients, is much less than that of D. The condensed meta knowledge D is highly compressed and representative. To enable the models trained the original data and the meta knowledge to achieve the comparable performance, we formulate the objective function of FMKE a
global model training on the server The cardinality of D, i.e., the size of
De
arg min L (" D9) = arg min ( 1/ Vr DD), DO).
Dr
Dr
where 7) denotes a learning rate for updating w by stochastic gradient descent (SGD).
To solve the above objective function, we employ a bi-level optimization solution (Rajeswaran et al. 2019: Wang al., 2018). To be specific, the meta knowledge extraction is formulated as nested optimization process: in the inner-loop, based on an initialized meta knowledge, a model is updated to minimize the training loss over the meta knowledge; in the outer-loop, given the updated model,
'The dimension of synthesized DE is the same as that of original data D. Assume one gradient descent step conducted to achieve w" here.
3

---

Published as a conference paper at ICLR 2023
the meta knowledge is renewed by minimizing the training loss over the original data. The iterative formulation of our bi-level optimization process on client c is written as:
m arg min [ (w", s.t. w" arg min (" DD),
(4)
w"
where (- Dr) denotes a loss function over the original data De on client c, ( D9) denotes loss function over the meta knowledge De on client c.
The inner-loop and outer-loop are implemented alternatingly and stochastic gradient descent is employed to update the model and meta knowledge. At first, on an active client c, we update the model parameter by the following formulation:
w f- w V L 2 (w", (W",
(5)
where denotes a learning rate for the inner-loop. An updated model w is obtained in this inner-loop. Secondly, given the updated model w", evaluate it original data D and calculate the loss ([(""""), Then the condensed meta knowledge De can be updated by:
D' + Dr a ( DD),
where a denotes a learning rate for the outer-loop, and the meta knowledge Dr is initialized based on a uniform distribution (i.e., Di ~ U[-1, +1). An updated meta knowledge Dr is obtained in this outer-loop. The inner-loop and outer-loop are conducted in alternatingly manner.
The presence of data heterogeneity in FL often results in biased meta knowledge extracted on each client. To address this issue, design two effective mechanisms within our FMKE approach, namely Dynamic Weight Assignment and Meta Knowledge Sharing.
Dynamic Weight Assignment:
Concretely, we dynamically assign weights for each training sample in D. The prediction confidence of each sample evolves during training, and the dynamic weights is determined based the varying contributions of each sample to the meta knowledge extraction. To be specific we calculate a dynamic weight value of for each sample Dj i.e, (f), based on its prediction loss f(w", x. v(). The formulation is defined as follows:
exp (-T l(ws, x7, k!))
where is a tlyper-paramedd to smooth the result, D [D S N and N denotes original data number on client c. We assign the dynamic weight to each sample to update meta knowledge:
Dr D
a. Φ0, &0, D.).
N
where (w Φ6, D
of l(w x8) Σ
Meta Knowledge Sharing: The initialization of De is crucial factor the extraction of informative meta knowledge. In prior bi-level optimization works (Rajeswaran et al.) 2019 Wang al. 2018). the initialization value was randomly sampled from constant distribution (i.e., De U(- +1), referred to as unconditional initialization. However, due to the heterogeneity issue in FI., this method may result in the extracted meta knowledge becoming biased towards the corresponding local data. To address this limitation, we propose a conditional initialization method incorporating a meta knowledge sharing mechanism.
Conditional initialization (Wang et al. 2020 Denevi et al. 2020) is obtained from data characteristics rather than random generation. To achieve conditional initialization, we design a simple yet effective strategy extracting meta knowledge De for client e at the current round t. Specifically, during the initialization process of client we randomly select another client c' and use its meta knowledge
DrL, extracted in the previous round - 1 as the initial value in Eq. 8 This changes the initialization for De from an unconditional manner to a conditional manner: Dr + DDD"" 1,, d' randint 11, C). In this manner, the meta knowledge condensation for client e is determined by both the local data on client e and the knowledge extracted on another client é', effectively mitigating the heterogeneity issue.
For the case of discussion, ("()) is slightly abused here.
4

---

Published as a conference paper at ICLR 2023
2.2.2 SERVER-SIDE CENTRAL MODEL TRAINING
After conducting FMKE, we upload the condensed meta knowledge D from clients to a server. On the server, the uploaded meta knowledge is used as normal training data to train global model Wa:
L. D)
Σ
((Wa, R,, Bi),
(9)
D
where É(.) is a cross-entropy loss function as in Eq. and ò = LDD.DDDs 1SCIC.
To further ameliorate data biases among diverse clients, introduce additional synthetic training samples into the central model training. Those introduced training samples are from the same distribution of upload meta knowledge D. Specifically, at first, we model the statistical distribution of uploaded meta knowledge D via a conditional generator, and then we sample additional data points based on the learned distribution. Thus, sampled data would share the same distribution as D. After the introduction of sampled synthetic data, we not only stabilize our training procedure but also achieve better performance.
To facilitate the discussion, we divide the model We into a feature extractor F with parameter W and classifier C with parameter W in which We (W a W Accordingly, we denote latent representation F(W .) and final prediction as y C(WC, G+ 2). The conditional generator G maps a label y into a latent representation Z ~ Gly, w and & is optimized by the objective:
g
arg max Bymp((s) Benlu.wo) logp(ulz, W
(10)
Gyy+z
where 8 denotes the parameter of g.
The trained generator G models the distribution of uploaded meta knowledge D. By sampling data from the distribution, we obtain set of "pseudo" meta knowledge Descu with corresponding labels. The generated "pseudo" meta knowledge Disse as well as uploaded D are utilized to train the global model by minimizing the following function:
Laveralt Wa, (D, (D,DDeeu)) L(Wo, +) + 3C(Wa, Doseu),
(1I)
where B is parameter and determined by the cardinality fraction
[D]
After central model training, we broadcast the obtained global We and meta knowledge D to clients. On each active client, the broadcasted model Wa as well as meta knowledge D are used for new round of FMKE. FMKE and CMT collaborate with each other in an iterative symbiosis paradigm, benefiting each other increasingly as the learning continues. The training process of our FedMK illustrated in Alg. [ used for inference.
After the completion of the training process, the trained global model is only
Computational Complexity: FedMK includes two parts: federated meta knowledge extraction on clients and global model training on the server. On clients, our method adopts bi-level optimization to extract the meta-knowledge. The bi-level optimization has running-time complexity O x n) (Fallah et al. 2020a). in which denotes the meta knowledge size, N denotes the number of samples on the client. It should be noted that the selected nodes can conduct FMKE in parallel. On the server, the global model training in our method has running-tiin time complexity of O(z). In total, the overall running-time complexity of our method is O(N x n..
3 EXPERIMENTS
3.1 DATASETS
We evaluate our algorithm and compare to the key related works on four benchmarks: MNIST (LeCun)
jet al. 2010), SVHN (Netzer et al. 2011), CIFARIO (Krizhevsky Hinton| 2009). and CIFAR100
(Krizhevsky & Hinton| 2009). MNIST a database of handwritten digits (0-9). In MNIST, there are 50, 000 images in the training set and 10, 000 images in the test set, and their size is x 28 x 28 pixels (c x h). There which there are 600, 000 color images collected from house numbers in Google Street View images. In SVHN each image is of size 3 32 32. The class number of SVHN is the same MNIST.
10 classes in MNIST dataset. SVHN is a real-world image dataset, in
5

---

Published as a conference paper at ICLR 2023
Algorithm 1: FedMK
Input:
Original data D; glohal parameters We: generator parameter w the communication budget.
Output: Optimal W
while within the communication budget do the server selects active clients C' uniformly at random, and broadcasts We to selected clients E-Federated Meta Knowledge Extraction on selected clients C:
for all user с E C in parallel do
w +- Wei for O, ... # #Round do
if [ == 0: Di... U[-1,+1];
elif 1: conduct the conditional initialization, i... Ding Di randint(1, C);
calculate dynamic weights by Eq.
10
generate Dj by Eq.
end
send the Dj to the server.
end 13
14
G-Global Model Training on the server:
15
update generator parameter w by Eq. 10
generate Desew by the updated generator 9;
16
17
update global parameter Wa by Eq. 11
18 end
19 return Wi as Wa:
Table 1: The classification accuracy with limited communication rounds (i.e, 10).
Setting
FedAvg
FedProx
FedDistill
FedEnsem
FedGen
FedMK
MNIST
a=0.50
74.61=2.79%
73.56=3.844 84%
75.04+22.32%
75.39+ 4.4
1..07+
92.95++3.81%
a=0.75
73.4913.62%
73.131268%
76.2111.59%
74.28.11.655
13.97.13.98%
02.86.0..28%
a=1.0
1.8%
73.35=0.10%
76.19#1.19%
74.45+2.25%
73.97
93.63-10.4%
SVHN
a=0.50
29.55+2.022%
28.52+22.4%
26.924-2.02%
29.12-42.99%
28.9442..62%
7..01+
a=0.75
31.7110.900%
25.7812..6%%
25.77....27%
29.53.11.91%%
30.40.1..86%
74.90.10.449
a=1.0
30.87=1411%
29.59=1.26%
25.84#1.21%
32.67:2.0075
33.6243.28%%
7..8%%20.32%
CIFARIO
a=0.50
26.63-00.50%
06.91+
24.38:-221255
27.56+00.006
25.42.4..
47.33 1.8%%
a=0.75
25.42:11.7%%
24.18:11.25% 24.85
26.42:10.59%
26.25:10.33%
49.04:11115%
6=1.0
26.80=0.800
89%%
26.66=0.37%
25.83+40.00 79%
26.74:0.87%
1..36+1..2%%
50.32:+0.69%
CIFARIO0
e=0.50
11.6621 1.2%
12.0900.22%
10.7620.779
13:2020.44%
26.74:0.60%% 10.3420.65%
a=0.75
11.6520.36% 12.1120.45%
11.55:0.796
13:1510.209%
10.2111102%%
27.43:10.54%
12.31+0.60%
11.34-0.48%
11.50+0.600
13.3140.54%
11.40.4.75%%
28.20-+0.10%
CIFARIO dataset consists of 60, 000 color images, each of which has a size of 3 32 32. There are 50, 000 images in the training set and 10, 000 images in the testing set. There are 10 classes in CIFAR1O. CIFAR100 dataset consists of 60, 000 color images, each of which has size of 3 x 32 32, There are 50, 000 images in the training set and 10, 000 images in the testing set. For each image in CIFAR 100, there are two kinds of labels, i.e., fine label and coarse label. We choose coarse labels and therefore we have 20 classes in the experiment on CIFAR100
3.2 IMPLEMENTATION DETAILS
We set the user number to 20, and the active-user number to 10. We use 50% of the training set and distribute it on all clients. All testing data is utilized for evaluation. We use LeNet (LeCun al.
1989) as the backbone for all methods: FedAvg (McMahan et al.) 2017). FedProx (Li et al. 2020). FedDistill (Seo al., 2020), FedEnsemble (Zhu et al.) 2021). FedGien (Zhu al.. 2021). lj. and FedMK. Dirichlet distribution Dir(a) ( is used to model data distributions. Specifically, we test three different values: 0.5, 0.75, and 1.0, respectively. We set the communication round number to 10. Learning with small communication round number (e.g.. 10) denotes learning under limited communication budget. For the methods conducting local model training on clients, i.e., FedAvg, FedProx, FedDistill, FedEnsemble, and FedGen, set the local updating number to 20, and the batch size number to 32. In our method, we set meta knowledge size for each datasets based on their different characteristics (e.g., R,, the class number, sample size, etc): 20 per class for MNIST; 100 per class for SVHN; 100 per

---

Published as a conference paper at ICLR 2023
SVHN
CIFAR100
25
(8))
60-
80-
60
40
20
Fodiug
000
15-
RedErsemble
Fedtan
20
FodDkct1
20
redki
50
100
150
200
50
100
150
200
200
30
100
150
200
Figure 2: Convergence rate comparisons MNIST, SVHN, CIFARIO, and CIFARI00.
1.0. The x-axis
represents communication round numbers.
MNIST
SVHN
CIFAR10
CIFAR100
55
30
95
52-
20
MAP 1967
50
26
90-
47-
70-
45
alpha-0.5
85
alpha-0.75
42-
alpha=1.0
80
20
10
20
30
100
20
60
80
Figure 3: Impact of meta knowledge size on final performance. #round=10. The x-axis represents meta knowledge sizes.
Table 2: Impact each designed mechanism.
w/o Iter
w/o Sharing
w/o pseudo knowledge
w/ dynamic weights
Ours
CIFARIO
28.15%
45.79%
46.71%
47.33% 47.16% 26.43%
CIFARI00
25.56%
26.10%
26.74%
class for CIFAR 10; 40 per class for CIFAR100. We run three trials and report the mean accuracy performance (MAP).
3.3 COMPARATIVE STUDIES
Compared with Prior Works: We evaluate FedMK and compare it with relevant prior works under limited communication budgets (10 rounds) on four datasets, including MNIST (LeCun et al. 2010).
SVHN (Netzer et al. 2011), CIFARIO (Krizhevsky & Hinton| (009), and CIFAR 100 (Krizhevsky & Hinton| 2009). The results are reported in Table As shown in the results, under limited communication budgets, all prior works fail to achieve good performance, while FedMK outperforms them by significant margin. For instance, MNIST, FedMK achieves MAP of 93. 63% (a 1), outperforming the competing methods. On SVHN FedMK reaches a MAP of 75.85% (a = 1) in 10 rounds, while the competing methods have not converged yet. The superior performance of FedMK can be attributed to two aspects. Firstly, unlike competing methods that conduct local model training with local private data, FedMK trains a global model based on meta-knowledge from all active clients, which reduces the bias in the model. Secondly, the dynamic weight assignment and meta knowledge sharing mechanisms designed in FedMK make the meta knowledge extraction more stable, leading to better performance than competing methods
Convergence rate comparisons: We compare the convergence rates of all methods and show the performance curves with respect to communication rounds in Figure As expected, our method achieves a high convergence speed compared to all competing methods. On all datasets, our method achieves satisfactory performance in much fewer communication rounds. The results shown in both Table I and Figure 2 demonstrate the effectiveness and efficiency of our proposed method.
Impact of the meta knowledge size: We study the effect of varying meta knowledge sizes on the performance of our method on four benchmark datasets. The results are presented Figure The results show that the final performance is influenced by the meta knowledge size. For instance, the MAP score on CIFARIO improves as the meta knowledge size increases. Furthermore, we find that the optimal meta knowledge sizes can differ between datasets. For example, the highest MAP score on MNIST is achieved when the meta knowledge size is set to 20, while the optimal meta knowledge sizes on the other three datasets are different. Further discussions on the relationship between meta knowledge size, communication rounds, and performance can be found in the Appendix.
'A more detailed comparison with FedGen can be found in the Appendix.
7

---

Published as a conference paper at ICLR 2023
CIFARIO
CIFAR100
90
Peshice
20
TedSan
TesMC
15
70
10
Figure 4: The performance curve w.r.t. the active client number. a= 0.5, #inode 20, #round = 10. The x-axis represents the active client numbers.
Feciua
Fedkisembe
12
32-
10
Figure 5:
Performance comparisons w.r.t. different round numbers. Left column: 20 rounds; middle column: 30 rounds; right column: 40 rounds. The x- x-axis represents different values.
Impact active client numbers: We examine the effect of varying the number active clients on proposed method and competing works. We conduct experiments with active client numbers set to 5, 7, and 10, respectively. The performance results are displayed in Figure The results illustrate that our proposed method consistently outperforms all comparison methods regardless of the number of active clients.
Impact of communication rounds: We conduct experiment on the CIFARIO dataset analyze the impact of communication rounds. We compare the performance of different methods when the number of communication rounds is 20, 30, and 40. The results of the comparison are presented in Figure 5 The results demonstrate that our proposed method consistently outperforms the competing methods by a remarkable margin.
Impact of designed mechanisms: We conduct study on CIFAR10 and CIFAR100 to explore the impact of each designed mechanism. We report the performance in Table 2 where "w/o Iter" means that we conduct local meta knowledge condensation and global model update only once "w/o Sharing" means that we do not adopt meta knowledge sharing between clients, "w/o pseudo meta knowledge" means that there is no learnable constraint on the server, "w/o dynamic weights" means that we update D by treating all samples in D equally.
As shown Table 2 there is drastic performance degradation when there is no iterative collab- oration between meta knowledge extraction and central model training ("w/o Iter"). Making the meta knowledge condensation and central model training running in an iterative manner improves the performance significantly, e.g. CIFAR10, MAP Moreover, the two designed mechanisms, ie.. the meta knowledge sharing between clients, dynamic weights assignment, significantly boost the MAP score.
increases from 28.15% to 45.79%.
Evaluation on the pathological non-iid setting: We conduct an experiment on MNIST under the pathological non-iid setting (Huang et al., (02T). In the experiment, we the node number 20, the active node number to 10, the number of classes on each client to 5. to 1.0. We compare all methods under limited budget communication (i.e., 10 rounds). As seen in Table our method achieves better performance compared to the competing methods.
The privacy concerns of meta knowledge: Dong et (2022) conducted theoretical analysis of the relationship between synthetic data and data privacy. Based on Proposition 4.10 in (Dong et al.) 2022), when using distilled data (meta knowledge), the leakage of membership privacy that directly
relates to personal privacy is O( ((card(()) card(p)) As indicated by Proposition 4.10, when the cardinality
w/o Iter" be regarded as one-shot FL. like Zhou al.
2020a
8

---

Published as a conference paper at ICLR 2023
Table 3: Result comparisons MNIST under the pathological non-iid setting.
FedAvg
FedProx
FedEnsem
FedDistill
FedGen
Ours
MAP (%)
75.11%
72.99%
76.43%
71.19%
86.44%
88.85%
of the synthetic data is much fewer than that of the original data, only limited (i.e., O card information can be obtained by the adversary with membership inference attack.
cardd (D)
In the Appendix, we provide more experimental analysis and comparisons with prior works.
4 RELATED WORK
Federated Learning. As a
solution, federated learning (Li et al. 2019) provides a new training manner to learn models over a collection of distributed devices. In federated learning, the data is located on distributed nodes and not shared. Learning a model without exchanging local data between nodes minimizes the risk of data leakage but increases the difficulty of training. To address this issue, FedAvg (McMahan et al. 2017) was proposed to obtain a global model by aggregating local models trained active clients. To further enhance local training. personalized FL methods
(Fallah et al.] (020bb] Dinh et al., 2020 Dai et al.) 2022) and knowledge distillation- FL methods
(Seo al. 20200] Zhu et al. 2202]] Zhang et 2022) have been proposed. For a comprehensive
understanding of FL, readers are referred to (Kairouz et al. 2021 Tan et al. 2022 Gao et al. (022).
Compact Data Representation. Generally, prior works compressing large scale data into a small set can be categorized into two main branches: data selection and data compression. Data selection methods (Rebuffi al., 2017; Castro et al. 2018 Aljunni et al. 2019 Sener & Savarese, 2018) select the most representative samples from the original data based on predefined criteria. How to select an appropriate criterion based on the given data and task is not a trivial issue. To overcome the aforementioned limitations, synthesizing new samples rather than selecting existing samples
becomes a more preferable solution. The methods (Wang et al. 2018 Zhao et al.] 2021 Nguyen
et al. 202 Zhao & Bilen
2021 Nguyen et al.] [021b; Vicol et al. 2022] Cazenavelte et al.) [2022]
Zhou et al.
2022 Loo et al. 2022] Wang et al.) 2022 Lee et al. 2022] 2ianniiang Jiang et al. [0222] Du et al.
2022 Zhao & Bilen 2023 Loo et al. (023) design different solutions for generating synthetie data
based on given datasets. In those methods, the generated data can replace the original data in the
model construction process. However, those prior synthetic data generation works require data be
localized in a centralized manner.
Federated Learning via Synthetic Data. A few attempts (Goetz & Tewari 2020 Zhou al.
2020b Yoon et al. 2021: Xiong et al., 2022 Hu et al. 2022 Kim & Choi 2022; Behera et al.
2022 Song al. (2022 have employed synthetic data generated from local data for FL. Zhou et al.
(2020b) and Song et al. (2022) introduced one-shot FL methods by utilizing distilled data. Goetz & Tewari (2020) proposed a method for FL that utilizes fata-poisoning to generate synthetic training data. Behera et al. (2022) utilized synthetic data generated by a Generative Adversarial Network
for FL. training. Concurrently, Xioog et al. (2022) used gradient matching to synthesize data from the original data and used the synthetic data to train global model. Hu et al. (2022) and Kim &
Choil (2022) also adopted the distilled data for training global model in FL. Compared to these concurrent works (Xiong al., 2022, Hu al. 2022 Kim & Choi 2022). our method introduces a mnowl-e sharing mechanism and a dynamic weight assignment strategy, significantly increasing the informative content of the meta knowledge. In this way,
further speed up the
convergence of the global model. In FedMix YYoon et al.] 2021). Yoon et al. proposed to share the averaged local data among clients for knowledge sharing. However, FedMix (Yoon al. 202T) still follows the training paradigm of FedAvg (i.e., training local models on clients and aggregating a global model on the server), incurring a higher communication cost.
CONCLUSION
In this paper, we present a new federated learning paradigm driven by meta knowledge, dubbed FedMK. to obtain an effective and fast-converging model. With the help of the proposed paradigm, FedMK can train a powerful model even under a limited communication budget (e.g., 10 communica- tion rounds), decreasing the communication cost significantly. Moreover, our designed mechanisms, i.e., meta knowledge sharing, dynamic weight assignment, and a learned constraint, collectively facilitate the central model training, benefiting FedMK outperforming all competing methods.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by S*SRAR Career Development Funding (CDF) Award (Grant No: 222D800031), SERC (Science and Engineering Research Council) Central Research Fund (Use- Inspired Basic Research), and in part by the Singapore Government's Research, and Innovation and Enterprise 2020 Plan (Advanced Manufacturing and Engineering Domain) under programmatic Grant A18AIb0045, Xin YU was supported by the Australian Research Council Discovery Project (DP220100800) and DECRA (DE:23100477).
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurlPS, 2019.
Roberto Battiti. First-and second-order methods for learning: between steepest descent and newton's method. Neural computation, 1992.
Monik Raj Behera, Sudhir Upadhyay, Suresh Shetty, Sudha Priyadarshini, Palka Patel, and Ker Farn Lee. Fedsyn: Synthetic data generation using federated learning. arXiv, 2022.
Francisco M Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, 2018.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR, 2022.
Xu Chencheng, Hong Zhiwei, Huang Minlie, and Jiang Tao. Acceleration of Federated Learning with Alleviated Forgetting in Local Training, In ICLR, 2022.
Rong Dai, Li Shen, Fengxiang He, Xinmei Tian, and Dacheng Tao. DisPFL: Towards communication- efficient personalized federated learning via decentralized sparse training. In ICML, 2022.
Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. The Advantage of Conditional Meta- Learning for Biased Regularization and Fine Tuning. In NeurlPS, 2020.
Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In NeurIPS, 2020.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In ICML, 2022.
Jiawei Du, Yidi Jiang, Vincent TF Tan, Jocy Tianyi Zhou, and Haizhou Li. Minimizing the accumu- lated trajectory error to improve dataset distillation. arXiv, 2022.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient-based model-agnostic meta-learing algorithms. In AISTATS, 2020a.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with
theoretical guarantees: A model-agnostic meta-learning approach. In NeurIPS, 2020b.
Dashan Gao, Xin Yao, and Qiang Yang. A survey on heterogeneous federated learning. arXiv preprint arXiv:2210.04505. 2022.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv, 2020.
Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu. Fedsynth: Gradient compression via synthetic data in federated learning. In NewrIPS Workshop, 2022.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In AAAI, 2021.
Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z Pan. Delving into effective gradient matching for dataset condensation. arXiv, 2022.
10

---

Published as a conference paper at ICLR 2023
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 2021.
Seong-Woong Kim and Dong-Wan Choi. Stable federated learning with dataset condensation. J. Comput. Sci. Eng., 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989.
Yann LeCun, Corinna Cortes, and Chris Burges. Mnist handwritten digit database, 2010.
Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensa- tion with contrastive signals. In ICML, 2022.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang. Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: vision, hype and reality for data privacy and protection. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2019.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys, 2020.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In NeurIPS, 2022.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communictiion-eficieen learning of deep networks from decentralized data. In AISTATS, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning.
Workshop, 2011.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-leaming from kernel ridge- regression. In ICLR, 2021a.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurlPs, 2021b.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In NeurIPS, 2019.
Sylvestre-AIvee Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. CVPR, 2017.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
Hyowoon Seo, Jihong Park. Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim. Federated knowledge distillation. arXiv preprint arXiv:20/1.02367 2020
Zhiwen Shao, Jianfei Cai, Tat-Jen Cham, Xuequan Lu, and Lizhuang Ma. Unconstrained facial action unit detection via latent feature domain. IEEE Transactions on Affective Computing, 2021.
Rui Song, Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll. Federated learning via decentralized dataset distillation in resource-constrained edge environments. arXiv, 2022.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions Neural Networks and Learning Systems, 2022.
11

---

Published as a conference paper at ICLR 2023
Junjiao Tian, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, and Zsolt Kira. Striking the right balance: Recall loss for semantic segmentation. In ICRA, 2022.
Paul Vicol, Jonathan P Lorraine, Fabian Pedregosa, David Duvenaud, and Roger B Grosse, On implicit bias in overparameterized bilevel optimization. In ICML, 2022.
Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang. Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022.
Ruohan Wang, Yiannis Demiris, and Carlo Ciliberto. Structured prediction for conditional meta- learning. In NeurlPS, 2020.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv, 2018.
Hongda Wu and Ping Wang. Node selection toward faster convergence for federated learning on non-iid data, arXiv preprint wwriiv:200.07000: 2021.
Zeke Xie, Xinrui Wang, Huishuai Zhang, Issei Sato, and Masashi Sugiyama, Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In ICML, 2022.
Yuanhao Xiong, Ruochen Wang, Minhao Cheng, Felix Yu, and Cho-Jui Hsieh. FedDM: Iterative distribution matching for communication--fficccenn federated learning. In NeurIPS, Workshop, 2022.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 2019a.
Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 2019b.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In ICLR, 2021.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR, 2020.
Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. CVPR, 2022.
Bo Zhao and Hakan Bilen. Dataset Condensation with Differentiable Siamese Augmentation. In ICMI., 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.07999 2020a.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning, arXiv, 2020b.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In NeurIPS, 2022.
Ligeng Zhu and Song Han. Deep leakage from gradients. In NeurlPS, 2020.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In ICML, 2021.
12

---

Published as a conference paper at ICLR 2023
Table 4: Results with 10 rounds.
Setting
FedAvg
FedProx
FedDistill
FedEnsem
FedGen
FedMK
MNIST
a=0.10
61.95%
61 .1.4%%
58.46%
67.89%
64.83%
77.37%
69.52%
72.23%
90.07%
SVHN
a=0.10
20.10%
18.39%
25.44%
24.60%
24.38%
57.24%
a=20.55
23.56%
25.01%
22.70%
23.21%
28.79%
65.66%
CIFARIO
a=0.10
23.71%
21.88%
24.93%
24.80%
20.16%
38.45%
23.98%
CIFARIOO
a=0.10
10.19%
9.41%
12.41%
10.64%%
10.79%
18,6218
a=00.5
11.73%
10.43%
8.73%
12.42%
8. 8,22%
22.14%
Appendix Meta Knowledge Condensation for Federated Learning
A ADDITIONAL EXPERIMENTAL RESULTS
Compared with Prior Works when setting a to 0.1 and 0.25: We set the value in Dirichlet distribution ((a) (Zhu et al. 2021) to 0.1 and 0.25, and run all methods under limited communication budgets (10 rounds) on four datasets. We report the results in Table 4 As shown in Table when communication budgets are limited (10 rounds) and the a value is set to 0.25 and 0.10, our method can still learn model outperforming competing works by a remarkable margin.
We investigate the impact of weight assignment mechanisms in our proposed method. Here, we compare our Dynamic Weight Assignment mechanism with Class Balance Weight strategy that has been used in prior works (Tian et al.) [0222 Shao et al. 202T]. This strategy assigns weights each sample during training based on the occurrence rate of the corresponding class. We perform an experiment on the MNIST dataset and present the results in Table 6) The results demonstrate that our proposed Dynamic Weight Assignment mechanism outperforms the Class Balance Weight strategy.
Impact of the weight assignment mechanism:
We conduct experiments on the SVHN dataset to analyze the effect of the number of nodes. Specifically, we set the node number to 200 for comparisons. Typically, as the number of nodes increases, the amount of data on each client becomes sparse, making learning more experiment shows that even
Impact of node numbers:
challenging and leading to decrease in performance. However, when the node number is increased to 200, our method (70.51 %) still outperforms FedGen (66.31%) by substantial margin.
Impact of the smooth parameter in Eq. 7 We conduct a study on four datasets to explore the We set the value 1.0, 5.0, and 10.0, respectively. As
impact of smooth parameter in Eq. 7 shown in Table 5 we achieve the highest performance in most cases when we set r to 5.0. Therefore, we set 7 in Eq. to 5.0 for all our experiments.
Communication cost comparisons: We analyze the communication cost of our method on MNIST. In each round of FedMK, we need to upload and download the meta knowledge, and download trained model. In the experiment, the meta image size is 20 per class, each of which is 4 Bytes x 28 x 28 (sizeof(fooat2)) x h). Thus, (meta knowledge), i.e., 4 Bytes x 28 x 28 x 20 x 10, and the downloading cost is 1677.2K (meta knowledge and a model), i.e., 627.2K + 105K x 10. In total, the communication cost for FedMK is 16M, i.c., (627. 11, 677.2K) x 10 (# rounds). As shown in Figure 2 to achieve comparable performance on MNIST, FedAvg has to run around 200 rounds. In this case, the communication cost for FedAvg is 420M, i.e., 105K 10 x 2 200 (model size #active node x 2 x #communication). This indicates that to obtain models with good performance, FedAvg and its variants require much higher communication costs.
each round, the uploading cost of FedMK is 627.2K
13

---

Published as a conference paper at ICLR 2023
Table 5: Impact of the smooth parameter
r=1.0
T=5.0
7=10.0
MNNST
a=0.50
91.70%
92.95%
91.79%
91.90%
92.86%
92.23%
a=1.0
91.53%
91.91%
SVHN
a=0.50
72.24%
74.11%
71.809%
a=0.75
71.47%
74.90%
74.03%
a=1.0
71.74%
74.84%
72.19%
CIFARIO
a=0.50
a=0.75
48.17%
49.04%
49.27%
a=1.0
47.82%
50.32%
48.54%
CIFARI00
a=0.50
26. 1..66%
26.74%
26.45%
a=0.75
26.93%
27.43%
26.98%
a=1.0
25.43%
28.20%
26.15%
Table 6: Comparisons on MNIST under different weight assignment mechanisms.
Method
0.5
0.75
1.0
Balance Weight
92.17% 92.95%
91.84%
91.69%.
Our method
92.86%
93.63%
Table 7: EMDs with respect meta-knowleege sizes.
Meta- Knowledge size(S)
10
30
50
70
Difference w.t.. S-10
0
10
20
40
50
60
70
EMD(metos, metesa)
0.000
0,023
0.054
0.046
0.053
0.045
0.045
0.043
B DISCUSSIONS
Why does utilizing meta-knowledge decrease the required communication rounds? Our method utilizes extracted meta knowledge as normal training data to train global model on the server. The meta knowledge is extracted from original data via a bi-level optimization, which encodes the "gradient of gradient" with respect to the model. The optimization methods based on the second order gradient generally have higher convergence speed than the methods using the first order gradient (Battiti 1992. Xie et al.) 2022). Therefore, utilizing meta- vetta-nnoleedge endows our algorithm with a fast convergence speed and decreases the communication round number.
Why increasing the meta knowledge size can not necessarily improve final performance? In the meta knowledge extraction process, the calculated meta-knowlest knowledge in each batch represents the average of the model update direction. The average gradient is stable when the batch number increases certain range (from x1 x10). As result, increasing the meta-knowledge sizes does not necessarily increase the performance. Intuitively, the meta knowledge is highly dense and compressed, encoding the knowledge from original data (Zhou et al.) 2022). In principle, using the meta knowledge approximates employing the original data. As the amount of information in the original data is constant, the training performance will not necessarily increase as the meta knowledge size increases.
We conduct an experiment on MNIST show the information change between meta knowledge with respect to different sizes. Concretely, we set the meta- knowllee size (S) as 10, 20, 30, 40, 50, 60, 70, and 80, respectively. As earth mover's distance (EMD) has been utilized to compute a structural distance between two data sets to determine their similarity (Zhang et al. 2020), use it to evaluate differences according to meta- knwwlede with different sizes. The results are listed Table 7 be seen that the EMDs with respect meta sittaa-nwwledgge sizes stable, indicating the amount of information in the meta-knan veta-knnoleede does not change significantly with respect to the meta knowledge size.
14

---

Published as a conference paper at ICLR 2023
67
2
3
4
7
9
original image
meta- feta-knowledge
restored data by deep leakage based on neta-knowledge
Figure 6: The visualization of original images (the row), extracted meta-knowledge (the middle row), and restored data by deep leakage based extracted meta-knnwledd (the bottom row).
The privacy concerns of meta knowledge. We think that the use of generated meta knowledge can protect original data privacy. This can be argued from three perspectives. Firstly, as shown in the function for the upper-level problem, ie.,, De +- De aV De D9), never returns the original local data D What receive in the outer loop (the upper-level problem) is the updated (or initialized when there is update) meta knowledge D
Secondly, based on Deep Leakage (Zhu Han 2020), we conduct an experiment on MNIST to explore the possibility of restoring data from extracted meta knowledge. The results shown in Figure The original images are shown the top row, and the extracted meta knowledge is shown in the middle row. We feed the extracted meta knowledge and trained model to Deep Leakage (Zhu & Han 2020), which is one of the state-of-the-ar methods for restoring data from leaked knowledge. The data restored by Deep Leakage is shown in the bottom row. can be seen that it is hard to construct correspondence between entries in restored data and original data.
Thirdly, Dong al. (Dong et al. 2022 conducted theoretical analysis of the relationship between synthetic data and data privacy. Based on Proposition 4.10 in (Dong et al. 2022), when using distilled data (meta knowledge), the leakage of membership privacy that directly relates personal privacy is
O card D cardd
As indicated by Proposition 4.10, when the cardinality of the synthetic data is much
fewer than that of the original data, only limited (i.e., O the adversary with membership inference attack.
cardd )) information can be obtained by card (D)
C THE COMPARISON WITH PRIOR WORKS
The difference between FedMix (Yoon et al. 2021 and FedMK: Our method differs from FedMix (Yonn et al. 2021) significantly in three aspects.
Firstly, our method adopts a different training paradigm. FedMix (Yoon et al. 2021 follows the training paradigm of FedAvg, i.e. training local models on clients and aggregating global model on the server. Our method conducts the meta knowledge extraction on clients, and trains global model on the server based on the uploaded meta knowledge. Our method significantly decreases the communication cost it only needs to upload the meta knowledge.
Secondly, our method has faster convergence speed. This is because our method utilizes the second- order gradient information. In FedMix (Yoon et al. 2021), only the first-order gradient information utilized. In our method, the meta knowledge encodes the "gradient of gradient" with respect to the model. The optimization methods based on the second-order gradient generally have higher convergence speed than the methods using the first-order gradient (Xie et al. 2022). We conduct an experiment MNIST (a 1.0) compare with FedMix (Yoon et al. 2021) When communication
15

---

Published as a conference paper at ICLR 2023
budgets are limited (i.e., 10 rounds), FedMix only achieves 74.68% MAP, whereas method can achieve 93.63% MAP.
Thirdly, to ameliorate the heterogeneity issue in FL, specifically design two mechanisms in federated meta knowledge extraction on local clients, i... dynamic weight assignment and meta knowledge sharing. There are such mechanisms in FedMix (Yoon al. 202T). We demonstrate
that the two mechanisms can improve the final accuracy significantly (e.g., 1.5% improvement on CIFARIO).
In summary, compared to FedMix (Yoon et al. 2021). our method adopts a different FL training paradigm, utilizes the second-order gradient information in model learning, and designs two specific mechanisms for ameliorating the heterogeneity issue.
The difference between FedSynth (Hu et al., 2022), FedDM (Xiong al. 2022), FedDDC (Kim &
Choi. 2022) and FedMK: Although our work and FedSynth (Hu et al. 2022). FedDM (Xiong et al.
2022 . FedDDC (Kim & Choi, 2022) all use condensed data in FL. there are significant differences between them. Please allow us to illustrate the differences from three aspects.
Firstly, on local clients, our method designs a different federated meta knowledge extraction solution. Prior works such as (Xiong et al., 2022, Kim & Choil 2022) directly utilize gradient matching strategy for data condensation. We adopt a bi-level optimization solution in our method. To further ameliorate the heterogeneity issue in FL., we specifically design two mechanisms in our meta knowledge extraction process, i.e., dynamic weight assignment and meta knowledge sharing. There no such mechanisms in prior works. In experiment, we demonstrate the effectiveness of the two mechanisms (e.g., 1.5% improvement on CIFAR10).
Secondly, on the server, prior works only the uploaded condensed data to finetune the aggregated global model. In our method, to further ameliorate data biases among diverse clients, we introduce additional synthetic training samples generated by conditional generator in the global model training.
Last but not the least, method adopts different FL learning paradigm compared to FedDC (Kim & Choil 2022). Kim et al. (Kim & Choil 2022) conduct local model training on clients, global model aggregation and fine-tuning
knowledge extraction clients, global model training on the server. As our method does not need to upload the local trained model each communication round, our method significantly decreases the communication cost.
the server. Our method conducts
In summary, there are significant differences between prior works and our proposed method in various aspects.
The difference between GedGen Zhu et al. 2021 and FedMK: There are significant differences between FedGen (Zhu et al. 2021 and our FedMK, which are listed as follows:
Firstly, FedGen utilizes original data on local clients to train local models, while our method conducts meta knowledge extraction synthesize meta knowledge, which is used for global model training on server. In FedGen, the trained local models might diverge due to the data distribution variations among clients.
Secondly. FedGen constructs global model by aggregating uploaded local models; while our method learns a global model based on meta knowledge uploaded from clients. our method, the global model learning utilizes knowledge from all active clients, therefore mitigating the bias issue compared to FedGen.
Thirdly, the conditional generator in FedGen is trained on the server and transmitted to clients. On clients, it is used as a constraint in the local model training. On the contrary, the conditional generator in our method is trained and utilized on the server, participating in the global model training. Compared to FedGen, our method has a less communication cost without performance deterioration.
In conclusion, compared to FedGen, our method performs more effectively and efficiently.
16


Reviews:
[{"review_id": "XwqqD_Ezovk", "reviewer": "Reviewer_ouFd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new learning scheme in Federated Learning that communicates knowledge between clients in the form of meta knowledge generated by dataset condensation. In this case, the global model aggregation in the cloud is replaced with a \"meta training\" process in the cloud using the combined meta knowledge from all participating clients in each communication round. The authors also proposed several modules that co-work with the main meta knowledge scheme, such as pseudo knowledge in the cloud using a generating model and dynamic weights.\n\nThe idea is clearly novel and the empirical effect of reducing communication rounds needed is impressive. The authors also show that other designed modules all contribute to the superiority of the overall framework.", "summary_of_the_review": "This is an outstanding work with novel technical contributions and impressive empirical performances.\n\n=====================================\n\nNote for the late submission of this review. I am very sorry that I submitted my review this late. I think I wrote the review locally on my computer but forgot to post it to OpenReview.\n\nMy original evaluation on this work without referring to other reviews was very positive, with only  some minor issues on the math formulation and some statements, such as:\n\n1. In equations (4,5,7) of the original draft and the equations (3,5,6,8) of the updated draft as of 11/30/2022, the $\\nabla$ operator misses subscripts that indicate with respect to which the derivatives are taken. The subscripts are needed because the derivatives are taken w.r.t. different inputs to the loss function $\\mathcal{L}^c$.\n2. In equation (6) of the original draft, the trailing parenthesis is missing. It is still missing in the updated draft.\n3. The $\\mathcal{L}^c$ notation is abused in the equation (7) of the original draft. The authors should mention that the notation comes with a little abuse.\n4. The authors said above equation (7) in the original draft that \"Apparently, the weight of each sample is inversely proportional to its prediction loss\", which is contrary to the formulation. It seems to have been fixed in the updated draft.\n5. There has been some unclarity of the bilevel optimization formula, but it has been addressed in the discussions between the authors and other reviewers.\n\nI encourage the authors to address the above points, while they have not influenced my overall evaluation of this work so I hope it won't make the rebuttal process messy.", "strength_and_weaknesses": "Clearly the meta knowledge sharing in place of the classic model weight aggregation is the most important and strong contribution of this work. In so heterogeneous scenarios in federated systems, model weight aggregation is really the pain point and potentially very bad in terms of communication efficiency. However, this work pointed out an alternative with clear empirical evidences that support the superiority of  it.\n\nThe only weakness I found in this work is the lack of discussion about the influence of the proposed condensation mechanism on the privacy preserving property of FL. This is crucial because the local meta knowledge will finally be sent to the cloud. While the authors didn't include discussions on privacy in the draft, I highly encourage them to at least add some references to existing analysis work about dataset condensation, which is not new technically in non-FL scenarios.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "V7xAoqQOKK", "reviewer": "Reviewer_StBv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new meta-knowledge driven FL paradigm to optimize the FL training with limited communication rounds. Besides, two strategies namely Dynamic Weight Assignment and Meta Knowledge Sharing are proposed to improve the performance. ", "summary_of_the_review": "Overall, the paper presents a new solution to tackle the FL training problem in case of limited communication rounds. The core performance is well evaluated. However, there still remains some issues that could be further improved. Thus, I recommend this paper a borderline accept score. ", "strength_and_weaknesses": "Strength:\nS1: A new FL paradigm is proposed to deal with the limited communication budgets problem by leveraging condensate meta knowledge instead of the raw training data.\nS2: The proposed paradigm is significantly better than existing FL solutions when facing limited communication rounds.\nS3: Extensive experiments are conducted to show the effectiveness of the proposed method in terms of accuracy, communication efficiency (rounds) and also data leakage.\n\nWeaknesses:\nW1: I wonder if it is fair to evaluate the communication budgets using the metric of communication rounds, because unlike the competitors that the clients and the servers transmit the gradients or model weights, the FMKE framework proposed in this paper needs the clients to send the meta data, which have different amount of size.\nW2: It is unknown if the bi-level optimization is optimal for FMKE framework. It is expected to compare the bi-level optimization with other baselines, e.g., simultaneously optimizing the two objectives.\nW3: While dynamically adjusting the weights of the training samples has been widely studied, it lacks of comparisons between the proposed Dynamic Weight Assignment with the existing methods, such as AdaBoost.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AG8ONROmyU", "reviewer": "Reviewer_FoXU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach for Federated learning. The idea is to first distill the private data of each client (with respect to the current initialization of their model weights) and send them to the server. On the server side, a global model will learn from these synthetic data and broadcast the learned weight back to the clients. The proposed method is demonstrated to outperform other FL benchmarks on several datasets.", "summary_of_the_review": "I recommend a rejection since I am not fully convinced about the distillation objective. I'm willing to change my score if the authors can clarify my concerns. I'm personally very curious about the exceptionally good results. I do hope that my assessment is wrong, because the method otherwise seems like a good contribution. I would suggest the authors to release the implementation on an anonymized github repo, so reviewers can properly investigate.\n\n-------\n\nUpon reading the authors' responses to my questions and revisions of the manuscript, I am convinced that the method is sound (Another minor comment: The revision is technically not a bilevel optimization anymore since both parameter updates happen simultaneously). As promised, I'm happy to change my scores accordingly.", "strength_and_weaknesses": "Strength: Idea is clear and straight forward. The empirical result seems great. \nWeakness: The objective formulation seems problematic and I cannot make sense why it works in practice.\n\n-- The notation in Eq. (3) does not make sense to me. $D_c$ is a constant, not a variable. Why is it being optimized? I take it as this is some random variable $D'$ that has the same desired dimension as the condensed dataset. If so, how is the local dataset used?\n\n-- Another major concern is that Eq. (3) seems to have a trivial solution? First off, i think it is improper to have loopy dependency in an optimization objective ($\\widehat{D}_c$ depends on $w_\\ast$ and vice versa). Maybe it's better to write it as $\\widehat{D}_c, w_\\ast = \\underset{w_c, D_c}{\\mathrm{argmin}} \\dots $, i.e., the bilevel optimization is an approximate procedure to solve for this objective and should not be the objective itself. Regardless of the notation, I think the optimization will converge when $\\widehat{D}_c$  and $w_\\ast$ are obtained such that $\\mathcal{L}(\\widehat{D}_c, w_\\ast) = 0$. This is probably simple to achieve when there is no constrained on the meta dataset. For example, we can construct a dataset where all datapoints are labelled as class 1, and a model that always predicts class 1 with zero uncertainty (setting the bias of the prediction layer to $\\mathbf{e}_1$ and zeroing out all other weights would probably suffice). Thus, a trivial solution exists and theoretically this could result in very bad performance. This seems intriguing given the very good result observed in the empirical studies. My best explanation for this good performance is that the local models only perform their bilevel optimization for a few steps, which is likely not enough to find such an exploit. \n\n-- This objective does not seem to explicitly prevent the condensed dataset from being exactly the same as the local data. After all, they live in the same input space, so it could happen. I understand that the paper conducts an empirical study to verify this is not the case, but when you deal with privacy, that is simply not good enough. What will happen when the global model somehow has zero loss on the client data? Then, the outer loop doesn't need to do anything but returning a subset of its local data (since loss can't be improved further). This seems like a big privacy vulnerability to me --- Imagine a client with only 10 data points on a binary classification task, then a malicious server would only need to send $2^10$ specifically constructed weights to eventually hit the perfect loss and hence recover exactly the client data? I think the meta dataset somehow has to have a different dimension than the original dataset for this to work.\n\nSome other (minor) concerns:\n\n--  We run three trials and report the mean accuracy performance (MAP) -- is the MAP averaged over both trials and clients? I would recommend showing some standard deviations.\n\n-- How many update steps per round of local data distillation?\n\n-- How is evaluation done? Does each local model of FMKE fine-tune the global weights before testing on their own data, or do they directly use the global weights trained on synthetic data? Is it the same for other benchmarks?\n\n-- Can the authors show the MAP curve of FMKE beyond iteration 10 in Fig. 2? It's not just about achieving high performance, but also about demonstrating the stability of the method. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1fOB5_dGsQ", "reviewer": "Reviewer_QKpf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a new FL solution. In the designed solution, the local clients distill meta knowledge based on local private data and shared knowledge from other clients, the distilled meta knowledge is uploaded to the server for global model training. As the global model training is based on meta knowledge from all active clients, the bias issue can be mitigated and the convergence can be accelerated experimentally. Additionally, the authors designed two novel mechanisms to improve the meta knowledge extraction on local clients, i.e., meta knowledge sharing and dynamic weight assignment. Both of the designed mechanisms are novel and technically correct. The authors conduct extensive experiments and ablation studies to demonstrate the efficacy and efficiency of their method.", "summary_of_the_review": "This work is interesting and novel. The authors designed a new learning paradigm and designed two novel mechanisms in the learning. The motivation is explained well and the technical details are clear. They conduct extensive experiments to demonstrate the efficacy and efficiency of the designed solution.", "strength_and_weaknesses": "Strength:\n \n-- The authors designed a new learning solution. Unlike prior FedAvg-based methods, the authors conduct meta knowledge extraction and upload the meta knowledge to a server. They use the meta knowledge from all active clients as normal training data for training a global model, mitigating the bias issue. The meta knowledge extraction and global model training are conducted alternatively.\n \n-- The designed mechanisms, i.e., meta knowledge sharing and dynamic weight assignment, are novel and technically correct.\n \n-- The authors conduct extensive experiments and ablation studies, and provide convincing experimental results to demonstrate the efficacy and efficiency of their method.\n \n-- The writing is clear and easy to understand. The whole work is organized well.\n \nWeakness:\nTwo minor issues:\n-- It would be better to place the Algorithm pseudo code in the draft rather than supplementary material.\n-- Why name the extracted knowledge as meta knowledge? Is there any specific reason? If yes, it would be better to provide the reason in the paper.\n \nClarity, quality, novelty and reproducibility\nClarity: The paper is well-written and easy-to-follow.\nQuality: The technical quality is good. The method proposed in this paper is extensively evaluated. Novelty: The idea of this paper is novel. I haven't seen the same idea in prior works. Reproducibility: This work is with good reproducibility.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster