DENSE: Data-Free One-Shot Federated Learning
Authors: Jie Zhang
Chen Chen
Bo Li
Lingjuan Lyu
Shuang Wu
Shouhong Ding
Chunhua Shen
Chao Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: federated learning
data-free knowledge distillation
one-shot FL

Abstract: One-shot Federated Learning (FL) has recently emerged as a promising approach, which allows the central server to learn a model in a single communication round. Despite the low communication cost, existing one-shot FL methods are mostly impractical or face inherent limitations, \eg a public dataset is required, clients' models are homogeneous, and additional data/model information need to be uploaded. To overcome these issues, we propose a novel two-stage \textbf{D}ata-fre\textbf{E} o\textbf{N}e-\textbf{S}hot federated l\textbf{E}arning (DENSE) framework, which trains the global model by a data generation stage and a model distillation stage. DENSE is a practical one-shot FL method that can be applied in reality due to the following advantages:
(1) DENSE requires no additional information compared with other methods (except the model parameters) to be transferred between clients and the server;
(2) DENSE does not require any auxiliary dataset for training;
(3) DENSE considers model heterogeneity in FL, \ie different clients can have different model architectures.
Experiments on a variety of real-world datasets demonstrate the superiority of our method.
For example, DENSE outperforms the best baseline method Fed-ADI by 5.08\% on CIFAR10 dataset. 


DENSE: Data-Free One-Shot Federated Learning
Jie Zhang
Chen Chen
Bo Li
Lingjuan Lyu
Shuang Wu
Shouhong Ding²
Chunhua Shen"
Chao Wu
'Zhejiang University
2 Youtu Lab, Tencent
Sony Al
zhangjee, chao.wu, cc33) ) @ @zju.edu.cn. Lingjuan Lv sony.com Ly (libraboli, calvinwu, ericshding) @tencent.com, chunhua @ me.com
Abstract
One-shot Federated Learning (FL) has recently emerged as promising approach, which allows the central server to learn a model single communication round. Despite the low communication cost, existing one-shot FL. methods are mostly impractical or face inherent limitations, e.g., public dataset is required, clients' models are homogeneous, and additional data/model information need to be up- loaded, To overcome these issues, we propose a novel two-stage Data-frelE oNe- Shot federated |Earning (DENSE) framework, which trains the global model by a data generation stage and model distillation stage. DENSE is a practical one-shot FL method that can be applied in reality due to the following advantages: (1) DENSE requires no additional information compared with other methods (except the model parameters) to be transferred between clients and the server, (2) DENSE does not require any auxiliary dataset for training; (3) DENSE considers model heterogeneity in FL., i.e., different clients have different model architectures. Experiments on a variety of real-world datasets demonstrate the superiority our method. For example, DENSE outperforms the best baseline method Fed-ADI by 5.08% on CIFARIO dataset.
Introduction
Deep neural networks (DNNs) have recently gained popularity powerful tool for advancing artificial intelligence both established and emerging fields [25, 22, 13, 9, 8, 51, 52, 58, 5, 4, 16]. Federated learning (FL) [42] has emerged as promising learning paradigm which allows multiple clients to collaboratively train a global model without exposing their private training data. In FL, cach client trains a local model on its own data and is required to periodically share its high-dimensional model parameters with a central server. Recent years, FL has shown its potential to facilitate real-world applications in many fields, including medical image analysis [36, 6]. recommender systems [34, 38), natural language processing [63, 46] and computer vision [27, 26].
The original FL. framework requires participants to communicate frequently with the central server in order to exchange models. In real-world FL, such high communication cost may be intolerable and impractical. Reducing the communication cost between clients and the server is desired both for system efficiency and to support the privacy goals federated learning [40, 41]. Recent research proposed some common methods to reduce communication costs, e.g., utilize multiple local up- dates [18], employ compression techniques [48], and one-shot FL [10]. Among them, one-shot FL is
Botth authors contributed equally to this work. Work done during Jie Zhang's 's internship at Tencent Youtu Lab and partly done at Sony AL.
"Work is completed during Chen Chen's internship at Sony AL.
a Cooresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

promising solution which only allows one communication round. There are several motivations behind one-shot FL: 1) First of all, multi-round training not practical in some scenarios such as model markets [45], in which users can only buy the pre-trained models from the market without any real data. 2) Furthermore, frequent communication poses a high risk of being attacked. For instance, frequent communication can be easily intercepted by attackers, who can launch
con-in-the-middl attacks [47] or even reconstruct the training data from gradients [54]. In this way, one-shot FL can reduce the probability of being intercepted by malicious attackers due to the one-round property. Thus, in this paper, we mainly focus on one-shot FL.
However, existing one-shot FL studies [10, 30, 62, 7] are still hard to apply in real-world applications, due to impractical settings. For example, Guha al. [10] and Li et al. [30] involved a public dataset for training, which may be impractical in very sensitive scenarios such as the biomedical domains. Zhu et al. [62] adopted dataset distillation [50] in one-shot FI., but they need to send distilled data to the central server, which causes additional communication cost and potential privacy leakage. Dennis et al. [7] utilized cluster-basee method in one-shot FI., which requires to upload the cluster means to the server, causing additional communication cost. Additionally, none of these methods consider model heterogeneity, i.e., different clients have different model architectures [3 J. which is very common in practical scenarios. For instance, in model market, models sold by different sellers are likely to be heterogeneous. Besides, when several medical institutions participate in FL, they may need to design their own model meet distinct specifications. Therefore, developing a practical one-shot FL method is in urgent need.
In this work, we propose a novel two-stage Data-freF oNe-Shot federated |Earning (DENSE) framework, which trains the global model by data generation stage and model distillation stage. In the first stage, we utilize the ensemble models (i.e., ensemble of local models uploaded by clients) to train a generator, which can generate synthetic data for training in the second stage. In the second stage, we distill the knowledge of the ensemble models to the global model. In contrast to traditional FL methods based on FedAvg [42], our method does not require averaging of model parameters, thus it can support heterogeneous models, i.e., clients can have different model architectures. In summary, our main contributions are summarized as follows:
We propose a novel data-free one-shot FL framework named DENSE, which consists of two stages. In the first stage, we train a generator that considers similarity, stability, and transferability at the same time. In the second stage, we use the ensemble models and the data generated by the generator train a global model.
The setting of DENSE is practical in the following aspects. First, DENSE requires no additional information (except the model parameters) to be transferred between clients and the server, Second, DENSE does not require any auxiliary dataset for training; Third, DENSE considers model heterogeneity, i.e., different clients can have different model architectures.
DENSE is compatible approach, which can be combined with any local training techniques to further improve the performance of the global model. For instance, we can adopt LDAM [1] to train the clients' local models, and improve the accuracy of the global model (refer to Section 2.3 and Section 3.2).
Extensive experiments on various datasets verify the effectiveness of our proposed DENSE. For example, DENSE outperforms the best baseline method Fed-ADI [55] by 5.08% on CIFARIO dataset.
2 Data-Free One-Shot Federated Learning
2.1 Framework Overview
To tackle the problems in recent one-shot FL methods as mentioned in Sec. 1, we propose novel method named DENSE, which conducts one-shot FL without the need to share additional information or rely on any auxiliary dataset, while considering model heterogeneity. To simulate real-world applications, we consider a more challenging yet practical setting where the data on each client are not independent and identically distributed (non-IID).
The illustration of the learning procedure demonstrated Figure 1, and the whole training process of DENSE is shown in Algorithm 1. After clients upload their local models to the server, the server

---

Data generation
Model distillation
Ensemble model
Ensemble model (fixed)
(fixed)
Random labels
Avg logits
Cross entropy
Generator
Avg logits
Random noise
Generator
Synthetic data
Hi+.
Medd
BN loss
loss
Back propagation
Back propagation
Synthetic data
Medd
Logits
Avg logits
Global model (fixed)
Logits
KL loss
Global model
Figure I: An illustration of training process DENSE on the server, which consists of two stages: (1) In data generation stage, we train an auxiliary generator that considers similarity, stability, and transferability at the same time; (2) In model distillation stage, we distill the knowledge of the ensemble models and transfer to the global model. Note that the fixed global model is used as additional discriminator in the divergence loss Laiv.
trains a global model with DENSE in two stages. In the data generation stage (first stage), we train auxiliary generator that generate synthetic data by the ensemble models, i.e., ensemble of local models uploaded by clients. In the model distillation stage (second stage), we use the ensemble models and the synthetic data (generated by the generator) train the global model.
2.2 Data Generation
In the first stage, aim train generator to generate synthetic data. Specifically, given the ensemble of well-trained models uploaded by clients, our goal is to train a generator that can generate data that have similar distribution the training data of clients. In addition, we aim not to leak private information from our generated data, i.e., attackers are not able to predict any sensitive information of clients from the generated data. Recent work [35] generated data by utilizing pre-trained generative adversarial network (GAN). However, such a method is unable to generate data as the pre-trained GAN is trained on public datasets, which is likely have different data distribution from the training data of clients. Morcover, we need to consider model heterogeneity, which makes the problem more complicated.
To solve these issues, we propose to train generator that considers similarity", stability, and transferability. The data generation process is shown in line 8 to 11 in Algorithm 1. In particular, given a random noise z (generated from a standard Gaussian distribution) and random one-hot label (generated from uniform distribution), the generator G(+) aims generate synthetic data x G(z) such that is similar to the training data (with label y) of clients.
Similarity. First, need to consider the similarity between synthetic data & and the training data. Since we are unable to access the training data of clients, we cannot compute the similarity between the synthetic data and the training data directly. Instead, we first compute the average logits (i.e.. outputs of the last fully connected layer) of & computed by the ensemble models.
1 Σ ph (x::0),
D(C; (0
KEC
where = [C), and D(;; (0*1m is the average logits of x, H* is the parameter of the k-thh client. And 8k {x x*) is the prediction function of client & that outputs the logits of x given parameter e
For simplicity, we use Dxx) denote (*;; (0*) [0*) m the rest of the paper.
Note that the ideal synthetic data should be visually distinct from the real data for visual privacy, but similar in distribution for utility.

---

Then, we minimize the average logits and the random label y with the following cross cross-entropy (CE) loss.
( y: 0a) = y))
(2)
It is expected that the synthetic images can be classified into one particular class with a high probability by the ensemble models. In fact, during the training phase, the loss between ()x) and can easily reduce to almost O, which indicates the synthetic data matches the ensemble models perfectly. Morcover, we do not directly compute the similarity between the synthetic data and the training data, which can reduce the probability of leaking sensitive information of the clients.
However, by utilizing only the CE loss, we cannot achieve high performance (please refer to Section 3.2 for detail). We conjecture this is because the ensemble models are trained on non-IID data, the generator may be unstable and trapped into sub-optimal local minima overfit to the synthetic data [2].
Stability. Second, to improve the stability of the generator, we propose add an additional regularization to stabilize the training. In particular, we utilize the Batch Normalization (BN) loss to make the synthetic data conform with the batch normalization statistics [55].
( 0c) =
1
E (11x1()) REC
-x,1ll + (x) -o,,1)
(3)
where (x) and (x) are the batch- wise mean and variance estimates of feature maps corresponding to the 1-th BN layer of the generator G(-),, uia, and o,, are the mean and variance of the I-th BN
layer [17] of (-). The BN loss minimizes the distance between the feature map statistics of the
synthetic data and the training data of clients. As a result, the synthetic data can have similar distribution to the training data of clients, no matter if the data is non-IID or IID.
Transferability. By utilizing the CE loss and BN loss, we can train a generator that can gen- erate synthetic data, but we observed that the synthetic data are likely to be far away from the decision boundary (of the ensemble models), which makes the ensemble models (teachers) hard to transfer their knowledge to the global model (student). We illustrate the observation in the left panel of Figure 2. S and T are the decision boundaries of the global model (the detail of the global model is introduced in Sec- tion 2.3) and ensemble models respectively. The essence of knowledge distillation is transferring the information of decision boundary from the teacher model to the student model [12]. We aim to learn the decision boundary of global model and have a high classification accuracy on the real test data (blue diamonds). However, the generated synthetic data (red circles) are likely to be on the same side of the two deci- sion boundaries and unhelpful to the transfer of knowledge [12]. To solve this problem, we argue to generate more synthetic data that fall between the decision boundaries of the ensemble models and the global model. We illustrate our idea the right panel of Figure 2. Red circles are synthetic data on the same side of the decision boundary, which are less helpful in learning the global model. Black circles are synthetic data between the decision boundaries, i.e., the global model and the ensemble models have different predictions on these data. Black circles can help the global model better learn the decision boundary of the ensemble models.
O O Sputhetic data
Real data
Student
TTachhr
Staseett Teacher
(a) Low diversity data easy learn
Train with desired synthetic
Figure 2: The illustration of generated data and decision boundary of ensemble models (teachers) and global model (student). Left panel: Synthetic data (red circles) are far away from the decision boundary, which is less helpful to the transfer of knowledge. Right panel: By utilizing our bound- ary support loss, we can generate more synthetic data near the decision boundaries (black circles), which helps the student better learn the decision boundary of the teacher.
Motivated by the above observations, we introduce a new boundary support loss, which urges the generator to generate more synthetic data between the decision boundaries of the ensemble models and the global model. We divide the synthetic data into 2 sets: (1) the global model and the ensemble
We assume the input is batch of data.
4

---

models have the same predictions on data in the first set (arg max, D(e) arg mag. SS (8: Os)): (2) different predictions on data in the second set (arg max, ( + arg max,, f (>x: 0 8s)). where Dle) (x) and fs (5;; 0s) are the logits for the c-th label of the ensemble models and the global model respectively. The data in the first set are on the same side of those two decision boundaries (red circles in Figure 2) while the data in the second set (black circles in Figure 2) are between the decision boundaries of the ensemble models and the global model. We maximize the differences of predictions of the global model and the ensemble models on data in the second set with Kullback-Leible divergence loss as follows.
Daie (x:; (x; 0g) -KL
(4)
where K ((.,)) denotes the Kullback-Leible (KL) divergence loss, w 1 (arg max, (x)) o (1)) (x: 0s)) outputs 0 for data in the first set and 1 for data in the second set, and [(a) the indicator function that outputs if a is true and outputs 0 if a is false. By maximizing the KL divergence loss, the generator can generate more synthetic data that are more helpful to the model distillation stage (refer to Section 2.3 for detail) and further improve the transferability of the ensemble models.
arg max, f&
By combining the above losses, we can obtain the generator loss follows,
Lyen (8. y; 0 = Lc + LDN (xc; 0g) + A2 0G),
(5)
where Å and A2 are scaling factors for the losses.
Algorithm 1 Training process of DENSE
generator Gi-) with parameter 0g. learning rate of the generator nJ. number of training rounds TG for generator in each epoch, global model fs() with parameter 0s, learning rate of the global model n/s global model training epochs T, and batch size b.
Input:
Number of client m, clients' local models { ..... Of......."' .f'.
for each client & E C in parallel do 0 + LocalUpdate(k)) end for Initialize parameter 0g and os
for epoch=1, T do
Sample a batch of noises and labels {z,. // data generation stage for jo 1,... To do Generate {x:)==1 with (2:)/-1 and G(.)
0c + 0g - 7G Ei Voc laenen (x,n(X;;1/ 0c)
end for
model distillation stage
Generate { with {z, 1/ and G(-) 0g = 0g 11 Eim1 Vos laii (x;; 0s)
end for
Note that the generated synthetic data have similar features but different from the training data (of clients), which reduces the probability of leaking sensitive information of clients. More discussions of the privacy issues are in Section 3.3.3.
2.3 Model Distillation
In the second stage, we train the global model with the generator (discussed in the previous section) and the ensemble models. Previous research [60, 35] showed that model ensemble provides general method for improving the accuracy and stability of learning models. Motivated by [60], we propose to use the ensemble models as teacher to train student (global) model. A straightforward method is to obtain the global model by aggregating the parameters of all client models (e.g., by FedAvg [42]). However, in real-world applications, clients are likely to have different model architectures [44], making FedAvg useless. Moreover, since the data in different clients are non-III, FedAvg cannot deliver good performance or even diverge [59, 32].
5

---

To this end, we follow [35] to distill the knowledge of the ensemble models to the global model by
minimizing the predictions between the ensemble models (teacher) and the global model (student) on the same synthetic data. The model distillation process ís shown in line 13 to 14 in Algorithm 1. First, we compute the average logits of the synthetic data according to Eq. (1); i.e., D(x)
L in Exec f xx: e*) In contrast to traditional aggregation methods (e.g., FedAvg) that are unable to aggregate heterogeneous models, averaging logits can be easily applied to both heterogeneous and homogeneous FL systems.
Then, we use the average logits to distill the knowledge of the ensemble models by minimizing the following objective function.
Lase (x: = KL (D(x), fs(:; 0z))
(6)
By minimizing the KL loss, can train global model with the knowledge of the ensemble models and the synthetic data regardless of data and model heterogeneity.
Note that DENSE has no restriction on the clients' local models, i.e., clients can train models with arbitrary techniques. Thus, DENSE a compatible approach, which can be combined with any local training techniques to further improve the performance of the global model. We further discuss the combination of local training techniques Section 3.2.
Discussions on privacy-preser where a malicious user uses GANs to recreate samples of another participant's is private datasets. Besides, in FL, exchanging models between the server and clients can result in potential privacy leakage. Note that our method prohibits the generator from seeing the real data directly, and there is only one communication round, which reduces the risk of privacy leakage. In addition, we display our generated images in Section 3.3.3, which does not directly reveal the information of real data. Several existing prreccy- methods can be incorporated into our framework to better protect clients from adversaries [37, 20J. We leave this as our future work.
Research [15] has shown that it is possible to launch an attack
Discussions on Knowledge Distillation in FL In traditional FL frameworks, all users have to agree on the specific architecture of the global model. To support model heterogeneity, Li al. [28] proposed a new federated learning framework that enables participants to independently design their models by knowledge distillation [14]. With the use of proxy dataset, knowledge distillation alleviates the model drift issue induced by non-IID data. However, the requirement of proxy data renders such method impractical for many applications, since a carefully designed dataset is not always available on the server. Data-free knowledge distillation is promising approach, which can transfer knowledge of a teacher model to student model without any real data [2, 55]. Lin et al. [35] proposed data-free ensemble distillation for model fusion through synthetic data in each communication round, which requires high communication costs and computational costs. However, in this paper, we are more concerned with obtaining good global model through only one round of communication in cases of heterogeneous models, which more challenging and practical. Zhu et al. [64] also proposed data-free knowledge distillation approach for FL, which learns a generator derived from the prediction of local models. However, the learned generator is later broadcasted to all clients, and then clients need to send their generators to the server, which increases the communication burden. More seriously, the generator has direct access to the local data (the generator can easily remember the training samples (39]), which can cause privacy concerns. As the generator used in our method is always stored in the central server, it never sees any real local data.
3 Experiments
3.1
Experimental Setup
3.1.1 Datasets
Our experiments are conducted on the following 6 real-world datasets: MNIST [24], FMNIST [53]. SVHN [43], CIFARIO [21], CIFARIO0 [21], and Tiny-ImageNet [23].MNIST dataset contains binary images of handwritten digits. There are 60,000 training images and 10,000 testing images in MNIST dataset. CIFARIO dataset consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images in CIFAR10 dataset. CIFAR100 dataset is similar to CIFARIO 10 dataset, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. Imay-ImageNet contains 100000 images of 200 classes (500 for each class) downsized to 64x64 colored images. Each class has 500 training images, 50 validation images and 50 test images.

---

Table 1: Accuracy different methods across a {0.1, 0.3, 0.5) on different datasets.
Dasaset
MNNST
TMIIST
CUARCO
SVIEN
CIEARIO)
Thy ImageNet
Method
0-0.1
ow4033
0-40.
0-001
0-4).3
0-411
---0.1
c-4.3
o-45
0-4.1
0-40.
0-005
o-413
0-0011
owe1.
Renkerg
72.04
27:73
FallFF
74.01
43.58
73.00
90.78
47:04
84.02
53.89
22.18
75.00
48.49
$6.19
54.08
59.34
50.45
77,45
70.85
25.18
19.59
25.34
DENSR
76.48
50.29
88.94
59.76
55.34
37.32
32.34
3.1.2 Data partition
To simulate real-world applications, we use Dirichlet distribution to generate non-lID data partition among clients [56, 29]. In particular, sample P& Dir(a) and allocate Pa proportion of the data of class to client i. By varying the parameter a, we can change the degree of imbalance. A small generates highly skewed data. We set 0.5 default.
3.1.3 Baselines
To ensure fair comparisons, we neglect the comparison with methods that require to download auxiliary models or datasets, such as FedBE [3] and FedGen [64]. Moreover, since there is only one communication round, aggregation methods that are based on regularization have no effect. Thus, we
also omit the comparison with these regularization based methods, e.g., FedProx [31], FedNova [49]. and Scaffold [18]. Instead, we compare our proposed DENSE with FedAvg [42] and FedDF [35]. Furthermore, since DENSE is data-free method, we derive some baselines from prevailing data-free knowledge distillation methods, including: 1) DAFL [2], a novel data-free learning framework based on generative adversarial networks; 2) ADI [55], an image synthesizing method that utilizes the image distribution to train deep neural network without real data. We apply these methods to one-shot FL,
and name these two baselines as Fed-DAFL and Fed-ADI.
3.1.4 Settings
For clients' local training, we the SGD optimizer with momentum=0.9 and learning rate=0.01. We set the batch size b 128, the number of local epochs E = 200, and the client numher m = 5. Following the setting [2], we train the auxiliary generator G(-) with deep convolutional network. 0.001. We set the number of training rounds in each epoch TG 30, and the scaling factor
We use Adam optimizer with learning rate
1 and A2 0.5. For the training of the server model fs(), we use the SGD optimizer with learning rate = 0.01 and momentum=0.9, The number of epochs for distillation 200. All baseline methods use the same setting as ours.
3.2 Results
3.2.1
Evaluation on real-world datasets
To evaluate the effectiveness of our method, we conduct experiments under different non-IID settings by varying a = {0.1. 0.3, 0.5] and report the performance on different datasets and different methods in Table 1. The re- sults show that: (1) Our DENSE achieves the highest accuracy across all datasets. In particular, DENSE outperforms the best base- line method Fed-ADI [55] by 5.08% when 0.3 on CIFARIO dataset. (2) FedAvg has the worst performance, which implies that directly averaging the model parameters can- not achieve a good performance under non- IID setting one-shot FL. (3) As a becomes smaller (i.e., data become more imbalanced), the performance of all methods decrease sig- nificantly, which shows that all methods suffer
Cleit
Clern
Dixt2
Okaro
20
Clert
Oivr
Fostig
Clase
100
230
330
400
100
200
303
400
Local spoch
Local epoch E
Figure Left panel: Accuracy of FedAvg and clients' local models across different local training epochs E {20, 40, 60, 400]. Right panel: The accuracy curve for local training. The dotted lines represent the best results of two one-shot FL methods (FedAvg and DENSE). Our DENSE out- performs FedAvg and local models consistently.
7

---

Table 2: Accuracy comparisons across heterogeneous client models CIFARIO. There are five clients in total, and each client has a personalized model.
Client
Server (ResNct-18)
Model
ResNet-18
CNNI
CNN2
WRN-16-1
WRN-40-1
FedDF
Fed-DAFL
Fed-AD1
DENSE (ours)
a=0.1
40.83
33.67
35.21
27.73
32,93
42.35
43.12
44.63
49.76
0=0.3
$1.49
52.78
44,96
47.35
37.24
52.72
57.72
58.96
63.25
0=0.5
59.96
58.67
54.28
53.39
58.14
60.05
61.56
63.24
67.42
0.5
0.6
0.3
0.0
60
60
Text Ansiney
170000 051 940
Test Accurer)
000
SM
Glem
46
Cluu ID
1M4
ADI
Fed ACI FAd
Fed (JAFL
100
200
205
400
0
100
200
300
400
4
Gloabl Epoch
Label
Gloabl Epoch
Epach
Label
Figure 4: Visualization of the test accuracy and data distribution for CIFARIO with {0.33
from highly skewed data. Even under highly skewed setting, DENSE still significantly outperforms other methods, which further demonstrates the superiority of our proposed method.
3.2.2 Impact of model distillation
We show the impact model distillation by comparing with FedAvg. We first conduct one-shot FL and use FedAvg to aggregate the local models. We show the results of the global model and clients' local models across different local training epochs E = {20, 40, 60, ... .000 in the left panel of Figure 3. The global model achieves the best performance (test accuracy=34% when E 40, while a larger value of E can cause the model to degrade even collapse. This result can be attributed to the inconsistent optimization objectives with non-Ill) data [49], which leads to weight divergence [61]. Then, we show the results of one-shot FI. when F. = 400 and report the performance of FedAvg and DENSE the right panel of Figure 3. We also plot the performance of clients' local models. DENSE outperforms each client's 's local model while FedAvg underperforms each client's 's local model. This validates that model distillation can enhance training while directly aggregating harmful to the training under non-IID setting in one-shot FL.
3.2.3 Results in heterogeneous FL
Note that our proposed DENSE can support heterogeneous models. We apply five different CNN models on CIFAR10 10 dataset with Dirichlet distribution a {0.1, 0.3, 0.5). The heterogeneous models include: 1) one ResNet- [11], 2) two small CNNs: CNN1 and CNN2: 3) two Rese-RssNets (WRN) [57]: WRN-16-1 and WRN-40-1. For knowledge distillation, we use ResNet-18 18 as the server's global model. Detailed architecture information of the given deep networks can be found in Appendix. Table 2 evaluates all methods in heterogeneous one-shot FL under practical non-llI) data settings. We omit the results for FedAvg as FedAvg does not support heterogeneous models. We remark that FL under both the non-IID data distribution and different model architecture setting is a quite challenging task. Even under this setting, our DENSE still significantly outperforms other baselines. In addition, report the accuracy curve of global distillation. As shown in Figure 4, our method outperforms other baselines by a large margin.
3.3
Analysis of Our Method
3.3.1 Impact of the number of clients
Furthermore, we evaluate the performance of these methods on CIFAR10 and SVHN datasets by varying the number of clients m = (5, 10, 20, 50, 100). According to [33], the server can become bottleneck when the number of clients is very large, we also concerned with the model performance
8

---

Table 3: Aceuracy across different number of clients m SVHN datasets.
{5, 10,10,20 20, 50, 100} on CIFARIO and
Dataser
CIFARIO
SVHN
n
FodAvg
FedDF
Fod-DAFL
Fed-ADI
DENSE (ours)
FodAvg
FedDF
Fed-DAFL
Fed-ADI
DENSE (ours)
5 43.67
53.56
55.46
58.59
62.19
56.09
73.98
78.03
78.85
80.03
10
38.29
$4.44
56.34
57.13
61.42
45.34
62.12
63.34
63.45
67.57
20
43.15
45.98
52.71
47.79
63.98
66.42
50
37.03
40.89
43.02
44.47
48.47
36.53
51.44
54.23
57.35
59.27
100
33.54
36.89
37.55
36.98
43.28
30.18
46.38
47.19
48.33
52.48
Table 4: Performance analysis of DENSE+LDAM.
Dataset
CIFARIO
SVHN
Method
6=0.1
o=0.3
6=0.5
x=0.1
x=0.3
x=0.5
DENSE
50.26
59.76
62.19
55.34
79.59
80.03
DENSE+LDAM
57.24
63.13
64.76
58.04
81.28
81.77
when increases. Table 3 shows the results of different methods across different The accuracy of all methods decreases as the number of clients m increases, which consistent with observations in [33, 42]. Even though the number of clients can affect the performance of one-shot FL., our method still outperforms other baselines. The increasing number clients poses new challenges for ensemble distillation, which leave for future investigation.
3.3.2 Combination with imbalanced learning
The accuracy of federated learning reduces sig- nificantly with non-IID data, which has been broadly discussed in recent studics [29, 49]. Ad- ditionally, previous studies [1, 19] have demon- strated their superiority on imbalanced data. The combination of method with these tech- niques to address imbalanced local data can lead effective FL system. For example, by using LDAM [1] in clients' local training, we can mitigate the impact of data imbalance, and thereby build a more powerful ensemble model. We compare the performance of the original DENSE and DENSE combined with LDAM (DENSE+LDAM) across a = 10.1, 0.3, 0.5) on CIFAR10 and SVHN datasets.
Test oo00 2oy
40
Ours
300
spoch
Figure 5: Left panel: Accuracy curves of DENSE and DENSE+LDAM, Right panel: Data distri- bution of different clients for 10 dataset (a=0.1).
As demonstrated in Table 4, DENSE+LDAM can significantly improve the performance, es- pecially for highly skewed non-IlI data (i.e.
3
CIFARIO
= 0.1). To help understand the performance gap and data skewness, in Figure 5, we visual- ize the accuracy curve and data distribution of CIFARIO 10=0.1) ) in the left panel and right panel respectively. The number in the right panel stands for the number of examples as- sociated with the corresponding label in one particular client. These figures imply that sig- nificant improvement can be achieved by com- bining DENSE with LDAM on highly skewed data.
Ours
SUHN
2
8
9
Durs
Figure 6: Visualization of synthetic data on Cl-
FARI0 and SVHN datasets.

---

3.3.3 Visualization of synthetic data
To compare the synthetic data with the training data, we visualize the synthetic data on CIFARIO and SVHN datasets in Figure 6, As shown in the figure, the first / third row is the original data of CIFARIO SVHN dataset, and the second last row is the synthetic data generated by the model trained on CIFARIO/ / SVHN dataset. The synthetic data not similar to the original data, which can effectively reduce the probability of leaking sensitive information of clients. Note that although the synthetic data look much different from the original data, method still achieves higher performance than other baseline methods by training with these synthetic data (as shown in Table 1). Note that the ideal synthetic data should be visually distinct from the real data.
3.3.4 Extend to multiple rounds
We also extend DENSE to multi-round FL to test its effectiveness, i.e., there are multiple communication rounds be- tween clients and server. Table 5 demon- strates the results of DENSE across different communication rounds T. 2,,2,3, 4, 5} on CIFARIO and SVHN datasets. The local training epoch is fixed as E = 10. The performance of DENSE improves as Te increases, and DENSE achieves the best performance when T. = 5. This shows that DENSE can be extended to multi-round FL and the performance can be further enhanced by increasing the communication rounds.
Table 5: Accuracy for multiple communication rounds.
Datnset
CIFARIO
SVHN
Communication munds
a=0.1
a=0.3
a-0.5
n=0.1
o=00.3
o=00.5
T.
50.72
59.41
63.89
54,344
79.87
80.14
T
63.08
65.90
71.16
56.13
79.75
85.18
T.
61.61
69.73
73.91
74.41
86.42
86.18
66.26
T, 5
67.65
71.42
76.01
80.28
86.25
86.55
3.3.5 Contribution of LBN and Ldiv
We investigate the contributions of differ- ent loss functions used data generation. We conduct leave-one-one testing and show the results by removing Laiv (w/o Ldiv), and removing LBN (wlo LBN). Additionally, we report the result by re- moving both Ldiv and LAN, i.e,, using only LeE (w/ LCr). As illustrated Table 6, using only LCg train the gen- erator leads to poor performance. Be- sides, removing either the LBN loss or Laio loss also affects the accuracy of the global model. A combination of these loss functions leads to a high performance of global model, which shows that each part of the loss function plays an important role enhancing the generator.
Table 6: Impact of loss functions in data generation.
Dataset
CIFARIO
SVHN
CIFARTO0
DENSE
62.19
80.03
42.07
w/ LoE 53.12
73.11
36.47
w/o LBN
61.05
78.36
39.89
w/o Laiv
59.18
77.59
39.14
4 Conclusion
In this paper, we propose an effective one-shot federated learning method called DENSE, which trains the global model by a data generation stage and model distillation stage. Extensive experiments across various settings validate the efficacy of our method. Overall, DENSE by far the most practical framework that can conduct data-free free one-shot FL with model heterogeneity. A promising future direction is consider the potential privacy attacks in one-shot FL.
5 Acknowledgement
This work was supported by Sony AI, the National Key Research and Development Project of China (2021ZD01 10400 No. 2018AAA0101900), National Natural Science Foundation of China (U19B2042), Zhejiang Lab (2021 KFOAC02), Academy Of Social Governance Zhejiang University, Fundamental Research Funds for the Central Universities.
10

---

References
[1] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distrbuttinnaware margin loss, 2019.
[2]
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Dafl: Data-free learning of student networks. In ICCV, 2019.
[3] Hong- You Chen and Wen Chao. Fedbe: Making bayesian model ensemble applicable to federated learning, 2021.
[4] Zhaoyu Chen, Bo Li, Shuang Wu, Jianghe Xu, Shouhong Ding, and Wenqiang Zhang. Shape matters: Deformable patch attack. In European conference computer vision, 2022.
[5] Zhaoyu Chen, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Wenqiang Zhang. Towards practical certifiable patch defense with vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15148-15158, June 2022.
[6] Zhen Chen, Meilu Zhu, Chen Yang, and Yixuan Yuan. Personalized retrogress-resilient frame- work for real-world medical federated learning. In Marleen de Bruijne, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert, editors, Medical Image Computing and Computer Assisted Intervention MICCAI 2021 24th Interna- tional Conference, Strasbourg, France, September 27 October 1, 2021, Proceedings, Part III, volume 12903 of Lecture Notes in Computer Science, pages 347-356. Springer, 2021.
[7] Don Kurian Dennis, Tian Li, and Virginia Smith. Heterogeneity for the win: One-shot federated clustering, 2021.
[8] Jiahua Dong, Yang Cong, Gan Sun, Zhen Fang, and Zhengming Ding. Where and how to transfer: Knowledge aggregation-indueed transferability perception for unsupervised domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(1):1-17, 2021.
[9] Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, and Xiaowei Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4022-4031, June 2020.
[10] Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. arXiv preprint 19Xiv:1902.11173 2019.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[12] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge distillation with adversarial samples supporting decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3771-3778 2019.
[13] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprini htt:::5533.2533 2(7), 2015.
[14] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.00531, 2015.
[15] Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz. Deep models under the gan: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 603-618, 2017.
[16] Hao Huang, Yongtao Wang, Zhaoyu Chen. Yuze Zhang, Yuheng Li, Zhi Tang. Wei Chu, Jingdong Chen, Weisi Lin, and Kai-Kuang Ma. Cmua-watermark: cross-model universal adversarial watermark for combating deepfakes. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):989-997, Jun. 2022.
[17] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.

---

[18] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
[19] Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to-minor translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[20] Aashish Kolluri, Teodora Baluta, and Prateek Saxena. Private hierarchical clustering in federated networks. In Yongdae Kim, Jong Kim, Giovanni Vigna, and Elaine Shi, editors, CCS '21: 202 I ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 19, 2021, pages 2342-2360. ACM, 2021.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 6066):84-90, 2017.
[23] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient based learning applied to document recognition. Proceedings of the IEEE,
2324, 1998.
[25] Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages 1I-104. IEEE, 2004.
[26] Bo Li, Zhengxing Sun, and Yuqi Guo. Supervae: Superpixelwise variational autoencoder for salient object detection. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - FFebuuryy 1, 2019, pages 8569-8576. AAAI Press, 2019.
[27] Bo Li, Zhengxing Sun, Lv Tang, Yunhan Sun, and Jinlong Shi. Detecting robust co- co-saliency with recurrent co- co-attention neural network. In Sarit Kraus, editor, Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 818 825. ijcai.org, 2019.
[28] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv: 1910.03581, 2019.
[29] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study, 2021.
[30] Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint arXiv:2010.01007, 2020.
[31] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2020.
[32] Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. In International Conference on Learning Representations, 2020.
[33] Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent, 2017.
[34] Feng Liang, Weike Pan, and Zhong Ming. Fedrec++: Lossless federated recommendation with explicit feedback. In Thirty-Fifih AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference Fleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, I, Virtual Event, February 2-9, 2021.
Innovative Applications of Artificial Intelligence, IAAI 2021, The
12

---

[35] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[36] Quande Liu, Cheng Chen. Jing Qin, Qi Dou, and Pheng-Am Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 1013-1023. Computer Vision Foundation IEFE, 2021.
[37] Ruixuan Liu, Yang Cao, Hong Chen, Ruoyang Guo, and Masatoshi Yoshikawa. FLAME: differentially private federated learning in the shuffle model. In Thirty-Fifihe AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thirl Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Fleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 8688-8696. AAAI Press, 2021.
[38] Shuchang Liu, Shuyuan Xu, Wenhui Yu, Zuohui Fu, Yongfeng Zhang. and Amélie Marian. Fedet: Federated collaborative transfer for recommendation. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 7 716-725, ACM, 2021.
[39] Yi Liu, Jialiang Peng. JQ James, and Yi Wu. Ppgan: network. In 2019 IEEE 25Th international conference on parallel and distributed systems (ICPADS), pages 985-989. IEEE, 2019.
generative adversarial
[40] Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and Philip Yu. Privacy and robustness in federated learning: Attacks and defenses. arXiv preprint arXiv:20/2.06333 2020.
[41] Lingjuan Lyu, Han Yu, Jun Zhao, and Qiang Yang. Threats to federated learning, In Federated Learning, pages 3-16. Springer, 2020.
[42] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aglier y Arcas. Communication--ffiiieee learning of deep networks from decentralized data, 2017.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[44] Lichao Sun and Lingjuan Lyu. Federated model distillation with noise-free differential privacy. In IJCAI, 2021.
[45] Manasi Vartak. MODELDB: A system for machine learning model management. In 8th Biennial Conference on Innovative Data Systems Research, CIDR 2017, Chaminade, CA, USA, Jamuary 8-11, 2017. Online Proceedings. www.cidrdb.org, 2017.
[46] Chenghong Wang, Jieren Deng, Xianrui Meng. Yijue Wang, Ji Li, Fei Miao, Sanguthevar Rajasekaran, and Caiwen Ding. A secure and efficient federated learning framework for NLP. In Franci-Fann Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, EMNLP 2021.
[47] Derui Wang, Chaoran Li, Sheng Wen, Surya Nepal, and Yang Xiang. Man-in-the-middle attacks against machine learning classifiers via malicious generative models. IEEE Transactions Dependable and Secure Computing, 2020.
[48] Hongyi Wang, Saurabh Agarwal, and Dimitris S. Papailiopoulos. Pufferfish: Communication- efficient models at extra cost. CORR, abs/2103.03936, 2021.
[49] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481,
2020. 2020.
13

---

[50] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros, Dataset distillation, 2020.
[51] Kun Wei, Da Chen, Yuhong Li, Xu Yang, Cheng Deng, and Dacheng Tao. Incremental embedding learning with disentangled representation translation. IEEE Transactions Neural Networks and Learning Systems, 2022.
[52] Kun Wei, Cheng Deng, Xu Yang, and Maosen Li. Incremental embedding learning via zero-shot translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10254-10262, 2021.
[53] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
[54] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16337-16346, 2021.
[55] Hongxu Yin, Pavlo Molchanov, Jose Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
[56] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks, 2019.
[57] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2017.
[58] Jie Zhang, Chen Chen, Jiahua Dong, Ruoxi Jia, and Lingjuan Lyu. Qekd: Query-efficient and data-free knowledge distillation from black-box models. arXiv preprint arXiv:2205.11155: 2022.
[59] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference Machine Learning, pages 26311-26329. PMLR, 2022.
[60] Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. Advances in Neural Information Processing Systems, 33, 2020.
[61] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, 2018.
[62] Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.07999 2020.
[63] Xinghua Zhu, Jianzong Wang, Zhenhou Hong, and Jing Xiao. Empirical studies of institutional federated learning for natural language processing. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 625-634. Association for Computational Linguistics, 2020.
[64] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou Data-free knowledge distillation for heteroge- neous federated learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 2888-12886 PMLR, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "PRrMzeha2kj", "reviewer": "Reviewer_fNZT", "summary": "This paper proposed a novel data-free one-shot FL framework named DENSE based on data generation and knowledge distillation techniques so that it can be applied to communication-efficient heterogeneous FL. Evaluation results on multiple benchmark datasets illustrate the good performance of the proposed FL framework.", "questions": "Why not compare the proposed method with the FedGen method that is data-free FL with knowledge distillation? I know it requires to send the model parameters of generator between server and clients multiple rounds.", "limitations": "Yes, the authors have described the limitations of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n[1] This paper proposed a novel data-free one-shot FL framework consisting of two stages: i) trains a generator based on ensemble models uploaded from clients and random labels; ii) adopts knowledge distillation to transfer the output from the teacher model to the global model. The idea of data-free one-shot FL is very interesting. \n[2] Extensive experiments have been conducted to evaluate the performance of the proposed framework\n\nWeakness:\n[1] I am concerned about the privacy as the ensemble models is uploaded to the central sever. As far as I know, attackers in the server side can still recover the privacy data based on the model parameters.\n[2] I am wondering if the labelled data in the server would leak the data privacy? Such as race and gender.\n[3] Please explain the important steps in algorithm 1 in Appendix with mode details. It seems hard to understand the algorithm.\n[4] Please conduct experiments with mutiple random seeds, and then report the mean and variance. Otherwise, the experimental results seem not very convincing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FmHfLYha6an", "reviewer": "Reviewer_oqiW", "summary": "The authors proposed a novel one-shot data-free Federated Learning algorithm:\n1. The algorithm only requires a single communication rounds. The clients locally train the model and uploads the model to the central server.\n2. In the server, the ensemble of the client model is utilized to train a generator model that generates synthetic data. The generator is trained by giving a random number and a random label, such that the logits computed from the generated image by averaging the client models minimize the cross entropy.\n3. Then the generator is used to generate a large amount of synthetic data to be labeled by the client ensemble. One then distill the knowledge of the ensemble to a global model by train it with synthetic data.\n4. The authors also includes additional loss terms for generator training to ensure similarity, stability, transferability of the data.\n5. Finally, extensive experiments are carried out, comparing with existing Federated learning algorithm as well as ablation study.", "questions": "There are a few questions regarding the technical details:\n\n1. The generator G(z) is trained by first sample random pairs of random number and label (z, y). Then the generated input x = G(z) is used such that x would minimize some loss. The author first shows cross entropy loss alone does not work well and conjecture that it is due to non-IID data. However, in ablation study one could certainly make the data IID for all clients (table 6). How does table 6 obtain the results? What is the hyperparameter used? \n\nIn my opinion the poor performance of cross entropy alone could also due to the sampling setup: we generate a random pair (z, y). There may not exist a proper transformation between z to x such that (z, y) is independent. In other words, (z, y) could be correlated. \n\n2. How does the author handle the case that for client models, their logits distributions are very different (line 105-106)? In extreme case, suppose one model produce a set of logits of value O(1e6) while other models produce logits of value (1e-6). Then one model could clearly dominate over the average of the ensemble. Is this a concern for the averaging? \n\n3. From algorithm 1, in the inner loop where generator is being updated, the global model's parameter theta_S is fixed. This would affect the quality of loss terms in equation (5) because the global model is still bad. Could the author comment on this issue?\n\n4. In global model training (distillation), the author uses KL divergence as objective (equation (6)). Have the authors tried other losses such as cross entropy? \n\n", "limitations": "The author has described limitations of current work and suggest for future investigation. They have not discussed negative societal impact, which I believe is OK.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper shows strength in the following perspectives:\n1. Originality. The algorithm proposed is novel to my best knowledge. It addresses several issues of applying federated learning to real world applications. First, the algorithm is genuinely data free: no client data, distillation data or other data summary will be uploaded to server. Second, the algorithm admits heterogeneity of client models, such that in real world applications clients have the freedom of picking local models. Thirdly, the algorithm is one-shot in the sense that only one round of communication is needed, although the author mentioned that more rounds can improve the quality of the model.\n2. Quality. The experiments conducted in the paper are extensive and sound. The author explains why some of the comparisons are not included such as regularization based methods. There are also plenty of experiments with a wide range of parameter space to demonstrate the effectiveness of the algorithm. Those include balancedness of the client data, number of clients, number of rounds and effects of added terms.\n3. Clarity. The paper is well-written. The author reviews the existing limitations of the literature, and gives a clear structure on the components of the algorithm.\n4. Significance. The paper's result is important both from academic and practical point of view. For the latter, it will stimulate the research community for real world application of FL, such that the whole infrastructure is secure, privacy-preserving, and high quality.\n\nThe weakness of the paper, in my opinion, has the following aspects:\n1. Quality. The paper does not have a lot of theoretical analysis, especially on why the algorithm would converge regardless of which dataset one is using, and data distributions among clients, etc. \n2. Clarity. While it is OK to move the backgrounds and preliminaries in appendix, in my opinion it would be better to move the full algorithm (Algorithm 1) in the main context - readers may constantly go to the table and refer to it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ChTm8qxpsaM", "reviewer": "Reviewer_e2Cu", "summary": "The paper focuses on one-shot federated learning, i.e., the server can learn a model with a single communication round. The proposed FedSyn method has two stages: first, training a generator from the ensemble of models from clients; second, distilling the knowledge of the ensemble into a global model with synthetic data. The authors validate the efficacy of FedSyn by conducting extensive experiments on 6 different datasets with various non-IID settings generated from Dirichlet distributions. Results can well support that the proposed method consistently outperforms all the baselines.", "questions": "1. What is the architecture of GAN? Is a larger GAN better for model distillation?\n\n2. In Fig.3, why the performance of Fedavg decreases with longer training epochs?\n\n3. In Table 6, how about the results for \"w/o l_CE\"? Is l_CE necessary?\n\n4. In the future direction, authors mentioned “defend privacy attacks in one-shot FL”, could you elaborate more on this? From my understanding, one-shot FL is already more secure than multi-round FL.  \n", "limitations": "The analysis, theory, and method are sound to me, but I didn't check the privacy concern.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper focuses on one-shot FL, an interesting but less explored topic. From my understanding, the proposed method is by far the most practical one considering that：1) DENSE requires no additional information to be transferred between clients and the server; 2) DENSE does not require any auxiliary dataset for training; 3) DENSE considers both model heterogeneity, i.e., different clients with different model architectures. Generally, I think the investigated problem is sound and interesting. I think this can be an extremely strong paper in one-shot FL.\n\n2. The method using data-free ensemble distillation is inspiring and novel. The experimental evaluation of applying the data-free ensemble distillation to one-shot FL significantly improves the performance of the global model. Fig. 1 illustrates the method clearly and is well drawn. \n\n3. The experimental settings are well motivated, and the related analysis is convincing. My favorite parts of the paper are the discussion and the ablation study, which offer sound support for the proposed DENSE. \n\nOverall, this paper is very interesting and to my knowledge novel. It seems like a pioneering contribution towards practical one-shot federated learning. Hence, I would like to vote for strong acceptance.  \n\nWeaknesses.\n\nBelow questions need to be addressed to further improve the quality of this paper.\n1. What is the architecture of GAN? Is a larger GAN better for model distillation?\n\n2. In Fig.3, why the performance of Fedavg decreases with longer training epochs?\n\n3. In Table 6, how about the results for \"w/o l_CE\"? Is l_CE necessary?\n\n4. Even though authors provide sufficient references, a few important papers are recommended. For example, a recent IJCAI paper “Data-Free Adversarial Knowledge Distillation for Graph Neural Networks” that also studied data-free distillation. These papers did not study exactly the same topic as this paper, but would certainly further enrich the literature review.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2y71PkQ1vb4", "reviewer": "Reviewer_4PjE", "summary": "This paper considers the problem of training a global model with only one-shot communication between the server and clients in the federated learning setting. This problem is addressed by designing a generator based on the ensemble of 'trained' local models, to generate synthetic data samples, which can then be further utilized to distill the ensemble models to the global model in a knowledge distillation paradigm.", "questions": "\n\nPlease refer to item **'Weaknesses'** in the previous section.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n- The one-shot federated learning is an important problem in both areas of machine learning and distributed computing.\n\n- Using a generator trained from local models can provide synthetic data samples for the knowledge distillation between the ensemble models and the global model, which is also a simple solution for data-free distillation.\n\n- The evaluation has been conducted on various datasets, which is good.\n\n[Weaknesses]\n\n- The novelty is limited. The setting of 'data-free distillation' + 'heterogeneous local models' has been already addressed by some previous works. For example, FEDGEN [1] also trains a generator to provide data-free knowledge distillation between the server and clients, wherein, their local models are also in heterogeneous architectures. The idea is almost the same.\n\n- The one-shot relies heavily on the 'well-trained' local models, to reduce the communication overhead between the server and clients. However, such a setting is not so practical when the local devices are resource constrained, in either data or computing capability.\n\n- Using the ensemble output results of uploaded local models may cause serious privacy issues. In that case, each individual local model can be examined on both input and output, thus the attackers can have the potential to reconstruct local models or local samples if the server is adversary-oriented.\n\n- The writing and presentation should be improved.\n\n[1] Data-Free Knowledge Distillation for Heterogeneous Federated Learning, ICML 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept