Federated Learning with GAN-based Data Synthesis for Non-IID Clients
Authors: Zijian Li
Jiawei Shao
Yuyi Mao
Jessie Hui Wang
Jun Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning
non-IID
generative model
data augmentation

Abstract: Federated learning (FL) has recently emerged as a popular privacy-preserving collaborative learning paradigm. However, it suffers from the non-IID (independent and identically distributed) data among clients. In this paper, we propose a novel framework, namely Synthetic Data Aided Federated Learning (SDA-FL), to resolve the non-IID issue by sharing differentially private synthetic data. Specifically, each client pretrains a local generative adversarial network (GAN) to generate synthetic data, which are uploaded to the parameter server (PS) to construct a global shared synthetic dataset. The PS is responsible for generating and updating high-quality labels for the global dataset via pseudo labeling with a confident threshold before each global aggregation. A combination of the local private dataset and labeled synthetic dataset leads to nearly identical data distributions among clients, which improves the consistency among local models and benefits the global aggregation. To ensure privacy, the local GANs are trained with differential privacy by adding artificial noise to the local model gradients before being uploaded to the PS. Extensive experiments evidence that the proposed framework outperforms the baseline methods by a large margin in several benchmark datasets under both the supervised and semi-supervised settings.

Under review as a conference paper at ICLR 2022
FEDERATED LEARNING WITH GAN-BASED DATA SYN- THESIS FOR NON-IID CLIENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL) has recently emerged as a popular laborative learning paradigm. However, suffers from the non-IID (independent and identically distributed) data among clients. In this paper, we propose a novel framework, namely Synthetic Data Aided Federated Learning (SDA-FL)), to re- solve the non-IID issue by sharing differentially private synthetic data. Specifi- cally, each client pretrains a local generative adversarial network (GAN) to gen- erate synthetic data, which uploaded to the parameter server (PS) to construct a global shared synthetic dataset. The PS is responsible for generating and up- dating high-quality labels for the global dataset via pseudo labeling with confi- dent threshold before each global aggregation. A combination of the local private dataset and labeled synthetic dataset leads nearly identical data distributions among clients, which improves the consistency among local models and benefits the global aggregation. To ensure privacy, the local GANs are trained with differ- ential privacy by adding artificial noise to the local model gradients before being uploaded to the PS. Extensive experiments evidence that the proposed framework outperforms the baseline methods by large margin in several benchmark datasets under both the supervised and semi-supervised settings.
col-
INTRODUCTION
The recent development of deep learning technologies has led to major breakthroughs in various domains. This results in a tremendous amount of valuable data that can facilitate the training of deep learning models for intelligent applications. traditional approach exploit these distributed data samples is to upload them to centralized server for model training. However, directly offload- ing data raises severe privacy concerns as data collected by mobile clients may contain sensitive information.
By decoupling model training from the need of transferring private data to the cloud, federated learning (FL) offers a promising approach to collaboratively learn a global model without directly sharing the local data. Particularly, McMahan et al. (2017) introduced the Federated Averaging (FedAvg) algorithm that synchronously aggregates the updates of local models, while satisfying the basic requirements for privacy protection. Later, FedProx was proposed by Li et al. (2020) that adds a proximal term in the loss function improve the statistical stability of the training process.
A key characteristic of federated learning is the non-IID (independent and identically distributed) data distribution among clients. Consequently, different clients learn from different data distribu- tions, which leads to high heterogeneity among local models and degrades the effectiveness of model aggregation (McMahan al. 2017 Wang al. 22200).. For extreme cases with highly skewed data distribution, traditional federated learning methods even lack a theoretical guarantee of convergence
Yu et al. 2020 Rothchild al. 2020). Recent studies have attempted address the non-III) issue by constructing a global shared dataset based on the local data to mitigate the data imbalance effect (Krizhevsky et al. [2009] Zhao et al. 2018 Yoshida et al. 2020 Oh al. 2020p. Particularly, several methods (Krizhevsky et al. 2009 Yoshida et al.) 2020) proposed data sharing strategies to distribute a percentage of raw data among clients, but naive data sharing seriously violate the privacy require- ment in FL. Besides, data augmentation approaches (Zhang al. 2021 Oh et al. 2020) generate synthetic samples without directly sharing private data. However, these methods are still susceptible to data leakage without a privacy-preserving mechanism. Thus, it is important to investigate how

---

Under review as a conference paper at ICLR 2022
to generate prraacypressrviing synthetic samples, well as how to effectively utilize the private dataset and synthetic dataset in the FL framework.
To alleviate the heterogeneity of the data distribution in FL without compromising privacy, we pro- pose a novel framework, named Synthetic Data Aided Federated Learning (SDA-FL), which re- solves the non-III issue by sharing differentially private synthetic data. Under this framework, each client pretrains a local generative adversarial network (GAN) (Goodffllow et al.] 2020 to gener- synthetic data, thus avoiding sharing the raw data directly. A parameter server (PS) performs pseudo-labeling for the uploaded synthetic data and constructs a global shared dataset. In each global aggregation, the PS updates the pseudo labels of the synthetic dataset based on the received models and updates the global model with the confident synthetic data. This iterative labeling pro- cess constitutes a synthetic dataset with high-confidence labels, which is fed back to the clients to construct their training samples, including the local private data and synthetic data, following an al- most identical distribution. Meanwhile, to provide a privacy guarantee for the generative models, we adopt the widely used differential privacy (Dwork 2008) to ensure that the actual training samples can not be revealed by adversaries, In particular, we use the Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al.] 2017) 7) as the backbone and introduce the privacy budget by adding artificial noise to the gradients in the training process. The SDA-FL framework is mostly compatible with many existing FL. methods and can be applied in both supervised and semi-supervised settings.
We evaluate the proposed framework on several standard datasets under both supervised learning and semi-supervised learning, which shows its effectiveness in resolving the non-IlI distribution problem even in the extreme case (with only one class of real data at each client). We also investi- gate the tradeoff' between the privacy budget and the FL performance, which demonstrates that this framework is fully functional under the strict differential privacy protection requirement. Our major contributions are summarized as follows:
We develop a global shared synthetic dataset by the differentially private GAN models. This data augmentation method effectively resolves the non-IID problem in FL and main- tains the local data privacy.
To utilize the global shared synthetic dataset efficiently, we propose a novel framework, SDA-FL, in which the PS provides and keeps updating high-confidence pseudo labels for the synthetic dataset iteratively, thus improving the performance of the global model.
We evaluate our framework several benchmark datasets under the high heterogeneous data distribution, where our framework consistently outperforms other baselines under both supervised and semi-supervised learning. Besides, our framework experiences slight performance degradation with stringent privacy budget.
2 RELATED WORK
Non-IID Challenges in Federated Learning The pioneering work in FL is FedAvg (MeMahan) et al. 2017). which simply averages model parameters trained by different clients in element- wise manner, weighted proportionately by the clients' data size. However, this algorithm often Ieads to slow convergence and performance degradation when data is heterogeneous across local clients (Li et al. 2019). To alleviate this issue, Wang et al. (2020a) leveraged the reinforcement learning to intelliiently choose the client devices to participate in each round of federated learning counter- balance the bias introduced by non-lID data. Besides, some studies proposed personalized layers to adjust the local model according to the private dataset (Arivazhagan al. 2019: Liang al. 2020).
For example, FedPer was introduced in AArvazhagan et al. (2019) that captures personalization as- pects in federated learning by only using the local data to train the personalized layers. Knowledge distillation (Deng et al.) 2020; Lin et al.) (2020) is also a promising idea of personalized federated learning, which transfers information from other clients to a certain client. Although the aforemen- tioned methods outperform FedAvg on strongly non-lll) data, the extra computational costs are not affordable by the resource-constrained loT devices (Zhu et al., 2021). Thus, client clustering has been proposed to construct a multi-center framework by grouping the clients into different clusters based on the model similarity (Ghosh et al., 2020; Kopparapu & Lin 2020). As clients avoid the negative knowledge transfer from the dissimilar models (Pan & Yang 2009), clustering methods can alleviate the negative effect of the non-IID data distribution. However, such methods are not scalable as the communication cost increases largely with the number of clients.
2

---

Under review as a conference paper at ICLR 2022
Data Augmentation and Privacy Preserving Recently, methods based some form of data sharing have received increasing attentions and achieved prominent performance in FL (Zhu et al,
2021 Zhao et al.] 20188) Particularly, data augmentation approaches showed the potential capability to address the non- issue by modifying the local distributions without sharing private data (Duan) al. 2019 Zhang al. 2018 Faton-Rosen et al. 2020). A vanilla data augmentation method was proposed by Duan et al. (2019). in which each client performs local data augmentation to reach a balanced global distribution. Besides, several methods (Oh al., 2020] Shin al. 2020 Yoon al.
2021 leveraged the Mixup technique (Zhang et al. (2018) to conduct data augmentation, where the clients share the blended local data and collaboratively construct a new global dataset to tackle the
non-III issue. However, frequent data exchange may be susceptible to privacy issues. Recently, the GAN-based data augmentation framework (Bowles et al. 2018 Yoshida et al.] 2020) was shown to be promising for reducing the degree of local data imbalance in FI.. The general idea is to train good generative model in the presence of non-IID data. Under this framework, each client sends its local seed samples to the server for the GAN training, and then well-trained generator will be sent to all the clients for model updating. However, directly sending local data samples to the server violates the data privacy requirement. FedDPGAN (Zhang al.,, (021) suggested an alternative that all the clients train a global GAN together based on the FL framework to supplement the scarce local Covid-19 19 data. Unfortunately, the white-box generators sent to the server are vulnerable to adversarial attacks (Chen et al. 2020).
3 PRELIMINARY
Federated Learning FedAvg (McMahan et al., 2017) is one of the representative training al- gorithms in federated learning, and it trains shared neural network without disclosing the raw data of clients. For every step / = 0,..., T - 1, every client h E & downloads the global model we and updates the local model with local dataset TD, via stochastic gradient descent (SGD), i.e., Wx+1* k + we - 1/ Ve. The updated local models are then sent back to the PS for a weighted ag- gregation Wt+1 f K Ekes, w These procedures repeat until convergence all the T training steps are exhausted.
Differentially Private Generative Adversarial network We utilize a differentially private gen- erative adversarial network (DPGAN) for data augmentation while protecting local data privacy. To avoid the gradients vanishing and mode collapse problems (Arjovsky & Bottou| [017 AArjvssy cet al. (011)) that are commonly encountered by many GAN models (Mirza & Osindero [001|] Rad- ford et al. 2015: Odena et al. 2017), we adopt the WGAN-GP (Gulrajani et 2017). which is up- graded based on the Wasserstein GAN (WGAN) (Arjovsky& & Bottou 2017
al. 2017), 7),
as the backbone of the generative model for a stable training procedure. Besides, we add noise to the gradient in the generator training process, which provides privacy control in terms of differential privacy (Torkzadehmahani et al. 2019)) The definition of differential privacy is as follows:
Definition 1. (Differential privaey p: A random mechanism Ap satisfies (e.f))-iifeeretiall privacy if for any output's 's subset (S) and for any two adjacent datasets D, D', the following probability inequality holds:
P (AA(DD) E S) S .
(Ap (D')
S) +
(1)
Particularly, is an optimal probability that guarantees the validity of the inequality, and is the pri- vacy budget indicating the privacy level, i.e., smaller value of implies stronger privacy protection. To satisfy the (c, 5) diffferentia privacy, we follow the work (Xie et al. 2018) and add the Gaussian noise to the gradients updated in each discriminator training iteration. The relationship between the noise variance and differential privacy is shown as follows:
On 2 na log
/e.
where g and nu denote the sample probability and the number of discriminator training iterations in each training round, respectively.
Although our work leverages the differential privacy to address the extra privacy problem introduced by the GA N-based data augmentation, the potential privacy risk of FL. is still open prohlem, which is out of the scope of our study.
3

---

Under review as a conference paper at ICLR 2022
Pseudo Labeling Although GAN-based data augmentation can generate synthetic data, the la- bels of the synthetic data are unknown, which hinders the training of the model through back- propagation. In semi-supervised learning, pseudo labeling is widely used method in classification tasks to generate confident pseudo labels (Sohn et al. 2020). hard pseudo label will be applied to the unlabeled data if the maximum class probability (MCP) of the inference result (Corbière al. 2019) exceeds predefined threshold т. In our experiments, the threshold is set to be 0.95.
(JMtodel upicading
Senver Updine
Poesdd Label Updode
Synthetic Docsset
Nloood Downloading
("Masia Aggregation
Figure I: Overview of the proposed SDA-FL framework. Before the federated learning, the syn- thetic data from all clients are sent to the PS to construct global synthetic dataset. In each iteration, every client first downloads the global model and updates the pseudo labels of the synthetic data for local training. The local models are then uploaded to the PS for pseudo label updating and model aggregation. Lastly, the PS updates the glohal model W1+11 with the updated synthetic dataset.
4 SYNTHETIC DATA AIDED FEDERATED LEARNING (SDA-FL)
We now introduce the SDA-FL framework that adopts GAN-based data augmentation to alleviate the negative effect of the non-IID data. The system diagram is shown in Fig. and key algorithmic innovations built upon the classic FL. framework are elaborated below.
Global Synthetic Dataset Construction At the beginning of the federated training, each client pretrains local GAN model generate synthetic samples based its private data. Then, the synthetic samples are sent to the PS to construct a global shared synthetic dataset. To effectively leverage the synthetic dataset for federated learning, perform pseudo labeling for these samples, which is critical for the effectiveness of the proposed framework.
In federated learning, the local model the most suitable network predict the pseudo labels for the synthetic data generated by the corresponding local GAN, as they are both trained with the same data. In our framework, improve the confidence level of the pseudo labels, utilize the local models to perform the pscudo labeling for the unlabeled synthetic data. Specifically, after receiving the local model w each aggregation step, the PS utilizes the model predictions assign pseudo label for each unlabeled synthetic instance if the maximum class probability is higher than a predefined threshold т. this way, we are gradually generating synthetic data samples with high-quality labels.
Synthetic Data Aided Model Training Augmented by the samples from the shared synthetic dataset that with the confident labels, the data available for local training at different clients reach approximately homogeneous distribution. To make good use the synthetic data, we leverage the M irup method (Zhang et al., 2018), which utilizes linear interpolation between the real sample (I;, yi) and the synthetic sample (æ,, fi) to augment the real data:
H = Aιă; + (1 -1-b0 A1)
(3)
X (1 Xι )1)1i.
where A follows the Beta distribution (Beta(a, (a, a:) with [0. Combined with the cross- entropy loss, the Mixup loss for local update becomes:
e, >ee(f(X; Y.) (1 xx))( Y.).
(4)
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Synthetic Data Augmented Fed- erated Learning (SDA-FL)
Algorithm 2: Model Update
LocalUpdate(k;; wi; Xk: Yi. XA, Ya):
Input: Local dataset D2 = {X Yz),
w t wt
Pretrained generators G = {GA}
for = .......-11 do
I (I,...,N)
Split D3 into batches of size for batch( X, Y) do
Initialize Wo. Synthetic dataset B. (X,, Y,} for server, i. {Xx, Y for
Sample batch( X, Y) from
clients
(X YR)
All clients upload the synthetic data generated by Gk to PS
Use (X, Y) and (X, Y)
update w following (6)
for t = 0...., T do
end end return ServerUppatt( (ir X,, Y.): + Wi+1 for e = 0,...,
S, < K clients are selected random Send wy and updated pseudo labels to
clients k E S
Clients Side:
for i E S do
Updata local synthesis dataset( Xx: Vi) wi+1 LocalUpdate(k; wi; Xu, Yk, XR, K
do Sample batch( X, Y) from (X., Y.) Evenly divide (X, Y into (X Yi) and (X2, 12)
end
Parameter Server Side:
Use (X1, Y1) and (X,, Y2) to
Pseudo Label Update
update w following (6)
w1++1 <- K
1 Ekes, w/+1
end return w
wi+1 < ServerUppaate ww11: X., &))
end
In addition, to keep more information of the local clients, another cross-entropy loss term is intro- duced for the real data follows:
2= (((x; Y.).
(5)
Then SGD is applied to update the local model:
w; f- w: 7 V(ei + )262),
(6)
where Ag is a hyperparameter to control the retention of the local information and will be set in different values in experiments. More details about the local update are shown in Algorithm
In contrast to the traditional federated learning where the PS does not have access to any data to update the global model, the PS in our framework can keep the entire global synthetic dataset D: and uses it to update the global model, thereby helping to enhance the performance.
Interplay between Model Training and Synthetic Dataset Updating Because of the non-IID problem, there is a distinct difference the weights of the local models among clients in feder- ated learning. Furthermore, simply sharing the synthetic data with the PS and other clients will not result in the good training performance of the global model due to the lack of confident pseudo labels. Our framework, SDA-FL, deeply integrates federated learning and synthetic data, which effectively alleviates the non-lID issue and helps to train a good global model. As shown in Fig. after creating the global synthetic dataset, will be shared with all the clients so that the lo- cal data, including both the real and the synthetic data, approach an IID distribution. Without doubt, the supplement of synthetic data results in the improvement of the local models. After being uploaded to the PS, the updated local models are used to perform the pseudo labeling and update the global synthetic dataset by the PS. The stronger local models can boost the confidence of the pseudo label. With the improvement of labeling for the global synthetic dataset, the PS can utilize to update the global model and all the clients can improve their local models subsequently at the next-round training. Thus there is interplay between the model update and the synthetic dataset at every training round: global synthetic dataset - global model update - local model update - labeling for global synthetic dataset, which results in converged and perfore model.
5

---

Under review as a conference paper at ICLR 2022
SDA-FL vs. Traditional FL From the above descriptions, the proposed SDA-FL framework in- troduces two additional operations at the server side, namely, pseudo label update of the shared synthetic dataset and server-side model update, as shown in Fig. 1. These two main innovations contribute to the performance improvement with non-IlI data. traditional FL. algorithms (Li et et 2020 Karimireddy al. (220), clients update their models based on local private data, which may lead to performance degradation when data are heterogeneous over clients. In our framework, the local dataset augmented by the GAN-based synthetic samples, which alleviates the non-IlI problem. Furthermore, the PS in the traditional FL system only performs model aggregation based on the uploaded local models. In contrast, our framework generates high-confidence synthetic data to update and improve the global model at the PS. Overall, with a shared synthetic dataset and an effective mechanism for pseudo label update, SDA-FL can overcome the heterogeneous data distri- butions among clients and enhance the model update at the server. We envision that this framework can be applied to develop other data augmentation- -based methods for more efficient federated learn- ing.
5 EXPERIMENTS
In this section, evaluate the proposed SDA-FL Fra-FL.framwwork in the presence of non-lID data for both supervised learning and semi-supervised learning. The experimental results on different benchmark datasets demonstrate the effectiveness of the proposed framework compared with baseline methods. Besides, ablation studies are conducted to illustrate the impacts of key factors on the generated samples and the model convergence.
5.1 EXPERIMENTAL SETUP
Datasets We implement the SDA-FL framework in the scenario where all the real data are stored in clients and the PS only has access to the synthetic data. We four benchmark datasets, in- cluding MNIST (LeCun et al., 1998). FashionMNIST (Xiao et al. 2017). Cifar- (Krizhevsky) 2009), and SVHN (Netzer al. 2011 to test the proposed method. The details about the datasets, models, and experimental settings are deferred to Appendix [ To demonstrate the effectiveness of our framework in handling different non-IID scenarios, we provide extensive experiments in the following five tasks.
Tasks 1) Federated supervised learning: We compare our framework with FedAvg (McMahan et al. 2017). Fedprox (Li et al. 2020), Scaffold (Karimireddy et al. 2020). Naivemix, and Fedmix Yoon al. 2021 on MNIST, FashionMNIST, Cifar-10, and SVHN datasets.
2) Federated semi-supervised learning: We extend our framework to the semi-supervised learn- ing setting in MNIST, FashionMNIST, and Cifar- 10 datasets by performing pseudo labeling for the unlabeled data. To show the effectiveness of our SDA-FL. method, we compare it with Local Fix- match+Mixup (Diao et al. 202I), Local Fixmatch (Sohn et al., 2020), and Local Mixup (Zhang
al. 2018).
3) The trade-off between the privacy budget and performance: To investigate the influence of the differential privacy mechanism on the quality of the synthetic data and the model convergence, we validate our framework under different privacy budgets. In particular, we select the Fréchet In- ception Distance (FID) (Heusel et al. 2017) as a metric quantify the quality (similarity) of the synthetic data.
4) The effectiveness of updating pseudo labels and the server update: Compared with the tra- ditional federated learning, SDA-FL leverages the pscudo labeling mechanism to generate high- confidence pseudo labels for the synthetic data, which are also used by the to further update the global model. To demonstrate the efficacy of these two mechanisms, we conduct ablation studies on a varying number of rounds for updating the pseudo labels and a varying number of server update steps.
5) Ablation studies of the computational cost and the number of samples for training the gen- erators: To evaluate the performance of the generative models, conduct ablation studies to investigate the influence of the training epochs and the number of samples on the quality (similarity) of the synthetic data.

---

Under review as a conference paper at ICLR 2022
Table 1: Test accuracy of different methods in various datasets. We report the best accuracy of Fedprox and Fedmix by tuning the parameters p = 0.001, 0.1, and X = 0.01, 0.2, 0.5, respectively.
Relasolelient
3
Algoittmm
Mnist
FnchionMiiit
Clari0
SVHN
Muist
FachionMiiist
Cifar10
SVHN
Minist
FashionMuist
Clarl0
SVHN
FodAvg
K3,4456
16.30%
13.36%
14,05%
97.61%
73.50%
61.28%
81.11%
9S.4216
82.4716
79.33%
54.15%
Fodprox
64.17%
57.14%
11.34%
17.538
97.55%
75.766%
63.16%
86.25%
96. 38.38%
83.41%
79.54%
90. 15%
Scaffold
25,395
56.80%
12.81%
11,62%
96.17%
70.82%
50.78%
73.34%
96.8915
77.68%
79.35%
50.13%
Naivemix
$4,3556
66.62%
14.39%
84.35%
79.54%
64.39%
11%1
$2.0999
Fediax
90,9096
16.78%
82.41%
92.61%
98.26%
96.87%
$4.56%
98.16%
5.2 PERFORMANCE OF SDA-FL ON NON-IID FEDERATED LEARNING
1) Federated Supervised Learning We compare the performance of various methods under the same federated setting. With varying numbers of classes per client, the experimental results in Table 1 show that our framework outperforms other methods by large margin, which is attributed to the GAN-based data augmentation that mitigates the detrimental effects of the data heterogeneity. Under the severe non-IID scenario (i.e., each client only has one class of data), our SDA-FL method maintains accuracy of 88 58.46% in the SVHN classification task, while other baselines suffer from great performance degradation. In the Cifar- 10 experiments, our framework outperforms the classic Naivemix and Fedmix algorithms at least 5.0% with three classes of data at each client. This indicates that augmentation via synthetic data is more effective than averaging real data samples, although Fedmix adds another derivative term when averaging real images explicitly handle the non-III problem.
Recently, GANs have been widely used solve the non-IID and data scarce issues based on the federated learning in the medical field (Nguyen et al. 2021 Zhang et al.) 2021), which can afford the additional computational cost for training the GANs. To further show the reasibility of SDA-FL, we include another comparison on a Covid-19 19 dataset (Chowdhury et al. 2020 See more details about the Covid-19 experiment Appendix B
Wwr hi Mayy Mww.MM MA
(a) MNIST
(b) FaShionMNIST
(c) Cifar-10
Figure 2: Test accuracy of different methods for federated semi-supervised learning on MNIST, FashionMNIST, and Cifar-10 classification tasks.
2) Federated Semi-Supervised Learning The results in Fig. show that our framework achieves faster convergence and better performance than other algorithms, indicating its robustness and gen- eralizability. Particularly, compared with Local Fixmatch+Locl Mixup, our method improves the accuracy by nearly 10% in the FashionMNIST classification task. the Cifar-10 dataset, the base- line methods framework converges in this challenging scenario and improves the test accuracy by a significant margin. This is because the proposed pseudo labeling mechanism can provide high-quality labels for the synthetic and unlabeled samples, which benefits the federated training process.
able train usable global model (with test accuracy below 40%), while our
3) The trade-off between the privacy budget and performance The privacy budget e provides the privacy protection for the generation of the synthetic dataset, but small value of affects the quality of the synthetic samples and thus degrades the training performance. To investigate the impact of the privacy budget, evaluate the performance of the proposed method under different
7

---

Under review as a conference paper at ICLR 2022
Test Accaracy (SQ)
10:0
Privacy Budget
Privacy Budget
Privacy Budget (c) Cifar-10
(a) MNIST
(b) FNSSI
Figure 3: Test accuracy and FID function of the privacy budget. We run three trails, and report the mean and the standard deviation of the test accuracy. The FIDs of the real samples MNIST, MNIS,, and Cifar-10 are 10.54, 23. and 42.70, respectively, which are much larger than those of the synthetic data. It illustrates that the differentially private synthetic data, while of low image quality, still effectively help to improve the training performance with non-IID data.
(a) Synthetic data
(b) Original Cifar- 10 dataset
Figure 4: Visual comparison between synthetic data and the original Cifar- 10 dataset. The synthetic data are sampled from the generator trained with privacy budget = 200.
values of e. As shown Fig the proposed framework suffers from performance degradation with small privacy budget. Compared with protection-free scenario (e oc), setting a strict privacy budget (c = 5) (Xie et al. 2018) leads to around 0.61% % and 2.59% accuracy drop in the MNIST and FashionMNiST datasets, respectively. Besides, utilize the FID to measure the quality of the generated samples, where smaller FID score denotes a better image quality. As illustrated in 3ig decreasing the privacy budget increases the FID score, which implies the degradation of the quality of the generated samples. Note that although our SDA-FL method is trained under the strict privacy requirement, compared with the results in Table our framework still maintains desired performance over other methods.
In the Cifar-10 10 classification task, as the generators are sensitive the Gaussian noise added the gradient, we adopt large privacy budget (e = 50) when training the proposed method. (c) shows that our framework outperforms Fedmix and Fedprox in federated supervised learning. To illustrate the privacy guarantee on the generators in our framework, we show some synthetic samples in Fig[4(a) which are extracted from the generators trained with c = 200. comparison with the samples from the private datasets, the synthetic images are blurry and noisy, which means the GAN models are not memorizing the real data, and the local privacy not undermined. Besides, the performance improvements shown in 3ic and Table [ indicate that the synthetic images with low perceptual quality can capture the necessary information for the model training.
4) The effectiveness of updating pseudo labels and the server update In comparison to the traditional federated learning, our framework can update the global network with the synthetic data at the server, which has the potential to further improve the performance. The results in 3 that the model performance on FashionMNIST will drop nearly 3% without any server update. However, too many server updates also degrade the performance because too much involvement of the synthetic data will relieve the effects the local update. The empirical results show that the model trained solely with the synthetic data (the steps) only get 66.0% accuracy, which highlights the necessity to effectively utilize the synthetic data as well as the local data for model training.
Besides, keeping updating pseudo labels for the synthetic data improves the model performance. As illustrated in Fig the accuracy gradually increases when continuously updating the pseudo labels,
8

---

Under review as a conference paper at ICLR 2022
$6.90
$6.5
15.01
14.00
16.99
15.00
15.93
80:2
18.00
25.80
$4.99
18
5481
Keyp Updeting
The Nww.brrbrrrdd The Surver (oqqe) crqpe)
Updatig
Figure 5: Test accuracy on FashionMNIST with varying server update steps. The oo steps mean that the model is only trained with the synthetic data.
Figure 6: Test accuracy on FashionMNIST with different rounds of pseudo label updating.
which demonstrates that our framework can improve the confidence level of the pseudo labels in the training process. Note that since framework only updates the pseudo labels instead of the samples, the extra communication overhead is neglectable.
5) The ablation studies of computational cost and the number of samples for training the generators present results Fig and training samples for generators result low-quality symthetic data and degraded performance. The generator only gets an FID score 217.81 (23.17 for real data) and an accuracy of 83.76% when training the generators with 30 rounds. In addition, shown in Fig.8) under the fixed 30- round computational cost, using fewer samples for training the generators reduces the quality of the synthetic data and thus affects the performance. Nonetheless, despite the low computational cost (30 rounds) and the small number of samples (1000 samples) used to train the generators individually, our framework still outperforms other baselines (82.41% in Fedmix), demonstrating the robustness of the proposed method.
7 and Fig which show that insufficient training rounds
In addition the influence of the quality of synthetic data on the performance, other factors, such as the threshold т, X2, and the size of the synthetic dataset, also affect the global model. More ablation studies about these factors are shown in Appendix
21730
54.83
14.59
240
34.39
$5.09
15.85
(M612
15.55
31.59
224.11
65.76
118.09
2483
33.53
53.56
1281
140
180
1499
2000
Conpetatined Costowash for rishing GANU)
The of sample for mrdiingggg the poenraee beads
Figure 7: Test accuracy and FID on Fashion- MNIST, under varying training rounds for the GANs.
Figure 8: Test accuracy and FID on Fashion- MNIST, under varying samples for training the GANs locally.
CONCLUSIONS
We proposed a new data augmentation method to resolve the heterogeneous data distribution prob- lem in federated learning, by introducing pretrained GANs to construct a differentially private global shared dataset. To effectively utilize the synthetic data, novel framework, named Synthetic Data Aided Federated Learning (SDA-FL), was developed, which generates and updates confident pseudo labels for the synthetic data samples. By supplementing the non-lID private datasets with high- quality synthetic data samples, this framework effectively improves the consistency among local models. Experiment results showed that - outperforms many existing baselines by remark- able margins in various image classification tasks and is still functional under strict differential privacy protection.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint 1olp:0992.00/1 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 1701 00002.04422.... 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Ham- mers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, and Daniel Rueck- ert. Gan augmentation: Augmenting training data using generative adversarial networks. arXiv preprint arXiv: 10863, 2018.
Dingfan Chen, Ning Yu. Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp. 343-362, 2020.
Muhammad EH Chowdhury, Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muhammad Ab- dul Kadir, Zaid Bin Mahbub, Khandakar Reajul Islam, Muhammad Salman Khan, Atif Iqbal, Nasser Al Emadi, al. Can ai help in screening viral and covid-19 pneumonia? IEEE Access, 8:
132665-132676, 2020.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In Conference on Neural Information Processing Systems (NIPS), pp. 2902-2913, 2019.
Yuyang Deng, Mohammad Mahdi Kamani, and Mchrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13461, 2020.
Enmao Diao, Jie Ding, and Vahid Tarokh. Semifl: Communication efficient semi-supervised feder- ated learning with unlabeled clients. arXiv e-prints, pp. arXiv-2106, 2021.
Moming Duan, Duo Liu, Xianzhang Chen, Yujuan Tan, Jinting Ren. Lei Qiao, and Liang Liang. Astraca: Self ballnbalnncing federated learning for improving classification accuracy of mobile deep learning applications. 2019 IEEE 37th international conference on computer design (ICCD), pp. 246-254. IEEF, 2019.
Cynthia Dwork. Differential privacy: A survey of results. In International conference on theory and applications of models of computation, 1-19. Springer, 2008
Z. Raton-Rosen, Felix S. Bragman, Sébastien Ourselin, and M. Cardoso. Improving data aug- mentation for medical image segmentation. Conference on Medical Imaging with Deep Learning (MIDI.), 2020.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Conference Neural Information Processing Systems (NIPS), 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144 2020.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In NIPS, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Conference on Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR, abs/1 706.08500, 2017. URL JRLhttp:: /ttp://aaiiiii http:///wwww.oog//////////0/ 08500
10

---

Under review as a conference paper at ICLR 2022
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020.
Kavya Kopparapu and Eric Lin. Fedfme: Sequential efficient federated learning on non-iid data. arXiv preprint arXiv:2006. 10937, 2020.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-22 1998.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems (MLSys), 2020.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov. and Louis-Philippe Philippe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.011 015523, 2020.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Conference on Neural Information Processing Systems (NIPS), 2020.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication-effiiiee learning of deep networks from decentralized data. International Con- ference Artificial Intelligence and Statistics (AISTATS), 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.1784 2014.
arXiv preprint
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, and Albert Y Zomaya. Fed- erated learning for covid-19 detection with generative adversarial networks in edge cloud com- puting. IEEE Internet of Things Journal, 2021.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with iliary classifier gans. International conference on machine learning, pp. 2642-2651. PMLR, 2017.
Seungeun Oh, Jihong Park, Eunjeong Jeong, Hyesung Kim, Mehdi Bennis, and Scong-Lyun Kim. Mix2fld: Downlink federated learning after uplink federated distillation with two-way mixup. IEEE Communication Letters, 2020.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345 1359, 2009.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0643: 2015.
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-eficien federated learning with sketching. In International Conference Machine Learning, pp. 8253-8265 PMLR, 2020.

---

Under review as a conference paper at ICLR 2022
MyungJae Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyum Kim. Xor mixup: Privacy-pre preseevie data augmentation for one-shot federated learning. Interna- tional Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML 2020 (FL.-ICMI 2020.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Hao Wang. Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning non-iid data with reinforcement learning. In IEEE INFOCOM 2020-IEEE Conference on Computer Commu- nications, pp. 1698-17880 1707. IEEE, 2020a.
Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 2020b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: (tti::1708.077. 07747, 2017.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint a721vv/880.006739, 2018.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. arXiv preprint wxXiv:2107000233 2021.
Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto, and Ryo Yonetani. Hybrid- for wireless networks: Cooperative learning mechanism using non-iid data. In ICC 2020-2020 IEEE International Conference on Communications (ICC), pp. 1-7. IEEE, 2020.
Felix Yu, Ankit Singh Rawat, Aditya Menon, and Sanjiv Kumar. Federated learning with only positive labels. In International Conference on Machine Learning, pp. 10946-10956. PMLR, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. International Conference on Learning Representations (ICLR), 2018.
Longling Zhang, Bochen Shen, Ahmed Barnawi, Shan Xi, Neeraj Kumar, and Yi Wu. Feddpgan: Federated differentially private generative adversarial networks framework for the detection of covid-19 pneumonia. Information Systems Frontiers, pp. 1-13, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint / 2018
Hangyu Zhu, Jinjin Xu, Shiqing I. Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. arXiv preprint 20X.::100606644. 2021.
A EXPERIMENTAL DETAIL
Datasets To demonstrate the generalizability of our framework, we use five widely used datasets to test our frameworks, where the details of the datasets are summarized experiments, we equally divide the training samples and assign them to the clients based the non- IID setting. Specifically, given the number of classes per client as C and with K clients, the whole dataset will be split into K C subsets. and each subset only has a single class. Then all the subsets of data will be randomly shuffled and distributed to clients. For example, in the MNIST experiment, under the scenario that each client only accesses two classes of data, given K 10 clients, each
Table 2 all the
12

---

Under review as a conference paper at ICLR 2022
client keeps 60000/10 6000 samples, each class with 3000 samples. The data distribution is kept the same in the comparison between our framework and other baselines. Except for the Covid- 19 dataset, set 10 clients for all the experiments, and all the clients participate in the training every communication round.
Models For fair comparisons, we adopt a simple CNN model that consists of two convolution lay- ers and two fully-connected layers for the MNIST and PashionMNIST classification tasks. Mean- while, ResNet18 (He et al., (016)) is used for classifying the Cifar- 10, SVHN and Covid-19 image datasets. To generate the qualified synthetic samples, we set a generator with four deconvolution layers and a discriminator with four layers convolution layers followed by a fully-connected layer.
Table 2: The statistics of the datasets
Dataset
MNIST
FashionMNIST
Cifar-10
SVHN
Covid-19
Training Samples
60000
50000
50000
73257
12000
Testing Samples
10000
10000
10000
26032
3153
Total dataset classes
10
10
10
10
Table 3: The details of parameters federated supervised learning and federated semi-superrised learning
Parameters
Supervised
Semi-supervise
Learning rate (ax
0.03
0.03
Batch size (I)
16 (labeled data)+64 (Unlabeled data)
Local steps( F!)
90
SDA-FL. (39 for Cifar- 10), baselines:
Server update steps
(10 for Cifar- 10)
(0 for Cifar-10)
Communication rounds(7')
200
200
Training rounds for GANs (90 steps/round) (400 for Cifar- 10)
200 (400 for Cifar-10)
Number of clients (K)
10
10
Fraction of clients
1.0
1.0
The size of the synthetic dataset
40000
40000
X (for our framework)
1.0 (0 for Cifar-10)
1.0 (Õ for Cifar- 10)
Threshold (r, for our framework)
0.95
0.95
(for Fedprox)
0.001, 0.01, 0.1, 1.0
À (for Naivemix and Fedmix)
0.01, 0.1, 0.2, 0.5
Mr. (for Naivemix and Fedmix)
Table 4: The experimental settings of the ablation studies.
Ablation study
Variable
Computational cost
Training rounds for GANs
30, 50, 100, 200
The samples for training GANs
The number of samples used for trining the GANs locally
1000, 2000, 3000, 6000
The size of the synthetic dataset
The size of the synthetic dataset
2000, 10000, 20000, 40000, 50000
0,25, 0.5, 0.75, 1.0, 2.0, 5.0
Threshold (r)
Threshold
0.5, 0.7.0.8. 0.95
Experimental settings of tasks 1) Federated supervised learning: In the federated supervised and semi-supervised experiments, the parameters always are kept the same in our framework, and they are shown in the Table 3 In the baseline experiments, we set = 0.001, 0.01, 0.1, 1.0 for Fedprox, and À 0.01, 0.1.0.1.... 0.2, 0.5 for Naivemix well Fedmix. The best performances of them are reported in the Table 2) Federated semi-supervised learning: In this experiment, we compare our framework with other baselines in MNIST. FashionMNIST,, and Cifar-10 10 datasets. Every client only keeps two classes of data, containing 50 labeled instances, and the rest are unlabeled. For fair comparisons, the total gradients steps of SDA-FL are the same as other methods.
3) The trade-off between the privacy budget and performance: We evaluate SDA-FL in MNIST and FaSHionMNIST with the privacy budget 5, 10, 20, and 30 training rounds (each round with 90 steps) for GANs. In the Cifar-10 experiment, we set the privacy budget e = 50, 100, 200, co to train the GANs with 400 training rounds (each round with 90 steps). Every client keeps two classes of data in all experiments.
4) The effectiveness of updating pseudo labels and the server update: In this task, we test
13

---

Under review as a conference paper at ICLR 2022
framework in the FashionMNIST dataset and distribute two classes of data to every client. In our framework, the PS updates the pseudo labels for the synthetic dataset every communication step to maintain the confidence of the synthetic data. Compared with the federated supervised learning, change to update the pseudo labels in the first 2, 10, 20 communication steps instead of updating the pseudo labels at all the communication steps in our framework. In the ablation study of the server update, we perform the experiments with different steps of server update (0, 20, 50, 100, oc) and batch size B = 64.
5) The ablation studies of computational cost and the number of samples for training the generators: The settings of all the ablation studies are the same as the federated supervised learning, and we test them in FashionMNIST with 2 classes of data at each client. All the experimental settings of ablation studies are shown in Table
Table 5: The data distribution of the Covid- 19 experiment
Hospital Data
0
2
3
Normal 2000
Covid-19 19 750
Pncumonia 250
Normal 2000
Covid-19 750
Pneumomia 250 Normal 2000
Covid-19 19 750
Pneumonia 250
Normal 2000
Covid-19 19 750
Pneumonia 250
Table 6: Test accuracy on Covid-19 dataset. The FedAvg (IID) represents the scenario where the training samples distributed to all the clients by average based the IID distribution.
Algorithms
FedAvg
Fedprox
Scaffold
Naivemix
Fedmix
FedDPGAN
FedAvg (IID)
SDA-FL
Test Accuracy
94.05%
95.03%
94.30%
94.14%
94.28%
94.57%
95.19%
96.25 %
B THE APPLICATION IN THE COVID-19 DATASET
Because of the scarcity of Pneumonia data, we only set up six clients in this experiment, each of which keeps two classes of data, and the data distribution is shown in Table We test SDA-FL with 30 local steps, 10 server steps, and 50 rounds for training the generators locally. In addition to the baselines in the federated supervised learning task, we also add FedDPGAN (Zhang et al. [02II) for comparison, which trains a global GAN to solve the non-lll) issue for medical applications, and the results shown in Table
Our framework outperforms the other baseline algorithms by a remarkable margin. Particularly, SDA-FL is superior FedDPGAN by 1.68% in the test accuracy, which indicates that the GANs which are trained individually generate higher-quality synthetic data than the global GAN trained based on the federated learning framework. Furthermore, in addition addressing the non-III issue, we also claim that the synthetic data can augment the data distribution, as SDA-FL is superior FedAvg (IID) by 1.14% in the test accuracy. a result, we assert that SDA-FL is able show its advantages of solving non-lll) problems and generating high-value data in medical applications.
C ABLATION STUDIES
The size of the synthetic dataset Before the federated learning, all the clients constitute the syn- thetic dataset together. Larger synthetic datasets can typically provide more qualified data to clients and PS, thus improving performance, but they require larger storage space for clients. The results in FigDssow that, although the test accuracy on FashionMNIST decreases as the size of the synthetic dataset declines, our framework achieves 85,44% test accuracy with each client only uploading 200 samples, which is acceptable in exchange for less storage space requirement.
The hyper-parameter X2 A2 is the parameter to control the use of the local real data. The results in Fig [0 show that compared with the normal settings of A2, the test accuracy on MNST drops by nearly 5.0% with X2 = 0.25, indicating that the training of the models is heavily reliant on the real data. Besides, an excessive X2 also degrades the performance since it relieves the effects of the synthetic data and cannot mitigate the non-lII problem much possible. In comparison, a moderate 22 is capable of combining the real data and the synthetic data intelligently and thus enhancing the models and solving the non-IID issue.
14

---

Under review as a conference paper at ICLR 2022
Threshold Threshold sets the criterion for selecting the high-confidence pseudo labels. The results on FashionMNIST are shown in Fig [T which indicates that a small threshold impairs the performance because some synthetic data with now-confidence pseudo labels are still considered qualified and used to train the models. An excessive threshold, on the other hand, filters out a lot of qualified data, which affects performance. Therefore, 0.95 is reasonable value to achieve balance between quality and the quantity of the pscudo labels.
synthetic dataset
Threshold
Figure 9: Test accuracy on FNShionMNIST, under varying sizes of the synthetic dataset.
Figure 10: Test accuracy on FashionMNIST, under varying X2 in the local update.
Figure 11: Test accuracy FashionMNIST, under varying thresholds.
15


Reviews:
[{"review_id": "m4cMHUCTSqZ", "reviewer": "Reviewer_ex8V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1. This paper addresses important non-IID and privacy problems in federated learning. It is a straightforward idea to share synthetic data instead of real data to avoid privacy leak. DP-GAN also has theoretically guarantee of privacy preserving. The framework seems to be promising. \n2. The performance of proposed framework is good. \n3. The effect of GAN-synthetic data is evaluated by a comparison with closely related algorithms. There is an improvement comapred to NaiveMix and FedMix. \n4. The sensitivity to privacy budget is evaluated. \n5. Each additional component seems not so novel, but it is a reasonable and potentially useful combination. \n6. Clear delivering of a complicated system. \n\nWeaknesses: \n\n1. Equation (4) seems to be wrong. If the algorithm use vanilla mixup, the input of classifier $$f(\\cdot; \\boldsymbol{w}_i)$$ should be the mixed data $$\\bar{\\boldsymbol{X}}_i$$, rather than $$(1 - \\lambda_1) \\boldsymbol{X}_i$$ or $$\\lambda_1 \\hat{\\boldsymbol{X}}_i$$.  $$\\lambda_1$$ should be $$1 - \\lambda_1$$ to align with equation (3), and vise versa. Same mistake in Algorithm 2. The author should correct them or explain. \n\n2. Inadequate evaluation of labelling process. It is not verified whether the interplaying labeling process is better than some naive method, e.g., \n\n   - Each client trains a local classifier to assign pseudo label to its synthesis data, once and for all. \n   - Each client uses conditional GAN or similar techniques to generate images directly given label. \n\n   The author should carefully design ablation study to evaluate this. \n\n3. Potential advantages introduced by ServerUpdate. An additional ServerUpdate step is introduced to the framework. Therefore, SDA-FL may take more gradient descent steps than baselines (FedAvg, FedProx, FedMix, etc.) in a communication round. People may be skeptical whether it is the additional gradient steps make the model converge faster. The author should make sure that same number of gradient steps in taken in each round. \n\n4. Lack of details and unclear hyperparameters in experiments. Number of clients, number of class per client for Cifar-10, $$\\alpha, \\lambda_2, E, B$$, hyperparameters for baselines... are not provided. More experiments details should be provided in main text, supplementary materals, or released code. \n\n5. Error bar is not provided. People may ask whether figure results are from only one run. It will be much better to provided result with standard deviation or confidence interval, together with the number of independent runs. \n\nOther concerns and questions: \n\n1. In SDA-FL, each participant train a GAN before training the classifier, introducing the following requirements: \n\n   - Each client must have sufficient data to train GAN. \n   - Each client must have adequate computing resources. (GAN often requires more resources. )\n   - In each communication round, clients are selected from *ones that uploaded their GANs*. This may implicitly requires a sustainable or high participation rate of client. \n\n   These requirements may limit the scope of SDA-FL's application. In cross-device settings, it's hard for personal devices (like mobile phones) to meet the above requirements. For me it seems like SDA-FL can only be applied to cross-silo settings. Please comment. \n\n2. DP-GAN guarantee privacy preserving during data synthesis. The training process of classifier is not protected and might leak personal information. This is an orthogonal problem but the author may mention this in the text. ", "summary_of_the_paper": "Federated learning suffers from non-IIDness across clients. A line of previous works address the non-IID problem by data sharing which violates the privacy requirement. In this paper, the author propose SDA-FL. Compare to the most basic FedAvg, the additional components includes\n\n- *Image synthesis with DP-GAN*. Before training classifier, each client trains a differentially private GAN to generate synthetic data, and upload these data to PS for future data-sharing. \n- *Synthetic image labeling*. Similar to self-training, PS use local models to assign pseudo-labels to unlabeled data. These labels are updated during training, i.e., interplay between model training and synthetic dataset updating. \n- *Mixup*. For each clients, private data and shared synthetic data are mixed to alleviate non-IIDness. \n- *ServerUpdate*. Data sharing also makes it possible for the server to conduct gradient descent. \n\nThis paper also empirically evaluate their framework under supervised and semi-supervised learning settings. Moreover, this paper study the sensitivity to privacy budget and effect of synthetic data. ", "summary_of_the_review": "This paper addresses important non-IID and privacy problems in federated learning, and proposes a frameworks with some components are evaluated. However, components of interplaying labeling and ServerUpdate are not well-evaluated in this paper. Moreover, the details of experiments are inadequate. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YV8WJpOy-7h", "reviewer": "Reviewer_UMsE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The studied problem is important. Non-IID data is a key challenge in federated learning.\n2. From the experiments, the improvement of the proposed approach is significant.\n\nWeaknesses:\n1. Some important details about the algorithm are missing. What is the size of the local synthetic data per client? Is it related to the size of the local data? \n2. The GAN seems to be an important part of the algorithm. The paper directly adopts WGAN without convincing explanation. The authors can add experiments to show the effect of the GAN architectures.\n3. The communication overhead may be significantly large than FedAvg since the synthetic data need to be communicated each round. The authors should theoretically or empirically analyze the communication cost.\n4. In Algorithm 1, why the client needs to update the synthesis dataset? The dataset has already been updated by the server.\n5. The authors claim that the local data including both the real and the synthetic data approach an IID distribution. However, there is no convincing explanation to support it. \n6. The experiments are weak. MNIST and Fashion-MNIST are two simple tasks, which are not challenging in federated learning. For the results on CIFAR-10, the differentially private version does not show superior performance compared with FedAvg.\n7. There is no experimental study on the hyperparameters including $\\lambda$ and the threshold. The authors should add them to understand the proposed algorithm more clearly.\n8. What is the non-IID data setting in the experiments of different privacy budget?\n9. Typo: the second paragraph of “Effect of privacy budget”: Table 5 -> Table 6.\n", "summary_of_the_paper": "The paper proposes a new federated learning algorithm called SDA-FL, which utilizes GANs to generate synthetic data for federated training on non-IID data. Specifically, each client pretrains a GAN to generate synthetic data and send them to the server. In each round, the server sends the global model and the synthetic data to the clients. Then, the clients update the label of the synthetic data and use both the local data and synthetic data to update the local model. The local models are sent to the server, which further averages the models and uses the averaged model to label the synthetic data. Experiments show that SDA-FL significantly outperforms the other federated learning approaches on non-IID data. The paper also studies the influence of different privacy budgets on the performance when applying differentially private GANs.", "summary_of_the_review": "The paper proposes a new federated learning algorithm on non-IID data. Although the proposed algorithm is interesting, many details in the algorithm and experiments are missing. I believe the paper needs to be significantly improved from both the theoretical and empirical perspectives.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Nwzt8uPCJVq", "reviewer": "Reviewer_Ena1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths are:\n1. Utilized the generative power of GAN to simulate the distribution of local data, without directly sharing it with peer and central server. Also used differential privacy, the two methods both maintained the local data privacy;\n2. Utilized pseudo labeling to further maintain the generated dataset.\n\nThe main weaknesses are:\n1. The privacy issue itself. Consider the extreme case, where each local client only keeps a single class of data. In this case, the first round locally pre-trained GAN-generated data uploaded to the central server will mostly contain the data which has the same label, which immediately break the privacy of the local data;\n2. Lack of citations. The differential private GAN is not an original idea. This idea has been broadly explored in the GAN community recently, thus proper citations are needed. Also, more citations of federated learning algorithms focusing on non-iid setting are also missing, for example, [1][2][3];\n3. The algorithms are not described clearly in the algorithm boxes. ServerUpdate is described as ''is the same as the\nLocalUpdate execpt the data'', which needs more clarification;\n4. Lack of details of experiments and code. Most of the details of the experiments, for example, training hyperparameters, network details are missing. Since no code is provided, it's kind of hard for one to repeat the experiments and confirm the effectiveness of the algorithm;\n5. Lack of experiments. More experiments should be conducted, for example for the first task \"Federated supervised learning,\"  different ways of partitioning the data should be considered, rather than the extreme case alone;\n6. Lack of theoretical analysis;\n7. Quality of GAN-generated images. This can be a minor issue, but the quality of the GAN-generated images provided in the paper is kind of low, not sure whether it's because of the noisy gradients;\n8. Another minor issue is why different choices of differential budget are only provided in the CIFAR-10 task? \n\n\n[1] Tackling the objective inconsistency problem in heterogeneous federated optimization. J Wang, Q Liu, H Liang, G Joshi, HV Poor - arXiv preprint arXiv:2007.07481, 2020\n\n[2] H. Wang, Z. Kaplan, D. Niu and B. Li, \"Optimizing Federated Learning on Non-IID Data with Reinforcement Learning,\" IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, 2020, pp. 1698-1707, doi: 10.1109/INFOCOM41043.2020.9155494. \n\n[3] On the convergence of fedavg on non-iid data. X Li, K Huang, W Yang, S Wang, Z Zhang - arXiv preprint arXiv:1907.02189, 2019", "summary_of_the_paper": "This paper focused on a classical federated learning setting where data was non-iid partitioned in different local servers, and came up with a method named SDA-FL, which combined GAN-generated data, differential privacy to both address the non-iid problem and keep local data privacy. The main contributions are:\n1. Utilized differential private GAN-generated data to solve the non-iid and local data privacy problems;\n2. Designed a label updating mechanism to increase the model performance;\n3. Tested the algorithm on CIFAR-10, MNIST, fashion-MNIST to confirm the performance of the algorithm.", "summary_of_the_review": "This paper focused on a challenging problem -- non-iid in federated learning and tried to combine methods from different areas to solve it, which is interesting itself, however, this paper mainly suffers from the following aspects:\n1. Technical correctness. The privacy is not well maintained;\n1. Lack of novelty. The proposed algorithm is more like a rough combination of several existing methods, with little innovative thoughts or theoretical explanations;\n2. Design of experiments. More experiments should be carefully designed and conducted;\n3. Missing proper citations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3AE9WtmbhOT", "reviewer": "Reviewer_t36G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength-\n\n* The proposed idea is clear and well-presented: The idea to use partially-private synthetic data to resolve the non-IID issue in federated learning is interesting and clearly presented. \n*Experimental result shows the improvement in performance: The authors compare their method with other federated learning methods and demonstrate the improvement in performance from the sharing of synthetic data. The trade-offs between privacy level and the quality of synthetic data are also presented. \n\nWeakness-\n\n* The overhead of training local GANs in each client can be huge: The proposed scheme achieves the performance gain from sharing of synthetic data, which models the private local data with some privacy protection. The performance gain highly depends on the quality of synthetic data, but in real scenarios, it could be unrealistic to train GANs in every client using their local data to generate high-quality synthetic data.\n* More empirical support would be necessary: the proposed scheme highly depends on the quality of pseudo-labels as well as the synthetic data. There are many performance measures to quantify the quality of synthetic data and pseudo labels, i.e., FID and Inception Score for synthetic data and label accuracy for the pseudo labels, which can be measured by training a classifier with the synthetic data. Quantifying such quality and examining trade-offs between privacy level, computational cost, quality of synthetic data and the overall performance will provide better insights on this work and make the presentation of empirical results more comprehensive.  \n", "summary_of_the_paper": "This paper proposes a new framework for federated learning to resolve the non-IID issue by sharing differentially private synthetic data. Each client pretrains a local GAN to generate synthetic data and upload the data to the parameter server. To effectively use the synthetic data, the server performs pseudo labeling and shares this information with the clients along with the global model parameters, to let the local data, including both the real and synthetic data, approach an IID distribution. The interplay between model training and synthetic data updating improves the convergence of the local models and resolves the non-IID issue. ", "summary_of_the_review": "The proposed idea of using synthetic data, generated by local GANs of each client, and pseudo labelling at PS seems to be effective to resolve the non-IID issue of federated learning. However, the overhead of training local GANs in each client can be huge, which makes this approach not very practical. The overhead should be clearly addressed to compare this work with other baselines. The current experimental results only measure the overall test accuracy, but since the trade-offs between quality of synthetic data, pseudo-labelling, and overall performance are main factors in the proposed scheme, presenting such trade-offs in more detailed manner would be helpful to support the idea. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an idea to address the non-IID issue that is well-known in federated learning. After the discussion with the authors, there are still some concerns remained about the proposed approach. First and foremost, the training of local GAN at each client can be demanding computationally and statistically, which limits the practicality of their approach. Secondly, there has been other work that aims to study the non-IID issues in federated learning, as suggested by the reviewers. The authors should consider citing some of the work in this literature and compare the prior approaches with their GAN-based approach. Thirdly, there is a lack of a formal statement about the privacy guarantee in this paper. In particular, it seems that the privacy guarantee would only make sense in the cross-silo setting, in which each client has many users' data. If each client corresponds to a single user, it does not make sense to train a local GAN. The authors should consider elaborating on the privacy guarantee in the next revision.
Decision: Reject