Federated Learning with GAN-based Data Synthesis for Non-IID Clients

1 INTRODUCTION . The recent development of deep learning technologies has led to major breakthroughs in various domains . This results in a tremendous amount of valuable data that can facilitate the training of deep learning models for intelligent applications . A traditional approach to exploit these distributed data samples is to upload them to a centralized server for model training . However , directly offloading data raises severe privacy concerns as data collected by mobile clients may contain sensitive information . By decoupling model training from the need of transferring private data to the cloud , federated learning ( FL ) offers a promising approach to collaboratively learn a global model without directly sharing the local data . Particularly , McMahan et al . ( 2017 ) introduced the Federated Averaging ( FedAvg ) algorithm that synchronously aggregates the updates of local models , while satisfying the basic requirements for privacy protection . Later , FedProx was proposed by Li et al . ( 2020 ) that adds a proximal term in the loss function to improve the statistical stability of the training process . A key characteristic of federated learning is the non-IID ( independent and identically distributed ) data distribution among clients . Consequently , different clients learn from different data distributions , which leads to high heterogeneity among local models and degrades the effectiveness of model aggregation ( McMahan et al. , 2017 ; Wang et al. , 2020b ) . For extreme cases with highly skewed data distribution , traditional federated learning methods even lack a theoretical guarantee of convergence ( Yu et al. , 2020 ; Rothchild et al. , 2020 ) . Recent studies have attempted to address the non-IID issue by constructing a global shared dataset based on the local data to mitigate the data imbalance effect ( Krizhevsky et al. , 2009 ; Zhao et al. , 2018 ; Yoshida et al. , 2020 ; Oh et al. , 2020 ) . Particularly , several methods ( Krizhevsky et al. , 2009 ; Yoshida et al. , 2020 ) proposed data sharing strategies to distribute a percentage of raw data among clients , but naive data sharing seriously violate the privacy requirement in FL . Besides , data augmentation approaches ( Zhang et al. , 2021 ; Oh et al. , 2020 ) generate synthetic samples without directly sharing private data . However , these methods are still susceptible to data leakage without a privacy-preserving mechanism . Thus , it is important to investigate how to generate privacy-preserving synthetic samples , as well as how to effectively utilize the private dataset and synthetic dataset in the FL framework . To alleviate the heterogeneity of the data distribution in FL without compromising privacy , we propose a novel framework , named Synthetic Data Aided Federated Learning ( SDA-FL ) , which resolves the non-IID issue by sharing differentially private synthetic data . Under this framework , each client pretrains a local generative adversarial network ( GAN ) ( Goodfellow et al. , 2020 ) to generate synthetic data , thus avoiding sharing the raw data directly . A parameter server ( PS ) performs pseudo-labeling for the uploaded synthetic data and constructs a global shared dataset . In each global aggregation , the PS updates the pseudo labels of the synthetic dataset based on the received models and updates the global model with the confident synthetic data . This iterative labeling process constitutes a synthetic dataset with high-confidence labels , which is fed back to the clients to construct their training samples , including the local private data and synthetic data , following an almost identical distribution . Meanwhile , to provide a privacy guarantee for the generative models , we adopt the widely used differential privacy ( Dwork , 2008 ) to ensure that the actual training samples can not be revealed by adversaries . In particular , we use the Wasserstein GAN with gradient penalty ( WGAN-GP ) ( Gulrajani et al. , 2017 ) as the backbone and introduce the privacy budget by adding artificial noise to the gradients in the training process . The SDA-FL framework is mostly compatible with many existing FL methods and can be applied in both supervised and semi-supervised settings . We evaluate the proposed framework on several standard datasets under both supervised learning and semi-supervised learning , which shows its effectiveness in resolving the non-IID distribution problem even in the extreme case ( with only one class of real data at each client ) . We also investigate the tradeoff between the privacy budget and the FL performance , which demonstrates that this framework is fully functional under the strict differential privacy protection requirement . Our major contributions are summarized as follows : • We develop a global shared synthetic dataset by the differentially private GAN models . This data augmentation method effectively resolves the non-IID problem in FL and maintains the local data privacy . • To utilize the global shared synthetic dataset efficiently , we propose a novel framework , SDA-FL , in which the PS provides and keeps updating high-confidence pseudo labels for the synthetic dataset iteratively , thus improving the performance of the global model . • We evaluate our framework on several benchmark datasets under the high heterogeneous data distribution , where our framework consistently outperforms other baselines under both supervised and semi-supervised learning . Besides , our framework experiences a slight performance degradation with a stringent privacy budget . 2 RELATED WORK . Non-IID Challenges in Federated Learning The pioneering work in FL is FedAvg ( McMahan et al. , 2017 ) , which simply averages model parameters trained by different clients in an elementwise manner , weighted proportionately by the clients ’ data size . However , this algorithm often leads to slow convergence and performance degradation when data is heterogeneous across local clients ( Li et al. , 2019 ) . To alleviate this issue , Wang et al . ( 2020a ) leveraged the reinforcement learning to intelligently choose the client devices to participate in each round of federated learning to counterbalance the bias introduced by non-IID data . Besides , some studies proposed personalized layers to adjust the local model according to the private dataset ( Arivazhagan et al. , 2019 ; Liang et al. , 2020 ) . For example , FedPer was introduced in Arivazhagan et al . ( 2019 ) that captures personalization aspects in federated learning by only using the local data to train the personalized layers . Knowledge distillation ( Deng et al. , 2020 ; Lin et al. , 2020 ) is also a promising idea of personalized federated learning , which transfers information from other clients to a certain client . Although the aforementioned methods outperform FedAvg on strongly non-IID data , the extra computational costs are not affordable by the resource-constrained IoT devices ( Zhu et al. , 2021 ) . Thus , client clustering has been proposed to construct a multi-center framework by grouping the clients into different clusters based on the model similarity ( Ghosh et al. , 2020 ; Kopparapu & Lin , 2020 ) . As clients avoid the negative knowledge transfer from the dissimilar models ( Pan & Yang , 2009 ) , clustering methods can alleviate the negative effect of the non-IID data distribution . However , such methods are not scalable as the communication cost increases largely with the number of clients . Data Augmentation and Privacy Preserving Recently , methods based on some form of data sharing have received increasing attentions and achieved prominent performance in FL ( Zhu et al. , 2021 ; Zhao et al. , 2018 ) . Particularly , data augmentation approaches showed the potential capability to address the non-IID issue by modifying the local distributions without sharing private data ( Duan et al. , 2019 ; Zhang et al. , 2018 ; Eaton-Rosen et al. , 2020 ) . A vanilla data augmentation method was proposed by Duan et al . ( 2019 ) , in which each client performs local data augmentation to reach a balanced global distribution . Besides , several methods ( Oh et al. , 2020 ; Shin et al. , 2020 ; Yoon et al. , 2021 ) leveraged the Mixup technique ( Zhang et al. , 2018 ) to conduct data augmentation , where the clients share the blended local data and collaboratively construct a new global dataset to tackle the non-IID issue . However , frequent data exchange may be susceptible to privacy issues . Recently , the GAN-based data augmentation framework ( Bowles et al. , 2018 ; Yoshida et al. , 2020 ) was shown to be promising for reducing the degree of local data imbalance in FL . The general idea is to train a good generative model in the presence of non-IID data . Under this framework , each client sends its local seed samples to the server for the GAN training , and then a well-trained generator will be sent to all the clients for model updating . However , directly sending local data samples to the server violates the data privacy requirement . FedDPGAN ( Zhang et al. , 2021 ) suggested an alternative that all the clients train a global GAN together based on the FL framework to supplement the scarce local Covid-19 data . Unfortunately , the white-box generators sent to the server are vulnerable to adversarial attacks ( Chen et al. , 2020 ) . 3 PRELIMINARY . Federated Learning FedAvg ( McMahan et al. , 2017 ) is one of the representative training algorithms in federated learning , and it trains a shared neural network without disclosing the raw data of clients . For every step t = 0 , . . . , T − 1 , every client k ∈ St downloads the global model wt and updates the local model with local dataset Dk via stochastic gradient descent ( SGD ) , i.e. , wt+1 k ← wtk − ηt+1∇ ` . The updated local models are then sent back to the PS for a weighted aggregation wt+1 ← 1K ∑ k∈St w k t+1 . These procedures repeat until convergence or all the T training steps are exhausted . Differentially Private Generative Adversarial network We utilize a differentially private generative adversarial network ( DPGAN ) for data augmentation while protecting local data privacy . To avoid the gradients vanishing and mode collapse problems ( Arjovsky & Bottou , 2017 ; Arjovsky et al. , 2017 ) that are commonly encountered by many GAN models ( Mirza & Osindero , 2014 ; Radford et al. , 2015 ; Odena et al. , 2017 ) , we adopt the WGAN-GP ( Gulrajani et al. , 2017 ) , which is upgraded based on the Wasserstein GAN ( WGAN ) ( Arjovsky & Bottou , 2017 ; Arjovsky et al. , 2017 ) , as the backbone of the generative model for a stable training procedure . Besides , we add noise to the gradient in the generator training process , which provides privacy control in terms of differential privacy ( Torkzadehmahani et al. , 2019 ) . The definition of differential privacy is as follows : Definition 1 . ( Differential privacy1 ) : A random mechanism Ap satisfies ( , δ ) -differential privacy if for any output ’ s subset ( S ) and for any two adjacent datasets D , D′ , the following probability inequality holds : P ( Ap ( D ) ∈ S ) ≤ e · P ( Ap ( D′ ) ∈ S ) + δ . ( 1 ) Particularly , δ is an optimal probability that guarantees the validity of the inequality , and is the privacy budget indicating the privacy level , i.e. , a smaller value of implies stronger privacy protection . To satisfy the ( , δ ) -differential privacy , we follow the work ( Xie et al. , 2018 ) and add the Gaussian noise to the gradients updated in each discriminator training iteration . The relationship between the noise variance and differential privacy is shown as follows : σn = 2q √ nd log ( 1 δ ) / , ( 2 ) where q and nd denote the sample probability and the number of discriminator training iterations in each training round , respectively . 1Although our work leverages the differential privacy to address the extra privacy problem introduced by the GAN-based data augmentation , the potential privacy risk of FL is still an open problem , which is out of the scope of our study . Pseudo Labeling Although GAN-based data augmentation can generate synthetic data , the labels of the synthetic data are unknown , which hinders the training of the model through backpropagation . In semi-supervised learning , pseudo labeling is a widely used method in classification tasks to generate confident pseudo labels ( Sohn et al. , 2020 ) . A hard pseudo label will be applied to the unlabeled data if the maximum class probability ( MCP ) of the inference result ( Corbière et al. , 2019 ) exceeds a predefined threshold τ . In our experiments , the threshold is set to be 0.95 .

Summary: This paper proposes a new framework for federated learning to resolve the non-IID issue by sharing differentially private synthetic data. Each client pretrains a local GAN to generate synthetic data and upload the data to the parameter server. To effectively use the synthetic data, the server performs pseudo labeling and shares this information with the clients along with the global model parameters, to let the local data, including both the real and synthetic data, approach an IID distribution. The interplay between model training and synthetic data updating improves the convergence of the local models and resolves the non-IID issue.