Federated Learning with GAN-based Data Synthesis for Non-IID Clients

1 INTRODUCTION . The recent development of deep learning technologies has led to major breakthroughs in various domains . This results in a tremendous amount of valuable data that can facilitate the training of deep learning models for intelligent applications . A traditional approach to exploit these distributed data samples is to upload them to a centralized server for model training . However , directly offloading data raises severe privacy concerns as data collected by mobile clients may contain sensitive information . By decoupling model training from the need of transferring private data to the cloud , federated learning ( FL ) offers a promising approach to collaboratively learn a global model without directly sharing the local data . Particularly , McMahan et al . ( 2017 ) introduced the Federated Averaging ( FedAvg ) algorithm that synchronously aggregates the updates of local models , while satisfying the basic requirements for privacy protection . Later , FedProx was proposed by Li et al . ( 2020 ) that adds a proximal term in the loss function to improve the statistical stability of the training process . A key characteristic of federated learning is the non-IID ( independent and identically distributed ) data distribution among clients . Consequently , different clients learn from different data distributions , which leads to high heterogeneity among local models and degrades the effectiveness of model aggregation ( McMahan et al. , 2017 ; Wang et al. , 2020b ) . For extreme cases with highly skewed data distribution , traditional federated learning methods even lack a theoretical guarantee of convergence ( Yu et al. , 2020 ; Rothchild et al. , 2020 ) . Recent studies have attempted to address the non-IID issue by constructing a global shared dataset based on the local data to mitigate the data imbalance effect ( Krizhevsky et al. , 2009 ; Zhao et al. , 2018 ; Yoshida et al. , 2020 ; Oh et al. , 2020 ) . Particularly , several methods ( Krizhevsky et al. , 2009 ; Yoshida et al. , 2020 ) proposed data sharing strategies to distribute a percentage of raw data among clients , but naive data sharing seriously violate the privacy requirement in FL . Besides , data augmentation approaches ( Zhang et al. , 2021 ; Oh et al. , 2020 ) generate synthetic samples without directly sharing private data . However , these methods are still susceptible to data leakage without a privacy-preserving mechanism . Thus , it is important to investigate how to generate privacy-preserving synthetic samples , as well as how to effectively utilize the private dataset and synthetic dataset in the FL framework . To alleviate the heterogeneity of the data distribution in FL without compromising privacy , we propose a novel framework , named Synthetic Data Aided Federated Learning ( SDA-FL ) , which resolves the non-IID issue by sharing differentially private synthetic data . Under this framework , each client pretrains a local generative adversarial network ( GAN ) ( Goodfellow et al. , 2020 ) to generate synthetic data , thus avoiding sharing the raw data directly . A parameter server ( PS ) performs pseudo-labeling for the uploaded synthetic data and constructs a global shared dataset . In each global aggregation , the PS updates the pseudo labels of the synthetic dataset based on the received models and updates the global model with the confident synthetic data . This iterative labeling process constitutes a synthetic dataset with high-confidence labels , which is fed back to the clients to construct their training samples , including the local private data and synthetic data , following an almost identical distribution . Meanwhile , to provide a privacy guarantee for the generative models , we adopt the widely used differential privacy ( Dwork , 2008 ) to ensure that the actual training samples can not be revealed by adversaries . In particular , we use the Wasserstein GAN with gradient penalty ( WGAN-GP ) ( Gulrajani et al. , 2017 ) as the backbone and introduce the privacy budget by adding artificial noise to the gradients in the training process . The SDA-FL framework is mostly compatible with many existing FL methods and can be applied in both supervised and semi-supervised settings . We evaluate the proposed framework on several standard datasets under both supervised learning and semi-supervised learning , which shows its effectiveness in resolving the non-IID distribution problem even in the extreme case ( with only one class of real data at each client ) . We also investigate the tradeoff between the privacy budget and the FL performance , which demonstrates that this framework is fully functional under the strict differential privacy protection requirement . Our major contributions are summarized as follows : • We develop a global shared synthetic dataset by the differentially private GAN models . This data augmentation method effectively resolves the non-IID problem in FL and maintains the local data privacy . • To utilize the global shared synthetic dataset efficiently , we propose a novel framework , SDA-FL , in which the PS provides and keeps updating high-confidence pseudo labels for the synthetic dataset iteratively , thus improving the performance of the global model . • We evaluate our framework on several benchmark datasets under the high heterogeneous data distribution , where our framework consistently outperforms other baselines under both supervised and semi-supervised learning . Besides , our framework experiences a slight performance degradation with a stringent privacy budget . 2 RELATED WORK . Non-IID Challenges in Federated Learning The pioneering work in FL is FedAvg ( McMahan et al. , 2017 ) , which simply averages model parameters trained by different clients in an elementwise manner , weighted proportionately by the clients ’ data size . However , this algorithm often leads to slow convergence and performance degradation when data is heterogeneous across local clients ( Li et al. , 2019 ) . To alleviate this issue , Wang et al . ( 2020a ) leveraged the reinforcement learning to intelligently choose the client devices to participate in each round of federated learning to counterbalance the bias introduced by non-IID data . Besides , some studies proposed personalized layers to adjust the local model according to the private dataset ( Arivazhagan et al. , 2019 ; Liang et al. , 2020 ) . For example , FedPer was introduced in Arivazhagan et al . ( 2019 ) that captures personalization aspects in federated learning by only using the local data to train the personalized layers . Knowledge distillation ( Deng et al. , 2020 ; Lin et al. , 2020 ) is also a promising idea of personalized federated learning , which transfers information from other clients to a certain client . Although the aforementioned methods outperform FedAvg on strongly non-IID data , the extra computational costs are not affordable by the resource-constrained IoT devices ( Zhu et al. , 2021 ) . Thus , client clustering has been proposed to construct a multi-center framework by grouping the clients into different clusters based on the model similarity ( Ghosh et al. , 2020 ; Kopparapu & Lin , 2020 ) . As clients avoid the negative knowledge transfer from the dissimilar models ( Pan & Yang , 2009 ) , clustering methods can alleviate the negative effect of the non-IID data distribution . However , such methods are not scalable as the communication cost increases largely with the number of clients . Data Augmentation and Privacy Preserving Recently , methods based on some form of data sharing have received increasing attentions and achieved prominent performance in FL ( Zhu et al. , 2021 ; Zhao et al. , 2018 ) . Particularly , data augmentation approaches showed the potential capability to address the non-IID issue by modifying the local distributions without sharing private data ( Duan et al. , 2019 ; Zhang et al. , 2018 ; Eaton-Rosen et al. , 2020 ) . A vanilla data augmentation method was proposed by Duan et al . ( 2019 ) , in which each client performs local data augmentation to reach a balanced global distribution . Besides , several methods ( Oh et al. , 2020 ; Shin et al. , 2020 ; Yoon et al. , 2021 ) leveraged the Mixup technique ( Zhang et al. , 2018 ) to conduct data augmentation , where the clients share the blended local data and collaboratively construct a new global dataset to tackle the non-IID issue . However , frequent data exchange may be susceptible to privacy issues . Recently , the GAN-based data augmentation framework ( Bowles et al. , 2018 ; Yoshida et al. , 2020 ) was shown to be promising for reducing the degree of local data imbalance in FL . The general idea is to train a good generative model in the presence of non-IID data . Under this framework , each client sends its local seed samples to the server for the GAN training , and then a well-trained generator will be sent to all the clients for model updating . However , directly sending local data samples to the server violates the data privacy requirement . FedDPGAN ( Zhang et al. , 2021 ) suggested an alternative that all the clients train a global GAN together based on the FL framework to supplement the scarce local Covid-19 data . Unfortunately , the white-box generators sent to the server are vulnerable to adversarial attacks ( Chen et al. , 2020 ) . 3 PRELIMINARY . Federated Learning FedAvg ( McMahan et al. , 2017 ) is one of the representative training algorithms in federated learning , and it trains a shared neural network without disclosing the raw data of clients . For every step t = 0 , . . . , T − 1 , every client k ∈ St downloads the global model wt and updates the local model with local dataset Dk via stochastic gradient descent ( SGD ) , i.e. , wt+1 k ← wtk − ηt+1∇ ` . The updated local models are then sent back to the PS for a weighted aggregation wt+1 ← 1K ∑ k∈St w k t+1 . These procedures repeat until convergence or all the T training steps are exhausted . Differentially Private Generative Adversarial network We utilize a differentially private generative adversarial network ( DPGAN ) for data augmentation while protecting local data privacy . To avoid the gradients vanishing and mode collapse problems ( Arjovsky & Bottou , 2017 ; Arjovsky et al. , 2017 ) that are commonly encountered by many GAN models ( Mirza & Osindero , 2014 ; Radford et al. , 2015 ; Odena et al. , 2017 ) , we adopt the WGAN-GP ( Gulrajani et al. , 2017 ) , which is upgraded based on the Wasserstein GAN ( WGAN ) ( Arjovsky & Bottou , 2017 ; Arjovsky et al. , 2017 ) , as the backbone of the generative model for a stable training procedure . Besides , we add noise to the gradient in the generator training process , which provides privacy control in terms of differential privacy ( Torkzadehmahani et al. , 2019 ) . The definition of differential privacy is as follows : Definition 1 . ( Differential privacy1 ) : A random mechanism Ap satisfies ( , δ ) -differential privacy if for any output ’ s subset ( S ) and for any two adjacent datasets D , D′ , the following probability inequality holds : P ( Ap ( D ) ∈ S ) ≤ e · P ( Ap ( D′ ) ∈ S ) + δ . ( 1 ) Particularly , δ is an optimal probability that guarantees the validity of the inequality , and is the privacy budget indicating the privacy level , i.e. , a smaller value of implies stronger privacy protection . To satisfy the ( , δ ) -differential privacy , we follow the work ( Xie et al. , 2018 ) and add the Gaussian noise to the gradients updated in each discriminator training iteration . The relationship between the noise variance and differential privacy is shown as follows : σn = 2q √ nd log ( 1 δ ) / , ( 2 ) where q and nd denote the sample probability and the number of discriminator training iterations in each training round , respectively . 1Although our work leverages the differential privacy to address the extra privacy problem introduced by the GAN-based data augmentation , the potential privacy risk of FL is still an open problem , which is out of the scope of our study . Pseudo Labeling Although GAN-based data augmentation can generate synthetic data , the labels of the synthetic data are unknown , which hinders the training of the model through backpropagation . In semi-supervised learning , pseudo labeling is a widely used method in classification tasks to generate confident pseudo labels ( Sohn et al. , 2020 ) . A hard pseudo label will be applied to the unlabeled data if the maximum class probability ( MCP ) of the inference result ( Corbière et al. , 2019 ) exceeds a predefined threshold τ . In our experiments , the threshold is set to be 0.95 .

Summary: Federated learning suffers from non-IIDness across clients. A line of previous works address the non-IID problem by data sharing which violates the privacy requirement. In this paper, the author propose SDA-FL. Compare to the most basic FedAvg, the additional components includes  - *Image synthesis with DP-GAN*. Before training classifier, each client trains a differentially private GAN to generate synthetic data, and upload these data to PS for future data-sharing.  - *Synthetic image labeling*. Similar to self-training, PS use local models to assign pseudo-labels to unlabeled data. These labels are updated during training, i.e., interplay between model training and synthetic dataset updating.  - *Mixup*. For each clients, private data and shared synthetic data are mixed to alleviate non-IIDness.  - *ServerUpdate*. Data sharing also makes it possible for the server to conduct gradient descent.   This paper also empirically evaluate their framework under supervised and semi-supervised learning settings. Moreover, this paper study the sensitivity to privacy budget and effect of synthetic data.