Training Federated GANs  with Theoretical Guarantees: A Universal Aggregation Approach

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have attracted much attention due to their ability to generate realistic-looking synthetic data ( Goodfellow et al. , 2014 ; Zhang et al. , 2018 ; Liu et al. , 2019b ; Shaham et al. , 2019 ; Dai et al. , 2017 ; Kumar et al. , 2017 ) . In order to obtain a powerful GAN model , one needs to use data with a wide range of characteristics ( Qi , 2019 ) . However , these diverse data are often owned by different sources , and to acquire their data is often infeasible . For instance , most hospitals and research institutions are unable to share data with the research community , due to privacy concerns ( Annas et al. , 2003 ; Mercuri , 2004 ; lex , 2014 ; Gostin et al. , 2009 ) and government regulations ( Kerikmäe , 2017 ; Seddon & Currie , 2013 ) . To circumvent the barrier of data sharing for GAN training , one may resort to Federated Learning ( FL ) , a promising new decentralized learning paradigm ( McMahan et al. , 2017 ) . In FL , one trains a centralized model but only exchanges model information with different data sources . Since the central model has no direct access to data at each source , privacy concerns are alleviated ( Yang et al. , 2019 ; Kairouz et al. , 2019 ) . This opens the opportunity for a federated GAN , i.e. , a centralized generator with multiple local and privately hosted discriminators ( Hardy et al. , 2019 ) . Each local discriminator is only trained on its local data and provides feedback to the generator w.r.t . synthesized data ( e.g. , gradient ) . A federated GAN empowers GAN with much more diversified data without violating privacy constraints . Despite the promises , a convincing approach for training a federated GAN remains unknown . The major challenge comes from the non-identical local distributions from multiple data sources/entities . The centralized generator is supposed to learn a mixture of these local distributions from different entities , whereas each discriminator is only trained on local data and learns one of the local distributions . The algorithm and theoretical guarantee of traditional single-discriminator GAN ( Goodfellow et al. , 2014 ) do not easily generalize to this federated setting . A federated GAN should integrate feedback from local discriminators in an intelligent way , so that the generator can ‘ correctly ’ learn the mixture distribution . Directly averaging feedbacks from local discriminators ( Hardy et al. , 2019 ) results in a strong bias toward common patternsowever , such non-identical distribution setting is classical in federated learning ( Zhao et al. , 2018 ; Smith et al. , 2017 ; Qu et al. , 2020 ) and characteristic of local data improves the diversity of data . In this paper , we propose the first theoretically guaranteed federated GAN , that can correctly learn the mixture of local distributions . Our method , called Universal Aggregation GAN ( UA-GAN ) , focuses on the odds value rather than the predictions of local discriminators . We simulate an unbiased centralized discriminator whose odds value approximates that of the mixture of local discriminators . We prove that by aggregating gradients from local discriminators based on the odds value of the central discriminator , we are guaranteed to learn the desired mixture of local distributions . A second theoretical contribution of this paper is an analysis of the quality of the federated GAN when the local discriminators can not perfectly learn with local datasets . This is a real concern in a federated learning setting ; the quantity and quality of local data can be highly variant considering the limitation of real-world institutions/sites . Classical theoretical analysis of GAN ( Goodfellow et al. , 2014 ) assumes an optimal discriminator . To understand the consequence of suboptimal discriminators , we develop a novel analysis framework of the Jensen-Shannon Divergence loss ( Goodfellow et al. , 2014 ; Lin , 1991 ) through the odds value of the local discriminators . We show that when the local discriminators behave suboptimally , the approximation error of the learned generator deteriorates linearly to the error . It is worth noting that our theoretical result on suboptimality also applies to the classical GAN . To the best of our knowledge , this is the first suboptimality bound on the federated or classical GAN . In summary , the contributions are threefold . • We propose UA-GAN , a novel federated GAN approach that aggregates feedback from local discriminators through their odds value rather than posterior probability . • We prove that UA-GAN correctly learns the mixture of local distributions when they are perfectly modeled by local discriminators . • We prove when the discriminators are suboptimal in modeling their local distributions , the generator ’ s approximation error is also linear . We also show that our bound is tight . We show with various experiments that our method ( UA-GAN ) outperforms the state-of-theart federated GAN approaches both qualitatively and quantitatively . Training on large scale heterogeneous datasets makes it possible to unleash the power of GANs . Federated GANs show their promise in utilizing unlimited amount of sensitive data without privacy and regulatory concerns . Our method , as the first theoretically guaranteed GAN , will be one step further in building such a foundation . Fig . 1 shows the workflow of UA-GAN . 2 RELATED WORK . The Generative Adversarial Networks ( GANs ) have enjoyed much success in various machine learning and computer vision tasks ( Zhang et al. , 2018 ; Liu et al. , 2019b ; Shaham et al. , 2019 ; Dai et al. , 2017 ; Kumar et al. , 2017 ) . Numerous methods are proposed for GAN training , such as Spectral Normalization ( SN ) ( Miyato et al. , 2018 ) , zero-centered gradient penalty ( Mescheder et al. , 2018 ; Thanh-Tung et al. , 2019 ) , WGAN ( Arjovsky et al. , 2017 ) , WGAN-GP ( Gulrajani et al. , 2017 ) , WGAN-TS ( Liu et al. , 2018 ) , WGAN-QC ( Liu et al. , 2019a ) etc . A common approach in practice is the conditional GAN ( cGAN ) ( Mirza & Osindero , 2014 ) , which uses supervision from data ( e.g. , class labels ) to improve GAN ’ s performance . Multi-discriminator/-generator GANs have been proposed for various learning tasks . To train these GANs , one common strategy is to directly exchange generator/discriminator model parameters during training ( Xin et al. , 2020 ; Hardy et al. , 2019 ) . This is very expensive in communication ; a simple ResNet18 ( He et al. , 2016a ) has 11 million parameters ( 40MB ) . Closest to us is MD-GAN ( Hardy Algorithm 1 Training Algorithm of UA-GAN . 1 : Input : Batch size m , datasets { Dj } , size of datasets { πj = njn } . 2 : Output : G , Dj , ∀j ∈ [ K ] . 3 : for t = 1 , · · · , T do 4 : { Work at the central server . } 5 : G generates synthetic data : x̂i = G ( zi ) , i = 1 , · · · , m. 6 : Send batch of synthetic data Dsyn = { x̂1 , · · · , x̂m } to all K sites . 7 : for j = 1 , · · · , K do 8 : { Work at each local site . } 9 : Update the local discriminator , Dj , using real samples from Dj and synthetic data batch , Dsyn , based on Eq . 2 . 10 : Output predictions and gradients for synthetic data Dj ( x̂i ) , ∂Dj ( x̂i ) /∂x̂i , i = 1 , · · · , m. Send them to the central server . 11 : end for 12 : { Work at the central server . } 13 : Simulate value of Dua ( x̂i ) via Eq . 4 , ∀i . 14 : Update G based on Eq . 5 , using gradients from Dj ’ s . 15 : end for et al. , 2019 ) , which aggregates feedbacks ( gradients ) from local discriminators through averaging . It also swaps parameters between discriminators . None of these methods provide theoretical guarantee as ours . Meanwhile , our method is the only one without model swapping , and thus is much more efficient in bandwidth consumption . Federated Learning ( FL ) ( Kairouz et al. , 2019 ; McMahan et al. , 2016 ) offers the opportunity to integrate sensitive datasets from multiple sources through distributed training . Many works have been done tackling practical concerns in FL , such as convergence under Non-IID data assumption ( Yu et al. , 2019 ; Lian et al. , 2017 ; Li et al. , 2020 ) , decentralized SGD without freezing parameters ( Recht et al. , 2011 ; Nguyen et al. , 2018 ) , communication efficiency ( Konečnỳ et al. , 2016 ; Li et al. , 2019 ) , provable privacy guarantees ( Alistarh et al. , 2017 ; Wei et al. , 2020 ) . Federated GAN is also of great interest from a federated learning perspective . A successful federated GAN makes it possible to train a centralized model ( e.g. , a classifier ) using data synthesized by the centralized generator . This becomes a solution when existing trained FL model needs to be replaced and updated by advanced machine learning approaches , as one can retrain the model at any time using the generator . It also alleviates some privacy concerns of FL , e.g. , the gradient leakage problem ( Zhu et al. , 2019 ) . 3 METHOD . To introduce our algorithm , we first introduce notations and formalize the mixture distribution learning problem . Next , we present our Universal Aggregation approach and prove that it is guaranteed to learn the target mixture distribution . We also analyze the suboptimality of the model when local discriminators are suboptimal . For ease of exposition , we mostly use ordinary GAN to illustrate the algorithm and prove its theoretical properties . At the end of this section , we extend the algorithm , as well as its theoretical guarantees , to conditional GAN ( cGAN ) Mirza & Osindero ( 2014 ) . The empirical results in this work are established on the cGAN since its training is much more controllable , thanks to the additional supervision by the auxiliary variable ( e.g. , classes of images ) . Notations and problem formulation . We assume a cross-silo FL setting Kairouz et al . ( 2019 ) , i.e. , K entities hosting K private datasets D1 , ... , DK , with size n1 , · · · , nK . The total data size n = ∑K j=1 nj . The overall goal is to learn a target mixture distribution p ( x ) = ∑K j=1 πjpj ( x ) , ( 1 ) in which a component distributions pj ( x ) is approximated by the empirical distribution from the j-th local dataset Dj . The mixing weight πj is computed using the fraction of dataset Dj : πj = nj/n . In general , different mixture components pi ( x ) and pj ( x ) may ( but not necessarily ) be non-identical , namely , ∃x , such that pi ( x ) 6= pj ( x ) . Universal Aggregation GAN : Now we are ready to introduce our multi-discriminator aggregation framework . A pseudo-code of UA framework can be found in Algorithm 1 . We have a centralized ( conditional ) generator G ( z ) seeking to learn the global distribution p ( x ) . In each local site , a discriminator Dj ( x ) has access to local dataset Dj . Note data in Dj are only sampled from pj ( x ) . During training , the generator generates a batch of synthetic data to allK sites . The j-th discriminator seeks to minimize the cross entropy loss of GAN from a local perspective Goodfellow et al . ( 2014 ) : max Dj V ( G , Dj ) = Ex∼pj ( x ) [ logDj ( x ) ] + Ez∼N ( 0 , Id ) [ log ( 1−Dj ( G ( z ) ) ) ] ( 2 ) To formalize the generator training , we first introduce odds value . It is an essential quantity for our algorithm and its analysis . Definition 1 ( odds value ) . Given a probability φ ∈ ( 0 , 1 ) , its odds value is Φ ( φ ) , φ1−φ . Note the definition requires φ 6= 1 . Also it is straightforward to see φ = Φ ( φ ) 1+Φ ( φ ) . The central idea of UA Framework is to simulate a centralized discriminator Dua ( x ) which behaves like the mixture of all local discriminators ( in terms of odds value ) . A well behaved Dua ( x ) can then train the centralized generator G using its gradient , just like in a classical GAN . We design Dua so that its odds value Φ ( Dua ( x ) ) is identical to the mixture of the odds values of local discriminators : Φ ( Dua ( x ) ) = ∑K j=1 πjΦ ( Dj ( x ) ) . ( 3 ) Given Φ ( Dua ( x ) ) , we can compute Dua ( x ) as Dua ( x ) = Φ ( Dua ( x ) ) 1 + Φ ( Dua ( x ) ) ( 4 ) Once the central discriminator Dua is simulated , the generator can be computed by minimizing the generator loss : min G V ( G , Dua ) = Ex∼p ( x ) [ logDua ( x ) ] + Ez∼N ( 0 , Id ) [ log ( 1−Dua ( G ( z ) ) ] ( 5 ) Note that mathematically , Eq . ( 5 ) can be directly written in terms of local discriminators Dj ’ s ( by substituting in Eqs ( 3 ) and ( 4 ) ) . In implementation , the simulated central discriminator can be written as a pytorch or tensorflow layer . Intuition . The reason we define Dua ’ s behavior using a mixture of odds values instead of a mixture of the predictions is mathematical . It has been shown in Goodfellow et al . ( 2014 ) that a perfect discriminator learning a data distribution p ( x ) and a fixed generator distribution q ( x ) satisfies D ( x ) = p ( x ) p ( x ) +q ( x ) . It can be shown that only with the odds value equivalency , this optimal solution of the central discriminator D ( x ) can be recovered if each individual discriminator is optimal , i.e. , Dj ( x ) = pj ( x ) pj ( x ) +q ( x ) . This is not true if we define the central discriminator behavior using the average prediction , i.e. , Dua = ∑ j πjDj . More details can be found in Theorem ( 4 ) and its proof . Remark 1 ( Privacy Safety ) . For federated learning , it is essential to ensure information of real data are not leaked outside the local site . This privacy safety is guaranteed in our method . To optimize G w.r.t . Eq . ( 5 ) , we only need to optimize the second term and use gradient on synthetic images G ( z ) from local discriminators . One important concern is about the optimal discriminator condition . Dua ( x ) is designed to be optimal only when Dj ’ s are optimal . We need to investigate the consequence if the local discriminators Dj ’ s are suboptimal . We will provide an error bound of the learned distribution w.r.t. , the suboptimality of Dj ’ s in Corollary ( 2 ) .

Summary: The paper proposes a new method, UA-GAN to train GANs in a federated learning setup. The method simulates a central discriminator D_ua such that the odds values of the central discriminator is equivalent to the weighted sum of the local discriminators. The central generator is then trained based on the simulated central discriminator. The paper provides theoretical analysis on its proposed method and conducts experiments on toy datasets and mixtures of real world datasets to simulate a federated learning setup.