Training Federated GANs  with Theoretical Guarantees: A Universal Aggregation Approach
Authors: Yikai Zhang
Hui Qu
Huidong Liu
Qi Chang
Dimitris N. Metaxas
Chao Chen
Venue:  (2021)
Keywords: Federated Learning
GAN
Deep Learning

Abstract: Recently, Generative Adversarial Networks (GANs) have demonstrated their potential in federated learning, i.e., learning a centralized model from data privately hosted by multiple sites. A federated GAN jointly trains a centralized generator and multiple private discriminators hosted at different sites. A major theoretical challenge for the federated GAN is the heterogeneity of the local data distributions. Traditional approaches cannot guarantee to learn the target distribution, which is a mixture of the highly different local distributions. This paper tackles this theoretical challenge, and for the first time, provides a provably correct framework for federated GAN. We propose a new approach called Universal Aggregation, which simulates a centralized discriminator via carefully aggregating the mixture of all private discriminators. We prove that a generator trained with this simulated centralized discriminator can learn the desired target distribution. Through synthetic and real datasets, we show that our method can learn the mixture of largely different distributions, when existing federated GAN methods fail to.

Under review as a conference paper at ICLR 2021
TRAINING FEDERATED GANS WITH THEORETICAL GUARANTEES: A UNIVERSAL AGGREGATION APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, Generative Adversarial Networks (GANs) have demonstrated their po- tential in federated learning, i.e., learning a centralized model from data privately hosted by multiple sites. A federated GAN jointly trains centralized generator and multiple private discriminators hosted at different sites. A major theoretical challenge for the federated GAN is the heterogeneity of the local data distributions. Traditional approaches cannot guarantee to learn the target distribution, which is a mixture of the highly different local distributions. This paper tackles this theoretical challenge, and for the first time, provides a provably correct framework for federated GAN. We propose a new approach called Universal Aggregation, which simulates centralized discriminator via carefully aggregating the mixture of all private discriminators. We prove that a generator trained with this simulated centralized discriminator can learn the desired target distribution. Through syn- thetic and real datasets, we show that our method can learn the mixture of largely different distributions where existing federated GAN methods fail.
INTRODUCTION
Generative Adversarial Networks (GANs) have attracted much attention due to their ability to generate realistic-looking synthetic data (Goodfellow et al. 2014 Zhang et al. 2018 Liu al. 2019b Shaham ett al. [219 Dai et al. 2017): Kumar et al. 2017)) In order to obtain powerful GAN model, one needs to use data with a wide range of characteristics (Qil 2019). However, these diverse data are often owned by different sources, and to acquire their data is often infeasible. For instance, most hospitals and research institutions are unable to share data with the research community, due to
privacy concerns (Annas et al., 20033 Mercuri, 2004; lex 2014 Gostin et al. 2009) and government
regulations (Kerikmäe) 2017 Seddon & Currie 2013).
To circumvent the barrier of data sharing for GAN training, one may resort to Federated Learning (FL), a promising new decentralized learning paradigm (McMahan et al. 2017). In FL, one trains centralized model but only exchanges model information with different data sources. Since the central model has no direct access to data at each source, privacy concerns are alleviated (Yang et al. 2019 Kairouz al., 2019). This opens the opportunity for a federated GAN, i.e., a centralized generator with multiple local and privately hosted discriminators (Hardy et al.) 2019). Each local discriminator is only trained on its local data and provides feedback the generator w.r.t. synthesized data (e.g., gradient). A federated GAN empowers GAN with much more diversified data without violating privacy constraints.
Despite the promises, a convincing approach for training a federated GAN remains unknown. The major challenge comes from the non-identical local distributions from multiple data sources/entities. The centralized generator is supposed to learn a mixture of these local distributions from different en- tities, whereas each discriminator is only trained on local data and learns one of the local distributions. The algorithm and theoretical guarantee of traditional 2014) do not easily generalize to this federated setting. A federated GAN should integrate feedback from local discriminators in an intelligent way, so that the generator can 'correctly" learn the mixture distribution. Directly averaging feedbacks from local discriminators (Hardy et al.) 2019) results in strong bias toward common patternsowever, such non-identical distribution setting is classical in federated learning (Zhao et al., 2018; Smith et al. 2017 Qu et al., 2020) and characteristic of local data improves the diversity of data.
GAN (Goodfellow et al.)

---

Under review as a conference paper at ICLR 2021
In this paper, we propose the first theoretically guaranteed federated GAN, that can correctly learn the mixture of local distributions. Our method, called Universal Aggregation GAN (UA-GAN), focuses on the odds value rather than the predictions of local discriminators. We simulate an unbiased centralized discriminator whose odds value approximates that of the mixture of local discriminators. We prove that by aggregating gradients from local discriminators based on the odds value of the central discriminator, we are guaranteed to learn the desired mixture of local distributions.
A second theoretical contribution of this paper is an analysis of the quality of the federated GAN when the local discriminators cannot perfectly learn with local datasets. This is a real concern in a federated learning setting: the quantity and quality of local data can be highly variant considering the limitation of real-world institutionsssiies. Classical theoretical analysis of GAN (Goodfellow et al.) 2014) assumes an optimal discriminator. To understand the consequence of suboptimal discriminators. we develop a novel analysis framework of the Jensen- 2014 Lin 199T) through the odds value of the local discriminators, We show that when the local discriminators behave suboptimally, the approximation error of the learned generator deteriorates linearly to the error.
Divergence loss (Goodfellow et al.
It is worth noting that our theoretical result on suboptimality also applies to the classical GAN. To the best of our knowledge, this is the first suboptimality bound on the federated or classical GAN. In summary, the contributions are threefold.
e
O
We propose UA-GAN, a novel federated GAN approach that aggregates feedback from local discriminators through their odds
Tomm Ld
U.C.
Seel Iniqus
Gemerate
value rather than posterior probability.
We prove that UA-GAN correctly learns the mixture of local distributions when they are perfectly modeled by local discriminators. We prove when the discriminators are sub- optimal in modeling their local distributions, the generator's 's approximation error is also linear. We also show that our bound is tight.
Universal (Eq.4)
Dg
We show with various experiments that our method (UA-GAN) outperforms the sfate-of-the- art federated GAN approaches both qualitatively and quantitatively.
Figure 1: UA-GAN framework: The data from multiple entities may share some common distri- bution while retain its own individual distribution. Here, we use the MNIST dataset that the 10 digits belong to different data center as common pattern, and MNIST Fashion dataset that randomly split into different data centers as distinct features. (see Sec. for details). We can see the UA-GAN carefully aggregates the feedback from different entities and manage to digest the universal distri- bution generate unbiased synthetic data.
Training on large scale heterogeneous datasets makes it possible to unleash the power of GANs. Federated GANs show their promise in utilizing unlimited amount of sensitive data without pri- vacy and regulatory concerns. Our method, as the first theoretically guaranteed GAN, will be one step further in building such a foundation. Fig. shows the workflow of UA-GAN
2 RELATED WORK
The Generative Adversarial Networks (GANs) have enjoyed much success in various machine learning and computer vision tasks (Zhang et al. 2018 Liu et al. 2019b] Shaham et al., 2019 Dai
al., 2017 Kumar et al. (2017). Numerous methods are proposed for GAN training, such as Spectral
Normalization (SN) (Miyato et al. 2018), zero-centered gradient penalty (Mescheder et al. 2018
Thanh-Tung et al., 2019), WGAN (Arjovsky et al. 2017)- WGAN-GP (Gulrajani et al.) 2017).
WGAN-TS (Liu et al.] 2018),, WGAN-QC (Liu et al. 2019a) etc. A common approach in practice is the conditional GAN (cGAN) (Mirza & Osindero 2014), which uses supervision from data (c.g..
class labels) to improve GAN's performance.
hultt-disrrimiintoo-ggennttooo GANs have been proposed for various learning tasks. To train these GANs, one common strategy is to directly exchange generaatorddisrriinnato model parameters during training (Xin et al. 2020; Hardy et al.) (011). This is very expensive in communication; a simple ResNet18 8 (He et al. 2016a) has 11 million parameters (40MB). Closest us is MD-GAN (Hardy
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Training Algorithm of UA-GAN.
Input: Batch size m, datasets {Dj}, size of datasets { 21 n
2: Output: G, Dj.jj [K].
3: for fort=1,.... , do
4: (Work at the central server.)
5: G generates synthetic data: 2; = G(zz), i 1,... m.
6: Send batch of synthetic data D Dayn (01,'". 3 to all K sites.
7:
for j-= -,.....
do [Work at each local site.}
8:
9:
Update the local discriminator, D,. using real samples from Dj and synthetic data batch,
Dayn: based Eq. 2
10:
Output predictions and gradients for synthetic data Dy(tt), (Dz( /0 i = -,,. r. m.
Send them to the central server.
11: end for
(Work at the central server.)
13: Simulate value of Dua (.;) via Eq. Vi.
Update based on Eq. using gradients from Dj's.
15:
end for
al. 2019). which aggregates feedbacks (gradients) from local discriminators through averaging. also swaps parameters between discriminators. None of these methods provide theoretical guarantee as ours. Meanwhile, our method is the only one without model swapping, and thus much more efficient in bandwidth consumption.
Federated Learning (FL) (Kairouz et al. 2019 McMahan et al. 2016) offers the opportunity to integrate sensitive datasets from multiple sources through distributed training. Many works have been done tackling practical concerns in FL, such as convergence under Non-IID data assumption Yu et al.] 2019 Lian et al.] 2017 Li et al. 2020), decentralized SGD without freezing parameters (Recht et al. 2011 Nguyen et al. 2018), communication efficiency (Konečný et al.) 2016 Li et al. 2019), provable privacy guarantees (Alistarh et al. 2017 Wei al. 2020). Federated GAN is also of great interest from a federated learning perspective. successful federated GAN makes it possible to train a centralized model (e.g., a classifier) using data synthesized by the centralized generator. This becomes a solution when existing trained FL model needs to be replaced and updated by advanced machine learning approaches, as one can retrain the model at any time using the generator. It also alleviates some privacy concerns of FL, e.g., the gradient leakage problem (Zhu et al. 2019)).
METHOD
To introduce our algorithm, first introduce notations and formalize the mixture distribution learning problem. Next, we present our Universal Aggregation approach and prove that it is guarantced to learn the target mixture distribution. We also analyze the suboptimality of the model when local discriminators are suboptimal. For ease of exposition, we mostly use ordinary GAN to illustrate the algorithm and prove its theoretical properties. At the end of this section, we extend the algorithm, as well as its theoretical guarantees, to conditional GAN (cGAN) Mirza & Osindero) (2014). The empirical results in this work established the cGAN since its training much more controllable, thanks to the additional supervision by the auxiliary variable (c.g., classes of images).
Notations and problem formulation. We assume cross-silo FL setting Kairouz al. (0119). i..., K entities hosting K private datasets Di...., Dk, with size n1,-. ,K.. The total data size プラ= Mj. The overall goal is to learn target mixture distribution
p(x) = E K p(0)=
(2)((x))
in which a component distributions ()(x) is approximated by the empirical distribution from the j-th local dataset Dy. The mixing weight πy computed using the fraction of dataset Dy: πо = ng/n. n. In general, different mixture components (x) and p, (I) may (but not necessarily) be non-identical, namely, 3z, such that (x) A (z).
Universal Aggregation GAN: Now we are ready to introduce our mullt-dissriminator aggregation framework. A pseudo-code of UA framework can be found in Algorithm [ We have centralized
3

---

Note the Under review as a conference paper at ICLR 2021
(conditional) generator ((2) seeking to learn the global distribution p(z). In each local site, discriminator (z has access to local dataset D,. Note data in D, are only sampled from (y(2). During training, the generator generates batch of synthetic data all K sites. The j-th discriminator seeks to minimize the cross entropy loss of GAN from a local perspective Goodffllow et al. (
max ((G, D2) Eu-p, (a)) Dlg Dy Eg~N((. [og((1 - D)(G(z))))
D;
To formalize the generator training, we first introduce odds value. It is an essential quantity for our algorithm and its analysis.
Definition (odds value). Given a probability & E (0, 1), its odds value is (()) I- definition requires Φ + 1.
Also it is straightforward to see Φ = I4-0(0)
(((h))
The central idea of UA Framework is to simulate centralized discriminator Dua (x) which behaves like the mixture of all local discriminators (in terms of odds value). well behaved Dua (z) can then train the centralized generator G' using its gradient, just like in classical GAN.
We design Dua so that its odds value (DDua (x)) is identical to the mixture of the odds values of local discriminators:
(D (x)( = E, (,)(D,(x)).
(3)
Given (x))) we can compute Dua (=)(I)
Φ(D (a) (1))
Dua (x) = + 4 Duc (x))
(4)
Once the central discriminator Dwa is simulated, the generator can be computed by minimizing the generator loss:
min V(G, Duna) Ez=p(=> (ua( Eg~ [log(1 D (G(=))
(5)
Note that mathematically, Eq. (5) can be directly written terms of local discriminators Dj (by substituting in Eqs (3) and (4). In implementation, the simulated central discriminator can be written as pytorch or tensorflow layer.
The reason we define D..'' 's behavior using mixture of odds values instead of a mixture of the predictions is mathematical. It has been shown in Goodfellow et al. (2014) that perfect discriminator learning a data distribution p(z) and a fixed generator distribution g(z) satisfies p(a)
Intuition.
D(z) p(z)+a(a) It can be shown that only with the odds value equivalency, this optimal solution of the central discriminator Dzx) can be recovered each individual discriminator is optimal, i.e., P3 (w)
D,(x) p, (()+)(())) This is not true if define the central discriminator behavior using the average prediction, i.e., Duoe = E, D.. More details can be found in Theorem (4) and its proof.
Remark (Privacy Safety). For federated learning, it is essential to ensure information of real data are not leaked outside the local site. This privacy safety guaranteed in our method. To optimize G w.r.t. Eq. (3), we only need to optimize the second term and use gradient on synthetic images G(2) from local discriminators.
One important concern is about the optimal discriminator condition. Dwa is designed be optimal only when Dy's are optimal. We need to investigate the consequence if the local discriminators D, 's are suboptimal. We will provide an error bound of the learned distribution w.r.t., the suboptimality of D, 's in Corollary (2).
3.1 THEORETICAL ANALYSTS OF UA-GAN
In this section, we prove the theoretical guarantees of UA-GAN. First, we prove the correctness of the algorithm. We show that if all local discriminators perfectly learn from their local data, the algorithm is guaranteed to recover the target distribution (Eq. (T). Second, we discuss the quality of the learned generator distribution when the local discriminators suboptimal, due real-world
4

---

Under review as a conference paper at ICLR 2021
constraints, e.g., insufficient local data. We show that the enror of the learned distribution is linear to the suboptimality the local discriminators. All results in this section are for original (unconditional) GANs. But they can be extended to conditional GANs (see Sec. (3.2). ). Due to space constraints, only state the major theorems and leave their proofs to the supplemental material.
CORRECTNESS UA-GAN
The correctness theorem assumes all local discriminators behave optimally.
Assumption (Optimal Local Discriminator). Assume all local discriminators are optimal, i.e., they learn predict whether data true/fake perfectly, Let q(z) be the probability of the current
generator G. A local discriminator is optimal iff Dz(z) = ((x))
P2(2)
Theorem states the unbiasedness of UA-GAN: with optimal local discriminators, the generator learns the target distribution.
Theorem (Correctness). Suppose all discriminators D2 's are optimal. (ua( is computed via Eq. (3. Denote by q the density function of data generated by G. Let q" (-) be the optimal distribution the Jenson Shannon divergence loss:
qº == arg min L(g) = -Ezppiz) Dua (x:) + Ex [log(1 - Dua(z)}-
(6)
We have qº equals to the true distribution, formally, q" p.
The proof mainly establishes that when Dj are optimal, Daa is also optimal. With the optimality of Duo proved, the rest follows the correctness proof of the classic GAN (Theorem Goodfellow et al. 2014)). More details are in the supplemental material.
ANALYSIS OF THE SUBOPTIMAL SETTING
In centralized learning setting, an optimal discriminator is a reasonable assumption since the model has access to all (hopefully sufficient) data. However, in federated GAN setting, available data in some site Dj may be limited. One natural question in this limited data scenario is: how would the generator behave if some local discriminators are suboptimal? We address this theoretical question.
We first focus on single discriminator case, We show the behavior of a perturbed version of Jensen- Shannon divergence loss Guha et al. (2007); Lin (T99T); Csiszár al. (2004). The suboptimality of central discriminator (zx) is measured by the deviation in temms of the odds value. Denote by ((x) the generator distribution of the current G. Ideally, the odds value of optimal discriminator should
be p(z)/pr (()//<()) We show that a suboptimal D with 5 deviation from the ideal odds value will result in O(8) suboptimality the target distribution.
Theorem 2 (Suboptimality Bound for a Single Discriminator). Suppose a discriminator D(x)
is perturbed version of the optimal discriminator D(r), s.t. & ((D(x)) +(D(=)){(z) with {(x) < 8 and 8 < 1/8. Let q" be the optimal distribution of the Jensen-Sharnon divergence loss based on the perturbed discriminator
q" = arg min L(g) (lopp())]og B
[og((1 -(()).
Then qº satisfies la* (x)/p(z) - 1 168, Vr.
This theorem shows that the ratio of the learned distribution q" is close to the target distribution when the suboptimality of D, is small. To the best of our knowledge, this is the first bound on the Shannon divergence with suboptimal discriminator, even for a classical GAN.
consistency of
Next, we show that the bound is also tight.
Theorem 3 (Tightness of the Bound in Theorem Given a perturbed discriminator D(x) of the optimal one D(x), s.I. (D())) ((D())) with [(x) 1| 2 7 and 7 1/8. The optimal distribution qf' as in Eq. (12) satisfes 1q" (x)/p(z) - 2 761, Vr.
Next extend these bounds for single discriminator to multi-discriminann setting. This is based on Theorem 5 and the linear relationship between the local discriminators and the central discriminator.
5

---

Under review as a conference paper at ICLR 2021
Corollary (Suboptimality Bound for UA-GANN. Assume suboptimal local discriminators Dj D2(2)
are the perturbed versions of the optimal ones (z And the suboptimality is bounded as: & (D)(2z) ((D,(2)){,()) with [5,(z) - 1| & 1/8, 1/8, Vr. The centralized discriminator Dua (2) computed using these perturbed local discriminators such that (Dua(()) = Σn= "y (D)((z) Let q" the optimal distribution of the Jensen- Maasen-Shannon divergence loss based on the perturbed UA discriminator Dua
q' arg min L(g) E2xp(=) (log Pua(x)) [e)gg())oog(( - Dua (1)]-
(8)
Then q' satisfies 14" (x)/p(2) -
- In particular, the optimal distribution q" (x)) has (
1|
total variation distance to the target distribution p(æ)..
Note that the lowerbound of the suboptimality for single discriminator (Theorem can also be extended to UA-GAN similarly.
Remark 2. The consistency gap in Corollary (2) assumes uniform suboptimality bounded for all local discrininators. In practice, such assumption may not be informative if the sizes Df data are highly imbalanced. It would be interesting to relax such assumption and investigate the guarantees of UA-GAN w.r.t. the expected suboptimality of Dj
3.2
UNIVERSAL AGGREGATION FRAMEWORK FOR CONDITIONAL GAN
Our algorithm and analysis on unconditional GANs can be generalized to the more practical Condi- tional GAN Mirza & Osindero 2014). A conditional GAN learns the joint distribution of p(z, y). Here x represents an image or a vectorized data, and y is an auxiliary variable to control the mode of generated data (e.g., the class label of an image/data). Conditional GAN is much better to train in practice and is the common choice in most existing works. This is indeed the setting we use in our experiments.
The target distribution of the learning problem becomes a joint mixture distribution:
p(r,y) Σ Typ (g)p
in which иj = =n;/n and (y) is the proportion of class data within the j-th local dataset Dj. We assume Tj, and the dictionary of y and its fractions in each Dj. wj (v) are known the public. In practice, such knowledge will be used for generating Formally, ~ Σソ= (V)
To design the UA-GAN for the conditional GAN, the odds value aggregation in formula Eq. needs be adapted to:
(DD (x = E (T)
The computation of Dua (zly) and the update of and Dj's need to be adjusted accordingly. The theoretical guarantees for unconditional GANs can also be established for a conditional GAN. Due to space limitation, we leave details to the supplemental material.
EXPERIMENTS
On synthetic and world datasets, we verify that UA-GAN learn the target distribution from both i.d and non-identical local datasets. We focus on conditional setting as it is the common choice in practice.
& Osindero (014)
4.1 SYNTHETIC EXPERIMENT
We evaluate UA-GAN on toy dataset. See Fig. 2 first row. The toy dataset has 4 datasets, generated
by 4 Gaussians centered at (10, 10), 10). (- (-10,10), (-10,-10) with variance 0.5. Data samples are shown as blue points. The central generator takes Gaussian noise centered at (0,0) with variance of
0.5 (green points) and learns to transform them into points matching the mixture distribution (orange points). The first figure shows the generator successfully recovers the Gaussian mixture. The contours

---

Under review as a conference paper at ICLR 2021
UA-GAN
Avg-GAN
MD-GAN
Figure 2: Results on toy dataset by UA-GAN, Avg-GAN and MD-GAN. UA-GAN can learn four Gaussians, whereas Avg-GAN and MD-GAN fail.
non-identical Mnist + Fashion
non-identical Mnist Font
Dataset
Accuracyt
ISt
FID
Accuracyt
ISt
FID
Real Centralized GAN
0.943
3.620 = 0.021
0.994
2.323 + 0.011
0
0.904
3.437 = 0.021
8.35
0.979
1.978 + 0.009
17.62
Avg GAN
0.421
4.237 + 0.023
72.80
0.822
1.517 + 0.004
85.81
MD-GAN
0.349
2.883 = 0.020
102.00
0.480
2.090 = 0.007
69.63
UA-GAN
0.883
3.606 = 0.020
24.60
0.963
1.839 + 0.006
24.73
Table I: Quantitative results on non-identical mixture datasets. UA-GAN achieves better result compared with the Avg GAN and MD-GAN's aggregation method.
show the central discriminator Duo calculated according to Eq. The generated points (orange) are evenly distributed near the 4 local Gaussians' centers. The 2nd to 5th figures show the prediction of the local discriminators (D, 's), overlaid with their respective samples Dj in blue.
Mcanwhile, show in the second row the results of the average scheme, called Avg-GAN, which averages local discriminators' outputs to train the generator. The first figure shows that the learned distribution Davg = K Σ, Dj is almost flat. The generated samples (orange) collapse origin. Although each individual discriminator can provide valid gradient information (see 2nd to 5th figures for D, 's), naively averaging their outputs cannot achieve the target distribution, when the distributions are symmetric. We show the results of the MD-GAN by Hardy al. (2019) in the third row. MD-GAN also adopts the average scheme, but randomly shuffle discríminators parameters during training. Similar to Avg-GAN, MD-GAN cannot learn the four Gaussians.
the
4.2 REAL-WORLD MIXTURE DATASETS
We evaluate our method on several mixture datasets, both i.i.d and non-identical.
Three real-world datasets are utilized to construct the mixture datasets: MNIST et al.] (1998), Fashion-MNIS' Xiao et al. (2017), and Font dataset. We create the Font dataset from 2500+ fonts of digits taken from the Google Fonts database Mo (2002). Similar to MNIST, it consists of 10 classes of 28 28 grayscale images, with 60k samples for training and 29k samples for test. To make the differences more clear between font and handwrite images, we highlight the Font images with a circle when build the dataset. Using these foundation datasets, create 2 different mixture datasets with non non-identical local datasets: (1) non-ide non-identical MNIST+Fashion; (2) non-identical MNIST+Font. We uniformly sample Fashion/Font data for all 10 distributed sites. These common patterns across all sites. Mcanwhile, for each individual site, we add MNIST data with distinct class among 9. These data distinguishable features for different sites. Ideally, federated GAN should be able to
Datasets.

---

Under review as a conference paper at ICLR 2021
1
2
2
2
2
3
3
3
3
3
3
E
3
3
R
3
3
E
3
3
Y
P
P
5
5
5
5
S
S
S
S
S
S
S
S
S
5
6
6
6
6
G
6
7
7
7
7
7
O
7
7
?
8
8
8
J
A
9
R 9 9 9
9 9 9
(a) Avg GAN
(b) MD-GAN
(c) UA-GAN
(d) Avg GAN
(c) MD-GAN
(1) UA-GAN
Figure 3: Synthetic images on the non-identical MNIST+Fashion ((a).(b.(c)) and MNIST+font datasets ((d).
learn both the common patterns and the site-specific data. Please see supplemental material for more details and samples from these mixture datasets.
Baselines. We compare UA-GAN with Avg-GAN and another SOTA federated GAN, MD- GAN Hardy et al.] (2019). We also include two additional baselines: Centralized GAN trains using the classic setting, i.c., one centralized discriminator that has access to all local datasets. Comparing with this baseline shows how much information we lose in the distributed training setting. Another baseline is Real, which essentially uses real data from the mixture datasets for evaluation. This is the upper bound of all GAN methods. More details can be found in supplementary material.
Evaluation metrics. We adopt Inception score (IS) Salimans al. (2016), Frechet Inception distance (FID) Heusel et al.] (017)) to measure the quality of the generated images. As GAN is often used for training downstream classifiers, we also evaluate the methods by training classifier using the generated data and report the Classification Accuracy as an evaluation metric. This is indeed very useful in federated learning; a centralized classifier can be trained using data generated by federated GANs without seeing the real data from private sites.
The quantitative results on the two non-identical mixture datasets are shown in Table I UA-GAN sigmificantly outperforms the other two federated GANs, Avg-GAN and MD-GAN Its performance even close to the centralized GAN, showing that our algorithm manages to mitigate the challenge of distributed training to a satisfying degree.
Discussion.
The superior performance of UA-GAN can be illustrated by qualitative examples in Fig. On MNIST+Fashion dataset (subfigures a-c), the average aggregation strategy used by Avg- WAgGGAN could not effectively aggregate outputs of 'non-identical local discriminators. Therefore, it only learns to generate the common patterns, e.g., Fashion images (Fig. a)). MD-GAN fails to produce high quality images (Fig. b)), probably because the discriminator switch makes the training not stable enough. Mcanwhile, our UA-GAN is able generate the mixture with both common patterns (Fashion images) and site-specific images (different digits from MNIST) with high quality. Similar phenomenon can be observed for MNIST+Font (subfigures d-f). Avg-GAN only learns the common pattern (computerized fonts from Font dataset), MD-GAN gives low quality images whereas UA-GAN can also learns the high-quality site-specific handwriting digits (MNIST).
Note that we also compare the methods on mixture datasets with i.i.d local distributions, i.e., all local datasets are sampled in same way from the real datasets. In an i.i.d setting, all federated GANs and the centralized GAN perform similarly. More results will be included in the supplemental material.
5 CONCLUSION AND FUTURE WORK
In this work, we proposed a provably correct federated GAN. It simulates a centralized discriminator via carefully aggregating the feedbacks of all local discriminators. We proved that the generator learns the target distribution. We also analyzed the error bound when the discriminator is suboptimal due to local dataset limitation. A well- well-trained federated GAN enpowers GANs to learn from diversified datasets. It can also be used as a data provider for training task-specific models without accessing or storing privacy sensitive data.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication- efficient sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems, pp. 1709-1720, 2017.
George J Annas et al. Hipaa regulations-a new era of medical-record privacy? New England Journal of Medicine. 348(15) (88((1))1866-1455) 2003.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875 2017.
Wasserstein gan. arXiv preprint
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Imre Csiszár, Paul C Shields, et al. Information theory and statistics: A tutorial. Foundations and Trendy Communications and Information Theory, 144):417-528, 2004.
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Russ R Salakhutdinov. Good semi- supervised learning that requires a bad gan. In Advances in neural information processing systems, pp. 6510-6520, 2017.
Ian Goodfellow, Jean Pouget-Abadic. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672 26802,2662222 2014.
Lawrence O Gostin, Laura Levit, Sharyl Nass, al. Beyond the HIPAA privacy rule: enhancing privacy, improving health through research. National Academies Press, 2009.
Sudipto Guha, Piotr Indyk, and Andrew McGregor. Sketching information divergences. In Interna- tional Conference on Computational Learning Theory (COLT), pp. 424443444 Springer, 2007.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, 2017.
Corentin Hardy, Erwan Le Merrer, and Bruno Sericola. Md-gan: Multi-discriminaoor generative adversarial networks for distributed datasets. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 866-88 677. IEEE, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016b. doi: 10.11099cx::2016.900
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp.
2017.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 2019.
Melita Sogomonjanl Tanel Kerikmäe. Challenges in the e-health regulatory policy. HOW DEEP IS YOUR LAW? BREXIT. TECHNOLOGIES MODERN CONFLICTS, 367, 2017.
Jakub Konečný, H Brendan McMahan, Felix X Yu. Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint /6100:/610.0544 2016.
Abhishek Kumar, Prasanna Sattigeri, and Tom Fletcher. Sumi-super learning with gans: Mani- fold invariance with improved inference. In Advances in Neural Information Processing Systems, pp. 5534-5544 2017.

---

Under review as a conference paper at ICLR 2021
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 -2224, 1998.
Milieu Ltd & Time. lex. Overview of the national laws electromic health records in the member states and their interaction with the provision of cross-border ehealth services: Final report and recommendations (contract 2013 63 02), viewed 18th march 2018. 2014.
Xiang Li, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Communication efficient decentralized training with multiple local updates. arXiv preprint arXiv: 1910.09916 2019.
Xiang Li. Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020. URL
https: /tta:///ppervveee ew net.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu, Can decentralized algorithms outperform centralized algorithms? case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems, pp. 5330-5340, 2017.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions Information theory, 37(1):145-151, 1991.
Huidong Liu, Xianfeng Gu, and Dimitris Samaras. A two-step computation of the exact GAN Wasserstein distance. In Proceedings of the International Conference on Machine Learning, 2018.
Huidong Liu, Xianfeng Gu, and Dimitris Samaras. Wasserstein gan with quadratic transport cost. In The IEEE International Conference on Computer Vision (ICCV), October 2019a.
Ming- YunggYYulu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10551-1056A 2019b.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-efficie learning of deep networks from decentralized data. Artificial Intelli- gence and Statistics, pp. 1273-1282, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication--fficieenn learning of deep networks from decentralized data. arXiv preprint arXiv: 00662.00.66........ 2016.
Rebecca T Mercuri. The hipaa-potamus in health care data security. Communications of the ACM, 47 (7):25-28, 2004.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In Proceedings of the International Conference on Machine Learning, 2018.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.1788 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference on Learning Representations, 2018
Lawrence Mo. Internet-based font server, January 24 2002. US Patent App. 09/863,250.
Lam M Nguyen, Phuong Ha Nguyen, Marten van Dijk, Peter Richtárik, Katya Scheinberg, and Martin Takáč. Sgd and hogwild! convergence without the bounded gradients assumption. arXiv preprint 18Xiv:1802.03800 2018.
Guo-Jun Qi. Loss-sensitive generative adversarial networks on lipschitz densities. International Journal of Computer Vision, pp. 1-23, 2019.
Hui Qu, Yikai Zhang, Qi Chang, Zhennan Yan, Chao Chen, and Dimitris Metaxas. Learn distributed gan with temporary discriminators. arXiy preprint arXiv:2007.0922 2020.
10

---

Under review as a conference paper at ICLR 2021
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693-701,2011 2011.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems, 2016.
Jonathan JM Seddon and Wendy I. Currie. Cloud computing and trans-border health data: Unpacking us and healthcare regulation and compliance. Health policy and technology, 2(4):229-241) 2013.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from single natural image. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4570-4580, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 140.::14091566 2014.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwaakarr Federated multi-task learning. In Advances in Neural Information Processing Systems, pp. 4424 2017.
Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. Improving generalization and stability of generative adversarial networks. In Proceedings of the International Conference on Learning Representations, 2019.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard Yang. Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, arXiv preprint arXiv:
2017.
Bangzhou Xin, Wei Yang, Yangyang Geng, Sheng Chen, Shaowei Wang. and Liusheng Huang. Private fl-gan: Differential privacy synthetic data generation based on federated learning, In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2927-2931 IEEE, 2020.
Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TTST), 10(2):1-19, 2019.
Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference on Machine Learning, pp. 7184-7199, 2019.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to few-shot learning. Advances Neural Information Processing Systems, pp. 2365-2374, 2018.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint 3.Kiv:/806.00533 2018.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems 32, pp. 14774-14784, Curran Associates, Inc., 2019. URL http // 7/papers.. httppppscccppppppee99e./ deep leakage from gradients.pdf
In this supplemental material, we provide proofs of the theorems the main paper (Sec. We also
provide additional experimental details and results (Sec. B
A PROOFS OF THEOREMS IN SECTION 3
We recall the definition of odds value. Definition 2 (odds value). Given probability & (0, 1), its odds value is (0)) definition requires Φ # 1.
Note the 1-8-
11

---

Under review as a conference paper at ICLR 2021
A.1 ANALYSIS OF OPTIMAL DISCRIMINATOR
We recall the theorem of the correctness of UA-GAN.
Theorem (Correctness). Suppose all discriminators D 's optimal. Pua(s) is computed via
Eq. M7P. The optimal distribution of the Jenson-Shannon divergence loss:
arg min L(g) (Px-ppas) log Dua (2)) (a [og(11 Dua (x))
is q" where q is the density (mass) function of G(z).
To prove the theorem, we first introduce the following Lemma, which is similar to Proposition in Goodfellow et al. (2014). We include the Lemma and Proof here for completeness.
Lemma 1. When generator is fixed, the optimal discriminator Dy D)))) is:
Dy(#) P2(x) P5(x)) (xx)
(10)
Proof:
/ + + - D2 (x)))dr
max Vj(D,) = max
Dj
Dj
(x) + (0)) Dr(x)))dr
Dj
(a) by setting (x) Py inequality hold with equality.
[)) we can maximize each component in the integral thus make the
Proof of Theorem Suppose in each training step the discriminator achieves its maximal criterion
in Lemma] the simulated Dua (z) becomes:
(x)x) Dua (2) = Σ₁ n, (y)
(II)
1+ Σ3 K (V(1) (-)
D7(2)
Given that discriminators D1, DR behave optimally, the value of (2( P2(x) Σ, (s(x) E, (1p)(1)
which implies
D2(z) <())
D (x)
By the aggregation formula the simulated discriminator
1=))(a) 9(00) 2, (m)
(z)
will be Dua (x) Σ, 3,P, (2) p())-g(2)) Suppose in each training step the discriminator
pla)
achieves its maximal criterion in Lemma the loss function for the generator becomes:
min L(g) Expp(z) [log ((=)] + E [log(1 D(+))
E Be-n(a (log D(x)] + Ea [log(1 D
p(x) + g(x) log
g(x) dr
p(x) log p(z) + ((x))
p(x)) + ((x)
The above loss function has optimal solution g due to Theorem
Goodfellow et al. (0014).
A.2 ANALYSIS OF SUB-OPTIMAL DISCRIMINATOR
We provide proofs of Theorems 6 and Corollary 2
Theorem 5 (Suboptimality Bound for Single Discriminator). Suppose discriminator D(x)
a perturbed version of the optimal discriminator D(x),, s.t. ((D(x)) 4 +(D(z)))(z)) with 1 - [(x) < 8 and 5 < 1/8. Let q" be the optimal distribution of the Jensen-Shannon divergence loss based on the perturbed discriminator
min L(g) = Ez D(x)) + Ex [lgg(1 - D(x)).
(12)
Then q" satisfies la" (x)/((x) 1 165, Vr.
12

---

Under review as a conference paper at ICLR 2021
Lemma 2. Suppose 0 < |a]. |6| - 5. log la) < we have 1 1-2/a|-22
_ 1 11+ 4 4|b]. I-p
Proof:
In the proof will use following fact:
1 6 11+ 2x, for 1
(13)
By equation log log(2 1 + a) + we have p ( a)e² thus:
1
[a)) - 2 p 2 + |a)) (1+ + 2|bl)
2
1 +
|a| b| s p
+||+ + |6| 2|ab|
2
1
1
|a|
b
2|ab|
- -
+ |a| [6|
(14)
2
2
2
|a|
bb|
p
1 + [a|
+6 2/ab|
2
[ +
b6|
- p
3
|a| |b 2/ab|
(1
2/a| 2/6))
I (1 + 4/a + 4(b))
1
Lemma 3. Suppose Mx)/p(x) 2 2. the following loss function is strongly convex:
h(x) + ((x) log ((x) h(x))
((x) dr
L(q) p(x) log h(x) + q(x)
(15)
Proof:
The first order derivative of L(g) is 8L(g) h(z)-p(a) ला(z) (())+((>) + log
((a) The second order derivative
+(())
8 L(g)) is 0q(x)= = ((e))
(There is no non-diagonal elements in the Hessian).
The proof for Theorem is shown below. The theorem focuses on Jensen- Shasnnnn Divergence loss
Guha al. (207): Lin 199T). We stress that the analysis is general and works for both general GAN
and conditional GAN.
Proof of Theorem
5
Note & (z))) = v(z)) The {(x) perturbed odds value gives 4 (D(()) = (D(()))((() p(=)((z)
p(z)
o(a)
Let f(x) ((x), we have - 8 S h(#)
p(z] < + 8 and 1 - d p( h(x)
- 1 26. The perturbed
h(w)
value of discriminator will be D(x) (()) The loss function that the generator distribution
q(x) seeks to minimize a perturbed Jensen-Shannon Divergence loss:
T(g)= [ p(x) log
((x) h(x) + qax) + ((x) log h(x)) ((x) dr X j g(x)dr 1oodr-1.. -1) (16)
M(x))
where represents the Lagrangian Multiplier. In Lemma 3 we verify the convexity of this loss function with regulated behavior of h(x). The derivative of L(g) w.r.t. q(x) is:
p(x) g(x) + h(T)
((x)
+ log(g (x)) + log(g(x) + h(x))
((x) + h(x)) +
which needs to be 0 for all value of x Boyd & Vandenberghe 2004). Thus we have following equation:
<(x) p(x) log g(x) + h(x))
((x))
(17)
((x) + h(x)
Since above equation holds for all æ. Let h(x)-p(z) = A(x) which has value bounded by -8 < A(x) < 8. we can multiply p(z) + q(x) on both side and integral over æ:
g(x)+h(x)
/ (M(z) + (h(x) + ((x)) log g(x)) + h(x) dx== -0- / A(h(x) + g(x))dz
((x)
13

---

Under review as a conference paper at ICLR 2021
which gives us A + A2 -2 where:
1 2 (h(x) + (z))))(x)dd, A
2 /M((x) + ((x)) log
1
((x) dx
A Plugging in above derived value of X back into Equation 21 we have:
g(x)) h(x))
g(x)
log
A + + A(x)
(18)
((x) p(x)
By the uniform upper bound on |x(x) I 26 we have
2
< c < + _ and
1 - o/4 8 v e v 1+ 56. By taking exponential operation on both sides of Equation 22 we have
((x) c * (1 + 46)
g(x) h(x)
Thus we can bound the value of e by the equation:
(xx (g(a:) (exr) (1 + 46))
(19)
Due to the fact that the equation holds for all value of x, by integrating Equation 19 have A2 2 45) which is equivalent to A2 log(} + 46). Thus log g(2)
g(0)+h(z) log( 46) A.
By Lemma we have g(w)) = 1 166.
p(a)
Theorem (Tightness of the Bound in Theorem 5 Given a perturbed discriminator D(x) the optimal one (xx), s.f. Φ ((x)) = + ((x))))((r with (5(x) 1| 2 y and y S 1/8. The optimal distribution q" in Eq. 12 satisfies 14" (x)/p(z) - /166, Vr.
Proof:
By similar steps to proof in Theorem we have:
h(x)
((x)
dx + Al/ g(z)do- - 1).
L(q) - p(x) log 1(x) + q(x)) + ((x) log h(x) ((x)
(20)
By setting the derivative of L(4) w.r.t. g(x) be 0 we have
h(x) p(x) (xx) h(x))
g(x)
log ((x)) h(x))
(21)
Let h(x)-p(s) (xx). By assumption that (5(x) - 1| 2 y we have
(())+h(a)
Aax)) [h(x) - p(x)
[E(x)
g(x) + l(x))
h(x)) ((x)
p(#) p(a)
Due to the fact that [g(x) 1 <
1 8. we have 2(x) p(x)
< 2. Thus we can bound |x(x) 2 3.
Next analyze following equation, can multiply p(z) + q(x) both side of 21 and integral over x:
J (h(z) + (xx)))((x)) - (h(#) + ((x)) log q(x) + h(x)
q(x) dx = X(h(x) + g(x))dzx
which gives us A + A2 22 where:
2 / (h(x) (z)))) A
2 / /(((x)) + ((x)) log q(xr) + h(x)
((x) dr
A Plugging in above derived value of À back into Equation 21 we have:
q(x)
log ((x) + p(z)
A + A. - A(x)
(22)
14

---

Under review as a conference paper at ICLR 2021
Next analyze the term A - A(x).
A1 - (x) = / (h(x) + (())) - ((x)
2 (h(x) + ((x)) M[)) p(x)
dx - A(x)
((2) + h(x)
A(z)
Due to the fact that |a 3. We can derive that Theorem 3 we have
1 Ts. By an analysis similar to
P 1
Corollary (Suboptimality Bound for UA-GAN), Assume suboptimal local discriminators D, (a)
the perturbed versions of the optimal ones (z(z). And the suboptimality is bounded as: & (x)) = (DD,(z)) with (z) 1| 8 S 1/8, Vz. The centralized discriminator Dua (z) is computed using these perturbed local discriminators such that 4 (ua (x)) = -Ø= Fy (D, (x))) Let q" be the optimal distribution of the Sensen-Shanoon divergence loss based on the perturbed UA discriminator Dua
G- arg min l(g) = Ezmp(()) (log Dava (x)) + Exwng( Dog(( Pua(2))
(23)
Then q' satisfies la (x)/p(z) - = =(0). In particular, the optimal distribution (xx) has O())
total variation distance the target distribution p(x).
Proof:
Let Ui 's be odds values of optimal discriminators Dy(x)'s:
(x( s: = 1- (x) and iy's be odds values of
suboptimal discriminators (y(x)'s. suffices to show Eg 7yps
1| S 8 and apply Theorem 5
B ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
Implementation Details: Here summarize details of the network we in the experiments. Our UA-GAN has one centralized generator and multiple local discriminators. The generator consists of two fully-connected layers (for input noise and label, respectively). five residual blocks He et al.) (2016b) and three upsampling layers. Each discriminator has two convolutional layers (for image and label, respectively), five residual blocks and three average pooling layers. LeakyReLU activation is used in both generator and discriminators. During training, we apply gradient update of the diseriminators each round. Each model is trained with Adam optimizer for 400 epochs with batch size of 256. The learning rate is initially 0.0002 and linear decays to 0 from epoch 200. The VGG (Simonyan & Zisserman (2014) 11-layer model used for the downstream classification task. We pad the image to 32 x 32 and then randomly crop them to 28 28 with batch size of 64 as input. The model trained with SGD optimizer using a learning rate 0.01 for 150 epochs.
Dataset Details: One of our foundational datasets is the Font dataset.
created from 2500+
fonts of digits taken from the Google Fonts database. Similar to MNIST, it consists of 10 classes of 28 x 28 grayscale images, with 60k samples for training and 29k samples for test.
Based on the MNIST, Fashion-MNIST and Font dataset, create both i.i.d mixture datasets and non-identical datasets. Details on non-identical datasets have been provided the main paper. Here we provide details on two i.i.d datasets. (1) i.i.d MNIST+Fashion: (2) i.id MNIST+Font. Each of the 10 distributed sites contains 10% of mixture dataset which is uniformly sampled (without replacement) from MNIST and Fashion/Font.
B. EMPIRICAL RESULTS ON I.L.. DATASETS
The quantitative results on the i...d mixture datasets shown in Table 2 One all three distributed GAN methods have comparable performance. can also observed from qualitative examples in Fig. 4 that all three methods achieve similar results. This suggests that all three
15

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Precise Training Algorithm of UA-GAN.
Input: Batch size m, datasets {Dj}, size of datasets { 21
2: Output: G, D,, Vj E [
3: for Number of total training iterations do
4: for Number of iterations to train discriminator do
5: [Work at the central server.)
6: G generates synthetic data: 24 G(zz), i= 1,
7: Send batch of synthetic data Dayn {01, ... Lm) to all sites.
8:
for j 1... K do
9:
{Work at each local site.}
10:
Uniformly randomly choose m real samples {zt,--. 2 from Dy:
Update the parameters of local discriminator Dy: fg using
Izg(D)(<()) + log(1 D2().)))
8
12:
end for
13:
end for
14:
(Work at each local site.}
15:
G generates synthetic data: á; = G(2)), 1,. m.
Send batch of synthetic data Dayn { mm) to all K sites.
17:
for 1, 1,.,.... K do
(Work at each local site.}
19:
Output predictions and gradients for synthetic data D,(ās), 0D,(2:)/02, i == - m.
Send them to the central server.
20:
end for
[Work at the central server.)
22:
Simulate value of Dua (æ;) via Eq. 4 Vi.
23:
Update parameter of G: 0g by descending its stochastic gradient:
a llog(1 Duo (i+))
0Dne 00 D
(D.) 0D, (d:)
8i:
0D (f,)
@D,( 0ā:
D6G
j=1
24:
end for The gradient- madde updates can use any standard gradient- baadee learning rule.
approaches can be used to train distributed GAN when datasets have i.i.d. distribution e.g., the data uniformly shuffled before sent each discriminator. Note that with similar performance, UA-GAN has much smaller communication cost compared to MD-GAN since the UA-GAN does not swap model parameters during training process.
i..d Mnist + Fashion
i.id Mnist + Font
Dataset
Accuracyf
ISt
FIDJ
Accuracyt
IST
FIDJ
Real
0.943
3.620 + 0.021
0
0.994
2.323 +: 0.011
0
Centralized GAN
0.904
3,437 + 0.021
8.35
0.979
1.978 + 0.009
17.62
Avg GAN
0.905
3.371 + 0.026
12.83
0.967
1.923 + 0.006
19.31
MD-GAN
0.884
3.364 + 0.024
13.63
0.971
1.938 + 0.008
19.65
UA-GAN
0.908
3.462 + 0.024
11.82
0.970
1.934 + 0.008
19.18
Table 2: The classification accuracy and IS, FID scores on two i.i.d mixture datasets. All of the three architecture could learn the right distribution with i.id datasets.
16

---

Under review as a conference paper at ICLR 2021
O
I
L
R
2
Z
2
2
2 2
2
2
2 E
2
2
S
S
3
3
3 3
3 3 3 E
3
3
4
H
Y
O
I K 4 A 4 D A 4 4 4
4
Y 4
S
5
9 S 5 S S S S
S S S
S
S
6
6 G
G
DOOOO G 6 6 6
G
G
G
G
7
7 1
7X
7 7
7
S
8
9
F
2 8 &
S
5
8
9
(a) Avg
GAN MNIST Fashion
(b) Avg GAN UNNIS++Fom
O
o
T
O
O
O
O
O
D D D O
/
3
2
A
2
3 J
2
2
2
3
3
3
5
3
2
S
3
3
3
3
3
3
Y
4
4
Y
H O Y 4 S
4
4
4 4 4 4
Y
F
5
S
S
6
K
S
S
S S
S
5
O
3
S 5
S
3
S
S
6
6
6
6
6
E
6
6
G
G &
G 6
S
6
G
6 E
6
G
G
6
6
6
7
7
7
2
7
e
X
Q
R
F
&
P
S
9
99
9
(c)
MD GAN MNIST
ANIST+Fashion Fashion
(d) MD GAN MNIST+Fom Font
O
O
0
O
O
0
D
O
O OOO DO 0
C
/
1
L
1
2
3
4
Y
4
5
S
S
S
S
B S
S
S
5
5
6
b
6
G
6
6
6
G
6 6
G
G
G
6 6 G
G
6
6
6
V
G
G 6
7
O O
7
7
7
7 7
S
8
Z
5
9
&
9
9
6
3 G
9
9 9 T
9 9 9 9 9
(e) UA-GAN MNIST+ Fashion
(f) UA-GAN MNIST+Font
Figure 4: Synthetic images on the identical MNIST+ Fashion dataset (ca). and MNIST+Font dataset (d)) using the average method, MD-GAN Hardy et al. (2019) and our UA-GAN method. All of the models could capture distributions over MNIST and Fashion/Font.
B.2 ADDITIONAL EMPIRICAL RESULTS ON NON-IDENTICAL DISTRIBUTION
We provide additional synthetic images in non-identical distribution cases. See Fig. By using average aggregation method, the synthetic image produced by Avg GAN and MD GAN only have Fashion images in (a), (c) and Font images in (b), (d). Our method in (c) and (f) could capture different patterns in MNIST + Fashion/Font and generate diverse images.
B.3 EMPIRICAL RESULTS OF MIXING THREE DATASETS
We report the results of mixing the three datasets MNIST, FashionMNIST and Font. In the non- identical setting, we add MNIST data with a distinct class among 0~-9. These data are distinguishable features for different sites. And uniformly sample Fashion and Font data for all 10 distributed sites. These are common patterns across all sites. In the identical setting, all three datasets are uniformly distributed across the 10 sites. The quantitative results are shown in Table 3 The synthtic images are shown in Fig. and Fig. 7 By using average aggregation method, the synthetic image produced by Avg-GAN and MD -GAN only have Fashion and Font images in Fig. a)a). (b). Our method in Fig. ((c) could capture different patterns in MNIST + Fashion + Font and generate diverse images.
B.4 EMPIRICAL RESULTS IN LARGER FEDERATED LEARNING SETTING
We report the results when using larger scale nodes(n 50) in distributed GAN methods(Avg-GAN, MD-GAN and UA-GAN). We uniformly split each individual site of the non-identical MNIST + Fashion dataset into distributed sites. In total, adopt 50 non-identical MNIST+Fashice datasets with 2380 MNIST and Fashion images each. The quantitative results are shown in Table and the synthetic images are shown in Fig 8
17

---

Under review as a conference paper at ICLR 2021
33 2
2
3 S
4
O 4
5 8 9 5 S 9 5 9 19 S P
S 5 5
66 O
6 OF
6 G 6
L Z 7 1O0D 7 7 7
(a) Avg GAN MNIST+ Fashion
(b) Avg GAN INIST+Foot
DDO
O
O
2
2
2
2
2
2
3 3 3
T
S
S
S
S
S
O
S S O S
S
S
S
S
S
C
C
C
G
G
G
G
G
C
& G
G
G
G
G
G
O
6 7
O
7
7
7
7
7
7
O
7
T
S
S
S
S
S
&
S
S
S
S
J
R
7
9
O
?
9
MD GAN MNIST +ASS++Fashhe Fashion
(d) MD GAN MNIST+ Font O
T
O
0
O
o
a
-
0
T
0
8
O
O
O O
O
Ø O 1
O
O
O
O
J
2
2
2
2
3
3 3
4
M
Y
X
5
5
5
SSS
S
5 S B S S 5
b
6
6
6
G
G
6
6
6 6 O 6 G G C G OF 6 6
6
G
7
7 2 7 7 Z O OO DU ?
Z D
8
8
S &
& R & 8
9
4
9 9 q DI 9 7
O 9
(e) UA-GAN MNIST+ Fashion
(f) UA-GAN MNIST+Font
Figure 5: Additional synthetic images on the non-identical MNIST+Fashion dataset ((a).C)) and
MNIST+Font dataset (f)) using the average method, MD-GAN Hardy et al. 2019 and our UA-GAN method.
non-i.i.d
i.i.d
Dataset
Accuracyt
ISt
FID4
Accuracyt
1St
FID
Real Centralized GAN
0.955
3.426 + 0.023
0.955
3.426 0.023
0.943
3.031 + 0.016
14.90
0.943
3.031 + 0.016
14.90
Avg GAN
0.822
3.144 = 0.013
41.63
0.936
2.877 0.013
17.90
MD-GAN
0.567
3.035 = 0.011
56.19
0.936
2.951 = 0.019
16.81
UA-GAN
0.933
2.949 + 0.023
20.80
0.923
2.875 + 0.013
17.34
Table 3: The classification accuracy and IS, FID scores mixture of three datasets.
non-i.id MNIST + Fashion (50 data sites)
Dataset
Accuracy?
ISt
FID
Real Centralized GAN
0.943
3.620 + 0.021
0
0.904
3.437 + 0.021
8.35
Avg GAN
0.489
3.755 + 0.023
90.36
MD-GAN
0.465
3.830 +-2 0.020
89.36
UA-GAN
0.626
3.531 + 0.018
53.26
Table 4: The classification accuracy and IS, FID scores on non-i.i.d mixture datasets for 50 distributed sites.
18

---

Under review as a conference paper at ICLR 2021
10
0
D
I
000
2
2
2
2
2
Z
33
9
3
3
3
3
3
3
3
3
3
4 4
4
R
36
5
9
5
S
S
S
S
5
S
5
6
0
6
G
6
G
2
A9
(a) Avg-GAN MNIST + Fon
(b) GAN MNISTTFaashin+-Fon
D
OOL
O
O
D
O
2
2
2
3
3
3 3
Y
4
4
O
4
4
O
S
S
2
S
5
S
S
S
G 5 S
S
E 6 G
6
G
6
B
G
6
7
i
7 7
/ &
F
(c) GAN MNIST+
Figure 6: Synthetic images on the non non-identical MNIST+Fashion+ Font dataset using the average method, MD-GAN Hardy et al. (2019) and our UA-GAN method.
U
0
O
10
10
2
2
2
R
3
3 3
3
9
3
R
3
3
3
4
4
4
4
4
4
Y
I
S G
5
P 6
S
S
S
D
3
L
S
S
5
5
3
6
C
6
6
O
6
6
6
6
6
6
G
6
O
7
G
G
G
6
6
7 7
7
7
7
F
7
7
3 8
3
5
8
B
8
9
9
Z
9
9
7
9
(a) Avg-GAN MNIST+ F Font
(b)
MD GAN -aAISs+Faasion-ee. Font O
0
0
O
O
1
2
2
3
K
3
3
4 #
Y
S
5
G
6
S
G
O
7
9
7
(c) UA-GAN MNIST+ Fassionnooo
Figure 7: Synthetic images on the identical KANIS-+Fashion-Fon dataset using the average method, D0--GN/Haarda, al. (2019) and our UA-GAN method.
B.5 EMPIRICAL RESULTS IN UNCONDITIONAL SETTING
We report the results when using unconditional GAN in all methods (centralized, Avg-GAN MD- GAN and UA-GAN). The quantitative results are shown in Table53and Table The synthetic images shown in Fig. 2 and Fig. 10 In the unconditional setting, the condition variable (labels) won't be given thus one can not directly apply the synthetic data in training classification model. Therefore we don't compute the classification accuracy Table 5 Table
B.6 EMPIRICAL RESULTS OF IMBALANCED DATASETS IN DIFFERENT SITES
We report the results when the sizes of the 10 sites are not the same. Based on the non-identical MNIST + fashionMNIST dataset, reduce the sample sizes of the first 5 sites by half and keep the
19

---

Under review as a conference paper at ICLR 2021
(a) Avg-GAN MNIST+Fashhon(S 50 data sites) (b) MI-GAN MINIST+Fashion5500 data sites)
O
2
3
3
S
5
S
L
C
7
F
S
(c) UA- GAN MNIST FashionnSO data sites)
Figure 8: Synthetic images on the non-identical MNIST+Fashion dataset for 50 distributed sites using the average method, MD-GAN Hardy et al. (2099) and our UA-GAN method.
non-i.id MNIST
Fashion
non-i.id MNIST + Font
Dataset
Accuracyt
1St
FID
Accuracyt
1St
FID
Real Centralized GAN
0.943
3.620 + 0.021
0.994
2.323 0.011 1.975 + 0.009
3.387 = 0.019
8.46
17.56
Avg GAN
4.068 = 0.020
61.56
1.547 0.005
80.07
MD-GAN
2,852 = 0.021
60.34
1.887 0.007
36.36
UA-GAN
3.280 + 0.022
22.34
1.985 + 0.013
22.17
Table 5: The classification accuracy and IS, FID scores on two non-i.i.d mixture datasets in uncondi- tional setting.
iid MNIST Fashion
ii.d MNIST + Font
Dataset
Accuracyt
ISt
FID|
Accuracyt
ISt
FID|
Real Centralized GAN
0.943
3.620 = 0.021
0
0.994
2.323 0.011
0
3.387 = 0.019
8.46
1.975 + 0.009
17.56
Avg GAN MD-GAN
3.326 + 0.016
9.68
1.918 + 0.006
19.52
3.428 = 0.025
12.04
1.934 0.006
18.85
UA-GAN
3.367 = 0.018
9.99
1.937 0.009
18.83
Table 6: The classification accuracy and IS, FID scores two i.i.d mixture datasets in unconditional setting.
other sites unchanged. In this case, the numbers of images each site are shown in Table The quantitative results shown in Table 8 The synthetic images are shown in Fig.
20

---

Under review as a conference paper at ICLR 2021
0
9
8
6
4
I
6
O
O
9
@
Avg
GAN MNIST+ MNIST+Fashion Fashion
(b)
Avg GAN MNIST+Font
7
9
3 C 7
D
Q
6
D
9
X F T 0
7
A
O 7
J L 200 3
I
I
O
Y
1 9
O 2 C
S 4
A
G
2
I H A
O Y 5 D 2 E VOED D I 9 S
P
D
4
J
/ O 400
1
7
T
7
4 3
0 J
1
S 9
7
D
6 5
X
CC)
MD GAN MNIST Fashion
(d)
MD GA GAN MNIST+Fom
2
3
3
AD
6
30 O
O 8
6
7
6
P
4
4
B
1
2
4 7
5
5 O
6
0
3
4
O
5
3
2
4
7
5 U
5
M
J
3
U
S
6
E
O T
Y 1
3 4 0
X
6
6
8 9
Q
+
4
O
2
3
4
7
7
Y
G
7
V
O < 5
0
4
S
D
5
O
2
7
O
7
6
7
7
4
9
4
5
8
2
2
2
(c) GAN MNIST+Fashion
(1) GAN MNIST+Font
Figure 9:
Synthetic images on the non-identical MNIST+Fashion dataset ((a)(c),(e) and
MNIST+Font dataset (f)) in unconditional setting.
Do
D1
D2
D3
D4
Dg
Dg
D7
Dg
Dg
MNIST
2917
3393
2944
3032
2939
5421
5918
6265
5851
5949
Fashion
3044
2978
3035
3033
2982
6000
6000
6000
6000
6000
Total
5961
6371
5979
6065
5921
11421
11918
12265
11851
11949
Table 7: The image numbers in each site in the imbalanced setting.
Accuracyt
IS
FID
Real Centralized GAN
0.939
3.580 + 0.039
0
0,886
3,486 = 0.033
10.87
Avg GAN
0.497
3.809 = 0.025
74.45
MD-GAN
0.443
3.877 - 0.034
85.61
UA-GAN
0.846
2.717 = 0.019
30.30
Table 8: The classification accuracy and IS, FID scores on the imbalanced non-i.i.d mixture MNIST + fashionMNIST dataset.
21

---

Under review as a conference paper at ICLR 2021
6
1
2
6
8
4
9
3
2
3
7
2
O
6
6
9
2
J
7
D
7
3
8
8
3
3
8
3
2 6
(a) Avg
GAN MNIST+ + Fashion
(b)
Avg GAN MNIST+Fom
O 2
4
9
3
9
2
6
6
R
S
7
5
9
X
4
O
T
O G
3
S
5
3
O
2
R
9
C
S R
2
O
5
S
4
4
G
L
O
P
I
I
O
I
4
4 G
7
G
3
S
4
O
B
O
Z 3
&
S
A
S
H
2
5
U
9
3 D
3
O
7
2
9
2
6
2
3
P
G
3
2
6
P
7
4
O
S
A
3
7
7
3
3
7
7
O
7
7
8
5
3
O
5
(c) MD-GAN MNIST+ + Fashion
(d)
MD- G AN MNIST+Fom
M
7
3
1
3
D
7
7
3
2
4
B
8
/
2
8
2
9
1
7
I
1
2
6
3
4
5
I 1
3 O I 5
8
O
2
3
F
R
7
4 0
D 7 5 OH 4 O
7 3
J
2
O
J
6
J
2
5
2
3
S & G 7 6 2 O O 2
V 5
4
Y
Z
J
2
5
6
0
O D
/ O O O 2 D
O
5
S .
D
0
3 S 6
3 O
6
3
8
G
3
1
O E
9
O (e) GA-
7
(f) GAN MNIST+Font
Figure 10: Synthetic images on the identical MNIST+Fashion dataset (())) and MNIST+Font dataset (f) in unconditional setting.
(a) Avg-GAN MNIST+Fashion
(b) MD-GAN MFISTna
0
O
O
Ø
9
2
G
2
B
S
0
H/
Y
5
J
5
6
S
S
5
S
7 6
6
6
G
6
6
7 7
7
9
g
R
V
S
?
F
Q
9 1
G
7
9
(c) UA-GAN MNIST+Fashion
Figure 11: Synthetic images on the imbalanced non-identical MNIST+Fashion dataset.
22


Reviews:
[{"review_id": "uHWVYdCzat", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes federated GAN with multiple private discriminators. The authors also analyze the optimality of the proposed federated GAN. The review has several concerns on the current submission:\n1.  Is minimax problem（5)  the total loss of UA-GAN that is optimized by Algorithm 1?  To solve the minimax problem (5),  D_{i} for i=1,2,..., K is coupled!.  Hence，why D_i defined in（2）is the solution of （5）when G is fixed？\n2. When and how does the Nash equilibrium of minimax problem （5）holds？ \n3. Lack of experiments for federated unconditional GAN to verify the theory. \n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QGN9ClnHyO3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n\nThis paper studies gederated Generative Adversarial Networks (federated GANs).\nIn particular, the authors propose a new method, UA-GAN, which is claimed to be better than earlier approaches.\n\nI have several concerns.  First, the writing can be improved significantly.  Second, the statements in the text are often rather vague.  This makes it hard to understand what are the results and to verify their correctness.   E.g., theorems are formulated either vaguely or unprecisely.\n\nMoreover, the theorems refer to \"optimal discriminators\", however there is no guarantee that one can find such an optimal discriminator or that one can even verify whether a discriminator is optimal.  (In fact, the paper doesn't define the term \"optimal\" precisely)   Even if it would be possible to learn an optimal discriminator, then there is no bound on the amount of time this would take (as that depends on various parameters of the learning problem and the learning algorithm).  \n\nThe paper contains more vague statements, e.g., Remark 1 claims the method preserves privacy, but doesn't define what information is kept private.   For example, it seems the size of the private datasets is needed for the central weighting and hence made public.  Moreover, there is no proof that from the information leaving the individual centers nothing about the sensitive data can be inferred.  For example, from the point of view of differential privacy, even if only aggregates are revealed, if their revealed value is exact then probably (epsilon,delta) - differential privacy is not guaranteed.\n\nIn conclusion, it is hard for the average reader to understand the paper due to a lack of precision, and the paper insufficiently specifies definitions, assumptions made, and precise formulations of results.\n\n\n\nSome details:\n\n* The first line of the abstract suggests that GANs are federated networks, while the second line of the abstract correctly states that this is called \"federated GAN\" and the first line of the introduction correctly describes GAN as generating realistic data.\n* Algorithm 1: \"Eq. equation 2.\" -> repetition\n* last line page 3: \"algorithm equation 1\" -> \"Algorithm 1\"\n* top of page 4: the generator G(z) seems to depend on an argument z of which the nature is not revealed immediately.  Later, Equation (2) suggests that z can be drawn from \\mathcal{N}(0,1) and hence is a real number.  It is unclear why the argument of G would be just one-dimensional.\n* top of page 4: while the word \"discriminator\" is used very frequently, no precise definition of this concept is provided, nor an explanation of how the discriminators are obtained.  With the help of Eq (2) some readers may be able to guess that D_j must have values in the open interval (0,1).\n* Definition 1: as the symbol q is already used for distributions, it is preferably to not use it for \"a probability in (0,1)\" too.\n* After definition 1: \"The central idea of UA framework\" -> \"framework\" needs an article\n* just before Section 3.1:  We will provide error bound -> ... an error bound\n* Theorem 1 uses \"Jenson Shannon divergence loss\", which isn't defined in the paper (and no definition is cited).  \"Jenson Shannon divergence\" is a somewhat well-known concept in probability theory, but even for those knowing this it is unclear how to get from it to \"Jenson Shannon divergence loss\".\n* Equation (6) in Theorem 1 seems to give an expression similar to the Jenson-Shannon divergence definition, but doesn't appear a statement the theorem is claiming to be true.  The rest of the sentence refers to q^* and q, but q is a bound variable in Eq (6), i.e., it has no meaning outside the scope of \"argmin_q\", and q^* doesn't occur in the formula (so why do you say \"where q^* = ...\" ?).  It is hence hard to parse the theorem statement and discover what is the claim exactly.\n* A proof is provided in appendix.  However, the proof first says \"To prove the theorem, we first introduce the following Lemma\", the text next states lemma 1, but never returns to the proof of Theorem 1 (as the next title says \"Proof of Theorem 4\").\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "O--dozDicNN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for training GANs in federated learning setups. The federated learning task is to train a centralized generative model using the samples distributed across a group of local nodes in a network. Toward this goal, the paper proposes Universal Aggregation GAN (UA-GAN), in which at every iteration the server communicates a batch of samples produced by the generator to the local nodes and then the local nodes optimize their discriminator using their observed real and received fake samples. Overall, the UA-GAN approach is technically-sound and potentially useful for various distributed learning problems. However, I still have a few comments regarding the paper's theoretical and numerical results. I look forward to the authors' response to my comments.  \n\nMy main concern is on training the discriminator in UA-GAN. According to Algorithm 1, UA-GAN trains K discriminator functions, i.e., one discrminator per local node in the network. In a typical federated learning problem, there might exist hundreds of nodes  where each node observes only tens or hundreds of training samples. In such scenarios, the generalization error of training one discriminator per local node can be very large, since every discriminator is trained using a limited number of real data points. On the other hand, the generalization error seems to be significantly smaller if one uses standard FedAvg for training a single discriminator network.  Currently, there are no theoretical guarantees or numerical analysis on the generalization properties of UA-GAN. I think analyzing the generalization behavior, either theoretically or numerically, is needed to address this comment, especially becuase the paper mostly focuses on the theoretical properties of UA-GAN. \n\nThe paper's numerical expeirments only consider setups with non-identical distributions across nodes. Also, in the experiments K is chosen to be moderately small while the number of training samples at every node is relatively large. I recommend providing some numerical results for larger federated learning settings with smaller training sets. For example, one can consider 100 local nodes with only 500 training samples available at every node. These numbers sound closer to a real federated learning problem than the ones used in the paper's experiments. In addition, lines 9 and 14 in Algorithm 1 should clearly state the optimization procedures for updating the discriminator and generator parameters instead of only referring to the optimization problems. In the current version, it remains unclear how many gradient steps are used to optimize the parameters at every iteration and how the performance of the trained GAN is affected by the number of gradient steps applied for optimizing the discriminator parameters at every communication round.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FT4rZYRlmz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a new method, UA-GAN to train GANs in a federated learning setup. The method simulates a central discriminator D_ua such that the odds values of the central discriminator is equivalent to the weighted sum of the local discriminators. The central generator is then trained based on the simulated central discriminator. The paper provides theoretical analysis on its proposed method and conducts experiments on toy datasets and mixtures of real world datasets to simulate a federated learning setup.\n\nPros:\n- The problem of training GANs in a federated learning setup is an important problem to tackle as it naturally provides a secure way to train a generative model.\n- The idea of simulated central discriminator based on the weighted odds value of the individual discriminators is quite interesting and seems to outperform a simple weighted sum of the discriminator gradients.\n- The results provided in the experimental section are reasonable and outperforms similar GAN setups on non-identical data setups. I also appreciate the inclusion of an accuracy metric in evaluating the different methods as it shows the effectiveness of generated data in training of downstream tasks.\n\nCons:\n- The paper provided a theoretical analysis of the performance of UA-GAN under conditions where some discriminators are trained sub-optimally. It would be nice to also have some experiments in the experimental section that shows the results of this setup i.e with smaller dataset size for some of the discriminators\n- The experimental results will also be more convincing if results of training the UA-GAN on the mixture of all three real world datasets (Font, MNIST and Fashion-MNIST) are shown\n\nOverall, the proposed method of training GANs in Federated Learning setup shows fairly convincing results. With some additional experimental results I believe this will be a good submission for ICLR.\n\n==================\n\nI thank the authors for the additional experiments which have marginally satisfied my initial concerns; ideally, more setup can be experimented. I keep my original rating.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a provable correct framework, namely Universal Aggregation, for training GANs in federated learning scenarios. It aims to address an important problem. The proposed solution is well-grounded with theoretical analysis and promising empirical results. 

The paper receives mixed ratings and therefore there were extensive discussions. One the positive end, some reviewers think that the authors' feedback provide clarification to confusing part of the paper; on the negative side, the authors feedback also confirms some of the concerns raised in the reviews: 

1.  It was confirmed that there is no guarantee that one can find an (nearly) optimal discriminator, which decreases the impact of the work, as in practice we work with non-optimal discriminators and hence some of the results couldn't apply. 

2. It was confirmed that no privacy guarantees can be given. This is concerning since the complexity of GANs won't prohibit skilled attackers from inferring some information. 

While it is true that some of the guarantees would be hard to achieve even for a traditional GAN, the paper sets up a high-expectation at the beginning of the paper,  but fails to satisfy the readers with enough evidence.

In addition, the writing can be significantly improved to ensure precise formulations and consistency; the added experiment results are useful, but stronger empirical results could help alleviate the issues in theoretical results. 

In summary, the paper has built solid foundations for a good piece of work, but the current version could benefit from one more round of revision to become a strong publication in the future. 
Decision: Reject