EP-GAN: Unsupervised Federated Learning with Expectation-Propagation Prior GAN

Abstract: Generative Adversarial Networks (GANs) are overwhelming in unsupervised learning tasks due to their expressive power in modeling fine-grained data distributions. However, it is challenging for GANs to model distributions of separate non-i.i.d. data partitions as it usually adopts an over-general prior, limiting its capability in capturing the latent structure of multiple data partitions and thus leading to mode collapse. In this paper, we present a new Bayesian GAN, dubbed expectation propagation prior GAN (EP-GAN), which addresses the above challenge of modeling non-i.i.d. federated data through imposing a partition-invariant prior distribution on a Bayesian GAN. Furthermore, unlike most existing algorithms for deep-learning-based EP inference that require numerical quadrature, here we propose a closed-form solution for each update step of EP, leading to a more efficient solution for federated data modeling. Experiments on both synthetic extremely non-i.i.d. image data partitions and realistic non-i.i.d. speech recognition tasks demonstrate that our framework effectively alleviates the performance deterioration caused by non-i.i.d. data. 

Reviews and author responses:

 
4. **“ ... the performance of the baseline model BayesGAN (2) was actually not reported in Table 1.”**

Actually, we did test the performance of both BayesGAN and ProbGAN on CIFAR10 in our preliminary experiments. We find that  ProbGAN outperforms BayesGAN in many aspects in terms of training stability and image generation performance. As such, we only include ProbGAN, the best performing baseline, in Table 1.

5. **“... the Table 1 did not consider the top performing case. Also, only having one natural image dataset is probably not enough ... ”**

Although the NS objective performs best in the ProbGAN paper, in our experiments it was relatively unstable, especially in the federated setting, which was never tested in the ProbGAN paper. In our preliminary experiments, when training with an NS objective with several random seeds, the variance of the results was large: the peak performance was slightly better than W and LS objectives, while the average performance was worse. In contrast, W and LS had more stable and satisfying performances. Therefore, we selected these two objectives as our baseline configurations. STL-10 only has 500 labeled samples for training and the rest of 100000 images are unlabeled. In our non-i.i.d. setting, we rely on the image labels to construct the synthetic cross-silo datasets. Therefore, STL-10 was not selected as our dataset. It is worth noting that we perform preliminary studies on toy and semi-synthetic image datasets. For large-scale real-world scenarios, we evaluated our proposed method on CHiME 5, one of the largest real-world conversational speech datasets with a natural cross-silo data split. 

6. **“ … EP-GAN variants on i.i.d. N=2 outperform oracle. What is the potential reason behind?”**

This is an interesting observation. The improvement might potentially come from two aspects: first, the EP-prior of EP-GAN provides regularization of models during training, improving the generalization ability of local models; second, the aggregation of two local models performs the role of the model ensemble, resulting in a more powerful global model. Compared to i.i.d. N=2, the global model in the setting of i.i.d N=10 is more seriously affected by the client drift (Want et al., 2021) (Note that $N$ represents the number of clients), overwhelming the improvement brought by the EP-prior/ensembling and therefore leading to worse performance. 

### Reference 

[1] Chong, Min Jin, and David Forsyth. "Effectively unbiased fid and inception score and where to find them." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Wang, Jianyu, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Maruan Al-Shedivat, Galen Andrew et al. "A field guide to federated optimization." arXiv preprint arXiv:2107.06917 (2021).

[3] Daxberger, Erik, et al. "Bayesian deep learning via subnetwork inference." International Conference on Machine Learning. PMLR, 2021.

[4] Miok, Kristian, et al. "To ban or not to ban: Bayesian attention networks for reliable hate speech detection." Cognitive Computation (2021): 1-19.

[5] Pasi Jylanki, Aapo Nummenmaa, and Aki Vehtari. Expectation propagation for neural networks with ¨ sparsity-promoting priors. Journal of Machine Learning Research, 15(54):1849–1901, 2014. URL http://jmlr.org/papers/v15/jylanki14a.html.

[6] Thomas P Minka. Expectation propagation for approximate bayesian inference. arXiv preprint arXiv:1301.2294, 2013.  

[7] Kang, Minguk, et al. "Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training." Advances in Neural Information Processing Systems 34 (2021).


 1. **“ … the scalability issue of Bayesian GANs can potentially hinder the use of EP-GAN ”**

Our work is orthogonal to the vast body of large-scale non-Bayesian GANs with state-of-the-art performance. This is because given any non-Bayesian GANs, one could always improve upon it with a Bayesian counterpart. In this work, as a proof-of-concept, we follow recent Bayesian GAN papers (e.g., ProbGAN at ICLR’19) to focus on a single backbone network and perform thorough studies. We expect similar results if we replace the backbone with larger-scale networks. 

As for the scalability issue of Bayesian GANs, it is still an open problem in the ML community.  Further study of this issue is interesting future work, but it is out of the scope of this paper.

Though solving the scalability issue of Bayesian GANs is not the focus of this paper, we understand the reviewer's concern about such a shortcoming.  We, therefore, are currently validating our EP-GAN with ReACGAN (Kang et al., 2021), the latest large-scale SOTA GAN. Due to the computation complexity and hardware resource constraints, it needs more time to run experiments, and we will update the results as soon as they are available.  

As the reviewer mentioned, Bayesian inference is a powerful framework, and Bayesian GANs are advantageous over their non-Bayesian counterparts in terms of robustness, generalization, and uncertainty quantification. Furthermore, Bayesian frameworks are usually theoretically grounded, which allows us to take advantage of the well-studied techniques of probability theory, e.g. expectation propagation (EP). It is these advantages that drive us to select ProbGAN, the best performing Bayesian GANs,  as our baseline for solving the mode collapse problem under non-i.i.d federated learning settings.  

2. **“ … eq. (5) lacks a justification on why it suffices the quality of likelihood modelling ... ” , “... fidelity of EP update...”**

First, the likelihood in EP refers to the likelihood contribution of a data; note that this is not the usual likelihood which describes the probability of observing a collection; since such a likelihood contribution can be treated as a probability, we use Sigmoid to ensure its value to be within the range of [0,1]. Practically, we adopted an auxiliary neural network $f(x_{k,i})$  taking as input the observation $x_{k,i}$ to assist in approximating the true likelihood contribution of $x_{k,i}$; intuitively, the auxiliary neural network f should be expressive enough to account for the likelihood contribution of each data point. Therefore, in our implementation, $f$ shared the backbone with the discriminator, which is trained to capture representation from data. 

Similar to the EP family (Jylanki et al., 2014, Thomas, 2013), when the likelihood term has a complex parametric form, the assessment of the approximation error is non-trivial and still an open question. Empirically, our numerical studies of learning mixture of Gaussians demonstrate that $f$ successfully helps train GANs under non-i.i.d federated learning settings (Figure 1).

Once the likelihood term is ready, we then follow Theorem 4.1 and Lemma 4.1 to analytically approximate the moments of the hybrid/tilted distribution. Notice that approximation comes from Lemma 4.1, where we use the probit function Φ (ζa(C+b)) to approximate $\sigma_k(C)$. Similar approximations are also adopted by Wang et al. (2016) and Wang & Manning (2013). Both the numerical and theoretical approximation analysis has been well studied in Wang et al. (2016).


3. **“  For EP-ProbGAN, how does the newly introduced EP prior affect the guarantee claimed by ProbGAN.”**

In EP-ProbGAN,  the prior is factorized by a Gaussian prior inferred by EP and a ProbGAN prior. As the convergence of ProbGAN is theoretically guaranteed,  the convergence of EP-ProbGAN requires the convergence of  EP.   Even though the convergence issue is still a  concern in the ML community, EP works well in handling various real-world tasks that require Bayesian inference.  This is because EP derives from loopy belief propagation (BP), which also may diverge. 

To empirically show the influence of EP prior in EP-ProbGAN, we evaluated EP-ProbGAN under the centralized setting and achieved marginal improvements compared to ProbGAN for CIFAR10, as shown in Table 1 of the revised paper. This verifies the effectiveness of the EP prior.



 1. **“... how and where do you ‘identify the mode collapse problem of Bayesian GANs... ‘”**

We evaluate WGAN, ProbGAN, and our EP-ProbGAN on a toy cross-silo non-i.i.d. dataset, where data are generated by a mixture of four Gaussian distributions (Section 1 and Section 5.1). Our task is to learn such a mixture of Gaussians. The results are reflected in Fig.1, which shows the learned density maps of all GANs.  *We find that both WGAN and ProbGAN (a variant of Bayesian GAN) suffer from severe mode collapse even after FedAvg when handling the non-iid data in federated learning*. In contrast, our EP-ProbGAN successfully captured all four modes.  We are the first to identify such an issue for Bayesian GAN under the federated learning setting and propose to tackle this problem by introducing EP-priors for Bayesian GANs. 

2. **“... What's the novelty over existing methods for training GAN under federated learning settings? e.g., [1-2] ” , “ … comparisons with existing methods  [1-2].”**

Actually, FedGAN [1] has been included in the discussion of our related work (Section 2). Besides, we did try our best to re-implement FedGAN ( as [1] does not provide open-source code ) and include it as a baseline of our experiment (Section 5.2, Vanilla GAN (federated) in Table 1 ).  The major focus of FedGAN is to improve the communication efficiency under the non-i.i.d federated learning setting. Such a model ignores another critical challenge: mode collapse of federated GANs. Addressing this problem is one of the main focuses of our model.

UA-GAN [2] is a recently submitted non-peer-reviewed preprint at Arxiv. We are sorry that we did not notice this work even with a very careful and comprehensive survey at the early stage of this research. Thanks to the reviewer’s comment, we analyze UA-GAN in the following discussion. We first attempt to implement UA-GAN in our problem settings. After checking their detailed implementation, we find that UA-GAN may not be a proper solution for federated learning settings. This is because their method is designed for **non-federated distributed** machine learning settings where the communication between the server and clients is in high frequency. However, in **federated learning** the number of communication rounds is much smaller [3]. We have added a similar discussion in the revised paper.  

3. **“ … q(θ) is not a distribution, i.e., it's not normalized, right? Similar questions for the following qept(θG).”**

In our paper, we assume the approximate factor of the EP prior has a Gaussian pdf. Having the form of the product of Gaussian pdfs means that both $q(\theta)$ and $q_{ep}^{(t)}(\theta_G)$  still have Gaussian pdfs ([4] and Section 4 in [5]), whose normalization constant is 1. Therefore, both $q(\theta)$ and $q_{ep}^{(t)}(\theta_G)$ are still valid distributions.

4. **“ In Eq 5, how will you train the auxiliary neural network f … How expensive is the proposed method, both in space and time?  ”**

The f-function in Eq. 5 is implemented with a backbone and an output module, where the backbone is shared with the discriminator and the output module is a two-layer neural network with the hidden size of 128.  Such a neural network is trained with FedAvg.
For computational efficiency, each θ parameter of the GAN generator shares the same f-function (tuning the number of independent f-functions can further improve the performance).  Therefore, the space overhead compared to ProbGAN is only the output module. Similarly, the time complexity is O(N) where N is the number of clients, as introduced in Section 4.1. 

5. **“ The notations starting from Eq 5 are quite confusing. Eqs 6-7 are not easy to follow. ”**

Due to the space limitation, we have to omit or simplify some explanations about the notations in Eqs 5-7. We are sorry for the confusion caused. We have simplified Section 4.2 while leaving the full details to Appendix, such that we have enough space to add more explanations to Eqs 5-7.   We have revised the paper to make it clear.  

6. **“ In the paragraph following Eq 14, I cannot see clearly why ``Theorem 4.1 shows that we are able to analytically approximate the prior of the global data distribution ... ”**

We have revised the paper to clarify this statement. 

7. **“The notation JG is not defined in Eq 16. ”**

$J_G$ denotes the objective functions of generators given the discriminator. 

### Reference

[1] FedGAN: Federated Generative Adversarial Networks for Distributed Data 

[2] Training Federated GANs with Theoretical Guarantees: A Universal Aggregation Approach\

[3] McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." PMLR, 2017.

[4] Williams, C. K., Agakov, F. V., & Felderhof, S. N. (2001, January). Products of gaussians. In NIPS (pp. 1017-1024).

[5] Wang, H., Chen, B., & Li, W. J. (2013, June). Collaborative topic regression with social regularization for tag recommendation. IJCAI


 6. **“ I didn't quite follow the need for a sigmoid in Eq. 5. "**

The likelihood in EP refers to the likelihood contribution of a data; note that this is not the usual likelihood which describes the probability of observing a collection; therefore we use sigmoid to ensure its value to be within the range of [0,1]. On the other hand, the sigmoid and auxiliary neural network f offers a nonlinear map from the data point to its likelihood contribution. 

7. **“Is this method generic enough to be applied for other Bayesian GAN settings beyond the ones considered here?  ”**

Yes, we believe our method should be generic enough to other Bayesian GANs with moderate adaptation. The intrinsic motivation of our method is to enhance the prior of Bayesian NNs under non-i.i.d cross-silo data. Our method provides an efficient way to compute the EP-prior for Bayesian NNs. In our paper, we adapted our method to two different Bayesian GANs: BayesianGAN, and ProbGAN. We believe our method should be applicable to other Bayesian GANs as well; the key is to identify the priors of the Bayesian GANs and augment them with our EP-prior.

8. **“ can this EP-prior help in the case of i.i.d data”,  “why should the proposed method aid in handing i.i.d data”**

From Table 1, we find that our EP-GANs consistently outperform ProbGAN (federated) in terms of Inception Scores, when they are trained with WGAN objectives. This suggests that EP-prior might help in handling i.i.d data as well.  This is possibly due to the better generalization of EP-GANs.  Notice that the data assigned to each client is limited for CIFAR10. This could result in the overfitting of the local ProbGANs. While ProbGAN (federated) adopts FedAvg to deterministically calculate a model average over local ProbGANs, EP-GAN uses latent variables of EP to capture the uncertainty of each local data distribution, leading to better generalization to unseen data.  

9. **“It would be good to include an i.i.d case too in Table 1, if possible.”**

We agree that including an i.i.d. case is useful. In fact, we do have results for the i.i.d cases in Table 1 (see the first two rows of each block). 

10. **“ More baselines could be added for the speech experiment. In addition, the task is not well defined in section 5.3 ”**

The CHiME5 dataset is the first large-scale corpus of real conversational speech in everyday home environments, which naturally contains multiple non-i.i.d conversational sessions of different speakers under different environments. Therefore, we adapted this dataset to our non-i.i.d federated learning settings and demonstrate the effectiveness to use our EP-GAN for unsupervised federated representation learning. It is worth noting that there are very few methods on unsupervised speech representation learning under the non-i.i.d federated learning setting. We have improved the definition of the task in section 5.3 in the revised paper.

### Reference

[1] Wang, Hao, Xingjian Shi, and Dit-Yan Yeung. "Natural-parameter networks: A class of probabilistic neural networks." Advances in Neural Information Processing Systems 29 (2016): 118-126.

 1. **“... it would be good to supplement figures with some quantifiable metrics …”**

This is a good point. According to your suggestion, we supplement the figures generated from oracle settings and the final figures generated from federated settings with D-MSE scores. The D-MSE score is defined as the MSE between two density maps, reflecting the similarity of the density estimation given by GANs and the ground-truth GMM. We uniformly sample 10000 points from a square area from (-5,-5) to (5,5). Then the D-MSE is calculated as $\frac{1}{10000}\sum_{x_i}(f_{gan}(x_i) - f_{gmm}(x_i))^2$, where $x_i$ is a sample from 10000 points. With the quantifiable metric, we could clearly observe the gains of our EP-GANs in the federated setting. This discussion has been added to the revised paper in Figure. 1 and Appendix F.2.2. 

2. **“ In the Related work section, the two large paragraphs on Federated Learning and Bayesian GANs seem to be disconnected… ”**

Thank you for the suggestion. We have added the following connecting paragraph in the revised paper:

In this work, we develop a new Bayesian GAN equipped with an expectation propagation (EP) prior for unsupervised federated learning under non-i.i.d settings; therefore, recent studies on federated learning, Bayesian GANs and EP with neural networks are reviewed in this section. 

3. **“The small paragraph on Federated Learning and deferring it to the appendix is not a good idea…”**

Thank you for your suggestion. Due to the tight restriction of 9 pages for the submission, we tried to cite all relevant references in the main paper with a brief discussion and defer detailed discussion to the appendix. We will address this issue in the camera-ready version.

4. **“ … why can't the approximated factor come from a non-exponential family? ”**

 If the approximate factors come from a non-exponential family,   the KL divergence in Step 3 of Section 3 may not have an analytical form, making it difficult to update the approximate factors.

5. **“ … why are Gaussian distributions used for approximate factors? What bearing does this have on the entire method? “**

Theorem 4.1 is derived based on Lemma 4.1 and Lemma B.1 (Stein’s Lemma), which require the approximate factors to follow the Gaussian distribution. Such Gaussianity allows us to have closed-form EP updates for Bayesian GANs under the federated learning settings.




 1. **“... whether the calculation in EP is dominant in the total framework ...”**

Thank you for the suggestion. The calculation of EP prior in EP-GANs is not dominant, but only thanks to our proposed closed-form update that improves the computation efficiency. To empirically confirm this, we kept a similar number of parameters of  ProbGAN and EP-ProbGAN and trained them on CIFAR-10. Both models took around 25 epochs during training. The average running time of ProbGAN and EP-ProbGAN are 4.3s and 5.9s per epoch, respectively; this shows that with the proposed closed-form update, our EP-ProbGAN can run almost as fast as ProbGAN, demonstrating its validity. We have added a similar discussion in Appendix 5.2 in the revised paper.   

2. **“... it can be easily improved by introducing client-specific parameters using a conditional module … The comparison with such a baseline is interesting.”**

Thank you for sharing the interesting intuition to use a client-specific architecture to guide local training. In fact, we did conduct a similar preliminary experiment by splitting the model parameters into the client-specific and client-invariant parts. The client-specific layers are normalized with a client-wise conditional BN layer. However, without knowing information about other clients, the prediction of gamma and beta (learnable parameters of BN) is unstable during federated learning. Therefore,  it did not work well. The major challenge is to capture qualified client embeddings under extremely non-i.i.d unlabeled cross-silo data from a view of global data distribution, as well as to avoid mode collapse. These observations motivated us to leverage EP to help capture the latent structure from non-i.i.d. distributions. Our experimental results on synthetic extremely non-i.i.d. cross-silo data as well as real-world image and speech datasets demonstrate the effectiveness of our proposed method.

3. **“... the number of clients is relatively small  (the order of 10) …”**

The order of 10 for the number of clients is commonly adopted by recent non-iid federated learning studies [1,2,3,4]. Though there exist a few studies that use millions of clients, their datasets are mostly unavailable to the public and thus suffer the issue of reproducibility.   

4. **“... significance test on the FID is not presented ...”**

Thank you for your suggestion, we have reported the statistical significance test results for FID in the revised paper.  

### Reference

[1] Lin, T., Kong, L., Stich, S. U., & Jaggi, M. (2020). Ensemble Distillation for Robust Model Fusion in Federated Learning. NeurIPS 2020.

[2] Li, Q., Diao, Y., Chen, Q., & He, B. (2021). Federated learning on non-iid data silos: An experimental study. arXiv preprint arXiv:2102.02079.

[3] Liu, Y., Kang, Y., Xing, C., Chen, T., Yang, Q., 2020b. A secure federated transfer learning framework. IEEE Intelligent Systems 35, 70–82.

[4] Li, D., Wang, J., 2019. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581


This paper aims to construct a GAN that can be applied to non-i.i.d. federated data. To achieve this aim, the authors propose an extension of Bayesian GAN called expectation propagation prior GAN (EP-GAN), which obtains a partition-invariant prior using expectation propagation. In particular, the authors introduce a closed-form solution for efficiency. The effectiveness of the proposed method is demonstrated using non-i.i.d data, including the toy data, image data, and speech data. [Strengths]
1. As far as I know, applying a GAN to non-i.i.d federated data is not actively studied in the previous studies and is an interesting research topic.

2. The proposed method is solid and mathematically grounded. The detailed derivatives are also provided in the supplementary materials.

3. The effectiveness of the proposed method is demonstrated using various data, including toy data, image data, and speech data.

[Weaknesses]
1. Improving the calculation speed of EP is included in the main contribution. However, its validity is not empirically demonstrated. At the bottom of Section 4.1, the difference in the calculation order is discussed; however, I cannot understand how significant the difference is in practice. In particular, I am curious whether the calculation in EP is dominant in the total framework, including the GAN training.

2. It seems that in the baseline models, simple parameter averaging is applied to all the layers when FedAvg is used. However, it can be easily improved by introducing client-specific parameters using a conditional module (e.g., conditional batch normalization), which is used in a typical conditional GAN. The comparison with such a baseline is interesting.

3. In practice, it is assumed that the number of clients is considerably large. However, in the experiments, the number of clients is relatively small (the order of 10). Therefore, in the current manuscript, I consider that the effectiveness in a practical setting is not sufficiently demonstrated, although some benchmark performance is provided.

4. In Appendix F.3.4, a statistical significance test on the inception score is provided; however, a statistical significance test on the FID is not presented in the current manuscript. If the authors intend to emphasize the utility of Mixture-EP-ProbGAN, this test also should be conducted. This paper addresses an interesting problem and proposes a reliable method that is mathematically grounded. However, I still have some questions regarding the experimental evaluation. I expect that the authors clarify them in the rebuttal.

This paper proposes a method for learning Generated Adversarial Networks (GANs) for non-i.i.d data in a federated learning setting. This is accomplished with the use of a partition-aware prior via an Expectation Propagation (EP) algorithm embedded into a Bayesian GAN setting. Additionally, it proposes a closed-form solution for EP updates aiding efficient federated learning. The claims are substantiated with experiments on both synthetic and real data. 



 



 The paper is well written and the use of EP prior to this purpose is a fine idea. However, I would like the Authors to address the following questions. 

1. While Figure 1 is a good way to motivate the problem, it would be good to supplement all figures with some quantifiable metrics (at least for a few ones). It is difficult otherwise to ascertain the gains. 

2. In the Related work section, the two large paragraphs on Federated Learning and Bayesian GANs seem to be disconnected. It would be good to have a connecting paragraph would be good. 

3. The small paragraph on Federated Learning and deferring it to the appendix is not a good idea. Some details on Fed-Learning here will be apt. 

4. I am not an expert on EP but I am curious - why can't the approximated factor come from a non-exponential family?

5. Related to the previous question, why are Gaussian distributions used for approximate factors? What bearing does this have on the entire method? 

6. I didn't quite follow the need for a sigmoid in Eq. 5.

7. Is this method generic enough to be applied for other Bayesian GAN settings beyond the ones considered here?

8. The other related question is, can this EP-prior help in the case of i.i.d data as well?

9. The other important question I have is - It is not clear from the paper why should the proposed method aid in handing i.i.d data in the federated learning setting. There is empirical evidence but can there be a more principled way of describing the same?

10. It would be good to include an i.i.d case too in Table 1, if possible. 

11. More baselines could be added for the speech experiment. In addition, the task is not well defined in section 5.3


 This paper addresses a very useful problem of federated GAN learning for non-i.i.d data. It is moderately novel to be considered for publication in ICLR. 

The goal of this paper is to train a Bayesian GAN on non-i.i.d. federated data. Specifically, the authors propose to adopt the newly-introduced expectation propagation (EP) prior, being partition-invariant, to address the non-i.i.d. federated challenge. Experiments on synthetic and real datasets are conducted. 



 The writing should be improved significantly; the current manuscript is quite challenging to understand what's really going on.

In the Introduction, how and where do you ``identify the mode collapse problem of Bayesian GANs under non-i.i.d. cross-silo unsupervised
learning scenarios?''

What's the novelty over existing methods for training GAN under federated learning settings? e.g., [1-2]

In the paragraph before Eq 1, $q(\boldsymbol \theta)$ is not a distribution, i.e., it's not normalized, right? Similar questions for the following  $q_{ep}^{t}(\boldsymbol \theta_G)$.

In Eq 5, how will you train the auxiliary neural network $f$ in federated learning settings? 

Also In Eq 5, it seems one should specify an $f$-function for each $\theta$ parameter of the GAN generator? How expensive is the proposed method, both in space and time?

The notations starting from Eq 5 are quite confusing. Eqs 6-7 are not easy to follow. 

In the paragraph following Eq 14, I cannot see clearly why ``Theorem 4.1 shows that we are able to analytically approximate the prior of the global data distribution with the datasets stored on different clients while following the cross-silo federated learning settings?''

The notation $J_G$ is not defined in Eq 16.

[1] FedGAN: Federated Generative Adversarial Networks for Distributed Data
[2] Training Federated GANs with Theoretical Guarantees: A Universal Aggregation Approach The writing should be improved significantly; the current manuscript is quite challenging to understand.
The contents in Section 4.1, i.e., the main novelties, are challenging to follow.  
Important comparisons with existing federated-GAN training methods are believed missing.

The authors targeted federated generative modelling in an unsupervised setting. Specifically, the work is built on top of Bayesian GANs. In order to aggregate the information from different clients, the authors proposed to use expectation propagation (EP). It makes sense. Despite being a well established Bayesian inference algorithm, EP operating on the neural network parameters can suffer from intractability. The authors presented a low complexity solution. The experiment results showed improved FID and IS over multiple baseline methods. However, the overall performance is quite poor on the rather simple dataset CIFAR10, owing to the scalability issue of Bayesian models. Federated learning with non-i.i.d. data partitions is particular challenging, as naive averaging does not work well. EP offers an information aggregation framework which can deal with different data partition styles in a unified way. In order to apply EP, it requires to have Bayesian GAN as the basis model at each client. While Bayesian inference is a powerful framework, the scalability issue of Bayesian GANs can potentially hinder the use of EP-GAN. For instance, the oracle baseline model still has FID on CIFAR10 above 25 while the best performing GAN on CIFAR10 is below 6. Moreover, the gap can potentially become even larger when the resolution is higher.

On the algorithm side, eq. (5) lacks a justification on why it suffices the quality of likelihood modelling besides being simple and thus permitting the closed-form EP update. For EP-ProbGAN, how does the newly introduced EP prior affect the guarantee claimed by ProbGAN.

In the experiment part, despite being introduced in the text, the performance of the baseline model BayesGAN (2) was actually not reported in Table 1. Furthermore, as reported by the authors of ProbGAN, the NS loss outperformed Wasserstein distance and LS for both ProbGAN and Bayesian GAN. However, the Table 1 did not consider the top performing case. Also, only having one natural image dataset is probably not enough, e.g., ProbGAN also considered STL-10 and ImageNet. Furthermore, EP-GAN variants on i.i.d. N=2 outperform oracle. What is the potential reason behind? Overall, the problem is trending, challenging and highly relevant. Within the Bayesian framework, the use of EP for federated learning is reasonable. Of course, complexity remains as a critical issue. The authors proposed some low-complexity solution and empirically showed the benefits of using EP over existing schemes, which however are not really developed for non-i.i.d. scenarios. Furthermore, the baseline models do not take the top performance configuration, which lead to my general concern on how strong the baselines are. On the algorithm side, Bayesian models need to deal with priors regardless of EP. Therefore, having or adding an EP prior in Bayesian models seems to be straightforward. The closed-form update is definitely interesting, but the authors shall (empirically) analyse its fidelity.

Meta-review: This paper presents a Bayesian GAN approach designed for a federated learning setting. In contrast to recent Bayesian GAN approaches that use Gaussian priors or iteratively-updated priors on GAN parameters, this paper proposes a more complex prior motivated by expectation propagation, dubbed as EP-GAN, and uses this formulation to construct a federated GAN. The paper claims that this prior better captures the multimodal distribution structure of the non-iid heterogeneous data across the different clients.

The paper looks at an interesting problem, i.e., federated training of GANs, which is indeed a problem that has received a lot of interest lately. The paper received mixed reviews. The reviewers raised several concerns, some of which included (1) weak baselines, (2) not considering what happens when we switch to more advanced GAN models, (3) performance of the approach when the number of clients is large, and (4) lack of clarity in the presentation. 

The authors responded to some of these concerns and it is commendable that they reported some additional results during the discussion phase. However, after an extensive discussion among the reviewers and between reviewers and authors, and after my own reading of the manuscript, concerns still lingers over many of the above-mentioned points. Another concern is the overly complex nature of the approach as compared to other recent federated GAN approaches which raises the question as to whether the actual improvements warrant the complexity of the proposed approach. From the report experiments, the improvements appear to be rather slim.

Considering these aspects, unfortunately, the paper in its current shape does not seem ready for acceptance. The authors are advised to consider the feedback from the reviewers which will strengthen the submission for a future submission.

Decision: Reject