Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides a new technique, called Folded HMC, to better sampling for the Bayesian treatments of GANs. The effectiveness of the method is demonstrated on several vision datasets, CIFAR-10, SVHN, ImageNet. 
main_review:  
---
Pros:
 
* The paper proposes a new HMC sampling method for Bayesian-based GAN. I find the method novel and effective. 
* The experiments are nice. On the synthetic dataset, the author shows that F-HMC is better than SGHMC when learning Bayesian GAN with high dimensional data. On the natural image dataset, the F-HMC’s effectiveness is further verified. 
 
---
Cons: 
 
* Although the proposed method looks effective, I am not sure about the motivation or insight behind the proposed design. Could the author elaborate on why think of changing SGHMC to F-HMC? What makes F-HMC a better sampler? Does the improvement application-specific? Can F-HMC apply to tasks other than generative modeling? 
 
* Since F-HMC is a general improvement on SGHMC. While ProbGAN improves the learning objective (Posterior formulation) of Bayesian GAN. So I am considering that F-HMC and ProbGAN should be two improvements on the original Bayesian GAN in orthogonal directions. Hence, I suggest the author check the performance of the combination of F-HMC and ProbGAN. Basically, comparing Bayesian GAN + SGHMC v.s. Bayesian GAN + F-HMC (which has been done) and ProbGAN + SGHMC v.s. ProbGAN + F-HMC (which I suggest doing) should give a better picture of the effectiveness of F-HMC. 
* The clarity of the paper could be improved. For example, I am not sure how the equation (8) is derived. In the derivation of equation (11), the author mentions Fokker-Planck Equation without explaining it. It would be nice if the author could articulate more on the background knowledge and equation derivation to make the text more accessible to the general audience. 
 
---
Some typos: 
* Page 5, equation (13) misses “[”.

summary_of_the_review: Overall, I vote for accepting. I like the idea of paralleling the HMC sampling for Bayesian-based GAN methods. My major concern is about the clarity of the paper and some additional ablation (see cons above). Hopefully, the authors can address my concern in the rebuttal period.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposed Folded Hamiltonian Monte Carlo (FHMC) for posterior sampling over the generator and discriminator parameters in Bayesian GAN.
main_review: The paper is not clear, especially for the core part of FHMC formulation.

1. How SGHMC is applied to the S components are not clearly described. I suggest a mathematical formulation instead of using only words.

2. $\mu$ and $\sigma$ are used without any definition beforehand, not sure which parameters they represent.

3. How are these parameters merged for F-HMC?

The theory part is also problematic. The stationary distribution of $V$ (equation 8) should be $\mathcal{N}(0, M)$. $\mathcal{N}(MB^{-1}\nabla U(X), M)$ is the stationary distribution for the second equation in Equation 7. If this is used, the scheme would reduce to SGLD, not SGHMC. Moreover, the proof after that seems to be identical for SGHMC, with a simple dimension decomposition.

The experiments seems to be fine, if done properly. Compared to SGHMC, using F-HMC for Bayes GAN does provide some marginal improvement, not sure if it is statistically significant though. Better provide standard deviations together with the mean estimates.
summary_of_the_review: The main contribution, the FHMC algorithm, is not presented clearly. The same for the theoratical justification. The paper would be strengthened if a formal and more detailed derivation of the proposed methodology is presented.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an extension of BayesGAN (Saatci and Wilson, 2017): BayesGAN learns to sample generator and discriminator parameters from a posterior distribution conditioned on target data using Hamiltonian Monte Carlo sampling; this paper proposes using a modified version dubbed folded-Hamiltonian Monte Carlo to speed up the sampling process and make it scalable for high-dimensional data.
main_review: Overall, the idea of parallelizing HMC is an interesting one. However, I do have some concerns about clarity and the results of the paper

1. Section 4, and particularly 4.1 (which expands on the main contribution of the paper) is confusing and very hard to follow:
- the algorithm uses a "parameter decomposition" to "divide tasks into blocks": how is parameter decomposition carried out? What does it mean to divide updating tasks into blocks? Is this decomposition different for every update of the networks, or is it determined before training begins?
- the components are then "run in parallel" -- Figure 1 implies that this HMC run simultaneously on different subsets of data dimensions, but Algorithm 1 implies that the subsets are for parameters.
- HMC fit a distribution with "Gaussian likelihoods" -- Equation (1) states that the likelihood is proportional to the exponential of the GAN loss function: does the algorithm use a Gaussian approximation for this likelihood?
- Finally, the "$\mu$s and $\sigma$s are given [...] cross-correlation between all components" -- it is very hard to understand what the algorithm does from this sentence.

Overall, from the explanation in section 4.1, it is quite difficult to figure out what exactly the algorithm does, and therefore very hard to judge its merits and demerits.


2. The theoretical analysis in section 4.2 looks at whether HMC performed in parallel on subsets of data dimensions converges to the same distribution as performing HMC on the full data. However, it is very hard to see what the different parts of the proof achieve, or how they are connected to HMC for the Bayesian GAN formulation without some more explanation. For example, it is very hard to follow the logic for deriving Equations 11-14, when all the meat in proving convergence seems to lie in the inline equation at the very end of section 4.2


3. The scores for F-HMC GAN are not better than BayesGAN or probGAN across the board on all applications presented in the paper, and are sometimes only marginally better:  e.g IS scores on CIFAR-10 and ImageNet in Table 2 differ in the 3rd significant digit, the JSD traces for BayesGAN and  FHMC GAN are quite similar and seem to converge to the the same value in Figure 5, etc. Although this does not automatically disqualify the use of FHMC-GAN since it certainly has an advantage in terms of lower runtime, it seems also a bit of an exaggeration to claim the "superiority of FHMC [...] in generating data"


### Minor comments:
- there were numerous typos throughout the manuscript: it would be good to have a thorough edit
- In Eqn 1, LHS ia posterior conditioned on $\alpha_d$ but RHS has a likelihood dependent on $\hat{\alpha}_d$
- The notation in section 3 is confusing: previously $x$ was used to denote target data, and $X$ as the set of all $x$
- The theorem statement in section 4.2 refers to the "dynamics described in Eqn 9" -- should this be Eqn 7?
- In Table 1, the score for probGAN with 4000 samples from ImageNet has the lowest value (25.8) and should be in bold.
summary_of_the_review: It was hard to judge the merits of the algorithm and the contributions of the paper due to a lack of clarity in the explanations, and (it seemed to me) an exaggeration of the results.
I would be happy to re-evaluate if the authors could provide a clearer explanation of the method and results.
review_title: null

Meta-review: The paper proposes a method to sample the parameters of the generator and discriminator in a BayesGAN (Saatci and Wilson, 2017) setting. The main innovation is a modified Hamiltonian Monte Carlo sampling scheme. Unfortunately the method is not clearly presented, to the point that all reviewers had difficulties understanding how the method works. The revision is making progress but still does not clearly explain the method. While the paper cannot be accepted for publication in its present form, the experimental results are encouraging so I encourage the authors to keep improving their manuscript.
Decision: Reject