Folded Hamiltonian Monte Carlo for Bayesian Generative Adversarial Networks
Authors: Narges Pourshahrokhi
Samaneh Kouchaki
Yunpeng Li
Payam M. Barnaghi
Venue: ICLR 2022 Submitted (2022)

Abstract: Generative Adversarial Networks (GANs) can learn complex distributions over images, audio, and data that are difficult to model. We deploy a Bayesian formulation for unsupervised and semi-supervised GAN learning. We propose Folded Hamiltonian Monte Carlo (F-HMC) within this framework to marginalise the weights of the generators and discriminators. The resulting approach improves the performance by having suitable entropy in generated candidates for generator and discriminators' weights. Our proposed model efficiently approximates the high dimensional data due to its parallel composition, increases the accuracy of generated samples and generates interpretable and diverse candidate samples. We have presented the analytical formulation as well as the mathematical proof of the F-HMC. The performance of our model in terms of autocorrelation of generated samples on converging to a high dimensional multi-modal dataset exhibits the effectiveness of the proposed solution. Experimental results on high-dimensional synthetic multi-modal data and natural image benchmarks, including CIFAR-10, SVHN and ImageNet, show that F-HMC outperforms the state-of-the-art methods in terms of test error rates, runtimes per epoch, inception score and Frechet Inception Distance scores.

Under review as a conference paper at ICLR 2022
FOLDED HAMILTONIAN MONTE BAYESIAN GENERATIVE ADVERSARIAL NETWORKS
CARLO FOR
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) can learn complex distributions over images, audio, and data that are difficult to model. We deploy Bayesian formu- lation for unsupervised and semi-supervised GAN learning. We propose Folded Hamiltonian Monte Carlo (F-HMC) within this framework to marginalise the weights of the generators and discriminators. The resulting approach improves the performance by having suitable entropy in generated candidates for genera- tor and discriminators' weights. Our proposed model efficiently approximates the high dimensional data due to its parallel composition, increases the accuracy generated samples and generates interpretable and diverse candidate samples. We have presented the analytical formulation as well as the mathematical proof of the F-HMC. The performance of our model in terms of autocorrelation of generated samples on converging to a high dimensional multi-modal dataset exhibits the ef- fectiveness of the proposed solution. Experimental results on -igeeniimnsi synthetic multi-modal data and natural image benchmarks, including CIFAR- 10, SVHN and ImageNet, show that F-HMC outperforms the state-of-thearr meth- ods in terms of test error rates, runtimes per epoch, inception score and Frechet Inception Distance scores.
INTRODUCTION
Generative adversarial networks (GANs) [Goodfellow et al. (2014)) have received traction in the field of deep generative models. The development GANs covers a wide range from multi-layer perceptrons the BigGAN framework [Brock et al. (2019)) with residual blocks and self-attention layers (Zhang et al. (2019)) to synthesise realistic images. Despite GAN's effectiveness in generat- ing realistic images, it experiences mode collapse, which occurs when the generator over-optimises for a particular discriminator and the discriminator never learns how to escape the trap. Recent work has focused on alternative metrics such f-diversities [Nowozin al. (2016)) or Wasserstein diver- gences [Arjovsky et al. (2017)) to substitute the Jensen- divergence inherent in traditional GAN training to alleviating several practical issues.
Saatci & Wilson (2017) recently proposed Bayesian GAN, a probabilistic framework for GANs based on Bayesian inference. demonstrates how modelling the distribution of generators alleviates mode collapse and motivates the interpretability of learned generators. GAN training measures the full posterior distribution across network weights a single-mode based on mini-max optimisation. Even if the generator does not recall training instances, samples from the generator are expected to be excessively compact compared with data distribution samples. fact, as demonstrated by Bayesian GAN, a posterior distribution over the generators' parameters can be vast and highly multi-modal to model the real data distribution by fully reflecting the posterior distribution over the generator and discriminator parameters. In addition. He et al. (2019) proposed the probGAN, which is similar to the Bayesian GAN, iteratively learns a distribution over generators but with a carefully crafted prior. Learning is triggered by tailored Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) to perform Bayesian inference.
In both Bayesian GAN and ProbGAN, SGHMC is deployed to marginalised the parameters of the generator and discriminators. The foundation of our idea in this paper is to utilise different samples for generator parameters in order to mitigate GAN collapse mode and more efficiently produce data samples with an adequate degree of entropy, especially with high nigh-diimnsional and highly correlated

---

Under review as a conference paper at ICLR 2022
data. To do so, propose Folded Hamiltonian Monte Carlo (F-HMC) replace the SGHMC part of the Bayesian framework. This proposal shares several desired properties with SGHMC, such 1) being experimentally well-adjusted in training the GAN due to its Hamiltonian dynamics and 2) directly importing parameters such as the learning rate from gradient descent into the sampler. Furthermore, it benefits from the following advantages:
1. F-HMC explores more accurately the target density, especially in the scenario with high- dimensional and highly correlated data.
2. F-HMC converges faster to the target density in terms of lag number.
3. F-HMC provides the practical advantage to the Bayesian GAN method by exploring a rich muulti-moda distribution over the weight parameters of generators an acceptable entropy level.
4. More importantly, because the parallel composition, F-HMC has an efficient run time in nigh-dimensional data approximation.
Our main contributions are listed below. We will return to them in the experiment section to highlight each of them.
1. We propose F-HMC to sample parameters of generators to create candidates from the multi- modal nigh-dimensional distribution.
2. We have mathematically verified the functionality of F-HMC.
3. Empirical results on a high-dimensional multi-mon modtil synthetic dataset show that the gen- erated samples from our proposed method cover target distribution with more similarity to the target density concerning entropy value.
4. We have shown that using our proposed model, the semi-supervised learning algorithms on natural image datasets (ImageNet, SVHN, and CIFARIO 10) outperform the state-of-the -ofthh--rt in terms of inception scores (IS) and Frechet Inception Distance scores (FID).
The structure of the paper is as follow: We begin with the problem formulation in Section 2 and enhance the framework by introducing the proposed F-HMC in Section 4. We also have provided the theoretical analysis of F-HMC generate samples from the desired target. Then, we have demonstrated the performance of our proposed model in Section 5. Finally, Section 6 concludes the paper and discusses future work.
PROBLEM FORMULATION
Suppose having observed data D (x/ ()N from unknown probability distribution Pdate where x' describes the input and y denotes the corresponding label. We would like to estimate Pdata which is a high-dimensional multi-modi modal distribution. Bayesian GAN [Saatci Wilson (2017)) vestigates the distributions over the weight parameters of the generators and creates distributions over an infinity space of generators and discriminators, corresponding every conceivable setting of these weight vectors. We build upon the problem formulation in Bayesian GAN [Saatci & Wilson (2017)) and formulate the posterior as ff [f'('a)))) .a) where f(x', a) Gen (2, ág). Here 2 repre- sents white noise derived from p(z), and â, represents distribution over generator parameters. We denote that parameter set a consisting of two sub parameter set à, related to the generator and å associated with the discriminators.
We require the weight candidates of generators and discriminators to create candidates for poste- rior estimation. In this regard, we need to estimate posterior over ågs, åa. Since generators and discriminators are performing min-max optimisation, the parameter over â, and Óæ are interdepen- dent. First, generator weights âg sampled from prior (( and a particular generative neural network is constructed conditioning on these samples. Then, white noise 2 derived from p(z) transformed through the network Gen(z; &g) generate candidate data samples. Conversely, discriminator conditioned on its weights Disc(,; â.a) produces the probability that these candidate samples are generated from the data distribution. This process can be stated as follow considering L as the likelihood:
p( 2, âra) x exp (enpp
2

---

Under review as a conference paper at ICLR 2022
From the discriminator side, we need to form classification likelihood that classifies actual data from the generated samples and can be formulated as:
p(oalz, X, dg) x ex dd)} x exp(L(1
(2)
Here p(áa)Ba) indicates prior for dtg By marginalising = from Equations and the equations get updated to p( (i.)) = p( ddz and p( p() = p( 2, X, (zz We can approximate the posterior over ag and ag by iteratively sampling from p(ág y 64) and p(ā Therefore we can have the corresponding generators and discriminators to generate candidate samples from the multi- modal high dimensional distribution (Pdata). This paper proposes F-HMC as an efficient sampling strategy, especially when the target is nigh-dimensional and highly correlated.
3 BACKGROUND
This section briefly reviews data sampling basics, such as the Hamiltonian Monte Carlo sampler and Stochastic Hamiltonian Monte Carlo derived from [Chen et al. (2014))
Suppose one wants to sample from the posterior distribution of X given set of independent obser- vations E D:
((X exp (-U(())
where the potential energy function U is given by
U= Σ
logr(x|X) (og
RED
Hamiltonian Monte Carlo (HMC) is method for efficiently exploring the state space by proposing samples of X in Metropolis- Hastings (MH) framework. By incorporating auxiliary momentum variables. V. these suggestions are generated from a Hamiltonian system. HMC creates samples from joint distribution of (X, V) to sample from p(X[D):
(X, V) & (exp(-U(X) - (<<(--U())-VVM--
(5)
The samples of X have marginal distribution D) if the resultant samples of V are simply discarded. Here M is a mass matrix that, coupled with V. defines a kinetic energy term. H(X, V) U(X) v M defines the Hamiltonian function. H intuitively calculates the total energy of physical system by utilising position variables X and momentum variables V. To propose samples, HMC simulates Hamiltonian dynamics.
[dX = M- dv = -V -VU U(X)dt
(6)
Furthermore, SGHMC is based on the idea of combining stochastic optimisation with first-order Langevin dynamic MCMC technique, demonstrating that adding the "right amount" of noise to stochastic gradient ascent iterates results in samples from the target posterior as the step size is annealed. SGHMC accomplishes this by including a "friction" term in the momentum update:
[dx= Vdtt dv = -VV-VVCC U()) 1M---M--Vaa d + 2Bdt)
4 F-HMC MODEL
This section proposes F-HMC as efficient and scalable sampling strategy, particularly when the target is multi-modal and highly correlated (Contribution 1). Additionally, we have presented the mathematical analysis of F-HMC (Contribution 2) in section 4.2 to prove that the F-HMC samples from the equivalent target distribution specified Equation 7.
3

---

Under review as a conference paper at ICLR 2022
Original Hlgh Dineusional
Decomposttio data into components
Rumning each component parallel
Extinaaiin parameters of each componeal correliortiitooon between each componeats
Ruaning reduced version of data and extracted paramettrr
somples from the target dinsity
Wil Miglley
Figure 1: The pipeline of the F-HMC model which is consists stages. The first stage is decom- posed into S regions using Gaussian mixture. The second stage is running the regions parallel and estimating their corresponding density parameter. The third stage is running another HMC on data coming from the last stage with respect the cross-correlation between components to ensure reliable flow between different regions and finally generating data from the target distribution.
4.1 METHODOLOGY
In the scenario where the gradient elements dramatically different scales and highly related, the SGHMC cannot efficiently explore the target density. As a pragmatic approach, we propose F-HMC, parallelised algorithm that, instead of finding single chain that samples from the whole distribution, combines samples from several chains. Chains are runming in parallel, each exploring a different region the state space (e.g., few modes only). F-HMC assumes that Gaus- sian mixture approximates the target density pi. In that case, each mixture component represents a reliable proposal specific region of the sample space S Rdd Assume pi has modes and consider its approximation via the mixture model as follows:
(x)) Σ Na(xx:pt o0)
where Na(2; o²) is the probability density of d-variate Gaussian distribution with l mean and covariance matrix. We define the partition $ _ using the mixture representation in 8, that pi is similar Na(x:; (al (d) each set 800 than any other distribution entering 8, which is formulated as follow:
s(3) {a: arrgmaaN ()) o =
The approximation along with the regions defined enable determine the proposal distri- bution of F-HMC. One may consider proposal distribution as:
Q(x)
SR ( (
(10)
In other words, we would utilise the dominating component of the mixture as a proposed distribution in each area 5هك While such a concept may have acceptable local properties, it may not ensure optimal flow across various regions. Therefore, are using a second (fold) HMC on top of these
samplers concerning the cross-correlation of partitions to allow flow between different regions. Thus the F-HMC proposal distribution models are as follow:
Q'(x)= = Σ (zx))(x,n00).0l
(I1)
Ø=1
4

---

Under review as a conference paper at ICLR 2022
where o is calculated with respect cross-correlation between all components. Figure demon- strates the functionality of the F-HMC; since the first part is rumning on parallel and the second fold reduced number of dimensions, the overall execution time, especially high di-
is running mensional setup, improves. Furthermore, because two HMC samplers thoroughly examine the data, the system's entropy will be satisfactory. Algorithm shows one iteration of the Bayesian learning for the generator parameters using our proposed model. Here 0 is the frictton term for HMC, and the learning rate. K shows the number of iterations in Bayesian GAN, and shows the number of F-HMC iterations, and S indicates the number of components in the F-HMC.
Algorithm One iteration of the generator in Bayesian GAN set up with our proposed F-HMC
I: Input: {ag and {a²z from preceding iteration and number of as HMC friction term, 1) for
the learning rate, and as the number of F-HMC iterations.
2: for each f to do
3: ~ー >p(2)
sampling white noise z from its prior D- running the decomposition
4: for each +- 1 to S do
5: H4g + HTM (logp( (a.kk.kk))))))) (z, 2,
6: append μ4 to the j
7: end for each
8: f- calculate covariance between S components
9:
- --MCC (is, o)
& merging all back to sample from the target density
end for each
11: n (0, 20n!)
V t (1 8) +
13: G t 2 +
b updating p()g)) sample set
Output: generated samples for āg
4.2 THEORETICAL ANALYSIS
The purpose of this section is to demonstrate that the probability density of data created using F- HMC is similar to the target distribution SGHMC. As a result, we can confirm that F-HMC samples are mathematically valid (Contribution 2). Therefore, we can replace the F-HMC as valid sampler with SGHMC the Bayesian GAN design. In the Experiment section, we illustrated how this replacement improved the outcomes. Considering earlier setup outlined in Background section 3.V Equation 7, converges a stationary distribution [Chen al. (2014)) given by the following formula:
V ~ N
M
(12)
By taking the F-HMC methodology into account, the V in Equation 7 can be decomposed as the following matrix:
V11
pi
1/2
)
N
(13)
ōss
Vs
jās
Theorem: *X, V)
exp H(X, V)) the unique stationary distribution of the dynamics
described in Equation 7.
0
[o
0
Proof:
let W
and R
Equation along with the proposed decomposition in
B
Equation 13 can be written in the following form:
X²
Xs
d
d
V
Vi
V2
Vs
-0
[0
VU(Xi)
(U(X2)
VU(Xs) +N((0, 2Rdt)
(14)
B
M
M
M V
M M
-- W + (H, didt + +W(0 2Rdt)

---

Under review as a conference paper at ICLR 2022
We employ the Fokker-Planck Equation (FPE) (Risken (1992)) to describe the temporal evolution of the probability density function in relation with a Stochastic Differential Equation (SDE) that defines the development of the distribution the random variable under specified stochastic dynamies. Under Hamiltonian dynamics, the random variables in our situation are position variable and momentum variable V. Consider the following SDE:
du g(w)dt + (0, 2R(w)d(t
(15)
Pr(w) is the distribution of w governed by Eq. 15. we also Consider J.(.) as d.(.) by using FPE the equation evolves under the following formula:
[(e(w)) = E In. + ==1==1
Ju, (J, (Ry(())))
(16)
1=1
We rewrite Equation 16 in the following compact form:
H((lw) (WW)(
(17)
E ( and (T Lis Jui (Rey (w) (Pr(()())))) (Pr(w)) = Eij Jwi (Ry((W)) Eij Jui (Jus (Rij ((](w))))(w) Eij Twi (Jw, (Rey (w))).
where ([(g(w)))(W))
From Equation 17 and considering the variable = (X and (سwو = - (R+ WXVHXXV) and
0
R(w) = R(X, V) = R = 0
the distribution evolution under dynamic system in Equation 14
can be written as follow:
(X, v)) TJ ([R W)),,,,,)
Vp(X,
(18)
Since V) exp (- (HX we can verify that the <Xx, V) is invariant under Equation 18 by calculating je- le- H(X HXXX V) + Ve-H(X.v)] = O. Therefore, we can confirm that the dynamics given Equation have similar invariance properties to that of the original Hamiltonian dynamics of Equation 7.
5 EXPERIMENTS
We implemented the proposed model using Pymc3 and report its performance generating sam- ples from complex distributions in section 5.1. We have examined the model's performance in marginalising the generators' parameters [Netzer et al. (2011)), CIFAR 10 [Krizhevsky (2009)] and ImageNet [Deng et al. (2009)) in section 5.2 and 5.3, respectively. We have compared our results with WDCGAN [Arjovsky et al. (2017)), DCGAN, 10DCGAN (which DCGANs constructed by ten random subsets with 80% of the size of the training set, [Radford al. (2016))) Bayesian GAN [Saatci & Wilson (2017)) and ProbGAN [He et al. (2019)) on supervised and semi-supervised tasks with four different numbers of labelled examples. For fair comparison, each model has the same number of generators and discriminator with the same architecture.
synthetic and natural image datasets such as SVHN
fully supervised convolutional neural network composed of ten
5.1 F-HMC PERFORMANCE
To evaluate performance of F-HMC and compare against SGHMC, we have designed a set of experiments. First, use Normalising Flows [Kobyzev et al. (2020); Rezende & Mohamed (2016) as a rich family of distributions to examine F-HMC and SGHMC's abilities to explore complex distribution. Figure 2 shows the potential energy of the rich target distribution and the generated candidates using SGHMC and F-HMC. F-HMC successfully covers the target distribution (Advantage 1). Second, we have measured auto-correlation between the samples generated in each sampler as metric to show the power of the sampler in exploring the target distribution. The more precise the sampler, the faster the auto-correlation reaches zero. Figure 3 shows auto-correlation in F-HMC drops faster to zero than SGHMC in terms of lag number (Advantage 2).
'https///dos.pymm..i

---

Under review as a conference paper at ICLR 2022
Figure 2: The graph on the left shows the potential distribution of the target. The middle graph shows F-HMC samples exploring the target, and the right graph shows the SGHMC samples. Both samplers converged the target distribution, but the F-HMC covered the target more accurately than SGHMC.
0.75
0.75
0.25
0.00
0.00
-0.50
-0.50
-0.75
-1.00
100
200
500
400
500
-1.00
100
500
Figure 3: Auto-correlation between samples while the samplers explore the target distribution with 100 dimensions. The left graph shows auto-correlation of samples using the F-HMC sampler while S is 25. The right graph shows the auto-correlation the same setup using SGHMC. F-HMC converges faster than SGHMC to the target in terms of lag number
5.2 NIGH-DIMENSIONAT. MULTI MODAL SYNTHETIC DATASET
We present experiments multi-modal synthetic dataset test inferring multi-modal posterior DD) This experiment shows F-HMC's ability to explore a set of generators' parameters with proper entropy and different complementary properties to encapsulate rich data distribution. We fit a regular GAN, Bayesian GAN, and our proposed model to a dataset with D = 100 and 500. The generator for all models is two-layer neural network: 10- 10-1000-100, fully connected, with ReLU activation. The red samples in Figure 4 depict the target data, whereas the green samples depict the corresponding generated data. The experiments the results on D = 500 are shown on two lower rows. The name of generating sampler is displayed on the right side of each row. We can ensure that both strategies (BGAN and F-HMC GAN) cover the intended distribution simply by comparing them visually. Even the visual comparison is insufficient D 100. Still, provides a more desirable match to the target distribution as the dimension increases (Advantage and Contribution 3). Figure 5 shows the comparison of the performance of GAN, F-HMC GAN, and Bayesian GAN in terms of Jensen-Shannon divergence. The experiment estimates the similarity of the probability distribution of generated data to the original data and the level of entropy and confirms that the F-HMC exceeds other models (Advantage 3 and Contribution 3).
D 100 shown in the first two rows, while
detect more outstanding performance
evident D 500 that F-HMC
5.3 NATURAL IMAGE DATASET
We used 5-laer network architecture for GAN's generator in all experiments on the natural im- ages' datasets. The corresponding discriminator for supervised GAN is 5-layer 2-class DCGAN, and we have used 5-layer, K class DCGAN for semi-supervised GAN performing classifica-

---

Under review as a conference paper at ICLR 2022
100
Iae 400
FHMC
Ta 400
Ru BCD
RLy 1000
BGAN
D= 500
Fax
FHMC
Fxr 400
far 900
Dte 1r00
BGAN
-12
-00
Figure 4: samples drawn from Pdata (x) and visualised in 2D. the red colour graph shows real data, and the green colour graph shows generated samples. The first upper rows show the experiment on D 100, and the two lower rows show D = 500. The name of generating sampler is shown on the right side of each row. The graphs generated by F-HMC have more visual similarity the actual target, especially when the D = 500.
J5 OIv, (D 1601
J5 div. ID 500)
0.6
0.6
Bayas GAN
Bayas GAN
0.5
GAN
0.5
GAN
piobGAN
probGAN
15 Divergence
0.4
[5 Dilvengence
0.4
0.3
0.3
0.1
0.1
0.0
0.0
600
800
600
800
1000
No. of Iterations
No. of Iterations
divergence between Patza (2r) and the number of iterations of model training. The left graph shows the experiment while D = 100, and the right graph shows the same experiment with D = 500. We can confirm that F-HMC exceeds other models in generating data more similar to the target concerning entropy.
Figure 5: The
tion over K classes (see Saatci & Wilson (2017) for further details about Bayesian GAN structure)). To evaluate the performance of our proposed model, we have employed experiments in three mea- surement levels: 1 --performance metric in supervised and semi-supervised learning using test error rate, 2-run time per epoch in minutes by running all the models on single GPU. 3-quality of gen- erated images in terms of IS and FID scores. FID [Heusel et al. (2017)) is a measure that calculates the distance between vectors derived for real and synthetic images. IS [Salimans et (2016) an objective measure for assessing the quality and diversity of generated images.
8

---

Under review as a conference paper at ICLR 2022
Table 1: Supervised and semi-supervised learning results for image benchmarks, the N₂ shows number of labelled examples.
N,
Supervised
DEGANIO W-DCGAN
BayesGAN
ProbGAN
F-HMC GAN
CIFAR-10 10
500
65.1
2.7
56.8 2.9
30.5
80.1 2.8
1000
51.
29.1
27.4 3.2
3.1
27.6 = 2.8
2000
52.4
26.3 3.3
3.59
24.2 11.2
28.3
23.9
2.3
48.1++14
24.7
2.9
21.7
SVHN
55.1
30.8
30.1
33.
26.4
2.2
2000
36.7
2.63
17.9
27.2 = 2.6
14.2 44.2::2. 2.8
14.1 2.6
13.7 1.8
4000
28.2 + 3.18
15.8 1.4
25.1 + 2.8
12.7 2.994
13.5 + 1.7
11.7 1.4
8000
21.1 2.2 2.2
15.1::
20.1 = 1.9
9.2
11.44 1.8
8.9 0.9
ImageNet
TO00
57.6 1: 4.2
53.4
55.7 =: 3.7
48.9 4.43
47.8
43.8
4.4
2000
42.3 3.5
2.5
4.1
34.6
3.8
33.6
3.7
4000
40.1 3.6
2.1
35.5
238
26.8 3.2
25.9
- 3.5
8000
36.8 +: 4.1
28.3 1.8
34.3 2.7
24.4
24.1 + 2.8
22.7
2.7
Table 2: IS (higher is better), FID (lower is better) both trained with WGAN objective and run time (epochs in minutes) results natural images datasets.
Dataset
Score
TODCGAN
BayesGAN
probGAN
F-HMC GAN
CIFARTO
IS
7.78
7.69
7.72
7.79
FID
23.81
24.75
24.63
23.73
Runtime
143
91
94
93
SVHN
8.27
8.19
8.31
49.61
$1.78
52.32
47.21
Runtime
151
98
89
94
ImageNet
IS
8.41
8.51
8,56
8.59
FID
30.2
29.78
28.12
27.83
Runtime
671
358
349
336
Table demonstrates supervised and semi-supervised learning results for all image benchmarks. Our proposed model mainly outperforms BayesGAN, probGAN, W-DCGAN, and 10-DCGAN in terms of test error rate (Contribution 4). F-HMC shows its substantial impact when running higher-dimensional data (lmageNet) due to the parallel composition of F-HMC; it can efficiently explore higher dimension data. Table 2 shows the generated images' quality and the time of the models. The quality of images increases by using F-HMC. is observed that the run time enhances when running the model on higher dimension. We perceive that in lower-dimensional data (CIFAR, SVHN), the Bayesian GAN and probGAN's run time more satisfying. Once we run the models on nigher-dimeesiooal data (ImageNet), the run time improves in F-HMC. The parallel composition of F-HMC makes the over epoch run time less than exploring the whole dimensions at once in Bayesian GAN and probGAN (Advantage 4).
CONCLUSION
Folded Hamiltonian Monte Carlo is presented in this paper as a scalable strategy in sampling high-dimensional, highly correlated data to improve Bayesian GAN in producing synthetic im- ages/generating data by marginalising the weights of the generators and discriminators. We demon- strated that F-HMC converges faster and adapts higher-dimensional inputs with more significant similarity to the target data. The theoretical and mathematical analysis of F-HMC is presented which confirms functionality. F-HMC outperforms the of-hearft in terms of test error rates, runtimes epoch, IS, and FID scores when evaluated on synthetic high-dimensional multi-modal data and natural image benchmarks, such as CIFAR- 10, SVHN, and ImageNet. Despite the notable improvement that F-HMC can bring in sampling target density, its hyperparameters affect its en- forcement. Nested sampling methods [Betancourt et al. (2011)) sample from the likelihood space instead of sample space and more likely to land on better set of parameters. Future direc- tions include the adoption of a nested sampling method, e.g. restricted Hamiltonian Monte Carlo methods, to help the improvement of the outcomes.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY
The experiments presented in the paper are designed to be reproducible and easy to extend. The notebook provides instructions on how install the packages well as the code required the experiments and the link to download the necessary datasets.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
Michael Betancourt, Ali Mohammad- Djjfari,
Bercher, and Pierre Bessiere. Nested sampling with constrained hamiltonian monte carlo. 2011. doi: 10. 10.1063/1.3573613. httht:::
//dx. org/10.1063/111 1063111..333111113 3573613.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
https::/cppereet www.://wwwweeeteeenteeeeaa
net forum? 1orrumidd=B1xssj0555
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. 31st International Conference on Machine Learning, ICML 2014, 5, 02 2014.
Jia Deng, R. Socher, Li Fei-Fei, Wei Dong, Kai Li, and Li-Jia Li. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recog- nition CVVR), volume 00, pp. 248-255, 06 2009. doi: 10.1109/CVPR.20095520660 http://sstttttttte abst:actt//buuuutt//22883
URL
atttps:/iiet httpss//ieeeexpccee..e ieee
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Y. Bengio. Generative adversarial networks. Advances in Neural Informa- tion Processing Systems, 3, 06 2014. doi: 10.1 10.1145/3422622.
Hao He, Hao Wang, Guang-He He Lee, and Yonglong Tian. Bayesian modelling and monte carlo inference for GAN. In International Conference on Learning Representations, 2019. URL
https:/ // /ttps://ppeeeeeeee wpwprevieeeeeeeeeeee/ee/ee net 1dmmm111d-11111111
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, pp- 6629-6640, 2017.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020. ISSN 1939-3539. doi: 10.1 109/ppami.2220.29999944 URL http: /ttp:pp//xxccoo doi. owtp://ooo..o//....ooooo 1109/TO /PTMM 2020 2992934.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Yuval Netzer, T. Wang, A. Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. F-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 271-279, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2016.
H. Risken. The fokker-planck equation methods of solution and applications. Berlin: Springer, 1992.
Yunus Saatci and A. G. Wilson. Bayesian gan. In NIPS, 2017.
10

---

Under review as a conference paper at ICLR 2022
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 2234 2242, Red Hook, NY, USA, 2016.
Curran Associates Inc. ISBN 9781510838819.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363, PMIR, 09-15 Jun 2019. URL https:: ptwpooooceeeeeeeeooooooooocc ngs. mlr. wtw.ppressv99/// zhang 9d. html.
A
APPENDIX
Figure 6: Data samples for the CIFAR10 generated using F-HMC strategy
181
55
12
Figure 7: Data samples for the SVHN generated using F-HMC strategy


Reviews:
[{"review_id": "nWk0ssMDoc5", "reviewer": "Reviewer_PuGd", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is not clear, especially for the core part of FHMC formulation.\n\n1. How SGHMC is applied to the S components are not clearly described. I suggest a mathematical formulation instead of using only words.\n\n2. $\\mu$ and $\\sigma$ are used without any definition beforehand, not sure which parameters they represent.\n\n3. How are these parameters merged for F-HMC?\n\nThe theory part is also problematic. The stationary distribution of $V$ (equation 8) should be $\\mathcal{N}(0, M)$. $\\mathcal{N}(MB^{-1}\\nabla U(X), M)$ is the stationary distribution for the second equation in Equation 7. If this is used, the scheme would reduce to SGLD, not SGHMC. Moreover, the proof after that seems to be identical for SGHMC, with a simple dimension decomposition.\n\nThe experiments seems to be fine, if done properly. Compared to SGHMC, using F-HMC for Bayes GAN does provide some marginal improvement, not sure if it is statistically significant though. Better provide standard deviations together with the mean estimates.", "summary_of_the_paper": "This paper proposed Folded Hamiltonian Monte Carlo (FHMC) for posterior sampling over the generator and discriminator parameters in Bayesian GAN.", "summary_of_the_review": "The main contribution, the FHMC algorithm, is not presented clearly. The same for the theoratical justification. The paper would be strengthened if a formal and more detailed derivation of the proposed methodology is presented.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aivLIyFG7XH", "reviewer": "Reviewer_Hh37", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " \n---\nPros:\n \n* The paper proposes a new HMC sampling method for Bayesian-based GAN. I find the method novel and effective. \n* The experiments are nice. On the synthetic dataset, the author shows that F-HMC is better than SGHMC when learning Bayesian GAN with high dimensional data. On the natural image dataset, the F-HMC’s effectiveness is further verified. \n \n---\nCons: \n \n* Although the proposed method looks effective, I am not sure about the motivation or insight behind the proposed design. Could the author elaborate on why think of changing SGHMC to F-HMC? What makes F-HMC a better sampler? Does the improvement application-specific? Can F-HMC apply to tasks other than generative modeling? \n \n* Since F-HMC is a general improvement on SGHMC. While ProbGAN improves the learning objective (Posterior formulation) of Bayesian GAN. So I am considering that F-HMC and ProbGAN should be two improvements on the original Bayesian GAN in orthogonal directions. Hence, I suggest the author check the performance of the combination of F-HMC and ProbGAN. Basically, comparing Bayesian GAN + SGHMC v.s. Bayesian GAN + F-HMC (which has been done) and ProbGAN + SGHMC v.s. ProbGAN + F-HMC (which I suggest doing) should give a better picture of the effectiveness of F-HMC. \n* The clarity of the paper could be improved. For example, I am not sure how the equation (8) is derived. In the derivation of equation (11), the author mentions Fokker-Planck Equation without explaining it. It would be nice if the author could articulate more on the background knowledge and equation derivation to make the text more accessible to the general audience. \n \n---\nSome typos: \n* Page 5, equation (13) misses “[”.\n", "summary_of_the_paper": "The paper provides a new technique, called Folded HMC, to better sampling for the Bayesian treatments of GANs. The effectiveness of the method is demonstrated on several vision datasets, CIFAR-10, SVHN, ImageNet. ", "summary_of_the_review": "Overall, I vote for accepting. I like the idea of paralleling the HMC sampling for Bayesian-based GAN methods. My major concern is about the clarity of the paper and some additional ablation (see cons above). Hopefully, the authors can address my concern in the rebuttal period. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FMq_BSoApV", "reviewer": "Reviewer_jjmZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the idea of parallelizing HMC is an interesting one. However, I do have some concerns about clarity and the results of the paper\n\n1. Section 4, and particularly 4.1 (which expands on the main contribution of the paper) is confusing and very hard to follow:\n- the algorithm uses a \"parameter decomposition\" to \"divide tasks into blocks\": how is parameter decomposition carried out? What does it mean to divide updating tasks into blocks? Is this decomposition different for every update of the networks, or is it determined before training begins?\n- the components are then \"run in parallel\" -- Figure 1 implies that this HMC run simultaneously on different subsets of data dimensions, but Algorithm 1 implies that the subsets are for parameters.\n- HMC fit a distribution with \"Gaussian likelihoods\" -- Equation (1) states that the likelihood is proportional to the exponential of the GAN loss function: does the algorithm use a Gaussian approximation for this likelihood?\n- Finally, the \"$\\mu$s and $\\sigma$s are given [...] cross-correlation between all components\" -- it is very hard to understand what the algorithm does from this sentence.\n\nOverall, from the explanation in section 4.1, it is quite difficult to figure out what exactly the algorithm does, and therefore very hard to judge its merits and demerits.\n\n\n2. The theoretical analysis in section 4.2 looks at whether HMC performed in parallel on subsets of data dimensions converges to the same distribution as performing HMC on the full data. However, it is very hard to see what the different parts of the proof achieve, or how they are connected to HMC for the Bayesian GAN formulation without some more explanation. For example, it is very hard to follow the logic for deriving Equations 11-14, when all the meat in proving convergence seems to lie in the inline equation at the very end of section 4.2\n\n\n3. The scores for F-HMC GAN are not better than BayesGAN or probGAN across the board on all applications presented in the paper, and are sometimes only marginally better:  e.g IS scores on CIFAR-10 and ImageNet in Table 2 differ in the 3rd significant digit, the JSD traces for BayesGAN and  FHMC GAN are quite similar and seem to converge to the the same value in Figure 5, etc. Although this does not automatically disqualify the use of FHMC-GAN since it certainly has an advantage in terms of lower runtime, it seems also a bit of an exaggeration to claim the \"superiority of FHMC [...] in generating data\"\n\n\n### Minor comments:\n- there were numerous typos throughout the manuscript: it would be good to have a thorough edit\n- In Eqn 1, LHS ia posterior conditioned on $\\alpha_d$ but RHS has a likelihood dependent on $\\hat{\\alpha}_d$\n- The notation in section 3 is confusing: previously $x$ was used to denote target data, and $X$ as the set of all $x$\n- The theorem statement in section 4.2 refers to the \"dynamics described in Eqn 9\" -- should this be Eqn 7?\n- In Table 1, the score for probGAN with 4000 samples from ImageNet has the lowest value (25.8) and should be in bold.", "summary_of_the_paper": "The paper proposes an extension of BayesGAN (Saatci and Wilson, 2017): BayesGAN learns to sample generator and discriminator parameters from a posterior distribution conditioned on target data using Hamiltonian Monte Carlo sampling; this paper proposes using a modified version dubbed folded-Hamiltonian Monte Carlo to speed up the sampling process and make it scalable for high-dimensional data.", "summary_of_the_review": "It was hard to judge the merits of the algorithm and the contributions of the paper due to a lack of clarity in the explanations, and (it seemed to me) an exaggeration of the results.\nI would be happy to re-evaluate if the authors could provide a clearer explanation of the method and results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to sample the parameters of the generator and discriminator in a BayesGAN (Saatci and Wilson, 2017) setting. The main innovation is a modified Hamiltonian Monte Carlo sampling scheme. Unfortunately the method is not clearly presented, to the point that all reviewers had difficulties understanding how the method works. The revision is making progress but still does not clearly explain the method. While the paper cannot be accepted for publication in its present form, the experimental results are encouraging so I encourage the authors to keep improving their manuscript.
Decision: Reject