Title: PROBGAN: TOWARDS PROBABILISTIC GAN WITH THEORETICAL GUARANTEES

Abstract: Probabilistic modelling is a principled framework to perform model aggregation, which has been a primary mechanism to combat mode collapse in the context of Generative Adversarial Networks (GAN). In this paper, we propose a novel probabilistic framework for GANs, ProbGAN, which iteratively learns a distribution over generators with a carefully crafted prior. Learning is efficiently triggered by a tailored stochastic gradient Hamiltonian Monte Carlo with a novel gradient approximation to perform Bayesian inference. Our theoretical analysis further reveals that our treatment is the first probabilistic framework that yields an equilibrium where generator distributions are faithful to the data distribution. Empirical evidence on synthetic high-dimensional multi-modal data and image databases (CIFAR-10, STL-10, and ImageNet) demonstrates the superiority of our method over both start-of-the-art multi-generator GANs and other probabilistic treatment for GANs.

1. INTRODUCTION:

Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) is notoriously hard to train and suffers from mode collapse. There has been a series of works attempting to address these issues. One noticeable thread focuses on objective design, which improves the original Jensen-Shannon divergence with more stable pseudo-metrics such as f -divergence (Nowozin et al., 2016) , χ 2 -divergence (Mao et al., 2017) , and Wasserstein distance (Arjovsky et al., 2017) . However, such treatment is inherently limited when a single generator does not include enough model capacity to capture the granularity in data distribution in practice. Clearly, such a generator can hardly produce accurate samples regardless of the choice of objectives.

An alternative remedy is to learn multiple generators instead of a single one. This type of methods (Hoang et al., 2018; Tolstikhin et al., 2017; Wang et al., 2016b ) is motivated by a straightforward intuition that multiple generators can better model multi-modal distributions since each generator only needs to capture a subset of the modes. To entail model aggregation, probabilistic modelling is a natural and principled framework to articulate the aggregation process.

Recently, Saatci & Wilson (2017) propose Bayesian GAN, a probabilistic framework for GAN under Bayesian inference. It shows that modelling the distribution of generator helps alleviate mode collapse and motivates the interpretability of the learned generators. This probabilistic framework is built upon Bayesian models for generator and discriminator, whose maximum likelihood estimation can be realized as a metaphor of typical GAN objectives.

While empirical study on semi-supervised image classification tasks shows the effectiveness of Bayesian GAN, a critical theoretical question on this framework remains unanswered: Does it really converge to the generator distribution that produces the real data distribution? Indeed, our theoretical analysis and experimental results on a simple toy dataset reveal that the current Bayesian GAN falls short of convergence guarantee.

With this observation, we follow the prior work to exploit probabilistic modelling as a principled way to realize model aggregation, but approach this problem from a theoretical perspective. We analyze the developed treatment, including the choice of priors, approximate inference, as well as its convergence property, and simultaneously propose a new probabilistic framework with the desirable convergence guarantee and consequently superior empirical performance.

Our main contributions are:

• We theoretically establish, to our best knowledge, the first probabilistic treatment of GANs such that any generator distribution faithful to the data distribution is an equilibrium.

• We prove the previous Bayesian method (Saatci & Wilson, 2017) for any minimax GAN objective induces incompatibility of its defined conditional distributions.

• We propose two special Monte Carlo inference algorithms for our probabilistic model which efficiently approximate the gradient of a non-differentiable criterion.

• Empirical studies on synthetic high-dimensional multi-modal data and benchmark image datasets, CIFAR-10, STL-10, and ImageNet, demonstrate the superiority of the proposed framework over the state-of-the-art GAN methods.

2. RELATED WORK:

Generative Adversarial Networks is a powerful class of methods to learn a generative model for any complex target data distribution. There is a game between a generator and a discriminator. Both of them adapt their strategies to maximize their own objective function involving the other:

EQUATION

Eqn. 1 gives a general mathematical form where p data is real data distribution and p gen (•; θ g ) are generated data distribution with generator parameter θ g . The objective functions φ 1 , φ 2 , φ 3 (termed as GAN objective in this paper) are elaborately chosen such that at the equilibrium, the generator generates the target data distribution. Table 1 summarizes several widely used GAN objectives, including the original min-max version, non-saturating version of original GAN (Goodfellow ( 2016)), LSGAN (Mao et al. (2017) ), and WGAN (Arjovsky et al. (2017) ). As reported in Table 1 , some GAN objectives, satisfying φ 3 (•) = -φ 2 (•), actually represent a min-max game, i.e. min θg max θg J d (θ d ; θ g ).

Training GAN with multiple generators is considered in several recent works to mitigate the mode collapse problem. In the spirit of boosting algorithm, Wang et al. (2016b) propose to progressively train new generator using a subset of training data that are not well captured by previous generators, while Tolstikhin et al. (2017) further propose a more robust mechanism to reweight samples in the training set for a new generator. From the perspective of game theory, MIX-GAN (Arora et al., 2017) extends the game between a single generator and discriminator to the multiple-player setting.

Other works resort to third-party classifiers to help multiple generators and discriminators achieve better equilibrium, such as MAD-GAN (Ghosh et al., 2017) and the recent state-of-art method, MGAN (Hoang et al., 2018) .

Bayesian GAN proposed by Saatci & Wilson (2017) adopts a different approach which models generator and discriminator distributions by defining the conditional posteriors (Eqn. 8). The likelihood model is specially designed such that maximizing it exactly corresponds to optimizing GAN objectives. The authors argue that compare to point mass ML estimation, learning the generator distribution which is multi-modal itself offers better ability to fit a multi-modal data distribution.

To facilitate discussion, we categorize GAN frameworks into the following taxonomy: optimizationbased methods and probabilistic methods. Optimization-based methods set up an explicit mini-max game between the generator and discriminator, where an ideal equilibrium typically characterize a generator faithful to data distribution. In probabilistic methods, generators and discriminators evolve as particles of underlying distributions, where an equilibrium is searched from a stochastic exploration in the distribution space (of the generators and discriminators).

3. METHODOLOGY:

We first summarize the notations. Second, we elaborate ProbGAN, our probabilistic modelling for GAN, and introduce its Bayesian interpretation by developing constituent prior and likelihood formulations. Finally, we develop inference algorithms for ProbGAN. A detailed discussion of the motivation of our modelling and the comparison with Bayesian GAN is included in Section 4.

3.1. NOTATIONS:

p data (x) over a sample space X is the target data distribution we want to learn. Our generator and discriminator are parameterized by θ g ∈ Θ g and θ d ∈ Θ d . A generator with parameter θ g defines a mapping from a random noise vector z ∼ p z to a random vector G(z; θ g ). The induced probability density of G(z; θ g ) is denoted as p gen (x; θ g ). A discriminator is a function that maps data to a real-valued score, i.e. D(x; θ d ) : X → [0, 1] (or X → R in some settings). Further, we use q g (θ g ) ∈ P Θg , q d (θ d ) ∈ P Θ d to denote the distribution over generators and discriminators respectively.

The total data distribution generated by generator following the density q g (θ g ) is naturally a mixture of data distribution given by every single generator, p model (x; q g ) = E θg∼qg(θg) [p gen (x; θ g )]. Our goal is to find a generator distribution q * g (θ g ) such that the total generated data distribution matches our target, i.e. p model (x; q * g ) p data (x). J g (θ g ; θ d ) and J d (θ d ; θ g ) denote objective functions of generator and discriminator as introduced in Eqn. 1. The common choices1 are listed in Table 1 . With a slight abuse of the notation, we extend the notation J g (θ g ; D * ) by replacing D(x; θ d ) in equation 1 with any score function D * . Then J g (θ g ; θ d ) can be viewed as an abbreviation of J g (θ g ; D(•; θ d )). Likewise, J d (θ d ; p gen (•)) represents discriminator objective given a virtual generator that generates data with density p gen (•).

3.2. PROBGAN:

Like Bayesian GAN, ProbGAN learns distributions of the generator and the discriminator. During training, the target data distribution is given (by samples) and treated as an fixed environment. While for the generator/discriminator, they observe each other and adapt their own parameters based on the observation. To facilitate the comparison to Bayesian GAN, we state ProbGAN in a Bayesian formulation. Every generator/discriminator distribution update can be viewed as the following posterior inference process.

Posterior. ProbGAN updates generator/discriminator distributions based on their distributions in previous time step and the target data distribution as shown in Eqn. 2. We we denote q (t) g and q (t) d as distributions at time step t.

EQUATION

To understand it is a posterior modelling, we further interpret the terms in Eqn. 2 as likelihood term and prior term seperately.

Likelihood. We call the exponential terms in Eqn. 2 as likelihood terms.

EQUATION

where

p (t) model (x) = E θg∼q (t)

g [p gen (x; θ g )] is the mixed data distribution under the current generator distribution q (t)

g and D (t) (•) = E θ d ∼q (t) d [D(•; θ d )] is the averaged discriminating score function under current discriminator distribution q (t) d .

These likelihoods indicate a preference for generators and discriminators, given current distributions of generator and discriminator. More specifically, likelihoods in Eqn. 3 encode the information that distributionally reflect the objective of generators J g and discriminators J d . Such quantities evaluate the fitness between the generator and the discriminator.

We emphasize, although sharing the same spirit of reflecting the GAN objectives in likelihood, there is a crucial difference between our likelihood model and that of Bayesian GAN. We will revisit it in the later theory section.

Prior. Unlike Bayesian GAN using normal distributions for both generator and discriminator, ProbGAN has less standard priors. As Eqn. 2 suggests, we set different priors for the two players.

For the generator, we use the generator distribution in the previous time step as a prior. The intuition is following. When the generated data distribution is increasingly close to the real data distribution, there will be less information for discriminator to distinguish between them; consequently, the discriminator tends to assign equal scores to all data samples, resulting in equal likelihoods for all generators. At that stage, a good strategy is to keep the generator distribution the same as the previous time step, since it already generates the desired data distribution perfectly. Hence, we use the generator distribution in the previous time step as a prior for the next Such dynamically evolving prior for generator turns out to be crucial. In Section 4.2, we show the Bayesian GAN suffers from bad convergence due to its fixed and weakly informative prior. In contrast, we set a uniform improper prior on the discriminator to pursuit unrestricted adaptability to evolving generators.

3.3. INFERENCE ALGORITHM:

So far we have introduced our ProbGAN model. In this section we develop novel inference algorithms to compute the posterior. Similar to most advanced Bayesian methods, exact calculation of the posterior is intractable. Following the strategy in Saatci & Wilson (2017) , we adopt Stochastic Gradient Hamiltonian Monte Carlo to generate samples from the posterior. In each iteration, M samples {θ

(t)

g,m } M m=1 are used to approximate the generator distribution q (t)

g .

EQUATION

∇ θg log q (t+1) g (θ g ) = ∇ θg (J g (θ g ; D (t) )+log q (t) g (θ g )) ≈ ∇ θg (J g (θ g ; 1 M d m D(•; θ (t) d,m ))+log q (t) g (θ g )).

(5)

Empowered by the adapted SGHMC (Algorithm 1 in the appendix), we are able to sample from q (t+1) g and q (t+1) d based on gradients in Eqn. 4 and Eqn. 5. The gradients come from two sides: the GAN objective J g , J d and the prior q (t) g . Getting GAN objective's gradient is easy while computing the prior's gradient, ∇ θg log q (t) g (θ g ), is actually non-trivial since we have no exact analytic form of q (t) g (θ g ). To address this challenge, we propose the following two methods to approximate ∇ θg log q (t) g (θ g ), leading to two practical inference algorithms.

Gaussian Mixture Approximation (GMA). Although the analytic form of the distribution q

(t) g (θ g ) is unknown, we have M g Monte Carlo samples {θ (t) g,m }

Mg m=1 which enables us to directly approximate the distribution as a Mixture of Gaussian in the left side of Eqn. 6, where σ is a hyper-parameter and C is the normalization constant. Then we derive the prior gradient approximation as shown in the right side of Eqn. 6.

EQUATION

Partial Summation Approximation (PSA). From Eqn. 5, actually we can make an interesting observation that the prior gradient can be recursively unfolded as a summation over all historical GAN objective gradients, shown as: Therefore if we store all historical discriminator samples {θ

EQUATION

(i) d,m } t,M d i=1,m=1

, the prior gradient can be computed accurately via simple summation. Practically, computing gradients with all discriminator samples costs huge amount of storage and computational time, which is unaffordable. Hence we propose to maintain a subset of discriminators by subsampling the whole sequence of discriminators.

4. THEORY:

In this section, we first present the good convergence property of ProbGAN. Second, we theoretically analyze the distribution evolution of Bayesian GAN (which will be referred to as BGAN in the rest of the paper) and compare BGAN with ProbGAN. All proofs are included in the appendix (Section A).

4.1. CONVERGENCE PROPERTY OF PROBGAN:

We say a generator distribution is ideal if the generator following this distribution produces the target data distribution. Theorem 1 shows that any ideal generator distribution is an equilibrium of the dynamics defined in Eqn. 2. Although its mathematical proof involves more elaboration, the idea behind is quite simple. When the generator distribution is ideal, the discriminator is not able to distinguish the synthetic data from real. Thus the averaged discriminator function will degenerate to a constant function. Afterwards, the generator distribution will remain unchanged since the discriminator essentially puts no preference over generators. Here, we note that the discriminator is only involved in the likelihood. The prior still needs to be carefully designed so that the model can converge to an equilibrium where the generator is ideal. Simply choosing a weakly informative prior as Bayesian GAN did will not give Theorem 1. Theorem 1. Assume the GAN objective and the discriminator space are symmetry. For any ideal generator distribution q * g (θ g ) satisfying p * model

E θg∼q * g [p gen (•; θ g )] = p data , there exists a dis- criminator distribution q * d such that D * (•) E θ d ∼q * d D(•; θ d ) ≡ c 2 .

Moreover, q * g and q * d is an equilibrium of the dynamic defined in Eqn. 2.

4.2. PROBGAN V.S. BAYESIAN GAN:

This section presents analyses of the BGAN algorithm, where we find a theoretical issue in its convergence and highlight the importance of our renovation of the prior and likelihood.

Corollary 1 states our derivation of posterior modelling in BGAN, where p g (θ g ; α g ), p d (θ d ; α d ) are the predefined priors. In practice, BGAN use a fixed Gaussian prior. Corollary 1. The Bayesian GAN algorithm actually performs distribution dynamics in Eqn. 8.

q (t+1) g ∝ exp{E θ d ∼q (t) d J g (θ g ; θ d )}p g (θ g ; α g ), q (t+1) d ∝ exp{E θg∼q (t) g J d (θ d ; θ g )}p d (θ d ; α d ) (8)

Difference in Likelihood. The subtle adjustment of our likelihood term lies in the order of taking expectation. As shown in Eqn. 3, our choice of likelihood yields a concrete physical meaning. Our discriminator likelihood explicitly evaluates the discriminator ability of distinguishing real data distribution and total data distribution generated by all generators. Hence, our approach matches the target data distribution with mixed data distribution produced by generators, while BGAN may not.

Difference in Prior. The choice of prior also plays an important role in convergence. A weakly/noninformative prior, as adopted in BGAN, prevents the generators from convergence even if they already produce the data distribution faithfully. The phenomenon arises from the fact that the information provided by discriminator vanishes when the generators are ideal and the resulting generator posterior will degenerate to the prior. To remedy this issue, our solution is to take generator distribution at previous time step as the prior. Then whenever the discriminator degenerates to a constant, the generator distribution will stay unchanged because the prior is itself.

An Analytical Case Study. We demonstrate the superior convergence property of our model on a categorical distribution, where analytic posterior computation of various choices of likelihood and prior are feasible; we compute the exact equilibrium of GAN models under all the four combinations of the priors and likelihoods(ProbGAN's choices v.s. BGAN's choices). The experiment details are in the appendix (section D). Figure 1 is an example of the data distributions generated by each model after it converges. Among all the combinations, our model is the only formulation that yields proper convergence, which validates our theoretical analysis.

Compatibility Issue. We further show BGAN's choice of likelihood and prior leads to theoretical issues. Specifically, BGAN is not suitable for any minimax-style GAN objective due to the incompatibility of its conditional posteriors. This problem may limit the usage of BGAN since many widely used GAN objective is in min-max fashion, such as the original GAN and WGAN.

Consider a simple case where we use only one Monte Carlo sample for the distributions q (t) g and q (t)

d .

Then the distribution evolution in Eqn. 8 will degenerate to a Gibbs sampling process.

EQUATION

Thus θ

(t)

g and θ

(t)

d are implicitly sampled from a joint distribution of θ d and θ g defined by the conditionals p(θ

g |θ d ) = exp{J g (θ g ; θ d )}p(θ g |α g ) and p(θ d |θ g ) = exp{J d (θ d ; θ g )}p(θ d |α d ).

However, our theoretical analysis shows that such a presumed joint distribution does not exist when J d (θ d ; θ g ) = -J g (θ g ; θ d ). Specifically, Lemma 1 shows the existence of a joint distribution satisfying the conditionals in Eqn. 9 requires the GAN objective to be decomposable, i.e.

∃φ g , φ d , s.t. J d (θ d ; θ g ) = φ g (θ g ) + φ d (θ d ).

Apparently, no valid GAN objective is decomposable. Therefore, conditionals in Eqn. 9 are actually incompatible. Sampling with incompatible conditional distribution is problematic and leads to unpredictable behavior (Arnold & Press, 1989) . Lemma 1. Consider a joint distribution p(x, y) of variable X and Y . Its conditional distributions can be represented in the forms of p(x|y) ∝ exp{L(x, y)}q x (x) and p(y|x) ∝ exp{-L(x, y)}q y (y) only if X and Y are independent, i.e., p(x, y) = p(x)p(y) and L(x, y) is decomposable, i.e., ∃L x and L y , L(x, y) = L x (x) + L y (y).

5. EXPERIMENTS:

In this section, we evaluate our model with two inference algorithms proposed in Section 3.3 (denoted as ProbGAN-GMA and ProbGAN-PSA). We compare with three baselines: 1) GAN (or DCGAN2 ): naively trained multiple generators in the vanilla GAN framework; 2) MGAN: Mixture GAN (Hoang et al., 2018) which is the start-of-art method to train GAN with multiple generators; 3) BGAN: Bayesian GAN (Saatci & Wilson, 2017) .

For each model, we conduct thorough experiments with the four different GAN objectives introduced in Table 1 , which are referred to as GAN-MM, GAN-NS, WGAN and LSGAN here. For a fair comparison, each model has the same number of generators with the same architecture. Discriminator architectures are also the same except for that of MGAN which has an additional branch of the classifier. To facilitate reproducibility, we report implementation and experiment details in Section B of the appendix.

5.1. HIGH-DIMENSIONAL MULTI-MODAL SYNTHETIC DATASET:

Dataset. Consider learning a data distribution in a high dimensional space X = R D , which is a uniform mixture of n modes. Each mode lies on a d-dimensional sub-space of X . We call this d-dimensional sub-space as mode-space of the i-th mode. Specifically, the data of the i-th mode is generated by the following process, 

EQUATION

In our experiment, n, D, and d are set to 10, 100, and 2. Hyper-parameters for A and b are set to be σ A = σ b = 5. Each model train ten generators.

Metric. We define projection distance p for generated data sample x as the minimum of Euclidean distance from x to any of mode-spaces i.e. p (x) = min i i (x)

x -A i (A T i A i ) -1 A T i x 2 .

Then we set a threshold3 η to test the belonging of x, i.e. the data samples whose Euclidean distance to the mode-space is below η are considered as belonging to that mode. The trained models are evaluated by the samples {x k } K k=1 ∼ p model it generates. We define hit set H i {x k | i (x k ) < η} to indicate the samples belong to each mode. We further define projected hit set, 

PH i {(A T i A i ) -1 A T i x-b i |x ∈ H i }

i ) U[-1, 1] d ).

The intuition is that if data generated is close to the ground truth distribution, they should be uniformly distributed in the square area of each mode.

Optimization-Based v.s. Probabilistic. The left part of Table 2 summarizes the results in terms of hit ratio and hit distance. Probabilistic methods including our algorithms and BGAN always achieve a hit ratio of 1, which means every data point generated from these models is very close to one mode of the target distribution. On the other hand, optimization based methods, both GAN and MGAN, consistently have a significantly larger hit error, and sometimes may even generate data samples that do not belong to any mode. Moreover, the data distribution generated by the optimization-based methods fits the target uniform distribution much worse than its probabilistic counterparts, which is quantitatively reflected by the cover error showed in the right side of Table 2 and visually demonstrated by the projected hit sets in Figure 2 . According to the visualization, data generated by GAN or MGAN tend to be under dispersed and hardly cover the whole square region of the true mode, while data generated by probabilistic methods align much better with the ground truth distribution. We attribute this superiority to stronger exploration power in the generator space coming from the randomness in probabilistic methods.

Bayesian GAN v.s. ProbGAN. The incompatibility issue of BGAN with minimax-style GAN objectives theoretically derived in Section 4.2 is empirically verified in our experiments. As visualized in Figure 2 , with the GAN-MM objective, BGAN is trapped in a local equilibrium and fails in capturing one mode of the true data. Besides, as shown in Table 2 , BGAN with the WGAN objective achieves much poorer coverage than with other GAN objectives, while our model is much more robust to the choice of GAN objectives (consistently lower cover errors). A qualitative comparison is made in Figure 9 (in the appendix) which shows the data distribution generated by BGAN trained with WGAN objective tends to shrink. More visual illustrations under different GAN objectives are placed in Section E of the appendix.

5.2. NATURAL IMAGE DATASET:

Datasets. We evaluate our method on 3 widely-adopted datasets: CIFAR-10 ( Krizhevsky et al., 2010) , STL-10 (Coates et al., 2011) and ImageNet (Deng et al., 2009) . CIFAR-10 has 50k training and 10k test 32x32 RGB images from 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. STL-10, containing 100k 96x96 RGB images, is a more diverse dataset than CIFAR-10. ImageNet has over 1.2 million images from 1,000 classes and presents the most diverse dataset. For a fair comparison with baselines, we use the same settings as MGAN. We resize the STL-10 and ImageNet images down to 48x48 and 32x32 respectively.

Evaluation Protocols. We employ two common image generation metrics: Inception Score (Salimans et al. ( 2016)) and Fréchet Inception Distance (Heusel et al. (2017) ). Inception Score computes exp(E x [KL(p(y|x) p(y))]) where p(y|x) standards the predicted label distribution by a pre-trained Inception model (Szegedy et al., 2015) and p(y) is the average of p(y|x) over all images in the dataset. Inception Score (IS) reflects the fidelity and diversity of images and is well-correlated with human judgment (Salimans et al., 2016) . However, it does not measure the similarity between the real data and the synthetic data. Therefore, as a complementary, we use Fréchet Inception Distance (FID). Specifically, it measures the Fréchet distance between two image distributions in the feature embedding space conduct by the Inception model. More details are included in the appendix Model Architectures. Inspired by Hoang et al. (2018) , we adapted the parameter sharing technique for both generators and discriminators. Specifically, generators (of one model) are not disjoint neural networks. They only differ at the first layer and sharing all parameters at the following layers. This tied parameter design reduced the model complexity and boost the performance. Especially for the probabilistic model such as Bayesian GAN, we observe a significant enhancement of generated image qualities compare to the original result (Figure 7 in Saatci & Wilson (2017) ). More implementation details such as hyper-parameters of network layers are reported in the appendix (Section B).

Quantitative Results. Table 3 and Table 4 summarize the Inception scores and FIDs on the three benchmark datasets for our model and baselines. For each model, we denote the epoch delivering highest 'IS-0.1FID' as the best epoch and report the scores of the model at this epoch. We design this 'IS-0.1FID' principle to neutralize the discrepancy between Inception score and FID. Because Inception score and FID may not be consistent with each other, it is possible that the Inception score improves but the FID gets worse when we evaluate the model at a different checkpoint. Hence, we need a rule to pick one point of the Pareto frontier of the model performance. Overall, our proposed ProbGAN outperforms the baselines on all three benchmark image datasets. Furthermore, according to results on CIFAR-10, our model achieves the best performance under all GAN objectives. Generally, probabilistic methods achieve better scores than optimization based methods, which indicates that injecting stochasticity into GAN training helps generate more multi-modal images. Note that the performance gap increases as the dataset gets more diverse. Specifically, when using GAN-NS objective, our model improves FID by 4.00 on CIFAR-10 (Table 3 ) in comparison to MGAN. While the FID improvements are 4.82 and 18.06 on STL-10 and ImageNet (Table 4 ), respectively.

Besides, we note that Bayesian GAN has a significant performance drop when accompanied by min-max style GAN objectives, which provides another empirical evidence for our theory analysis in Section 4.2. By contrast, ProbGAN fits any GAN objectives and constantly performs well.

Qualitative Results. Figure 3 displays the samples randomly generated by the baselines and our ProbGAN. Each row in the figure contains samples of one learned generator. In Figure 3 and Figure 10 (in the appendix), all the three baselines noticeably suffer from mode collapse. Indeed, almost in every training trial, one or two generators of the baseline models degenerate during the training. Hoang et al. (2018) already notice that mode collapse in one of the generators could happen after a long training procedure (around 250 epochs). Our experiment shows Bayesian GAN also have this issue. However, ProbGAN is robust to mode collapse. Visual results for the entire ablation study on CIFAR-10 are included in Section F of the appendix. Each row in these two figures contains generated images from one generator. The results show that our method is robust to 'single generator mode collapse' in both STL-10 and ImageNet. We also include cherry-picked results on STL-10 to exhibit the capability of our model to generate visually appealing images with complex details while improving the robustness against mode collapse.

6. DISCUSSION:

In this paper, we propose ProbGAN, a novel probabilistic modelling framework for GAN. From the perspective of Bayesian Modelling, it contributes a novel likelihood function establishing a connection to existing GAN models and a novel prior stabilizing the inference process. We also design scalable and asymptotically correct inference algorithms for ProbGAN. In the future work, we plan to extend the proposed framework to non-parametric Bayesian modelling and investigate more theoretical properties of GANs in the probabilistic modelling context.

Developing Bayesian generalization for deep learning models is not a recent idea and happens in many fields other than generative models such as adversarial training (Ye & Zhu, 2018) and Bayesian neural networks (Wang et al., 2016a) . By this work, we emphasize the importance of going beyond the intuition and understanding the theoretical behavior of the Bayesian model. We hope that our work helps inspire continued exploration into Bayesian deep learning (Wang & Yeung, 2016 ) from a more rigorous perspective.

Proof.

EQUATION

EQUATION

)

Eqn. 11 and Eqn. 12 prove that q

* d (θ d ) ∝ exp(J d (θ d ; p * model )) satisfies E θ d ∼q * d D(•; θ d ) ≡ c

2 . Note that, in the above equations, S(θ d ) denotes the symmetric discriminator of θ d satisfying D(x; S(θ d )) ≡ c -D(x; θ d ). Hence according Eqn. 13, we know J g (θ g ; D * ) is a constant.

EQUATION

Thus the generator distribution will not change based on the dynamics in Eqn. 2 since q * g (θ g ) ∝ exp{J g (θ g ; D * )}q * g (θ g ).

Corollary 1:

Proof.

log q (t+1)

EQUATION

Algorithm 1 from the original paper (Saatci & Wilson, 2017) implies that Eqn. 14 where θ k d ∼ q (t) d (θ d ) are Monte Carlo samples of discriminator and every z (i) is a mini-batch containing n g noise samples. Be definition p(θ g |z (i) 

, θ k d ) ∝ ng j=1 D(G(z (i) j ; θ g ); θ k d )p(θ g |α g ) (Eqn. 1 in the original paper), each term 4 log p(θ g |z (i) , θ k d ) = ng j=1 φ 3 (D(G(z (i) j ; θ g ); θ k d ))+log p(θ g |α g )

. Hence, the total summation is a Monte Carlo approximation of the expectation in the right side of Eqn. 15. The same derivation can be done to q (t+1) d

. Together, we get Corollary 1.

Lemma 1:

Proof.

p(x|y) = α(y) exp{L(x, y)}q x (x), p(y|x) = β(x) exp{-L(x, y)}q y (y), =⇒ p(x, y) 2 = p(x|y)p(y) × p(y|x)p(x) = p(x)p(y)α(y)β(x)q x (x)q y (y) =⇒ X, Y are independent. =⇒ p(x) = p(x|y) =⇒ L(x, y) = log p(x) -log q x (x) -log α(y) =⇒ L(x, y) is decomposable. Setup. In this toy example, we consider the case where X , Θ g , and Θ d are all finite sets, specifically

m = 1 to M d do θ d,m ← θ (t) d,m for l = 1 to L do v ← (1 -α)v + η∇ θ d log q (t+1) d (θ d,m ) + n; n ∼ N (0, 2αηI) θ d,m ← θ d,m + v end for θ (t+1) d,m ← θ d,m end for for m = 1 to M g do θ g,m ← θ (t) g,m for l = 1 to L do v ← (1 -α)v + η∇ θg log q (t+1) g (θ g,m ) + n; n ∼ N (0, 2αηI) θ g,m ← θ g,m + v end for θ (t+1) g,m ← θ g,

X = {x 1 , • • • , x N }, Θ g = {θ 1 g , • • • , θ Ng g }, Θ d = {θ 1 d , • • • , θ N d d }.

The target data distribution is a categorical distribution Cat(λ 1:N ) where λ i = p data (x i ) is the probability of generating data x i . Generator G i generates data following the categorical distribution p data (x; θ i g ) = Cat(α i 1:N ).

Further, the probability distributions of generator and discriminator are categorical distributions q g (θ g ) = Cat(β 1:Ng ) and q d (θ d ) = Cat(γ 1:N d ).

In practice, we set N = 10, N g = 20, N d = 100. In each experiment trial, target data distribution λ 1:N and data distributions for each generator α i 1:N are fixed and initialized by the following categorical distribution generating procedure. We first independently sample from the uniform distribution U[0, 1] to get { λj } N j=1 and then normalize them to get a categorical distribution λ j = λj N j=1 λj .

For the discriminators, their function values are randomly generated from a uniform distribution, i.e.

{D(x:

i ; θ j d )} N,N d i=1,j=1 ∼ U[0, 1].

Models in comparison.:

As listed in Table 5 , we compare four models with different pairs of likelihood and prior. There are two choices of likelihood, expectation of objective value and objective value of expectation. Mathematically, expectation of objective value likelihood has the following formula:

• Generator likelihood l g (θ g ) of observing discriminator distribution q d (θ d ) is proportion to exp{E θ d ∼q d (θ d ) J g (θ g ; θ d )}. • Discriminator likelihood l d (θ d ) of observing generator distribution q g (θ g ) is proportion to exp{E θg∼qg(θg) J d (θ d ; θ g )}.

While the objective value of expectation likelihood is different as follows:

• Generator likelihood l g (θ g ) of observing discriminator distribution q d (θ d ) is proportion to exp{J g (θ g ; E θ d ∼q d (θ d ) D(•; θ d ))}. • Discriminator likelihood l d (θ d ) of observing generator distribution q g (θ g ) is proportion to exp{J d (θ d ; E θ d ∼q d (θ d ) p gen (•; θ g ))}.

Thus model A indeed is the design of Bayesian GAN, while model D is our proposed model. Model B and C are introduced to conduct the ablation study.

Metric. We employ l 1 distance for evaluation which can be directly computed on categorical distributions as follows.

EQUATION

Evaluation. In the categorical distribution settings, all the likelihood, prior and posterior computing can be done analytically. For each model, we update the generator and discriminator distributions iteratively and monitor the distance between the target data distribution and the data distribution generated by the model. We experiment with two different choices of GAN objectives, GAN-MM and GAN-NS. Note GAN-MM is a minimax-style objective.

Result. Three probabilistic models performs perfectly in this case, while both the two optimization-based methods miss one mode of the true distribution. This experiment illustrates that although MGAN employs an additional classifier to force the data generated by different generators to be disjoint, it still suffers from mode collapsing problem. This is because in MGAN, generators may still generate disjoint data samples in the same mode and fail in capturing other modes. 

:

The concepts of minimax version and non-saturating version of vanilla GAN are first introduced in Goodfellow (2016).

We use original GAN for synthetic dataset and DCGAN for image generation task

Based on the fact that the average distance between the data from two different modes is 800, we set a threshold of η =

Note that in BGAN paper, the GAN objective is GAN-NS. Thus φ3 equals log(•).

MGAN Code: https://github.com/qhoangdl/MGAN

https://github.com/mseitzer/pytorch-fid

https://github.com/bioinf-jku/TTUR

Figures and Tables:

Figure 1: An example of data distributions produced by converged models in the toy experiment on categorical distribution. We examined four possible combinations of likelihoods and priors. L our , L bgan stand for the likelihoods of our ProbGAN model and BGAN model. P our and P bgan stand for the priors. Only our model (Figure 1(d)) learns the target data distribution (Figure 1(e)).

Figure 2: Visualization of projected hit sets. Rows from top to bottom correspond to MGAN, BGAN, and ProbGAN-PSA trained with GAN-MM objective. See Figure 6 (in appendix) for results of all models. In one row, projected hit sets for each mode are plotted in different panels, where the red boxes indicate real data regions U[-1, 1] 2 . Different colors indicate samples from different generators.

by projecting data in each hit set back to the canonical low dimensional space. Now we introduce three evaluation metrics: hit ratio, hit distance, and cover error. Hit ratio H r n i=1 |H i | /K is the percentage of generated data belonging to any of the modes of real data. Hit distance H d n i=1 x∈Hi i (x)/ n i=1 |H i | is the averaged projection distance over all data in the hit set. Lastly, cover error C e evaluates how well the generated data covers each mode. Essentially it computes the KL-divergence between the estimated distribution of samples in PH i and the uniform distribution over [-1, 1] d . Formally, it is defined as the averaged KL-divergence on n modes i.e. C e

Figure 3: Images generated by MGAN, BGAN and our model trained on CIFAR 10 with GAN-NS objective. The tenth generator of MGAN (Figure 3(a)) and the first of BGAN (Figure 3(b)) collapse while generators of our method all work well. DCGAN (Figure 10 in the appendix) also presents 'single generator collapse' issue. Note that, mode collapse also happens when baseline models trained with other GAN objectives.

Figure 4: Images generated by ProbGAN trained on ImageNet (left) and STL-10 (middle, right).Figure 4(c) are cherry-picked synthetic images on STL-10.

Figure 4(a) and Figure 4(b) display images generated by our model trained on ImageNet and STL-10.Each row in these two figures contains generated images from one generator. The results show that our method is robust to 'single generator mode collapse' in both STL-10 and ImageNet. We also include cherry-picked results on STL-10 to exhibit the capability of our model to generate visually appealing images with complex details while improving the robustness against mode collapse.

rate η, SGHMC noise factor α, number of updates in SGHMC procedure L, number of updating iterations T . for t = 1 to T do for

Figure 5: The l 1 distances between model generated data distribution and target data distribution at each iteration step. Results of model A, B, C, D are denoted by different colors.

Figure 8: Visualization of the projected hit sets of different models trained with the LSGAN objective.Three probabilistic models performs perfectly in this case, while both the two optimization-based methods miss one mode of the true distribution. This experiment illustrates that although MGAN employs an additional classifier to force the data generated by different generators to be disjoint, it still suffers from mode collapsing problem. This is because in MGAN, generators may still generate disjoint data samples in the same mode and fail in capturing other modes.

Figure 12: Images generated by ProbGAN-PSA model trained on CIFAR-10 with LSGAN objective.Images in different rows are generated by different generators.

Figure 13: Images generated by ProbGAN-PSA model trained on CIFAR-10 with GAN-MM objective.Images in different rows are generated by different generators.

Figure 14: Images generated by ProbGAN-PSA model trained on CIFAR-10 with WGAN objective.Images in different rows are generated by different generators.

Figure 15: Images generated by Bayesian GAN trained on CIFAR-10 with GAN-NS objective.Images in different rows are generated by different generators.

Figure 16: Images generated by Bayesian GAN trained on CIFAR-10 with LSGAN objective.Images in different rows are generated by different generators.

Figure 17: Images generated by Bayesian GAN trained on CIFAR-10 with GAN-MM objective.Images in different rows are generated by different generators.

Figure 18: Images generated by Bayesian GAN trained on CIFAR-10 with WGAN objective.Images in different rows are generated by different generators.

Figure 19: Images generated by MGAN trained on CIFAR-10 with GAN-NS objective.Images in different rows are generated by different generators.

Figure 20: Images generated by MGAN trained on CIFAR-10 with LSGAN objective.Images in different rows are generated by different generators.

Figure 21: Images generated by MGAN trained on CIFAR-10 with GAN-MM objective.Images in different rows are generated by different generators.

Figure 22: Images generated by MGAN trained on CIFAR-10 with WGAN objective.Images in different rows are generated by different generators.

Figure 23: Images generated by DCGAN trained on CIFAR-10 with GAN-NS objective.Images in different rows are generated by different generators.

Figure 24: Images generated by DCGAN trained on CIFAR-10 with LSGAN objective.Images in different rows are generated by different generators.

Figure 25: Images generated by DCGAN trained on CIFAR-10 with GAN-MM objective.Images in different rows are generated by different generators.

Common GAN objective functions.

Table content: <table/>

Hit ratios (H r ), hit distances (H d ), cover errors (C e ) results. Note, if the model failed to capture all the modes of real data, by definition its cover error is ∞. In that case, we report the averaged KL-divergence on modes captured by the model in brackets.

Table content: <table><tr><td/><td colspan="4">Hr (HIGHER IS BETTER), Hd (LOWER IS BETTER)</td><td/><td colspan="2">Ce (LOWER IS BETTER)</td><td/></tr><tr><td/><td>GAN-MM</td><td>GAN-NS</td><td>WGAN</td><td>LSGAN</td><td colspan="3">GAN-MM GAN-NS WGAN</td><td>LSGAN</td></tr><tr><td>GAN</td><td colspan="4">0.86, 22.6 0.85, 23.1 0.78, 26.7 0.74, 23.1</td><td>12.11</td><td>8.86</td><td>7.20</td><td>∞ (12.07)</td></tr><tr><td>MGAN</td><td colspan="4">0.82, 24.2 0.84, 25.5 0.67, 31.7 0.81, 23.6</td><td>5.46</td><td>6.31</td><td>5.00</td><td>∞ (4.25)</td></tr><tr><td>BGAN</td><td>1.0, 5.5</td><td>1.0, 6.4</td><td>1.0, 12.1</td><td>1.0, 6.3</td><td>∞ (1.73)</td><td>1.76</td><td>4.32</td><td>1.80</td></tr><tr><td>PROBGAN-GMA</td><td>1.0, 7.4</td><td>1.0, 7.7</td><td>1.0, 15.5</td><td>1.0, 5.3</td><td>1.84</td><td>1.73</td><td>3.01</td><td>1.79</td></tr><tr><td>PROBGAN-PSA</td><td>1.0, 5.8</td><td>1.0, 6.4</td><td>1.0, 12.5</td><td>1.0, 6.4</td><td>1.75</td><td>1.75</td><td>2.28</td><td>1.74</td></tr></table>

Inception score and FID results on CIFAR-10. Results of each model trained with 4 different GAN objectives are all reported.

Table content: <table><tr><td/><td colspan="4">INCEPTION SCORES (HIGHER IS BETTER)</td><td/><td colspan="2">FIDS (LOWER IS BETTER)</td><td/></tr><tr><td/><td colspan="8">GAN-MM GAN-NS WGAN LSGAN GAN-MM GAN-NS WGAN LSGAN</td></tr><tr><td>DCGAN</td><td>6.53</td><td>7.21</td><td>7.19</td><td>7.36</td><td>35.57</td><td>27.68</td><td>28.31</td><td>29.11</td></tr><tr><td>MGAN</td><td>7.19</td><td>7.25</td><td>7.18</td><td>7.34</td><td>30.01</td><td>27.55</td><td>28.37</td><td>30.72</td></tr><tr><td>BGAN</td><td>7.21</td><td>7.37</td><td>7.26</td><td>7.46</td><td>29.87</td><td>24.32</td><td>29.87</td><td>29.19</td></tr><tr><td>PROBGAN-PSA</td><td>7.75</td><td>7.53</td><td>7.28</td><td>7.36</td><td>24.60</td><td>23.55</td><td>27.46</td><td>26.90</td></tr></table>

Inception score and FID results on STl-10 and ImageNet. Each model is trained with GAN-NS objective.

Table content: <table><tr><td>DATASET</td><td>STL-10</td><td/><td>IMAGENET</td><td/></tr><tr><td/><td>INCEPTION SCORES</td><td>FIDS</td><td>INCEPTION SCORES</td><td>FIDS</td></tr><tr><td>DCGAN</td><td>8.05 ± 0.101</td><td>51.01</td><td>7.66 ± 0.113</td><td>48.99</td></tr><tr><td>MGAN</td><td>8.72 ± 0.096</td><td>51.56</td><td>7.77 ± 0.108</td><td>45.75</td></tr><tr><td>BGAN</td><td>8.84 ± 0.100</td><td>47.35</td><td>8.52 ± 0.075</td><td>29.68</td></tr><tr><td>PROBGAN-PSA</td><td>8.87 ± 0.095</td><td>46.74</td><td>8.57 ± 0.073</td><td>27.69</td></tr></table>

Peer review (iclr2019):
 PRIOR COMMENT: This paper should be rejected based on the experimental work. Experiments need to be reported for larger datasets. Note the MGAN paper reports results on STL-10 and ImageNet as well. NOTE: this was addressed by the 27/11 revision, which included good results for these other data sets, thus I now withdraw the comment Note, your results on CIFAR-10 are quite different to those in the MGAN paper. Your inceptions scores are worse and FIDs are better!! I expect you have different configurations to their paper, but it would be good for this to be explained. NOTE: explained in response! NOTE: this was addressed by the 27/11 revision I thought the related work section was fabulous, and as an extension to BGAN, the paper is a very nice idea. So I benefited a lot from reading the paper. I have some comments on Bayesian treatment. In Bayesian theory, the true distribution cannot appear in any evaluated formulas, as you have it there in Eqn (1) which is subsequently used in your likelihood Eqn (2). Likelihoods are models and cannot involve "truth". Lemma 1: Very nice observation!! I was trying to work that out, once I got to Eqn (3), and you thought of it. Also, you do need to explain 3.2 better. The BGAN paper, actually, is a bit confusing from a strict Bayesian perspective, though for different reasons. The problem you are looking at is not a time-series problem, so it is a bit confusing to be defining it as such. You talk about an iterative Bayesian model with priors and likelihoods. Well, maybe that can be *defined* as a probabilistic model, but it is not in any sense a Bayesian model for the estimation of . NOTE: anonreviewer2 expands more on this What you do with Equation (3) is define a distribution on and (which, confusingly, involves the "true" data distribution ... impossible for a Bayesian formulation). You are doing a natural extension of the BGAN papers formulation in their Eqs (1) and (2). This, as is alluded to in Lemma 1. Your formulation is in terms of two conditional distributions, so conditions should be given that their is an underlying joint distribution that agrees with these. Lemma 1 gives a negative result. You have defined it as a time series problem, and apparantly one wants this to converge, as in Gibbs sampling style. Like BGAN, you have just arbitrarily defined a "likelihood". To me, this isn*t a Bayesian model of the unsupervised learning task, its a probabilistic style optimisation for it, in the sense that you are defining a probability distribution (over and ) and sampling from it, but its not really a "likelihood" in the formal sense. A likelhood defines how data is generated. Your "likelihood" is over model parameters, and you seem to have ignored the data likelihood, which you define in sct 3.1 as . Anyway, I*m happy to go with this sort of formulation, but I think you need to call it what it is, and it is not Bayesian in the standard sense. The theoretical treatment needs a lot of cleaning up. What you have defined is a probabilistic time-series on and . Fair enough, thats OK. But you need to show that it actually works in the estimation of . Because one never has , all your Theorem 1 does is show that asymptotically, your method works. Unfortunately, I can say the same for many crude algorithms, and most of the existing published work. Thus, we*re left with requiring a substantial empirical validation to demonstrate the method is useful. Now my apologies to you: I could make somewhat related statements about the theory of the BGAN paper, and they got to publish theirs at ICLR! But they did do more experimentation. Oh, and some smaller but noticable grammar/word usage issues. NOTE: thanks for your good explanation of the Bayesian aspects of the model ... yes I agree, you have a good Bayesian model of the GAN computation , but it is still not a Bayesian model of the unsupervised inference task. This is a somewhat minor point, and should not in anyway influence worth of the paper ... but clarification in paper would be nice.