LDMGAN: Reducing Mode Collapse in GANs with Latent Distribution Matching
Authors: Zhiwen Zuo
Lei Zhao
Huiming Zhang
Qihang Mo
Haibo Chen
Zhizhong Wang
AiLin Li
Lihong Qiu
Wei Xing
Dongming Lu
Venue:  (2020)
Keywords: Deep Learning
Unsupervised Learning
Generative Adversarial Networks
Mode Collapse
AutoEncoder

Abstract: Generative Adversarial Networks (GANs) have shown impressive results in modeling distributions over complicated manifolds such as those of natural images. However, GANs often suffer from mode collapse, which means they are prone to characterize only a single or a few modes of the data distribution. In order to address this problem, we propose a novel framework called LDMGAN. We ﬁrst introduce Latent Distribution Matching (LDM) constraint which regularizes the generator by aligning distribution of generated samples with that of real samples in latent space. To make use of such latent space, we propose a regularized AutoEncoder (AE) that maps the data distribution to prior distribution in encoded space. Extensive experiments on synthetic data and real world datasets show that our proposed framework signiﬁcantly improves GAN’s stability and diversity.

Under review as a conference paper at ICLR 2020
LDMGAN: REDUCING MODE COLLAPSE IN GANS WITH LATENT DISTRIBUTION MATCHING
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) have shown impressive results in mod- eling distributions over complicated manifolds such as those of natural images. However, GANs often suffer from mode collapse, which means they are prone to characterize only single or a few modes of the data distribution. In order to address this problem, we propose novel framework called LDMGAN. We first introduce Latent Distribution Matching (LDM) constraint which regularizes the generator by aligning distribution of generated samples with that of real samples in latent space. To make use of such latent space, we propose regularized Au- toEncoder (AE) that maps the data distribution to prior distribution in encoded space. Extensive experiments synthetic data and real world datasets show that our proposed framework significantly improves GAN's stability and diversity.
INTRODUCTION
Generative models Smolensky 1986 Salakhutdinov & Hinton 2009 Hinton & Salakhutdinov
2006 [Hinton| 2007 Kingma & Welling 2013 Rezende al. 2014 Goodfellow et al. 2014) pro-
vide powerful tools for unsupervised learning of probability distributions over difficult manifolds such as those of natural images. Among these models, instead of requiring explicit parametric specification of the model distribution and a likelihood function, Generative Adversarial Networks (GAN) (Goodfellow et al., 20144 only have a generating procedure. They generate samples that are sharp and compelling, which have gained great successes on image generation tasks (Denton et al.) 2015 Radford et al. 2015 Karras et al. 2017 Zhang et al. 2018) recently. GANs are composed of two types of deep neural networks that compete with each other: generator and a discriminator. The generator tries to map noise sampled from simple prior distribution which is usually a multi- variate gaussian to data space with the aim of fooling the discriminator, while the discriminator learns to determine whether a sample comes from the real dataset or generated samples.
In practice, however, GANs are fragile and in general notoriously hard train. On the one hand, they are sensitive to architectures and prper-parameters (Gooodfeloow et al. 2014)). For example, the imbalance between discriminator and generator capacities often leads convergence issues. On the other hand, there is a common failure issue in GANs called mode collapse. The generator tends to produce only single sample or few very similar samples in that case, which means GANs put large volumes of probability mass onto a few modes.
We conjecture the mode missing issue GANs is probably because GANs lack a regularization term that can lead the generator to produce diverse samples. To remedy this problem, in this work, we first propose a regularization constraint called Latent Distribution Matching. suppresses the mode collapse issue in GANs by aligning the distributions between true data and generated data in encoded space. To obtain such encoded space, we introduce a regularized autoencoder which maps data distribution to a simple prior distribution, eg.. a gaussian. As shown in Figure I| we collapse the decoder of the regularized AE and generator of GAN into one and propose LDMGAN, Our frame- work can stabilize GAN's 's training and reduce mode collapse issue in GANs. Compared to other AE-based methods on 2D synthetic, MNIST, Stacked-MNIE CIFAR-10 10 and CelebA datasets, our method obtains better stability, diversity and competitive standard scores.

---

Under review as a conference paper at ICLR 2020
2
BACKGROUND
GANs were initially proposed by Goodfellow et al. (2014). They contain two neural networks: generator and a discriminator. Let { J.2 denote the training data, where each æ; RD is drawn from unknown data distribution (a)).. Generator is a neural network G4 that maps the noise vector z E IR typically drawn from a multi-variate gaussian, to data space a E RD. The discriminator Da is classification network that distinguishes real samples from generated samples. The parameters of these networks are optimized by solving the following minmax game:
min max V(G, D) = Emp log D(x)) E2 log(1 - D(G(z)))
where Ex indicates an expectation over the data distribution pa(æ)) and Ez indicates an expec- tation over the prior noise distribution p(z). Given generator G, the optimal discriminator D is pa(æ)
(z(=) p=(x))+,((a)> where (z( is the model distribution. With the help of the optimal dis- criminator, the GAN objective is actually minimizing the Jensen Shannon divergence between the model distribution and the data distribution. And the global minimum is achieved if and only if Pg (z( = (m)).
3 RELATED WORKS
There are many works that try to stabilize GAN's training and alleviate mode collapse in GANs. DCGAN (Radford al. 2013) designed a class of deep convolutional architectures which has be- come standard architecture for training GANs. Improved GANs (Salimans et al.) (016) proposed several techniques like feature matching, mini-batch discrimination and historical averaging which stabilized GAN's training and reduced mode collapse. Unrolled GAN (Metz et al., 2016) proposed unrolling optimization of the discriminator objective train generator. TTUR (Heusel et al. 2017) introduced two time-scale update rule and proved its convergence to local Nash equilibrium. Ar- jovsky al. (2017) analyzed the convergence properties of GANs and proposed WGAN which leveraged the Wasserstein distance and demonstrated its better convergence than Jensen Shannon divergence. However, WGAN required that the discriminator must lie on the space of 1-Lipschitz functions. It used a weight clipping trick to enforce that constraint. WGAN-GP stabilized WGAN by alternating the weight clipping by penalizing the gradient norm of the interpolated samples. SN- GAN (Miyato et al. 2018) proposed spectral normalization which controls the Lipschitz constraint of netowork layers.
Some works make efforts towards integrating AEs into GANs. AEs can be used in discrimina- tors. EBGAN (Zhao et al. 2016) employed an autoencoder structure in its discriminator and intro- duced repelling regularization prevent mode collapse. BEGAN (Berthelot al. 2017) extended EBGAN by optimizing Wasserstein distance between AE loss distributions of real and generated samples. MDGAN (Che et al. 2016) contained manifold step and diffusion step that combined plain autoencoder with GANs to suppress mode missing problem. However, MDGAN required two discriminators and it did not impose regularization on its AE. VAEGAN (Larsen al. 2015 unified VAE and GAN into one model, and utilized feature-wist wise distance in intermediate features learned by discriminator to replace similarity metric in data space VAE to avoid blur. AAE (Makhzani et al. 2015) used adversarial learning in its encoded space to perform variational inference. BiGAN (Donahue al., 2016) and Adversarial Learned Inference (ALI) (Dumoulin et al., (016) jointly trained an inference model and generative model through adversarial game. VEEGAN (Srivastaval ct al. 2017) employed a reconstructor network to both map the true data distribution pa( to a gaussian and to approximately invert the generator network. AGE (Ulyanov et al. 2018) performed adversarial game between a generator and an encoder, and proposed to align the model distribution with data distribution in encoded space. They eliminated discriminator in their framework and used an encoder to extract statistics to align model distribution with data distribution.
The most related works to ours are VAEGAN and VEEGAN. VAEGAN's motivation was combine VAE with GAN, and they relied on an ELBO, namely E,(x æe) [logg(///)) p(x|=)] - DKL (<<])))))))))
While we perform marginal distribution matching, namely (x(=(=) log p [ DKL(q (()))))((()) and view the second term as regularization term. Therefore there is need of re- parameterization trick in our framework. Also we compare the distributions of the real samples and generated sam- ples in the encoded space to suppress mode collapse in GANs which is different from VAEGAN.
2

---

VEEGAN had reconstructor network which mapped the data distribution gaussian well. However, VEEGAN autoencoded noise vectors rather than data items, and its training objective was formed adversarial game in joint space (a, =) like BiGAN and ALL Our regularized au- toencoder is similar to AAE, But instead of using adversarial learning in encoded space which also explicit divergence align Under review as a conference paper at ICLR 2020
has possibility of falling into mode collapse in data space,
aggregated posterior with the prior distribution.
4
METHOD
GAN's training can be viewed an adversarial game between two players, in which the discrim- inator tries to distinguish between real and generated samples, while the generator tries to fool the discriminator by pushing the generated samples towards the direction of higher discrimination val- ues. However, there are many points data manifold that have high discrimination values. When collapse to single mode is imminent, the gradient of the discriminator may point in similar di- rections for many similar points (Salimans et al.] 2016). Since the discriminator iteself processes each sample independently, there is no mechanism tell the outputs of the generator to become more dissimilar to each other. In this paper, we propose a framework called LDMGAN to address this problem. See Figure for glance of our model's architecture. Instead of exposing the dis- similarity of generated samples to the discriminator (Salimans et al.) 2016) Zhao et al. 2016), we introduce regularized autoencoder in our framework. We perform data dimensionality reduction on the training data to obtain well-shaped manifold in encoded space, e.g.. a factorized gaussian, by learning regularized autoencoder. Then when sampling noise from a factorized gaussian to feed into the generator to produce generated samples, we encode the generated samples to latent space to enforce them be coincident with the factorized gaussian. In this way, the generator receives signal from that constraint to tell it to produce diverse samples. We call this constraint Latent Distribution Matching (LDM). In the following, we describe LDM and the regularized autoencoder in detail.
loss
E
E (x)
G (x))
Z p(z)
A(E()) II 2)
G
A(E G(z) l1
~ (2))
G(z)
E
E o ((z)
fake
D
Teal
Figure 1: Our LDMGAN's architecture. The thick lines denote added losses to GANs,
4.1 LATENT DISTRIBUTION MATCHING
The main idea of LDM is to introduce an encoder function F, that can map the true data distribution p(æ) to a prior Gaussian p(z). To understand why such an encoder helps alleviate address mode collapse, consider the example in Figure 2, The middle vertical panel represents the data space, where this example the true data distribution Pa(x) is a mixture of two Gaussians. The bottom panel depicts the input to the generator (latent code space), which is drawn from p(z), and the top panel depicts the results of applying the encoder network to the generated data and true data. Assume that the encoder approximately maps the data distribution the prior gaussian (see q(2)
3

---

Under review as a conference paper at ICLR 2020
above the top panel in Figure 2). The purple arrows above the middle panel show the action of the encoder on the true data, whereas the green arrows show the same action on the generated data. mode collapse occurs it does in this example that the generator G,, captures only one of the two modes of Pa(æ)). would cause distribution mismatch between the two marginal distributions g(z) and (=). The definitions of (=) and ('() go follow:
((2) = / ((2|2))pp(e))))) (p)
(2)
(z)
(3)
where Po (a) denotes the model distribution of G. in Equation 3 This mismatch is evaluated by in the latent space. We only require this divergence to be non-
divergence measure A negative and zero if and only if the distributions are identical (1)(()))l((z)) = - (z)=) q())). Therefore, this mismatch value function is defined as:
Vi V missmatch ((d(=))ll(zz)
(4)
However, comparing two empirical (hence non-parametric) distributions g'(z) and q(z) is difficult. We avoid this issue by introducing an intermediate reference distribution which also the prior distribution p(2), resulting in:
2 masmatchh = Ald (=)l=))))())) ((o(=)ll((2))
As we have assumed that the encoder network can approximately map the data distribution gaussian, we can get rid of ((g(=)lp((2)) with minor error. Finally, this mismatch term turns out to be
V
mismmtch
It can be used to detect mode collapse in generator. We call this mismatch term as Latent Distribution Matching (LDM), and use this term to regularize the generator in GANs to combat mode collapse.
q' d'(2)
((2)
Pa(x)
Fø
p(z)
G
Figure 2: An example of how LDM helps to address mode collapse. See text for explaination.
4.2 REGULARIZED AUTOENCODER
AutoEncoders have long been used for data dimensionality reduction and feature learning, We im- pose the regularization that the distribution in the encoded space of real data coincident with the prior distribution p(z) on the autoencoder. We call this autoencoder the regularized autoencoer in this paper. There are two ways of utilizing the regularized autoencoder. One is to pretrain a regular- ized autoencoder, then use the encoder of the regularized autoencoder only. The other is combine

---

Under review as a conference paper at ICLR 2020
the autoencoder training with GAN objective which has already been proposed in many GAN vari- ants (Larsen et al., 2015; Che et al. 2016, Srivastava et al., 2017). We combine autoencoder with GANs in our model for three reasons: (1) to guarantee the learning is grounded on all training sam- ples; (2) to regularize the generator in producing samples to resemble real ones; (3) last but not least, obtain encoder that approximately map the data distribution gaussian. In the following, we will explain the reasons for combining autoencoders in our model and describe our framework formally.
4.2.1 REASONS OF COMBINING AUTOENCODERS WITH GANS
One of the reasons that mode collapse occurs is probably that the areas near the missing modes are rarely visited by the generator, therefore providing very few samples to improve the generator around those areas. And only when G(z) is very close the missing modes the generator get gradients to push itself towards the missing modes. Consider a model combining GANs with reconstruction loss in data space a(x,, G(F(a:))). Given enough data examples available in training data for minor mode Mo. we know G o E' is a good autoencoder, then G (xx) will be located very close the minor mode Mo. Therefore the gradient from the discriminator V log DGG E(x)) might be able to push these points towards the minor mode Mg and the learning is grounded on all training
samples. For that purpose, the following loss is added to the generator:
LG A2d(E Go E(x)) log(D(G B(x))))
As the regularized autoencoder combined with GAN has imposed gaussian on the encoded space of real data samples, this guarantees such ((x)) for minor modes can be sampled with fair possibility when sampling z from gaussian that are feeded to the generator.
addition, as lots of image synthesis works (Isola et al. 2017 Zhu al. 2017) have used L' or L loss in data space and achieved successes in producing realistic images. We use the autoencoder loss d(x, G o E(x)) in data space well for the purpose of regularizing the generator in producing samples resemble real ones.
4.2.2 REGULARIZATION ON THE ENCODED SPACE
In this section, we formally demonstrate the regularized autoencoder we combine with GANs. Let be the data sample vector, E be the reconstructed sample vector of æ by the autoencoder and be the latent code vector of the autoencoder. Let p(=) the prior distribution we want to impose on the codes, q ((2(z)) be an encoding distribution and p( [2) be the decoding distribution. Also let pa(x) be the data distribution, and (x) be the model distribution. We desire an autoencoder whose aggregated posterior 7(2) is coincident with prior distribution p(z). In this paper, a factorized gaussian as the prior distribution. The encoding function of the autoencoder g(2(x=) defines the aggregated posterior Equation 2) We choose a deterministic function form for q(z which means the stochasticity of q(z) comes only from the the data distribution pa(x). We impose regularization term on the encoded space of the autoencoder to minimize: A(g(2) (=)()))))).)..))))))))))))))){)) The autoencoder, meanwhile, attempts to minimize the reconstruction error d(x, .) (here d is some similarity metric, besides L or L distance in data space, it can also be distance metric in the reature-wise space like the intermediate layers of well-known VGG (Simonyan & Zisserman 2014) network or the discriminator of GANs).
Therefore, the regularized autoencoder is trained in two phases: the reconstruction phase and the reg- ularization phase. In the reconstruction phase, the regularized autoencoder uses SGD to update the encoder and decoder's parameters to reconstruct the input data vector. In the regularization phase, the autoencoder uses SGD to update the encoder's 's parameters to match the aggregated posterior with the prior distribution. Implemented by deep neural networks, with arbitrary capacity, the universal approximation theorem (Hornik et al.) 1989 Cybenkoo] 1989) guarantees the regularized autoencoder is capable of approximately matching the aggregated posterior q(z) with the prior distribution p(z).
5

---

Under review as a conference paper at ICLR 2020
4.3 TRAINING OBJECTIVE
Given the encoder Es which is capable of approximately mapping the data distribution to a gaussian, consider a standard GAN training loss plus the LDM (Latent Distribution Matching) constraint on the generator:
min max G) = En log ((x) + Ez log(1 + A Gy
Dw
This training objective E is what we apply in GAN framework when not concerning com- bining the regularized autoencoder into GAN, And after unifying the regularized autoencoder and GAN described above, the objective becomes:
min max VDD,G. G, F. = E log ((x) + log(1 - D(G o E(x)))) + Eg log(1 - D(G(z)))
G. Fi D2
+ A1d(x, G B(+))] +
where Å and A2 are hyper-parameters that control the loss weights of reconstruction error and LDM constraint respectively. Note that the LDM term Agq' (=)][p(=)) is minimized only with respect to G,,, which means the parameters of the encoder E' are fixed during backpropagating the LDM term. The divergence measure between the empirical distribution and the prior distribution in the latent M-dimensional space that we use in this paper goes as follow:
M log o;
M
+
DKL (QIANO I)) M Σ
(10)
where 14; and a; are the means and the standard deviations of the fitted empirical distribution Q along various dimensions. The training algorithm and architecture of our proposed model is depicted in Algorithm and Figure resseettvvlly
Algorithm Our LDM-GAN's training algorithm
1: Da, Eo, G. + initialize network parameters respectively
repeat
3: X f- Random mini-batch of m data points from dataset
4: 7 +- Random samples from prior distribution p(2)
5:
Ž < Ex(X)
X + G-(Z))
7:
Xrec G2(Z)
8:
Zrec + Ex
9:
11 Training discriminator D
10:
gw + -- |log (x) + log(1 D2(X)) + +log(1 (X
11:
11 Training encoder E
12:
9o <- Vod(X, Xrec) DKL (xa(((2)(l((2)))
13:
Training generator G
14:
9- &- V. Volo((1 (x)( log(1 Du Aid(X, Xree) + (Zreec(
Using SGD to update model's 's parameters
16:
w +- -g-: 0 +- 796: & 7/-
17:
until convergence
18:
return D,
5 EXPERIMENTS
In this section, compare model with four different models 2D Synthetic, MNIST, Stacked- MNSIT. CIFAR- 10 and CelebA datasets. We do not finetune the hyper-parameters X and X2 in our model, and by default set A 1 and Xa

---

Under review as a conference paper at ICLR 2020
2D Grid
2D Ring
Modes
% High Quality Samples
Modes
% High Quality Samples
(Max 25)
(Max 8)
GAN
3,4 = 0.70
11.10 0.02
I 0.00
99.96 = 0.01
BiGAN
20.4 1044:..1 2.18
66,45 0.11
0.93
38.07 = 0.07
VEEGAN
1 0.00
99.45 +.45+0.0 0.01
1 0.00
99.89 + 0.01
VAEGAN
8.9 -99+0.33 0.33
98.60 + 0.01
1 0.00
100.00 + 0.00
LDMGAN
23.1 = 1.05
15.36
8.00
97.86 + 0.01
Table 1: Modes captured and percentage of high quality sample on 2D synthetic data. Our model consistently captures the highest number of modes and produces competing percentage of high qual- ity samples.
5.1 SYNTHETIC 2D DATA
Evaluation of mode collapse in GANs is very difficult especially when trained on natural images. However, the missing modes can be calculated precisely when using synthetic data for training. In this section we compare five different models on two synthetic datasets: mixture of eight 2D Gaussian distributions arranged in a ring, and a mixture of twenty-five 2D Gaussian distributions ar- ranged in grid. The covariance matrices and centroids have been chosen such that the distributions exhibit lots of modes separated by large low-probability regions, which make them decently hard tasks despite the 2D nature of the dataset.
All models share the same network architectures for fair comparison. The encoder and the generator network are consisted of two hidden layers, while the discriminator network only has one hidden layer. All networks are implemented by fully-connected layers and each layer has 64 neurons. The dimension of the prior input is 2. For VEEGAN and BiGAN, the discrininator's input augumented by the size of the latent code. Note that the VAEGAN we used here is slightly different from the original paper, since the network architectures are shallow and the data is low-dimensional: we used L loss on the data space for simplicity instead of on features learned by the discriminator.
To quantify the mode collapse behavior we report two metrics: the number of modes that each model captures and the percentage of high quality samples. We draw 2048 samples from each model for evaluation. The high quality samples are defined as the points which are within three standard deviations of the nearest mode, and if there are more than twenty such samples registered to a mode, the mode is considered as captured. The results are depicted in Table Tand the numbers are averaged over eight runs. We also show the generated distribution of each model in Figure 3
As we can see from Table I our model captures the greatest number of modes on both synthetic datasets. The generator learned by our model is sharper and closer to the true distribution as shown in Figure 3] Though other models may obtain greater percentage of high quality samples, for ex- ample, vanilla GAN generates 99.96% high quality samples on 2D Ring dataset. However, it runs into severe mode missing problem. Vanilla GAN only captures one mode on 2D Ring dataset I VEEGAN is able to capture most of the modes in
settings (LDMGAN also performs well as VEEGAN under those settings), but it fails to generalize well under our settings. In particular, while our model resembles VABGAN in some ways, VAEGAN suffers from some degree of mode missing problem as well and it is far from competitive compared to our proposed model.
5.2 EXHAUSTIVE GRID SEARCH ON MNIST
In order to systemically explore the effect of our proposed LDMGAN on alleviating mode collapse in GANs, we use large scale grid search of different GAN hyper-parameters on the MNIST dataset. We use the same hyper-parameters settings for both vanilla GAN and our LDMGAN, and the en- coder in our model is the "inverse" of the generator. The search range is listed in Table 2, Our grid search is similar to those proposed in Che et al.] (2016); Zhao et al. (2016). Please refer to it for detailed explanations regarding these hyper-parameters.
Note that under certain settings of parameters, vanilla GAN can also recover all modes 2D Ring and 2D Grid datascts.
7

---

Under review as a conference paper at ICLR 2020
(a) Data
(b) GAN
(c) VAEGAN
(d) VEEGAN
(e) BiGAN
LDMGAN
(2) True Data
(h) GAN
(i) VAEGAN
( VEEGAN
(k) BiGAN
() LDMGAN
Figure 3: Density plots of the true data and generator distributions from different GAN methods
trained on mixtures of Gaussians arranged in a ring (top) or grid (bottom).
LLayerG
[2,3,4]
nLayerD
[2,3,4]
sizeG
sizel)
(256,512,1024
dropoutD
False]
optimG
[dGD.Adam]
optimD
[SGD,Adam]
[Ie-2.le33IIe-]
Table 2: Grid Search for Hyper-parameters
In order to estimate both the missing modes and the sample qualities, we first train a regular CNN classifier the MNIST digits, and then apply it compute the MODE Score which is proposed in Che et al. (2016). MODE Score is variant of Inception Score (IS) (IS)SSlaa et al. 2016) and its definition goes follow:
exp (E (ply)))) DKL(P" (y (wl(p()))))
(11)
where p(y a:) is the softmax output of trained classifier of the labels, and (y) and p(y) are the marginal distributions of the label of the generated samples and training samples respectively. We train each architecture for 50 epochs and after that draw 10K samples for evaluation. The resulting distribution MODE Score shown in Figure 4 It can be seen that our model clearly improves the sample quality and diversity compared to GANs.
5.3 STACKED MNIST
Following Metz et al. 2016), we evaluate our method on the stacked MNIST dataset, a variant of the MNIST data specifically designed to increase the number of discrete modes. The data synthesized by stacking three randomly sampled MNIST digits along the color channel resulting in a 28 x 28 x image. We now expect 1000 modes on this dataset, corresponding to the number of possible triples of digits.
We generate dataset of 50000 images for training. And for fair comparation we use the same architecture for all models. The generator and discriminator networks are implemented following DCGAN. For encoder network, we use a simple two layer MLP network without any regularization layers. For BiGAN and VEEGAN, with the latent code which follows by two fully-connected layers in order to perform joint space discrimination. The dimension of the prior input is 64, Each model is trained for 50 epochs.
concatenate the last convolutional layer of the discriminator
As the true locations the modes in this data are unknown, the number of modes are estimated using trained classifier described in subsection 5.2) We draw 26000 samples from each model for
8

---

Under review as a conference paper at ICLR 2020
248
144
Figure 4: The distribution of MODE scores for GAN and our LDM-GAN. Higher MODE Score is better.
8
$
9
4
8
9 7
7
3
2
2
7
9 4
8
6
9
4
f
5
9 2
4
2
4
2
0
S
8
?
7
8
5
2
8
@
2
8
69
3
7
6
f
8
2
5
9
9
f
3
(a) DCGAN
(b) BiGAN
(c) VEEGAN
(d) VAEGAN
(c) LDMGAN
Figure 5: Random samples of different methods on Stacked-MNIST. Zoom in to see details.
evaluation. As a measure of quality, following Metz et al. (2016), we also report the KL divergence between the model distribution and the data distribution. The results averaged over 5 runs. As reported in Table 3, our model recovers greatest number of modes and better matches the data distribution than any other methods.
Modes
KL
(Max 1000)
Divergence
DCGAN
165,0 49.06
3.36 = 0.21
BIGAN
135.6 - 89.64
3.22 0.23
VEEGAN
178.8 48.07
3.35 0.32
VAEGAN
206.4 + 32.62
3.21 0.34
LDMGAN
988.8 + 2.40
0.15 0.06
Table Degree of mode collapse, measured modes captured and sample quality (as measured by KL) on Stacked-MNID Our model captures the most modes and also achieves the highest quality.
5.4 CIFAR-10 AND CELEBA
For CIFAR-10 and CelebA datasets, we measure the performance with FID scores (Heusel et al., 2017]. FID can detect intra-class mode dropping, and measure the diversity as well as the quality of generated samples. FID computed from 10K generated sampels and the pre-calculated statistics on all training data provided in the code repository Our default parameters are used for all exper- iments. And the model architecture is the same as the standard DCGAN. We train each model for CelebA. The dimension of the prior input is 100. All of
50 epochs on CIFAR-10 10 and 30 epochs our experiments are conducted using the unsupervised setting. We evaluate the FID score for each model per epoch during training, and leave the best per run. As and BiGAN on these datasets using the same architecture, we do not report FID scores on these two models.
reproduce VEEGAN
Phans://itthbb.cm/mminfff//////

---

Under review as a conference paper at ICLR 2020
The reported results are depicted in Table 4 and averaged over 5 runs. It is important to mention that we do not pursue the state-of ofth--of FID scores on both datasets as we only want to prove that model improves GANs. It can be seen that model has both the best FID scores and averaged FID scores on CIFAR-10 and CelebA. The improvement on CelebA is smaller compared to that CIFAR-10. We suspect that this is due to the less complicated structure of face images, but anyway our model still consistently performs better than DCGAN and VAEGAN, The random samples generated from different models on both datasets are shown in Appendix A] Finally, to show that our proposed model are not just simply memorizing the training data, we interpolate between random vectors on generator of our model trained on CelebA dataset and the results are depicted in Appendix A
CIFAR-I0
CelebA
best FID
FID
best FID
FID
DCGAN VAEGAN
51.19
51.79 + 0.38
14.28
14.82 + 0.88
49.50
50.82 + 0.71
13.88
14.98 + 0.77
LDMGAN
44.87
47.02 + 1.18
12.79
13.32 + 0.45
Table 4: FID scores for different models, lower is better. Best FID is the best run in 5 trial.
CONLUSION
We propose robust AE-based GAN model with novel Latent Distribution Matching constraint, called DMMGAN, that can address the mode collapse effectively. Our model different from pre- vious works: (1) We propose novel regularization constraint called Latent Distribution Matching to align the distributions of real data and generated data in encoded space. (2) We propose a new regularized autoencoder for approximately mapping data distribution a novel AE-based GAN that drastically alleviate mode collapse in GANs. Extensive experiments demonstrate that our method is stable and does not suffer from mode collapse problem in Synthetic 2D, MNIST, Stacked-MNIS" datasets. Furthermore, we achieve better FID scores on CIFAR-10 and CelebA datasets compared to baseline models without any other tricks and techniques. These demonstrate that our proposed LDMGAN indeed improves GAN's stability and diversity.
gaussian. (3) We designed
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan.
arXiv preprint
arXiv: 07875, 2017.
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative ad- versarial networks. arXiv preprint arXiv: 2017.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. arXiv preprint arXiv: 2016.
George Cybenko. Approximation by superpositions sigmoidal function. Mathematics control, signals and systems, 2(4):303-314, 1989.
Emily L Denton, Soumith Chintala, Rob Fergus, al. Deep generative image models using laplacian pyramid of adversarial networks. In Advances neural information processing systems, pp. 1486-1494, 2015.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1907072. 2016.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606.000 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
10

---

Under review as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Geoffrey Hinton. Learning multiple layers of representation. Trends in cognitive sciences, 11(10): 428 434, 2007.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-577, 2006.
Kurt Homik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-360 -66, 1989.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125- 1134, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Diederik Kingma and Max Welling. 1r21v:1312.6114, 2013.
Auto-encoding variational bayes. arXiv preprint
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther, Autoen- coding beyond pixels using learned similarity metric. arXiv preprint arXiv:1512.09300, 2015.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprint 1rXiv:1511.05666 2015.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohk-Dickstein. Unrolled generative adversarial networks. arXiv preprint 1rXiv:661 1,02633,
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: (rXiv:/802.0595, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint arXiv: 1401.4040 2014.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pp. 148-455. 2009.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Infor- mation Processing Systems, pp. 3308-3318, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. It takes (only) two: Adversarial generator- encoder networks. In Second AAAI Conference on Artificial Intelligence, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv: 1805.0838 2018.

---

Under review as a conference paper at ICLR 2020
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arxiv preprint
2016.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A APPENDIX
(a) DCGAN
(b) VAEGAN
(c) LDMGAN
(d) DCGAN
(c) VAEGAN
(f) LDMGAN
Figure 6: Random samples of different methods. Zoom in to see details.
Figure 7: Generated samples by interpolating between two different vectors (from left to right).
12


Reviews:
[{"review_id": "SyeUi4nhKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "LDMGAN: Reducing Mode Collapse in GANs with Latent Distribution Matching\n\nSummary:\n\nThis paper proposes a modification to the VAE-GAN model where mode coverage is encouraged by passing samples G(Z) through the encoder and minimizing the forward KL between E(G(Z)) and the prior over Z. Results are presented on synthetic MoG datasets, MNIST variants, CIFAR and CelebA.\n\nMy Take:\n\nThis paper’s only point of novelty over a vanilla VAE-GAN implementation is the inclusion of the KL(E(G(Z)) || p) term in the generator loss, which is very similar to the idea behind VEEGAN. The relative novelty over VEEGAN is also limited, the description of the method is exceptionally similar to the description used in the VEEGAN paper (going so far as to copy-paste a figure straight from VEEGAN without attribution), and the comparison to VEEGAN in the related work section is not sufficiently fleshed out. The difference in performance over a vanilla VAE-GAN on CIFAR and CelebA is negligible (6-9% relative reduction in FID on an already weak baseline), so there is no compelling empirical reason to adopt this method. I argue strongly in favor of rejection.\n\nNotes:\n\n-Mode collapse (when many points in z map to an unexpectedly small region in G(z)) is a different phenomenon from mode dropping (when many points in x are not represented in G(z), i.e. no point in z maps to a cluster of x’s, as is the case if e.g. a celebA model generates frowning and neutral faces but no smiling faces). While these phenomena often co-occur (especially during complete training collapse), they are not the same thing, and this paper conflates them throughout. \n\n-For the synthetic dataset examples the comparison against VEEGAN appears to be unfair—it’s one thing to report robustness to hyperparameters, but this seems more like the authors have intentionally picked settings for which VEEGAN happens to fail (by halving the width). If the authors wish to use settings different from the standard ones used by most other papers which test on the MoG datasets they should justify it thoroughly and include this justification in their analysis—do we have a compelling reason to believe that the LDM method is better suited to learning lower capacity models in general?\n\nMinor:\n\nTypos throughout, like “regularized autoencoer.” Please thoroughly proofread your paper for grammar and spelling mistakes.\n\nThere are formatting errors in the PDF, such as at the top of page 8. Please examine your paper for formatting mistakes.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeOJOq45S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript contains typing and grammatical errors. I also think that the presentation is not clear enough. The notation is not clearly defined also. I hope I am not missing it, but the function combination operation is not defined (e.g. f(x) o g(x), to denote f(g(x)), and used a lot. \n\nJust under equation 4, you are saying that comparing empirical distributions is difficult, and hence propose an alternative objective. But you could actually compare the two empirical distributions with a sample based divergence measure such maximum mean discrepancy, or Frechet Inception distance. Why don't you do this? \n\nIn the algorithms section, I don't understand why you don't compare with widely used WGAN-GP model. DCGAN, and VAEGAN are relatively old models. Your only results in Celeb-A are compared against these relatively older models. Also by looking at the provided samples, I think the quality of the results are far away from what is obtained with state of the Generative models currently. \n\nOverall, I think the idea is sound and sensible, but I don't think that results are convincing enough for a conference paper. Furthermore, the writing needs to be improved.  \n\nminor comments: \n\nIn Figure 2, I think you should provide the explanations also in the caption to make the concept easier to follow for the reader. \n\nIn Algorithm 1, I think it would be easier for the reader to follow if you don't use shorthand notations defined in lines 5, 6, 7, 8.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylXNvH1qr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It is argued in this paper that GANs often suffer from mode collapse, which means they are prone to characterize only a single or a few modes of the data distribution. In order to address this problem, the paper proposed a framework called LDMGAN which constrains the generator to align distribution of generated samples with that of real samples in latent space by introducing a regularized AutoEncoder that maps the data distribution to prior distribution in encoded space.\n\nThe major difference of this paper from many traditional GANs is to constrain the distributions of generated data same as distributions of true data in latent space instead of constrain the ability of discriminator. The authors detailed their motivation, the algorithm, and also reported a series of evaluation results on several datasets. \n\nGenerally, this paper was well written. However, this paper has the following major concerns: \n\t（1） Though somewhat new, the novelty of this paper may be incremental to me. It looks like a combination of VEEGAN and AAE.  Though the authors mentioned that VEEGAN autoencoded the noise vectors rather than data items, and AAE exploited the adversarial learning in the encoded space rather than using an explicit divergence,   it appears not significant to me between the proposed model and these two models. At least the authors did not  address sufficiently how significant the proposed method would be.\n\n\t （2） The paper tested the proposed algorithm with a 2D Synthetic dataset. However, I found a lot of discrepancies in the results presented in Table 1 with other published works. The authors show 1 of mode captured on 2D Grid and 2D Ring using the VEEGAN method. However, the VEEGAN paper shows they get 24.6 and 8 on 2D Ring and 2D Grid respectively. Such discrepancies were also observed in Figure 3. These discrepancies must be explained.\n\n\t（3） In Figure 4, the authors showed the distribution of MODE scores for GAN and LDMGAN. From the figure, it seemed that LDMGAN improved the sample quality and diversity compared to GANs, but it is still prone to characterizing only a single or a few modes of the data distribution. In another word,  this may alleviate the problem but may not fully solve the problem.  Another minor point, the coordinate and legend are too small in this figure. It would be better if they become bigger.\n\n\t（4） The results of Table 4 is not convincing because the comparative methods are truly out-of-date. It would be more convincing if more latest methods can be compared with LDMGAN method. Those results reported are far lower than the state-of-the-art performance in these datasets. \n\n         (5) There is a mistake in the second term of equation 11, it should be ⋯〖-D〗_KL (p^* (y)||p(y))  )\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to mitigate mode collapse in GANs by encouraging distribution matching in the latent space. Reviewers 1 and 3 expressed concerns that the methodology is too incremental in the context of the existing literature (VEEGAN, VAE-GAN, AAE). This, combined with the lack of up-to-date baselines, makes it difficult to access the significance of the proposed modifications. The quality and precision of the writing can also be improved to meet the standards of publication at a top-tier conference. 

Decision: Reject