``"Best-of-Many-Samples" Distribution Matching
Authors: Apratim Bhattacharyya
Mario Fritz
Bernt Schiele
Venue:  (2020)
Keywords: Distribution Matching
Generative Adversarial Networks
Variational Autoencoders

Abstract: Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poorer sample quality. Recent works have proposed hybrid VAE-GAN frameworks which integrate a GAN-based synthetic likelihood to the VAE objective to address both the mode collapse and sample quality issues, with limited success. This is because the VAE objective forces a trade-off between the data log-likelihood and divergence to the latent prior. The synthetic likelihood ratio term also shows instability during training. We propose a novel objective with a ``"Best-of-Many-Samples" reconstruction cost and a stable direct estimate of the synthetic likelihood. This enables our hybrid VAE-GAN framework to achieve high data log-likelihood and low divergence to the latent prior at the same time and shows significant improvement over both hybrid VAE-GANS and plain GANs in mode coverage and quality.

Under review as a conference paper at ICLR 2020
"BEST-OF-MANY--AMPPES" DISTRIBUTION DISTRBBUTIO MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) can achieve state-of-the art sample quality in generative modelling tasks but suffer from the mode collapse prob- lem. Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-basee data log-likelihood foreing it to cover all modes, but suf- fer from poorer sample quality. Recent works have proposed hybrid VAE-GAN frameworks which integrate a GAN-based synthetic likelihood to the VAE objec- tive to address both the mode collapse and sample quality issues, with limited success. This is because the VAE objective forces a trade-off between the data log- lieliikei and divergence to the latent prior. The synthetic likelihood ratio term also shows instability during training. We propose a novel objective with (Best-o-MaanySSamples reconstruction cost and stable direct estimate of the synthetic likelihood. This enables our hybrid VAE-GAN framework to achieve high data log- li-lliikeoood and low divergence the latent prior at the same time and shows significant improvement over both hybrid VAE-GANS and plain GANs in mode coverage and quality.
INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow al.) 2014) have achieved stale-of-the-art sample quality in generative modeling tasks. However, GANs do not explicitly estimate the data likelihood. Instead, aims to "fool" an adversary. so that the adversary unable to distinguish between samples from the true distribution and the generated samples. This leads to the generation of high quality samples (Adler & Lunz, 2018 Brock et al., 2019). However, there is no incentive to cover the whole data distribution. Entire modes of the true data distribution can be missed commonly referred to as the mode collapse problem.
In contrast, the Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) explicitly maximize data likelihood and can be forced to cover all modes (Bozkurt et al.) 20, 88 [hhu et al.) 2018)). VAEs enable sampling by constraining the latent space to unit Gaussian and sampling through the latent space. However, VAEs maximize a data likelihood estimate based on the L,/L2 reconstruction cost which leads to lower overall sample quality - blurriness in of image distributions. Therefore, there has been a spur of recent work (Donahue et al.) [017]] [ Larsen et al.) 2016) Rosca et al.) (019) which aims integrate GANs in VAE framework to improve VAB generation quality while covering all the modes. Notably in Rosca et al. (2019), GANs are integrated in a VAE framework by augmenting the L2LL2 data likelihood term in the VAE objective with GAN discriminator based synthetic likelihood ratio term.
Rosca et (2019) reports that in case of hybrid VAE-GANs, the latent space does not usually match the Gaussian prior. This is because, the reconstruction ligeliild in the VAE objective at odds with the divergence to the latent prior (Tabor et al.) 2018) (also in case of alternatives proposed by Makhhaai et al. (2016): Arjovsky et al. (2017). ) This problem is further exacerbated with the addition of the synthetic likelihood term in the hybrid VAE-GAN objective - the encoder/decoder. This leads to the degradation the quality and diversity of samples. Moreover, the synthetic likelihood it is the ratio of outputs of classifier, any instability
However,
is necessary for sample quality but it introduces additional constraints
ratio term is unstable during training in the output of the classifier magnified. We directly estimate the ratio using a network with a controlled Lipschitz constant, which leads to significantly improved stability. Our contributions

---

Under review as a conference paper at ICLR 2020
in detail are, 1. We propose a novel objective for training hybrid VAE-GAN frameworks, which relaxes the constraints on the encoder by giving the encoder multiple chances to draw samples with high likelihood enabling it to generate realistic images while covering all modes of the data distribution, 2. Our novel objective directly estimates the synthetic likelihood term with a controlled Lipschitz constant for stability, 3. Finally, we demonstrate significant improvement over prior hybrid VAE-GANs and plain GANs on highly muti-modal synthetic data, CIFAR-I0 and CelebA
2 RELATED WORK
Generative Autoencoders. VAEs (Kingma & Welling 2014) allow for generation by maintaining a Gaussian latent space. In Kingma & Welling (2014), the Gaussian constraint in applied point-wise and latent representation of each point is forced towards zero. Adversarial Auto-encoders (AAE) (Makhzani al.) 2016) and Wasserstein Auto-encoders (WAE) (Arjovsky al., 2017) tackle this problem by an approximate estimate of the divergence which only requires the latent space to be Gaussian as a whole. But, the Gaussian constraint in (Arjovsky et al. 2017 Kingma Welling 2014] Makhhani et al. 2016 Mahajan et al., 2019) is still at odds with the data log-likelihood. In this work, we enable the encoder to maintain both the latent representation constraint and high data log-likelihood using a novel objective. Furthermore, we integrate GAN-based synthetic likelihood term to the objective to enhance the sharpness of generated images.
Mode Collapse in Classical GANs. The classic GAN formulation (Goodfellow et al. 2014 Radford al. 2016) has several shortcomings importantly mode collapse. Denoising Feature Matching (Warde-Farley & Bengio, (017) deals with the mode collapse by regularizing the discriminator using an auto-encoder. MDGAN (Che et al., 2017 7) uses two separate discriminators and regularizes using a auto-encoder. In EBGAN (Zhao et al. 2017a)) the discriminator is interpreted as energy functional and is also cast in an auto-encoder framework, leading to improvements in semi-supervised learning tasks. BEGAN (Berthelot al. 2017) proposes Wasserstein distance based objective to train such GANs with auto-encoder based discriminators. The proposed approach leads to smoother convergence. InfoGAN (Chen et al., 2016) maximizes the mutual information between small subset of latent variables and observations in a Information Theoretic framework. This leads to disentangled and more interpretable latent representations. PacGAN (Lin et al.) 2018) proposes to deal with the mode collapse problem by using the discriminator to distinguish between product distributions. D2GAN (Nguyen al. 2017) proposes to use two discriminators one for the forward KI. divergence between the true and generated distributions and one for the reverse. BourGAN (Xiao) al. 2018) proposes to learn the distribution of the latent space (instead of assuming Gaussian) which reflects the distribution of the data. In (Srivastava et al. 2017), inverse mapping from from latent to data space is learned and the generator is penalized based on the inverted distribution to cover all modes. Ravuri et al. 2018)) proposes moment matching paradigm different from VAEs or GANs, However, as the presented moment matching network involves an order of magnitude more parameters compared to VAEs or GANs, we do not consider them here. As propose hybrid VAE-GAN framework these techniques can be applied on top to potentially improve results. However, in hybrid VAE-GANs the reconstruction loss already incentivizes the coverage of all modes.
Wasserstein Loss based Formulations. Arjovsky et al. 2017); Gulrajani et al. (2017) proposes GANs which minimize the Wasserstein distance between true and generated distributions. MMyato ct al. (2018) demonstrates improved results by applying Spectral Normalization on the weights In Tran et al. (2018), distance constraints are applied on top. In Adler & Lunz (2018) WGANs were extended to Banach Spaces to emphasize edges or large scale behavior. Orthogonally, Karras al. (018)) focus on progressively learning to use more complex model architectures improve performance. We use the regularization techniques developed for WGANs to improve stability of our hybrid VAE-GAN framework. [Brock et al. (2019) shows very high quality generations at high resolutions but these class conditional. However, diverse class conditional generation is considerably easier as intra-class variability is generally much lower than inter-class variability. Here, we focus the more complex unconditional image generation task.
Hybrid VAE-GANs. In Larsen et al. (2016) a VAE-GAN hybrid is proposed with discriminator feature matching the VAE decoder trained to match discriminator features instead of / reconstruction loss. ALI (Dumoulin et al. 2016) proposes to instead match the encoder and decoder joint distributions with limited success on diverse datasets. BiGAN (Donahue et al. 2017). builds
2

---

Under review as a conference paper at ICLR 2020
upon ALI to learn inverse mappings from the data to the latent space and demonstrate effectiveness on various discriminative tasks. Rosca et al. (2019) extends standard VAEs by replacing the log- likelihood term with hybrid version based synthetic likelihoods. The KL-divergence constraint to the prior is also recast to a synthetic likelihood form, which can be enforced by discriminator (as in Makhzani al. (2066); Tolstikhin et al. (2018). The second improvement is crucial in generating realistic images at par with Blassic/Wasserseein GANs. We further improve upon Rosca et al. (2019) by allowing the encoder multiple chances draw desired samples and enforcing stability enabling it to maintain low divergence to the prior while generating realistic images.
NOVEL OBJECTIVE FOR HYBRID VAE-GANS
We begin with brief overview of hybrid VAE-GANS followed by details of our novel objective.
Overview. Hybrid VAE-GANS Fiiurre 1) are generative models for data distributions x ~ p(() that transform a latent distribution Z ~ p(z) to a learned distribution ~ pe(x) approximating p(x). The GAN (Ga.Dr alone can generate realistic samples, but has trouble covering all modes. The VAE (Ro.Gg.D) can cover all modes of the distribution, but generates lower quality samples overall. VAE-GAN: leverage the strengths of both VAEs and GANs to generate high quality samples while capturing all modes. We begin with a discussion of the prior hybrid VAE-GAN objectives (Rosca) objective with novel reconstruction term and regularized stable direct estimate of the synthetic likelihood.
et al. 2019) and its shortcomings, followed by novel (Best-of-Many-Samee
3.1 SHORTCOMINGS OF HYBRID VAE-GAN ORJECTIVES
Hybrid VAE-GANs (Dumoulin al. 2016 Makhzani et al. 2016 Rosca et al., 2019: Zhao et al. 2017b) maximizes the log-likelihood of the data (x ~ p(x)) akin to VAEs. The log-likelihood, assuming the latent space to be distributed according to p(z),
lx)( = log (f po p=(x))))(2))d)
Here, p(z) is usually Gaussian. This requires the generator Go to generate samples that assign high likelihood to every example x in the data distribution for a likely Z ~ p(z). Thus, the decoder 0 can be forced to cover all modes of the data distribution x p(x). contrast, GANs never directly maximize the data likelihood and there is no direct incentive to cover all modes.
However, the integral in I is intractable. VAEs and Hybrid GAE-SANs use amortized variational inference using recognition network qo (7|x) (Rz). The final hybrid VAE-GAN objective of the state-of-the-art o-GAN (Rosca et al., 2019) which integrates a synthetic likelihood ratio term is,
Di(x)z) - (KL(p(z) || (x(x))))
LaNGAN = A W. (210) log(po(x|z)) + E. (x1x) log 1 -i-(()())
(2)
This objective has two important shortcomings. Firstly, as pointed in (Bhattacharyya et al. 2018 Tolstikhin et al. 2018), this objective severely constrains the recognition network as the average likelihood of the samples generated from the posterior 4 (z)x) ) is maximized. This forces all samples from go (z(x) to explain equally well, penalizing any variance in qs ( x) and thus forcing it away from the Gaussian prior p(z). Therefore, this makes it difficult to match the prior in the latent space and the encoder forced to trade-off between a good estimate of the data log-likeliile log-likelihood and the divergence to the latent prior.
Secondly, the synthetic likelihood ratio term is the ratio of the output of D,, any instability (non- smoothness) in the output of the classifier is magnified. Moreover, there is no incentive for Dj to be smooth (stable). For two similar images, (X1, X2} with [x] X2l I e, the change of output (x, (x, |z2)l can be arbitrarily large. This means that small change in the generator output (e.g. after a gradient descent step) can have a large change in the discriminator output.
Next, describe how we can effectively leverage multiple samples from qo (zix) deal with the first issue. Finally, we derive a stable synthetic likelihood term (Rosca et al. 2019 Wood 2010) to deal with the second issue.
3

---

Under review as a conference paper at ICLR 2020
Best Match
Our "Best of Many"- Objective
max log (x
p(x)
+
R
C'
Dr
Real Fake
X Po ( (2)
GAN
+
VAE
~ (0(2)x)
DL
Real Fake
Figure 1: Overview of our BMS-VAE GAN framework. The terms of our novel objective 7 are highlighted the right. We consider only the best sample from the generator Ge while computing the reconstruction loss.
3.2 LEVERAGING MULTIPLE SAMPLES
Building upon Bhattacharyya et al. 20 we derive an alternative variational approximation of ( which uses multiple samples to relax the constrains on the recognition network (full derivation in Appendix A),
Lms log (f po(x)z)go(((x)de) - (L(gg(z)x) |I p(z)).
(3)
In comparison to the a-GAN objective 2 where the expected likelihood assigned by each sample the data point x was considered, we see that in (3) the likelihood is computed considering all generated samples. The recognition network gets multiple chances to draw samples which assign high likelihood x. This allows (g to have higher variance, helping better match the prior and significantly reducing the trade-off with the data log-likelihood. Next, we describe how we can integrate a synthetic likelihood term in to help us generate sharper images.
3.3 INTEGRATING STABLE SYNTHETIC LIKELIHOOD WITH THE "BEST-OF-MANY" SAMPLES
Considering only L reconstruction based likelihoods (x/z)) (as in Bhattacharyya al. 2018
Kingma & Welling 2014) Tolstikhin et al. 2018) might not sufficient in case complex high dimensional distributions e.g. in case of image data this leads to blurry samples. Synthetic estimates of the likelihood (010) leverages a neural network (usually classifer) which is jointly trained distinguish between real and generated samples. The network is traiend assign low likelihood to generated samples and higher likelihood to real data samples. Starting from (3, we integrate synthetic likelihood term with weight encourage our generator to generate realistic samples. The L2 reconstruction likelihood (with weight a) forces the coverage of all modes. However, unlike prior work (Bhattachrryya al., 2019 Rosca et al.) 2019). our synthetic likelihood estimator D₁ is not classifier. We first convert the likelihood term to likelihood ratio form which allows for synthetic estimates,
Lms =a log (Bao(c)))P + 33 log (Bast())P(((2)) (L(go(z)x) || p(z))
(4)
po (x(2)) o log (())( p(x)
+ 3 log (Bgo(1))P0x(22) - KL(gs(zix) p(z)).
To enable the estimation of the likelihood ratio pe (x(1)//p(s) using neural network, introduce the auxiliary variable where, denotes that the sample was generated and denotes that the sample is from the true distribution. We can now express (4) (using Bayes theorem, Appendix A),
y
-1)) =
log (x)g)
+ log (log
(Fasse)ssPo(())))) (I((o(Z(xx) p(z)).
pxx = 0)
(5)
po (y 1 1|2,x) + B log (Eaofcl))o(((2)) (L(gg(z)x) p()))
=alog (gga(e)s))
1- ply 1[x)
The ratio p=(y=11zx\/11pp<==1 should be high for generated samples which are indistinguishable from real samples and low otherwise. In case of image distributions, find that direct estimation of
4

---

Under review as a conference paper at ICLR 2020
the numerator/denominator (as in Rosca et al. (2019) exacerbates instabilities (non- moootmmothhsss) of the estimate. Therefore, we estimate this ratio directly using the neural network Di(x) trained to produce high values for images indistinguishable from real images and low otherwise.
LMss log (x) + ,3 log (BasC)))PP(x12)) - (KL(s(Z(x) ll p(z)).
(6)
To further unsure smoothness, we directly control the Lipschitz constant K of D,. This ensures, (x X2,
|2,) (x(2|22) I K|x, X2 1il-the the function is strictly smooth everywhere. Small changes in generator output cannot arbitrarily change the synthetic likelihood estimate, hence allowing the generator smoothly improve sample quality. We constrain the Lipschitz constant K to I using Spectral Normalization Miyato et al. (2018). Note that the likelihood py (x(z) takes the form e -Xijx>ill in (6 a log-sum-exp which is numerically unstable. As we perform stochastic gradient descent, can deal with this after stochastic (MC) sampling of the data points. We can well estimate the log-sum-exp using the max the "Best-of- -Best-of-Many-eeee Many-Samples" (Nielsen & Sun 2016).
X2|
log T Σ po(x 2 max moxxlo(pp(xx(2) - log(T)
In practice, we observe that we can improve sharpness of generated images by penalizing generator Gg, using the least realistic of the T samples,
log
Di(x)2)) 2 min log (Di(xlz")
Our final "Best-of-Manyy- objective takes the form (ignoring the constant log(T) term).
LBMS&S a min (Di()(2)) max log (L(gg(z)x) p(z)).
We use the same optimization scheme in Rosca al. 2019). We provide the algorithm in detail in Appendix B.
Approximation Errors. The "Best-of-Many Many-Samples' scheme introduces the log(T)) error term. However, this error term is dominated by the low data likelihood term in the beginning of optimization (Bhattacharyya al. 2018). Later, as generated samples become more diverse, the log likelihood term is dominated by the Best of T samples "Best of Many-Samplee is equivalent.
Classifier based estimate the prior term. Recent work (Makhzani al. 2016 Arjovsky et al. 2017, Rosca et al., (019)) has shown that point-wise minimization of the KL-divergence using its analytical form leads to degradation in image quality. Instead, KL-divergence term recast in synthetic likelihood ratio form minimized "globally" using a classifier instead of point-wise. Therefore, unlike Bhattacharyya et al. (2018). . here we employ a classifier based estimate of the KL-divergence to the prior. However, pointed out by prior work hybrid VAE-GANs (Roscal et al. 2019), a classifier based estimate with still leads to mismatch to the prior as the trade-off with the data log- likelihood still persists without the use of the Saatoof-Man-SSampple Therefore, as we shall demonstrate next, the benefits of using the Meest-of-Many-Samplee extends to case when a classifier based estimate of the KL-divergence is employed.
EXPERIMENTS
Next, we evaluate on multi-modal synthetic data as well as CIFAR-10 and CelebA. We perform all experiments single Nvidia V100 GPU with 16GB memory. We many samples during training as would fit in GPU memory so that we make the same number of forward/backwardd passes other approaches and minimize the computational overhead of sampling multiple samples.
5

---

Under review as a conference paper at ICLR 2020
Table 2: Visualization of samples.
Table 1: Evaluation on multi-modal synthetic data.
Target
WAE
o-
BMS-VAE-GAN
2D (25 modes)
2D Ring (8 modos)
Modes
HQS
Modes
HQ'
VERGAN Stivattvs
24.6
GDFP GAN FiRcE.
Aug
24,8
68.5
71.2
SN-GAN Mivanr Fal pir 2lnd
238+11.5
90.0+4.0
6.8+1.1
MD-GAN Lupeer Zareh orar 2019
25
99.3122.2
89.0.33.6
WAE Astevsky el 2017
25
65.4+338
35.8+4.77
a-GAN
2017
25
70.5:443
83.-1..3
BMS
99.720.2
9.66:0.3
WAE
o-GAN
BMS- VAE-GAN
WAE
o-GAN
BMS-VAE-GAN
Latent space samples 2, (x)x) & p(c)
Latent space
samples (x) & p(x)
Conrsponding data space Pe (x(2)
Conrepanding data space samples. Pg (x]2)
Table 3: Effect of our novel objective in the latent space. Top Row: The standard WAE and a-GAN objectives leads to mismatch to the prior in the latent space. We show samples z (in red) which are highly likely under the standard Gaussian prior (blue) but have low probability under the learnt marginal posterior do (z). Bottom Row: We show that such points z lead to low quality data samples (in red), which do correspond any of the modes.
4.1 EVALUATION ON MULTI-MODAL SYNTHETIC DATA.
We evaluate in Tables and 2 on the standard 2D Grid and Ring datasets, which are highly challenging due to their multi-modalin The metrics considered are the number of modes captured and % of high quality samples (within standard deviations of mode). The generator/diseriinaato architecture is same in Srivasstaa al. (2017). We see that our BMS-VAE- VAE-GAN (using the best of T = 10 samples) outperforms state of the art GANs e.g. (Eghbal-zadch et al. 2019) and the WAE and a-GAN and the
baselines. The explicit maximization of the data likeliioddo enables our BMS-VAE-GA WAE and a-GAN baselines to capture all modes both the grid and ring datasets. The significantly increased proportion of high quality samples with respect to WAE and o-GAN baselines is due to our novel Trest-of-Many-Samllee
objective. We illustrate this in Table 3 Following Rosca et al. (20T9) we analyze the learnt latent spaces in detail, in particular we check for points (in red) which are likely under the Gaussian prior p(z) (blue) but have low probability under the marginal posterior (z) 9o (z)xx)d.. We use (SNE to project points from our 32-dimensional latent space to 2D. In
Table 3 (Top Row) we clearly see that there are many such points in case of the WAE and a- a-GAN baselines (note that this low probability threshold common across all methods). In Table (Bottom Row) we see that these points lead to the generation of low quality samples (in red) in the data space. Therefore, we see that our "Best-of-Many Many-Samples" samples objective helps us match the prior in the latent space and thus this leads to the generation of high quality samples and outperforming both state of the art GANs and hybrid VAE-GAN baselines.
Table 5: Closest generated images found using IvOM.
Table IvOM on Cifar10.
Sample
SN-GAN
c-GAN SN
BMS-VAE-GAN
Method
IvOM |
DCGAN RRdffrd el al. 20166)
10084410.002
VEEGAN
Snivastava ef al. 2017
0.0068=0.0001
SN-GAN Miyato et al. 2018
0.0055=000006
a GAN (N
0,0048
BMS-VAE-COE VAE-GAN (Ours) 7' = 30
0.0037=000005

---

Under review as a conference paper at ICLR 2020
4.2
EVALUATION ON CIFAR-10
Next, we evaluate on the CIFAR-10 dataset. Auto-encoding based approaches (Kingma & Welling 2014
Makhzani et al. 2016) do not perform well on this dataset, simple Gaussian reconstruction based likelihood is insufficient for such highly multi-modal image data. This makes CIFAR-10 very
challenging dataset for hybrid VAE-GANs.
Architecture Details. We use two different types of architectures for the generator/diseriinatto pair Gg, D1 DCGAN based (Radford al. 2016) as used in Rosca et al. (2019) and the Standard CNN used in Miyato et al. ( et al. (2018).
(Kigmaa & Ba 2015) and use learning rate of 2 x 10 10-4, 31 = 0.0 and B2 = 0.9 for all components. We use the same architecture of the latent space discriminator Dr. as in a-GAN Rosea al. (2019) (3-layer MI.P with 750 neurons each). Values of log( Dr) [0 work well.
Experimental Details and Baselines. We use the ADAM optimizer
We consider the following baselines for comparison against our BMS- VAE-GAN with a DCGAN generatorddisriminatoo: 1. A standard DCGAN (Radford et al. 2016), 2. The a-GAN model of (Rosca et al. 2019). Furthermore, we compare our BMS-GAN with the Standard CNN genera- tor/discriminator to. 1. SN-GAN (Miyato et al. 2018), 2. BW-GAN (Adler & Lunz 2018), 3. Dis- t-GAN (Tran et al.) 2018), 4. Our a-GAN + SN is an improved version of the a-GAN which includes Spectral Normalization for stable estimation of synthetic likelihoods. Again, the a-GAN and a-GAN + SN baselines are identical to the corresponding BMS- VAE-GAN except for the "Best-of-Many" Many-SampylS""" reconstruction likelihood.
Discussion of Results. We report results in Table 6 Please note that the higher latent space dimensionality (100) makes the latent spaces much harder to reliably analyze. Therefore, we rely on the FID and Io metrics. We follow evaluation protocol of Miyato et al. (018): Tran et al. (2018) and use 10k/5k real/generated samples to compute the FID score. The a-GAN (Rosca et al.] 2019) model with (DCGAN architecture) demonstrates better fit the true data distribution (29.3 vs 30.7 FID) compared to plain DCGAN. This again shows the ability of hybrid VAE-GANs in improving the performance of plain GANs. We observe that novel "Best-of-Me Many- Samples" " optimization scheme outperforms both the plain DCGAN and hybrid a-GAN((8.8 vs 29.4 FID), confirming Furthermore,
Method
FID
DCGAN Architecture
WAE (Tolstikhin et al. 2018
89.3++0.3
BMS- ASS-VAETOUSS
87.940.4
DCGAN Radford al. 2016
30.7::0.2
a GAN Rosca el al. 2019
29.4.10.3
BMS-GAN (ouns)
28.8:10.4
Standard CNN Architecture
SN-GAN Miyato et al 2018
25.5
BW-GAN Adler
2018
a-GAN SN (Uur)) T
24.6+t0.3
BMS-VAE-GAN VAE-GAN (Ours) T 10
BMS-
23.8:10.2
BMS-VAE MAN (Ours) T 30
23.4.:0.2
Dist-GAN [Tran et al. 2018
22.9
the advantage of using -Best-of-Maay-SSmmpee we see that our BMS-VAL VAE outperforms the state-of-the-art plain auto-encoding WAE (Tolstikhin et al. 2018).
BMS-VAE-GAN (Dun) 7
10
21.8:002
Table 6: FID on CIFAR-10.
We further compare our BMS VAD-GAN -tate-of-the-art GANs using the Standard CNN architec- ture in Table 6) with 100k generator iterations. Our a-GAN + SN ablation significantly outperforms the state-of-the-arr plain GANs (Adler & Lunz 2018 Miyato et al., 2018), showing the effectiveness of hybrid VAE-GANs with a stable direct estimate of the synthetic likelihood on this highly diverse dataset. Furthermore, our BMS-VAF-GE VAE-GAN model trained using the best of T = 30 samples signif- icantly improves over the a-GAN SN baseline (23.4 vs 24.6 FID), showing the effectiveness of our "Best-of-Many-Samee
We also compare to Tran et al. (2018) using 300k generator iterations, again outperforming by a significant margin (21.8 vs 22.9 FID). The Io metric of Srivastava et al. (2077) (Tables 4 and 5). illustrates that we are also able to better reconstruct the image distribution. The improvement in both sample quality measured by the FID metric and data reconstruction as measured by the lo VM metric shows that our novel "Best-of-Mann Many-Sanpllllae objective helps us both match the prior in the latent space and achieve high data log-likelihood at the same time.
4.3 EVALUATION ON CELEBA
Next, we evaluate on CelebA at resolutions 64x 64 and 128 x 128.
Training and Architecture Details. As the focus is to evaluate objectives for hyrid VAE-GANs, use simple DCGAN based generators and discriminators for generation at both 64 64 and 128 x 128. Approaches like progressive growing (Karras et al. 2018) orthogonal and can be applied top.
7

---

Under review as a conference paper at ICLR 2020
(a) Our a-GAN SN (T 1, 128x 128)
(b) Our BMS- VAE-GAN (T 10. 128x 128)
Figure 2: CelebA Random Samples. Our "Best of Many" reconstruction cost leads to sharper results.
Baselines and Experimental Details. We consider the following baselines for comparison with our BMS-GAN with 7' {10, 30} samples. 1. WAE (Tolstikhin al., 2018) the state-of-the-art plain auto-encoding generative model, 2. a-GAN (Rosca et al.] (019) the state-of-the-art hybrid VAE-GAN, Our a-GAN SN is an improved version of the a-GAN which includes Spectral Normalization for stable estimation of synthetic likelihoods. Note, the a- a-GAN baseline is identical to our except for the "Best-of-Many" reconstruction likelihood. Moreover, we include several plain GAN baselines, 1. Wasserstein GAN with gradient penalty (WGAN-GP) Gulrajani et al. (2017). 2. Spectral Normalization GAN (SN-GAN) Miyato et al. (2018). 3. Dist-GAN (Tran et al. 2018)).
To train our DAS-VAE-GAN and cR-GAN models use the two time-scale update rule (Heusel ctt al. 2017) with learning rate of 1 x 10 for the generator and 4 x 10 4 for the discriminator, We the Adam optimizer with $1 0.0 and 8g 0.9. We use three layer MLP with 750 neurons as the latent space discriminator Dr (as in Rosca et al. (2019) and a DCGAN based recognition network Rg. We use the hinge loss to train Dj produce high values for real images and low values for generated images, (Dr) E [-0.5, 0.5] works well.
Discussion of Results. We train all models for 200k itera- tions and report the FID scores (Heusel et al., 2017) for all models using 10k/10k real/generated samples in Table 7 The pure auto-encoding based WAE (Tolstikhin et al. 2018) has
Method
FID L
Resolution: 64
WAE Tolstikkin et al. 2018
41.2+0.3
the weakest performance due to blurriness. Our pure auto- encoding BMS- VAE (without synthetic likelihoods) improves upon the WAE (39.8 8 vs 41.2 FID), already demonstrating the effectiveness of using "Best-of-Many-Samye
BMS. (AUS))
39.8.0.33
DCGAN
31.110.9
WGAN-GP Guirajani
2017
26.8+:12
BEGAN Beitneeo el ain 2017
26.3+-0.9
We see that the base DCGAN has the weakest performance among the GANs. BEGAN suffers from partial mode collapse. The SN- GAN improves upon WGAN-GP, showing the effectiveness of Spectral Normalization. However, there exists considerable artifacts in its generations. The a- a-GAN of Rosca et al. (2019). which integrates the base DCGAN in its framework performs significantly better (31.1 vs 19.2 FID). This shows the effec- tiveness of VAE-GAN frameworks in increasing quality and diversity of generations. Our enhanced a-GAN + SN regu-
Dist-GAN
23.710.3
SN-GAN Mayalo el al 2013
21,910.8
GAN Rosca
2019
19.24-.88
a. GAN + SN (Ouns)
15.1
BMS VAE-GAN (Ours) T=
14.310.4
BMS-VAE-GAN (Ours) = 30
13.620.4
Resolution: 128x128
SN-GAN Miyato et al. 2018
6..5+1.5
oR.GAN (DISS) BMS GAN (Ouns)
45.8+ 1.4
1.2
Table 7: FID on CelebA.
larized with Spectral Normalization performs significantly better (15.1 vs 19.2 FID). This shows the effectiveness of a regularized direct estimate of the synthetic likelihood. Using the gradient penalty regularizer of Gulraaani
al. (2017) lead to drop of 0.4 FID. Our BMS- VAE-GAN improves significantly over the a-GAN + SN baseline using the "Best-of-Many-Samee The results at 128 x 128 resolution mirror the results at 64. We additionally evaluate using the loVM metric in Appendix C. We see that by using the "Best-of-Mang Man-aaay-Samplees we obtain sharper (Figure 4d) results that cover more of the data distribution as shown by both the FID and lo VM.
(13.6 vs 15.1 FID).
5 CONCLUSION
We propose a new objective for training hybrid VAE-GAN frameworks which overcomes key limitations of current hybrid VAE-GANs. We integrate, 1. A "Best-of-Many-Samplee likelihood which helps in covering all the modes of the data distribution while maintaining a latent space as close to Gaussian possible, 2. A stable estimate of the synthetic likelihood ratio.. Our hybrid VAE-GAN framework outperforms state-of-the hybrid VAE-GANs and plain GANs in generative modelling on CelebA and CIFAR-10, demonstrating the effectiveness of our approach.
reconstruction
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jonas Adler and Sebastian Lunz. Banach wasserstein gan. NeurlPS, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. ICML, 2017.
David Berthelot, Thomas Schumm, and Luke Metz. Began: boundary equilibrium generative adversarial networks. arXiv preprint arXiv: 10717, 2017.
Apratim Bhattacharyya, Bernt Schiele, and Mario Fritz. Accurate and diverse sampling of sequences based on best many sample objective. CVPR, 2018.
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Bayesian prediction future street scenes using synthetic likelihoods. ICLR, 2019.
Alican Bozkurt, Babak Esmaeili, Dana Brooks, Jennifer G Dy, and Jan-Willem van de Meent. Can vaes generate novel examples? NeurIPS Workshop, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. ICLR, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. NIPS, 2016.
Michael Comenetz. Calculus: the elements. World Scientific Publishing Company, 2002.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. ICLR, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv: 1606.00704, 2016.
Hamid Eghbal-zadeh, Werner Zellinger, and Gerhard Widmer. Mixture density generative adversarial networks. CVPR, 2019.
Mohamed Elfeki, Camille Couprie, Morgane Riviere, and Mohamed Elhoseiny. Gdpp: Learning diverse generations using determinantal point process. ICML, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. NIPS, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NIPS, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. ICML, 2016.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. NeurIPS, 2018.
Shweta Mahajan, Teresa Botschen, Iryna Gurevych, and Stefan Roth. Joint wasserstein autoencoders for aligning multimodal embeddings. ICCV Workshop, 2019.

---

Under review as a conference paper at ICLR 2020
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. ICLR, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets. NIPS, 2017.
Frank Niclsen and Ke Sun. Guaranteed bounds the kullback-leibbee divergence of univariate mixtures. IEEE Signal Processing Letters, 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR, 2016.
Suman Ravuri, Shakir Mohamed, Mihaela Rosca, and Oriol Vinyals. Learning implicit generative models with the method of learned moments. JCML, 2018.
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in varia- tional inference. arXiv preprint www.iv:1802.066411 2019.
Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer, and Stefano Ermon. Amortized inference regularization. NeurlPS, 2018.
Akash Srivastava, Lazar Valkoz. Chris Russell, Michael Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. NIPS, 2017.
Jacek Tabor, Szymon Knop, Przemyslaw Spurek, Igor Podolak, Marcin Mazur, and Stanislaw Jastrzebski. Cramer-wold autoencoder. arXiv preprint arXiv: NiKKi:/800... 1805.09235, 2018.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. ICLR, 2018
Ngoc-Trung Tran, Tuan-Anh Bui, and Ngai-Man Cheung. Dist-gan: An improved gan using distance constraints. ECCV, 2018.
David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. ICI.R, 2017.
Simon N Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 2010.
Chang Xiao, Peilin Zhong, and Changxi Zheng. Bourgan: Generative networks with metric embed- dings. NeurlPS, 2018.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. ICLR. 2017a.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders, arXiv preprint 1706:1706002666 2017b.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX A. ADDITIONAL DERIVATIONS
We begin with a derivation of the multi-sample objective 3
We maximize the log-likelihood of the data (x p(x)). The oog-likelhhood, assuming the latent space be distributed according to p(z),
(x)( log (ʃ ( px(x)2))(()d)z
Here, p(z) is usually Gaussian. However, the integral in 8 is intractable. VAËS and Hybrid VAE- GANs use amortized variational inference using an (approximate) variational distribution q6 (z)x) (jointly learned),
(x)( log (ʃ (o(()2)) go (x(z)x)
p(2)
go (z(z)))
To arrive a tractable objective, the standard VAE objective applies the Jensen incquality this stage. but this forces the final objective to consider the average data- lata-likelihood. Following Bhattacharyya al. (2018), we apply the Mean Value theorem of Integration (Comenetz samples,
2002) leverage multiple
(x)(po()) 2 log (s. (o(1x))) dz + log
p(zz) (1((2(xx)
z' E |a, b).
We can lower bound 9 with the minimum value of z',
(og(po(x)) log (f." (x(2))) go (z(x)) dz)
p(z') (a(z') [x)
min log z'E(o,b]
(S2)
As the term on the right is difficult to estimate, we approximate it using the KL divergence (as in Bhattacharyya et al.) (0118)). Intuitively, as the KL divergence heavily penalizes (z)x) if it is high for low values p(z), this ensures that the ratio p(z') (//((///)))) is maximized. Similar Bhattacharyya
al. 2018). this leads to the "many-sample" objective (4) of the main paper,
Lms log (Ens(s))po(((z) (L((o(z)x)) || p(z)).
(4)
Next, we provide detailed derivation of (5). Again, to enable the estimation of the likelihood ratio pe(x))/pp(s) using a neural network, we introduce the auxiliary variable y where, y = 1 denotes that the sample was generated and y denotes that the sample is from the true distribution. We can now express (5) as (using Bayes theorem),
(x y =)) + 3 log (Ex.cc)))o(x(22) - (L(a((Z)x) || p(z)).
a log (xlx))
p(x 0)
Po = 1|2. x)
=a log (gs(t)s))
1 ply = 1|x) + Bog (En0(1)))((x12)) (L(Ga(z(x) || p(z)).
This is because, (assuming independence p(z, = p(z)p(x))
Po (x|z,,y 1) ply = 1 1/2, x)x())
ply 1)
and,
po(xly = 0) ply 0xx)p(x)
p(y = 0)
Assuming, p(y = 0) = p(y- = 1) (equally likely to be true or generated),
px(x/z,y=11 Poly = 1/2, x)
p(xly = ply 0

---

Method Under review as a conference paper at ICLR 2020
Algorithm 1: BMS VAE-GAN Trainnnn
Initialize parameters of Rg, Go, Dr. Dr;
for i <- to max_iters do Update Rg. Ge (jointly) using our LBMS-S objective;
3
Update D1 using hinge loss produce high values (2 a) for real images and low ($ 6)
otherwise: Ep(x) max {0, a log( (x)) + Bp(()) max (0, b+ log(Di(Go(<)))):
Update Dr. using the standard cross-entropy loss:
Ep(x (z)) + Ep(x) log(1 (R (x))):
end
APPENDIX B. TRAINING ALGORITHM
We detail in algorithm how the components Rg. Gg. D1. DL of our BMS- VAE-GAN (see Figure Figure T) are trained. We follow Rosca et al. (2019) in bessgning(algorittin 1 However, unlike Rosca et al. (2019), we train RR. Go jointly as we found it to be computationally cheaper without any loss of performance. Also unlike Rosca et al. (2019), we use the hinge loss to update Dj as it leads to improved stability (as discussed in the main paper).
APPENDIX C. ADDITIONAL RESULTS USING THE IOVM METRIC
We additionally evaluate using the IoVM on CelebA in Table 8, using the base DCGAN architecture at 64x64 64 resolution. We observe that our BMS- VAE-GAN performs better. The improvement is smaller compared to CIFAR- because CelebA is less multi-modal compared to CIFAR-10. However, still observe better overall sample quality from our EMS-VAE-GIE This means that although difference in data reconstruction is smaller, our BMS- VAE-GAN enables better match the prior in the latent space. Finally, we provide additional examples of closest matches found using Io VM in Figure 3. illustrating regions of the data distribution captured by CAS-VAEEGAN but not captured by SN-GAN or a-GAN SN.
IoVM |
SN-GAN (Miyato al. 2018 a- c-GAN- (N 7
0.02211:0.0003
0.0036::0.0001
BMS- VAE-GAN (Ours) T 10
0.0034:100001
Table 8: Evaluation on CelebA using the IoVM metric.
APPENDIX D. ADDITIONAL QUALITATIVE EXAMPLES ON CELEBA AND CIFAR-10
In Figure 4 we compare qualitatively our BMS-VAE-GA against other state-of-the-art GANs. We use the same settings as the main paper and use the same DCGAN architecture across methods (as the aim is to evaluate training objectives). Again note that, approaches like Karras et al. (0118) use more larger generator/discrimnaator architectures and can be applied on top. We see that BEGAN (Berthelot et al. 2017) produces sharp images (with only a few very minuscule artifacts), but lack diversity also reflected by the lower FID score in Table 2 of the main paper. comparison, both SN-GAN MMyyato et al., 2018) and Dist-GAN (Tran et al. 2018) produce sharp and diverse images (again reflected by the FID score in Table 2 of the main paper) but also introduce artifacts. Bist-GAN (Tran et al.) 2018) introduces relatively fewer artifacts in comparison to SN-GAN (Miyato al. 2018). In comparison, our BMS VAE-GAR strikes the best balance generating sharp and diverse images with few if any artifacts (also again reflected by the FID scores in the main paper).
We also provide additional qualitative examples on CIFAR- 10 in Figure 5] highlighting sharper images compared to o-GAN +NN
12

---

Under review as a conference paper at ICLR 2020
Test Sample
SN-GAN
a-GAN + SN BMS-VAE-GAN
Test Sample
SN-GAN
a-GAN SN BMS-VAE-GAN
Figure 3: Closest images found by optimising in the latent space. Left: CIFAR-10, Right: CelebA
APPENDIX E. ADDITIONAL DIVERSITY EVALUATION USING LPIPS
include diversity using the LPIPS metric. To compute the LPIPS diversity score 5k samples were randomly generated and the similarity within the batch was computed. We see that our BMS- VEE--AE--AAA generates the most diverse examples both datasets, further highlighting the effectiveness of our "Best-of- "Best-of-Many-Sanny Many-Samples" objective.
In Table 9
Method
CelebA
CIFAR-I0 J
SN-GAN
0.160
0.148
a-GAN SN (Ours)
0.162
0.145
BMS VAE-GAN (Ours) 7 10
0.151
0.140
Table 9: Evaluation using the LPIPS metric.
13

---

Under review as a conference paper at ICLR 2020
(a) BEGAN Berthelot al. 2017 (64x64)
(b) SN GAN Miyato et al. 2018 (64 64)
(c) Dist-GAN
Tran et al.
2018 (64x 64)
(d) Our BMS-VMM--EEEEE VAE-GAN (T = 30, 64x64)
Figure 4: CelebA Random Samples of state-of-the-art GANs versus our BMS- VAE-GAN (using DCGAN architecture).
(a) a-GAN +SN (T = 1)
(b) BMS-VAE VAE-GAN (T' = 30)
Figure 5: CIFAR-10 Random Samples (using Standard CNN architecture).
14


Reviews:
[{"review_id": "r1ejzBAX5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new objective function for hybrid VAE-GANs. To overcome a number of known issues with VAE-GANs, this work uses multiple samples from the generator network to achieve a high data log-likelihood and low divergence to the latent prior.\nIn the experimental section, the ``\"Best-of-Many-Samples\" approach is shown to outperform other state-of-the-art methods on CIFAR-10 and a synthetic dataset.\n\nThanks for submitting code with your submission!\n\nCaveat: I'm not an expert in this domain and did my best to review this paper.\n\nQuestions:\n- Considering the smaller gap between α-GAN+SN and BMS-VAE-GAN, I was wondering how much of the improvement is due to spectral normalization vs using multiple samples. Did you do an ablation study of BMS-VAE-GAN without SN?\n- I noticed some minor typos in the text. Please fix (3.2 \"constrains\" -> \"constraints\", 3.3 \"traiend\", 3.3 \"unsure\" -> \"ensure\").", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SkxNbzojYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "“Best of Many Samples” Distribution matching\n\nSummary:\n\nThis paper proposes to a novel VAE-GAN hybrid which, during training, draws multiple samples from the reparameterized latent distribution for each inferred q(z|x), and only backpropagates reconstruction error for the resulting G(z) which has the lowest reconstruction. The authors appear to use the AAE method instead of analytically matching KL(p||q) for enforcing that the latents q(z|x) match the prior. The authors present results on MoG toy datasets, CIFAR-10, and CelebA, and compare  against several other models.\n\nMy take:\n\nThe idea in this paper is moderately interesting, well-founded, has plenty of precedent in the literature (while still being reasonably novel),  but the results present only a minimal improvement (a 5% relative improvement in FID over the baseline model from Rosca et al on CIFAR, especially when including SN [which is not a contribution of this paper]) and come at a substantial compute cost, requiring up to 30 extra samples per batch in order to attain this minimal increase. While I think the idea is interesting, the change in results over Rosca et. al does not seem to justify its increased computational expense (which is also not characterized in sufficient thoroughness). I am pretty borderline on this paper ( about a 5/10) but under the 1-3-6-8 scoring constraint I tend to lean reject because while I like the idea, I do not think the results are significant enough to support its adoption; I think the relative compute and implementation cost limit this method’s potential impact. I am keen to discuss this paper with the other reviewers.\n\nNotes:\n\n-The results on the 2D MoG toy datasets are good but are also suspect—the authors state that they use a 32-dimensional latent space, but the original code provided for VEEGAN uses a 2-dimensional latent space. The authors should re-run the experiment for BMS-VAE-GAN  using a 2D latent space (this should be very easy and take less than an hour on a GPU to get several runs in).\n\n-“again outperforming by a significant margin (21.8 vs 22.9 FID)” This is not a significant margin; this is less than a 5% margin and, at those FID scores, represents an imperceptible change in sample quality.\n\n-The authors seem to suggest that applying spectral norm to the GAN of Rosca et. al. is somehow a contribution (e.g. having “ours” next to this model in the tables); I would advise against even appearing to suggest this as it is clearly not a contribution.\n\n-Characterize the increase in compute cost. “. We use as many samples during training as would fit in GPU memory so that we make the same number of forward/backward passes as other approaches and minimize the computational overhead of sampling multiple samples” is a qualitative description; I would like to see this quantitatively described. How do the runtimes differ between your baseline and the T=10 and T=30 runs? If they don’t differ, why? Are the authors e.g. reducing the batch size by a factor of 10 or 30 to make this computationally tractable?\n\n-The latent space discriminator D_L should be referred to in section 3; its formal introduction is deferred to later in the paper, hampering the presentation and flow. \n\n-CelebA is not multimodal; it is in fact, highly constrained, and primarily only has textural variation (virtually no pose variation).\n\n-ALI and BiGAN are listed under Hybrid VAE-GANs. These models are not VAE-GAN hybrids. Additionally, this section states that BiGAN builds upon ALI. This is not true, these papers are in fact proposing the same thing and were released at nearly the exact same time.  Do not mischaracterize or incorrectly summarize papers. Please re-read both papers and refer to them correctly.\n\n-Mode collapse (when many points in z map to an unexpectedly small region in G(z)) is a different phenomenon from mode dropping (when many points in x are not represented in G(z), i.e. no point in z maps to a cluster of x’s, as is the case if e.g. a celebA model generates frowning and neutral faces but no smiling faces). While these phenomena often co-occur (especially during complete training collapse), they are not the same thing, and this paper conflates them in several places.\n\nMinor:\n\nSection 3, paragraph 2: “The GAN (Gθ,DI…” There’s a close parenthesis missing here. \n\nSection 3.3: “The network is traiend…” \n\nPlease thoroughly proofread your paper for typos and grammatical mistakes.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1lD7rTsFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1.  Using Discriminator to estimate the likelihood ratio is a commonly used approach, which was first proposed in [1]. This is also generalized as a reversed KL based GAN in [2] [3]. The authors failed to discuss this with these previous works in Section 3.3 and in Related works.\n\n2. How is the best of many comparing with importance sampling method? I think using importance sampling is the most intuitive baseline.\n\n3. this paper is not well written. L_1/L_2 has never explained throughout this paper, also has typos such as \"taiend\".\n\n\n[1] Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks\n[2] Variational Annealing of GANs: A Langevin Perspective\n[3] Symmetric variational autoencoder and connections to adversarial learning", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an improvement on VAE-GAN which draws multiple samples from the reparameterized latent distribution for each inferred q(z|x), and only backpropagates reconstruction error for the resulting G(z) which has the lowest reconstruction.  While the idea is interesting, the novelty is not high compared with existing similar works, and the improvement is not significant.
Decision: Reject