``"Best-of-Many-Samples" Distribution Matching

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have achieved state-of-the-art sample quality in generative modeling tasks . However , GANs do not explicitly estimate the data likelihood . Instead , it aims to “ fool ” an adversary , so that the adversary is unable to distinguish between samples from the true distribution and the generated samples . This leads to the generation of high quality samples ( Adler & Lunz , 2018 ; Brock et al. , 2019 ) . However , there is no incentive to cover the whole data distribution . Entire modes of the true data distribution can be missed – commonly referred to as the mode collapse problem . In contrast , the Variational Auto-Encoders ( VAEs ) ( Kingma & Welling , 2014 ) explicitly maximize data likelihood and can be forced to cover all modes ( Bozkurt et al. , 2018 ; Shu et al. , 2018 ) . VAEs enable sampling by constraining the latent space to a unit Gaussian and sampling through the latent space . However , VAEs maximize a data likelihood estimate based on the L1/L2 reconstruction cost which leads to lower overall sample quality – blurriness in case of image distributions . Therefore , there has been a spur of recent work ( Donahue et al. , 2017 ; Larsen et al. , 2016 ; Rosca et al. , 2019 ) which aims integrate GANs in a VAE framework to improve VAE generation quality while covering all the modes . Notably in Rosca et al . ( 2019 ) , GANs are integrated in a VAE framework by augmenting the L1/L2 data likelihood term in the VAE objective with a GAN discriminator based synthetic likelihood ratio term . However , Rosca et al . ( 2019 ) reports that in case of hybrid VAE-GANs , the latent space does not usually match the Gaussian prior . This is because , the reconstruction log-likelihood in the VAE objective is at odds with the divergence to the latent prior ( Tabor et al. , 2018 ) ( also in case of alternatives proposed by Makhzani et al . ( 2016 ) ; Arjovsky et al . ( 2017 ) ) . This problem is further exacerbated with the addition of the synthetic likelihood term in the hybrid VAE-GAN objective – it is necessary for sample quality but it introduces additional constraints on the encoder/decoder . This leads to the degradation in the quality and diversity of samples . Moreover , the synthetic likelihood ratio term is unstable during training – as it is the ratio of outputs of a classifier , any instability in the output of the classifier is magnified . We directly estimate the ratio using a network with a controlled Lipschitz constant , which leads to significantly improved stability . Our contributions in detail are , 1 . We propose a novel objective for training hybrid VAE-GAN frameworks , which relaxes the constraints on the encoder by giving the encoder multiple chances to draw samples with high likelihood enabling it to generate realistic images while covering all modes of the data distribution , 2 . Our novel objective directly estimates the synthetic likelihood term with a controlled Lipschitz constant for stability , 3 . Finally , we demonstrate significant improvement over prior hybrid VAE-GANs and plain GANs on highly muti-modal synthetic data , CIFAR-10 and CelebA . 2 RELATED WORK . Generative Autoencoders . VAEs ( Kingma & Welling , 2014 ) allow for generation by maintaining a Gaussian latent space . In Kingma & Welling ( 2014 ) , the Gaussian constraint in applied point-wise and latent representation of each point is forced towards zero . Adversarial Auto-encoders ( AAE ) ( Makhzani et al. , 2016 ) and Wasserstein Auto-encoders ( WAE ) ( Arjovsky et al. , 2017 ) tackle this problem by an approximate estimate of the divergence which only requires the latent space to be Gaussian as a whole . But , the Gaussian constraint in ( Arjovsky et al. , 2017 ; Kingma & Welling , 2014 ; Makhzani et al. , 2016 ; Mahajan et al. , 2019 ) is still at odds with the data log-likelihood . In this work , we enable the encoder to maintain both the latent representation constraint and high data log-likelihood using a novel objective . Furthermore , we integrate a GAN-based synthetic likelihood term to the objective to enhance the sharpness of generated images . Mode Collapse in Classical GANs . The classic GAN formulation ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ) has several shortcomings – importantly mode collapse . Denoising Feature Matching ( Warde-Farley & Bengio , 2017 ) deals with the mode collapse by regularizing the discriminator using an auto-encoder . MDGAN ( Che et al. , 2017 ) uses two separate discriminators and regularizes using a auto-encoder . In EBGAN ( Zhao et al. , 2017a ) , the discriminator is interpreted as an energy functional and is also cast in an auto-encoder framework , leading to improvements in semi-supervised learning tasks . BEGAN ( Berthelot et al. , 2017 ) proposes a Wasserstein distance based objective to train such GANs with auto-encoder based discriminators . The proposed approach leads to smoother convergence . InfoGAN ( Chen et al. , 2016 ) maximizes the mutual information between a small subset of latent variables and observations in a Information Theoretic framework . This leads to disentangled and more interpretable latent representations . PacGAN ( Lin et al. , 2018 ) proposes to deal with the mode collapse problem by using the discriminator to distinguish between product distributions . D2GAN ( Nguyen et al. , 2017 ) proposes to use two discriminators – one for the forward KL divergence between the true and generated distributions and one for the reverse . BourGAN ( Xiao et al. , 2018 ) proposes to learn the distribution of the latent space ( instead of assuming Gaussian ) which reflects the distribution of the data . In ( Srivastava et al. , 2017 ) , a inverse mapping from from latent to data space is learned and the generator is penalized based on the inverted distribution to cover all modes . Ravuri et al . ( 2018 ) proposes a moment matching paradigm different from VAEs or GANs . However , as the presented moment matching network involves an order of magnitude more parameters compared to VAEs or GANs , we do not consider them here . As we propose a hybrid VAE-GAN framework these techniques can be applied on top to potentially improve results . However , in hybrid VAE-GANs the reconstruction loss already incentivizes the coverage of all modes . Wasserstein Loss based Formulations . Arjovsky et al . ( 2017 ) ; Gulrajani et al . ( 2017 ) proposes GANs which minimize the Wasserstein distance between true and generated distributions . Miyato et al . ( 2018 ) demonstrates improved results by applying Spectral Normalization on the weights . In Tran et al . ( 2018 ) , distance constraints are applied on top . In Adler & Lunz ( 2018 ) WGANs were extended to Banach Spaces to emphasize edges or large scale behavior . Orthogonally , Karras et al . ( 2018 ) focus on progressively learning to use more complex model architectures to improve performance . We use the regularization techniques developed for WGANs to improve stability of our hybrid VAE-GAN framework . Brock et al . ( 2019 ) shows very high quality generations at high resolutions but these are class conditional . However , diverse class conditional generation is considerably easier as intra-class variability is generally much lower than inter-class variability . Here , we focus on the more complex unconditional image generation task . Hybrid VAE-GANs . In Larsen et al . ( 2016 ) a VAE-GAN hybrid is proposed with discriminator feature matching – the VAE decoder is trained to match discriminator features instead of a L1/L2 reconstruction loss . ALI ( Dumoulin et al. , 2016 ) proposes to instead match the encoder and decoder joint distributions – with limited success on diverse datasets . BiGAN ( Donahue et al. , 2017 ) , builds upon ALI to learn inverse mappings from the data to the latent space and demonstrate effectiveness on various discriminative tasks . Rosca et al . ( 2019 ) extends standard VAEs by replacing the loglikelihood term with a hybrid version based on synthetic likelihoods . The KL-divergence constraint to the prior is also recast to a synthetic likelihood form , which can be enforced by a discriminator ( as in Makhzani et al . ( 2016 ) ; Tolstikhin et al . ( 2018 ) ) . The second improvement is crucial in generating realistic images at par with classic/Wasserstein GANs . We further improve upon Rosca et al . ( 2019 ) by allowing the encoder multiple chances to draw desired samples and enforcing stability – enabling it to maintain low divergence to the prior while generating realistic images . 3 NOVEL OBJECTIVE FOR HYBRID VAE-GANS . We begin with a brief overview of hybrid VAE-GANs followed by details of our novel objective . Overview . Hybrid VAE-GANs ( Figure 1 ) are generative models for data distributions x ∼ p ( x ) that transform a latent distribution z ∼ p ( z ) to a learned distribution x̂ ∼ pθ ( x ) approximating p ( x ) . The GAN ( Gθ , DI alone can generate realistic samples , but has trouble covering all modes . The VAE ( Rφ , Gθ , DL ) can cover all modes of the distribution , but generates lower quality samples overall . VAE-GANs leverage the strengths of both VAEs and GANs to generate high quality samples while capturing all modes . We begin with a discussion of the prior hybrid VAE-GAN objectives ( Rosca et al. , 2019 ) and its shortcomings , followed by our novel “ Best-of-Many-Samples ” objective with a novel reconstruction term and regularized stable direct estimate of the synthetic likelihood . 3.1 SHORTCOMINGS OF HYBRID VAE-GAN OBJECTIVES . Hybrid VAE-GANs ( Dumoulin et al. , 2016 ; Makhzani et al. , 2016 ; Rosca et al. , 2019 ; Zhao et al. , 2017b ) maximizes the log-likelihood of the data ( x ∼ p ( x ) ) akin to VAEs . The log-likelihood , assuming the latent space to be distributed according to p ( z ) , log ( pθ ( x ) ) = log ( ∫ pθ ( x|z ) p ( z ) dz ) . ( 1 ) Here , p ( z ) is usually Gaussian . This requires the generator Gθ to generate samples that assign high likelihood to every example x in the data distribution for a likely z ∼ p ( z ) . Thus , the decoder θ can be forced to cover all modes of the data distribution x ∼ p ( x ) . In contrast , GANs never directly maximize the data likelihood and there is no direct incentive to cover all modes . However , the integral in ( 1 ) is intractable . VAEs and Hybrid VAE-GANs use amortized variational inference using a recognition network qφ ( z|x ) ( Rφ ) . The final hybrid VAE-GAN objective of the state-of-the-art α-GAN ( Rosca et al. , 2019 ) which integrates a synthetic likelihood ratio term is , Lα-GAN = λ Eqφ ( z|x ) log ( pθ ( x|z ) ) + Eqφ ( z|x ) log ( DI ( x|z ) 1−DI ( x|z ) ) − KL ( p ( z ) ‖ qφ ( z|x ) ) . ( 2 ) This objective has two important shortcomings . Firstly , as pointed in ( Bhattacharyya et al. , 2018 ; Tolstikhin et al. , 2018 ) , this objective severely constrains the recognition network as the average likelihood of the samples generated from the posterior qφ ( z|x ) is maximized . This forces all samples from qφ ( z|x ) to explain x equally well , penalizing any variance in qφ ( z|x ) and thus forcing it away from the Gaussian prior p ( z ) . Therefore , this makes it difficult to match the prior in the latent space and the encoder is forced to trade-off between a good estimate of the data log-likelihood and the divergence to the latent prior . Secondly , the synthetic likelihood ratio term is the ratio of the output of DI , any instability ( nonsmoothness ) in the output of the classifier is magnified . Moreover , there is no incentive for DI to be smooth ( stable ) . For two similar images , { x1 , x2 } with |x1 − x2| ≤ , the change of output |DI ( x1|z1 ) −DI ( x1|z2 ) | can be arbitrarily large . This means that a small change in the generator output ( e.g . after a gradient descent step ) can have a large change in the discriminator output . Next , we describe how we can effectively leverage multiple samples from qφ ( z|x ) to deal with the first issue . Finally , we derive a stable synthetic likelihood term ( Rosca et al. , 2019 ; Wood , 2010 ) to deal with the second issue .

Summary: 1.  Using Discriminator to estimate the likelihood ratio is a commonly used approach, which was first proposed in [1]. This is also generalized as a reversed KL based GAN in [2] [3]. The authors failed to discuss this with these previous works in Section 3.3 and in Related works.