AIM: Adversarial Inference by Matching Priors and Conditionals

1 INTRODUCTION . Deep directed generative models like variational autoencoder ( VAE ) ( Kingma and Welling , 2013 ; Rezende et al. , 2014 ) and generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) have been proved to be powerful for modeling complex high-dimensional distributions . While both VAE and GAN can learn to generate realistic images , their underlying mechanisms are fundamentally different . VAE maps the data into low-dimensional codes using an encoder , and then reconstructs the original data by a decoder . This allows it to perform both generation and inference . GAN , on the other hand , trains a generator and a discriminator adversarially . The generator learns to fool the discriminator by mapping low-dimensional noise vectors to the data space ; at the same time , the discriminator evolves to detect the generated fake samples from the true ones . These two methods have complementary strengths and weaknesses . VAE can learn a bidirectional mapping between data and code spaces , but relies on over-simplified parametric assumptions on the complex data distribution , thereby causing it to generate blurry images ( Donahue et al. , 2016 ; Goodfellow et al. , 2014 ; Larsen et al. , 2015 ) . GAN generates more realistic samples than VAE ( Radford et al. , 2015 ; Larsen et al. , 2015 ) because the adversarial regime allows it to learn more complex distributions . However , note that GAN only learns a unidirectional mapping for data generation , and does not allow inferring the latent codes from given samples . This is limiting because the ability of inference is very crucial for several downstream applications , such as classification , clustering , similarity search , and interpretation . Furthermore , GAN also suffers from the model collapse problem ( Che et al. , 2016 ; Salimans et al. , 2016 ) – many modes of the data distribution are not represented in the generated samples . Therefore , one may wonder on whether it is possible to develop a generative model that enjoys the strengths of both GAN and VAE without their inherent weaknesses . Such model should be able to generate high-quality samples as good as GAN , have an inference mechanism as effective as VAE , and also avoid the mode collapse issue . Many recent efforts have been devoted to combining VAE with adversarial discriminator ( s ) ( Brock et al. , 2016 ; Che et al. , 2016 ; Larsen et al. , 2015 ; Makhzani et al. , 2015 ; Mescheder et al. , 2017 ) . However , VAE-GAN hybrids tend to manifest a compromise of the strengths and weaknesses of both the approaches . The main reason is that all of them retain the autoencoder structure , which requires an explicit metric to measure the data reconstruction and assumes over-simplified parametric data distributions . To overcome such limitations , adversarially learned inference ( ALI ) ( Donahue et al. , 2016 ; Dumoulin et al. , 2016 ) was recently proposed , wherein the discriminator is trained on the joint distribution of data and latent codes . In this way , under a perfect discriminator , one can successfully match joint distributions of the decoder and encoder , thereby , performing inference by sampling from the encoder ’ s conditional distribution that also matches the decoder ’ s posterior . However , in practice the equilibrium of the jointly adversarial game is hard to approach as the dependency structure between data and codes is not explicitly specified . ALI ’ s reconstructions of samples are thus not always faithful ( Dumoulin et al. , 2016 ; Li et al. , 2017 ) implying that its inference mechanism is not always effective . To overcome the aforementioned issues , in this paper , we propose a novel approach , Adversarial Inference by Matching priors and conditionals ( AIM ) , that integrates efficient inference to GAN and overcomes the limitations of prior approaches . The approach keeps the structure simple , involving only one generator , one encoder , and one discriminator . Furthermore , AIM ’ s objective is directly derived from our goal of matching both prior and conditional distributions of the generator and encoder , instead of a heuristic combination with lk norm regularization . Compared to regular GANs , AIM has the ability to conduct inference , and also does not suffer from the mode collapse problem . Moreover , AIM also abandons the unrealistic parametric assumption on the conditional data distribution , and does not require any reconstruction in the data space . This is fundamentally different from VAE or VAE-GAN hybrids in which the lk norm is used to measure the data reconstruction . The usage of simple data-fitting metrics on the complex data distribution leads to worse generation performance . Different from ALI , AIM decomposes the hard problem of matching the joint distributions into two sub-tasks – explicitly matching the priors on the latent codes and the conditionals on the data . As a consequence of more restrictive constraint , it achieves better generation and more faithful reconstruction than ALI . Last but not the least , note that GAN variations with inference mechanism usually achieve worse generation performance as compared to regular GANs . However , as demonstrated in the experiments , AIM not only performs best on inference , but also improves the generation performance over GAN . 2 RELATED WORK . The most straightforward way to learn an inference mechanism is to learn the inverse mapping of GAN ’ s generator post-hoc ( Zhu et al. , 2016 ) . However , since its training process is the same as GAN , it still suffers from mode collapse problem . AGE ( Ulyanov et al. , 2017 ) encourages encoder and generator to be reciprocal by simultaneously minimizing an l1 reconstruction error in the data space and an l2 error in the code space . This is closely related to the cycle-consistency criterion ( Zhu et al. , 2017 ; Kim et al. , 2017 ; Yi et al. , 2017 ; Li et al. , 2017 ) . Although the pairwise reconstruction errors help reduce mode collapse , the data reconstruction is still measured by l1 or l2 norm , which brings the same problem of VAE and VAE-GAN hybrids . InfoGAN ( Chen et al. , 2016 ) minimizes the mutual information between a subset c of the latent code and the generated samples , and hence can only do partial inference on c. Different from the heuristic combination of VAE and GANs , Mescheder et al . ( 2017 ) theoretically derived an adversarial game to replace the KL-divergence term in the variational lower bound ( also called ELBO ) , and gives the new method , adversarial variational Bayes ( AVB ) , much more flexibility in its dependence on latent z . However , the reconstruction term on x still exists and so is the parametric assumption on the conditional data distribution , leading to the blurriness in their reconstructed and generated samples . ALI ( Dumoulin et al. , 2016 ; Donahue et al. , 2016 ) is an elegant approach to bring inference mechanism into adversarial learning without assuming parametric distribution on the data . Different from our work , it directly plays an adversarial game to match the joint distributions of the decoder and encoder . But in practice , ALI ’ s reconstructions are not necessarily faithful because the dependency structures within the two joint distributions are not specified ( Li et al. , 2017 ) . ALICE ( Li et al. , 2017 ) tries to solve this problem by regularizing ALI using an extra conditional entropy constraint on the data . The conditional entropy is either explicitly measured by lk norm , or implicitly learned by adversarial training . However , when the data distribution becomes complicated ( e.g . CIFAR-10 ) , the lk metric may lead to blurry reconstructions and the adversarial training is hard to achieve ( Li et al. , 2017 ) . Compared with ALI and ALICE , our method is proven to minimize the KL-divergence between both priors and conditionals of generator and encoder , and can provide consistent effective inference even on complicated distribution ( see Section 5.1 ) . Srivastava et al . ( 2017 ) proposed VEEGAN to tackle the mode collapse issue of GANs by adding an implicit variatinoal learning on the latent z . To our best knowledge , this is by far the only approach that is also reconstructing z . Different from VAEs , VEEGAN autoencodes the latent variable or noise z . By doing so , it enforces the generator not to collapse the mappings of z to a single mode , because otherwise , the encoder will not be able to recover all the noise z . The details of their model can be summarized as ALI regularized by an extra reconstruction of latent z . Therefore , VEEGAN is similar to ALICE in the sense that they are both adversarial games on the joint distribution with an extra regularization on either data or latent reconstruction . Our model AIM instead only plays the adversarial game on the marginal data distribution , and reconstructs the latent z by maximizing its log-likelihood under the latent posterior distribution . 3 BACKGROUND We consider the generative model pθ ( z ) pθ ( x|z ) , where a latent variable z ( i ) is first generated from the prior distribution pθ ( z ) , and then the data x ( i ) is sampled from the conditional distribution pθ ( x|z ) . The parameter θ stands for the ground truth parameter of the distribution . The prior pθ ( z ) is always assumed to be a simple parametric distribution ( e.g . N ( 0 , I ) ) , but the generative conditional pθ ( x|z ) is always very complex and not known to us . Moreover , the posterior pθ ( z|x ) is intractable but stands for an important inference procedure : given a data x ( i ) , it allows us to infer its latent variable z ( i ) . 4 METHODOLOGY . In our method , we will model the generating process by a neural network called generator , and the inference process by another neural network encoder . Consider the following two distributions of the generator and encoder , and their corresponding sampling procedures : • the generator distribution : p ( z ) p ( x|z ) ; z ∼ p ( z ) , x ∼ p ( x|z ) . • the encoder distribution : q ( x ) q ( z|x ) ; x ∼ q ( x ) , z ∼ q ( z|x ) . The generator ’ s conditional p ( x|z ) approximates the generating distribution pθ ( x|z ) . The encoder ’ s conditional q ( z|x ) approximates the posterior distribution pθ ( z|x ) , which is what we need for inference . The marginal distribution q ( x ) stands for the empirical data distribution , and the other marginal p ( z ) is taken to be pθ ( z ) , which is always a known distribution like standard Gaussian . 4.1 MATCHING PRIORS AND CONDITIONALS . The ultimate goal is to match the joint distributions , p ( x , z ) and q ( x , z ) . If this is achieved , we are guaranteed that all marginals match and all conditionals match as well . In particular , the conditional q ( z|x ) matches the posterior p ( z|x ) . We propose to decompose this goal into two sub-tasks – matching the priors p ( z ) and q ( z ) , and matching the conditionals p ( x|z ) and q ( x|z ) . There are two advantages . Firstly , we explicitly define the dependency structure z → x. Secondly , the explicit constraints on both priors and conditionals are stronger than merely one constraint on the joint distributions . In order to match the distributions , we will minimize the KL-divergence between the priors and between the conditionals : Ep ( z ) DKL ( p ( x|z ) ||q ( x|z ) ) +DKL ( p ( z ) ||q ( z ) ) . ( 1 ) By the properties of KL-divergence , when the minimum of equation 1 is attained , we have both p ( z ) = q ( z ) and p ( x|z ) = q ( x|z ) , for all x and z .

Summary: The goal this is work is to develop a generative model that enjoys the strengths of both GAN and VAE without their inherent weaknesses. The paper proposes a learning framework, in which a generating process p is modeled by a neural network called generator, and an inference process q by another neural network encoder. The ultimate goal is to match the joint distributions, p(x, z) and q(x, z), and this is done by attempting to match the priors  p(z) and q(z) and matching the conditionals p(x|z) and q(x|z). As both q(z) and q(x|z) are impossible to sample from, the authors mathematically expand this objective criterion and rewrite to be dependent only on p(x|z), q(x) and q(z|x), that can be easily sampled from. In the main part of the work, the authors use the f-divergence theory (Nowozin et al., 2016) to present the optimization problem as minmax optimization problem, that is learned using an adversarial game, using training and inference algorithms that are proposed by the authors. In experiments, the authors consider both reconstruction and generation tasks using the MNIST, CIFAR10 and CelebA datasets. Results show that the proposed method yields better MSE reconstruction error as better as a higher inception scores for the generated examples, compared to a standard GAN and a few other methods.