Perceptual Generative Autoencoders

1 INTRODUCTION . Recent years have witnessed great interest in generative models , mainly due to the success of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ; Karras et al. , 2018 ; Brock et al. , 2019 ) . Despite their prevalence , the adversarial nature of GANs can lead to a number of challenges , such as unstable training dynamics and mode collapse . Since the advent of GANs , substantial efforts have been devoted to addressing these challenges ( Salimans et al. , 2016 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) , while non-adversarial approaches that are free of these issues have also gained attention . Examples include variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) , reversible generative models ( Dinh et al. , 2014 ; 2017 ; Kingma & Dhariwal , 2018 ) , and Wasserstein autoencoders ( WAEs ) ( Tolstikhin et al. , 2018 ) . However , non-adversarial approaches often have significant limitations . For instance , VAEs tend to generate blurry samples , while reversible generative models require restricted neural network architectures or solving neural differential equations ( Grathwohl et al. , 2019 ) . Furthermore , to use the change of variable formula , the latent space of a reversible model must have the same dimensionality as the data space , which is unreasonable considering that real-world , high-dimensional data ( e.g. , images ) tends to lie on low-dimensional manifolds , and thus results in redundant latent dimensions and variability . Intriguingly , recent research ( Arjovsky et al. , 2017 ; Dai & Wipf , 2019 ) suggests that the discrepancy between the intrinsic and ambient dimensionalities of data also contributes to the difficulties in training GANs and VAEs . In this work , we present a novel framework for training autoencoder-based generative models , with non-adversarial losses and unrestricted neural network architectures . Given a standard autoencoder and a target data distribution , instead of matching the target distribution in the data space , we map both the generated and target distributions to the latent space using the encoder , and train the generator ( or decoder ) to minimize the divergence between the mapped distributions . We prove , under mild assumptions , that by minimizing a form of latent reconstruction error , matching the target distribution in the latent space implies matching it in the data space . We call this framework perceptual generative autoencoder ( PGA ) . We show that PGAs enable training generative autoencoders with maximum likelihood , without restrictions on architectures or latent dimensionalities . In addition , when combined with VAEs , PGAs can generate sharper samples than vanilla VAEs.1 1Code is available at https : //bit.ly/2U0kRYL . We summarize our main contributions as follows : • A training framework , PGA , for generative autoencoders is developed to match the target distribution in the latent space , which , we prove , ensures the matching in the data space . • We combine the PGA framework with maximum likelihood , and remove the restrictions of reversible generative models on neural network architectures and latent dimensionalities . • We combine the PGA framework with VAE , which solves the problem of blurry samples , without introducing any auxiliary models or sophisticated model architectures . 2 RELATED WORK . Autoencoder-based generative models are trained by minimizing an data reconstruction loss with regularizations . As an early approach , denoising autoencoders ( DAEs ) ( Vincent et al. , 2008 ) are trained to recover the original input from an intentionally corrupted input . Then a generative model can be obtained by sampling from a Markov chain ( Bengio et al. , 2013 ) . To sample from a decoder directly , most recent approaches resort to mapping a simple prior distribution to a data distribution using the decoder . For instance , adversarial autoencoders ( AAEs ) ( Makhzani et al. , 2016 ) and Wasserstein autoencoders ( WAEs ) ( Tolstikhin et al. , 2018 ) attempt to match the aggregated posterior and the prior , either by adversarial training or by minimizing their Wasserstein distance . However , due to the use of deterministic encoders , there can be “ holes ” in the latent space that are not covered by the aggregated posterior , which would result in poor sample quality ( Rubenstein et al. , 2018 ) . By using stochastic encoders and variational inference , variational autoencoders ( VAEs ) are likely to suffer less from this problem , but are known to generate blurry samples ( Rezende & Viola , 2018 ; Dai & Wipf , 2019 ) . Nevertheless , as we will show , the latter problem can be addressed by moving the VAE reconstruction loss from the data space to the latent space . In a different line of work , reversible generative models ( Dinh et al. , 2014 ; 2017 ; Kingma & Dhariwal , 2018 ) are developed to enable exact inference . Consequently , by the change of variables theorem , the likelihood of each data sample can be exactly computed and optimized . However , to avoid expensive Jacobian determinant computations , reversible models can only be composed of restricted transformations , rather than general neural network architectures . While this restriction can be relaxed by utilizing recently developed neural ordinary differential equations ( Chen et al. , 2018 ; Grathwohl et al. , 2019 ) , they still rely on a shared dimensionality between latent and data spaces , which remains an unnatural restriction . In this work , we use the proposed training framework to trade exact inference for unrestricted neural network architectures and arbitrary latent dimensionalities , generalizing maximum likelihood training to autoencoder-based models . 3 METHODS . 3.1 PERCEPTUAL GENERATIVE MODEL . Let fφ : RD → RH be the encoder parameterized by φ , and gθ : RH → RD be the decoder parameterized by θ . Our goal is to obtain a decoder-based generative model , which maps a simple prior distribution to a target data distribution , D. A summary of notations is provided in Appendix A . Throughout this paper , we use N ( 0 , I ) as the prior distribution . For z ∈ RH , the output of the decoder , gθ ( z ) , lies in a manifold that is at most H-dimensional . Therefore , if we train the autoencoder to minimize Lr = 1 2 Ex∼D [ ‖x̂− x‖22 ] , ( 1 ) where x̂ = gθ ( fφ ( x ) ) , then x̂ can be seen as a projection of the input data , x , onto the manifold of gθ ( z ) . Let D̂ denote the reconstructed data distribution , i.e. , x̂ ∼ D̂ . Given enough capacity of the encoder , D̂ is the best approximation to D ( in terms of ` 2-distance ) , that we can obtain from the decoder , and thus can serve as a surrogate target distribution for training the decoder-based generative model . Due to the difficulty in directly matching the generated distribution with the data-space target distribution , D̂ , we reuse the encoder to map D̂ to a latent-space target distribution , Ĥ . We then transform the problem of matching D̂ in the data space into matching Ĥ in the latent space . In other words , we aim to ensure that for z ∼ N ( 0 , I ) , if fφ ( gθ ( z ) ) ∼ Ĥ , then gθ ( z ) ∼ D̂ . In the following , we define h = fφ ◦ gθ for notational convenience . To this end , we minimize the following latent reconstruction loss w.r.t . φ : Lφlr , N = 1 2 Ez∼N ( 0 , I ) [ ‖h ( z ) − z‖22 ] . ( 2 ) Let Z ( x ) be the set of all z ’ s that are mapped to the same x by gθ , we have the following theorem : Theorem 1 . Assuming E [ z | x ] ∈ Z ( x ) for all x generated by gθ , and sufficient capacity of fφ ; for z ∼ N ( 0 , I ) , if Eq . ( 2 ) is minimized and h ( z ) ∼ Ĥ , then gθ ( z ) ∼ D̂ . We defer the proof to Appendix B.1 . Note that Theorem 1 requires that different x ’ s generated by gθ ( fromN ( 0 , I ) andH ) are mapped to different z ’ s by fφ . In theory , minimizing Eq . ( 2 ) would suffice , since N ( 0 , I ) is supported on the whole RH . However , there can be z ’ s with low probabilities in N ( 0 , I ) , but with high probabilities in H that are not well covered by Eq . ( 2 ) . Therefore , it is sometimes helpful to minimize another latent reconstruction loss onH : Lφlr , H = 1 2 Ez∼H [ ‖h ( z ) − z‖22 ] . ( 3 ) In practice , we observe that Lφlr , H is often small without explicit minimization , which we attribute to its consistency with the minimization of Lr . By Theorem 1 , the problem of training the generative model reduces to training h to map N ( 0 , I ) to Ĥ , which we refer to as the perceptual generative model . In the subsequent subsections , we present a maximum likelihood approach , a VAE-based approach , and a unified approach to train the perceptual generative model . The basic loss function of PGAs is given by Lpga = Lr + αL φ lr , N + βL φ lr , H , ( 4 ) where α and β are hyperparameters to be tuned . Eq . ( 4 ) is also illustrated in Fig . 1a . 3.2 A MAXIMUM LIKELIHOOD APPROACH . We first assume the invertibility of h. For x̂ ∼ D̂ , let ẑ = fφ ( x̂ ) ∼ Ĥ . We can train h directly with maximum likelihood using the change of variables formula as Eẑ∼Ĥ [ log p ( ẑ ) ] = Ez∼H [ log p ( z ) − log ∣∣∣∣det ( ∂h ( z ) ∂z ) ∣∣∣∣ ] . ( 5 ) Ideally , we would like to maximize Eq . ( 5 ) only w.r.t . the parameters of the generative model ( i.e. , θ ) . However , directly optimizing the first term in Eq . ( 5 ) requires computing z = h−1 ( ẑ ) , which is usually unknown . Nevertheless , for ẑ ∼ Ĥ , we have h−1 ( ẑ ) = fφ ( x ) and x ∼ D , and thus we can minimize the following loss function w.r.t . φ instead : Lφnll = −Ez∼H [ log p ( z ) ] = 1 2 Ex∼D [ ‖fφ ( x ) ‖22 ] . ( 6 ) To avoid computing the Jacobian in the second term of Eq . ( 5 ) , which is slow for unrestricted architectures , we approximate the Jacobian determinant and derive a loss function to be minimized w.r.t . θ : Lθnll = H 2 Ez∼H , δ∼S ( ) [ log ‖h ( z + δ ) − h ( z ) ‖22 ‖δ‖22 ] ≈ Ez∼H [ log ∣∣∣∣det ( ∂h ( z ) ∂z ) ∣∣∣∣ ] , ( 7 ) where S ( ) can be either N ( 0 , 2I ) , or a uniform distribution on a small ( H−1 ) -sphere of radius centered at the origin . The latter choice is expected to introduce slightly less variance . We show below that the approximation gives an upper bound when → 0 . Eqs . ( 6 ) and ( 7 ) are illustrated in Fig . 1b . Proposition 1 . For → 0 , log ∣∣∣∣det ( ∂h ( z ) ∂z ) ∣∣∣∣ ≤ H2 Eδ∼S ( ) [ log ‖h ( z + δ ) − h ( z ) ‖22 ‖δ‖22 ] . ( 8 ) The inequality is tight if h is a multiple of the identity function around z . We defer the proof to Appendix B.2 . We note that the above discussion relies on the invertibility of h , which , however , is not required by the resulting method . Indeed , when h is invertible at some point z , the latent reconstruction loss ensures that h is close to the identity function around z , and hence the tightness of the upper bound in Eq . ( 8 ) . Otherwise , when h is not invertible at some z , the logarithm of the Jacobian determinant at z becomes infinite , in which case Eq . ( 5 ) can not be optimized . Nevertheless , since ‖h ( z + δ ) − h ( z ) ‖22 is unlikely to be zero if the model is properly initialized , the approximation in Eq . ( 7 ) remains finite , and thus can be optimized regardless . To summarize , we train the autoencoder to obtain a generative model by minimizing the following loss function : Llpga = Lpga + γ ( Lφnll + L θ nll ) . ( 9 ) We refer to this approach as maximum likelihood PGA ( LPGA ) .

Summary: The paper considers generative models and proposes to change the VAE approach in the following way. While VAEs assume a generative model (prior on the latent space + stochastic generator) and aim at learning its parameters so as to maximize the likelihood of the (training) data distribution, the authors propose to derive the learning objective from a different view. In my understanding, they consider the two composed mappings generator(encoder) and encoder(generator) and require the first one to have the data distribution as a fixpoint and the second one to have the latent prior as a fixpoint. Starting from this idea they derive an objective function for the case that the mappings are deterministic and then further enrich the objective by either likelihood based terms or VAE based terms. The new approach is analysed experimentally on MNIST, CIFAR and CelebA datasets. The authors report quantitative improvements in terms of Frechet inception distance (FID) as well as sharper samples (when compared to standard VAEs).