Generative Latent Flow

1 INTRODUCTION . Generative models have attracted much attention in the literature on deep learning . These models are used to formulate the distribution of complex data as a function of random noise passed through a network , so that rendering samples from the distribution is particularly easy . The most dominant generative models are Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , as they have exhibited impressive performance in generating high quality images ( Radford et al. , 2015 ; Brock et al. , 2018 ) and in other vision tasks ( Zhu et al. , 2017 ; Ledig et al. , 2017 ) . Despite their success , training GANs can be challenging , partly because they are trained by solving a saddle point optimization problem formulated as an adversarial game . It is well known that training GANs is unstable and sensitive to hyper-parameter settings ( Salimans et al. , 2016 ; Arora et al. , 2017 ) , and sometimes training leads to mode collapse ( Goodfellow , 2016 ) . Although there have been multiple efforts to overcome the difficulties in training GANs ( Arjovsky et al. , 2017 ; Metz Luke & SohlDickstein , 2017 ; Srivastava et al. , 2017 ; Miyato et al. , 2018 ) , researchers are also actively studying non-adversarial methods that are known to be less affected by these issues . Some models explicitly define p ( x ) , the distribution of the data , and training is guided by maximizing the data likelihood . One approach is to express the data distribution in an auto-regressive pattern ( Papamakarios et al. , 2017 ; Oord et al. , 2016 ) ; another is to express it as an invertible transformation of a simple distribution using the change of variable formula , where the invertible transformation is defined using a normalizing flow network ( Dinh et al. , 2014 ; 2016 ; Kingma & Dhariwal , 2018 ) . While being mathematically clear and well defined , normalizing flows keep the dimensionality of the original data in order to maintain bijectivity . Consequently , they can not provide low-dimensional representations of the data and training is computationally expensive . Considering the prohibitively long training time and advanced hardware requirements in training large scale flow models such as ( Kingma & Dhariwal , 2018 ) , we believe that it is worth exploring the application of flows in the low dimensional representation spaces rather than for the original data . Another class of generative models employs an encoder-decoder structure and low dimensional latent variables to represent and generate the data . An encoder is used to produce estimates of the latent variables corresponding to a particular data point , and samples from a predefined prior distribution on the latent space are passed through a decoder to produce new samples from the data distribution . We call these auto-encoder ( AE ) based models , of which variational auto-encoders ( VAEs ) are perhaps the most influential ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . VAEs use the encoder to produce approximations to the posterior distribution of the latent variable given the data , and the training objective is to maximize a variational lower bound of the data log likelihood . VAEs are easy to train , but their generation quality still lies far below that of GANs , as they tend to generate blurry images ( Dosovitskiy & Brox , 2016 ) . Whereas the original VAE uses a standard Gaussian prior , it can be extended by introducing a learnable parameterized prior distribution . There have been a number of studies in this direction ( see section 2 ) , some of which use a normalizing flow parameterization , where the prior is modeled as a trainable continuous bijective transformation of the standard Gaussian . We carefully study this method , and make the surprising novel observation that in order to produce high quality samples , it is necessary to significantly increase the weight on the reconstruction loss . This corresponds to decreasing the variance of the observational noise of the generative model at each pixel , where we are assuming the data distribution is factorial Gaussian conditioned on the output of the decoder , which yields the MSE as the reconstruction loss . It is important to note that increasing this weight alone without access to a trainable prior does not consistently improve generation quality . We show that as this weight increases , we approach a vanishing noise limit that corresponds to a deterministic auto-encoder . This leads to a new algorithm we call Generative Latent Flow ( GLF ) , which combines a deterministic auto-encoder that learns a mapping to and from a latent space , and a normalizing flow that matches the standard Gaussian to the distribution of latent variables of the training data produced by the encoder . Our contributions are summarized as follows : i ) we carefully study the effects of equipping VAEs with a normalizing flow prior on image generation quality as the weight of the reconstruction loss is increased . ii ) Based on this finding , introduce Generative Latent Flow , which uses auto-encoders instead of VAEs . iii ) Through standard evaluations , we show that our proposed model achieves state-of-the-art sample quality among competing AE based models , and has the additional advantage of faster convergence . 2 RELATED WORK . In general , in order for an AE based model with encoder-decoder structure to generate samples resembling the training data distribution , two criteria need to be ensured : ( a ) the decoder is able to produce a good reconstruction of a training image given its encoded latent variable z ; and ( b ) the empirical latent distribution q ( z ) of z ’ s returned by the encoder is close to the prior p ( z ) . In VAEs , the empirical latent distribution is often called aggregated or marginal posterior : q ( z ) = Ex∼pdata [ q ( z|x ) ] . While ( a ) is mainly driven by the reconstruction loss , satisfying criterion ( b ) is more complicated . Intuitively , criterion ( b ) can possibly be achieved by designing mechanisms that either modify the empirical latent distribution q ( z ) , or conversely modify the prior p ( z ) . There is plenty of previous work in both directions . Modifying the empirical latent distribution q ( z ) : In the classic VAE model , DKL ( q ( z|x ) ‖p ( z ) ) in the ELBO loss can be decomposed as DKL ( q ( z ) ‖p ( z ) ) plus a mutual information term as shown in ( Hoffman & Johnson , 2016 ) . Therefore , VAEs modify q ( z ) indirectly through regularizing the posterior distribution q ( z|x ) . Several modifications to VAE ’ s loss ( Chen et al. , 2018 ; Kim & Mnih , 2018 ) , which are designed for the task of unsupervised disentanglement , put a stronger penalty specifically on the mismatch between q ( z ) and p ( z ) . There are also attempts to incorporate normalizing flows into the encoder to provide more flexible approximate posteriors ( Rezende & Mohamed , 2015 ; Kingma et al. , 2016 ; Berg et al. , 2018 ) . However , empirical evaluation shows that VAEs with flow posteriors do not reduce the mismatch between q ( z ) and p ( z ) ( Rosca et al. , 2018 ) . Furthermore , as of yet , all these modifications to VAEs have not been shown to improve generation quality . Adversarial auto-encoders ( AAEs ) ( Makhzani et al. , 2015 ) and Wasserstein auto-encoders ( WAEs ) ( Tolstikhin et al. , 2017 ) use an adversarial regularizer or MMD regularizer ( Gretton et al. , 2012 ) to force q ( z ) to be close to p ( z ) . WAEs are shown to improve generation quality , as they generate sharper images than VAEs do . Modifying the prior distribution p ( z ) : An alternative to modifying the approximate posterior is using a trainable prior . ( Tomczak & Welling , 2017 ; Klushyn et al. , 2019 ; Bauer & Mnih , 2018 ) propose different ways to approximate q ( z ) using a sampled mixture of posteriors during training , and then use the approximated q ( z ) as the prior in the VAE . This is a natural way to let the prior match q ( z ) , however , these methods have not been shown to improve generation quality . Two-stage VAE ( Dai & Wipf , 2019 ) introduces another VAE on the latent space defined by the first VAE to learn the distribution of its latent variables . VQ-VAE ( Oord et al. , 2017 ) first trains an auto-encoder with discrete latent variables , and then fits an auto-regressive prior on the latent space . GLANN ( Hoshen et al. , 2019 ) learns a latent representation by GLO ( Bojanowski et al. , 2017 ) and matches the densities of the latent variables with an implicit maximum likelihood estimator ( Li & Malik , 2018 ) . RAE+GMM ( Ghosh et al. , 2019 ) trains a regularized auto-encoder ( Alain & Bengio , 2014 ) and fits a mixture of Gaussian distribution on the latent space . Note that all these methods involve two stage-training , which means that the prior distribution is fitted after training the variational or deterministic auto-encoder . They have been shown to improve the quality of the generated images . VAEs with a normalizing flow as a learnable prior ( Chen et al. , 2016b ; Huang et al. , 2017 ) also fall into this category . Since these are the main focus of this paper , we discuss them in detail in Section 3.2 . We note that modifications of VAEs with a normalizing flow posterior have been extensively studied . In contrast , VAEs with flow prior have attracted much less attention . ( Huang et al. , 2017 ) briefly discusses this model to solve the distribution mismatch in the latent space , and recently ( Xu et al. , 2019 ) shows the advantages of learning a flow prior over learning a flow posterior . However , these papers only focus on improvements of the data likelihood . Here we study the model from the perspective of the effects of the normalizing flow prior on sample generation quality , leading to some important and novel observations . 3 COMBINING NORMALIZING FLOW WITH AE BASED MODELS . In this section , we discuss the combination of normalizing flow priors with AE based models in detail . We first review normalizing flows in section 3.1 , then in section 3.2 we introduce VAEs with normalizing flow prior and present some novel observations with respect to this model . Finally in section 3.3 we propose Generative Latent Flow ( GLF ) to further simplify the model and improve performance . 3.1 REVIEW : NORMALIZING FLOWS . Normalizing flows are carefully-designed invertible networks that map the training data to a simple distribution . Let z ∈ Z be an observation from an unknown target distribution z ∼ p ( z ) and p be the unit Gaussian prior distribution on E . Given a bijection fθ : Z → E , we define a probability model pθ ( z ) with parameters θ on Z . The negative log likelihood ( NLL ) of z is computed by the change of variable formula : LNLL ( fθ ( z ) ) , − log ( pθ ( z ) ) = − ( log p ( fθ ( z ) ) + log ∣∣∣∣det ( ∂fθ ( z ) ∂z ) ∣∣∣∣ ) , ( 1 ) where ∂fθ ( z ) ∂z is the Jacobian matrix of fθ . In order to learn the flow fθ , the NLL objective of z is minimized , which is equivalent to maximizing the likelihood of z . Since the mapping is a bijection , sampling from the trained model pθ ( z ) is trivial : simply sample ∼ p and compute z = f−1θ ( ) . The key to designing a normalizing flow model is defining the transformation fθ so that the inverse transformation and the determinant of the Jacobian matrix can be efficiently computed . Based on ( Dinh et al. , 2016 ) , we adopt the following layers to form the flows used in our model . Affine coupling layer : Given D dimensional input data z and d < D , we partition the input into two vectors z1 = z1 : d and z2 = zd+1 : D. The output of one affine coupling layer is given by y1 = z1 , y2 = z2 exp ( s ( z1 ) ) + t ( z1 ) where s and t are functions from Rd → RD−d and is the element-wise product . The inverse of the transformation is explicitly given by z1 = y1 , z2 = ( y2 − t ( y1 ) ) exp ( −s ( y1 ) ) . The determinant of the Jacobian matrix of this transformation is det∂y∂z = ∏d j=1 ( exp [ s ( z1 ) j ] ) . Since computing both the inverse and the Jacobian of an affine coupling layer does not require computing the inverse and Jacobian of s and t , both functions can be arbitrarily complex . Combining coupling layers with random permutation : Affine coupling layers leave some components of the input data unchanged . In order to transform all the components , two coupling layers are combined in an alternating pattern to form a coupling block , so that the unchanged components in the first layer can be transformed in the second layer . In particular , we add a fixed random permutation of the coordinates of the input data at the end of each coupling block . See Figure 1b for an illustration of a coupling block used in our model .

Summary: The authors propose a model that combines a simple Auto-Encoder (AE) together with a Normalizing Flow (NF) model, such that to derive a generative model. In particular, the AE is used to learn a low-dimensional representation of the given data in a latent space. Then, a NF model learns under a maximum likelihood principle, the distribution of these latent codes by applying an invertible transformation on a easy to sample distribution.