Generative Latent Flow

1 INTRODUCTION . Generative models have attracted much attention in the literature on deep learning . These models are used to formulate the distribution of complex data as a function of random noise passed through a network , so that rendering samples from the distribution is particularly easy . The most dominant generative models are Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , as they have exhibited impressive performance in generating high quality images ( Radford et al. , 2015 ; Brock et al. , 2018 ) and in other vision tasks ( Zhu et al. , 2017 ; Ledig et al. , 2017 ) . Despite their success , training GANs can be challenging , partly because they are trained by solving a saddle point optimization problem formulated as an adversarial game . It is well known that training GANs is unstable and sensitive to hyper-parameter settings ( Salimans et al. , 2016 ; Arora et al. , 2017 ) , and sometimes training leads to mode collapse ( Goodfellow , 2016 ) . Although there have been multiple efforts to overcome the difficulties in training GANs ( Arjovsky et al. , 2017 ; Metz Luke & SohlDickstein , 2017 ; Srivastava et al. , 2017 ; Miyato et al. , 2018 ) , researchers are also actively studying non-adversarial methods that are known to be less affected by these issues . Some models explicitly define p ( x ) , the distribution of the data , and training is guided by maximizing the data likelihood . One approach is to express the data distribution in an auto-regressive pattern ( Papamakarios et al. , 2017 ; Oord et al. , 2016 ) ; another is to express it as an invertible transformation of a simple distribution using the change of variable formula , where the invertible transformation is defined using a normalizing flow network ( Dinh et al. , 2014 ; 2016 ; Kingma & Dhariwal , 2018 ) . While being mathematically clear and well defined , normalizing flows keep the dimensionality of the original data in order to maintain bijectivity . Consequently , they can not provide low-dimensional representations of the data and training is computationally expensive . Considering the prohibitively long training time and advanced hardware requirements in training large scale flow models such as ( Kingma & Dhariwal , 2018 ) , we believe that it is worth exploring the application of flows in the low dimensional representation spaces rather than for the original data . Another class of generative models employs an encoder-decoder structure and low dimensional latent variables to represent and generate the data . An encoder is used to produce estimates of the latent variables corresponding to a particular data point , and samples from a predefined prior distribution on the latent space are passed through a decoder to produce new samples from the data distribution . We call these auto-encoder ( AE ) based models , of which variational auto-encoders ( VAEs ) are perhaps the most influential ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . VAEs use the encoder to produce approximations to the posterior distribution of the latent variable given the data , and the training objective is to maximize a variational lower bound of the data log likelihood . VAEs are easy to train , but their generation quality still lies far below that of GANs , as they tend to generate blurry images ( Dosovitskiy & Brox , 2016 ) . Whereas the original VAE uses a standard Gaussian prior , it can be extended by introducing a learnable parameterized prior distribution . There have been a number of studies in this direction ( see section 2 ) , some of which use a normalizing flow parameterization , where the prior is modeled as a trainable continuous bijective transformation of the standard Gaussian . We carefully study this method , and make the surprising novel observation that in order to produce high quality samples , it is necessary to significantly increase the weight on the reconstruction loss . This corresponds to decreasing the variance of the observational noise of the generative model at each pixel , where we are assuming the data distribution is factorial Gaussian conditioned on the output of the decoder , which yields the MSE as the reconstruction loss . It is important to note that increasing this weight alone without access to a trainable prior does not consistently improve generation quality . We show that as this weight increases , we approach a vanishing noise limit that corresponds to a deterministic auto-encoder . This leads to a new algorithm we call Generative Latent Flow ( GLF ) , which combines a deterministic auto-encoder that learns a mapping to and from a latent space , and a normalizing flow that matches the standard Gaussian to the distribution of latent variables of the training data produced by the encoder . Our contributions are summarized as follows : i ) we carefully study the effects of equipping VAEs with a normalizing flow prior on image generation quality as the weight of the reconstruction loss is increased . ii ) Based on this finding , introduce Generative Latent Flow , which uses auto-encoders instead of VAEs . iii ) Through standard evaluations , we show that our proposed model achieves state-of-the-art sample quality among competing AE based models , and has the additional advantage of faster convergence . 2 RELATED WORK . In general , in order for an AE based model with encoder-decoder structure to generate samples resembling the training data distribution , two criteria need to be ensured : ( a ) the decoder is able to produce a good reconstruction of a training image given its encoded latent variable z ; and ( b ) the empirical latent distribution q ( z ) of z ’ s returned by the encoder is close to the prior p ( z ) . In VAEs , the empirical latent distribution is often called aggregated or marginal posterior : q ( z ) = Ex∼pdata [ q ( z|x ) ] . While ( a ) is mainly driven by the reconstruction loss , satisfying criterion ( b ) is more complicated . Intuitively , criterion ( b ) can possibly be achieved by designing mechanisms that either modify the empirical latent distribution q ( z ) , or conversely modify the prior p ( z ) . There is plenty of previous work in both directions . Modifying the empirical latent distribution q ( z ) : In the classic VAE model , DKL ( q ( z|x ) ‖p ( z ) ) in the ELBO loss can be decomposed as DKL ( q ( z ) ‖p ( z ) ) plus a mutual information term as shown in ( Hoffman & Johnson , 2016 ) . Therefore , VAEs modify q ( z ) indirectly through regularizing the posterior distribution q ( z|x ) . Several modifications to VAE ’ s loss ( Chen et al. , 2018 ; Kim & Mnih , 2018 ) , which are designed for the task of unsupervised disentanglement , put a stronger penalty specifically on the mismatch between q ( z ) and p ( z ) . There are also attempts to incorporate normalizing flows into the encoder to provide more flexible approximate posteriors ( Rezende & Mohamed , 2015 ; Kingma et al. , 2016 ; Berg et al. , 2018 ) . However , empirical evaluation shows that VAEs with flow posteriors do not reduce the mismatch between q ( z ) and p ( z ) ( Rosca et al. , 2018 ) . Furthermore , as of yet , all these modifications to VAEs have not been shown to improve generation quality . Adversarial auto-encoders ( AAEs ) ( Makhzani et al. , 2015 ) and Wasserstein auto-encoders ( WAEs ) ( Tolstikhin et al. , 2017 ) use an adversarial regularizer or MMD regularizer ( Gretton et al. , 2012 ) to force q ( z ) to be close to p ( z ) . WAEs are shown to improve generation quality , as they generate sharper images than VAEs do . Modifying the prior distribution p ( z ) : An alternative to modifying the approximate posterior is using a trainable prior . ( Tomczak & Welling , 2017 ; Klushyn et al. , 2019 ; Bauer & Mnih , 2018 ) propose different ways to approximate q ( z ) using a sampled mixture of posteriors during training , and then use the approximated q ( z ) as the prior in the VAE . This is a natural way to let the prior match q ( z ) , however , these methods have not been shown to improve generation quality . Two-stage VAE ( Dai & Wipf , 2019 ) introduces another VAE on the latent space defined by the first VAE to learn the distribution of its latent variables . VQ-VAE ( Oord et al. , 2017 ) first trains an auto-encoder with discrete latent variables , and then fits an auto-regressive prior on the latent space . GLANN ( Hoshen et al. , 2019 ) learns a latent representation by GLO ( Bojanowski et al. , 2017 ) and matches the densities of the latent variables with an implicit maximum likelihood estimator ( Li & Malik , 2018 ) . RAE+GMM ( Ghosh et al. , 2019 ) trains a regularized auto-encoder ( Alain & Bengio , 2014 ) and fits a mixture of Gaussian distribution on the latent space . Note that all these methods involve two stage-training , which means that the prior distribution is fitted after training the variational or deterministic auto-encoder . They have been shown to improve the quality of the generated images . VAEs with a normalizing flow as a learnable prior ( Chen et al. , 2016b ; Huang et al. , 2017 ) also fall into this category . Since these are the main focus of this paper , we discuss them in detail in Section 3.2 . We note that modifications of VAEs with a normalizing flow posterior have been extensively studied . In contrast , VAEs with flow prior have attracted much less attention . ( Huang et al. , 2017 ) briefly discusses this model to solve the distribution mismatch in the latent space , and recently ( Xu et al. , 2019 ) shows the advantages of learning a flow prior over learning a flow posterior . However , these papers only focus on improvements of the data likelihood . Here we study the model from the perspective of the effects of the normalizing flow prior on sample generation quality , leading to some important and novel observations . 3 COMBINING NORMALIZING FLOW WITH AE BASED MODELS . In this section , we discuss the combination of normalizing flow priors with AE based models in detail . We first review normalizing flows in section 3.1 , then in section 3.2 we introduce VAEs with normalizing flow prior and present some novel observations with respect to this model . Finally in section 3.3 we propose Generative Latent Flow ( GLF ) to further simplify the model and improve performance . 3.1 REVIEW : NORMALIZING FLOWS . Normalizing flows are carefully-designed invertible networks that map the training data to a simple distribution . Let z ∈ Z be an observation from an unknown target distribution z ∼ p ( z ) and p be the unit Gaussian prior distribution on E . Given a bijection fθ : Z → E , we define a probability model pθ ( z ) with parameters θ on Z . The negative log likelihood ( NLL ) of z is computed by the change of variable formula : LNLL ( fθ ( z ) ) , − log ( pθ ( z ) ) = − ( log p ( fθ ( z ) ) + log ∣∣∣∣det ( ∂fθ ( z ) ∂z ) ∣∣∣∣ ) , ( 1 ) where ∂fθ ( z ) ∂z is the Jacobian matrix of fθ . In order to learn the flow fθ , the NLL objective of z is minimized , which is equivalent to maximizing the likelihood of z . Since the mapping is a bijection , sampling from the trained model pθ ( z ) is trivial : simply sample ∼ p and compute z = f−1θ ( ) . The key to designing a normalizing flow model is defining the transformation fθ so that the inverse transformation and the determinant of the Jacobian matrix can be efficiently computed . Based on ( Dinh et al. , 2016 ) , we adopt the following layers to form the flows used in our model . Affine coupling layer : Given D dimensional input data z and d < D , we partition the input into two vectors z1 = z1 : d and z2 = zd+1 : D. The output of one affine coupling layer is given by y1 = z1 , y2 = z2 exp ( s ( z1 ) ) + t ( z1 ) where s and t are functions from Rd → RD−d and is the element-wise product . The inverse of the transformation is explicitly given by z1 = y1 , z2 = ( y2 − t ( y1 ) ) exp ( −s ( y1 ) ) . The determinant of the Jacobian matrix of this transformation is det∂y∂z = ∏d j=1 ( exp [ s ( z1 ) j ] ) . Since computing both the inverse and the Jacobian of an affine coupling layer does not require computing the inverse and Jacobian of s and t , both functions can be arbitrarily complex . Combining coupling layers with random permutation : Affine coupling layers leave some components of the input data unchanged . In order to transform all the components , two coupling layers are combined in an alternating pattern to form a coupling block , so that the unchanged components in the first layer can be transformed in the second layer . In particular , we add a fixed random permutation of the coordinates of the input data at the end of each coupling block . See Figure 1b for an illustration of a coupling block used in our model .

Summary: The paper proposes a new model combining an auto-encoder (AE) and a normalising flow (NF). The model, Generative Latent Flow (GLF), uses the AE to map the inputs to a latent space, which is then transformed using the NF. The approach is intuitively beneficial in that the AE can reduce the dimensionality of the inputs such that the NF mapping becomes much faster, computationally. The proposed method is compared to related methods that use a variational AE (VAE) in combination with an NF, and the similarities are pointed out and studied empirically.