Title: GENERATIVE LATENT FLOW

Abstract: In this work we propose Generative Latent Flow (GLF), an algorithm for generative modeling of data distributions. GLF uses an auto-encoder to learn latent representations of the data, and a normalizing flow to map the distribution of the latent variables to that of a standard Gaussian. GLF can be seen as a variational auto-encoder, with normalizing flow prior, and a vanishing limit of the pixel-wise variance of the data. We carefully study this relationship and the pros and cons of using an auto-encoder vs. a variational auto-encoder. In contrast to a number of other auto-encoder based generative models, which use various regularizers to encourage the encoded latent distribution to match the prior distribution, our model explicitly constructs a mapping between these two distributions, leading to better density matching while avoiding over regularizing the latent variables. We compare our model with several related techniques, and show that under standard quantitative evaluations, our method achieves state-of-the-art sample quality and diversity among AE based models on commonly used datasets and is competitive with GANs' benchmarks.

1. INTRODUCTION:

Generative models have attracted much attention in the literature on deep learning. These models are used to formulate the distribution of complex data as a function of random noise passed through a network, so that rendering samples from the distribution is particularly easy. The most dominant generative models are Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) , as they have exhibited impressive performance in generating high quality images (Radford et al., 2015; Brock et al., 2018) and in other vision tasks (Zhu et al., 2017; Ledig et al., 2017) . Despite their success, training GANs can be challenging, partly because they are trained by solving a saddle point optimization problem formulated as an adversarial game. It is well known that training GANs is unstable and sensitive to hyper-parameter settings (Salimans et al., 2016; Arora et al., 2017) , and sometimes training leads to mode collapse (Goodfellow, 2016) . Although there have been multiple efforts to overcome the difficulties in training GANs (Arjovsky et al., 2017; Metz Luke & Sohl-Dickstein, 2017; Srivastava et al., 2017; Miyato et al., 2018) , researchers are also actively studying non-adversarial methods that are known to be less affected by these issues. Some models explicitly define p(x), the distribution of the data, and training is guided by maximizing the data likelihood. One approach is to express the data distribution in an auto-regressive pattern (Papamakarios et al., 2017; Oord et al., 2016) ; another is to express it as an invertible transformation of a simple distribution using the change of variable formula, where the invertible transformation is defined using a normalizing flow network (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018) . While being mathematically clear and well defined, normalizing flows keep the dimensionality of the original data in order to maintain bijectivity. Consequently, they cannot provide low-dimensional representations of the data and training is computationally expensive. Considering the prohibitively long training time and advanced hardware requirements in training large scale flow models such as (Kingma & Dhariwal, 2018) , we believe that it is worth exploring the application of flows in the low dimensional representation spaces rather than for the original data.

Another class of generative models employs an encoder-decoder structure and low dimensional latent variables to represent and generate the data. An encoder is used to produce estimates of the latent variables corresponding to a particular data point, and samples from a predefined prior distribution on the latent space are passed through a decoder to produce new samples from the data distribution. We call these auto-encoder (AE) based models, of which variational auto-encoders (VAEs) are perhaps the most influential (Kingma & Welling, 2013; Rezende et al., 2014) . VAEs use the encoder to produce approximations to the posterior distribution of the latent variable given the data, and the training objective is to maximize a variational lower bound of the data log likelihood. VAEs are easy to train, but their generation quality still lies far below that of GANs, as they tend to generate blurry images (Dosovitskiy & Brox, 2016) .

Whereas the original VAE uses a standard Gaussian prior, it can be extended by introducing a learnable parameterized prior distribution. There have been a number of studies in this direction (see section 2), some of which use a normalizing flow parameterization, where the prior is modeled as a trainable continuous bijective transformation of the standard Gaussian. We carefully study this method, and make the surprising novel observation that in order to produce high quality samples, it is necessary to significantly increase the weight on the reconstruction loss. This corresponds to decreasing the variance of the observational noise of the generative model at each pixel, where we are assuming the data distribution is factorial Gaussian conditioned on the output of the decoder, which yields the MSE as the reconstruction loss. It is important to note that increasing this weight alone without access to a trainable prior does not consistently improve generation quality.

We show that as this weight increases, we approach a vanishing noise limit that corresponds to a deterministic auto-encoder. This leads to a new algorithm we call Generative Latent Flow (GLF), which combines a deterministic auto-encoder that learns a mapping to and from a latent space, and a normalizing flow that matches the standard Gaussian to the distribution of latent variables of the training data produced by the encoder.

Our contributions are summarized as follows: i) we carefully study the effects of equipping VAEs with a normalizing flow prior on image generation quality as the weight of the reconstruction loss is increased. ii) Based on this finding, introduce Generative Latent Flow, which uses auto-encoders instead of VAEs. iii) Through standard evaluations, we show that our proposed model achieves state-of-the-art sample quality among competing AE based models, and has the additional advantage of faster convergence.

2. RELATED WORK:

In general, in order for an AE based model with encoder-decoder structure to generate samples resembling the training data distribution, two criteria need to be ensured: (a) the decoder is able to produce a good reconstruction of a training image given its encoded latent variable z; and (b) the empirical latent distribution q(z) of z's returned by the encoder is close to the prior p(z). In VAEs, the empirical latent distribution is often called aggregated or marginal posterior: q(z) = E x∼p data [q(z|x)]. While (a) is mainly driven by the reconstruction loss, satisfying criterion (b) is more complicated. Intuitively, criterion (b) can possibly be achieved by designing mechanisms that either modify the empirical latent distribution q(z), or conversely modify the prior p(z). There is plenty of previous work in both directions.

Modifying the empirical latent distribution q(z): In the classic VAE model, D KL (q(z|x) p(z)) in the ELBO loss can be decomposed as D KL (q(z) p(z)) plus a mutual information term as shown in (Hoffman & Johnson, 2016) . Therefore, VAEs modify q(z) indirectly through regularizing the posterior distribution q(z|x). Several modifications to VAE's loss (Chen et al., 2018; Kim & Mnih, 2018) , which are designed for the task of unsupervised disentanglement, put a stronger penalty specifically on the mismatch between q(z) and p(z). There are also attempts to incorporate normalizing flows into the encoder to provide more flexible approximate posteriors (Rezende & Mohamed, 2015; Kingma et al., 2016; Berg et al., 2018) . However, empirical evaluation shows that VAEs with flow posteriors do not reduce the mismatch between q(z) and p(z) (Rosca et al., 2018) . Furthermore, as of yet, all these modifications to VAEs have not been shown to improve generation quality. Adversarial auto-encoders (AAEs) (Makhzani et al., 2015) and Wasserstein auto-encoders (WAEs) (Tolstikhin et al., 2017) use an adversarial regularizer or MMD regularizer (Gretton et al., 2012) to force q(z) to be close to p(z). WAEs are shown to improve generation quality, as they generate sharper images than VAEs do.

Modifying the prior distribution p(z): An alternative to modifying the approximate posterior is using a trainable prior. (Tomczak & Welling, 2017; Klushyn et al., 2019; Bauer & Mnih, 2018) propose different ways to approximate q(z) using a sampled mixture of posteriors during training, and then use the approximated q(z) as the prior in the VAE. This is a natural way to let the prior match q(z), however, these methods have not been shown to improve generation quality. Two-stage VAE (Dai & Wipf, 2019) introduces another VAE on the latent space defined by the first VAE to learn the distribution of its latent variables. VQ-VAE (Oord et al., 2017) first trains an auto-encoder with discrete latent variables, and then fits an auto-regressive prior on the latent space. GLANN (Hoshen et al., 2019) learns a latent representation by GLO (Bojanowski et al., 2017) and matches the densities of the latent variables with an implicit maximum likelihood estimator (Li & Malik, 2018) . RAE+GMM (Ghosh et al., 2019) trains a regularized auto-encoder (Alain & Bengio, 2014) and fits a mixture of Gaussian distribution on the latent space. Note that all these methods involve two stage-training, which means that the prior distribution is fitted after training the variational or deterministic auto-encoder. They have been shown to improve the quality of the generated images. VAEs with a normalizing flow as a learnable prior (Chen et al., 2016b; Huang et al., 2017 ) also fall into this category. Since these are the main focus of this paper, we discuss them in detail in Section 3.2.

We note that modifications of VAEs with a normalizing flow posterior have been extensively studied. In contrast, VAEs with flow prior have attracted much less attention. (Huang et al., 2017) briefly discusses this model to solve the distribution mismatch in the latent space, and recently (Xu et al., 2019) shows the advantages of learning a flow prior over learning a flow posterior. However, these papers only focus on improvements of the data likelihood. Here we study the model from the perspective of the effects of the normalizing flow prior on sample generation quality, leading to some important and novel observations. 

3. COMBINING NORMALIZING FLOW WITH AE BASED MODELS:

In this section, we discuss the combination of normalizing flow priors with AE based models in detail. We first review normalizing flows in section 3.1, then in section 3.2 we introduce VAEs with normalizing flow prior and present some novel observations with respect to this model. Finally in section 3.3 we propose Generative Latent Flow (GLF) to further simplify the model and improve performance.

3.1. REVIEW: NORMALIZING FLOWS:

Normalizing flows are carefully-designed invertible networks that map the training data to a simple distribution. Let z ∈ Z be an observation from an unknown target distribution z ∼ p(z) and p be the unit Gaussian prior distribution on E. Given a bijection f θ : Z → E, we define a probability model p θ (z) with parameters θ on Z. The negative log likelihood (NLL) of z is computed by the change of variable formula:

EQUATION

where ∂f θ (z) ∂z is the Jacobian matrix of f θ . In order to learn the flow f θ , the NLL objective of z is minimized, which is equivalent to maximizing the likelihood of z. Since the mapping is a bijection, sampling from the trained model p θ (z) is trivial: simply sample ∼ p and compute z = f -1 θ ( ). The key to designing a normalizing flow model is defining the transformation f θ so that the inverse transformation and the determinant of the Jacobian matrix can be efficiently computed. Based on (Dinh et al., 2016) , we adopt the following layers to form the flows used in our model.

Affine coupling layer: Given D dimensional input data z and d < D, we partition the input into two vectors z 1 = z 1:d and z 2 = z d+1:D . The output of one affine coupling layer is given by y 1 = z 1 , y 2 = z 2 exp(s(z 1 )) + t(z 1 ) where s and t are functions from R d → R D-d and is the element-wise product. The inverse of the transformation is explicitly given by z 1 = y 1 , z 2 = (y 2 -t(y 1 )) exp(-s(y 1 )). The determinant of the Jacobian matrix of this transformation is

det ∂y ∂z = d j=1 (exp[s(z 1 ) j ]

). Since computing both the inverse and the Jacobian of an affine coupling layer does not require computing the inverse and Jacobian of s and t, both functions can be arbitrarily complex.

Combining coupling layers with random permutation: Affine coupling layers leave some components of the input data unchanged. In order to transform all the components, two coupling layers are combined in an alternating pattern to form a coupling block, so that the unchanged components in the first layer can be transformed in the second layer. In particular, we add a fixed random permutation of the coordinates of the input data at the end of each coupling block. See Figure 1b for an illustration of a coupling block used in our model.

3.2. VAES WITH NORMALIZING FLOW PRIOR:

We begin by introducing the training loss of the model. Consider the ELBO loss of standard VAEs with Gaussian prior and posterior (η, φ denote the parameters of encoder and decoder, respectively):

ELBO(η, φ) = E p data (x) E qη(z|x) [β • log p φ (x|z) + log p(z) -log q η (z|x)] .

(2)

The first term is related to the reconstruction loss, while the last two terms can be combined as D KL (q(z|x) p(z)). β > 0 is a hyper-parameter that controls the relative weight of the reconstruction loss and the KL divergence loss. In the standard formulation of VAEs, p φ assumes an independent Gaussian distribution with variance 1 at each pixel. The parameter β allows us to adjust this variance as 1/β.

If we introduce a normalizing flow f θ for the prior distribution, then the prior

p θ becomes p θ (z) = p (f θ (z)) det ∂f θ (z)

∂z:

, where p is the standard Gaussian density. Substituting this prior into equation 2, we obtain ELBO(η, φ, θ) for VAEs with flow prior:

E p data (x) E qη(z|x) β • log p φ (x|z) + log p (f θ (z)) + log det ∂f θ (z) ∂z -log q η (z|x) . (3)

The second and third terms together are the log-likelihood of z under the prior distribution modeled by the flow (corresponding to the negative of equation 1). The last term corresponds to the entropy of the posterior distribution returned by the encoder. Both the VAE and the normalizing flow are trained by minimizing -ELBO(η, φ, θ).

Previous work on VAEs with a flow prior did not consider tuning β (which means the reconstruction loss and the KL loss are weighted equally) as they focused on comparing the obtained log likelihoods with those from plain VAEs. However, we observe that when β = 1, VAEs with a flow prior do not significantly improve the generation quality (see section 4.2 and Table 1 ). The reason might be that although p(z) is matched with q(z) due to the flow transformation, the decoder is not good enough to reconstruct sharp images (i.e, criterion (a) is not ensured). In contrast, we find that increasing β in the objective produces samples with significantly higher quality (see Figure 3 ). Intuitively, larger weight on the reconstruction loss forces the decoder to produce sharper reconstructed images, while the normalizing flow prior is flexible enough to match the latent distribution.

To the best of our knowledge, we are the first to observe such a relation between the weight of the reconstruction loss and the generation quality of VAEs with flow prior. As β increases, two things occur as demonstrated empirically in Section 4.2.1. First the estimated variances from the encoder decrease, and second the generation quality consistently improves. In the limit, as the posterior variance goes to zero, we obtain a deterministic encoder, leading to a deterministic auto-encoder and a normalizing flow that is used to match the distribution of the latent variables obtained from the data. This is described in detail in the next section.

3.3. GENERATIVE LATENT FLOW (GLF):

In an auto-encoder, z = E η (x) is deterministic so that q(z|x) in equation 3 becomes a delta distribution and the entropy term in equation 3 can be removed. The overall training loss is then

EQUATION

where

L recon x i , G φ (E η (x i )) corresponds to -log p φ (x i |z i ) and L NLL f θ (E η (x i )) to -log p θ (z)

in the negative of equation 3.

As noted in section 3.2, larger β's yield better results, in which case the parameters of the auto-encoder are affected almost exclusively by L recon , while L NLL only affects the parameters θ of the normalizing flow. Therefore, optimizing (4) with extremely large β is approximately equivalent to optimizing

EQUATION

where sg [•] is the stop gradient operation. The weight parameter β is no longer needed because the two loss terms affect independent sets of parameters. We name the model trained by equation 5 as Generative Latent Flow (GLF), to highlight that our model applies normalizing flows on latent variables. See Figure 1a for an illustration of the GLF model. We call the model trained by equation 4, without stopped gradient, regularized GLF, since the flow acts as a regularizer on the encoder.

Note that when stopping the gradients, GLF can also be trained in two stages, namely an auto-encoder is trained first, and then the flow is trained to map the distribution of estimated latent variables to the standard Gaussian. Empirically, we find that the two-stage training strategy leads to similar performance, so we only focus on one-stage training as it follows our derivation more naturally.

3.3.1. NECESSITY OF STOPPING THE GRADIENTS:

The stop gradient operation is necessary when using deterministic auto-encoders. In VAEs with flow prior, the entropy term, which encourages the posterior to have large variance, prevents the degeneracy of the z's. However, when using a deterministic encoder, if we let gradients of L NLL back propagate into the latent variables, training can lead to degenerate z's produced by the encoder E η . This is because f θ has to transform the z's to unit Gaussian noise, so the smaller the scale of the z's, the larger the magnitude of the log-determinant of the Jacobian. Since there is no constraint on the scale of the output of E η , the Jacobian term can dominate the entire objective. While the latent variables cannot become exactly 0 because of the presence of the reconstruction loss, the extremely small scale of z may cause numerical issues that lead to severe fluctuations. In summary, we stop the gradient of L NLL at the latent variables, preventing it from modifying the values of z and affecting the parameters of the encoder. We demonstrate the issues with regularized GLF in Section 4.2.1.

4. EXPERIMENTS:

To demonstrate the performance of our method, we present both quantitative and qualitative evaluations on four commonly used datasets for generative models: MNIST (Lecun, 2010) , Fashion MNIST (Xiao et al., 2017) , CIFAR-10 ( Krizhevsky et al., 2009) and CelebA (Liu et al., 2015) .

Throughout the experiments, we use 20-dimensional latent variables for MNIST and Fashion MNIST, and 64-dimensional latent variables for CIFAR-10 and CelebA. (Lucic et al., 2018) adopted a common network architecture based on InfoGAN (Chen et al., 2016a) to evaluate GANs. In order to make fair comparisons without designing arbitrarily large networks to achieve better performance, we use the generator architecture of InfoGAN as our decoder's architecture, and the encoder is set symmetric to the decoder. For details of the AE network structures, see Appendix A. For the flow applied on the latent variables, we use 4 affine coupling blocks as shown in Figure 1b , where each block contains 3 fully connected layers each with k hidden units.

For MNIST and Fashion MNIST, k = 64, while for CIFAR-10 and CelebA, k = 256. Note that the flow only adds a small parameter overhead on the auto-encoder (less than 3%).

4.1. METRICS:

Estimated test data log likelihood is a popular metric to evaluate models based on VAEs. It is not trivial to estimate the log likelihood obtained from GLF, as it uses an deterministic auto-encoder. More importantly, as shown in (Grover et al., 2018; Theis et al., 2015) , likelihood is not well correlated with sample quality. We use the Fréchet Inception Distance (FID) (Heusel et al., 2017) as a metric for image generation quality. FID is computed by first extracting features of a set of real images x and a set of generated images g from an intermediate layer of the Inception network (Szegedy et al., 2015) . Each set of features is fitted with a Gaussian distribution, yielding means µ x , µ g and co-variances matrices Σ x , Σ g . The FID score is defined to be the Fréchet distance between these two Gaussians:

FID(x, g) = µ x -µ g 2 2 + Tr Σ x + Σ g -2 (Σ x Σ g ) 1 2

It is claimed that the FID score is sensitive to mode collapse and correlates well with human perception of generator quality (Lucic et al., 2018) . Recently, (Sajjadi et al., 2018) proposed using Precision and Recall for Distributions (PRD) which can assess both the quality and diversity of generated samples.

We also include PRD in our studies. 

4.2. RESULTS:

Table 1 summarizes the main results of this work. We compare the FID scores obtained by GLF with the scores of the VAE baseline and several existing AE based models that are claimed to produce high quality samples. Instead of directly citing their reported results, we re-ran the experiments because we want to evaluate them under standardized settings so that all models adopt the same AE architectures, latent dimensions and image pre-processing. We report the results of VAE+flow prior/posterior with Note that the authors of WAE propose two variants, namely WAE-GAN and WAE-MMD. We only report the results of WAE-GAN, as we found it consistently outperforms WAE-MMD. Note also that, GLANN (Hoshen et al., 2019) obtains impressive FID scores, but it uses perceptual loss (Johnson et al., 2016) as the reconstruction loss, while other models use MSE loss. The perceptual loss is obtained by feeding both training images and reconstructed images into a pre-trained network such as VGG (Simonyan & Zisserman, 2014) , and computing the L 1 distance between some of the intermediate layers' activation. We also train our method with perceptual loss and compare with GLANN in the last two rows of Table 1 .

As shown in Table 1 , our method obtains significantly lower FID scores than competing AE based models across all four datasets. In particular, GLF greatly outperforms VAE+flow prior with the default setting of β = 1. A more detailed analysis and comparison between the two methods will be done in Section 4.2.1. We also confirm that VAE+flow posterior cannot improve generation quality. Perhaps the competing model with the closest performances to ours is RAE+GMM, which shares some similarity with GLF in that both methods fit the density of the latent variables of an AE explicitly. To compare our method with GANs, we also include the results from (Lucic et al., 2018) in Appendix D. In (Lucic et al., 2018) , the authors conduct standardized and comprehensive evaluations of representative GAN models with large-scale hyper-parameter searches, and therefore, their results can serve as a strong baseline. The results indicate that our method's generation quality is competitive with that of carefully tuned GANs.

In Table 3 , Appendix C, we present the Precision and Recall scores of our method and several competing methods. As shown in the table, GLF obtains state-of-the-art Precision and Recall scores across all datasets, indicating that our method outperforms competing methods in terms of both sample quality and diversity.

Some qualitative results are shown in Figure 2 . Besides samples of the datasets used for quantitative evaluation, samples of CelebA-HQ (Karras et al., 2017) with the larger size of 256 × 256 are also included in Figure 2e to show our method's ability to scale up to images with higher resolution.

Qualitative results show that our model can generate sharp and diverse samples in each dataset. In Figure 2f , we show CelebA images generated by linearly interpolating two sampled random noise vectors. The smooth and natural transition shows that our model can generate samples that have not been seen during training. To provide further evidence that our model does not overfit or 'memorize' the training set, we show nearest neighbors in the training set for some generated samples in Appendix G. For more qualitative results, including samples from models trained with perceptual loss, see Appendix H. We observe that samples from models trained with perceptual loss have higher quality.

4.2.1. COMPARISONS: GLF VS. REGULARIZED GLF AND VAE+FLOW PRIOR.:

As discussed in section 1 and section 3.2, we underline the novel finding regarding the relation between the weight on the reconstruction loss and the sample quality of VAEs with flow prior. In this section, we present detailed experiments on this relation. We train VAEs+flow prior on CIFAR-10 for different choices of β, plus one with a learnable β (Dai & Wipf, 2019) . We record the progression of FID scores of these models in Figure 3a . In Figure 3b , we plot the entropy term, which is the last term in equation 3, the objective of VAE+flow prior. The entropy is expressed as -d j=1 log(σ j )/2, where σ j is the standard deviation of the approximate posterior on the j th latent variable. Higher entropy means that the latent variables have lower variances. In Figure 3c , we plot the NLL loss. We omit the results for β = 1 because the obtained FID scores are too high to fit the scale of the plot. Settings for the experiments in this subsection can be found in Appendix B.6. From Figure 3a , we clearly observe the trend that the generation quality measured by FID scores improves as β increases. We also observe that as β increases, the performance gap between VAE+flow prior and GLF closes, indicating that GLF captures the limiting behavior of VAE+flow prior. We also find that learnable β is not effective, probably due to the relatively small values of β at the early stages of training. When β is large, as indicated by Figure 3b , the posterior variances of VAEs become very small, so that effectively we are training an AE. For example, as shown in Figure 3b , when β = 400, the corresponding average posterior variance is around 10 -4 . This motivates us to use a deterministic auto-encoder in GLF, which as we have said above can be seen as the vanishing observational variance limit of VAE+flow prior. It is important to note that the relation between β and generation quality only exists for VAEs with a trainable prior (such as normalizing flow), as we verify empirically that increasing β on plain VAEs leads to worse FID scores.

As discussed in section 3.3.1, training regularized GLF is unstable because of the degeneracy of the latent variables driven by the NLL loss. We empirically study the effect of latent regularization as a function of β and present results in Figure 4 . For low values of β = 1 and 10, the NLL loss completely dominates the learning signal and the reconstruction loss quickly diverges, therefore we omit them in the plot. For larger values of β = 50, 100, 400 we observe that the NLL loss decreases to a negative value of very large magnitude, and although overall performance is reasonable, it oscillates quite strongly as training proceeds. In contrast, for GLF, where the flow does not modify z, the NLL loss does not degenerate, resulting in stable improvements of FID scores as training progress.

In contrast to regularized GLF, which uses a deterministic encoder, no degeneracy in the latent variables is observed for VAE+flow prior, thanks to the noise introduced in the stochastic encoder and the corresponding entropy term. Indeed, Figure 3c shows that the training of VAE+flow prior does not over-fit the NLL loss, as opposed to regularized GLF where severe over-fitting to NLL loss occurs as shown in Figure 4c . Comparing Figure 3a and 4a, we observe that unlike regularized GLF, VAE+flow prior does not suffer from divergence or fluctuations in FID scores, even with relatively small β. In summary, the results of FID scores show that regularized GLF is unstable, while as β increases, the performance of VAE+flow prior converges to that of GLF. Note that although GLF only slightly outperforms VAE+flow prior even when β is very large, it has the advantage that there is no need to tune β. 

4.3. TRAINING TIME:

Besides better performance, our method also has the advantage of faster convergence among competing methods such as GLANN and Two-stage VAE. In For all experiments in Section 4.2.1, we use batch size 256 and initial learning rate 10 -3 for both AE and flow. We train all models for 500 epochs with learning rates decaying by a factor of 2 every 150 epochs.

C PRECISION AND RECALL:

In this section, we report the precision and recall (PRD) evaluation of samples on each dataset in Table 3 . include WAE-GAN, Two-stage VAE, RAE+GMM and GLANN (with perceptual loss) for comparisons. As the case of FID scores, for GLANN, we directly cite their reported results, and we compute the results ofr other models. We report the PRD of models trained under the settings introduced in Appendix B.

The two numbers in each entry are F 8 , F 1 8 that capture recall and precision, respectively. See (Sajjadi et al., 2018) for more details. Higher numbers are better. 

D COMPARISON WITH GANS:

In Table 4 we combine our reported results of AE based models and the FID scores of GANs cited from (Lucic et al., 2018) . 

F TRAINING TIME COMPARISONS:

In Table 5 , we report the number of training epochs of our method, two-stage VAE and GLANN. In Table 6 , we report the clock training time per epoch of these methods. Note that for methods using perceptual loss, the per epoch training time is longer because VGG activations need to be computed. These two tables show that GLF needs much shorter training time than the two competing methods.

In GLF, training the flow does not add much computational time due to the low dimensionality. 

H MORE QUALITATIVE RESULTS:

In Figure 6 , we show more samples of each dataset generated by GLF, using either MSE or perceptual loss as reconstruction loss. In Figure 7 , we show samples of CelebA-HQ datasets from GLF trained with perceptual loss. In Figure 8 , we show examples of interpolations between two randomly sampled noises on CelebA from GLF trained with perceptual loss. 

:



Figures and Tables:

Figure 1: (a) Illustration of the GLF model. The red arrow contains a stop gradient operation (see section 3.3). (b) Structure of one flow block. The input is split into two parts y = (y 1 , y 2 ), go through two coupling layers C (see section 3.1). Finally a random permutation P is applied.

Figure 2: (a)-(e): Randomly generated samples from our method trained on different datasets. (f): Random noise interpolation on CelebA.

Figure 3: (a) Record of FID scores on CIFAR-10 for VAEs+flow prior with different values of β and GLF. (b) Record of entropy losses for corresponding models. (c) Record of NLL losses for corresponding models.

Figure 4: (a) Record of FID scores on CIFAR-10 for regularized GLF with different values of β and GLF. β = 1 and 10 are omitted because they lead to divergence in the reconstruction loss. (b) Record of reconstruction loss for the corresponding models. (c) Record of NLL loss for the corresponding models.

Figure 6: (a)-(d) Randomly generated samples from our method with MSE loss. (e)-(h) Randomly generated samples from our method with perceptual loss.

FID scores obtained from different models. For our reported results, we executed 10 independent trials and report the mean and standard deviation of the FID scores. Each trail is computing the FID between 10k generated images and 10k real images. For other methods, we largely follow their proposed experimental settings. Details of each experiment are presented in Appendix B.

Table content: <table><tr><td/><td>MNIST</td><td>Fashion</td><td>CIFAR-10</td><td>CelebA</td></tr><tr><td>VAE</td><td colspan="4">28.2 ± 0.3 57.5 ± 0.4 142.5 ± 0.6 71.0 ± 0.5</td></tr><tr><td>WAE-GAN</td><td colspan="3">12.4 ± 0.2 31.5 ± 0.4 93.1 ± 0.5</td><td>66.5 ± 0.7</td></tr><tr><td>Two-Stage VAE</td><td colspan="3">10.9 ± 0.7 26.1 ± 0.9 96.1 ± 0.9</td><td>65.2 ± 0.8</td></tr><tr><td>RAE + GMM</td><td colspan="3">10.8 ± 0.1 25.1 ± 0.2 91.6 ± 0.6</td><td>57.8 ± 0.4</td></tr><tr><td>VAE+flow prior</td><td colspan="4">28.3 ± 0.2 51.8 ± 0.3 110.4 ± 0.5 54.3 ± 0.3</td></tr><tr><td>VAE+flow posterior</td><td colspan="4">26.7 ± 0.3 55.1 ± 0.3 143.6 ± 0.8 67.9 ± 0.3</td></tr><tr><td>GLF (ours)</td><td>8.2 ± 0.1</td><td colspan="2">21.3 ± 0.2 88.3 ± 0.4</td><td>53.2 ± 0.2</td></tr><tr><td colspan="2">GLANN with perceptual loss 8.6 ± 0.1</td><td colspan="2">13.0 ± 0.1 46.5 ± 0.2</td><td>46.3 ± 0.1</td></tr><tr><td>GLF+perceptual loss (ours)</td><td>5.8 ± 0.1</td><td colspan="2">10.3 ± 0.1 44.6 ± 0.3</td><td>41.8 ± 0.2</td></tr><tr><td>β = 1.</td><td/><td/><td/></tr></table>

Appendix F, we compare the number of training epochs to obtain the FID scores in Table1. We also compare the per epoch training clock time in Table6, Appendix F. The combined results indicate that GLF requires much less training time while generating samples with higher quality.5 CONCLUSIONIn this paper, we introduce Generative Latent Flow, a novel generative model which uses an autoencoder to learn a latent space from training data and a normalizing flow to match the distribution of the latent variables with the prior. Under standardized evaluations, our model achieves state-of-the-art results in image generation quality and diversity among several recently proposed auto-encoder based models. While we are not claiming that our GLF model is superior to GANs, we do believe that it opens the door to realizing the potential of AE based models to produce high quality samples just as GANs do. Our proposed model is motivated by our novel finding on the relation between large reconstruction weight and generation quality of VAEs with normalizing flow prior. The finding itself is important, as it can potentially motivate future work to study the trade-off between reconstruction and density matching in the objective of VAEs with learnable priors.B.6 SETTINGS FOR EXPERIMENTSIN SECTION 4.2.1

Table content: <table/>

Evaluation of sample quality by precision/recall.

Table content: <table><tr><td/><td>MNIST</td><td>Fashion</td><td>CIFAR-10</td><td>CelebA</td></tr><tr><td>WAE-GAN</td><td colspan="4">(0.978, 0.956) (0.901, 0.837) (0.414, 0.723) (0.501, 0.512)</td></tr><tr><td>Two-stage VAE</td><td colspan="4">(0.982, 0.977) (0.937, 0.845) (0.382, 0.669) (0.452, 0.558)</td></tr><tr><td>RAE+GMM</td><td colspan="4">(0.988, 0.971) (0.922, 0.924) (0.370, 0.733) (0.333, 0.445)</td></tr><tr><td>GLF (ours)</td><td colspan="4">(0.982, 0.985) (0.932, 0.926) (0.485, 0.767) (0.542, 0.618)</td></tr><tr><td>GLANN+perceptual loss</td><td colspan="4">(0.971, 0.979) (0.985, 0.963) (0.860, 0.825) (0.574, 0.681)</td></tr><tr><td colspan="5">GLF+perceptual loss (ours) (0.990, 0.992) (0.987, 0.980) (0.765, 0.845) (0.760, 0.778)</td></tr></table>

FID score comparisons of GANs and various AE based modelsE ISSUES WITH LATENT REGULARIZATIONIn this appendix section, we present the plots of FID scores, reconstruction loss and NLL loss of regularized GLF. Related results are discussed in Section 4.2.1.

Table content: <table><tr><td/><td>MNIST</td><td>Fashion</td><td>CIFAR-10</td><td>CelebA</td></tr><tr><td>MM GAN</td><td>9.8 ± 0.9</td><td colspan="2">29.6 ± 1.6 72.7 ± 3.6</td><td>65.6 ± 4.2</td></tr><tr><td>NS GAN</td><td>6.8 ± 0.5</td><td colspan="2">26.5 ± 1.6 58.5 ± 1.9</td><td>55.0 ± 3.3</td></tr><tr><td>LSGAN</td><td>7.8 ± 0.6</td><td colspan="3">30.7 ± 2.2 87.1 ± 47.5 53.9 ± 2.8</td></tr><tr><td>WGAN</td><td>6.7 ± 0.4</td><td colspan="2">21.5 ± 1.6 55.2 ± 2.3</td><td>41.3 ± 2.0</td></tr><tr><td>WGAN GP</td><td colspan="3">20.3 ± 5.0 24.5 ± 2.1 55.8 ± 0.9</td><td>30.3 ± 1.0</td></tr><tr><td>DRAGAN</td><td>7.6 ± 0.4</td><td colspan="2">27.7 ± 1.2 69.8 ± 2.0</td><td>42.3 ± 3.0</td></tr><tr><td>BEGAN</td><td colspan="3">13.1 ± 1.0 22.9 ± 0.9 71.4 ± 1.6</td><td>38.9 ± 0.9</td></tr><tr><td>VAE</td><td colspan="4">28.2 ± 0.3 57.5 ± 0.4 142.5 ± 0.6 71.0 ± 0.5</td></tr><tr><td>WAE-GAN</td><td colspan="3">12.4 ± 0.2 31.5 ± 0.4 93.1 ± 0.5</td><td>66.5 ± 0.7</td></tr><tr><td>Two-Stage VAE</td><td colspan="3">10.9 ± 0.7 26.1 ± 0.9 96.1 ± 0.9</td><td>65.2 ± 0.8</td></tr><tr><td>RAE + GMM</td><td colspan="3">10.8 ± 0.1 25.1 ± 0.2 91.6 ± 0.6</td><td>57.8 ± 0.4</td></tr><tr><td colspan="2">GLANN (with perceptual loss) 8.6 ± 0.1</td><td colspan="2">13.0 ± 0.1 46.5 ± 0.2</td><td>46.3 ± 0.1</td></tr><tr><td>VAE+flow prior</td><td colspan="4">28.3 ± 0.2 51.8 ± 0.3 110.4 ± 0.5 54.3 ± 0.3</td></tr><tr><td>VAE+flow posterior</td><td colspan="4">26.7 ± 0.3 55.1 ± 0.3 143.6 ± 0.8 67.9 ± 0.3</td></tr><tr><td>GLF (ours)</td><td>8.2 ± 0.1</td><td colspan="2">21.3 ± 0.2 88.3 ± 0.4</td><td>53.2 ± 0.2</td></tr><tr><td>GLF+perceptual loss (ours)</td><td>5.8 ± 0.1</td><td colspan="2">10.3 ± 0.1 44.6 ± 0.3</td><td>41.8 ± 0.2</td></tr></table>

Number of training epochs for Two-stage VAE, GLANN, and GLF NEAREST NEIGHBORS OF GENERATED SAMPLES IN THE TRAINING SET Quantitative measurements of sample quality, such as FID score and Precision/Recall can be minimized by letting the generative model memorize the training set. The smooth transition of noise interpolation shown in Figure2provides evidence that our model can generalize, i.e., it generate samples that have not be seen during training. Here we provide additional evidence showing that our generative model generalizes well.We randomly generate some samples from the models trained on MNIST and CelebA datasets. Then we present the 5 nearest neighbors of each generated samples in the training set. The nearest neighbor is defined in terms of L 2 distance. Results are shown in Figure5. By inspecting the figure, we find that we can easily differentiate each generated sample from the closest training data. This indicates that our model generalizes well.

Table content: <table><tr><td/><td colspan="3">MNIST/Fashion CIFAR-10 CelebA</td></tr><tr><td colspan="2">Two-stage VAE First/Second 400/800</td><td colspan="2">1000/2000 120/300</td></tr><tr><td>GLANN First/Second</td><td>500/50</td><td>500/50</td><td>500/50</td></tr><tr><td>GLF</td><td>100</td><td>200</td><td>40</td></tr><tr><td colspan="3">Table 6: Per-epoch training time in seconds</td><td/></tr><tr><td/><td colspan="3">MNIST/Fashion CIFAR-10 CelebA</td></tr><tr><td>2-stage VAE 1st/2nd</td><td>5/2</td><td>6/2</td><td>60/28</td></tr><tr><td>GLF</td><td>10</td><td>13</td><td>108</td></tr><tr><td colspan="2">GLANN with perceptual loss 14</td><td>16</td><td>292</td></tr><tr><td>GLF with perceptual loss</td><td>16</td><td>19</td><td>343</td></tr></table>

Peer review (iclr2020):
The paper proposes a deterministic Auto-Encoder with a trained marginal distribution over latent variables, p(z), to be able to sample from the model. For this purpose, the authors propose to use a flow-based model for p(z), and regularize the AE objective (i.e., MSE) with a cross-entropy between q(z) = 1/N sum_n E(x_n) and p(z). In general, I find the idea quite interesting. The construction of the objective and motivation behind is rather clear. The experiments are rather convincing (however, the FID metric is subjective, but it*s impossible to calculate the log-likelihood scores). The main disadvantage of the paper is its language. There are many typos and difficult to follow sentences. But besides that, the main ideas are well explained. Please find more specific remarks below. Remarks - The language in the paper is a bit off. There are sentences that sound very peculiar, e.g., *Deep generative models can be roughly classified into explicit and implicit models. The former class assumes explicit parametric specification of the distribution, whereas the latter does not.* The introduction should be re-written. Similarly, Section 2 is hard to follow. There are places where a word or a punctuation mark is missing, for instance: * The first line misses a word: *(...) on deep learning.* * First paragraph, Section 2: *This distribution is unknown and possibly The model also has a predefined prior distribution p(z) on Z.* * Section 3.2, below Eq. 2: *(...) assumptio (...)*. * Section 3.4, last paragraph: *Our work is differs in two ways (...)*. - The authors should refer to the following very relevant paper: * Xu, H., Chen, W., Lai, J., Li, Z., Zhao, Y., & Pei, D. (2019). On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder. arXiv preprint arXiv:1905.13452. - It is unclear how the authors dequantize image data that are typically represent as integers from 0 to 255 (or 0 and 1 in the binary case). The authors mention in Section 3.2 that they use the MSE loss for mathcal{L}_{recon}. This corresponds to taking a Gaussian decoder in the VAE framework. However, the manner how the images are dequantized is extremely important to properly evaluate all results. For instance, if the authors add a uniform noise, then it highly influences the final quality of a model. - I do not fully follow why the authors call the objective part for learning the marginal distribution over latents *the NLL loss*. It is rather the cross entropy between q(z) = 1/N sum_n E(x_n) and p(z). This follows from the notation and the description that we stop the gradient for E(.). I find it confusing. - Figure 4 is extremely important for understanding why we should stop gradient. However, jumping between Section 4.2 and E is extremely annoying. The authors can use up to 10 pages, so adding a half of a page would not make a difference, but it would help a reader a lot. - The main difference between the objective in Eq. 2 and the ELBO lies in considering a deterministic encoder and entails skipping the entropy term for the variational posterior. Could the authors comment why this is so important to choose a deterministic encoder? I can easily imagine taking q(z|x) = Normal(z | mu(z), a1), i.e., fixing the variance to some value a (e.g., a=1), that would result in Entropy[q] = const. Hence, we can skip entropy from the objective, but still we use a stochastic encoder. ===== AFTER REBUTTAL ===== I would like to thank the authors for their response and new version of the paper. I must admit that I had a very hard nut to crack. From one side, I really appreciate all the effort the authors put to improve the paper. However, on the other hand, I tend to agree with the reviewer #3 that the paper is interesting from the engineering perspective and it lacks novelty. Eventually, I decided to sustain my score. There are two reasons for that. First, more analysis of the model would be helpful. For instance, analyzing the latent space would be interesting (a suggestion from the reviewer #3). Second, considering a pure AE with a trainable flow-based prior is interesting, but it is also very limiting in the sense of obtaining nicely looking pictures instead of being able to provide (approximate) probability of an image. In my opinion, from the decision making perspective, having a probability of an object is much more important than being able to generate a crisp image.