Generative Latent Flow
Authors: Zhisheng Xiao
Qing Yan
Yali Amit
Venue:  (2020)
Keywords: Generative Model
Auto-encoder
Normalizing Flow

Abstract: In this work, we propose the Generative Latent Flow (GLF), an algorithm for generative modeling of the data distribution. GLF uses an Auto-encoder (AE) to learn latent representations of the data, and a normalizing flow to map the distribution of the latent variables to that of simple i.i.d noise. In contrast to some other Auto-encoder based generative models, which use various regularizers that encourage the encoded latent distribution to match the prior distribution, our model explicitly constructs a mapping between these two distributions, leading to better density matching while avoiding over regularizing the latent variables. We compare our model with several related techniques, and show that it has many relative advantages including fast convergence, single stage training and minimal reconstruction trade-off. We also study the relationship between our model and its stochastic counterpart, and show that our model can be viewed as a vanishing noise limit of VAEs with flow prior.  Quantitatively, under standardized evaluations, our method achieves state-of-the-art sample quality and diversity among AE based models on commonly used datasets, and is competitive with GANs' benchmarks. 

Under review as a conference paper at ICLR 2020
GENERATIVE LATENT FLOw
Anonymous authors Paper under double-blind review
ABSTRACT
In this work we propose Generative Latent Flow (GLF), an algorithm for gener- ative modeling of data distributions. GLF uses an auto-encoder to learn latent representations of the data, and normalizing flow to map the distribution of the latent variables to that of a standard Gaussian. GLF can be seen as a variational auto-encoder, with normalizing flow prior, and vanishing limit of the pixel-wise variance of the data. We carefully study this relationship and the pros and cons using an auto-encoder vs. a variational auto-encoder. In contrast to a number of other auto-encoder based generative models, which use various regularizers to encourage the encoded latent distribution to match the prior distribution, our model explicitly constructs mapping between these two distributions, leading better density matching while avoiding over regularizing the latent variables. We compare our model with several related techniques, and show that under standard quantitative evaluations, our method achieves state-of-the-ar the sample quality and diversity among AE based models on commonly used datasets and is competitive with GANs' benchmarks.
INTRODUCTION
Generative models have attracted much attention in the literature on deep learning. These models are used to formulate the distribution of complex data as function of random noise passed through network, that rendering samples from the distribution is particularly easy. The most dominant generative models are Generative Adversarial Networks (GANs) [Goodfellow et al., 2014). as they have exhibited impressive performance in generating high quality images (Radford et al.] 2015)] Brock et al., 2018)) and in other vision tasks (Zhu et al., 2017 Ledig et al.) 2017). Despite their success, training GANs can be challenging, partly because they are trained by solving saddle point optimization problem formulated as an adversarial game. It is well known that training GANs is unstable and sensitive to hyper-parameter settings (Salimans et al. 2016 Arora et al. 2017), and sometimes training leads to mode collapse (Goodfellow (016)) Although there have been multiple efforts to overcome the difficulties in training GANs (Arjovsky et al. 2017: Metz Luke & Sohl Dickstein Srivastaaa et al.) 2017 Miyato et al.) 2018), researchers also actively studying non-adversarial methods that are known to be less affected by these issues.
Srivastava
Some models explicitly define p(x:), the distribution of the data, and training is guided by maximizing the data likelihood. One approach to express the data distribution in an auto-regressive pattern express it as an invertible transformation of a simple distribution using the change of variable formula, where the invertible transformation
(Papamakarios et al. 2017 Oord al. (0016); another
is defined using normalizing flow network (Dinh al. 2014 2016 Kingma & Dhariwal, 2018).
While being mathematically clear and well defined, normalizing flows keep the dimensionality of the original data in order to maintain bijectivity. Consequently, they cannot provide -i-deemnnional representations of the data and training is computationally expensive. Considering the prohibitively long training time and advanced hardware requirements in training large scale flow models such as (Kingma & Dhariwal 2018), we believe that it is worth exploring the application of flows in the low dimensional representation spaces rather than for the original data.
Another class of generative models employs encoder-decoder structure and low dimensional latent variables to represent and generate the data. An encoder is used to produce estimates of the latent variables corresponding
particular data point, and samples from predefined prior distribution on the latent space are passed through a decoder to produce new samples from the data distribution. We call these auto-encoder (AE) based models, of which variational auto-encoders (VAEs) are perhaps

---

Under review as a conference paper at ICLR 2020
the most influential (Kingma & Welling 2013: Rezende et al. 2014). VAEs use the encoder to produce approximations to the posterior distribution of the latent variable given the data, and the training objective is to maximize variational lower bound of the data log likelihood. VAEs are easy to train, but their generation quality still lies far below that of GANs, as they tend to generate blurry images (Dosovitskiy & Brox 2016).
Whereas the original VAE uses a standard Gaussian prior, it can be extended by introducing a learnable parameterized prior distribution. There have been number of studies in this direction (see section 2). some of which use a normalizing flow parameterization, where the prior is modeled as trainable continuous bijective transformation of the standard Gaussian. We carefully study this method, and make the surprising novel observation that in order to produce high quality samples, is necessary to significantly increase the weight on the reconstruction loss. This corresponds to decreasing the variance of the observational noise of the generative model at each pixel, where we are assuming the data distribution is factorial Gaussian conditioned on the output of the decoder, which yields the MSE as the reconstruction loss. It is important to note that increasing this weight alone without access to a trainable prior does not consistently improve generation quality.
We show that as this weight increases, we approach vanishing noise limit that corresponds to deterministic auto-eneoder. This leads to a new algorithm we call Generative Latent Flow (GLF), which combines a deterministic auto-encoder that learns a mapping to and from latent space, and a normalizing flow that matches the standard Gaussian to the distribution of latent variables of the training data produced by the encoder.
Our contributions are summarized as follows: i) we carefully study the effects of equipping VAEs with a normalizing flow prior on image generation quality the weight of the reconstruction loss is increased. ii) Based on this finding, introduce Generative Latent Flow, which uses auto-encoders instead of VAEs. iii) Through standard evaluations, we show that our proposed model achieves state-of-thee -hrttrrrrrtttt art sample quality among competing AE based models, and has the additional advantage of faster convergence.
2 RELATED WORK
In general, in order for an AE based model with encoder-decoder structure to generate samples resembling the training data distribution, two criteria need to be ensured: (a) the decoder is able
to produce good reconstruction of a training image given its encoded latent variable 2; and (b)
the empirical latent distribution g(=) of z's returned by the encoder is close to the prior p(2). In
VAEs, the empirical latent distribution is often called aggregated or marginal posterior: q(z)
Egepdses [](2[z)]. While (a) is mainly driven by the reconstruction loss, satisfying criterion (b) is
more complicated. Intuitively, criterion (b) can possibly be achieved by designing mechanisms that either modify the empirical latent distribution (2), or conversely modify the prior p(z). There is plenty of previous work in both directions.
Modifying the empirical latent distribution (z): In the classic VAE model, DKL (</z>)[p(()) in the BL.BO loss can be decomposed as DKL. (q(2))lp(( plus mutual information term as shown in (Hoffman & Johnson| 2016 Therefore, VAES modify q(2) indirectly through regularizing
VAE's loss (Chen et al., 2018 Kim & Mnih, 2018), which are designed for the task of unsupervised disentanglement, put a stronger penalty specifically on the mismatch between q(2) and p(z). There are also attempts incorporate normalizing flows into the encoder to provide more flexible approximate posteriors (Rezende & Mohamed 2013 Kingma al.) 2016 Berg et al. 2018). However, empirical evaluation shows that VAËS with flow posteriors do not reduce the mismatch between g(=) and p(=) (Rosca et al. 20 8). Furthermore, as of yet, all these modifications to VAEs have not been shown to improve generation quality. Adversarial auto-encoders (AAEs) (Makhzani et al. 2015) and Wasserstein auto-encoders (WAEs) (Tolstikhin et al., 2017)) use an adversarial regularizer or MMD regularizer (Gretton et al..
the posterior distribution ((]Ø). Several modifications
2012) to force (2)) be close to p(z). WAEs are shown to improve generation quality, they generate sharper images than VAEs do.
Modifying the prior distribution p p(z): : An alternative to modifying the approximate posterior is using a trainable prior. (Tomezak & Welling 2017: Klushyn et al.) 2019 [auer & Mnih, 2018 propose different ways approximate q(z) using a sampled mixture of posteriors during training,
2

---

Under review as a conference paper at ICLR 2020
and then the approximated ((2) as the prior in the VAE. This is natural way to let the prior match g()), however, these methods have not been shown to improve generation quality. Two-stage VAE (Dai & Wipf 2019) introduces another VAE on the latent space defined by the first VAE to learn the distribution of its latent variables. VQ-VAE (Oord et al., 2017) first trains an auto-encoder with discrete latent variables, and then fits auto-regressive prior the latent space. GLANN (Hoshen et al.) 2019) learns a latent representation by GLO (Bojanowski et al., 2017) and matches the densities of the latent variables with an implicit maximum likelihood estimator (Li & Malik 2018). RAE+GMM (Ghosh et al. (2119) trains regularized auto-encoder (Alain & Bengio 2014) and fits mixture Gaussian distribution on the latent space. Note that all these methods involve two stage-training, which means that the prior distribution is fitted after training the variational or deterministic auto-encoder. They have been shown to improve VAEs with normalizing flow as learnable prior (Chen et al. 201 Huang et al. (017)) also fall into this category. Since these are the main focus of this paper. we discuss them in detail Scction 3.2
quality the generated images.
note that modifications VAEs with a normalizing flow posterior have been extensively studied. In contrast, VAEs with flow prior have attracted much less attention. (Huang al., 2017) briefly discusses this model to solve the distribution mismatch in the latent space, and recently (Xu et al. 2019) shows the advantages of learning flow prior learning flow posterior. However, these papers only focus on improvements of the data likelihood. Here we study the model from the perspective of the effects of the normalizing flow prior on sample generation quality, leading some important and novel observations.
Roccnn
MI
31
Encodeer
Decoder
Likelihood
loss
(a)
(b)
Figure (a) Illustration of the GLF model. The red arrow contains stop gradient operation (see
section 3.3). (b) Structure of one flow block. The input is split into two parts y = (31, U2), go through
two coupling layers C (see section 3.1b. Finally random permutation applied.
3 COMBINING NORMALIZING FLOW WITH AE BASED MODELS
In this section, we discuss the combination of normalizing flow priors with AE based models in detail. We first review normalizing flows in section 3.1 then in section 3.2 we introduce VAEs with normalizing flow prior and present some novel observations with respect to this model. Finally in section 3.3 we propose Generative Latent Flow (GLF) to further simplify the model and improve performance.
3.1 REVIEW: NORMALIZING FLOWS
Normalizing flows are creffully-dsiinne invertible networks that map the training data to a simple distribution. Let E Z be observation from unknown target distribution the unit Gaussian prior distribution on E. Given a bijection fo - E, we define probability model (=) with parameters on Z. The negative log likelihood (NLL.) of is computed by the change of variable formula:
p(2) and p. be
(x))(fo(2)) & -log(po())) -(1og-0111100 (log)c(fo(=)) log det (8fo(2)
)) Oz
3

---

Under review as a conference paper at ICLR 2020
where (fe(s) is the Jacobian matrix of fa. In order to learn the flow fe. the NLL objective of is (3= minimized, which is equivalent to maximizing the likelihood of z. Since the mapping a bijection, sampling from the trained model po(=) is trivial: simply sample e ~ Pr and compute = So (c).
The key to designing a normalizing flow model is defining the transformation fo so that the inverse transformation and the determinant of the Jacobian matrix can be efficiently computed. Based on Dinh et al. 2016), we adopt the following layers to form the flows used in our model.
Affine coupling layer: Given D dimensional input data z and d < D, we partition the input into two vectors 21 = 21:d and 22 = 1:41: The output of one affine coupling layer is given by 9/1 21. 8/2 22 @ exp(s(21). + t(21) where and are functions from F + DD and @ is the element-wise product. The inverse of the transformation is explicitly given by 21 = 91. 22 (y/2 t (y11) ) O exp((s(y1)). The determinant of the Jacobian matrix of this transformation 2z - (exp(s(z))i)) Since computing both the inverse and the Jacobian of an affine coupling layer does not require computing the inverse and Jacobian and t, both functions can be arbitrarily complex.
is del By
Combining coupling layers with random permutation: Affine coupling layers leave some compo- nents of the input data unchanged. In order to transform all the components, two coupling layers are combined in an alternating pattern to form a coupling block, so that the unchanged components in the first layer be transformed in the second layer. In particular, we add fixed random permutation of the coordinates of the input data at the end of each coupling block. See Figure Tb for an illustration of a coupling block used in our model.
3.2 VAES WITH NORMALIZING FLOW PRIOR
We begin by introducing the training loss of the model. Consider the ELBO loss of standard VAEs with Gaussian prior and posterior (7, o denote the parameters of encoder and decoder, respectively):
ELBO Φ)
[ log (x(z)) log p(z) log o. (z/x)]
The first term is related to the reconstruction loss, while the last two terms can be combined as (g(]) a))) [=)()())))) 3 > O is a hyper-parameter that controls the relative weight of the reconstruction loss and the KL divergence loss. In the standard formulation of VAEs, Ps assumes an independent Gaussian distribution with variance 1 at each pixel. The parameter 33 allows us to adjust this variance as 1/3.
If we introduce a normalizing flow So for the prior distribution, then the prior po becomes po(z)
Pe (z) det [ffa(z) where P. is the standard Gaussian density. Substituting this prior into equation we obtain (L db B) for VAEs with flow prior:
tx
log P (xlz) + Pc (fo(z)) + log det (ff((z) | log 9, (4/xx]
E Bo (xix)
(3)
Oz
The second and third terms together the likelhhodd of under the prior distribution modeled by the flow (corresponding to the negative of equation []]. The last term corresponds to the entropy of the posterior distribution returned by the encoder. Both the VAE and the normalizing flow are trained by minimizing - ELBO(1) (7, é, 0).
Previous work on VAES with a flow prior did not consider tuning 3 (which means the reconstruction loss and the KL loss are weighted equally) as they focused on comparing the obtained log likelihoods VAEs with a flow prior do not significantly improve the generation quality (see section 4.2 and Table The reason might be that although p(z) is matched with (2)) due to the flow transformation, the decoder is not good enough to reconstruct sharp images (i.e, criterion (a) is not ensured). In contrast, find that increasing B in the objective produces samples with significantly higher quality (see Figure 3). Intuitively, larger weight the reconstruction loss forces the decoder to produce sharper reconstructed images, while the normalizing flow prior is flexible enough to match the latent distribution.
with those from plain VAEs. However, we observe that when /
To the best of our knowledge, we are the first observe such relation between the weight of the reconstruction loss and the generation quality of VAEs with flow prior. As 3 increases, two things occur as demonstrated empirically in Section 4.2.1 First the estimated variances from the encoder
4

---

Under review as a conference paper at ICLR 2020
decrease, and second the generation quality consistently improves. In the limit, the posterior variance goes to zero, we obtain a deterministic encoder, leading to deterministic auto-encoder and normalizing flow that is used to match the distribution of the latent variables obtained from the data. This is described in detail in the next section.
3.3 GENERATIVE LATENT Fr.ow (GLF)
In an auto-encoder, 2 = (x) is deterministic so that ((2[z) in equation 3 becomes a delta distribution and the entropy term in equation 3 can be removed. The overall training loss is then
6 [n.
Σ (Breeom (x, (x(E, + (fo
(4)
where Frecon (x, Go (x, in the negative of equation 3
corresponds to - -o 24) and LNLL (fo(E, to log Pa(2)
As noted in section 3.2 larger 3's yield better results, in which case the parameters of the auto-encoder
are affected almost exclusively by frecon- while LNLL only affects the parameters with extremely large 33 is approximately equivalent to optimizing
normalizing
flow. Therefore, optimizing 4
N Σ (Crecom (x,;
D(n, Φ.
Go Go (x:()) + CNLL (fo(sg|E,
(5)
where sgl:] is the stop gradient operation. The weight parameter is no longer needed because the two loss terms affect independent sets of parameters. We name the model trained by equation as Generative Latent Flow (GLF), to highlight that our model applies normalizing flows on latent variables. See Figure Ta for illustration of the GLF model. We call the model trained by equation without stopped gradient, regularized GLF, since the flow acts as a regularizer on the encoder.
Note that when stopping the gradients, GLF also be trained in two stages, namely auto-encoder is trained first, and then the flow is trained to map the distribution of estimated latent variables to the standard Gaussian. Empirically, we find that the two-stage training strategy leads to similar performance, so we only focus on one-stage training as it follows our derivation more naturally.
3.3.1 NECESSITY OF STOPPING THE GRADIENTS
The stop gradient operation is necessary when using deterministic auto-encoders. In VAEs with flow prior, the entropy term, which encourages the posterior to have large variance, prevents the degeneracy of the z's. However, when using a deterministic encoder, if we let gradients of LNU.. back propagate into the latent variables, training can lead to degenerate z's produced by the encoder Ery. This is because fe has to transform the z's to unit Gaussian noise, the smaller the scale of the z's, the larger the magnitude the log-determinant of the Jacobian. Since there is no constraint on the scale of the output of F,y. the Jacobian term can dominate the entire objective. While the latent variables cannot become exactly 0 because of the presence of the reconstruction loss, the extremely small scale of may cause numerical issues that lead to severe fluctuations. In summary, we stop the gradient of LNLI at the latent variables, preventing it from modifying the values of and affecting the parameters of the encoder. We demonstrate the issues with regularized GLF in Section 4.2.1
EXPERIMENTS
To demonstrate the performance of our method, we present both quantitative and qualitative eval- uations four commonly used datasets for generative models: MNIST (Lecun 2010). Fashion MNIST (Xiao et al.) 2017). CIFAR-10 (Krizhevsky et al. 2209) and CelebA (Liu et al.) 2015). Throughout the experiments, use 20-dimensional latent variables for MNIST and Fashion MNIST, and 64-dimenslonal latent variables for CIFAR-10 10 and CelebA.
(Lucic et al. 2018) adopted common network architecture based on InfoGAN (Chen et al. 2016a) to evaluate GANs. In order to make fair comparisons without designing arbitrarily large networks achieve better performance, use the generator architecture of InfoGAN decoder's
5

---

Under review as a conference paper at ICLR 2020
architecture, and the encoder is set symmetric to the decoder. For details of the AE network structures, see Appendix A For the flow applied on the latent variables, we use affine coupling blocks as shown in Figure lb where each block contains 3 fully connected layers each with & hidden units. For MNIST and Fashion MNIST, k = 64, while for CIFAR-10 and CelebA, k = 256. Note that the flow only adds small parameter overhead on the auto-encoder (less than 3%).
4.1 METRICS
Estimated test data log likelihood a popular metric to evaluate models based on VAEs. It is not trivial to estimate the log likelihood obtained from GLF, as it uses an deterministic auto-encoder. More importantly, as shown (Grover et al., 2018 Theis al., 2013), likelihood is not well correlated with sample quality. We use the Fréchet Inception Distance (FID) (Heusel et al.] 2017) image generation quality. FID is computed by first extracting features set real images æ and set of generated images g from intermediate layer of the Inception network (Szegedy et al. 2015). Each set of features is fitted with a Gaussian distribution, yielding means Pax Hy and no-vanianees matrices E.z, Dg. The FID score is defined to be the Fréchet distance between these two Gaussians:
metric for
FID g) = 11,z Mg 12
Eg + E - (
is claimed that the FID score is sensitive to mode collapse and correlates well with human perception of generator quality (Lucic al. 2018). Recently, (Sajadi al. 2018) proposed using Precision and Recall for Distributions (PRD) which can assess both the quality and diversity of generated samples. We also include PRD in our studies.
8
2
4
0
8
5
4
3
7
3
9 9 5 6
0
6
9
3
0 7
5
2
2
4
4
3
4
6 1
0
9
6
0
2
(a) MNIST
(b) Fashion MNIST
(c) CIFAR-10
(d) CelebA
(c) CelebA-HQ
(f) Noise interpolation
Figure 2: (a)-(e): Randomly generated samples from our method trained on different datasets. (f):
Random noise interpolation on CelebA.
4.2 RESULTS
Table summarizes the main results of this work. We compare the FID scores obtained by GLF with the scores of the VAE baseline and several existing AE based models that are claimed to produce high quality samples. Instead of directly citing their reported results, re-ran the experiments because want to evaluate them under standardized settings so that all models adopt the same AE architectures, latent dimensions and image pre-proces.ing. We report the results of VAE+flow prior/posterior with

---

Under review as a conference paper at ICLR 2020
Table 1: FID scores obtained from different models. For our reported results, we executed 10 independent trials and report the mean and standard deviation of the FID scores. Each trail is computing the FID between 10k generated images and 10k real images.
MNIST
Fashion
CIFAR-10
CelebA
VAE WAE-GAN
28.2 + 0.3
57.5 0.4
142.5
71.0 0.5
12.4 0.2
0.4
9.5
66.5
Two-Stage VAE
1.7
2.9
9.91
6.82
RAE+ + GMM
10
2.2
9.66
5..8
VAE+flow prior
2.2
0.3
1.0.4
5.3
VAE+flow posterior
2.3
55.1
0.8
67.9 0.3
GLF (ours)
+.2
21.3
88.3 0.4
53.2 + 0.2
GLANN with perceptual loss
8.6 0.1
13.0
46,5 0.2
46.3 + 0.1
Gl Eepercepptlll loss (ours)
5.8 + 0.1
10.3 0.1
44.6 + 0.3
41.8 + 0.2
3 = 1. For other methods, we largely follow their proposed experimental settings. Details of each experiment are presented in Appendix B
Note that the authors of WAE propose two variants, namely WAE-GAN and WAE-MMD. We only report the results of WAE-GAN, we found it consistently outperforms WAE-MMD. Note also that, GLANN (Hoshen et al. 2019) obtains impressive FID scores, but it uses perceptual loss (Johnson al. 2016) as the reconstruction loss, while other models use MSE loss. The perceptual loss obtained by feeding both training images and reconstructed images into pre-trained network such as VGG (Simonyan& & Zisserman 2014), and computing the I, distance between some of the intermediate layers' activation. We also train method with perceptual loss and compare with GLANN in the last two rows of [
As shown in Table D our method obtains significantly lower FID scores than competing AE based models across all four datasets. In particular, GLF greatly outperforms VAE+ prior with the default setting of 3 = 1. A more detailed analysis and comparison between the two methods will be done in Section 4.2.1 We also confirm that VAE+flow posterior cannot improve generation quality. Perhaps the competing model with the closest performances to ours is RAE+GMM, which shares some similarity with GLF in that both methods fit the density of the latent variables of an AE explicitly. To compare our method with GANs, we also include the results from (Lucic et al., 2018) in Appendix D In (Lucic et al. 2018), the authors conduct standardized and comprehensive evaluations of representative GAN models with large-scale their results can serve as strong baseline. The results indicate that our method's s generation quality competitive with that of carefully tuned GANs.
searches, and therefore,
In Table 3. Appendix C we present the Precision and Recall scores of our method and several competing methods. As shown in the table, GI.F obtains state-of-the-arr Precision and Recall scores across all datasets, indicating that our method outperforms competing methods in terms of both sample quality and diversity.
Some qualitative results are shown in Figure] 2 Besides samples of the datasets used for quantitative evaluation, samples CelebA-HQ (Karras et al. 2017) with the larger size of 256 x 256 are also included in Figure 2 to show method's 's ability to scale up to images with higher resolution. Qualitative results show that our model can generate sharp and diverse samples in each dataset. In Figure 2 show CelebA images generated by linearly interpolating two sampled random noise vectors. The smooth and natural transition shows that our model can generate samples that have not been seen during training. To provide further evidence that our model does not overfit or 'memorize' the training set, we show nearest neighbors in the training set for some generated samples in Appendix For more qualitative results, including samples from models trained with perceptual loss, see Appendix H
G
We observe that samples from models trained with perceptual loss have higher quality.
4.2.1 COMPARISONS: GLF VS. REGULARIZED GLF AND VAE+FLOW PRIOR.
As discussed in section T and section 3.2
we underline the novel finding regarding the relation between the weight the reconstruction loss and the sample quality of VAES with flow prior. In this
7

---

Under review as a conference paper at ICLR 2020
section, we present detailed experiments this relation. We train VAEs+flow prior CIFAR-10 1- for different choices of 8. plus one with a learnable 3 (Dai & Wipi 2019). We record the progression of FID scores of these models in Figure 3a In Figure 3b we plot the entropy term, which is the last
term in equation 3. the objective of VAE+flow prior. The entropy is expressed as -j //2
where oj is the standard deviation of the approximate posterior on the jth latent variable. Higher
entropy means that the latent variables have lower variances. In Figure 3c we plot the NLL loss. We omit the results for / because the obtained FID scores are too high to fit the scale of the plot. Settings for the experiments in this subsection can be found in Appendix B.6
10
8 =
50
8
100
3 = 400
3 Icarnablc non-reg GLF
160
entropy loss
FID score
100
150
NLL loss
150
95
90
60
90
100
36
100
200
300
400
600
100
2200
300
400
500
100
200
300
400
600
training epochs
training epochs
training epochs
(a)
(b)
(c)
Figure 3: (a) Record of FID scores on CIFAR-I0 for VAEs+flow prior with different values of B and GLF (b) Record of entropy losses for corresponding models. (c) Record of NLL losses for corresponding models.
From Figure 3a| clearly observe the trend that the generation quality measured by FID scores improves as 3 increases. We also observe that as 3 increases, the performance gap between VAE+flow prior and GLF closes, indicating that GLF captures the limiting behavior of VAE+flow prior. We also find that learnable 3 is not effective, probably due to the relatively small values of B at the early stages of training. When 3 is large, as indicated by Figure 3b the posterior variances of VAEs become very small, so that effectively we are training an AF. For example, shown in Figure 3b when 3 400, the corresponding average posterior variance is around 10-4, This motivates us to use deterministic auto-encoder in GLF, which as we have said above can be seen as the vanishing observational variance limit of VAE+flow prior. It is important to note that the relation between and generation quality only exists for VABs with trainable prior (such normalizing flow), verify empirically that increasing 33 on plain VAEs leads to worse FID scores.
As diseussed in section 3.3.1 training regularized GLF is unstable because of the degeneracy of the latent variables driven by the NLL loss. We empirically study the effect of latent regularization function of 33 and present results in Figure 4 For low values of B and 10, the NLL loss completely dominates the learning signal and the reconstruction loss quickly diverges, therefore we omit them in the plot. For larger values of 50, 100, 400 we observe that the NLL loss decreases to a negative value of very large magnitude, and although overall performance is reasonable, it oscillates quite strongly as training proceeds. In contrast, for GLF, where the flow does modify z, the NLL loss does not degenerate, resulting in stable improvements of FID scores as training progress.
In contrast to regularized GLF, which uses a deterministic encoder, no degeneracy the latent variables is observed for VAE+flow prior, thanks to the noise introduced in the stochastic encoder and the corresponding entropy term. Indeed, Figure 3c shows that the training of VAE+flow prior does not over-fit the NLL loss, as opposed to regularized GLF where severe over-fitting to NLL loss occurs as shown in Figure 4c Comparing Figure 3a and 4a we observe that unlike regularized GLF, VAE+flow prior does not suffer from divergence or fluctuations in FID scores, even with relatively small 3. In summary, the results FID scores show that regularized GLF is unstable, while as B increases, the performance of VAE+flow prior converges to that of GLE. Note that although GLF only slightly outperforms VAE+flow prior even when 3 is very large, it has the advantage that there is no need tune 3.
8

---

Under review as a conference paper at ICLR 2020
B
50
100
3
400
non-reg GLF
reconstruction loss
30
110
FID score
NLL loss
20
400
500
300
400 600
training epochs
training epochs (b)
training epochs (c)
Figure 4: (a) Record of FID scores on CIFAR-I0 for regularized GLF with different values of 3 and GLE B = 1 and 10 are omitted because they lead to divergence in the reconstruction loss. (b) Record of reconstruction loss for the corresponding models. (c) Record of NLL loss for the corresponding models.
4.3 TRAINING TIME
Besides better performance, our method also has the advantage of faster convergence among compet- ing methods such as GLANN and Two-stage VAE. In Table Appendix FF compare the number of training epochs obtain the FID scores in Table We also compare the per epoch training clock time in Table 6 Appendix F The combined results indicate that GLF requires much less training time while generating samples with higher quality.
CONCLUSION
In this paper, we introduce Generative Latent Flow, a novel generative model which uses an auto- encoder to learn latent space from training data and normalizing flow to match the distribution of the latent variables with the prior. Under standardized evaluations, our model achieves state-of-the-an results in image generation quality and diversity among several recently proposed auto-encoder based models. While we are not claiming that our GLF model is superior to GANs, we do believe that opens the door to realizing the potential of AE based models to produce high quality samples just as GANs do. Our proposed model is motivated by our novel finding on the relation between large reconstruction weight and generation quality VAEs with normalizing flow prior. The finding itself is important, as it can potentially motivate future work to study the trade-off between reconstruction and density matching in the objective of VAEs with learnable priors.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593, 2014.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXfv:1701.07875. 2017.
Wasserstein gan.
arXiv preprint
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning- coanninggVooumee 70, pp. 224-232. JMLR. org, 2017.
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. arXiv preprint arXiv: 1428, 2018.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. arXiv preprint arXiv:1803.05649 2018.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv preprint arXiv: 1707.07707.05776. 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXi: 1096, 2018
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016a.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXi::6611002311, 2016b.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.0522 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1rXiv:1605.08803 2016.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, pp. 658-666, 2016.
Partha Ghosh, Mchdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint 1903::/9033.24336 2019.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks.
arXiv preprint
arXiv:1701.00160, 2016.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(MMr):723-773, 2012.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
10

---

Under review as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, pp. 6626-6633 2017.
Matthew Hoffman and Matthew Johnson. Elbo surgery; yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, 2016.
Yedid Hoshen and Lior Wolf. Nam: Non-adversarial unsupervised domain mapping. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 436-451, 2018.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative latent nearest neighbors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5811-5819, 2019.
Chin-Wei Huang, Ahmed Touati, Laurent Dinh, Michal Drozdzal, Mohammad Havaei, Laurent Charlin, and Aaron Courville. Learnable explicit density for continuous latent space and variational inference. arXiv preprint arXiv:1710.02244 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 17Xiv:1710.1011 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances neural information processing systems, pp. 4743 4751, 2016.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchical priors in vaes. arXiv preprint arXiv:1905.04 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Y. Lecun. The mnist database of handwritten digits. http://yaam..ccc..co///ddmm/m/m 2010.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv preprint 0Xiv:180900908/ 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015,
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. In Advances in Neural Information Processing Systems 31, pp. 700-709, 2018.
11

---

Under review as a conference paper at ICLR 2020
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint 11%xiv/511100544/ 2015.
Pfau David Metz Luke, Poole Ben and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint 1rXiv:1611.02.633 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 2018.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:/601.067553 2016.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.066 2015.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505. 05770,, 2015,
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: urXiv:/401.4000 1401.4082, 2014.
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in variational inference. arXiv preprint arXiv: 2018
Mchdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems 31, pp. 2228-5233 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, pp.
2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 1-9, 2015.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. note on the evaluation of generative models. arXiv preprint arXhv:1511.01885 2015.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv: 1711.01558, 2017.
Jakub M Tomezak and Max Welling. Vae with vampprior. arXiv preprint arXiv:1005.07120, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Haowen Xu, Wenxiao Chen, Jinlin Lai, Zhihan Li, Youjian Zhao, and Dan Pei. On the necessity and effectiveness of learning the prior of variational auto-encoder. arXiv preprint 1905v:1905.1344 2019.
Zhu, T. Park, Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consister adversarial networks. In 2017 IEEE International Conference Computer Vision (ICCV), 2017.
12

---

Under review as a conference paper at ICLR 2020
Table 2: Network structure for auto-encoder based on InfoGAN
Encoder
Decoder
Input J 4 x 4 Conv64, ReLU 4 Conv 128, BN, ReLU
Input 2
FC nz - 1024, BN, ReLU
FC 1024 - 128 M x M, BN, ReLU
Flatten, FC 128 x M x M 1024, BN, ReLU FC 1024 - nz
4 x 4 Deconver, BN, ReLU
X 4 DeconV12s. Sigmoid
A NETWORK ARCHITECTURES
In this section we provide Table 2 that summarizes the auto-encoder network structure. The network structure is adopted from InfoGAN(Chen et al. (016a)), and the difference between the networks we used for each dataset is the size of the fully connected layers, which depends on the size of the image. All convolution and deconvolution layers have stride 2 and padding
to ensure the spatial dimension decreases/increases by factor of 2. M is simply the size of an input image divided by 4. Specifically, for MNIST and Fashion MNIST, = 7; for CIFAR-I0. 10, 8; for CelebA, M = 16. BN stands for batch normalization.
For VAEs, the final FC layer the encoder will have doubled output size to return both the mean and standard deviation of latent variables.
B EXPERIMENT SETTINGS
In this section, present the details of our experimental settings for results in Table I Since the settings for MNIST and Fashion MNIST are the same, we only mention MNIST for simplicity, For GLANN, we directly cite the results from (Hoshen et al. 2019). their experimental settings is very similar to ours.
We use the original images in the training sets for MNIST, Fashion MNIST and CIFAR-10. For CelebA, we follow the same pre-processing as in (Lucic et al. 2018): center crop to 160 x 160 and then resize 64 x 64. We normalize the pixel values [0, 1J. without adding noise to pixels (i.e, no de-quantization).
B. SETTINGS FOR TRAINING GLF
For all datasets (except CelebA-HQ), we use batch size 256 and Adam (Kingma & Ba 2014) optimizer with initial learning rate 10-3 for the parameters of both the AE and the flow. We add weight decay 2 10 to the optimizer for the flow. For MNIST, we train our model for 100 epochs, with learning rate decaying by factor of after 50 epochs. For CIFAR-10, 200 epochs, with the learning rate decaying by factor of every 50 epochs. For CelebA, we train our model for 40 epochs with no learning rate decay.
train model for
For GL. with perceptual loss, we compute the perceptual loss as suggested (Hoshen & Wolf 2018). See https: / thub. com/ cebook wwwrereeeeehaaaae (AAM/bl. ht/ll/oo/////////////// ht//master/code/ perceptual_lee los__osss_. py for their implementation. Other settings are the same.
For CelebA-HQ dataset, we adopt our AE network structure based on DCGAN (Radford al. 2015). Note that this is relatively simple network for high resolution imgaes. We use batch size 64, with initial learning rate 10-3 for both the AE and the flow. We train our model for 60 epochs, with learning rate decaying by factor of 2 after 40 epochs.
B.2 SETTINGS FOR TRAINING VAES AND VAE VARIANTS
We adopt common settings for our reported results of VAE, VAE+flow prior and VAE+flow posterior. We use 3 1 for all three VAE variants. We still use batch size 256, and Adam optimizer with initial learning rate 10 for both the VAE and the flow, if applicable. We find VAEs need longer time to converge, double the training epochs. We train MNIST for 200 epochs, with learning rate
13

---

Under review as a conference paper at ICLR 2020
decaying by factor of 2 after 100 epochs. We train CIFAR- 10 for 400 epochs, with the learning rate decaying by a factor of 2 every 100 epochs. We train CelebA for 80 epochs with learning rate decaying by factor of 2 after 40 epochs.
B.3 SETTINGS FOR TRAINING WAE-GAN
We follow the settings introduced in the original WAE paper Tolstikhin et al. 2017). The adversary in WAE-GAN has the following architecture:
2 Rd + FC512 + ReLU
+ FC512 ReLU
+ FC512 + ReLU
+ FC512 ReLU + FC1
where d is the dimension of the latent variables.
WAR has major hyper-parameters: À that controls the weight coefficient of the adversarial o
regularizer, and o² which is the variance of the prior. Batch size is 100 for all datasets. For MNIST, 10 and x² 1, and the model is trained for 100 epochs. The initial learning rate is 10-3 for the AE and 5 x 10 10-4 for the adversary. After 30 epochs both learning rates decreased both by factor of and after first 50 epochs further by factor of 5. For CIFAR, À 10 and o² and the model is trained for 200 epochs. The initial learning rates are the same as training MNIST, and the learning rate decays by a factor of 2 after first 60 epochs, and further by factor of 5 after 120 epochs. For CelebA, x = and o² = 2. The model is trained for 55 epochs. The initial learning rate is 3 x 10-1 for the AE and 10-3 for the adversary. Both learning rates decays by factor of 2 after epochs, further by factor of 5 after 50 first epochs.
B.4 SETTINGS FOR TRAINING TWO STAGE VAE
We adopt the settings in the original paper Dai & Wipi 2019). For all datasets, the batch size is set to be 64, and the initial learning rate for both the first and the second 10-4, For MNIST, the first VAE is trained for 400 epochs. with learning rate halved every 150 epochs; the second VAE is trained for 800 epochs with learning rate halved every 300 epochs. For CIFAR-10, 1000 and 2000 epochs trained for the two VAEs respectively, and the learning rates are halved every 300 and 600 epochs for the two stages. For CelebA, 120 and 300 epochs are trained for the two VAEs respectively, and the learning rates are halved every 48 and 120 epochs for the two stages.
Explaining the discrepancy between our reported results and the results in the original paper: The original Two stage VAE paper adopts similar settings with experiments, but observe large discrepancies on the results of CIFAR-10 and CelebA. After carefully reviewing their published codes, we find that there is issue in their FID score computation particularly for CIFAR-10 dataset. Specifically, the true images used for computing the FID on CIFAR- 10 is obtained from saving the original data file in jpg format and reading them back, and the saving will cause some errors in pixel values. After fixing this issue, we re-ran their published codes and obtained similar results as we reported. We also run through their original FID computation protocol using samples from our models, and we obtain scores around 65. For CelebA, one particular detail worth noting is that, (Dai & Wipf, 2019) applies 128 x 128 center-crop before re-sizing on CelebA while 160 x 160 center-crop is used in our evaluations. With smaller center-crops the human faces occupy larger portion of the image with less background, making the generative modeling easier.
B.5 SETTINGS FOR TRAINING RAE+GMM
The settings of batch size, learning rate scheduling and number of epochs for training RAE are the same those of GLF. The objective of the RAE is reconstruction loss plus penalty on the norm of the latent variable. Since the author does not report their choices for the penalty coefficient 7 search over 7 E 0.1, 0.5, 1, 2, and we find that 3 0.5 leads to the best overall performances. and therefore we let y = 0.5. After training the RAE, we a 10-component Gaussian mixture distribution the latent variables.
14

---

Under review as a conference paper at ICLR 2020
B.6 SETTINGS FOR EXPERIMENTS IN SECTION 4.2.
For all experiments in Section 4.2.1 and flow. We train all models for 500 epochs with learning rates decaying by a factor of 2 every 150 epochs.
batch size 256 and initial learning rate 10-3 for both AE
C PRECISION AND RECALL
In this section, we report the precision and recall (PRD) evaluation of samples on each dataset in T We include WAE-GAN, Two-stage VAE, RAE+GMM and GLANN (with perceptual loss) for comparisons. As the case of FID scores, for GLANN, we directly cite their reported results, and compute the results ofr other models. We report the PRD of models trained under the settings introduced in Appendix B
The numbers in each entry are Fs, Fg that capture recall and precision, respectively. See (Sajjadi al. 2018 for more details. Higher numbers are better.
Table Evaluation of sample quality by precision/recall.
MNIST
Fashion
CIFAR-I0
CelebA
WAE-GAN Two-stage VAE
(0.978, 0.956)
(0.901, 0.837)
(0.414, 0.723)
(0.501, 0,512)
(0.982, 0.977)
(0.937, 0.845)
(0.382, 0.669)
(0.452, 0.558)
RAE+GMM
(0.988, 0.971)
(0.922, 0.924)
(0.370, 0.733)
(0.333, 0,445)
GLF (ours)
(0.982, 0.985)
(0.932, 0.926)
(0.485, 0.767)
(0.542, 0.618)
percepuulal loss
(0.971, 0.979)
(0,985, 0,963)
(0.860, 0.825)
(0.574, 0.681)
GLF GLF+perceptual loss (ours)
(0.990, 0.992)
(0.987, 0.980)
(0.765, 0.845)
(0.760, 0.778)
D COMPARISON WITH GANs
In Table we combine our reported results AE based models and the FID scores of GANs cited from (Lucic et al. 2018).
Table 4: FID score comparisons of GANs and various AE based models
MNIST
Fashion
CIFAR-10
CelebA
MM GAN
9.8 0.9
29.6 1.6
72.7 3.6
65.6 4.2
NS GAN
0.5
58.5 1.6
1.9
55.0 + 3.3
LSGAN
0.6
87.1 3.2
47.5
53.9 2.8
WGAN
6.7
1.6
2.3
4.0
WGAN GP
5.0
2..
0.9
30.3
DRAGAN
0.6
2.2
2.0
4.0
BEGAN
2.99 1.
1.
3.9
VAE
57.5 0.4 0.3
1.2
0.5
WAE-GAN
31.5 0.2
0.4
9.5
6.75
Two-Stage VAE
26.1 0.9 1.79
96.1
6..2
RAE+ GMM
25.1= 0.2 1..
0.6
5..8
GLANN (with perceptual loss)
0.1
13.0
4..
4.1
VAB+flow prior
51.8 2.2
1.0.4
54.3
VAB+flow posterior
2.3
55.1 0.3
143.6
6.39
GLF (ours)
8.2
21.3
8..
532
GLF GLF+perceptual loss (ours)
0.1
10.3 10.3= 0.1
0.3
4.2
15

---

Under review as a conference paper at ICLR 2020
E ISSUES WITH LATENT REGULARIZATION
In this appendix section, we present the plots FID scores, reconstruction loss and NLL loss of regularized GLF. Related results are discussed in Section 4.2.1
F TRAINING TIME COMPARISONS
In Table we report the number of training epochs of our method, two-stage VAE and GLANN. In Table 6 we report the clock training time per epoch of these methods. Note that for methods using perceptual loss, the per epoch training time longer because VGG activations need be computed. These two tables show that GLF needs much shorter training time than the two competing methods. In GLF, training the flow does not add much computational time due to the low dimensionality.
Table 5: Number of training epochs for sto-stage VAE, GLANN, and GLF
MNIST/Fashion
CIFAR-10
CelebA
Two-stage VAE First/Second 400/800
1000/2000
120/300
GLANN First/Second GLF
500/50
500/50
500/50
100
200
40
Table 6: Per-epoch training time in seconds
MNIST/Fashion
CIFAR-10
CelebA
2-stage VAE Ist/2nd GLF
5/2
6/2
60/28
10
13
108
GLANN with perceptual loss
14
16
292
GLF with perceptual loss
16
19
343
G NEAREST NEIGHBORS OF GENERATED SAMPLES IN THE TRAINING SET
Quantitative measurements sample quality, such as FID score and Precision/Recnt can be min- imized by letting the generative model memorize the training set. The smooth transition of noise interpolation shown in Figure 2 provides evidence that our model can generalize, i.e., generate samples that have not be seen during training. Here provide additional evidence showing that our generative model generalizes well.
We randomly generate we present the 5 nearest neighbors of each generated samples the training set. The nearest neighbor defined in terms of L2 distance. Results are shown in Figure By inspecting the figure, we find that we can easily differentiate each generated sample from the closest training data. This indicates that our model generalizes well.
samples from the models trained on MNIST and CelebA datasets. Then
H MORE QUALITATIVE RESULTS
In Figure we show more samples of each dataset generated by GLF, using either MSE or perceptual loss as reconstruction loss. In Figure 7 we show samples of CelebA-HQ datasets from GLF trained with perceptual loss. Figure we show examples of interpolations between two randomly sampled noises on CelebA from GLF trained with perceptual loss.
16

---

Under review as a conference paper at ICLR 2020
5
5
5
5
5
1
1
1
1
6
6
6
6
6
6
9
9
9
9
9
9
4
44441
4
4
71
2
2
2
2
4
4
4
4
9
9
6
6
6
6
6
6
7
7
2 4
7 4
7 4
7
4
4
4
(a) MNIST
(b) CelebA
Figure 5: Some randomly generated samples are presented the leftmost column in each picture. The other 5 columns of each picture show the top 5 nearest neighbors of the corresponding sample in training set.
17

---

Under review as a conference paper at ICLR 2020
9
0
3
7
3
8
1
7
2
7
0
0
9
4
3
9
4
6
4
9
5
7
3
9
4
9
5
9
5
0
9
3
8
9
2
6
3
2
3
A
8
2
9
5
2
3
9
3
4
8
O
8
2
O
7
|
(a) MNIST
Fashion MNIST
(c) CIFAR-10
(d) CelebA
8
7
0
7
9
5
N
0
S
3
6
3
3
9
7
9
0
5
5
8
0
6
2
3
9
3
7
6
6
3
9
2
4
5 7 9 6
/ /
6
0
4
7
4
3
2
0 1 6
5
7
9
8
0
3
9
(c) MNIST
(1 Fashion MNIST
(g) CIFAR-10
(h) CelebA
Figure 6: (a)-(d) Randomly generated samples from method with MSE loss. (e)-(h) Randomly generated samples from our method with perceptual loss.
18

---

Under review as a conference paper at ICLR 2020
Figure 7: Randomly generated samples from our method with perceptual loss on CelebA-HQ dataset
19

---

Under review as a conference paper at ICLR 2020
Figure 8: Noise interpolation on CelebA
20


Reviews:
[{"review_id": "rJltgrq2FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a model that combines a simple Auto-Encoder (AE) together with a Normalizing Flow (NF) model, such that to derive a generative model. In particular, the AE is used to learn a low-dimensional representation of the given data in a latent space. Then, a NF model learns under a maximum likelihood principle, the distribution of these latent codes by applying an invertible transformation on a easy to sample distribution.\n\nGeneral Comments:\n\nThe paper is ok written, but some parts probably can be improved (see comment #5). As regards the proposed idea of the paper, I think that it is closer to an engineering practical approach than a consistent modeling choice. In particular:\n\n1. With the current modeling the likelihood p(x|z) is not defined, which means that the density of the p(x) can not be evaluated. So this should be considered as an explicit or implicit generative model? In that case the comparison with explicit density models where we can evaluate the p(x) is a bit unfair. Otherwise, the test log-likelihood should be provided.\n\n2. Learning the prior is already a debatable choice. However, in the proposed idea I have the feeling that there is a strong overfiting issue. Since the NF model is trained in a maximum likelihood principle, it will try to put all the mass from the simple distribution p(e) on the training latent codes z_i. Consequently, I am a bit sceptical as regards the generalization power of the generative model. Does the latent distribution learn something meaningful (an illustration could have been informative) or just how to re-generate the training data? \n\n3. The authors claim that the training is end-to-end. However, I think that the model which performs better is actually a 2 stage training model. More specifically, due the sg[.], the L_NLL term does not have any influence in the L_recon term. Therefore, the AE is trained independently, and simply the NF at every step \"follows\" and tries to capture the latent (empirical) distribution of the encoded training data.\n\n4. From the experiments is argued that the proposed model provides better samples than the competitive methods. I have the feeling that the generated samples are basically very similar to the training samples. Because the learned prior essentially learns to generate the latent codes of the training data. Does the model generalize i.e. can it generate samples that have not be seen during training?\n\n5. I think that the first paragraph of Sec. 2 and some parts of the next paragraph need improvement. Also, how the decoder implies the distribution \\tilde{p}(z) in the latent space? I would expect the encoder to be responsible for the latent distribution. Moreover, in the classic VAE the KL divergence is used between the approximate posterior q(z|x) and p(z), while the KL between the aggregated posterior q(z) and p(z) is not the default choice, and usually, is not straightforward to optimize.\n\nIn general, I think that the proposed model is a rather good practical approach, but probably not a very well defined modeling choice. As a practical approach, the experiments is most of the times the only way to support the argued behavior. The conducted experiments mainly focus on the FID score. However, I think that it would have been interesting to include examples that show the latent distribution and why is this better from other models, for example less regularized from the p(z) in VAE. Also, another crucial issue is the level of overfiting the current approach might have, because learning the prior implies this behaviour. Does the generated samples cover only the training distribution or can it generalizes to unseen test samples?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xybyqCKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new model combining an auto-encoder (AE) and a normalising flow (NF). The model, Generative Latent Flow (GLF), uses the AE to map the inputs to a latent space, which is then transformed using the NF. The approach is intuitively beneficial in that the AE can reduce the dimensionality of the inputs such that the NF mapping becomes much faster, computationally. The proposed method is compared to related methods that use a variational AE (VAE) in combination with an NF, and the similarities are pointed out and studied empirically.\nThe authors compare the performance of GLF to a large number of competing methods, showing very competitive results. In particular, for $\\beta = 1$, GLF significantly outperforms its VAE+flow prior sibling in terms of the Fréchet Inception Distance (FID) measuring the quality of the generated images.\n\nThe paper presents a well-motivated method, which is extensively and thoroughly evaluated. The experimental part is the paper's main strength, as the method itself is quite incremental (replacing a VAE with an AE). The authors do, however, spend considerable effort comparing the two versions of the method (using VAE and AE, respectively) both mathematically and experimentally. This is very well done and provides the reader with a good understanding of the behaviour of both models. My main concern is the lack of novelty in the proposed method.\n\nI am also slightly concerned that the paper tries to oversell the method a bit. Among the claimed benefits of the method are 1) better density mapping without over-regularised latent variables, 2) single-stage training, 3) minimal reconstruction trade-off, and 4) faster convergence. As far as I understand, benefits 1, 2, and 3 are shared with similar models (VAE+flow) and are as such not unique to GLF. I am not convinced that benefit 4 is true either, as from figure 3, the convergence rates seem to be similar. GLF clearly reaches a better FID, but this seems to happen before epoch 100, which is the earliest shown. It is not clear from the plot if GLF simply starts out being better or when it gains the advantage. Furthermore, in the discussion just below figure 3, the authors note that \"even with large $\\beta$, GLF still slightly outperforms VAE+flow prior\". I find this to be a stretch - had the training stopped at epoch 400, the conclusion would have been that the methods perform identically.\n\nWhile the authors have clearly put a lot of effort into the paper, they seem to have been rushing for the deadline. There are numerous typos and half-missing sentences (too many to list all, but the worst are pointed out below), so the text needs some polishing before publication. I think it would also make sense to rework section 1 and 2 as, currently, they both present introduction, motivation, and related works.\n\nQuestions:\n- Just above section 3.2, you say that you add a random permutation after each coupling layer. This is not shown in figure 1(b) if I understand it correctly. Here, only a permutation after the entire block is shown. Did I misunderstand the model?\n- Were the model runs in figure 3 also repeated as in table 1? If so, are the standard deviations just too small to be seen or nor shown at all?\n\nMinor comments:\n- \"Auto-encoder\" has inconsistent capitalisation throughout the paper.\n- The very first sentence of the introduction misses an ending.\n- Fourth sentence of section 2 misses an ending.\n- Fifth sentence of section 2 changes the notation - I believe it should be z ~ p(z) here to be consistent.\n- There are many examples of a whitespace missing between a reference and the preceding word.\n- p 2: \"detremine\" -> \"determine\"\n- p 4: \"assumptio\" -> \"assumption\"\n- p 6: \"Frchet\" -> \"Fréchet\"\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxSFyR6Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a deterministic Auto-Encoder with a trained marginal distribution over latent variables, p(z), to be able to sample from the model. For this purpose, the authors propose to use a flow-based model for p(z), and regularize the AE objective (i.e., MSE) with a cross-entropy between q(z) = 1/N \\sum_n E(x_n) and p(z). In general, I find the idea quite interesting. The construction of the objective and motivation behind is rather clear. The experiments are rather convincing (however, the FID metric is subjective, but it's impossible to calculate the log-likelihood scores). The main disadvantage of the paper is its language. There are many typos and difficult to follow sentences. But besides that, the main ideas are well explained. Please find more specific remarks below. \n\nRemarks\n- The language in the paper is a bit off. There are sentences that sound very peculiar, e.g., \"Deep generative models can be roughly classified into explicit and implicit models. The former class assumes explicit parametric specification of the distribution, whereas the latter does not.\" \nThe introduction should be re-written. Similarly, Section 2 is hard to follow.\nThere are places where a word or a punctuation mark is missing, for instance:\n* The first line misses a word: \"(...) on deep learning.\"\n* First paragraph, Section 2: \"This distribution is unknown and possibly The model also has a predefined prior distribution p(z) on Z.\"\n* Section 3.2, below Eq. 2: \"(...) assumptio (...)\".\n* Section 3.4, last paragraph: \"Our work is differs in two ways (...)\".\n\n- The authors should refer to the following very relevant paper:\n* Xu, H., Chen, W., Lai, J., Li, Z., Zhao, Y., & Pei, D. (2019). On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder. arXiv preprint arXiv:1905.13452.\n\n- It is unclear how the authors dequantize image data that are typically represent as integers from 0 to 255 (or 0 and 1 in the binary case). The authors mention in Section 3.2 that they use the MSE loss for \\mathcal{L}_{recon}. This corresponds to taking a Gaussian decoder in the VAE framework. However, the manner how the images are dequantized is extremely important to properly evaluate all results. For instance, if the authors add a uniform noise, then it highly influences the final quality of a model.\n\n- I do not fully follow why the authors call the objective part for learning the marginal distribution over latents \"the NLL loss\". It is rather the cross entropy between q(z) = 1/N \\sum_n E(x_n) and p(z). This follows from the notation and the description that we stop the gradient for E(.). I find it confusing.\n\n- Figure 4 is extremely important for understanding why we should stop gradient. However, jumping between Section 4.2 and E is extremely annoying. The authors can use up to 10 pages, so adding a half of a page would not make a difference, but it would help a reader a lot.\n\n- The main difference between the objective in Eq. 2 and the ELBO lies in considering a deterministic encoder and entails skipping the entropy term for the variational posterior. Could the authors comment why this is so important to choose a deterministic encoder? I can easily imagine taking q(z|x) = Normal(z | mu(z), a1), i.e., fixing the variance to some value a (e.g., a=1), that would result in Entropy[q] = const. Hence, we can skip entropy from the objective, but still we use a stochastic encoder. \n\n===== AFTER REBUTTAL =====\nI would like to thank the authors for their response and new version of the paper. I must admit that I had a very hard nut to crack. From one side, I really appreciate all the effort the authors put to improve the paper. However, on the other hand, I tend to agree with the reviewer #3 that the paper is interesting from the engineering perspective and it lacks novelty. Eventually, I decided to sustain my score. There are two reasons for that. First, more analysis of the model would be helpful. For instance, analyzing the latent space would be interesting (a suggestion from the reviewer #3). Second, considering a pure AE with a trainable flow-based prior is interesting, but it is also very limiting in the sense of obtaining nicely looking pictures instead of being able to provide (approximate) probability of an image. In my opinion, from the decision making perspective, having a probability of an object is much more important  than being able to generate a crisp image.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose generative latent flow which uses autoencoder to learn latent representations and normalizing flows to map that distribution. The reviewers feel that there is limited novelty since it is a straightforward combination of existing ideas. 
Decision: Reject