Variational Autoencoders with Normalizing Flow Decoders
Authors: Rogan Morrow
Wei-Chen Chiu
Venue:  (2020)

Abstract: Recently proposed normalizing flow models such as Glow (Kingma & Dhariwal, 2018) have been shown to be able to generate high quality, high dimensional images with relatively fast sampling speed. Due to the inherently restrictive design of architecture , however, it is necessary that their model are excessively deep in order to achieve effective training. In this paper we propose to combine Glow model with an underlying variational autoencoder in order to counteract this issue. We demonstrate that such our proposed model is competitive with Glow in terms of image quality while requiring far less time for training. Additionally, our model achieves state-of-the-art FID score on CIFAR-10 for a likelihood-based model.

Under review as a conference paper at ICLR 2020
VARIATIONAL AUTOENCODERS WITH NORMALIZING FLOW DECODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Recently proposed normalizing flow models such as Glow (Kingma & Dhariwal (018)) have been shown to be able to generate high quality, high dimensional images with relatively fast sampling speed. Due to their inherently restrictive architecture, however, is necessary that they excessively deep in order to train effectively. In this paper we propose to combine Glow model with an underlying variational autoencoder in order counteract this issue. We demonstrate that such our proposed model is competitive with Glow in terms of image quality while requiring far less time for training. Additionally, our model achieves state-of-the-art FID score on CIFAR- 10 for likelihood based model.
INTRODUCTION
The field of deep generative models is experiencing rapid progress, with state-of-the-an results being achieved regularly. Currently, state-of-the- in image generation is dominated by adversarial training methods such as Karras et al. (2019) and Brock et al. (2019). However, Generative Adversarial Networks (GANs) notoriously suffer from problems with training stability and mode collapse due to the dynamics of an adversarial objective. Many ideas have been proposed for dealing with these issues, such as Arjovsky et al. (2017) and Miyato et al. (2018), however issues inherently remain. Due to this, there is still signficant attention being given to pure likelihood models despite their poorer qualitative performance, mode coverage on account of its heavy penalization of models that do not allocate probability mass to space containing training data.
maximum likelihood grants stable training and is able to achieve good
There are numerous classes of likelihood-based deep generative models in the contemporary literature. Autoregressive models such as PixeIRNN (van den Oord et al. (016) have been shown to achieve impressive test likelihood scores on various datasets, however such models typically perform poorly when evaluated using quantitative measures of image quality, and in addition take long time to generate samples. Variational autoencoders (VAE) (Kingma & Welling, 2014), a form of latent variable model and one of the earlier proposed deep generative models, samples of poor quality when implemented in their most basic form and as such require combining with more powerful generative models in order be competitive. More recently, it was shown in Kingma & Dhariwal (2018) that a class of likelihood models known as normalizing flows can be used generate high quality images with much faster sampling times than autoregressive models. Heavy restrictions must be imposed on the model architecture however in order to be able to efficiently calculate the determinant of the model Jacobian, forcing implementations of such models to be extremely deep and wide to make up for deficiencies of restricted architecture. As result, training requires an exorbitant amount of time; as an example 40 GPUs and ~ week training was required in order to produce the 256x256 CelebA-HQ samples in Kingma & Dhariwal (2018). Finding ways to cut down on these resource requirements is therefore pertinent in order to make such models accessible to the average practitioner. Towards achieving this goal, in this work we propose combining the standard variational autoencoder model with an overlying Glow layer and demonstrate sfate-of-the--n FID (Heusel et al. 2017) results while simultaneously achieving much lower training time.
well known to produce

---

Under review as a conference paper at ICLR 2020
2 BACKGROUND AND RELATED MODELS
2.1 VARIATIONAL AUTOENCODERS
Variational Autoencoders (VAEs), first proposed in Kingma & Welling (2014) are popular class of deep latent variable model inspired by traditional variational inference. Given dataset X (x(1), (2)) ,..,,,!!11 x(1) ). the goal is to maximize the log-likelihood lower bound
E (2 [lo p(x [z]] DKL [x=l((()="0)1((a))
where q(2/z) is an encoder distribution and p(z|z) is a decoder distribution, both parameterized by deep neural networks. The prior distribution p(z) may be fixed or learned. Typically the encoder, decoder and prior distributions are all assumed to be some simple distribution, usually Gaussian. In this case the model enjoys the properties of stable training and fast convergence, well fast sampling. VAES are well known to produce sample images of poor quality, however, owing to a number of previously identified factors.
It has been pointed out that the typical Gaussian assumption of the posterior ((2[2)) leads to poor posterior approximations, as the true posterior is often far from Gaussian in shape. As the optimizer is forced to fit mismatching distributions, this results in the marginal encoder distribution g(z) diverging from the prior p(2). In Jimenez Rezende Mohamed (2015) and many subsequent works such as Kingma et al. (2016), it was proposed that the posterior be made more flexible via the use of normalizing flows, which are discussed detail in the next section. In this way, the approximate posterior can achieve a better fit to the true posterior.
In Dai & Wipt (019) the authors pointed out that the failure to optimize the decoder variance in typical implementations of VAEs is major cause of poor quality samples, and that by optimizing decoder variance is possible to achieve an optimal VAE cost in the case where the encoder and decoder distributions are Gaussian (specifically. diagonal Gaussian encoder and isotropic Gaussian decoder). Achieving the optimal VAE cost does not necessarily recover the ground truth distribution, though, and they additionally proposed learning the distribution of the latent space using second- stage VAF, and proved that by doing so it is theoretically possible to recover the ground truth distribution.
Despite there being a theoretically obtainable optimal cost in the case where the decoder variance is learned, in practice using a decoder that assumes independence between pixels is not ideal, is well understood that this leads to blurry reconstructions due to the optimizer favouring solutions that are a weighted mean of data points pixel space. Additionally, because Gaussian decoders are a poor fit for natural images, they may introduce substantial salt-and- -nndee noise into the generated samples, and so it is commonplace to restrict decoder sampling to the mean. This artificially restricts the generative distribution to manifold of dimensionality equal to that of the latent space, and potentially limits sample diversity. It is precisely these deficits of simple decoder distributions used in standard VAE models that we attempt to address in proposal.
Complex, non-Gaussian decoder distributions have been proposed several previous works. In Gulrajani et al. (2017). the authors propose using autoregressive PixeICNN layers (Van den Oord et al. (016) in the decoder, and are able to achieve impressive test likelihood scores. An adversarially trained autoencoder was proposed in Larsen et al. (0166). with the decoder objective being defined by a standard GAN loss over the generated image plus a representation loss using 1.2 distance between discriminator hidden features. However, such model cannot be used to evaluate likelihood, and being an adversarial method it suffers from problems with training stability. [Shmelkov et al. (2019) propose very similar model our own, with a Real NVP (Dinh et al. (2016) normalizing flow model stacked on top of VAE with Gaussian decoder. They combine likelihood maximization with adversarial training to achieve competitive FID scores on various datasets, however their model achieves poor results when using purely likelihood based approach.
2.2 NORMALIZING FLOWS
Normalizing flows take advantage of the change of variables formula for probability distributions, given by
( = p(=) det (xxx)())))
2

---

Under review as a conference paper at ICLR 2020
(2z)
P& (i
91 (2)
Glow Layer
9r (2)
h
u
Phase-1
Phase-2
Figure 1: Overview of our proposed method, which combines simple VAE with normalizing flow model in the pixel space. The model training is split into two phases, please refer to Section 3.4 for more details.
where f defines bijection between the spaces of x and They have found use with application to
VAEs order to achieve flexible priors/posteriors (Jimenez Rezende & Mohamed 2013 Chen et al. 2017), well being used as standalone generative models directly in data space (Dinh et al. 2014)
in which case exact likelihood evaluation is possible. One of the most common implementations that of autoregressive flows, whereby the bijection f is made be auto-regressive resulting in a triangular Jacobian. This is important for efficiency, as it allows the determinant in Eq. to be
calculated in (nn) rather than Onz²). Composition of multiple autoregressive functions allows for the construction of high capacity models.
3 METHOD AND MODEL DEFINITION
At high level, our idea is to combine simple VAE model with normalizing flow model in the pixel space. As such, we hope to obtain the benefits of both: the smaller model size, faster training, and compact latent representation of VAEs, well as the high quality samples of normalizing flows. We use a simple VAE to learn a base distribution, and train a normalizing flow using this base distribution, such that end up with conditional normalizing flow. As the VAE should be able to learn distribution that is already close to the ground truth, our normalizing flow, when conditioned on this base distribution, should not need to do much work full marginal normalizing flow model such as Glow (Kingma & Dhariwal 2018). Our normalizing flow therefore need not be as deep as such models, saving training time and model size. Our model is trained using pure likelihood maximization, and is capable of evaluating lower bound on the likelihood of arbitrary data samples.
For our approach we use an underlying Gaussian decoder learn the base distribution of a normalizing flow. We denote the random variable produced by this underlying decoder as
T ~ P~Pa 2 = (T (2), g5(2))
(3)
where decodes the mean of pa (z) and gy: decodes the diagonal covariance. The normalizing flow layer then defines bijection f X + X and random variable
$ pa (+)=) = Pa (f dee
(4)
where i E & and & e X.
3.1 UNDERLYING GAUSSIAN DECODER
In Dai & Wipi 2019) the authors propose to add learnable parameter y to the model, with the decoder likelihood given by (x)g 7T), and prove that doing so enables the model to always be
3

---

Under review as a conference paper at ICLR 2020
capable of achieving better VAE cost by lowering the value of y. This has an additional benefit in the case of our model, as it facilitates training of the overlying Glow layer. By allowing the VAE decoder to shrink its variance it gains confidence in its estimates, the Glow layer has to do less work to remove any erroneous noise in the decoder distribution.
We extend this idea by allowing the decoder distribution to be Gaussian with diagonal covariance, rather than simply an isotropic Gaussian. By doing so, we allow the decoder to express confidence in its estimates at the pixel level. In the case of MNIST, for example, the decoder is likely be highly confident in its estimate of the uniformly black background pixels, while the pixels on the border of the digit are likely to be the source of lower confidence. When sampling from the underlying VAE (while keeping decoder samples restricted to the mean) this does not result in any significant improvement over the simpler isotropic Gaussian decoder. For our full model, however, we found in our experiments that this small change resulted in significant improvements to FID score.
3.2 PRIOR DISTRIBUTION
As noted previously, we allow the decoder to learn the variance of its distribution. However, as shown in Dai & Wipl (2019), this results in the decoder likelihood quickly dominating the KL divergence term, resulting the model favouring learning of the data manifold over learning the ground truth distribution. In other words, the model is able to learn much more accurate reconstructions at the expense of the encoder distribution matching the prior. Their proposed solution is to learn the encoder distribution using a second second-stage VAE, and they prove that by doing so they are theoretically capable of recovering the ground truth distribution. We note, however, that by using a flexible learned prior in the form of an autoregressive normalizing flow, we are able to calculate exact prior likelihood and are also able to recover the ground truth distribution in a single stage of training. Recovery of the ground truth distribution is possible as autoregressive normalizing flows are universal approximators (Huang et al. 2017). This gives us a second latent distribution p(u) = NV [wu[0, I) and bijection h : 2 2 with 2 and E U. The prior distribution is then given by p(=) p(u) det det(Oh())/0z")
Such a model was originally proposed in Chen et al. (2017). where the authors show that it is
equivalent Inverse Autoregressive Flow (IAF) (Kingma et al. 2016) along the encoder path, while being deeper along the decoder path. We therefore obtain the benefit of using a flexible posterior while simultaneously being able to close the gap between the encoder and prior distributions. We found in our experiments that by using normalizing flow prior we were able to obtain better FID scores than by using a second-stage VAE. Results are shown in the experiment section.
In our implementation we use a normalizing flow similar in structure to Real NVP Dinh et al. (016)
(which is a special case of autoregressive normalizing flows PPapamakarios et al. 2017 ). as allows both efficient training and sampling.
3.3 NORMALIZING FLOW LAYER
We use the additive version of the Glow model proposed in Kingma & Dhariwal (2018). More specifically, our flow is comprised of several blocks, with each block containing an activation normalization followed by an invertible 1xl convolution, which is then followed by additive coupling layer (Dinh et al. 2014). Since our flow does not need to be as deep on account of it being conditioned on the underlying VAE distribution, we do not utilize any split or squeeze operations.
3.4 TRAINING
We split the model training into two phases. In the first phase, we train the underlying VAE and its prior. In the second phase, keep all components of the underlying VAE including its prior fixed while training the Glow layer on top of it. That is, in the second stage we optimize the parameters of
the Glow layer to maximize ( [log Pa (z(()))) [2]]. This continues to maximize likelihood, as the first term in Eq. iis maximized while the second term is kept fixed. It is also possible to train the
entire model jointly in one phase, however we found that this resulted in poorer quality images, likely
because the Glow layer is unable to train efficiently with changing base distribution.
4

---

Under review as a conference paper at ICLR 2020
3.5 SAMPLING
Sampling from the model is performed straightforwarly by first sampling T from the underlving marginal VAE distribution, and then calculating the final sample via (z). In Kingma Dhariwal sampling in order to improve image quality. We
(2018). the authors make use of reduced-temperatte note however that, since this operation results in significantly reduced sample diversity, it will have negative impact distance measures such as the FIID score. In our model, because the underlying VAE is trained first and then subsequently kept fixed during training of the Glow layer, the VAE ends up being responsible for sample diversity while the Glow layer "sharpens" the VAE output. Due to this, we are able to perform lemperature reduction in the Glow layer without significantly sacrificing sample diversity. We found that using temperature of 0.5 in the Glow layer produced the highest FID scores across all our experiments.
4 EXPERIMENTS
4.1 NORMALIZING FLOW PRIOR VS TWO-STAGE VAE
We begin our experiments by first demonstrating our choice of normalizing flow prior advantagcous over the second-stage stage VAE proposed in Dai & Wipl (2019). We conduct experiments on Fashion- MNIST (Xiao et al. 2017) and CIFAR-10 (Krizhevsky et al., 2009) using decoder distribution given by X gz (2),7Z), 7I), where is a learnable parameter. The experimental results are shown in Table where the benefit of using normalizing flow prior in our proposed method is well provided, in comparison to the 2-stage VAE (Dai & Wipf 2019). Please note that here we especially the same network architecture for DDai & Wipt 2019) and the VAE of our proposed model, in order to have fair comparison.
FID
Fashion (2nd VAE)
31.87
Fashion (Flow)
19. 70
CIFAR- 10 (2nd VAE)
76.04
CIFAR- 10 (Flow)
71.85
Table 1: FID scores comparing normalizing flow prior with second-stage VAE.
4.2 RESULTS FOR FULL MODEL
We now present our experimental results on both the CIFAR-10 (Krizhevsky et al., 2009) and CelebA (Liu et al. 2015) datasets. For both datasets we use single layer Glow (i.e. no squeeze split operations) of depth 32. For the underlying VAE we use the Resnet model described in Dai & Wipf (2019), with depth of 3 for CIFAR-10 and depth of 4 for CelebA. The prior was implemented using a 16 layer normalizing flow with each layer comprised of two fully connected layers of width 1024, each followed by ReLU. For both datasets trained the underlying VAE for 1 1,000 epochs before training the Glow layer also for 1,000 epochs. We use an initial learning rate of 0.0001 and halve it every 250 epochs. In the case of CIFAR-I0. 10, we found that retraining the prior with early stopping after training the VAE helped to improve FID. Early stopping was done after 50 epochs. We note that we do not make use of data augmentation in any of our experiments.
4.2.1 QUANTITATIVE EVALUATION
We report Fréchet Inception Distance (Heusel et al.) 2017) calculated against a held-out test set for quantitative image quality evaluation, well as test bits per dimension. For our CIFAR-10 10 results we include FID scores and biss-per-dimension reported in previous works involving likelihood- models, in order to provide more comprehensive list of baselines; results were taken from Chen et al.) (2019) and Ostrovski et al. (0188). In Chen et al. (2019) the authors report an FID score of 46.90 for Glow, however after training the official Glow model ourselves using additive coupling layers for 2,000 epochs we achieved an FID score of 45.23, so we report this our table instead. Results for CIFAR-10 can be found in Table 2 and results for CelebA can be found in Table 3 We achieve
5

---

Under review as a conference paper at ICLR 2020
FID
bits/dim
PixeICNN
65.93
3.03
FID
bits/dim
PixelIQNT
49.46
Glow
19.00
2.25
Glow
45.23
3.35
Ours
20.64
2.81
Residual Flow
46.37
3.28
Ours
42.14
3.17
Table 3: FID score on CelebA
Table 2: FID score and bits per dimen- sion on CIFAR-10 (f uses quantile regression)
what is to our knowledge state-of-the-art FID CIFAR-10 for pure likelihood model, while also outperforming Glow on test likelihood. For CelebA dataset, we have competitive FID to the Glow model, but with much less time for training, as what is going to be discussed in the Section 4.3
(a) Glow Kingma & Dhariwal
2018
(b) Ours
Figure 2: Samples from models trained CIFAR-10.
(a) Glow
Kingma Dhariwal 2018
(b) Ours
Figure 3: Samples from models trained on CelebA.

---

Under review as a conference paper at ICLR 2020
4.2.2 QUALITATIVE EXPERIMENTS
First, some qualitative results for CIFAR-10 10 and CelebA datasets are provided in Figure Figure respectively, based on the Glow model and our proposed one. We can see that our proposed model able to produce competitive or better results in terms of image quality and diversity. Then, we present qualitative experiments to demonstrate the behaviour of our model. As mentioned earlier, the Glow layer essentially becomes a "sharpening" layer as result of our two-stage training procedure. To demonstrate this, in Figure 4 we show random samples from the VAE alongside samples from the same latent code after having been passed through the Glow layer. It can be seen that the VAE samples are somewhat blurry expected, and are considerably sharper after being passed through Glow. The content is entirely preserved by the Glow layer, with stochasticity resulting in small changes to the fine details that are only visible upon close inspection.
Additionally, we show images obtained by interpolating between randomly sampled points in the VAE latent space in Figure This is common practice for latent variable models in the literature to demonstrate the model's ability to generalize and learn a useful compact representation.
(a) CIFAR-10
(b) CelebA
Figure 4: Example of how the Glow layer modifies VAE samples. Left column: samples from the underlying VAE. Right columns: decoder samples after being passed through the Glow layer.
T
S
20
2
2
2
2
25
25
2
Figure 5: Interpolation between randomly sampled points in the VAE latent space, demonstrating that our model learns a useful compact representation.
4.3 TRAINING TIME
One of the main claims made regarding our model is that can significantly reduce training time when compared with standalone Glow model. In order to validate this claim, measure wall clock
7

---

Under review as a conference paper at ICLR 2020
CIFAR-10
Time/epoch (seconds)
Glow
512
Ours (VAE)
118
Ours (Glow)
162
Ours (Avg.)
140
Table 4: Training time per epoch for different models (dataset:CIFAR-1O).
time per epoch when training each model and report the results in Table Times were measured using a GeForce GTX 1080 Ti. We measure VAE and Glow layer times separately for our model and, since we train an equal number of epochs for each, include the average of the two for the overall time per epoch. We make a note of the fact that we did not use gradient checkpointing when evaluating training time. Results indicate that our model trains over 3.5x faster than Glow.
CONCLUSION AND FUTURE RESEARCH
We have presented model combining standard VAEs with Glow, and demonstrated that it is capable of achieving better sample image quality than a standalone Glow model while being significantly faster train. our hope that our results are useful any future works attempting to bridge the gap between image quality of likelihood- involve investigating whether our results continue to hold when scaling up to higher dimensional images.
models and GANs. Future research directions may
REFERENCES
Siddharth Agrawal and Ambedkar Dukkipati. Deep Variational Inference Without Pixel-Wise Reconstruction. arXiv e-prints, Nov 2016.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. In Proceedings of the International Conference Machine Learning (ICML), 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual Flows for Invertible Generative Modeling. In Advances in Neural Information Processing Systems (NIPS), 2019.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Bin Dai and David P. Wipf. Diagnosing and enhancing VAE models. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components
Estimation. In International Conference on Learning Representations (ICLR) Workshop, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Advances in Neural Information Processing Systems (NIPS) Deep Learning Symposium, 2016.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. In Advances in Neural Information Processing Systems (NIPS), 2017.
8

---

Under review as a conference paper at ICLR 2020
Chin-Wae Wei Huang, Ahmed Touati, Laurent Dinh, Michal Drozdzal, Mohammad Havaei, Laurent Charlin, and Aaron C. Courville. Learnable explicit density for continuous latent space and variational inference. In International Conference on Machine Learning (ICML) Workshop, 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Proceedings of the International Conference on Machine Learning (ICML), 2015.
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Advances in Neural Information Processing Systems (NIPS), 2018.
Diederik P Kingma and Max Welling, Auto-Encoding Variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Diederik Kingma, Tim Salimans, and Max Welling. Improving variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems (NIPS), 2016.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. In Proceedings of the International Conference Machine Learning (ICML), 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference Learning Representations (ICLR), 2018.
Georg Ostrovski. Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
George Papamakarios, Theo pavlakou, and lain Murray. Masked Autoregressive Flow for Density Estimation. In Advances in Neural Information Processing Systems (NIPS), 2017.
Konstantin Shmelkov, Thomas Lucas, Karteek Alahari, Cordelia Schmid, and Jakob Verbeek. Cover- age and quality driven training of generative image models. ArXiv:1901.0109 2019.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems (NIPS), 2016.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. doshion-mnise a novel image dataset for benchmarking machine learning algorithms. ArXiv: /708.07777 2017.


Reviews:
[{"review_id": "rJgx7oKM5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes adding additional flow layers on the decoder of VAEs. The authors make two claims\n1. The proposed model achieves better image quality than a standalone Glow.\n2. The proposed model is faster to train than Glows.\nThe intuition is a VAE can learn a distribution close enough to be target distribution, and the Glow only needs to do much less work than standalone Glow, hence faster. Some positive results are reported in the experiments, including better image quality, faster training time, and the Glow indeed sharpens the output of VAEs. \n\nThe paper indeed has some good results, particularly they can achieve it only with single-scale Glows with additive coupling layers. However, I think the claims are not sufficiently supported. Taking point 1 as an example, it is not clear to me why VAE+Glow is better than a standalone Glow. Imagine two models\n\nM1: VAE+Glow (proposed in the paper)\nM2: Glow0+Glow (standalone Glow)\n\nsharing the last \"Glow\" part. M1 is better than M2 implies \"VAE\" is more powerful than \"Glow0\", which I doubt. Similarly, for point 2, it is not clear to me why \"VAE\" is faster than \"Glow0\". I think comparing the proposed model with IAF make more sense, because the proposed model just adds flows to the decoder and the prior. But the relationship with Glows needs to be considered more thoroughly.\n\nAnother confusing detail for me is the two-stage training in Sec. 3.4. The explanation \"likely because the Glow layer is unable to train efficiently with a changing base distribution\" doesn't make sense. Because IAF does successfully train their q-net without 2-stage training. There might be other reasons?\n\nThe baselines are not strong enough. Most importantly, Flow++ [1] reports a likelihood 3.08 on Cifar10 with standalone flows, which should also be a part of the baseline. I also wonders whether the proposed model benefits from deeper model, like standalone flows do. Will standalone flows surpasses the proposed model as the number of layers goes to infinity?\n\n[1] Ho, Jonathan, et al. \"Flow++: Improving flow-based generative models with variational dequantization and architecture design.\" arXiv preprint arXiv:1902.00275 (2019).\n\nFinally, the paper is somewhat incremental. Particularly comparing with VAE-IAF, where this paper just adds flow layers to not only q but also p.\n\nUpdate\n=====\n\nAfter reading the rebuttal I found some of my concerns are unaddressed. (regarding to the two-stage training, and the novelty)\n\nPoint 1 is still not answered. The answer I expect to have is how \"The interaction between two models when they are being stacked may affect the overall performance in such a way that it is more than just the sum of its parts.\" Why are these two models perform particularly well when combined? The purpose of my initial question is for some in-depth analysis and intuition / theory.\n\nTherefore I will keep my score unchanged.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJex9qu3tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a combination of a conditioned flow-based model with a VAE. The main contribution of the paper is a two-phase training that allows to train the model. Unfortunately, a joint training of the model failed. In general, combining VAEs with flow-based models is an important research direction. Unfortunately, the paper is not clearly written. A lot of details are missing that makes the paper impossible to reproduce and fully understand. Moreover, the most interesting part about training procedure, is discussed superficially. Finally, I find the comparison to SOTA methods disappointing. The authors skipped many recent papers. I do not expect to see SOTA results among all models, but at least comparable results within a group of approaches. However, the authors provide only a small subset of models that makes me wonder whether they are aware of other papers.\n\nRemarks:\n- The following statement is not fully correct:\n\"In our implementation we use a normalizing flow similar in structure to Real NVP Dinh et al. (2016) (which is a special case of autoregressive normalizing flows Papamakarios et al. (2017)), as it allows both efficient training and sampling\"\nReal NVP is a bipartite flow, while Masked Autoregressive Flow is an autoregressive flow. In special cases, these two flows coincide, but their original implementations are different.\n\n- The paper misses a lot of important details. For instance, a reader needs to figure out what is the objective function. Further, the authors do not mention how they deal with images represented by integers. Do they use dequantization as in other papers (e.g., Theis, L., van den Oord, A., and Bethge, M.  A note on the evaluation of generative models. In Workshop Track,ICLR, 2016)?  These are very important details to fully understand the proposed approach. Providing a very general diagram (Figure 1) and generic equations (2-4) are not sufficient. Currently, there are different implementations of flow components, and describing them would be definitely beneficial. Moreover, it is important to show how conditioning is used in the flow model.\n\n- Section 3.4 is incredibly interesting part of the paper and it should be further analyzed. The proposed two-phase is reasonable, but it leaves an open question why a joint training fails. I agree with the authors that a possible explanation is training instability. Nevertheless, I would be more than interested in seeing a more thorough analysis of this phenomenon.\n\n- I do not understand why the authors skipped reporting bpd for CelebA. \n\n- In general, the comparison in Table 2 is far from being complete. For instance, please see the following paper:\nHo, J., Chen, X., Srinivas, A., Duan, Y., & Abbeel, P. (2019). Flow++: Improving flow-based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.00275.\nOn Cifar10, current non-autoregressive models are able to achieve 3.08 bpd (Flow++)  and 3.11 bpd (IAF-VAE). This is much better than the reported 3.17 bpd.\n\n- In Section 4.2.2, first line, a number to a figure is missing.\n\n======== AFTER REBUTTAL ========\nI would like to thank the authors for their rebuttal. However, I am not fully satisfied with some answers. First of all, the paper should be clearly written so that a reader could be able to find all necessary details and seamlessly implement the presented idea. If some details do not fit the main text, then they should be included in an appendix. Second, the objective function is an important component of any ML problem and, therefore, it should be included in the paper. Sometimes a verbal description is not sufficient. Third, by conditioning a flow I meant whether the based distribution was conditional or/and a single flow layer was conditioned on z. Currently, there are multiple ways to condition and I was curious which of them was used by the authors. Fourth, I do not believe that achieving nicely looking images is the ultimate task for generative models. However, this is an open discussion. Fifth, I agree, the discussion on why the joint training failed is an incredibly interesting question.\nAgain, I really appreciate all the answers, and I believe that the authors did their best to improve the paper. However, as a reviewer I must look for novelty and evaluate how the paper is readable for others. In my opinion, combining VAE with GLOW is not sufficiently novel idea. If the paper was very clearly written, I would think about rising the paper. However, right now, I decide to keep my original score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1e3Kw7atH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a variant of the variational autoencoder, specifically by using a simplified normalizing flow model as the decoder. When compared with Glow, the proposed method is simpler and more efficient to train. The authors applied their algorithm on a few datasets, and showed better or competitive performance, both qualitatively and quantitatively.\n\nThis paper is in general well written. I think the idea looks interesting, although the novelty is a bit incremental, as it basically combined the two well-known models (VAE and Glow). The experimental results showed the promise of the new method, which could be more convincing if applied on larger scale datasets. My detailed comments and questions are as follows.\n1. Regarding training, the authors decomposed it into two phases, i.e., training VAE first and then Glow. The authors also mentioned that jointly training resulted in images with poor qualities. I am curious about how the authors designed the Glow model: Intuitively a larger model may have more modeling capacity, but at the cost of computational cost. Some ablation studies or explanation could be helpful.\n2. The authors claimed at the beginning of Section 3 that the their normalizing flow \"should not need to do as much work as a full marginal normalizing flow model such as Glow\". I am wondering how the performance will be for the Glow used, if without the VAE part? \n3. For the bits/dim results for Glow in Table 2, was it computed by yourself or just from the Glow paper? I saw the FID score was obtained by yourself.\n4. For the Glow used in experiments, how does its architecture compare with the one used in the original paper?\n5. I am a bit surprised to see the results in Table 3 and Figure 3, as Glow has a better FID score but the overall image quality is worse. Is it related to the size of the Glow used?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews: WR (R1,R3) and WA (R2). AC has carefully read reviews and rebuttal and examined the paper. Unfortunately, the AC sides with R1 & R3, who are more experienced in this field than R2, and feels that paper does not quite meet the acceptance threshold. The authors should incorporate the comments of the reviewers and resubmit to another venue. 
Decision: Reject