Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
Authors: Rewon Child
Venue: ICLR 2021 Spotlight (2021)
Keywords: VAE
generative modeling
deep learning
likelihood-based models

Abstract: We present a hierarchical VAE that, for the first time, generates samples quickly $\textit{and}$ outperforms the PixelCNN in log-likelihood on all natural image benchmarks. We begin by observing that, in theory, VAEs can actually represent autoregressive models, as well as faster, better models if they exist, when made sufficiently deep. Despite this, autoregressive models have historically outperformed VAEs in log-likelihood. We test if insufficient depth explains why by scaling a VAE to greater stochastic depth than previously explored and evaluating it CIFAR-10, ImageNet, and FFHQ. In comparison to the PixelCNN, these very deep VAEs achieve higher likelihoods, use fewer parameters, generate samples thousands of times faster, and are more easily applied to high-resolution images. Qualitative studies suggest this is because the VAE learns efficient hierarchical visual representations. We release our source code and models at https://github.com/openai/vdvae.

Published as a conference paper at ICLR 2021
VERY DEEP VAES GENERALIZE AUTOREGRESSIVE MODELS AND CAN OUTPERFORM THEM ON IMAGES
Rewon Child OpenAl
San Francisco, CA rewone ieconnopenai.om
ABSTRACT
We present a hierarchical VAE that, for first time, generates samples quickly and outperforms the PixeICNN in log-likelihood on all natural image benchmarks. We begin by observing that, in thcory. VAEs can actually represent autoregressive models, well as faster, better models if they exist, when made sufficiently deep. Despite this, autoregressive models have historically outperformed VAEs in log- likelihood. We test if insufficient depth explains why by scaling VAE to greater stochastic depth than previously explored and evaluating CIFAR-10, ImageNet, and FFHQ. comparison the PixeICNN, these very deep VAEs achieve higher likelihoods, use fewer parameters, generate samples thousands of times faster, and are more easily applied high-resolution images. Qualitative studies suggest this is because the VAE learns efficient hierarchical visual representations. We release our source code and models at wtthtts://// /topps:/giii wiigthtbbb.... com wwwp/o/w//ppaeeeee//////ee opena 1/vvvae
Low resolution
High resolution
Figure 1: Selected samples from our very deep VAE on FFHQ-256, and demonstration of
the learned generative process. VAEs can learn to first generate global features at low resolution, then fill local details in parallel at higher resolutions. When made sufficiently deep, this learned, parallel, multiscale generative procedure attains higher log-likelihood than the PixeICNN.

---

Published as a conference paper at ICLR 2021
INTRODUCTION
One potential path increased data-efficiency, generalization, and robustness of machine learning methods is to train generative models. These models can learn useful representations without hu- man supervision by learning to create examples of the data itself. Many types of generative models have flourished in recent years, including likelihood-basee toregressive models (Uria et al., 2013). variational autoencoders (VAEs) (Kingma & Welling 20144 Rezende et al. 2014), and invertible flows (Dinh et al., 2014 2016). Their objective, the nega- tive log-likelihood, is equivalent to the KL divergence between the data distribution and the model distribution. A wide variety of models can be compared and assessed along this criteria, which corresponds to how well they fit the data in an information -theoretic sense.
generative models, which include au-
Starting with the PixeICNN (Van den Oord et al. 2016), autoregressive models have long achieved the highest log-likelihoods across many modalities, despite counterintuitive modeling assumptions. For example, although natural images are observations of latent scenes, autoregressive models learn dependencies solely between observed variables. That process can require complex function approx- imators that integrate long-range dependencies (Oord et al. 2016 Child et al.) 2019). In contrast, VAEs and invertible flows incorporate latent variables and can thus, in principle, learn a simpler model that mirrors how images are actually generated. Despite this theoretical advantage, on the landmark ImageNet density estimation benchmark, the Gated PixelCNN still achieves higher likeli- hoods than all flows and VAEs, corresponding to a better fit with the data.
Is the autoregressive modeling assumption actually a better inductive bias for images, or can VAEs, sufficiently improved, outperform autoregressive models? The answer has significant practical stakes, because large, compute-intensive autoregressive models (Strubell et al. 2019) are increas- ingly used for a variety of applications (Oord et al. 2016 Brown et al. 2020; Dhariwal et al. 2020) Chen et al., 2020). Unlike autoregressive models, latent variable models only need to learn depen- dencies between latent and observed variables; such models can not only support faster synthesis and higher-dimensional data, but may also do so using smaller, less powerful architectures.
We start this work with a simple but (to the best of our knowledge) unstated observation: hierar- chical VAFs should be able to at least match autoregressive models, because autoregressive models are equivalent to VAEs with a powerful prior and restricted approximate posterior (which merely outputs observed variables). In the worst case, VAEs should be able replicate the functionality of autoregressive models; in the best case, they should be able to learn better latent representations, possibly with much fewer layers, if such representations exist.
We formalize this observation in Section 3 layers than previous work has explored. Then we experimentally test on competitive natural image benchmarks. Our contributions are the following:
showing it is only true for VAEs with more stochastic
We provide theoretical justification for why greater depth (up to the data dimension D, but also low some value K & D) could improve VAE performance (Section
We introduce an architecture capable of scaling past 70 layers, when previous work ex- plored at most 30 (Section
We verify that depth, independent of model capacity, improves log likellil,oo,d, and allows VAFs to outperform the PixeICNN on all benchmarks (Section 5.1)
Compared to the PixeICNN, we show the model also uses fewer parameters, generates samples thousands of times more quickly, and can be scaled to larger images. We show evidence these qualities may emerge from the model learning an efficient hicrarchical rep- resentation of images (Section 5.2)
We release code and models http::///gtttab..cmm//aaaaaaaa/a
2 PRELIMINARIES
We review prior work and introduce some of the basic terminology used in the field.
2

---

Published as a conference paper at ICLR 2021
Latent variables identical observed variables
Latent variables allow for parallel generation
Pe (s)
Pe (x)
4p (8)
p=(x
Input
Output
Input
Output
Figure 2: Different possible learned generative models in a VAE. Left: A hierarchical VAE can learn autoregressive model by using the deterministic identity function the autoregression the prior. Right: Learning the encoder can lead to efficient hierarchies of latent variables (black). If the bottom group of three latent variables is conditionally independent given the first, they can be generated in parallel within a single layer, potentially leading to faster sampling.
encoder, and learning
2.1 VARIATIONAL. AUTOENCODERS
Rezende al. 2014) consist of a generator p=(x))), a prior po(z), and an approximate posterior qa x2 Neural networks á and f are trained end-to-end with backpropagation and the reparameterization trick in order to maximize the evidence lower bound (ELBO):
Variational autoencoders (Kingma Welling
2014
log Pa 2 Ex-gg (x)x)) log (x( D
(1)
See Kingma & Welling (211)) for an in-depth introduction. There are many choices for what net- works are used for pe z). (0 and whether (=()) is also learned or set simple distribu-
tion.
We study VAES with independent pe (ax =) that where each observed z; is output without condi- tioning on any other zj. This ensures generation time does not increase linearly with the dimension- ality the data, and requires that these VAEs learn to incorporate the complexity of the data into rich distribution over latent variables 2. It is possible to have autoregressive (= 2²) (Gulrajani al., 2016), but generation slow for these models. They also sometimes ignore latent variables entirely, becoming equivalent to normal autoregressive models (Chen et al. (2016)).
2.2 HIERARCHICAL VARIATIONAL AUTOENCODERS
Much of the early work on VAEs incorporate fully-factorized Gaussian 40 (z|x²) (x) and po(z). This can lead to poor outcomes the latent variables required for good generation take on more com- plex distribution, greater expressivity in both distributions to use a hierarchical VAE, which has several stochastic layers of latent variables. These variables are emitted in groups 2o, 21...., NN, which are condi- tionally dependent upon each other some way. For images, latent variables are typically output in feature maps of varying resolutions, with corresponding to small number of latent variables at low resolution at the "top" the network, and nn corresponding to a larger number of latent variables at high resolution at the "bottom".
is common with independent (o(222). One of the simplest methods of gaining
One particularly elegant conditioning structure is the top-down VAE, introduced Sønderby et al. (2016). In this model, both the prior and the approximate posterior generate latent variables in the same order:
Po (z( Pe (2 (2 2o) |=xN)
(2)
( (z| = q (z | 2o. qo. (ZN x<N,
(3)
diagram of this process appears in Figure 3. A typical implementation of this model has first perform deterministic "bottom-up" pass on the data generate features, then processes the groups

---

Published as a conference paper at ICLR 2021
Battom-up Top-down path
res block
oont 3x3
lopdown block
bultmmsp))
cone 1w1
pool
unpool
363
res block
lopdown block
res block
topdown block
cone 1x1
res block
lopdown block
res block
Input
Reconsiruction
Figure 3: A diagram of our top-down VAE architecture. Residual blocks are similar to bottleneck ResNet blocks (He et al. 2016). Each convolution is preceded by the GELU nonlinearity (Hendrycks) & Gimpel 2016). (.(.) and (.).) are diagonal Gaussian distributions. z is sampled from go during training, and pe(.) when sampling. We use average pooling and nearest-neighbor upsampling for pool and unpool layers.
of latent variables from top to bottom, using feedforward networks to generate features which are shared between the approximate posterior, prior, and reconstruction network pa(x)22 We adopt this base architecture as it is simple, empirically effective, and has been postulated to resemble biological processes of perception (Dayan et al.) 1995).
WHY DEPTH MATTERS FOR HIERARCHICAL VAES
We find that hierarchical VAEs with sufficient depth can not only learn arbitrary orderings over observed variables, but also learn more effective latent variable distributions, if such distributions exist. present these results below.
Definition (N llylayer VAE). A deep hierarchical VAE with N stochastic layers, independent p(æ]2), and the top-nown factorization of the prior and approximate posterior in Equations 2
Proposition N-layer VAEs generalize autoregressive models when N is the data dimension Proposition 2. N-layer VAEs are universal approximators of N-dimensional latent densities
Proposition 1 (proof in Appendix, also visualized in Figure 2 left) leads possible explanation of why autoregressive models to date have outperformed VAEs: they are deeper, in the sense of statistical dependence. VAE must as deep as the data dimension D (3072 layers the case of 32x32 images) if the images truly require D steps generate.
Luckily, however, Proposition (proof and further technical requirements in Appendix) suggests that shorter procedures, if they exist, are also learnalle. N D is an extreme case, where the most effective latent variables z E R may simply be copies of the observed variables. But if for some to efficiently compress
K D there exist latent variables z E IR that the generator the data, Proposition states K- Kllayer VAE can learn the posterior and prior distribution over those variables.
4

---

Published as a conference paper at ICLR 2021
Such shorter generative paths could emerge in two ways. First, depicted in Figure (right), if the model discovers that certain variables are conditionally independent given others, the model can generate them in parallel inside single layer, where [0(=N =<NN, x) II go (d) N x). We hypothesize these efficient hierarchies should emerge in images, as they contain many spatially independent textures, and study this in Section 5.2 Second, the model could learn oow-dimensional representation of the data. Dai & Wipt (2019) recently showed that when a VAE is trained on data distributed K-dimensional manifold embedded in R.,, VAE will only activate K dimensions in its latent space, meaning that the VAE will require fewer layers unless the manifold dimension is D, which is unlikely to be the for images.
difficult to ascertain the lowest possible value of K for a given dataset, but it may be deeper than most hierarchical VAEs date. Images have many thousands of observed variables, but early hierarchical VAEs did not exceed layers, until MMalée et al. (2019) investigated a Gaussian VAE with 15 layers and found it displayed impressive performance along variety of measures. Kingma iet al. (2016) and Vahdat & Kautz| (2020) additionally explored networks up to 12 and 30 layers. (These additionally incorporated additional statistical dependencies in the approximate posterior through the usage of inverse autoregressive flow (Kingma et al. 2016). an alternative approach which we contrast with our approach in Section A.44). Nevertheless, given these results we hypothesize that greater depth may improve the performance of VAEs. In the next section, we introduce an architecture capable of scaling to a greater number of stochastic layers. Section 5.1 depth indeed improves performance.
we show
4 AN ARCHITECTURE FOR VERY DEEP VAES
We consider "very deep" VAE to simply be one with greater depth than has previously been explored (and do not define it to be a specific number of layers). As existing implementations of VAEs did not support many more stochastic layers than they were trained on, we reimplemented minimal VAE with the sole aim increasing the number of stochastic layers. This VAE consists only of convolutions, nonlincarities, and Gaussian stochastic layers. It does not exhibit posterior collapse even for large numbers of stochastic layers. We describe key architectural choices here and refer readers to our source code for more details.
4.1 ARCHITECTURAL COMPONENTS AND INITIALIZATION
A diagram of our network appears in Figure It resembles the ResNet VAE in Kingma al. (0166). but with bottleneck residual blocks. For each stochastic layer, the prior and posterior are diagonal Gaussian distributions, as used in prior work (Maaløe al. 2019).
As an alternative to weight normalization and data-dependent initialization (Salimans & Kingma 2016), we adopt the default PyTorch weight intialization. The exception is the final convolu- tional layer in each residual bottleneck block, which we scale by 1 where N is the depth (similar VN"
to Radford et al. )2019); Child et al. (2019): Zhang et al. (2019)). This residual scaling improves stability and performance with many layers, as we show in the Appendix (Table
Additionally, we neirest-neighhor upsampling for "unpool" layer, which when paired with our ResNet architecture, allows us to completely remove the "free bits" and KL "warming up" terms that appear in related work. As detail in the Appendix (Figure 5). when upsampling is done through transposed convolutional layer, the network may ignore layers at low resolution (for instance, 1x1 or 4x4 layers). We found no evidence of posterior collapse in any networks trained with nearest neighbor interpolation.
4.2 STABILIZING TRAINING WITH GRADIENT SKIPPING
VAEs have notorious "optimization difficulties." " which are not frequently discussed in the litera- ture but nevertheless well-known by practitioners. These manifest extremely high reconstruction or KL losses and corresponding large gradient norms (up to le15). We address this by skipping updates with gradient norm above a certain threshold, set by hyperparameter. Though high thresholds that affect fewer than 0.01% % of updates, this technique almost entirely eliminates divergence, and allows networks train smoothly. We plot the evolution of grad norms and the
select
5

---

Published as a conference paper at ICLR 2021
Loss by network with different configurations of stochastic layers on ImageNet-32 (similar trends appear on CIFAR-10). Left: Networks with equal number of layers, but with lower stochastic depth as described in Section [5.]] Increasing depth up to 48 layers still shows gains, which is farther than previous work has explored. Right: Networks with 48 layers, but distributed at different resolutions. We find higher resolutions benefit more from layers.
Table 1:
Depth
Params
Test Loss
Distribution of 48 layers
Test Loss
41M
4.30
32x32
16x16
8x8
4x4
1xl
41M
4.18
10
10
10
10
3.98
12
41M
4.06
12
12
10
8
6
3.97
24
41M
3,98
14
10
6
3,96
48
41M
3,95
16
16
10
4
2
3.95
values select in (Figure 6 An alternative approach to stabilizing networks may be the spectral regularization method introduced in Vahdat & Kautz (2020).
EXPERIMENTS
We trained very deep VAEs challenging natural image datasets. All hyperparameters for experi- ments are available in the Appendix and in our source code.
5.1 STATISTICAL DEPTH, INDEPENDENT OF CAPACITY, IMPROVES PERFORMANCE
We first tested whether greater statistical depth, independent of other factors, can result in improved performance. We trained network with 48 layers for 600k steps on ImageNet-32, grouping layers to output variables independently instead conditioning on each other. the input for the ith topdown block is z,, can make K consecutive blocks independent by setting 2(+1, .... Zi+K all equal to z;- (Normally, 24+1 = 2, + (block(x;)) ()))). This technique reduces the stochastic depth without affecting parameter count. Stochastic depth shows clear correlation with performance, even up to 48 layers, which past what previous work has explored (Table left).
We then tested our hypothesis scale. We trained networks CIFAR-10, ImageNet-32, and ImageNer-64 with greater numbers of stochastic layers, but with fewer parameters than related work (see Table D. On CIFAR-IO, we trained model with 45 stochastic layers and only 39M parameters, and found it achieved a test log-likelihood of 2.87 bits per dim (average of 4 seeds). On ImageNet- 32 and ImageNet-64, we trained networks with 78 and 75 stochastic layers and only approximately 120M parameters, and achieved likelihoods of 3,80 and 3.52.
On all tasks, these results outperform all GatedPixeCCNN/ixxcCNN autoregressive models, while using similar or fewer parameters. These results support our hypothesis that stochastic depth, as opposed to other factors, explains the gap between VAEs and autoregressive models.
models, and all non-
5.2 VERY DEEP VAES LEARN AN EFFICIENT IERRRCIIICA ORDERING
One question that emerges from the analysis in Section 3 is whether VAES need to be as deep as autoregressive models, or whether they can learn a latent hierarchy of conditionally independent variables which are able to be synthesized in parallel. We qualitatively show this is true in Figure For FFHQ-256 images, the first several layers low resolution almost wholly determine the global features of the image, even though they only account for less than 1% of the latent variables. The rest of the high-resolution variables appear to be spatially independent, meaning they can be emitted in parallel in number of layers much lower than the dimensionality of the image. This efficient hierarchical representation may underlie the VAE's ability to achieve better likeliiisso than the PixelCNN while simultaneously sampling thousands of times faster. This can be viewed as a learned parallel multiscale generation method, unlike the handcrafted approaches of Kolesnikov
& Lampert 2017) Menick & Kalchbrenner (2018); Reed et al. (2017).

---

Published as a conference paper at ICLR 2021
4x4
8x8
16x16
32x32
64x64
128x128
<0.001% %
0.03%
0.15%
1.1%
9.5%
30.1%
74%
Figure 4: Cumulative percentage latent variables at a given resolution, and reconstructions of samples on FFHQ-256. We sample latent variables from the approximate posterior until the given resolution, and sample the rest from the prior low temperature. This shows what images are likely given a subset of latent variables. Low-resolution latents comprise small fraction of the total latents, but encode significant portions of the global structure. This suggests deep VAEs learn efficient, hierarchical representations of the data.
Additionally, found that on all datasets tested, very deep VAEs used roughly 30% fewer parameters than the PixeICNN (Table 2 One possible explanation is that the learned hierarchical generation procedure involves fewer long-range dependencies, may otherwise be simpler to learn.
We found that networks in general benefited from more layers at higher resolutions (Table right). This suggests that global features may account for smaller fraction of information than local details and textures, and that it is important to have many latent variables at high resolution.
5.2.1 VERY DEEP VAES ARE EASILY SCALED HIGH DIMENSIONAL DATA
Scaling autoregressive models to higher resolutions presents several challenges. First, the sampling time and memory requirements of autoregressive models increase linearly with resolution. This scaling makes datasets like FFHQ-256 and FFHQ-1024 intractable for naive approaches. Although clever factorization techniques have been adopted for 256x256 images (Menick & Kalchbrenner, 2018), such factorizations may not be effective for alternate datasets higher-resolution images.
Our VAE, in contrast, readily scales to higher resolutions. The same network used for 32x32 images can be applied to 1024x 1024 images by introducing greater number of upsampling layers through- out the network. We found we could train an equal number of steps (1.5M) using a similar number of training resources (32 GPUs for 2.5 weeks) both 32x32 and 1024x1022 1024 images with few hy- perparameter changes (see Appendix for hyperparameters). Samples from both models (displayed Appendix) require a single forward pass of the model generate, with only minor differences in runtime. An autoregressive model, on the other hand, would require a thousand times more network evaluations to sample 1024x 1024 images and likely require custom training procedure.
6 RELATED WORK AND DISCUSSION
Our work is inspired by previous and concurrent work in hierarchical VAEs (Sonderby et al. 2016 Maaløe et al. 2019 Vahdat & Kautz, 2020). Relative these works, we provide some justification for why deeper networks may perform better, introduce new architecture, and empirically demon- strate gains in log-likelihood Many aspects of prior work are complementary with ours and could be combined. Maaløe et al. (2019), for instance, incorporates a "bottom-up" stochastic path that doubles the depth of the approximate posterior, and Vahdat & Kautz| (2020) introduces a number of powerful architecture components and improved training techniques. We seek here not in- troduce a significantly better method than these alternatives, but to demonstrate that depth is a key overlooked factor in most prior approaches to VAEs.

---

Published as a conference paper at ICLR 2021
Table 2: Our main results on standard benchmark datasets. Very deep VAEs outperform PixeICNN-based autoregressive models with fewer parameters while maintaining fast sampling. "Depth" refers the number of stochastic layers for hierarchical VAEs (although BIVA and IAF- based networks have additional statistical dependencies). Sampling refers to the number of network evaluations per sample, and D designates the dimensionality of the data. An asterisk (") denotes our estimate of parameters. Samples for ImageNet and CIFAR-10 10 are in the Appendix.
Model type
Params
Depth
Sampling
NLL
CIFAR-10
PixcICNN+ Sallimans et al.
2017
AR
53M*
DR DD
2.92
PixeISNAIL (Chnn et al. 2017)
AR
2.85
Sparse Transformer (Child et al.
2019)
AR
59M
2.80
VLAE Chen al.
2016
VAE
CILIEL
S 2.95
(Kingma el al
2016
VAE
12
3.11
Flow++ (Ho
2019
Flow
31M
3.08
BIVA Maalóe et al. 2019
VAE
103M
15
3,08
NVAE Vahdat & Kautz
2020
VAE
131M
30
2.91
I
Very Deep VAE (ours)
VAE
39M
45
2.87
ImageNet-322
Gated PixelCNN
AR
177M
10
D
3.83
D
Image Transformer Parmar et al.
2018
AR
DIRI
3.77
BIVA
VAE
103M
15
3.96
NVAE
VAE
268M
28
< 3.92
Flow++
Flow
169M
S 3.86
Very Deep VAE (ours)
VAE
119M
78
3.80
ImageNet-64
Gated PixeICNN
AR
177M
3.57
SPN (Menick & Kalchbrenner 2018
AR
150M
3.52
Sparse Transtormer
AR
152M
3,44
Glow (Kingma Dhariwal 2018)
Flow
3.81
Flow++
Flow
73M
3.669
I
Very Deep VAE (ours)
VAE
125M
75
$ 3.52
FFHQ-256 (5 bit)
NVAE
VAE
36
0.68
Very Deep VAE (ours)
VAE
115M
62
< 0.61
F0H (8 bit) Very Deep VAE (ours)
VAE
115M
72
2.42
Diffusion models can be seen deep VAEs that, like autoregressive models, have specific analyti- cal posterior. Ho et al. (2020) showed that such models achieve impressive sample quality with great depth, which is in line with our observations that greater depth is helpful for VAEs. One benefit of the VAES we outline this work over diffusion models that our VAEs generate samples with single network evaluation, whereas diffusion models currently require large number of network evaluations per sample.
Inverse autoregressive flows (IAF) also closely related, and discuss the differences with hierarchical models in Section A.4 The work of Zhao et al. (2017) may also appear to contradict findings, and we discuss that work in Section A.5
8

---

Published as a conference paper at ICLR 2021
7 CONCLUSION
We argue deeper VAES should perform better, introduce a deeper architecture, and show it outper- forms all PixelCNN- autoregressive models in likelihood while being more efficient. We hope this encourages work in further improving VAEs and latent variable models.
ACKNOWLEDGMENTS
We thank Aditya Ramesh, Pranav Shyam, Johannes Otterbach, Heewoo Jun, Mark Chen, Prafulla Dhariwal, Alec Radford, Yura Burda, Bowen Baker, Raul Puri, and Ilya Sutskever for helpful dis- cussions. We also thank the anonymous reviewers for helping improve our work.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. Proceedings of the 37th International Conference Machine Learning, 2020.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731, 2016.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. arXiv preprint arXiv:1712.09763, 2017.
Rewon Child, Scott Gray, Alee Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 19X,:19904.1009, 2019.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv: 1903:09903..5789 2019.
Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine. Neural computation, 7(5):889-9044 1995.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and llya Sutskever. Jukebox: generative model for music. arXiv preprint arXiv:2005.00341,, 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 180500160,.0880,, 2016.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05001 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error arXiv:1606.0844 08415, 2016.
linear units
(gelus).
arXiv preprint
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint arXiv: /90.. 2019.

---

Published as a conference paper at ICLR 2021
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint wwxi::200.11239, 2020.
Chin-Wei Huang, Ahmed Touati, Laurent Dinh, Michal Drozdzal, Mohammad Havaei, Laurent Charlin, and Aaron Courville. Learnable explicit density for continuous latent space and varia- tional inference. arXiv preprint urXiv:1110.0222: 2017.
Chin-Wa Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. arXiv preprint arXiv: (oKKv:1804..000 1804.00779, 2018.
Diederik P Kingma and Max Welling. Stochastic gradient vb and the variational auto-encoder. In Second International Conference Learning Representations, ICLR, volume 19, 2014.
Diederik P Kingma and Max Welling. An introduction to variational autoencoders. arXiv preprint arXiv:1906. 02691, 2019.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751. 2016.
Alexander Kolesnikov and Christoph H Lampert. Pixelenn models with auxiliary variables for natural image modeling. In International Conference Machine Learning, pp. 1905-1914.
PMLR, 2017.
Lars Maalée, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In Advances neural information processing systems, pp. 6548-6558, 2019.
Jacob Mcnick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv: arXiv:/8/2.0/60 18/2.0/688. 2018.
Aaron van den Oord, Sander Dieleman, Heiga Zen Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: (et:::660003999 2016.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint 1921v2191 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv:1802.0575: 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner. Sergio Gómez Colmenarejo, Ziyu Wang, Dan Belov, and Nando De Freitas, Parallel multiscale autoregressive density estimation. arXiv preprint 173iv:1703.03664, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: /001:::/0014082. 2014.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization to accelerate training of deep neural networks. In Advances in neural information processing systems, pp. 901 909, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05511 2017.
10

---

Published as a conference paper at ICLR 2021
Table 3: Effects of scaling residual initialization on very deep VAEs. We trained networks with varying depths for 80k iterations. Scaling the last layer in the residual block by vv results in higher losses for shallower networks, but lower losses and greater stability for deeper networks. The number of updates which are skipped because the gradient norm would destabilize the network is significantly reduced with scaling.
Depth
Without scaling
With scaling
Loss
Skipped Updates
Loss
Skipped Updates
15
2.50
13
2.51
30
2.36
41
2.38
45
2.31
48
2.30
60
2.30
76
2.29
75
Diverged
2.28
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-3746, 2016.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint (rXih::906.02244 2019.
Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. In Advances in Neural Information Processing Systems, pp. 2175-2183, 2013.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelcnn decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv: /901, 093211, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. arXiv preprint arXiv: 700.00836, 2017.
A APPENDIX
A.1 ABLATIONS OF ARCHITECTURAL COMPONENTS
First, we visualize data that suggests upsampling layers and residual connections have an impact on posterior collapse (Figure 5p. Architectural differences may explain why our VAEs do not need "free bits" or KL warmups to avoid posterior collapse.
In Table we show residual initialization leads to smoother and better training of very deep VAEs. Without residual initialization, very deep VAEs encounter a high number of unstable updates and have higher losses.
In Figure 6, we show the max gradient norms experienced throughout training, and show that our skipping criterion avoids small number of updates that would destabilize the network.
A.2 PROPOSITION 1: N-LAYER VAES GENERALIZE AUTOREGRESSIVE MODELS WHEN IS THE DATA DIMENSION
Proposition shows that an autoregressive model with an arbitrary ordering over observed variables in æ E RNN is equivalent to an N N-layer VAE with an approximate posterior that simply outputs the observed variables in the given order, and a generator that performs the identity function (see Figure 2).
11

---

Published as a conference paper at ICLR 2021
Average rate (KL) per layer by network type at 80k updates
16
ResNeet (Nearest Neighbors), 2.
1.4
ResNet (Convolution), ELB0-3.30
FFN, ELB0=3.42
1.2
Curuletive rate (KL) (bpd)
1.0
06
02
10
40 50
70
Layer
Figure 5: Relationship between architecture and posterior collapse. We visualize the cumulative KL divergence (or "rate", in bits per dimension) for several different architectures across 73 layer network on ImageNet-32. When residual connections are removed from the "res block" in the top- down path (Figure 3). the model encodes no information in the first 45 layers of the network and the loss is highest ("FFN"). When learned convolutional upsampler is used as the "unpool" layer, the first 13 layers of the network encode no information. When nearest-neighbor upsampling is used, the first layers all encode information, and the loss is the lowest.
Without loss of generality, we simplify notation by assuming each vector-valued latent variable x; only has one element, which write 2; E IR. We assume prior and approximate posterior distribution following [anaatio[aan]]
Proof. Let g(2; = zu =<),
1, and p(z; 21/²) = 1. Then p(z[x)
((
which well-known to imply equality in the evidence lower bound (ELBO) of Eq. D logg(=)>)) = logp(x)>) = O, the ELBO becomes (og) = log (og))(2> = Σ (<)
Since
En log (x(x) which is equivalent an autoregressive model the observed vari- ables.
A.3 PROPOSITION 2: N-LAYER VAES ARE UNIVERSAL. APPROXIMATOR N-DIMENSIONAL LATENT DENSITIES
Proposition 2 shows that hierarchical VAEs learn depthwise autoregressive flows, and under certain conditions (described in Huang et al. (2017) can express any density over latent variables of N dimensions, given enough capacity.
Proof. We omit full proof. and refer readers to Huang et al. (0077); Papamakarios et al. 20199) where universality is established for autoregressive flows. Here we only note that the prior and ap- proximate posterior in an N-layer VAE are autoregressive flows: Let (z(2) be the prior distribution. po(z) can written using the reparameterization trick as deterministic function of noise c drawn from known base density Px= (z(z) (c) det @f(c.0) where is neural network that im- De
plements the factorization in Eq. 2 Since f is autoregressive and its Jacobian is lower triangular, Po(2) can approximate any p(z) that fits the criteria in Huang et al. (2017). The same logic applies to (z æ:) and p(z[2=) It should be noted that this result depends on being able to implement the inverse CDF an arbitrary probability density, and so using Gaussian distributions will restrict the densities the VAB can express in practice. This is limitation of our architecture that never- theless adopt since we hypothesize depth, not the elementwise density, is the more important factor. More discussion this subject, and options for removing this restriction, are described in Huang eet al. (2017) and Huang et al. (2018), and we defer studying more expressive elementwise densities future work.
12

---

Published as a conference paper at ICLR 2021
200880
40008
100000
100000 D20600 128000
Figure 6: Effect of gradient skipping. We plot the gradient encountered per 500 updates for our best models across datasets. The dashed black line indicates the "skip threshold", or value above which the update is skipped. We choose a high threshold that affects fewer than 0.01 percent of training updates. Without this skip heuristic, networks will diverge when extreme updates are encountered. These updates can have norm as high as 1e15.
A.4 A NOTE INVERSE AUTOREGRESSIVE FLOW
Inverse autoregressive flows (IAF, Kingma et al. (2016) and are similar to very deep VAEs in that they are universal approximators of posterior distributions in VAEs, even with single layer and sufficiently expressive univariate density (Huang et 2018).
There are several practical differences between IAFs and deep hierarchical VAEs, however, which can result in qualitatively very different behavior. First, the masked autoregressive components in IAF build statistical dependencies spatially, whereas very deep hierarchical VAE builds depen- dencies depthwise, and these inductive biases may better suit different domains. Additionally, IAFs spend an equal amount of computation and parameters on each variable. In contrast, deep VAE can specify a structure, like a hierarchy of global-to- and modeling capacities for each stage. For images, these differences may result in qualitatively different behavior, and it is not clear whether a single layer IAF can readily learn the sort of rich hierarchical decomposition of images that appear with very deep VAEs.
variables, which have different computational
13

---

Published as a conference paper at ICLR 2021
Figure 7: Non-cherrypicked, temperature 1.0 samples on FFHQ-256. Cover images were each cherrypicked from a batch of 16 (unadjusted temperature) samples. Here we show a random batch of 16 images for comparison.
Nevertheless, the two techniques are complementary IAF was introduced in a deep hierarchical VAE (Kingma et al. 2016), in fact, and it is likely that introducing IAF into our architecture (as in Vahdat Kautz (0200) would improve performance.
A.5 A NOTE ON LEARNING HIBRARCHICAL FEATURES
The work Zhao et al. (217)) may appear to contradict our work, by suggesting that additional layers in hierarchical VAEs do not lead to additional expressivity, based off their finding that Gibbs sampling from the last stochastic layer is sufficient to recover the data. For high dimensional data like images, however, the last stochastic layer may have many thousands of variables, and Gibbs sampling may take unacceptably long to converge. A hierarchy of latent variables as in our model allows efficient and tractable sampling from this distribution. Additionally, assumptions regarding global maximization of the ELBO may not apply in practice. Nevertheless, we think further clarify- ing these contradictory statements would be useful future work.
A.6 BROADER IMPACT
Broadly speaking, any generative model will reflect the biases of the datasets they trained deployed without careful consideration, generative models (including but not limited to VAEs) trained on research datasets like ImageNet, CIFAR- 10, and FFHQ may inadvertently cause harm by propagating or otherwise reinforcing harmful biases the dataset. Further work is required to improve and debias research benchmark datasets mitigate this source of negative impact.
14

---

Published as a conference paper at ICLR 2021
Figure 8: Non-cherrypicked, temperature 0.85 samples on FFHQ-256. Lower temperature sam- ples result greater regularity in images.
Some VAEs are distinguished from other generative models by their fast synthesis of new data amples. Generative models with fast synthesis can allow for realtime synthesis of high dimensional data, such music, speech, and video. These models could be used augment human creativity and lead to number of helpful applications in real-time media applications. Such models could also be used for compression, which could assist in delivering content bandwiddt-coonstraine regions of the world. They can also be used for spreading disinformation, generally making it less possible distinguish real from generated data. An additional potential harm is that fast, high quality syn- thesis of data could end up economically displacing individuals who rely upon creative work, such as musicians, visual artists, and more.
VAEs also are distinguished by their usage of latent variables. Generative models with useful latent variables could have positive impacts in scientific domains, where density estimation could lead to novel insights about chemical, physical, or biological data. Latent variable representations of data could also be helpful in efforts to debias, interpret, or otherwise increase understandibility of models and their representations.
15

---

Published as a conference paper at ICLR 2021
Figure 9: Non-cherrypicked, temperature 0.60 samples on FFHQ-256. We visualize temperature 0.60 samples for comparison with Vahdat & Kautz (2020)
Table Key hyperparameters for experiments. We detail here the main hyperparameters used in
training. FFHQ-1024 has reduced hidden size for higher resolutions; code for details.
Parameter
CIFAR-10
ImageNet-32
ImageNet-64
FFHQ-256
FFHQ-1024
Num layers
45
78
75
62
72
Hidden size
384
512
512
512
Varies
Bottleneck size
96
128
128
128
Varies
Latent dim per layer
16
16
16
16
16
Batch size
32
256
128
32
32
Learning rate
0.0002
0.00015
0.00015
0.00015
0.00007
Optimizer
Adam
Adam
Adam
Adam
Adam
Skip threshold
400
300
380
180
500
Weight Decay
0.01
0.0
0.0
0.0
0.0
EMA rate
0.0002
0.00015
0.00015
0.00015
0.00015
Training iterations
1.1M
1.7M
1.6M
1.7M
1.7M
GPUs
x V100
32 x V100
32 V100
32 x V100
32 x V100
Training time
6 days
2.5 weeks
2.5 weeks
2.5 weeks
2.5 weeks
Parameters
39M
119M
125M
115M
115M
16

---

Published as a conference paper at ICLR 2021
Figure 10: ImageNet-32 (left) and ImageNet-66 (right) reconstructions and samples. Recon- structions of validation images from various stages in the latent hierarchy (top), and unconditional samples from the model at temperature 1.0 (bottom).
Figure 11: FFHQ-1024 samples. These are generated with reduced temperature (top) and tempera- ture 1.0 (bottom). The model we train has similar capacity smaller ones we use 32x32, 64x64, and 256x256 images, and so fails to capture the intricacies of this more complex distribution well. larger model, trained for longer, may achieve better sample quality.
17


Reviews:
[{"review_id": "qnHZKFQl651", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\nThis paper shows that deep hierarchical VAEs can outperform state-of-the-art autoregressive models on images. The authors first argue that autoregressive models are special cases of hierarchical VAEs and that hierarchical VAEs are universal approximators. They introduce a simple top-down (LVAE) architecture that scales past 70 layers. Furthermore, the model can be trained without using freebits or KL annealing -- although additional tricks are required (gradient skipping and prior warmup). They demonstrate that likelihood performance is correlated with depth and report state-of-the-art performances on multiple image datasets.\n\n2. a Strong Points\n- the contribution is clear: the idea the hierarchical models can outperform autoregressive models is clearly stated and supported by a short theoretical section and relevant experiments (likelihood vs. depth + benchmark)\n- the proposed model is simpler than existing methods and is trained using a simpler objective (no freebits/KL-warmup):\n- the authors demonstrate state-of-the-art likelihood on multiple datasets: the Very Deep VAE indeed outperforms large autoregressive models\n- the authors demonstrate that the method is scalable: proof of concept on FFHQ 256 and 1024.\n\n2. b Weak Points\n- the part on parallel generation is unclear: are you referring to each of the latents for one layer being sampled independently? (i.e. $q(\\mathbf{z}^l | \\mathbf{z}^{l-1}, \\mathbf{x}) = \\prod_d q(z^l_d | \\mathbf{z}^{l-1}, \\mathbf{x})$)\n- section 5.2.1 is unclear: \"unlike autoregressive models, this scaling does not require greater training resources\". In my opinion, using larger images does requires greater training resources because of the increased image definition.\n- Although the code is provided, the experimental protocol is not well described in the paper (learning rate, number of epochs, hidden size, ...)\n\n3. Recommendation\nI recommend accepting this paper.\n\n4. Recommendation Arguments \nThis paper presents a simple, yet solid, story. The empirical results strongly support that \"very deep VAEs can outperform autoregressive models on images\" and the authors introduced a minimal architecture allowing to do so.\n\n5. Questions to the Author\nPlease elaborate on the weak points.\n\n6. Feedback \nFigure 2: the sentence \"latent variables are observed variables\" is odd. Latent variables are unobserved by definition. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pwW9loHjo-f", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**GENERAL**\nThe paper claims that high quality of generated samples and SOTA bpds are achievable by VAEs if the model is deep enough (deep in terms of the number of stochastic layers). The authors explain the architecture that resemblances the U-net architecture, and explain its building blocks. Interestingly, they are able to learn VAEs with up to 78 stochastic layers, and achieve SOTA bpds on CIFAR-10, ImageNet-32, ImageNet-64, FFHQ-256 (5-bit), and setting a great result on FFHQ-1024 (8bit).\n\n**Strengths:**\nS1: The authors are capable of training VAEs with over 45 stochastic layers (up to 78).\n\nS2: The proposed architecture does not contain any extra \"tricks\", it is relatively simple. This is a great plus for the paper!\n\nS3: The presented theorems are interesting additions to this rather practical paper.\n\nS4 The experiments are well performed and the ablation studies are insightful.\n\nS5: Generated images are of very high quality! Even a reflection in a glass of a generted lady is better than samples of CIFAR10 in many papers.\n\n**Deficiencies:**\nD1: The prior is not explained in the paper! Without this information, it is hard to properly understand what kind of problems occur during training q(z|x) and p(z).\n\n**Remarks:**\nR1: The proposed heuristic method for training q(z|x) reminds of the following paper:\nHe, J., Spokoyny, D., Neubig, G., & Berg-Kirkpatrick, T. (2019). Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv:1901.05534.\nIt would be interesting to compare at the conceptual level both heuristics.\n\nR2: It seems that the authors do not use BatchNorm. Is it correct? This would be also interesting to discuss, because in the following paper:\nVahdat, A., & Kautz, J. (2020). Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898.\nthe BatchNorm is indicated as an important component for achieving a deep VAE.\n\n**Questions:**\nQ1: What kind of prior was used in this paper?\n\nQ2: Is BatchNorm indeed irrelevant for VAEs?", "extra_scores": {}, "extra_text": {}}, {"review_id": "DDVq9bkjgRp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------------\n\nThis paper provides evidence that \"very deep\" hierarchical VAEs can outperform autoregressive and flow-based models albeit using less parameters on image density estimation tasks.\n\nIt seems natural to think that a hierarchy of latent variables progressively compressing information would be useful for image modelling, with top latent variables capturing more abstract/general features and bottom latent variables capturing lower-level details. However, recent success of flow based and autoregressive based models such as PixelCNN seemed to invalidate the need of such hierarchy of latent variables and to \"compress\" pixel-level information. Here, the authors show that a simple hierarchical VAE architecture inspired by previously proposed ones can outperform autoregressive models if it's made sufficiently \"deep\". I think this is an important contribution. With respect to previous work, this work relates to the concurrently proposed \"Nouveau VAE\" but obtains better results with less parameters and considerably less involved customization of the architecture.  The authors report impressive results on multiple datasets generally using less parameters than competing models. Additionally, sampling from very deep VAEs is considerably cheaper than in autoregressive models.\n\nThe authors also attempt at showing that learnt latent variables implement a hierarchy of information which could be useful to have in general. This point is a bit weak and not well demonstrated in the paper.\n\nPros\n------\n\n- Strong results on multiple tasks with a method that was previously thought to have plateaued in performance\n\nCons\n-------\n\n- Originality / novelty is a bit weak\n- Clarity can be improved\n\n\nDetailed Remarks\n-------------------------\n\n- Figure 4 is not totally convincing as high-level features in the first image are not always maintained in higher-resolution realizations (sample in the first row seem to have glasses then they disappear?). Could you include more samples to back this claim? Do you think of a way of understanding whether high-level variables maintain general info (maybe by probing the posterior samples for some downstream attribute ?)\n\n- I find it hard to understand what is going on in Table 1 (left). In Section 5.1, referring to Table 1, what do you mean by \"grouping layers to output variables independently instead of conditioning on each other\" ? In Table 1, what do you mean by \"masking\" in the sentence \"with masking introduced such that the effective stochastic depth is lower\" ? I cannot find any other references to masking. \n\n- In Figure 3, what do you use as the pooling operation? (2, 2) max pooling ?\n\n- An ablation study of the proposed modifications to the architecture and training tricks would be useful, e.g. what's the most single important modification that makes the model work ? Is it the neighbour upsampling ? Is it the 1/\\sqrt(N) init of the last layer ? Is it the skipping gradient trick ?\n\n- How is Figure 4 obtained ? When you say \"The rest of the high-resolution variables can be output in parallel, largely independent of each other\", are you referring to the fact that you sample from the top 1x1 layer and then sample independently the other zs from the learnt prior e.g. p(z_4x4) ... p(z_64x64) without ancestral sampling ?\n\n- Section A.1: \"Without loss of generality, we simplify notation by assuming each vector-valued latent variable zi\nonly has one element, which we write as zi\", do you mean each latent variable is \\in R ? It'd be good to mention that\nyou assume an architecture with an auto-regressive learnable prior p_\\theta(z_i | z_<i) or refer to Eq. 2.\n\n- Section A.2: I am not sure you need this sentence: \"Without loss of generality, we simplify notation by assuming each vector-valued latent variable zi only has one element, which we write as zi\", as it seems copied from A.1.\n\nGrammar\n-------------\n- Section 5.2: \"an learned\" -> \"a learned\"\n- Section 4.1: \"the the\" -> \"the\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "5frSzxKhDB8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**summary** \nthe paper puts forward an idea that deep-enough VAE should perform at least as well as autoregressive models. Authors explore this in the context of image generation, and construct VAE model that is a generalisation of typical autoregressive architectures. They use several tricks to ensure stable training of very deep VAEs and show that final performance exceeds all autoregressive models. This experimentally supports their claim that very deep VAEs encompass autoregressive models.\n\n**pros**\nThe idea of perceiving VAE architectures as strictly more powerful and potentially efficient is very appealing. Given the recent work on improving deep VAE training(like Vahdat & Kautz (2020)) this paper takes another step in this direction by effectively, as it seems from the text, removing the depth limitation for training such VAEs. The tricks used to stabilise training are pretty ad hoc, but their effectiveness, showed experimentally, is important in advancing the field.\n\n\n**cons**\n* The main criticism I have is around ablation studies that justify the proposed architecture choices and training stabilisation tricks, as well as comparison to other tricks in the literature (e.g. Vahdat & Kautz (2020)). Of course the positive result speaks for itself, but the paper would be even more convincing with some details on the exploration that led to the final model.\n\n\n\n**questions**\n* it would be good to clarify in the text how exactly sampled latent variables from lower layers are decoded into the images to produce Fig. 4: is the idea to pass those latents down the top-bottom path and just not add new latents in the node \"+\" within the topdown block?\n* In Section 5.2.1, it is unclear why models with 32x32 and 1024x1024 resolutions have equal number of parameters: is this because ResNet blocks used at different resolutions share parameters?\n* Did the authors experiment with methods of slowing down the training of the prior, other then stopping it for the first half of training? It seems that exponentially averaging prior parameters might be another way of doing it, although the exponent will become another hyperparameter.\n\n**comments**\n* Further investigating the relation between using NN interpolation in upsampling and having active latents in all layers would be very useful. \n* I particularly enjoyed the perceptional shift that the paper advocates for, i.e. that VAE and autoregressive models are not competing approaches, but rather VAE is a more general one and it encompasses the latter.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper posits that VAEs, if made sufficiently deep, are able to implement autoregressive models, and could possibly outperform them. Experimentally, the authors attempt make VAEs sufficiently deep so that they are able to outperform autoregressive models on image generation. The authors use a variety of tricks to scale the depth of the model to up to 78 stochastic layers, and achieve SOTA, or near-SOTA NLLs on a number of datasets. Furthermore, in comparison to other models (in particular the recently proposed Nouveau VAE), the models achieve these scores using far fewer parameters.

Although the tricks are a bit ad-hoc and the novelty is a bit weak, the experimental results are quite strong and would be of interest to anyone working on VAE research. Moreover, one of the weakness of the paper, a lack of ablations, was addressed during the rebuttal. All reviewers believed that the paper should be accepted, and I see nothing in the paper or the reviews to suggest otherwise.
Decision: Accept (Spotlight)