Mind Your Bits and Errors: Prioritizing the Bits that Matter in Variational Autoencoders
Authors: Rui Shu
Stefano Ermon
Venue: ICLR 2022 Submitted (2022)
Keywords: deep generative models
variational autoencoders

Abstract: Good likelihoods do not imply great sample quality. However, the precise manner in which models trained to achieve good likelihoods fail at sample quality remains poorly understood. In this work, we consider the task of image generative modeling with variational autoencoders and posit that the nature of high-dimensional image data distributions poses an intrinsic challenge. In particular, much of the entropy in these natural image distributions is attributable to visually imperceptible information. This signal dominates the training objective, giving models an easy way to achieve competitive likelihoods without successful modeling of the visually perceptible bits. Based on this hypothesis, we decompose the task of generative modeling explicitly into two steps: we first prioritize the modeling of visually perceptible information to achieve good sample quality, and then subsequently model the imperceptible information---the bulk of the likelihood signal---to achieve good likelihoods. Our work highlights the well-known adage that "not all bits are created equal" and demonstrates that this property can and should be exploited in the design of variational autoencoders.

Under review as a conference paper at ICLR 2021
MIND YOUR BITS AND ERRORS: PRIORITIZING THE BITS THAT MATTER IN VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Good likelihoods do not imply great sample quality. However, the precise manner in which models trained to achieve good likelihoods fail at sample quality remains poorly understood. In this work, consider the task of image generative mod- eling with variational autoencoders and posit that the nature of high-dimensional image data distributions poses an intrinsic challenge. entropy in these natural image distributions is attributable to visually impercepti- ble information. This signal dominates the training objective, giving models an easy way to achieve competitive likelihoods without successful modeling of the visually perceptible bits. Based on this hypothesis, we decompose the task generative modeling explicitly into two steps: we first prioritize the modeling of visually perceptible information to achieve good sample quality, and then subse- quently model the imperceptible information the bulk of the likelihood signal- to achieve good likelihoods. Our work highlights the well-known adage that "not all bits are created equal" and demonstrates that this property and should be exploited in the design of variational autoencoders.
particular, much of the
INTRODUCTION
The task of generative modeling of high-dimensional image data has inspired the development of many successful deep generative models such as autoregressive models (Uria et al., 2016; Oord et al., 2016b), flow-based models (Dinh et al., 2014; 2016). generative adversarial networks (Goodfellow et 201,4) 2014), variational autoencoders (Kingma& & Welling, 2013; Rezende et al., 2014), and di diffusion models (Ho et al., 2020; Song & Ermon, 2019). These models have since been applied to many other modeling tasks such as speech synthesis (Oord et al., 2016a; Donahue et al., 2018), reinforcement learning (Zhang et al., 2019; Levine et al.. 2019), and scene-understanding (Eslami al., 2018). In this work, we focus on the application VAEs to image modeling. Despite the success of variational autoencoders (VAEs) in numerous fields (Akuzawa et al., 2018: Hsu et al., 2017: Gómez-Bombaree et al., 2018; Sultan et al., 2018; Van Hoof et al., 2016), the application of VAES to image modeling has been lukewarm best-plagued by optimization issues (Rezende & Viola, 2018; Child, 2020; Vahdat & Kautz, 2020) and models achieving good Evidence Lower Bounds (ELBOs) but poor sample quality. Our work examines the phenomenon that good ELBOs do not impy great sample quality in VAEs (Theis et al., 2015). We argue that the primary cause of this issue stems from the model being overwhelmed by the vast volume of visually- -ipplly-impereppibble information contained in natural image distributions. Our contributions are thus as follows.
We analyze the rate (the amount of information encoded in the latent space) VAEs trained with different rate-distortion trade-offs and show that low-rate models can achieve percep- tually high-quality reconstructions and sampling. In contrast, the standard ELBO objective favors high-rate models with good reconstructions but poor sampling quality.
2. However, low-rate models have much worse ELBOs due to poor modeling of the visu- ally imperceptible information. To overcome this issue, we propose two-stage training process that trains a secondary high-rate model on top of the low-rate model. Since the sec- ondary model is restricted to modeling visually imperceptible information, it can improve the ELBO significantly with minimal impact on the sample quality achieved by the initial low-rate model.

---

Under review as a conference paper at ICLR 2021
2
THE CHALLENGE OF TRAINING VARIATIONAL AUTOENCODERS
Standard WAE
Standard VAE
Standard VAE
Distortion-lergeted VAE
105
Distortion-Trgeteed VAE
Distortion-Targeted VAE
Mean- Squared Emror
Negative ELBO
Gredient: Norm
10
10a
10
10
0.4
0.6
1.0
heeraiions
Figure 1: Mean-squared error, BLBO, and gradient norm for the standard VAE with trainable vari- ance versus a VAE targeted achieve the same distortion. The former exhibits unstable training.
2
2
(a) Standard VAE
(b) Distortion- Targeted VAE
Distortion-Targeted
Figure 2: Samples from a standard VAE with trainable variance versus VAE targeted achieve the same distortion. Both have poor sample quality despite achieving good ELBOs (Figure I).
In this section, we highlight the challenges of applying variational autoencoders with Gaussian ob- servation models continuous data. We seek train model pe (2r, with observation æ and latents 70:7. We use a hierarchical prior po (=):7) = II. Po(2e =<x) and a Gaussian observation model Pe (x Zo:r). As many practitioners may be aware and demonstrated in Figures and 2, training Gaussian observation VAEs with a trainable variance presents key challenges.
Optimization difficulty. As past works have noted, training variational autoencoders (VAE) is challenging (Rezende & Viola, 2018; Vahdat & Kautz, 2020; Child, 2020). It seems, however, that VAEs with canaaale-vaiinne Gaussian observation models exacerbates the issue. Even with modeling heuristics such as gradient skipping and pregision-weighted merging that are intended to ease the optimization landscape (Child, 2020: Sønderby et al., 2016), we observe that optimization easily becomes unstable (Figure 1). To overcome this issue, we instead train a distortion-targetee VAE more more precisely, we use a mean- squared error reconstruction loss but tune the reconstruction coefficient to achieve the same mean-squared error as the standard VAE.
Good ELBOs but poor sample quality. In Figure 1, we show that both the standard VAE and distortion-targetel VAE achieve small distortion and as shall demonstrate in subsequent sections good Evidence Lower Bound (ELBO). However, this does not translate to good sample quality (Figure 2).
Since distortion distortion-targeting appears to largely resolve the optimization issue, the primary goal of our work is to explain- and ultimately fix-why VAEs achieve good ELBOS but poor sample quality.
2

---

Under review as a conference paper at ICLR 2021
3 PERCEPTUALLY IMPORTANT INFORMATION IN VAES
3.1
MEAN SAMPLING IN VAES
41
8
0
3
30
88
2A
8
16
0
6
7
5
664
9
5
5
3
7
3
5
9
2
42
61
24
0
5
3
6
6
18
1
2
34
360
5
80
6
8
3
2
6 5
4
84
5
4
6
3
6 7
23
51
3
8
5
8
58
8
(a) Standard Sampling
(b) Mean Sampling
Figure 3: Distortion-targeted VAE with non-negligible distortion target y 16, trained SVHN.
Standard sampling from po (z) versus mean sampling from pa Zo:r) Standard sampling with a
Gaussian observation model is noisier since essentially adds Gaussian noise to the mean samples.
A common practice for sampling from variational autoencoders involve sampling the latent code zo:r followed by decoding the mean of po 20:T), as described by
20:T po (20:2)
(1)
u(Zo:r)) = pe [x].
(2)
In other words, letting Zur denote the random variable distributed according to po (20:T) prac- titioners often report the sampling from μ (Z.))) rather than actually sampling æ po( especially when the choice of the decoder makes μ(z0:T) easy to calculate, such as in the case when po(x 20:T) is Gaussian observation model. The design choice to sample the mean μ Zo.r) instead of the full generative process pa (x) is of technical importance, since the implicitly defined distribution u(Zo:r) does not necessarily have a well-defined density. This is because μi u(Zor)) is not guaranteed to cover a non-zero measure in X.
Crucially, the benefit of sampling from u(Zorr) instead of Po (x: comes down to how choose to handle the information not encoded by Zu:r. Since the goal of po (x z0:r) is to model the residual information in the data distribution lost during the lossy encoding via (o(2o: Iz), the mean sample yu( (Zorr) chooses to handle this information by averaging over it. In contrast, po(x Z0:T) attempts explicitly model this information. In the case where (1 zo:T) a Gaussian observation model, the variational autoencoder is essentially attempting to crudely model the residual information via Gaussian distribution. Figure 3. we compare the resulting images from sampling po(x) versus Iu (Z.rr). Whereas the former yields noisy image, the latter averages over the noise and yields a blurry image. In cases where we are willing to accept the information loss, mean sampling offers a more visually-appealing alternative to sampling from Ps (z).. We commit to using mean sampling in all subsequent figures.
3.2 How MUCH INFORMATION ACTUALLY MATTERS PERCEPTUALLY?
In this section, we scrutinize the behavior of the mean decoding function sa(-). In particular, we are interested in bounding the mutual information 1(X: Z) necessary for the variational autoencoder to achieve perceptually high-quality reconstructions, Recalling that (= := Epa(s 2o [x]], we consider the objective
minimize Epacca (x) [DLL (gs (20.T x)) || po(2oo))]] 0,0
(3)
subject to Epanaa (x) (2o. (so. -o(20:)))
(4)
3

---

Under review as a conference paper at ICLR 2021
24
SVHN Data
(b) 7 64, rate 1.1
7 32, rate 7.47
25
5
14
77
55
221
15
221
2
6
B8
98
88
1655
50
TH
170
8 70
2
10
(d) y = 8, rate = 37.5
(c) 7 = rate rate = 116.0
(f) 0.5, rate 355.3
Figure 4: Reconstructions from distortion-targee targeted VAEs for various choices %. Choosing ? suffices to achieve a high-quality reconstruction and requires encoding significantly less information (rate) into the latent space than 7 = 0.5.
22
(a) SVHN Data
(b) 64, rate 1.1
(c) 32, rate 7.47
62
3
0
24102
BIS
28
14
00
2 8158
2
20
8
( = 8, rate = 37.5
(e) = 2, rate = 116.0 ( = 0.5, rate 355.3
Figure 5: Samples from distortion-targeted VAEs for various choices of 7. Lower rates result in easier modeling but blurrier samples. Higher rates make the modeling task much more challenging, causing poor sample quality despite good reconstructions (Figure
We optimize this constrained objective using a Lagrange multiplier (Zhao et al., 2018). We note that the rate term DRL. (4o (1a(T x)) po (=o.T) is upper bound on the mutual information I(X: Zo.r) under the distribution defined by po (z: 96 Zy:r x)) (Alemi et al., 2018). By varying the choice of y, modulate the amount of information about x: that is stored in in the latent code 20:7- Figure shows the reconstructions along with the rate the learned model. Models with high rate contain considerably more information about in the latent space and adds additional burden that the prior
4

---

Under review as a conference paper at ICLR 2021
3.5
3.0
144
2.5
1.3
Negalive ELBO
1.3
Rate
2.0
Negelive log
1.2
1.2
1.1
1.1
1.0
1.0
0.5
0.9
0.9
0.0
0.8
Mean-Squared Error
Mean-Squared Error
Mean-Squared Error
Figure 6: Rate, conditional log-likelihood In po z0.r). ) and ELBO as function of the distortion achieved by various VAEs on SVHN. The ELBO favors high-rate models, which has poor sample quality (Figure 5).
(o(:T) needs to model correctly. In Figure 5, we show the corresponding samples generated by Hol Zo:r) when Zo:T is distributed according to p=(zo:))- Note that smaller the rate, the more distributionally-sinnii
the reconstructions (Figure 4) are to the samples (Figure 5). Furthermore, Figure shows that the ELBO objective favors high-rate models, which-when naively trained- have poor sample quality as shown in Figure 5.
4 PRIORITIZING THE BITS THAT MATTER
In contrast to naive training, which favors extremely small distortions and high rates, our exper- iments demonstrate that only small amount of information required to achieve high sample- quality reconstructions. Furthermore, by limiting the the amount of information encoded into the latent space, the full modeling capacity the prior po (=o:T) can dedicated to the small amount of information necessary for perceptual quality, which in turn improves the sample quality of ju (o:r) when Zo.r is drawn from the prior (20:T))
The drawback, however, is that the large amount of entropy that remains is now crudely modeled by Gaussian observation model, causing the model to achieve poor likelihoods. To address this issue, we now train second variational autoencoder- with parameters (ω) 0) and latent variables 2o:r conditioal on the mean decoding ž == p(20:r) achieved by the first model, via the objective
minimize (1.(2) 0,6 subject to ( Bug M4 (a 2)12 7.
DKI (žo:r (r, Pa ( [8))
(6)
(Zo:r For national simplicity, we subsumed Pdata by defining (1, 2o.r) Paata (1) (2o:1 ax).
Our two key insights are as follows. First, the reconstructions from the first variational autoencoder augments the initial dataset by creating the paired data of the form (x, 2). This paired data subse- quently defines a new conditional density estimation problem that seeks to predict x based on the learned reconstructions. Second, the secondary variational autoencoder can trivially recover the ini- tial distortion 5 by setting the decoder as Hgg Mg("orr, .) f. And any further improvement is thus bounded by 7. Furthermore, since the initial distortion y was chosen to achieve perceptually- acceptable reconstructions, the secondary variational autoencoder is thus restricted to making small perturbations to the existing decoding ž.
Taken together, argue that since the perceptible bits already accounted for by the primary variational autoencoder, the secondary variational autoencoder is thus restricted to modeling the remaining imperceptible bits. As such, hypothesize that not only will the secondary variational autoencoder improve upon the initial ELBO by replacing the crude Gaussian observation model with more expressive conditional VAE observation model, it will also do with minimal impact the sample quality achieved by the primary variational autoencoder. Figure confirms this hypothesis, and Figure 8 further demonstrates that the secondary variational autoencoder significantly improves the initial ELBO without sacrificing sample quality.
5

---

Under review as a conference paper at ICLR 2021
3
6
0
22
60
8
2
62
9
38
2
6
8
3
9
5
2
3
8
5
60
8
2
62.9
56
2
6
10
5
3
9
3
8
3
3
0
12
160
8
2
6269
22
6
B
3
9
5 2
3
3
0
8
2
62
6
10
6
3
9
8 8
6
3
8
2
160
8
2
62
53 2
6
3
3
9
8
2
3
8
3
?
39
(a) SVHN. Row 1:
6.6. Rows 2-S: 6.6, 7
[2, 0.58, 0.17, 0.05]. Row 6: 7 0.05.
(b) CelebA. Row 1: 7 = 13.8. Rows 2-5: 7 = 13.8, f [, 2, 0.58, 0.3). Row 6: y = 0.3.
Figure 7: Samples from two-stage training for SVHN and CelebA. The first shows samples from the primary VAF. The next four rows show samples from secondary VAEs with various choices of 5. The secondary VAE makes cearriiperceppible visual changes to the primary VAE's samples. The last row shows samples from a VAE that directly targets a small distortion. In contrast to two-stage training, directly targeting small distortion yields poorer sample quality.
1e4
le4
1.2
gamma=6.6
gamma 13.8
4.25
1.1
gamma=0.05
gamma=0..
Negative ELBO
Negative ELBO
4.00
1.0
3.75
0.9
3.50
0.8
3.25
0.7
3.00
0.6
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Mean-Squared Enor
Mean-Squared Error
(a) SVHN
(b) CelebA
Figure 8: Test set ELBO from two-stage training function of the final distortion achieved. ELBOs for single-stage training are shown as horizontal lines for comparison. For the same final distortion target, two-stage training achieves similar ELBO single-stage training while also having better sample quality (Figure 7).

---

Under review as a conference paper at ICLR 2021
5
NoT ALL MEAN-SQUARED ERRORS ARE CREATED EQUAL
gamma=32
gamma=120
gamma=16
gamma-60
Mean Squared Error
gamma=8
Mean Squared Errer
gamma=33.
201
gamma-2
102
gamma-13.8 13.8
gamma-0:5
gamma-7.4
10"
10a
20000
40000
60000
BO000
100000
20000
40000
60000
80000
100000
Iterations
Iterations
(a) SVHN
(b) CelebA
Figure 9: Mean Squa-- Error over the first 100000 iterations. All models quickly converge to the targeted distortion values.
20
2
3
20
8
3
4
2
N
0
13
24
3
0
1
15
15
14
5
14
(a) 7
16, iteration 10000
(b)
16, iteratiion 270000
(c) = 33, iteration 10000
(d) 7 = 33, iteration 350000
Figure 10: Reconstructions for SVHN and CelebA function of the distortion target y and train- ing iterations. Despite achieving the same distortion (Figure 9), earlier iterations capture higher- frequency information.
While the primary focus of our work is on prioritizing the modeling of visually perceptible informa- tion, we also observe over the course of training notable characteristic of how mean-squared error relates to the visual quality of the reconstructions. Figure 9 shows characteristic distortion over time curves when training the primary variational autoencoder SVHN and CelebA for various choices of y. All models quickly achieve the targeted distortion value; the remainder of the training thus focuses minimizing the rate while keeping the distortion constant.
7

---

Under review as a conference paper at ICLR 2021
A natural consideration, then, is how the reconstruction differs visually near the beginning versus the end of training despite having the exact same mean-squared error. In Figure 10, we see that the reconstructions have much greater global consistency near the end of training than at the beginning of training. We interpret these results as indicating that the models rapidly achieve the desired distor- tion targets by encoding high-frequency information, but gradually shifts to using lower-frequency information (and thus achieving lower rate) over the course of training.
To draw an analogy to principal components analysis, for the bottom k eigenvectors to achieve the same reconstruction cost as the top-k' eigenvectors, it tends to be the case that k >>>>>>> k' for natural images. Similarly, while there are many encoding schemes that can achieve the reconstruction cost, they may differ greatly in the amount of information stored in the latent space. Masking out part of the image, for example, is an encoding scheme that still preserves significant amount of fine- grained texture information in the image and will likely result in much higher rate. As result, the constrained objective favors blurrier, but globally-consisten reconstructions over (locally) sharper but globally-inconsistent reconstructions.
RELATED WORK
Given the extensive literature on deep generative models, we highlight several works most relevant our paper. Theis al. (2015) pointed out that good likelihoods do not imply good sample qual- ity, and showed that it is possible to construct pathological distributions that exhibit such behavior (by mixing real with data with noise data). However, such a pathology does not reflect the failure mode that variational autoencoders exhibit in practice, which is the focus of our work. Meng et al. (2021); Dai & Wipf (2019) both proposed two-stage training processes, but with different underly- ing motivations. Meng et al. (2021) proposed to train primary model on a noised version of the original dataset to ease optimization in autoregressive models, whereas our two-stage training proce- dure explicitly characterizes the relation between likelihood and sample quality. Dai & Wipf (2019) motivated their two-stage training process to handle data manifolds with intrinsic dimensionality lower than the dimensionality of the ambient space X. In contrast, both our primary and secondary models are deep hierarchical VAES whose number of latent variables exceed the dimensionality of X. Menick & Kalchbrenner (2018) also decomposes the original image modeling problem into bits that are more versus less significant. However, the Subscale Pixel Network prescribes a hand- designed encoding scheme, whereas we allow the model learn an encoding scheme based on what minimizes rate most for a particular distortion target. Higgins et al. (2016) bears resemblance to our primary model, but differ in how the distortion targeted; whereas 3-VAE tunes the coeffi cients of reconstruction and rate terms, we directly target mean-squared eπror value, which is more interpretable. Finally, both VDVAE and NVAE (Child, 2020; Vahdat & Kautz, 2020) highlight the optimization challenges in VAE training: our work builds on of these works by further addressing the challenge of building VAFs that simultaneously achieve good sample quality and good likelihoods.
7 CONCLUSION
In this work, we critically examined why variational autoencoders tend to exhibit good likelihoods but poor sample quality for image generative modeling. Our experiments demonstrate that only a small amount of information in natural image distributions pertinent to perceptual sample quality. However, conventional ELBO optimization does not distinguish the bits that matter perceptually from the bits that do not. And since the imperceptible bits dominate the ELBO signal. the model does not dedicate enough modeling capacity to the perceptually relevant information, thus caus- ing poor sample quality. In addition to demonstrating this phenomenon, we also propose a simple two-stage training procedure that prioritizes the modeling the perceptible information. By doing so, we can reliably train VAžS with good sample quality while still achieving ELBOs comparable to conventionally-trainn
VAEs. While our work focuses on variational autoencoders, we believe that this general phenomenon of imperceptible information dominating the likelihood signal is relevant to other likelihood- based models for image modeling. Our work demonstrates not all bits cre- ated equal, and we encourage researchers and practitioners alike to explicitly prioritize the bits that "matter" in the design of deep generative models.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Expressive speech synthesis via modeling expressions with variational autoencoder. arXiv preprint WrXiv:1804002155 2018.
Alexander Alemi, Ben Poole. Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy, Fixing broken elbo. In Internatiinna Conference on Machine Learning, pp. 159-168. PMLR, 2018.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011. 2020.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:/103.0053 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. arXiv preprint arXiv: 2018.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, al. Neural scene representation and rendering. Science, 3606639)):2044-1110, 2018.
Rafael Gómez-Bombaree Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling. Dennis Sheberla, Jorge aguilera-Iparragune Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268 276, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Wei-Ning Hsu, Yu Zhang, and James Glass. Learning latent representations for speech generation and transformation. arXiv preprint arXiv:1704.04277 2017.
Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.688 6980, 2014.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv preprint
arXiv:1312.61 14, 2013.
Nir Levine, Yinlam Chow, Rui Shu, Ang Li, Mohammad Ghavamzadeh, and Hung Bui. Predic- tion, consistency, curvature: Representation learning for locally-lineal control. arXiv preprint arXiv:1909.01506, 2019.
Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao, and Stefano Ermon. Improved autore- gressive modeling with distribution smoothing. arXiv preprint arXiv:2103. 2021.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint 1xI.v:1112.066600 2018.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.0349, 201

---

Under review as a conference paper at ICLR 2021
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint 165:::6600005328,8 2016bb. 2016b.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint
2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation And Approximate Inference In Deep Generative Models. arXiv preprint arXiv: 1401.4022, 2014.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaac Sønderby, and Ole Winther. Ladder variational autoencoders. Advances neural information processing systems, 29:3738-3746, 2016.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint 1xXiv:197.00600 2019.
Mohammad M Sultan, Hannah K Wayment-Steet and Vijay S Pande. Transferable neural networks for enhanced sampling of protein dynamics. Journal of chemical theory and computation, 14(4): 1887-1894, 2018.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. arXiv preprint arXiv:1511.0188 2015.
Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184- 7220, 2016.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03899, 2020.
Herke Van Hoof, Nutan Chen, Maximilian Karl, Patrick van der Smagt, and Jan Peters. Stable rein- forcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 3928-3934. IEEE, 2016.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444-7453, PMLR, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. The information autoencoding family: A la- grangian perspective on latent variable generative models. arXiv preprint reXiv:1800 06510, 2018.
10

---

Under review as a conference paper at ICLR 2021
A ARCHITECTURE
A.1 PRIMARY VAE
Our model makes use of the VDVAE architecture (Child, 2020) in conjunction with precision- weighted merging (Sønderby et al., 2016). For SVHN, we use model with 48 layers of stochastic- ity, consisting of two resolution layers, four 4 4, ten 8, sixteen 16 16, and sixteen layers. For CelebA, we simply added an additional eight 64 x 64 resolution layers. Each layer of latent variables at any n n resolution has 16 channels. The hidden dimensionality of the residual network is fixed at 128 channels, and all bottleneck residual blocks uses 32 channel bottleneck.
A.2 SECONDARY VAE
The secondary VAE has the extract same depth-structure as the primary VAE and simply conditions on the additional image ū generated by the primary VAE. For the inference model, simply con- catenated [z, 2 as the input. For the generative model, we use U-Net structure to inject information about E all resolutions in the generative process.
B TRAINING INFORMATION
B. DEQUANTIZATION
We dequantize both SVHN and CelebA by adding uniform noise in the interval (0, 1) to the original [0, 255] pixel intensities. We only use the dequantized images when computing In In (Zdequantized 20:T). The inference model is always given the clean image. This practice of feeding the clean image into the inference model still admits a valid ELBO."
B.2 LAGRANGIAN
Letting R(0,0 0) and ( () denote the rate and distortion (as measured by mean-squared error) re- spectively, we can follow Zhao et al. (2018) 8) and perform ? distortion-targeting by jointly optimizing the following two objectives with gradient descent.
minimize R(Ø, 0) + (),,0) A (D(
0.0
maximize ())
(8)
In practice, instead perform
minimize - A) R( 0) + X D(
(9)
0,8
maximize ((D(0, 7) 7)
(10)
The general intuition still holds: if D > 7, then Equation (10) is maximized by increasing X, which in turn encourages Equation (9) reduce D. We restrict À E (0.0001, 0.9999] via projected gradient descent and initialize at = 0.9999 to mimic KI. annealing Sønderby et al. (2016).
B.3 OPTIMIZATION
We use the Adam optimizer (Kingma & Ba, 2014). We use a learning rate of 2 x 10 1 for both Equations (9) and (10). However, we use (2) (0.9, 0.999) for Equation (9) but (8,,8)) = (0.0, 0.999) for Equation (10).
see http:///iiishuiin22000////////10 for the lower bound interpretation.
11


Reviews:
[{"review_id": "zeFApaHliSv", "reviewer": "Reviewer_KKon", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of exploring over-parameterized VAEs is really important. The authors try to analyze the reasons of model’s poor performance. Over-parameterized hierarchical VAEs are able to achieve perfect reconstruction on the training set but become extremely vulnerable to overfitting. So it is important to understand how one can mitigate this issue. Authors suggest an interesting form of regularization and show that it improves the quality of generation at the expense of ELBO value. To deal with the latter they build secondary VAE upon the primary and show that now both good reconstruction and ELBO are achieved.\nAlthough I liked the general concept I still recommend rejection of the paper in its current form due to several reasons:\nThe authors claim that the reason of poor performance is “model being overwhelmed by the vast volume of visually-imperceptible information”. The claim itself may be right (although I disagree with it – see below) but is never shown in experiments. All that is observed/reported can be explained by more simple and clear reason: overfitting on training data. It may happen in over-parameterized VAE that are perfect in reconstructing training objects. They achieve arbitrary low reconstruction error and very high ELBO but have a mismatch between marginalized encoder $ \\int q(z|x) p_{data}(x) dx$ and prior $p(z)$. The reason of this mismatch is the domination of reconstruction term that can be made very large in case of (almost) perfect reconstruction since std in MSE loss goes to zero. In this case the mismatch is not penalized and hence further regularization is necessary. This is exactly what authors do but I do not think it is related to overwhelming of model. I have not found any experiments that confirm that VAE is concentrating on visually imperceptible information at the expense of visually important details. So I would say that the main claim remains unproven if not wrong.\nDoes your model suffer from mode-collapse (that is when you posterior q(z|x) collapses to prior p(z))? This problem was reported for over-parameterized VAE and it seems that minization of the rate given reconstruction error fixed can make the situation even worse increasing the number of «dead latents». It would nice to see the comparison between their quantity in initial VAE and in the proposed two-stage model. \nThe paper looks like it was written in a rush. There are many figures and few experiments. The phenomena and suggested modifications of the model is important and should be studied in more details. There is enough space for that in the paper. In particular I think it is important to see:\nThe comparison with less over-parameterized VAE that directly optimizes ELBO and is able to achieve the same value as the primary VAE that minimizes rate.\nThe relation between the values of gamma and values of beta in beta-VAE. What should be the value of beta in order to achieve the same rate and reconstruction as is achieved by primary VAE with given value of gamma?\nCurrent scheme is two-stage. Is it possible to derive single model that could be trained end-to-end and where losses from primary and secondary VAE are combined? That would be a nice regularization method for over-parameterized VAE.\nThe value of gamma is a crucial hyperparameter. It is important to see the strategies for its selection for different datasets.\nHow the rate changes in the last experiment? Is there any significant change between iterations 10000 and 270000/350000?\n\n\n", "summary_of_the_paper": "The paper investigates the possible reasons for poor performance of over-parameterized VAE that are able to achieve good train ELBO but fail to generate realistically-looking objects. The authors formulate a hypothesis and try to prove it experimentally deriving a novel two-stage scheme for VAE that provides good ELBO values and generates high-quality images. In particular they suggest to minimize rate (that is KL divergence between approximate posterior and prior over the latents) given the reconstruction error does not exceed predefined threshold gamma. This serves as a kind of regularizer that prevents overfitting and allows for better generation.", "summary_of_the_review": "Overall I find the paper interesting but in its current form it is more a workshop paper. Many important experiments are missing. The main claim is not justified and may be wrong. I encourage the authors to improve the paper, to better study the reasons that cause poor generation and to resubmit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "wjwSkEdhvn", "reviewer": "Reviewer_Vnek", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is quite good and I'm very excited by the idea.  Reconstruction losses have been a huge part of neural networks going back several decades, and they've always performed somewhat poorly, despite a great deal of research.  \n\nThe idea in this paper, that we can reframe reconstruction losses as a constraint, is intriguing.  At the same time, I feel that the constraint formulation in equations 3/4 feels a little bit off to me, since it seems like there will be some examples which we can reconstruct perfectly, so it feels odd to force them to have reconstruction error of \\gamma.  Maybe if the data is homogeneous enough we can simply find a \\gamma which works for all examples, but I think this will be problematic if the data is diverse.  For example if some images are solid color, I see no reason why I need to have a small error \\gamma instead of letting these examples have perfect reconstructions.  \n\nI also think this paper's presentation is too limiting, in that the ideas here are clearly applicable everywhere that we use reconstruction errors and not just VAEs.  However this makes the presentation more concrete, so I suppose it's okay.  \n\nI also wonder how much the visually imperceptible details can be removed simply by lowering the resolution (and then up-sampling).  I suspect this is imperfect and does remove more important information than the constrained formulation, but it feels like a good baseline to study.  I also think that it reflects one of the major ways that people have tried to make reconstruction losses perform better, is to do modeling at a low resolution in the first stage and then a higher resolution at the next stage.  \n\nSmall Comments: \n  -Section 3.1 is very nicely written but it's not clear to me that it's essential to the actual story.  \n\n  -It's just a matter of style, but I feel like Figures 4 and Figures 5 are the most striking, so using them to construct the teaser figure at the start of the paper would be compelling.  \n", "summary_of_the_paper": "This paper argues that much of the likelihood in a VAE is taken up by visually imperceptible information.  Thus a VAE may prefer to model things which are unimportant before learning important visual details.  The authors propose to address this by learning a multi-stage model where the first stage learns coarse visual details and the second stage learns to produce original images.  The technique for learning the first stage involves reframing reconstruction error as a constrained optimization problem where the reconstruction error is not reduced to zero but rather is reduced to some hyperparameter value \\gamma.  \n\nThe improvement in visual samples is impressive, especially on CelebA.  ", "summary_of_the_review": "This paper deals with a decades old question in neural networks research, of how we can produce high-quality outputs using reconstruction errors.  Their solution is logical but I have some questions about the details.  Nonetheless I think it represents progress on this problem and should be accepted.  I also found the samples to be impressive.  I believe that this paper will have very high impact.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gyG86hghxsU", "reviewer": "Reviewer_LBJj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is interesting. It resembles the Laplace pyramid approach to image modelling (high-frequency information conditioned on lower-frequency information), but in the proposed method we model \"imperceptible\" information conditioned on the \"perceptible\" information, motivating it with the fact that it's the \"imperceptible\" information that contributes most to the likelihood.\n\nI have a few issues with the paper, however. First, evaluation of the proposed method is not rigorous enough. Experiments do show that the model improves on the single-stage baseline in terms of ELBO-at-given-rate and sample-quality-at-given-rate, but 1) no quantitative perceptual quality metrics (e.g. FID) are used 2) no third-party baselines are used, especially other VAEs that aim to improve perceptual quality of samples (two-stage VAE by Dai & Kipf (2019), beta-VAE, VAEs with perceptual loss, etc.). In addition, the computational/parameter complexity is not discussed: we have to train *two* models instead of one in the proposed method, which is more expensive and increases the number of parameters when compared to the single-stage baseline. This should be accounted for in comparisons.\n\nThird, while I do commend the authors for building a narrative to motivate the proposed method, I've struggled to relate much of the content to prior work, and found some of it not to be sufficiently novel. A few examples follow.\n\nThe notion of the rate-distortion (RD) trade-off/curve in relation to VAE learning (Alemi et al., 2018) is not properly introduced, even though what eqs. (3)-(4) achieve is they select a particular point on the RD Pareto front. Doing so was also proposed by Alemi et al. in Section 2, hence the novelty of this should be discussed.\n\nThe finding that both low-rate and high-rate models result in poor samples is not surprising to me: in the former case the posterior doesn't deviate enough from the prior to store any information about the input, hence the decoder has to output an \"average\" blurry image; in the latter case the posterior is *too* different from the prior, so samples from the prior are likely out of the decoder's distribution. The finding that ELBO favours high-rate models is also not ironclad: standard ELBO can be seen as an instance of beta-weighted ELBO with beta=1 (Higgins et al., 2017), and using beta > 1 can make ELBO favour lower-rate/higher-distortion solutions. \n\nFinally, Section 2 and Section 3.1 are of questionable value. Section 2 ends with \"distortion-targeting appears to largely resolve the optimization issue\", which is much too broad a conclusion for the results presented. Figure 3 shows a that samples from a Gaussian are \"noisier\" than the mean of that Gaussian, which is hardly surprising.\n\nMinor points:\n- \"Rate\" and \"distortion\" are used throughout the write-up, but neither is at any point defined precisely.\n- Consider discussing [1], a more theoretical treatment of the relationship between the RD curve and perceptual quality.\n- Avoid figures immediately following a section title: makes for difficult reading. \n- \"MSE\" and \"distortion\" are used interchangeably: I'd stick to one.\n- Typos: \"for _national_ simplicity\"; in sec. 7: \"the modeling _ the perceptible information\".\n\nReferences:\n- [1]: https://arxiv.org/abs/1901.07821", "summary_of_the_paper": "Authors explore a disconnect between likelihood and perceptual quality in likelihood-based generative models --- a well-known phenomenon (Theis et al., 2015). Authors take the rate-distortion perspective of VAEs due to Alemi et al. (2018), and demonstrate that high-rate models (i.e. models that store more information about the input in the latent variables) tend to produce samples with poor perceptual quality. This is problematic, as authors also show that the standard ELBO training objective prefers such higher-rate models. In attempt to alleviate this authors propose a two-stage method, where one VAE is trained to be low-rate (i.e. to have high-quality samples), and a second VAE is trained to be high-rate and model the distribution of *imperceptible information* conditioned on the output from the first VAE. Authors run experiments to demonstrate that the proposed method produced samples with high-perceptual quality, while also having high test likelihoods.", "summary_of_the_review": "While I do find the proposed method interesting and thought-provoking, I don't think it's evaluated with sufficient rigour. I also find the rest of the story in the paper to not be sufficiently novel or well-connected to prior work. In the end, I consider the paper to be below the bar for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D52ninjThn1", "reviewer": "Reviewer_sh3z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe rate-distortion and related analysis has been a classic topic in understanding VAE and ELBO. This paper adds some interesting perspectives to this topic, especially from the angle of perception. This then motivated a two-stage training method that shows bits of information can be prioritized to capture more perceptible information during VAE training. \n\nSome results are intriguing and surprising, such as the speculation that higher-frequency information is being captured during earlier iteration of VAE training (Fig 10)\n\n\nWeakness:\n\n\nThe main aspect that needs to be clarified about this paper is its fundamental contribution. Neither the rate-distortion analysis nor the two-stage VAE is a new concept. In [1], a dedicated rate-distortion analysis has been made to show the competition and balance between these two terms in ELBO. It has been shown that a low distortion will lead to a good reconstruction, but limited generation / sampling (which is referred to as the auto-encoding limit). The findings presented from the current study seems to add little new insights to this. In general, the balance between these two terms have been widely studied, and it is not clear how much new insights this study is adding to the existing body of literature.The paper should also add better discussion of [1], which is only mentioned in passing in the current manuscript.\n\n\nThe presented two-stage training of VAE is also similar to existing works. In [2], for instance, a teacher-student scheme is used to prioritize the learning of the main factors of variations in the latent space (similar to prioritize the learning of the \"perceptible information\" in this work), while additional bits are then used to improve the image generation quality. It is true the teacher-student model is different from a two-stage model as proposed, and the motivation was posed differently -- but the fundamental contribution and insights are not clear to me given these existing works. [2] was also not discussed in this paper\n\nInterestingly, in this paper, the authors shows that the second stage VAE decreases ELBO loss without changing perceptible quality change of the generated images. In [2], the point of the additional bits was to improve the quality of the generated images. Intuitively, one would also expect that adding more bits will allow adding more details to the images. If the second-stage VAE does not improve the perception of the image, then what is the benefit of the second-stage VAE? Just to decrease the ELBO loss? But for what actual benefits?\n\n\n\nExperiment wise, this paper did not have a systematic experimental section. Results were presented to support statements, without a systematic description of the experimental settings. There was also no consideration or comparison to related works, further obscuring the contribution of the work compared to the state of the arts.\n\n[1] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing\na broken elbo. In International Conference on Machine Learning, pp. 159–168. PMLR, 2018.\n\n[2] Jose Lezama, Overcoming the disentanglment vs reconstruction trade-off via Jacobian supervision, ICLR, 2019", "summary_of_the_paper": "his paper presents a rate-distortion analysis about VAE to examine why good reconstruction may not be associated with good sampling/generation. Based on the analysis, a two-stage type of training strategies that prioritizes the modeling of the \"perceptible information\". Analyses and presented training strategies were supported by experimental results on benchmark data such as SVHN and CelebA.\n\n", "summary_of_the_review": "This paper discusses an interesting and long-standing topic with a perspective somewhat different from existing ones (i.e., whether information is perceptible), with statement supported by a two-stage VAE scheme. The contribution, both in insights or methods, however appears limited in terms of adding to existing knowledge. The paper can also strengthen discussion and comparison with related works that have examined the same topic.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9qMQNUGk6bx", "reviewer": "Reviewer_7Pio", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and easy to follow, and I think that further attempting to understand how likelihoods and sample quality are related is an important research direction. That being said, I do have two main problems with this paper:\n\n1. The work of Alemi et al. [1] is merely cited in passing, and I actually believe that the objective in the reviewed paper is not really novel as compared to this work. In other words, the presented relationship between likelihoods and sample quality seems to be just a rephrasing of the rate/distortion decomposition of [1]. I would be happy to be corrected about this point if I misunderstood anything, but I strongly believe that at the very least a careful discussion about [1] and how the current work differs from it is warranted.\n\n2. Given the previous point, I think the novel part of this paper is actually just the way in which they train VAEs in two steps. However, I don't think there is strong motivation for this: It feels like the main point of the paper is that likelihood is not important, and the only purpose of the second VAE is achieving good likelihoods. In other words, the authors spend a lot of time trying to convince the reader that likelihoods are dominated by high-frequency patters that are imperceptible to humans, and that low-likelihood-achieving VAEs are enough; only to then suggest adding a second VAE to \"fix\" the likelihood problem.\n\nMinor things:\n\n-The paper is using the ICLR 2021 format, not ICLR 2022.\n\n-The sentence \"the application of VAEs to image modeling has been lukewarm at best\" is a pretty strong statement.\n\n[1] Fixing a Broken ELBO, Alemi et al.\n\n========================================================================================================\n\nUPDATE 1 AFTER REBITTAL\n\n========================================================================================================\n\nI have read the author's response, and while I agree that this paper more closely looks at sample quality than Alemi et al., I still believe that a much more careful discussion is needed before the paper is accepted. I will thus keep my current score.", "summary_of_the_paper": "This paper studies the relationship between likelihood and sample quality in Gaussian VAEs. The authors argue that high likelihoods are obtained through high frequency signals which are not perceptible to humans, and that low-likelihood models can actually produce good samples. The authors support this by maximizing a varational upper bound to the mutual information between latents and osbservables, with the added constraint that reconstruction errors should have a given magnitude. Finally, the authors propose a two-step procedure; where a VAE is first trained attempting to achieve reasonable samples even if it does not have good accuracy, and then a second (conditional, on the reconstructions from the first VAE) VAE tries to learn the patterns that will give large likelihoods.", "summary_of_the_review": "While this paper studies an interesting problem in a well-presented way, I have doubts about both the novelty and the motivation of the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors discuss the disconnect between log-likelihood and sample quality of VAEs and relate it to an undesirable focus of the model on high-frequency signals. They propose to alleviate it through a two-stage training scheme for VAEs.
As it is, the paper does not explain well its contributions, especially compared to the rate-distortion balance discussion in "Fixing a Broken ELBo" by Alemi et al. (2018) (see [reviews sh3z](https://openreview.net/forum?id=-0LuSWi6j4&noteId=D52ninjThn1), [7Pio](https://openreview.net/forum?id=-0LuSWi6j4&noteId=9qMQNUGk6bx), and [LBJj](https://openreview.net/forum?id=-0LuSWi6j4&noteId=gyG86hghxsU)), and lacks the experiments to back up its claim (see [LBJj](https://openreview.net/forum?id=-0LuSWi6j4&noteId=gyG86hghxsU), and [KKon](https://openreview.net/forum?id=-0LuSWi6j4&noteId=zeFApaHliSv)). While the authors have made a more precise statement about their contributions in their rebuttal, the writing remains unclear.
I recommend this submission for rejection.
Decision: Reject