Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper argues that much of the likelihood in a VAE is taken up by visually imperceptible information.  Thus a VAE may prefer to model things which are unimportant before learning important visual details.  The authors propose to address this by learning a multi-stage model where the first stage learns coarse visual details and the second stage learns to produce original images.  The technique for learning the first stage involves reframing reconstruction error as a constrained optimization problem where the reconstruction error is not reduced to zero but rather is reduced to some hyperparameter value \gamma.  

The improvement in visual samples is impressive, especially on CelebA.  
main_review: This paper is quite good and I'm very excited by the idea.  Reconstruction losses have been a huge part of neural networks going back several decades, and they've always performed somewhat poorly, despite a great deal of research.  

The idea in this paper, that we can reframe reconstruction losses as a constraint, is intriguing.  At the same time, I feel that the constraint formulation in equations 3/4 feels a little bit off to me, since it seems like there will be some examples which we can reconstruct perfectly, so it feels odd to force them to have reconstruction error of \gamma.  Maybe if the data is homogeneous enough we can simply find a \gamma which works for all examples, but I think this will be problematic if the data is diverse.  For example if some images are solid color, I see no reason why I need to have a small error \gamma instead of letting these examples have perfect reconstructions.  

I also think this paper's presentation is too limiting, in that the ideas here are clearly applicable everywhere that we use reconstruction errors and not just VAEs.  However this makes the presentation more concrete, so I suppose it's okay.  

I also wonder how much the visually imperceptible details can be removed simply by lowering the resolution (and then up-sampling).  I suspect this is imperfect and does remove more important information than the constrained formulation, but it feels like a good baseline to study.  I also think that it reflects one of the major ways that people have tried to make reconstruction losses perform better, is to do modeling at a low resolution in the first stage and then a higher resolution at the next stage.  

Small Comments: 
  -Section 3.1 is very nicely written but it's not clear to me that it's essential to the actual story.  

  -It's just a matter of style, but I feel like Figures 4 and Figures 5 are the most striking, so using them to construct the teaser figure at the start of the paper would be compelling.  

summary_of_the_review: This paper deals with a decades old question in neural networks research, of how we can produce high-quality outputs using reconstruction errors.  Their solution is logical but I have some questions about the details.  Nonetheless I think it represents progress on this problem and should be accepted.  I also found the samples to be impressive.  I believe that this paper will have very high impact.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: his paper presents a rate-distortion analysis about VAE to examine why good reconstruction may not be associated with good sampling/generation. Based on the analysis, a two-stage type of training strategies that prioritizes the modeling of the "perceptible information". Analyses and presented training strategies were supported by experimental results on benchmark data such as SVHN and CelebA.


main_review: Strength:

The rate-distortion and related analysis has been a classic topic in understanding VAE and ELBO. This paper adds some interesting perspectives to this topic, especially from the angle of perception. This then motivated a two-stage training method that shows bits of information can be prioritized to capture more perceptible information during VAE training. 

Some results are intriguing and surprising, such as the speculation that higher-frequency information is being captured during earlier iteration of VAE training (Fig 10)


Weakness:


The main aspect that needs to be clarified about this paper is its fundamental contribution. Neither the rate-distortion analysis nor the two-stage VAE is a new concept. In [1], a dedicated rate-distortion analysis has been made to show the competition and balance between these two terms in ELBO. It has been shown that a low distortion will lead to a good reconstruction, but limited generation / sampling (which is referred to as the auto-encoding limit). The findings presented from the current study seems to add little new insights to this. In general, the balance between these two terms have been widely studied, and it is not clear how much new insights this study is adding to the existing body of literature.The paper should also add better discussion of [1], which is only mentioned in passing in the current manuscript.


The presented two-stage training of VAE is also similar to existing works. In [2], for instance, a teacher-student scheme is used to prioritize the learning of the main factors of variations in the latent space (similar to prioritize the learning of the "perceptible information" in this work), while additional bits are then used to improve the image generation quality. It is true the teacher-student model is different from a two-stage model as proposed, and the motivation was posed differently -- but the fundamental contribution and insights are not clear to me given these existing works. [2] was also not discussed in this paper

Interestingly, in this paper, the authors shows that the second stage VAE decreases ELBO loss without changing perceptible quality change of the generated images. In [2], the point of the additional bits was to improve the quality of the generated images. Intuitively, one would also expect that adding more bits will allow adding more details to the images. If the second-stage VAE does not improve the perception of the image, then what is the benefit of the second-stage VAE? Just to decrease the ELBO loss? But for what actual benefits?



Experiment wise, this paper did not have a systematic experimental section. Results were presented to support statements, without a systematic description of the experimental settings. There was also no consideration or comparison to related works, further obscuring the contribution of the work compared to the state of the arts.

[1] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing
a broken elbo. In International Conference on Machine Learning, pp. 159–168. PMLR, 2018.

[2] Jose Lezama, Overcoming the disentanglment vs reconstruction trade-off via Jacobian supervision, ICLR, 2019
summary_of_the_review: This paper discusses an interesting and long-standing topic with a perspective somewhat different from existing ones (i.e., whether information is perceptible), with statement supported by a two-stage VAE scheme. The contribution, both in insights or methods, however appears limited in terms of adding to existing knowledge. The paper can also strengthen discussion and comparison with related works that have examined the same topic.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the relationship between likelihood and sample quality in Gaussian VAEs. The authors argue that high likelihoods are obtained through high frequency signals which are not perceptible to humans, and that low-likelihood models can actually produce good samples. The authors support this by maximizing a varational upper bound to the mutual information between latents and osbservables, with the added constraint that reconstruction errors should have a given magnitude. Finally, the authors propose a two-step procedure; where a VAE is first trained attempting to achieve reasonable samples even if it does not have good accuracy, and then a second (conditional, on the reconstructions from the first VAE) VAE tries to learn the patterns that will give large likelihoods.
main_review: This paper is well written and easy to follow, and I think that further attempting to understand how likelihoods and sample quality are related is an important research direction. That being said, I do have two main problems with this paper:

1. The work of Alemi et al. [1] is merely cited in passing, and I actually believe that the objective in the reviewed paper is not really novel as compared to this work. In other words, the presented relationship between likelihoods and sample quality seems to be just a rephrasing of the rate/distortion decomposition of [1]. I would be happy to be corrected about this point if I misunderstood anything, but I strongly believe that at the very least a careful discussion about [1] and how the current work differs from it is warranted.

2. Given the previous point, I think the novel part of this paper is actually just the way in which they train VAEs in two steps. However, I don't think there is strong motivation for this: It feels like the main point of the paper is that likelihood is not important, and the only purpose of the second VAE is achieving good likelihoods. In other words, the authors spend a lot of time trying to convince the reader that likelihoods are dominated by high-frequency patters that are imperceptible to humans, and that low-likelihood-achieving VAEs are enough; only to then suggest adding a second VAE to "fix" the likelihood problem.

Minor things:

-The paper is using the ICLR 2021 format, not ICLR 2022.

-The sentence "the application of VAEs to image modeling has been lukewarm at best" is a pretty strong statement.

[1] Fixing a Broken ELBO, Alemi et al.

========================================================================================================

UPDATE 1 AFTER REBITTAL

========================================================================================================

I have read the author's response, and while I agree that this paper more closely looks at sample quality than Alemi et al., I still believe that a much more careful discussion is needed before the paper is accepted. I will thus keep my current score.
summary_of_the_review: While this paper studies an interesting problem in a well-presented way, I have doubts about both the novelty and the motivation of the proposed approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Authors explore a disconnect between likelihood and perceptual quality in likelihood-based generative models --- a well-known phenomenon (Theis et al., 2015). Authors take the rate-distortion perspective of VAEs due to Alemi et al. (2018), and demonstrate that high-rate models (i.e. models that store more information about the input in the latent variables) tend to produce samples with poor perceptual quality. This is problematic, as authors also show that the standard ELBO training objective prefers such higher-rate models. In attempt to alleviate this authors propose a two-stage method, where one VAE is trained to be low-rate (i.e. to have high-quality samples), and a second VAE is trained to be high-rate and model the distribution of *imperceptible information* conditioned on the output from the first VAE. Authors run experiments to demonstrate that the proposed method produced samples with high-perceptual quality, while also having high test likelihoods.
main_review: The proposed method is interesting. It resembles the Laplace pyramid approach to image modelling (high-frequency information conditioned on lower-frequency information), but in the proposed method we model "imperceptible" information conditioned on the "perceptible" information, motivating it with the fact that it's the "imperceptible" information that contributes most to the likelihood.

I have a few issues with the paper, however. First, evaluation of the proposed method is not rigorous enough. Experiments do show that the model improves on the single-stage baseline in terms of ELBO-at-given-rate and sample-quality-at-given-rate, but 1) no quantitative perceptual quality metrics (e.g. FID) are used 2) no third-party baselines are used, especially other VAEs that aim to improve perceptual quality of samples (two-stage VAE by Dai & Kipf (2019), beta-VAE, VAEs with perceptual loss, etc.). In addition, the computational/parameter complexity is not discussed: we have to train *two* models instead of one in the proposed method, which is more expensive and increases the number of parameters when compared to the single-stage baseline. This should be accounted for in comparisons.

Third, while I do commend the authors for building a narrative to motivate the proposed method, I've struggled to relate much of the content to prior work, and found some of it not to be sufficiently novel. A few examples follow.

The notion of the rate-distortion (RD) trade-off/curve in relation to VAE learning (Alemi et al., 2018) is not properly introduced, even though what eqs. (3)-(4) achieve is they select a particular point on the RD Pareto front. Doing so was also proposed by Alemi et al. in Section 2, hence the novelty of this should be discussed.

The finding that both low-rate and high-rate models result in poor samples is not surprising to me: in the former case the posterior doesn't deviate enough from the prior to store any information about the input, hence the decoder has to output an "average" blurry image; in the latter case the posterior is *too* different from the prior, so samples from the prior are likely out of the decoder's distribution. The finding that ELBO favours high-rate models is also not ironclad: standard ELBO can be seen as an instance of beta-weighted ELBO with beta=1 (Higgins et al., 2017), and using beta > 1 can make ELBO favour lower-rate/higher-distortion solutions. 

Finally, Section 2 and Section 3.1 are of questionable value. Section 2 ends with "distortion-targeting appears to largely resolve the optimization issue", which is much too broad a conclusion for the results presented. Figure 3 shows a that samples from a Gaussian are "noisier" than the mean of that Gaussian, which is hardly surprising.

Minor points:
- "Rate" and "distortion" are used throughout the write-up, but neither is at any point defined precisely.
- Consider discussing [1], a more theoretical treatment of the relationship between the RD curve and perceptual quality.
- Avoid figures immediately following a section title: makes for difficult reading. 
- "MSE" and "distortion" are used interchangeably: I'd stick to one.
- Typos: "for _national_ simplicity"; in sec. 7: "the modeling _ the perceptible information".

References:
- [1]: https://arxiv.org/abs/1901.07821
summary_of_the_review: While I do find the proposed method interesting and thought-provoking, I don't think it's evaluated with sufficient rigour. I also find the rest of the story in the paper to not be sufficiently novel or well-connected to prior work. In the end, I consider the paper to be below the bar for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the possible reasons for poor performance of over-parameterized VAE that are able to achieve good train ELBO but fail to generate realistically-looking objects. The authors formulate a hypothesis and try to prove it experimentally deriving a novel two-stage scheme for VAE that provides good ELBO values and generates high-quality images. In particular they suggest to minimize rate (that is KL divergence between approximate posterior and prior over the latents) given the reconstruction error does not exceed predefined threshold gamma. This serves as a kind of regularizer that prevents overfitting and allows for better generation.
main_review: The topic of exploring over-parameterized VAEs is really important. The authors try to analyze the reasons of model’s poor performance. Over-parameterized hierarchical VAEs are able to achieve perfect reconstruction on the training set but become extremely vulnerable to overfitting. So it is important to understand how one can mitigate this issue. Authors suggest an interesting form of regularization and show that it improves the quality of generation at the expense of ELBO value. To deal with the latter they build secondary VAE upon the primary and show that now both good reconstruction and ELBO are achieved.
Although I liked the general concept I still recommend rejection of the paper in its current form due to several reasons:
The authors claim that the reason of poor performance is “model being overwhelmed by the vast volume of visually-imperceptible information”. The claim itself may be right (although I disagree with it – see below) but is never shown in experiments. All that is observed/reported can be explained by more simple and clear reason: overfitting on training data. It may happen in over-parameterized VAE that are perfect in reconstructing training objects. They achieve arbitrary low reconstruction error and very high ELBO but have a mismatch between marginalized encoder $ \int q(z|x) p_{data}(x) dx$ and prior $p(z)$. The reason of this mismatch is the domination of reconstruction term that can be made very large in case of (almost) perfect reconstruction since std in MSE loss goes to zero. In this case the mismatch is not penalized and hence further regularization is necessary. This is exactly what authors do but I do not think it is related to overwhelming of model. I have not found any experiments that confirm that VAE is concentrating on visually imperceptible information at the expense of visually important details. So I would say that the main claim remains unproven if not wrong.
Does your model suffer from mode-collapse (that is when you posterior q(z|x) collapses to prior p(z))? This problem was reported for over-parameterized VAE and it seems that minization of the rate given reconstruction error fixed can make the situation even worse increasing the number of «dead latents». It would nice to see the comparison between their quantity in initial VAE and in the proposed two-stage model. 
The paper looks like it was written in a rush. There are many figures and few experiments. The phenomena and suggested modifications of the model is important and should be studied in more details. There is enough space for that in the paper. In particular I think it is important to see:
The comparison with less over-parameterized VAE that directly optimizes ELBO and is able to achieve the same value as the primary VAE that minimizes rate.
The relation between the values of gamma and values of beta in beta-VAE. What should be the value of beta in order to achieve the same rate and reconstruction as is achieved by primary VAE with given value of gamma?
Current scheme is two-stage. Is it possible to derive single model that could be trained end-to-end and where losses from primary and secondary VAE are combined? That would be a nice regularization method for over-parameterized VAE.
The value of gamma is a crucial hyperparameter. It is important to see the strategies for its selection for different datasets.
How the rate changes in the last experiment? Is there any significant change between iterations 10000 and 270000/350000?



summary_of_the_review: Overall I find the paper interesting but in its current form it is more a workshop paper. Many important experiments are missing. The main claim is not justified and may be wrong. I encourage the authors to improve the paper, to better study the reasons that cause poor generation and to resubmit.
review_title: null

Meta-review: The authors discuss the disconnect between log-likelihood and sample quality of VAEs and relate it to an undesirable focus of the model on high-frequency signals. They propose to alleviate it through a two-stage training scheme for VAEs.
As it is, the paper does not explain well its contributions, especially compared to the rate-distortion balance discussion in "Fixing a Broken ELBo" by Alemi et al. (2018) (see [reviews sh3z](https://openreview.net/forum?id=-0LuSWi6j4&noteId=D52ninjThn1), [7Pio](https://openreview.net/forum?id=-0LuSWi6j4&noteId=9qMQNUGk6bx), and [LBJj](https://openreview.net/forum?id=-0LuSWi6j4&noteId=gyG86hghxsU)), and lacks the experiments to back up its claim (see [LBJj](https://openreview.net/forum?id=-0LuSWi6j4&noteId=gyG86hghxsU), and [KKon](https://openreview.net/forum?id=-0LuSWi6j4&noteId=zeFApaHliSv)). While the authors have made a more precise statement about their contributions in their rebuttal, the writing remains unclear.
I recommend this submission for rejection.
Decision: Reject