Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **GENERAL**

The paper presents a more in-sight analysis of using learnable variance in Gaussian decoders in the Variational Auto-Encoder framework. The authors follow up on other papers in the literature and try to answer the following research question:
- If a Gaussian decoder is used, is it better to fix the variance (e.g., \sigma=1), or learn it?
In general, the paper is fine, however, it lacks novelty and it does not highlight that the VAE framework is a probabilistic framework, and a Gaussian decoder is not appropriate for modeling images. Nevertheless, I must admit that the analysis is properly performed.

**Strengths:**

S1: An in-depth analysis of using a shared variance across dimensions in Gaussian decoders.

S2: Comparing both NLL and FID is a good indication of showing importance of learning variance (or, more generally speaking, modeling uncertainty) in decoders in VAEs.


**Remarks:**

R1: VAEs constitute a sub-class of latent variables models with prescribed distributions (namely, all distributions are known in advance). In opposition to implicit models, the distributions must be picked accordingly to observed quantities (e.g., images). I am totally aware that many authors use Gaussian decoders for modeling images, however, doing it naively is simply wrong. The support of a normal distribution is [-\infty, +\infty], while images are typically represented by integers in [0, ..., 255]. Therefore, even if pixel values are normalized to [0, 1], they take one of 256 possible values. As a result, using Gaussian decoders is inappropriate.

It is possible to use dequantization as it is typically done in flow-based models, see:
- Theis, L., van den Oord, A., and Bethge, M. A note on the evaluation of generative models. ICLR 2016
- Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. Flow++: Improving flow-based generative models with variational dequantization and architecture design. ICML 2019
- Winkler, C., Worrall, D., Hoogeboom, E., and Welling, M. Learning Likelihoods with Conditional Normalizing Flows. arXiv preprint, 2019
- Hoogeboom, E., Cohen, T. S., and Tomczak, J. M. Learning Discrete Distributions by Dequantization. arXiv preprint, 2020

However, this is not the case in this paper.
VAEs have a strong probabilistic foundation, and it should be treated as a probabilistic model. However, taking inappropriate distributions to model data, it results in:
(i) a wrong model;
(ii) propagating a wrong message in the deep learning community that any loss function works for VAEs.

I would highly appreciate if the authors would make it very clear in the paper, and start with a remark about choosing appropriate distribution for observed data. Afterwards, it could be explained that we need some sort of dequantization to utilize Gaussian decoders.

R2: The authors indicate a connection between a VAE with a Gaussian decoder and a \beta-VAE framework. The connection is very clear from the optimization perspective, but it is not the case from the modeling standpoint. If we take a look at the objective and consider the optimization process, then indeed, there is no difference in potential optima, because multiplying by \sigma or \beta results in the same objective. However, this has different consequences from the modeling perspective. \beta-VAE is a class of stochastic Auto-Endoders where the objective, the reconstruction error, is expanded by adding a Lagrangian multiplier. In other words, adding the equality constraint KL(q(z)||p(z)) = 0. However, the authors propose a valid VAE, i.e., the objective is a valid lower-bound to the log-likelihood function. I find it very confusing to indicate the connection without being very precise in what sense these two approaches are related to each other. Similarly to the remark R1, it could be very confusing in the DL community, and it could propagate a message that the VAE framework is a DL framework, while neural networks are an important (or even crucial) component of a probabilistic framework.

R3: As indicated by the authors, a similar approach was already discussed in the literature. Sharing a single variance across all dimensions was discussed in other papers. However, I must admit, not in-depth as here.

R4: It would be beneficial to discuss the following paper:
- Ghosh, P., Sajjadi, M. S., Vergari, A., Black, M., & Schölkopf, B. From variational to deterministic autoencoders. ICLR 2020

It is closely related and it could be see as an important other perspective on the problem.

*AFTER REBUTTAL*
I would like to thank the authors for their hard work! I increase my score to 6.
review_title: An analysis of learning variance in Gaussian decoders

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper claims two contributions: 1. Proposes a connection between beta-VAE with fixed variance Gaussian decoder and VAE with variable variance Gaussian decoder 2. Proposes to optimize the variance of a Gaussian decoder

This paper uses a simple but useful method, but I have some concerns about novelty and soundness of the experiments. I will give a score on the low side for now. 

Pro: 

The connection between beta-VAE and variable variance Gaussian decoder is somewhat interesting. 

The proposal that the variance can be adjusted for each individual image seems novel, but currently there does not seem to be a good theoretical/empirical justification for this. 

The writing is quite good. Easy to follow and clear. 

Con: 

I think the main claimed contribution has limited novelty. The main difference between the current work and the prior common practice is to learn a single variance shared by all the pixels (instead of learning a variance per pixel). This by itself is fine, the lack of novelty can be made up by very convincing experiments and good practical guidance. My major concern is that the paper does not sufficiently justify its choice with experiments:

The baselines are surprisingly bad. For example, the paper says it uses the model in Maaloe et al, 2019 for celebA; comparing the samples in the current paper and the original paper, there is a very big difference. Granted, the original paper has a somewhat different setup, but the sample quality is too unreasonable to be convincing (I think VAE samples looked like that in 2013). In addition, the FID score of baselines also seem to be much worse than typically reported in the literature. For example, a FID score of around 50 is already typical for VAE models in 2019 (e.g. see Dai et al, 2019), while the current paper report an FID of 186 for baselines. Such a big discrepancy does not instill confidence that the baselines are properly trained with current recommended practices. 

Following up on the above point, typically some kind of annealing of the beta parameter is the standard practice to avoid converging to a poor local minimum (which seems to be what is happening to the baselines here). It would be nice to have some discussion / comparison. In fact, I think it is completely fine if the argument is to show that reasoning effort to choose an annealing schedule do not lead to optimal results. However, I think this point does not come out from the experiments. 

The criticized alternative (learning a variance per pixel) has been widely used in the literature and the samples look quite reasonable (e.g. see http://ruishu.io/2018/03/14/vae/) unlike the baselines in the paper. Granted there are a few implementation tricks (such as bounding the output to avoid numerical instability), but these are quite mild. It is difficult to conclude that the proposed approach uses less practical implementation techniques to achieve these results. 

Minor comments:

The terminology calibration can have many different but precise meanings. For example, when people say a probability forecaster is calibrated, there is a specific property the forecaster must satisfy. I don’t think the usage of the terminology calibration matches any standard usage, and hence can be confusing. 

The title is not very informative. In particular, with the current interpretation of "calibration" which is to "learn better probabilities", I think any VAE paper could have used that title. 


--------- 

Thank you for the detailed reply and revisions. I have increased my review score because several of my concerns have been addressed. I am not entirely convinced that the baselines use current best practices, and several claims in the paper regarding the inductive biases and calibration. Nevertheless I think the algorithms proposed in the paper is practically useful. 
review_title: Some useful ideas, but lacks solid experimental verification

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper discusses a well-known problem of VAE training that decoder produces blurry reconstruction with constant variance. While much existing work addressed this problem by introducing independent variance training (as of the original VAE model) or additional hyper-parameters, those approaches usually come with additional training/tuning difficulty and even break the ELBO assumption. This paper proposed a simple $\sigma$-VAE that addresses the above problem by optimizing a single variance variable. This also could be easily connected to the well known $\beta$-VAE works. The experiment results in Tables 2 and 3 show the proposed model obtains a better FID score than the existing works on multiple datasets.

In general, I am not surprised that shared variance would work well in practice as it is somewhat obvious as lots of works with beta-VAE to get better performance. However, it is surprising that no one aligns beta-VAE with the variance of observation before. 

The most valuable knowledge I learned from this paper is written in one small section about variance implementation details where a new variable $lambda$ is introduced to avoid numeric problems. It is more helpful than the main claim of the paper somewhat in practice.

Pros
===
1. The paper is well written, and the authors provide sufficient evidence to show the proposed model works well in practice. The experiment design is also intuitive.
2. Mutual information analysis is great for reading.

Cons
===
1. The proposed approach is too simple to validate its novelty since it seems a rewritten of the existing formula by assigning the beta hyper-parameter with another meaning. 
2. Figure 8 is hard to believe as Gaussian VAE on CIFAR-10 should perform much better than it is shown here. Many GitHub implementation shows far better-generated images.

Again, it is quite surprising that the method is not introduced before the year 2020. I have an impression where some of the VAE implementations did include things like proposed in this paper (but fail to find them now).


review_title: a simple but interesting discovery (not sure if there is similar work before this)

Meta-review: This paper proposes to (re-)examine VAEs with calibrated uncertainties for the likelihood, which is say VAEs in which the variance is learned rather than chosen as a fixed hyperparameter. The authors argue that doing so provides a reasonable means of automatically navigating the tradeoff between minimizing the distortion (the reconstruction loss) and the rate (the KL loss) in the variational objective. In particular, the authors propose to use a diagonal covariance  Σ = σ^2 Ι that is shared across pixels, and note that it is trivial to define  σ(z) = MSE(x, μ(z)) on a per-image basis to minimize the reconstruction loss. 

This is very much a borderline paper. Reviewers appreciate that the writing is clear, and acknowledge that revisiting the idea of learning calibrated is of interest to the community. At the same time, the reviewers note that the proposed approach has very limited technical novelty, and note problems with the experimental evaluation. 

The metareviewer has read the paper, and is critical of the framing of this work. The manuscript in its current form does not do a sufficiently good job of discussing the large and detailed literature that exists on this topic. Learning calibrated decoders is by no means new, which this submission could and should acknowledge much more clearly. The two seminal papers on VAEs both considered learning calibrated decoders. Moreover there is a lack of thoughtful discussion of the reasons why learning a pixel-wise σ(z) is not common practice. The authors note that this can lead to problems with training stability, but fail to note that this problem is mathematically ill-posed; A well-known property of VAEs is that high-capacity models will memorize the training data, in the sense that the optimal learned marginal likelihood is equal to the empirical distribution over the training set (i.e. a mixture over delta peaks). 

The metareviewer would expect to see a more thoughtful discussion of  the long line of work on navigating the trade-off between rate and distortion, as well as the role of model capacity. A good place to start would be a more careful discussion of the autoencoding and autodecoding limits (Alemi et al 2018) and the GECO paper (Rezende et al 2018). More broadly, the metareviewer would expect some discussion of approaches that improve the quality of generation such as [1], and work that considers effect of model capacity on generalization, such as [2].  

In terms of experimental evaluation, this paper also somewhat falls short. As R4 notes, some of the results look worryingly bad, which may be due to the fact that the authors train for only 10 epochs (as indicated in  Appendix B). Moreover, what is once again lacking in experiments is a systematic consideration of the role of model capacity. Some comparison to more recent baselines than the β-VAE (e.g. GECO) would also be helpful here.
 
The metareviewer is sympathetic to the basic premise of this paper, which is the claim that learning a σ that is shared across pixels is a pretty good best practice in terms of finding a reasonable balance between rate and distortion. There is certainly room for a paper that communicates this idea. However, such a paper should (a) more explicitly position itself as revisiting this idea rather than introducing this idea, (b) include a more thoughtful discussion of related work, and (c) include a more robust empirical evaluation. 

[1] Engel, J., Hoffman, M. & Roberts, A. Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Models. arXiv:1711.05772 [cs, stat] (2017).

[2] Shu, R., Bui, H. H., Zhao, S., Kochenderfer, M. J. & Ermon, S. Amortized inference regularization. in Proceedings of the 32nd International Conference on Neural Information Processing Systems 4398–4407 (Curran Associates Inc., 2018).
Decision: Reject