Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper tries to establish an explanation for the posterior collapse by linking the phenomenon to local minima. If I am understanding correctly, the final conclusion reads along the lines of 'if the reconstruction error is high, then the posterior distribution will follow the prior distribution'. They also provide some experimental data to suggest that when the reconstruction error is high, the distribution of the latents tend to follow the prior distribution more closely. 

Although I really liked section 3 where authors establish the different ways in which `posterior collapse' can be defined, overall I am not sure if I can extract a useful insight or solution out of this paper. When the reconstruction error is large, the VAE is practically not useful.  

Also, I don't think I am on board with continuing to use the standard Gaussian prior. Several papers (such as the cited Vampprior paper) showed that one can very successfully use GMM like priors, which reduces the burden on the autoencoder. Even though I liked the exposition in the first half of the paper, I don't think I find the contributions of this paper very useful, as one can actually learn the prior and get good autoencoder reconstructions while obtaining a good match between the prior and the posterior, without having degenerate posterior distribution which is independent from the data distribution. All in all, I think using a standard gaussian prior is not a good idea, and that fact renders the explanations provided in this paper obsolete in my opinion. Is there any reason why we would want to utilize a simplistic prior such as the standard Gaussian prior? Do you have any insights with regards to whether the explanations in this paper would still hold with more expressive prior distributions? 

In general, I have found section 5 hard follow. And to reiterate, the main arguments seem to be centered around autoencoders which cannot reconstruct well, as the authors also consider the deterministic autoencoder. If the autoencoder can not reconstruct well, it is not reasonable to expect a regularized autoencoder such as VAE to reconstruct, better, and therefore the VAE is already is a regime where it is not useful anyhow. I think the authors should think about the cases where the reconstruction error is low, and see if there is an issue of posterior collapse in those setups. 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Summary:

This paper is clearly written and well structured. After categorizing difference causes of posterior collapse, the authors present a theoretical analysis of one such cause extending beyond the linear case covered in existing work. The authors then extended further to the deep VAE setting and showed that issues with the VAE may be accounted for by issues in the network architecture itself which would present when training an autoencoder.



Overall:

1) I felt that Section 3 which introduces categorizations of posterior collapse is a valuable contribution and I expect that these difference forms of posterior collapse are currently under appreciated by the ML community. I am not certain that the categorization is entirely complete but is nonetheless an excellent step in the right direction. One source of confusion for me was the difference between sections (ii) and (v) --- in particular I believe that (ii) and (v) are not mutually exclusive.

Additionally, the authors wrote "while category (ii) is undesirable, it can be avoided by learning $\gamma$". While this is certainly true in the affine decoder case it is not obvious that this is true in the non-linear case.

2) Section 4 provides a brief overview of existing results in the affine case and introduces a non-linear counter-example showing that local minima may exist which encourage complete posterior collapse.

3) On the proof of Proposition 4.1. In A.2.1 you prove that there exists a VAE whose ELBO grows infinitely (exceeding the local maxima of (7)). While I have been unable to spot errors in the proof, something feels odd here. In particular, the negative ELBO should not be able to exceed the entropy of the data which in this case should be finite. I've been unable to resolve this discrepancy myself and would appreciate comments from the authors (or others). The rest of the proof looks correct to me.

4) I felt that section 5 was significantly weaker than the rest of the paper. This stemmed mostly from the fact that many of the arguments were far less precise and less rigorous than those preceding. I think the presentation of this section could be significantly improved by focusing around Proposition 5.1.

a) Section 5 depends on the decoder architecture being weak, though this is not clearly defined formally. I believe this is a sensible restriction which enables analysis beyond the setting of primary concern in Alemi et al. (and other related work).

b) In the third paragraph, you write "deep AE models can have bad local solutions with high reconstruction [...]". I feel that this doesn't align well with the discussion in this section. In particular, I believe it would be more accurate to say that IF the autoencoder has bad local minima then the VAE is also likely to have category (v) posterior collapse.

c) Equation (8) feels a little too imprecise. Perhaps this could be formalized through a bias-variance decomposition of the right hand side similar to Rolinek et al.?

d) The discussion of optimization trajectories was particularly difficult to follow. It is inherently difficult to reason about the optimization trajectories of deep auto-encoding models and is potentially dangerous to do so. For example, perhaps the KL divergence term encourages a smoother loss landscape and encourages the VAE to avoid the local stationary points that the auto-encoder falls victim to.

e) It is written, "it becomes clear that the potential for category (v) posterior collapse arises when $\epsilon$ is large". This is not clear to me and in fact the analysis seems more indicative of collapse presented in category (ii) (though as mentioned above, I am not convinced these are entirely separate). Similarly, later in this section it is written, "this is more-or-less tantamount to category (v) posterior collapse". I was also unable to follow this reasoning.

f) "it is actually the AE base architecture that is effectively the guilty party when it comes to posterior collapse". If the conclusions are to be believed, this only applies to category (v) collapse.

g) Unfortunately, I did not buy the arguments surrounding KL annealing at the end of section 5. In particular, KL warm start will change the optimization trajectory of the VAE. It is possible that the VAE has a significantly worse loss landscape than the autoencoder initially and so warm-start may enable the VAE to escape this difficult initial region.

Minor:

- The term "VAE energy" used throughout is not typical within the literature and seems less explicit than the ELBO (e.g. it overlaps with energy based models).
- Equation (4) is missing a factor of (1/2).
- Section 3, in (ii), typo: "assumI adding $\gamma$ is fixed", and "like-likelihood". In (v), typo: "The previous fifth categories"
- Section 4, end of para 3, citep used instead of citet for Lucas et al.
- Section 4, eqn 6 is missing a factor of 1/2 and a log(2pi) term.
- Section 5, "AE model formed by concatenating" I believe this should be "by composing".
- Section 5, eqn 10, the without $\gamma$ notation is confusing and looks as though the argmin does not depend on gamma. Presumably, it would make more sense to consider $\gamma^*$ as a function of $\theta$ and $\phi$.
- Section 5 "this is exactly analogous". I do not think this is _exactly_ analogous and would recommend removing this word.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 1. Summary
The paper theoretically investigates the role of “local optima” of the variational objective in ignoring latent variables (leading to posterior collapse) in variational autoencoders. The paper first discusses various potential causes for posterior collapse before diving deeper into a particular cause: local optima. The paper considers a class of near-affine decoders and characterise the relationship between the variance (gamma) in the likelihood and local optima. The paper then extends this discussion for deeper architecture and vanilla autoencoders and illustrate how this can arise when the reconstruction cost is high. The paper considers several experiments to illustrate this issue.

2. Opinion and rationales
I thank the authors for a good discussion paper on this important topic. However, at this stage, I’m leaning toward “weak reject”, due to the reasons below. That said, I’m willing to read the authors’ clarification and read the paper again during the rebuttal to correct my misunderstandings if there is any. The points below are all related.

a. I would like to understand the use of “local optima” here. I think the paper specifically investigate local optima of the likelihood noise variance, and there are potentially other local optima. Wouldn’t this be an issue with hyperparameter optimisation in general? For example, for any regression tasks, high observation noise can be used to explain the data and all other modelling components can thus be ignored, so people have to initialise this to small values or constrain it during optimisation.

b. I think there is one paper that the paper should discuss: Two problems with variational expectation maximisation for time-series models by Turner and Sahani. In this paper, the paper considers optimising the variational objective wrt noise likelihood hyperparameters and illustrates the “bias” issue of the bound towards high observation noise.
 
c. I think it would be good to think about the intuition of this as well: “unavoidably high reconstruction errors, this implicitly constrains the corresponding VAE model to have a large optimal gamma value”: isn’t this intuitive to improve the likelihood of the hyperparameter gamma given the data?

d. If all above are sensible and correct, I would like to understand the difference between this class of local minima and that of (ii). Aren’t they the same?

e. The experiments consider training AEs/VAEs with increasingly complex decoders/encoders and suggest there is a strong relationship between the reconstruction errors in AEs and VAEs, and this and posterior collapse. But are these related to the minima in the decoder’s/encoder’s parameter spaces and not the hyperparameter space? So that is the message that the paper is trying to convey here?

3. Minor:

Sec 3
(ii) assumI -> assuming
(v) fifth -> four, forth -> fifth

review_title: Official Blind Review #2

Meta-review: This manuscript investigates the posterior collapse in variational autoencoders and seeks to provide some explanations from the phenomenon. The primary contribution is to propose some previously understudied explanations for the posterior collapse that results from the optimization landscape of the log-likelihood portion of the ELBO.

The reviewers and AC agree that the problem studied is timely and interesting, and closely related to a variety of recent work investigating the landscape properties of variational autoencoders and other generative models. However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the technical difficulty and importance of the results. In reviews and discussion, the reviewers noted issues with clarity of the presentation and sufficient justification of the results. There were also concerns about novelty. In the opinion of the AC, the manuscript in its current state is borderline, and should ideally be improved in terms of clarity of the discussion, and some more investigation of the insights that result from the analysis.
Decision: Reject