The Usual Suspects? Reassessing Blame for VAE Posterior Collapse

Abstract: In narrow asymptotic settings Gaussian VAE models of continuous data have been shown to possess global optima aligned with ground-truth distributions.  Even so, it is well known that poor solutions whereby the latent posterior collapses to an uninformative prior are sometimes obtained in practice.  However, contrary to conventional wisdom that largely assigns blame for this phenomena on the undue influence of KL-divergence regularization, we will argue that posterior collapse is, at least in part, a direct consequence of bad local minima inherent to the loss surface of deep autoencoder networks.  In particular, we prove that even small nonlinear perturbations of affine VAE decoder models can produce such minima, and in deeper models, analogous minima can force the VAE to behave like an aggressive truncation operator, provably discarding information along all latent dimensions in certain circumstances.  Regardless, the underlying message here is not meant to undercut valuable existing explanations of posterior collapse, but rather, to refine the discussion and elucidate alternative risk factors that may have been previously underappreciated.

Reviews and author responses:

We have recently become aware that no paper revisions can be uploaded after November 15.  Therefore we are now uploading a new version of our paper that addresses many of the reviewer comments.  Among other things, we have clarified Section 5 and significantly expanded Section 7.  Beyond this, we have also included new empirical results involving KL warm-start in Section 6, Figure 1.  These results exactly conform with expectations per the analysis in our paper.  Of course any additional comments/suggestions by reviewers are greatly appreciated, and can be incorporated into a later update once available.

We have recently become aware that no paper revisions can be uploaded after November 15.  Therefore we are now uploading a new version of our paper that addresses many of the reviewer comments.  Among other things, we have clarified Section 5 and significantly expanded Section 7.  Beyond this, we have also included new empirical results involving KL warm-start in Section 6, Figure 1.  These results exactly conform with expectations per the analysis in our paper.  Of course any additional comments/suggestions by reviewers are greatly appreciated, and can be incorporated into a later update once available.

Question 4g):  "It is possible that the VAE has a significantly worse loss landscape than the autoencoder initially and so warm-start may enable the VAE to escape this difficult initial region."

Response:  As we have argued and demonstrated via experiments, deep AEs can have a bad loss landscape with local minima exhibiting high reconstruction error.  Using KL warm-start makes the VAE optimization trajectory actually more similar to that of an AE near the initialization.  And as the KL annealing parameter changes, the trajectories are merely pushed towards collapsed solutions which will also have high reconstruction error.  Regardless, we have never empirically found a single instance where a VAE with KL warm-start leads to better reconstruction error than the corresponding AE.

---------------------------

Other minor points:

Regarding the list of typos and other minor issues, we are extremely appreciative.  While it is obviously tedious for a reviewer to include such details, it is of course invaluable in clarifying and polishing a revision.

--------------------------- 

Thanks for the constructive comments pertaining to our submission.  Please let us know if there are any unresolved concerns.

*** Response to Reviewer 3 comments ***
---------------------------

Question 1): "One source of confusion for me was the difference between sections (ii) and (v) --- in particular I believe that (ii) and (v) are not mutually exclusive."

Response:  Categories (ii) and (v) are not necessarily mutually exclusive in the sense of the final outcome for certain scenarios, but they are nonetheless conceptually quite different.  For (ii) we are assuming that gamma is explicitly set too large an not changed during training.  The optimal value of remaining model parameters then leads to posterior collapse, i.e., the global optimum of the VAE cost conditioned on gamma being fixed too large can lead to collapse.  Category (ii) can occur even with simple affine decoders, but by virtue of the stated definition, it cannot occur if we learn gamma.

In contrast, category (v) describes the situation where all model parameters are optimized simultaneously, and yet the model gets stuck at a point where gamma is large and the posterior has collapsed.  In principle, this can happen even with arbitrarily small deviations from the affine case (Proposition 4.1), but in practice we argue that this is more likely to happen with deep, complex decoders as shown via experiments.

---------------------------

Question 3):  "On the proof of Proposition 4.1. In A.2.1 you prove that there exists a VAE whose ELBO grows infinitely (exceeding the local maxima of (7)). While I have been unable to spot errors in the proof, something feels odd here ..."

Response:  Actually, the ELBO will not be bounded when using models with sufficient capacity and data that lies on a low-dimensional manifold relative to the ambient space.  In this case, there can be infinite density on the manifold and the ELBO can be infinite as well.  For more on this issue, please see (Dai and Wipf, 2019).  As a special illustrative example, consider the case where the decoder is affine and the data lie on a subspace.  Loosely speaking, the VAE then collapses to probabilistic PCA, the ELBO bound is tight, and the VAE assigns infinite density to the data subspace.

---------------------------

Question 4b):  "... I believe it would be more accurate to say that IF the autoencoder has bad local minima then the VAE is also likely to have category (v) posterior collapse."

Response:  The reviewer's statement is definitely true; however, the full message we would like to make actually has two key parts: (1) If the AE has bad local minima then the analogous VAE is also likely to have category (v) collapse,  and (2) relatively deep AE models are likely to have bad local minima as evidenced by our experiments.

---------------------------

Question 4c):  "Equation (8) feels a little too imprecise. Perhaps this could be formalized through a bias-variance decomposition of the right hand side similar to Rolinek et al.?"

Response:  Unfortunately, we do not know of a natural way to further decompose this expression without additional approximations.  However, intuitively we expect this inequality to hold, and in all the experiments we have ever tried related to this paper, empirical estimates of eq. (8) hold true.

---------------------------

Question 4d):  "... perhaps the KL divergence term encourages a smoother loss landscape and encourages the VAE to avoid the local stationary points that the auto-encoder falls victim to."

Response:  It is of course always possible that the KL term somehow steers a trajectory away from solutions with poor reconstruction error.  However, if without this term the model produces bad reconstructions, it seems quite unlikely that adding a regularizer that explicitly favors ignoring the latent code would improve results.  And in fact, as mentioned above, we have never found a case where the VAE has a better reconstruction error than the AE.

---------------------------

Question 4e): "... it is written, 'this is more-or-less tantamount to category (v) posterior collapse.' I was also unable to follow this reasoning."

Response: Please see comments above regarding the difference between category (ii) and (v) collapse.  Also, if the gradients mentioned in this section are equal to zero, then there is no signal passing through the decoder, which is essentially equivalent to posterior collapse.  Moreover, it is category (v) by definition since gamma is learned in the present context.

---------------------------

Question 4f):  "If the conclusions are to be believed, this only applies to category (v) collapse."

Response:  This is correct.

---------------------------

Question e.:  "... So that is the message that the paper is trying to convey here?"

Response:  The reviewer's summary is partially correct; however, there is a bit more nuance involved in the full story.  In particular, a key thread of the paper can be enumerated as follows:

(1) Deeper AE architectures are essential for modeling high-fidelity images or similar.
(2) But counter-intuitively, increasing the depth of AE models can actually produce worse reconstruction errors even on the training data because of bad local minima.
(3) This then implies that the analogous VAE model may also have worse reconstruction error because the added KL regularization is likely to compound the problem, i.e., regularization usually just makes the reconstructions even worse as we have empirically verified.
(4) At any such bad minima, the value of gamma will necessarily be large, i.e., if it is not large, we cannot be at a local minimum.
(5) Whenever gamma is sufficiently large, the VAE will provably exhibit full/exact posterior collapse.
(6) Forcing gamma to be small does not fix this problem, since in some sense the "implicit" gamma is still large.
(7) Avoiding this overall scenario requires the development of better AE architectures, initializations, or training procedures such that bad reconstruction errors do not occur.  This prescription is quite different from existing remedies in the literature such as KL warm-start, which can help with relatively shallow models, but which is often incapable of significantly improving the deeper models we consider.

We can expand the discussion section (Section 7 of the submission) to more comprehensively present these points.  We can also reinforce other details such as the independent value of Propositions 4.1 and 5.1 in building upon existing results in the literature.

---------------------------

Thanks for the constructive comments pertaining to our submission.  Please let us know if there are any unresolved concerns.

*** Response to Reviewer 2 comments ***
---------------------------

Question a. (first part):  "I would like to understand the use of “local optima” here. I think the paper specifically investigate local optima of the likelihood noise variance, and there are potentially other local optima."

Response:  There may be a degree of misunderstanding here.  Our work is not just about local optima with respect to the noise variance.  For example, Proposition 4.1 involves the quantification of a bad local minimum with respect to all model parameters, including the noise variance gamma.  Similarly, Proposition 5.1 implies that if a noise variance estimate happens to be large (which will necessarily occur at bad local minima as we argue in Section 5), then the optimal solution with respect to the remaining model parameters will provably involve exact posterior collapse.  Please see our response to Question e. below for additional information related to this issue.

---------------------------

Question a. (second part): "... for any regression tasks, high observation noise can be used to explain the data and all other modelling components can thus be ignored, so people have to initialise this to small values or constrain it during optimisation."

Response:  High observation noise can be used to explain the data to some degree, but this will not generally produce a high likelihood as we desire when training a generative model like a VAE.  In fact, relying on observation noise while ignoring the latent variables is the intrinsic problem with posterior collapse that we would like to avoid.  Note that under certain conditions the VAE *global* optimum will provably avoid bad collapsed solutions exhibiting large observation noise (Bin and Wipf, 2019); however, the problem that still remains is collapsed *local* minima as we have detailed in our submission.  And importantly, such collapsed local minima can still exist even if we force the noise variance gamma to be small as we have argued in Section 5.

---------------------------

Question b.: "I think there is one paper that the paper should discuss ..."

Response:  While the paper by Turner and Sahani the reviewer mentions provides an interesting perspective on variational inference, the central message is not directly related to our work.  In particular, the bias towards large observation noise discussed in Turner and Sahani is a structural phenomena of the relatively simple set-up they examine, namely, the global optima of their low-dimensional energy function favors a large noise estimate.  In contrast, we consider deep VAE models of high-dimensional data whereby, under certain conditions from (Dai and Wipf, 2019), the global minimum will be provably characterized by gamma -> 0 (negligible noise), and yet bad local minima can lead to gamma being large and posterior collapse.

---------------------------

Question c.:  “... unavoidably high reconstruction errors, this implicitly constrains the corresponding VAE model to have a large optimal gamma value ..."

Response:  Yes, intuitively we would expect that with higher reconstruction errors, the optimal gamma value should be larger.  However, what is not at all intuitive is that, if this optimal gamma is large enough but still finite, the optimal solution with respect to all other model parameters is a fully degenerate posterior collapsed solution.  In this regard, the model behaves analogously to an exact thresholding operator, which is not obvious from inspection.  Hence our Proposition 5.1 is a significant contribution in and of itself.

---------------------------

Question d.:  "If all above are sensible and correct, I would like to understand the difference between this class of local minima and that of (ii)."

Response:  The class of minimum defined by category (ii) posterior collapse is where we explicitly set gamma to some fixed value that happens to be too large (i.e., gamma is not learned) and then train all other parameters.  This class of minimum can be avoided by simply treating gamma as a free parameter that can be learned along with all the others as mentioned in Section 3.  In contrast, the more insidious type of local minima we highlight in our submission occurs when gamma is simultaneously learned from the data along with all other parameters, but gets stuck at a large value because of poor reconstructions from deep autoencoder architectures.  This is the category (v) situation.  And for reasons described in Section 5, simply forcing gamma to be smaller does not solve this issue.

---------------------------


Question: "Is there any reason why we would want to utilize a simplistic prior such as the standard Gaussian prior? Do you have any insights with regards to whether the explanations in this paper would still hold with more expressive prior distributions?"

Response:  While non-Gaussian priors are certainly a promising enhancement, Gaussian priors are still arguably the most commonly-used latent-space distribution for VAE applications involving both continuous and discrete datasets.  Moreover, there are many ongoing effects to analyze VAEs with Gaussian priors, e.g., references (He et al., 2019; Lucas et al., 2019; Rolinek et al., 2019), and better understanding of their use can lead to targeted enhancements, e.g., reference (Dai and Wipf, 2019).

And finally, insights drawn from Gaussian models often do in fact translate to non-Gaussian models as well, so the analysis of the former is especially relevant.  For example, non-Gaussian priors do not immunize VAE models from posterior collapse and, as mentioned above, category (v) collapse as highlighted in our submission remains a significant concern regardless of how the aggregated latent posterior is modeled.  In this sense, our observations will certainly benefit those working with non-Gaussian priors such as Vampprior.  For further reference, the shared limitations of Gaussian and non-Gaussian priors are also discussed in reference (Mattei and Frellsen, 2018).

---------------------------

Question:  "If the autoencoder can not reconstruct well, it is not reasonable to expect a regularized autoencoder such as VAE to reconstruct better, and therefore the VAE is already in a regime where it is not useful anyhow."

Response:  We largely addressed this point in previous responses.  However, just to reiterate the message, the issue here is not just the acknowledgement that VAEs with bad reconstructions are not useful.  Rather, it is the examination of *why* do VAEs frequently enter such a regime where they are not useful in the first place.  The existing literature mostly suggests that it is the KL term that is to blame, which naturally suggests practical remedies such as KL warm-start.  In contrast, we have presented a distinct alternative explanation supported by both theory and experiments.  Regardless, we can further clarify Section 5, as well as add a more comprehensive enumeration of the central points to the discussion in Section 7 (please also see comments to R2, Question e. for additional details).

---------------------------

Question:  "I think the authors should think about the cases where the reconstruction error is low, and see if there is an issue of posterior collapse in those setups."

Response:  Our focus has been on isolating and examining category (v) posterior collapse, as presently this form of collapse stands in the way of handling high-dimensional data and there is currently limited analysis or understanding of this phenomena.  However, if the reconstruction error is low, the only type of posterior collapse that is possible is category (i), and this form of collapse is actually beneficial for downstream tasks like generating good samples as mentioned in Section 3.

---------------------------

Thanks for the constructive comments pertaining to our submission.  Please let us know if there are any unresolved concerns.

*** Response to Reviewer 1 comments ***
---------------------------

Question:  "Not sure if I can extract a useful insight or solution out of this paper.  When the reconstruction error is large, the VAE is practically not useful."

Response:  It is of course true that when the reconstruction error is large, the VAE may not be useful.  But our contributions go well beyond this observation and provide key explanations for why VAEs may at times behave poorly.  For example, we prove that even minor deviations from an affine decoder can produce local minima with bad reconstructions (Proposition 4.1).  Previous analysis of this kind has shown that no such bad minima can exist for the strictly affine case, e.g., reference (Lucas et al., 2019).

Beyond this we also demonstrate that, quite counter-intuitively, training deeper autoencoders will eventually lead to larger training data reconstruction errors even when using residual blocks and skip connections.  Such solutions must represent bad local minima (or saddle points) since increased capacity in these models can otherwise only improve such reconstructions.  Moreover, we prove that if the training reconstruction error is raised to a sufficiently high but finite value, exact posterior collapse will necessarily occur in the analogous full VAE model (Proposition 5.1).  This indicates that the VAE behaves as a canonical thresholding operator, a phenomena that has never been formally proven in generic architectures and yet provides a direct explanatory link to posterior collapse in deeper models.   These insights are largely orthogonal (yet still at times complementary) to current explanations for posterior collapse in the literature.

---------------------------

Question: "I don't think I am on board with continuing to use the standard Gaussian prior. Several papers (such as the cited Vampprior paper) showed that one can very successfully use GMM like priors, which reduces the burden on the autoencoder."

Response:  It is critical to point out that more sophisticated latent space priors (e.g., Vampprior) do not actually mitigate the type of posterior collapse we highlight in this paper.   In fact, VAE models will be similarly vulnerable to category (v) posterior collapse whether a Gaussian prior is used or not.  This is because to improve the reconstruction error for sophisticated high-dimensional data, a deeper or more complex decoder will be needed, and this can introduce a new constellation of bad local minima as we have shown.  A trainable non-Gaussian prior merely grants greater flexibility to modeling the aggregated posterior in the latent space, but bad local minima from deep decoders remain problematic.  Of course models like Vampprior can still be very helpful in generating better samples, but only when paired with a deep architecture capable of good reconstructions.

Note also that in (Tomczak and Welling, 2018), the Vampprior model is only tested on small black-and-white images, and so a deep decoder is not even needed.  Another more recent representative example is Bauer and Mnih, "Resampled Priors for Variational Autoencoders," AISTATS 2019, which again, involves shallow models and simple images.  While both of these papers (and others like them) involve elegant procedures for learning richer priors, this is a separate issue from our submission.  In particular, the category (v) posterior collapse we address will not generally materialize until larger, more complex decoders are adopted as required for dealing with high-resolution color images and the like.

---------------------------

1. Summary
The paper theoretically investigates the role of “local optima” of the variational objective in ignoring latent variables (leading to posterior collapse) in variational autoencoders. The paper first discusses various potential causes for posterior collapse before diving deeper into a particular cause: local optima. The paper considers a class of near-affine decoders and characterise the relationship between the variance (gamma) in the likelihood and local optima. The paper then extends this discussion for deeper architecture and vanilla autoencoders and illustrate how this can arise when the reconstruction cost is high. The paper considers several experiments to illustrate this issue.

2. Opinion and rationales
I thank the authors for a good discussion paper on this important topic. However, at this stage, I’m leaning toward “weak reject”, due to the reasons below. That said, I’m willing to read the authors’ clarification and read the paper again during the rebuttal to correct my misunderstandings if there is any. The points below are all related.

a. I would like to understand the use of “local optima” here. I think the paper specifically investigate local optima of the likelihood noise variance, and there are potentially other local optima. Wouldn’t this be an issue with hyperparameter optimisation in general? For example, for any regression tasks, high observation noise can be used to explain the data and all other modelling components can thus be ignored, so people have to initialise this to small values or constrain it during optimisation.

b. I think there is one paper that the paper should discuss: Two problems with variational expectation maximisation for time-series models by Turner and Sahani. In this paper, the paper considers optimising the variational objective wrt noise likelihood hyperparameters and illustrates the “bias” issue of the bound towards high observation noise.
 
c. I think it would be good to think about the intuition of this as well: “unavoidably high reconstruction errors, this implicitly constrains the corresponding VAE model to have a large optimal gamma value”: isn’t this intuitive to improve the likelihood of the hyperparameter gamma given the data?

d. If all above are sensible and correct, I would like to understand the difference between this class of local minima and that of (ii). Aren’t they the same?

e. The experiments consider training AEs/VAEs with increasingly complex decoders/encoders and suggest there is a strong relationship between the reconstruction errors in AEs and VAEs, and this and posterior collapse. But are these related to the minima in the decoder’s/encoder’s parameter spaces and not the hyperparameter space? So that is the message that the paper is trying to convey here?

3. Minor:

Sec 3
(ii) assumI -> assuming
(v) fifth -> four, forth -> fifth


Summary:

This paper is clearly written and well structured. After categorizing difference causes of posterior collapse, the authors present a theoretical analysis of one such cause extending beyond the linear case covered in existing work. The authors then extended further to the deep VAE setting and showed that issues with the VAE may be accounted for by issues in the network architecture itself which would present when training an autoencoder.



Overall:

1) I felt that Section 3 which introduces categorizations of posterior collapse is a valuable contribution and I expect that these difference forms of posterior collapse are currently under appreciated by the ML community. I am not certain that the categorization is entirely complete but is nonetheless an excellent step in the right direction. One source of confusion for me was the difference between sections (ii) and (v) --- in particular I believe that (ii) and (v) are not mutually exclusive.

Additionally, the authors wrote "while category (ii) is undesirable, it can be avoided by learning $\gamma$". While this is certainly true in the affine decoder case it is not obvious that this is true in the non-linear case.

2) Section 4 provides a brief overview of existing results in the affine case and introduces a non-linear counter-example showing that local minima may exist which encourage complete posterior collapse.

3) On the proof of Proposition 4.1. In A.2.1 you prove that there exists a VAE whose ELBO grows infinitely (exceeding the local maxima of (7)). While I have been unable to spot errors in the proof, something feels odd here. In particular, the negative ELBO should not be able to exceed the entropy of the data which in this case should be finite. I've been unable to resolve this discrepancy myself and would appreciate comments from the authors (or others). The rest of the proof looks correct to me.

4) I felt that section 5 was significantly weaker than the rest of the paper. This stemmed mostly from the fact that many of the arguments were far less precise and less rigorous than those preceding. I think the presentation of this section could be significantly improved by focusing around Proposition 5.1.

a) Section 5 depends on the decoder architecture being weak, though this is not clearly defined formally. I believe this is a sensible restriction which enables analysis beyond the setting of primary concern in Alemi et al. (and other related work).

b) In the third paragraph, you write "deep AE models can have bad local solutions with high reconstruction [...]". I feel that this doesn't align well with the discussion in this section. In particular, I believe it would be more accurate to say that IF the autoencoder has bad local minima then the VAE is also likely to have category (v) posterior collapse.

c) Equation (8) feels a little too imprecise. Perhaps this could be formalized through a bias-variance decomposition of the right hand side similar to Rolinek et al.?

d) The discussion of optimization trajectories was particularly difficult to follow. It is inherently difficult to reason about the optimization trajectories of deep auto-encoding models and is potentially dangerous to do so. For example, perhaps the KL divergence term encourages a smoother loss landscape and encourages the VAE to avoid the local stationary points that the auto-encoder falls victim to.

e) It is written, "it becomes clear that the potential for category (v) posterior collapse arises when $\epsilon$ is large". This is not clear to me and in fact the analysis seems more indicative of collapse presented in category (ii) (though as mentioned above, I am not convinced these are entirely separate). Similarly, later in this section it is written, "this is more-or-less tantamount to category (v) posterior collapse". I was also unable to follow this reasoning.

f) "it is actually the AE base architecture that is effectively the guilty party when it comes to posterior collapse". If the conclusions are to be believed, this only applies to category (v) collapse.

g) Unfortunately, I did not buy the arguments surrounding KL annealing at the end of section 5. In particular, KL warm start will change the optimization trajectory of the VAE. It is possible that the VAE has a significantly worse loss landscape than the autoencoder initially and so warm-start may enable the VAE to escape this difficult initial region.

Minor:

- The term "VAE energy" used throughout is not typical within the literature and seems less explicit than the ELBO (e.g. it overlaps with energy based models).
- Equation (4) is missing a factor of (1/2).
- Section 3, in (ii), typo: "assumI adding $\gamma$ is fixed", and "like-likelihood". In (v), typo: "The previous fifth categories"
- Section 4, end of para 3, citep used instead of citet for Lucas et al.
- Section 4, eqn 6 is missing a factor of 1/2 and a log(2pi) term.
- Section 5, "AE model formed by concatenating" I believe this should be "by composing".
- Section 5, eqn 10, the without $\gamma$ notation is confusing and looks as though the argmin does not depend on gamma. Presumably, it would make more sense to consider $\gamma^*$ as a function of $\theta$ and $\phi$.
- Section 5 "this is exactly analogous". I do not think this is _exactly_ analogous and would recommend removing this word.

This paper tries to establish an explanation for the posterior collapse by linking the phenomenon to local minima. If I am understanding correctly, the final conclusion reads along the lines of 'if the reconstruction error is high, then the posterior distribution will follow the prior distribution'. They also provide some experimental data to suggest that when the reconstruction error is high, the distribution of the latents tend to follow the prior distribution more closely. 

Although I really liked section 3 where authors establish the different ways in which `posterior collapse' can be defined, overall I am not sure if I can extract a useful insight or solution out of this paper. When the reconstruction error is large, the VAE is practically not useful.  

Also, I don't think I am on board with continuing to use the standard Gaussian prior. Several papers (such as the cited Vampprior paper) showed that one can very successfully use GMM like priors, which reduces the burden on the autoencoder. Even though I liked the exposition in the first half of the paper, I don't think I find the contributions of this paper very useful, as one can actually learn the prior and get good autoencoder reconstructions while obtaining a good match between the prior and the posterior, without having degenerate posterior distribution which is independent from the data distribution. All in all, I think using a standard gaussian prior is not a good idea, and that fact renders the explanations provided in this paper obsolete in my opinion. Is there any reason why we would want to utilize a simplistic prior such as the standard Gaussian prior? Do you have any insights with regards to whether the explanations in this paper would still hold with more expressive prior distributions? 

In general, I have found section 5 hard follow. And to reiterate, the main arguments seem to be centered around autoencoders which cannot reconstruct well, as the authors also consider the deterministic autoencoder. If the autoencoder can not reconstruct well, it is not reasonable to expect a regularized autoencoder such as VAE to reconstruct, better, and therefore the VAE is already is a regime where it is not useful anyhow. I think the authors should think about the cases where the reconstruction error is low, and see if there is an issue of posterior collapse in those setups. 


Meta-review: This manuscript investigates the posterior collapse in variational autoencoders and seeks to provide some explanations from the phenomenon. The primary contribution is to propose some previously understudied explanations for the posterior collapse that results from the optimization landscape of the log-likelihood portion of the ELBO.

The reviewers and AC agree that the problem studied is timely and interesting, and closely related to a variety of recent work investigating the landscape properties of variational autoencoders and other generative models. However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the technical difficulty and importance of the results. In reviews and discussion, the reviewers noted issues with clarity of the presentation and sufficient justification of the results. There were also concerns about novelty. In the opinion of the AC, the manuscript in its current state is borderline, and should ideally be improved in terms of clarity of the discussion, and some more investigation of the insights that result from the analysis.

Decision: reject