Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The paper presents a characterization of failure modes of Gaussian VAEs. It is known that Gaussian VAEs can fail to produce good models either by failing to match the data distribution or by learning latent variables that are uninformative. The paper builds upon prior work that suggests that the VAE objective can cause the inference model to over-regularize the generative model. The paper characterizes the conditions under which this over-regularization occurs with corresponding. Furthermore, the paper examines the affect of VAE pathologies on downstream tasks a number of unsupervised and supervised downstream tasks.

----------------------------

The paper is clearly written and the authors have attempted cover all cases of the theoretical reasons they have identified for VAE failures with corresponding experiments. The attempt to cover a number of downstream tasks is also a positive since this is a main goal of unsupervised learning.

As for the core of the paper the authors have identified two types of VAE failures: 1) when the generative quality is traded off for simple posteriors and 2) when the learning of output variance is biased. The first case is presented as occuring when the true posterior is difficult to estimate and there is no good likelhood function with a simple posterior. An expression of this is given as theorem 1.

This brings me to my main concern: Theorem 1 seems to say that if 1) the model cannot match the true posterior (Gaussian assumption is false) and 2) the model cannot match p(x) without matching the true posterior (decoder is not too powerful) then the model will not match p(x). If this is the case, then I would suggest that the statement does not present any new insight into VAE failures since a good VAE model of p(x) would either match the true simple posterior or it will model p(x) without matching the posterior.

Another concern is that although the authors mention prior work on posterior collapse, they do not consider this when characterizing the conditions that cause pathological behvaviour in VAEs. In particular it seems that the "No good, simpler alternative" condition explicitly excludes the case of a strong decoder that is able to ignore the latent code to model p(x) directly. This seems to be at odds with the authors' claim that they consider "fully flexible generative and inference models" when comparing with the prior work of Yacoby et al. Also, the case of strong decoder able to ignore the latent code is difficult to see as a case of the inference model over-regularizing the generative model.

If my understanding above is correct, then I suggest that the authors revise their claim of considering fully flexible models. In particular, they should state whether or not the case of posterior collapse is handled by their characterization.

The authors also present theorem 2 which suggests that using the ELBO to choose output variance results in a biased estimate. As far as I can tell this is new and of interest.

The authors present a number of experiments to show VAE failures: when the true posterior is far from Gaussian, when the true posterior is Gaussian and the decoder is weak, when the posterior is far from Gaussian and but the decoder is strong enough. They also show experiments to show that the output noise is overestimated when the ELBO is used to compute it. One limitation is that the authors have chosen very simple datasets with 1-d latent spaces for illustration. This leads to another question of interest of whether there are problems with VAEs that only appear with higher dimensional latent spaces?

Overall I found the paper to be interesting but in light of the questions mentioned above I do not recommend acceptance.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper exposes the pathologies of VAEs and characterizes them with concrete conditions. The synthetic data experiments well summarize the conditions that we might meet in real-world applications. The authors also analyze the corresponding effects for specific downstream tasks and give insightful suggestions to avoid these problems.

The trade-off between the generative distribution and inference distribution in VAEs has been studied for a long time and has been revealed from several perspectives such as information theory, etc. It is good to see in this paper that the two conditions in theorem 1 summarize well why VAEs work well or poorly.  

Questions:
For theorem 1, when only condition 1 holds, we know $p(x)$ might still be approximated but we may get an unwanted generative distribution since there exist several generative models that fit the data equally well. Is there any solution to avoid the duplicated solution?

In the paper, IWAE is applied to avoid the issues mentioned in VAEs. However, in Figure 9, the learned generative model seems to be different than the ground truth, and the posterior is thus simpler than the true posterior. Is there any possible explanation for this multi-modal case? 

---

In general, this is an interesting paper that well analyzes the training issues in VAEs and provides insightful guidance to the VAE studies. The paper is well written (the figure labels are too small to read) and easy to follow. 


----
Update after the discussion stage

I appreciate the authors' responses to address my questions in the experiments. However, I agree with the concerns of the other reviewers, especially the redundancy of Theorem 1 raised by Reviewer3. After reading the reviews and the discussion, the authors seem not to provide convincing responses to this part and this raises my concerns for this paper. Thus I change my rating below the borderline.


review_title: Interesting paper that exposes the MLE and posterior matching issues of VAEs

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The paper presents two analysis: (1) Characterization of when the training of VAEs using the ELBO leads to suboptimal generative models (biased towards ones with simple posteriors); and (2) How this suboptimality may affect downstream tasks that use the learned models. Specifically, the work focuses on VAEs using mean-field Gaussians as variational distributions, and explores how their limited modeling capacity affects the final generative model learned. They present some theoretical results and simple and illustrative scenarios. In addition, they present an analysis regarding how the suboptimal models learned affect other tasks, such as learning disentangled or compressed representations.


Pros:
- The paper is very clearly written.
- The theoretical results are novel, although based on well known ideas and maybe not very relevant from a practical perspective (see "cons").
- It presents an interesting exploration of how the known failures of VAEs affect subsequent tasks that use the suboptimal models. I think this is quite relevant, and often does not receive as much attention as new training methods for VAEs.


Cons:
- As mentioned in the paper, the failure modes of VAEs are known. The paper's first contribution is a characterization of when they happen. While the theorems give precise conditions and expressions, it is not clear to me whether they are useful in practice for real scenarios. I think most of the analysis of downstream tasks do not really require the precise conditions given in the theorems, but the ideas behind them, which were already known. For instance, the analysis regarding disentangled representations states that given several models with equal likelihood, optimization will choose the one with lower KL divergence (which may be disentangled or not, depending on the scenario). This does not require the conditions from theorem 1, but knowing that for a given log p(x), decreasing the KL divergence increases the ELBO (which is known, given that the tightness of the ELBO bound is exactly the KL divergence).
- Something similar happens with the analysis regarding compressed representations. Here, it is mentioned that increasing K (dimensionality of latent space) leads to simpler functions and thus to simpler posteriors, and is therefore preferred when using mean-field Gaussians approximating distributions. Again, the precise conditions from the theorems do not seem to be used here. (To the authors, please let me know if I am wrong; that is, if the precise results from the theorems are actually used in these analysis and I missed it.)
- The relevance of the result in theorem 2 is not clear to me. The fact that the optimal noise parameters is not recovered is expected (it is known that optimizing the ELBO leads to biases in the parameters of the generative model), albeit the expression that precisely quantifies the bias in terms of the approximating distribution was not known. However, this exact expression is not used in the subsequent analysis, only the fact that it is biased/suboptimal. Thus the relevance of this result is not clear to me.
- I think that some parts of the second analysis could use further justification. For instance, consider the compressed representations part. It is not clear to me why increasing K leads to generative models with simpler posteriors. I saw the empirical analysis in the Appendix, but did not find an analysis justifying the claim. Am I missing something simple here?


All in all, I do not find the first part of the analysis to be very relevant to the community. This is not the case find the second part of the analysis; I consider that studying how these suboptimalities affect other downstream tasks is important. Despite the fact that I believe the analysis could be expanded, I think it represents a first step in this direction. Therefore, I'm inclined to recommend acceptance. (I updated the score after the discussion.)


One last comment, I would suggest using larger fonts in all plots.
review_title: Interesting analysis in the second part

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper looks to investigate when VAEs fail to learn the maximum marginal likelihood (MML) model and some of the implications this can have for downstream tasks.  In particular, it introduces a theorem (Theorem 1) that provides assumptions under which MML will not be found, and then performs experiments to assess behavior in scenarios where the authors believe these assumptions are satisfied or not.

Though the topic of the paper is interesting and the long term aims are a good line of research, I believe the actual approach taken is rather misguided and that the arguments and conclusions are not properly supported.  In short, I do not believe that the key claimed contribution of describing *when* pathologies occur in VAEs is actually accurate or that the paper adds notable additional insight on this compared to previous work.  As such, I do not believe it is suitable for publication at ICLR in its present state.

*Strengths*
- The problem the work is trying to tackle is important.
- The supplement is very comprehensive and a lot of experiments have been run.
- The paper is generally well written (though the hand-waviness of some of the arguments and reliance on the supplement do significantly detract from the clarity in the latter sections).
- The work is mostly well referenced.  One important missing reference that should be added is  https://arxiv.org/abs/2006.10102 which already makes important related arguments about trade-offs in M2-style semi-supervised VAEs (in particular, they discuss the fact this type of semi-supervision imposes a mismatch between the marginal posterior q(z) and the prior p(z), which closely relates to this paper's discussion of mismatch between p(z|x) and q(z|x) but without needing to assume a ground truth posterior).  Direct discussion about the fact that a VAE need not learn a "ground truth likelihood" to perfectly match the data distribution should also be added (see e.g. http://ruishu.io/2017/01/14/one-bit/).
- Though I am not sure I agree with all the associated conclusions or assumptions being made, it is clear that the authors have put noticeable effort into trying to link the findings of the paper to practical implications.

*Weaknesses*
- I believe the core result of the paper (Theorem 1) is redundant (see below).
- The work makes lots of unnecessary and restrictive assumptions about a "ground truth" model that do not match up to typical real-world situations where such a model does not generally exist (in particular, the latent variables are generally an arbitrary construction such that the concept of a ground truth is actually meaningless).  Moreover, it is already well established that VAEs are not, in general, capable of uncovering a ground truth model even when this does exist (due to equivalence classes amongst other things); the paper offers little beyond this already well-known fact.  In fact, it is written in a way that often implies that we would expect such a ground truth to be uncovered even though various works have explained why this is not a reasonable expectation (e.g. Locatello et al 2019).  
- Most of the paper is extremely hand-wavy and imprecise.  For example, various claims are made about the assumptions in Theorem 1 holding or not for different experiments, but the justifications for these are never really properly explained, let alone formally demonstrated.  Furthermore, I actually do not necessarily agree that the assertions made are always correct.  At the very least, it is certainly not the case that all the conclusions of the work have been fully demonstrated.
- The paper constantly implicitly implies that a failure to satisfy the assumptions of Theorem 1 means that the associated pathologies will be avoided.  However, this logic does not hold as the assumptions in the theorem are sufficient not necessary conditions.  Moreover, I show later in the review that actually the result holds under far weaker assumptions, strongly imply that these inverse assertion that the paper relies on are likely to be false.
- The paper is written in a way that incorrectly implies various well-known behaviors are surprising insights (e.g. "the ELBO can prefer learning likelihood functions $f_{\theta}$ that reconstruct $p(x)$ poorly, even when learning the ground truth likelihood is possible!").
- Too many of the experimental results are partially relegated to the appendices.  The paper would be stronger for being more focused on certain aspects of the experiments gone through carefully (with the results themselves actually in the paper!) and some completely relegated to the appendices.  At the moment the paper feels almost unfinished because it is not at all self-contained without the supplement, which is not really acceptable in this format.
- The paper often talks in overly general terms that are not properly justified.  For example, the abstract just talks about generic "pathologies" whereas the topic of the paper is about some quite specific issues rather than a general analysis of different VAE pathologies.
- The figures are scruffy and rather difficult to read (in particular their font size is ridiculously small).

*Redundency of Theorem 1*

I believe that the core result, Theorem 1, is a rather convoluted way of making somewhat simpler points (see below) and does not provide any particular insights; its first assumption is very much reversed engineered rather than a natural starting point.  As such, it is difficult to take intuitions from it or to understand what is required to satisfy it.  Moreover, the experiments seem to demonstrate that the conditions cannot formally be demonstrated in practice as only very hand-wavy explanations are given rather than concrete demonstrations. 

Even more problematically, I believe that the Theorem itself is actually a vacuous result in light of simpler, well-known, ideas.  In short, it is obvious and well-known that we will not achieve maximum marginal likelihood (MML) solutions by maximizing the ELBO if the expected KL cannot be driven to zero at the MML value of theta (except in the bizarre edge case where the expected KL does not vary with theta).  It is also straightforward and well-known that this will give a non-zero KL(p(x)||p_{theta}(x)) except in the special case where there is an alternative likelihood that gives the same p_{\theta}(x) while allowing an expected KL of zero (interestingly, this special case may well occur with infinite data though as this allows the encoder variance to tend to zero).   As such, the result of the Theorem is somewhat obvious even without the first assumption holding: this core assumption is much stronger than it needs to be (it is only a sufficient condition, not a necessary one), which not only makes the theorem predominantly redundant, it also undermines most of the subsequent conclusions the paper derives from this result (e.g. the suggestion that pathologies will not occur if this assumption does not hold).

To demonstrate my misgivings more concretely, and verify that the result is indeed vacuous, I developed the following, which I believe to be a stronger and more intuitive alternative result (that also has the key advantage of not making any assumptions about some ground truth model)

*Alternative Theorem*

Consider a VAE with a fixed prior $p(z)$. Define the sets
$$\Theta_{MML} = argmin_{\theta} KL(p(x)||p_{\theta}(x)),$$
and
$$\Theta_{opt} = argmin_{\theta\in\Theta_{MML}} \min_{\phi} E[KL(q_{\phi}(z|x) || p_{\theta}(z|x))],$$
such that $\Theta_{opt} \subseteq \Theta_{MML}$ (note both of these will often be the same single point). If the following assumptions hold:
1. $\nexists \theta \in \Theta_{MML} : E_{p(x)}[KL(q_{\phi}(z|x) || p_{\theta}(z|x))]=0$, i.e. the encoder cannot match the posterior (this assumption is not strictly necessary but is included for intuition because the next assumption cannot hold if this one does not, while it will almost always hold in practice when this assumption does hold),
2. $\exists \theta \in \Theta_{opt} : \lVert \nabla_{\theta} \min_{\phi} E[KL(q_{\phi}(z|x) || p_{\theta}(z|x))] \rVert >0$ and $\nabla_{\theta} \min_{\phi} E[KL(q_{\phi}(z|x) || p_{\theta}(z|x))]$ is locally absolutely continuous for this $\theta$ (see discussion of this assumption below)

then the global minima of the negative ELBO, {$\theta',\phi'$}, is non-optimal for the marginal maximum likelihood in the sense that $\theta'\notin\Theta_{MML}$ and thus:
$$KL(p(x)||p_{\theta'}(x))>KL(p(x)||p_{\theta_{MML}}(x))\ge0$$
where $\theta_{MML}$ be an arbitrary element in $\Theta_{MML}$.

*Proof*:
The proof follows by considering an arbitrary $\theta$ for which the second assumption is satisfied.  Here $\nabla_{\theta} KL(p(x)||p_{\theta}(x))=0$ but $\nabla_{\theta} \min_{\phi} E[KL(q_{\phi}(z|x) || p_{\theta}(z|x))] \neq 0$ so, by our continuity assumption, it must be possible to improve the ELBO by moving $\theta$ in the direction $-\nabla_{\theta} \min_{\phi} E[KL(q_{\phi}(z|x) || p_{\theta}(z|x))]$.  However, because we are improving $\min_{\phi} E[KL(q_{\phi}(z|x) || p_{\theta}(z|x))]$, it impossible for this change to produce a $\theta$ that remains in the set $\Theta_{MML}$  (as this would imply our original point was not in $\Theta_{opt}$).  As we have a $\theta$ that improves the ELBO but is no longer in $\Theta_{MML},$ we can conclude that the optimum of the ELBO is no longer optimal from the perspective of the maximum marginal likelihood. ☐

Here that the second assumption in my Theorem above is very weak whenever the first assumption holds because it effectively equates to saying that the global optima for the MML are not all also local optima for the attainable expected KL divergence between the encoder and posterior.  For example, one would usually not expect the $\theta \in \Theta_{MML}$ to be connected, in which case the assumption can only be violated if the gradient of the attainable expected KL divergence coincidentally happens to be zero for every MML optimal $\theta$.  In light of this, my suggested theorem is making far weaker and (arguably) more intuitive assumptions than Theorem 1 in the paper, while it is also a slightly stronger final result as it does not require there to be a "ground truth" (which is a massive assumption to be able to drop).  

Now, of course, the existence of an alternative theorem does not undermine the contributions of the work in itself, but the problem here is that my result above shows that Theorem 1 in the paper is vacuous because its result effectively always holds in practice if the encoder cannot exactly match the posterior (or more precisely, $\nexists \theta \in \Theta_{MML} : E_{p(x)}[KL(q_{\phi}(z|x) || p_{\theta}(z|x))]=0$).  As such, the complex first assumption in Theorem 1 is generally not necessary and offers little insight.  Moreover, because it is obvious that optimizing the ELBO does produce the MML parameters if the encoder can match the posterior, we see that Theorem 1 provides very little insight other than this already well-known fact.  In my opinion, this severely undermines the contribution of the work.
review_title: Interesting topic for paper, but serious shortfalls in approach

Meta-review: This paper investigates various pathologies that occur when training VAE models. There was quite a bit of discussion (including "private" discussion between the reviewers) about the theory presented. Particular concerns included: For Theorem 1, while the required conditions formalise the setting in which the learned likelihoods are poor, it's unclear whether these particular conditions they are useful in practice or provided deep insight; for Theorem 2, its relevance and importance was not necessarily clear. In general the results in these two theorems are closely related to known challenges (e.g. that using the ELBO to optimise parameters may lead to bias), without necessarily providing as much new insight as one might hope.

I would note that all the reviewers included positive feedback as to the quality of the experiments, showing the impact of these pathologies on downstream tasks. However, as written much of the paper focuses on the theory — too many of the (very interesting!) figures and experimental results are relegated to the appendix.
Decision: Reject