Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks
Authors: Yaniv Yacoby
Weiwei Pan
Finale Doshi-Velez
Venue:  (2021)
Keywords: Variational Autoencoders
Variational Inference
VAE
Approximate Inference
Semi-Supervision

Abstract: Variational Auto-encoders (VAEs) are deep generative latent variable models that are widely used for a number of downstream tasks. While it has been demonstrated that VAE training can suffer from a number of pathologies, existing literature lacks characterizations of exactly when these pathologies occur and how they impact down-stream task performance. In this paper we concretely characterize conditions under which VAE training exhibits pathologies and connect these failure modes to undesirable effects on specific downstream tasks, such as learning compressed and disentangled representations, adversarial robustness and semi-supervised learning.

Under review as a conference paper at ICLR 2021
FAILURE MODES OF VARIATIONAL AUTOENCODERS AND THEIR EFFECTS ON DOWNSTREAM TASKS
Anonymous authors
Paper under double-blind review
ABSTRACT
(VAEs) are deep generative latent variable models that are widely used for number of downstream tasks. While has been demonstrated that VAE training can suffer from a number of pathologies, existing literature lacks characterizations of exactly when these pathologies occur and how they impact down-stream task performance. In this paper we concretely characterize conditions under which VAE training exhibits pathologies and connect these failure modes to undesirable effects on specific downstream tasks, such as learning compressed and disentangled representations, adversarial robustness and semi-supervised learning.
Variational
INTRODUCTION
Variational Auto-encoders (VAEs) are deep generative latent variable models that transform simple distributions over latent space to model complex data distributions Kingma & Welling (2013).
They have been used for a wide range of downstream tasks, including: generating realistic looking
synthetic data (e.g Pu et al. (2016)). learning compressed representations (e.g. MMao & Blunsom
(2016): Gregor et al. (2016); Alemi et al. (2017). adversarial defense using de-noising (Luo & Piister
2018 Ghosh et al. 2018), and, when expert knowledge is available, generating counter-factual data using weak or semi-supervision (e.g. Kingma et al. (2014); Siddharth et al. 2017); Klys et (2018)) Variational auto-encoders are widely used by practitioners due to the ease of their implementation and simplicity of their training. In particular, the common choice of mean- mean-field Gaussian (MFG) approximate posteriors for VAEs (MFG VAB) results an inference procedure that is straight-forward to implement and stable in training.
Unfortunately, a growing body of work has demonstrated that MFG- VAEs suffer from variety of pathologies, including learning un-informative latent codes (e.g van den Oord et al. (2017): Kim et al. (2018) and unrealistic data distributions (e.g. [Tomezak & Welling (017)). When the data consists of images or text, rather than evaluating the model based on metrics alone, we often rely on "gut checks" to make sure that the quality of the latent representations the model learns and the synthetic data (as well as counterfactual data) generated by the model is high (e.g. by reading generate text inspecting generated images visually (Chen et al.] 2018 Klys et al., 2018 However, as VAEs are increasingly being used in application where the data is numeric, e.g. in medical or financial domains (Pfohl et af. 2019 Doshi Doshi et al.] 2019 Way & Greene| 2017). these intuitive qualitative checks no longer apply. For example, in many medical applications, the original data features themselves (e.g. biometric reading) are difficult to analyze by human experts in raw form. In these cases, where the application touches human lives and potential model error/pathologies are particularly consequential, we need to have a clear theoretical understanding of the failure modes of our models as well as the potential negative consequences on strnn-tteam tasks.
Recent work Yacobb et al. 2020) attributes a number of the pathologies of MFG-VAES VAEs to properties of the training objective; in particular, the objective may compromise learning a good generative model in order to learn good inference model in other words, the inference model over-regularizes the generative model. While this pathology has been noted in literature (Burda et al., 2016 Zhao et al. 2017: Cremer et al. 2018). no prior work has characterizes the conditions under which the MFG- VAE objective compromises learning a good generative model order to learn a good inference model; moreover,
prior work has related MFG-VA VAE pathologies with the performance on downstream tasks. Rather, existing literature focuses on mitigating the regularizing effect of the inference model the VAE generative model by using richer variational families (e.g. Kingma et al.

---

Under review as a conference paper at ICLR 2021
(2016) : Nowozin (2018) Luo et al. )020)). While promising, these methods introduce potentially significant additional computational costs to training. as well as new training issues (e.g. noisy gradients Rocder et al. (2017); Tucker et al. (0188); Rainforth et al. (2019). As such, it is important to understand precisely when MFG-VAHS exhibit pathologies and when alternative training methods worth the computational trade-off. In this paper, we characterize the conditions under which MFG-VAEs perform poorly and link these failures to effects on a range of downstream tasks, While we might expect that methods designed mitigate VAE training pathologies (e.g. methods with richer variational families (Kingma et al. 2016)), will also alleviate the negative downstream effects, we find that this is not always so. Our observations point to reasons for further studying the performance VAE alternatives in these applications. Our contributions are both theoretical and empirical:
I. When VAE pathologies oceur: (1) We characterize concrete conditions under which learning the in- ference model will compromise learning the generative model for MFG-VARs More problematically, we show that these bad solutions are globally optimal for the training objective, the ELBO. (2) We demonstrate that using the ELBO to select the output noise variance and the latent dimension results in biased estimates. (3) We propose synthetic data-sets that trigger these two pathologies and can be used to test future proposed inference methods.
II. Effects on tasks: (4) We demonstrate ways in which these pathologies affect key downstream tasks, including learning compressed, disentangled representations, adversarial robustness and semi- supervised learning. In semi-supervised learning, we are the first to document the instance of "functional collapse", in which the data conditionals problematically collapse to the same distribution. (5) Lastly, we show that while the use of richer variational families alleviate VAE pathologies on unsupervised learning tasks, they introduce new ones in the semi-supervised tasks.
These contributions help identify when MFG-VAEs suffice, and when advanced methods are needed.
RELATED WORK
Existing works that characterize MFG-VAEs pathologies largely focus on relating local optima of the training objective to single pathology: the un-informaatveness of the learned latent codes (posterior collapse) (He al., 2019 Lucas et al. 2019: Dai al., 2019). In contrast, there has been little work to characterize pathologies at the global optima of the VAE's training objective. Yacoby al. (2020) show that, when the decoder's 's capacity is restricted, posterior collapse and the mismatch between aggregated posterior and prior can occur as global optima of the training objective. In contrast, we focus on global optima of the MFG- objective in fully general settings: with fully flexible generative and inference models, as well as with and without learned observation noise.
Previous works (e.g. Yacoby et al. (2200) have connected VAE pathologies like posterior collapse to the over-regularizing effect of the variational family on the generative model. However while there are many works that mitigate the over-regularizztion issue (e.g. |Burda et al. (2016); Zhao et al. (2017); Cremer et al. (0118) Shu al. (2018)) none have given a full characterization of when the learned generative model is over-regularized nor have they related the quality of the learned model to its performance down-stream lasks. In particular, these works have shown that their proposed methods have higher test log-likelihood relative to a MFG-VAES, but as we show in this paper, high test log- logelikelioood is not the only property needed for good performance on downstream tasks. Lastly, these works all propose fixes that require a potentially significant computational overhead. For instance, works that use complex variational families, such as normalizing flows (Kingma et al.) 2016), require significant number of parameters to scale (Kingma & Dhariwal 2018). In the case of the Importance Weighted Autoencoder (IWAE) objective (Burda al., 2016). which can be interpreted as having more complex variational family (Cremer et al.] 2017), the complexity of the posterior scales with the number of importance samples used. I Lastly, works that de-bias existing bounds (Nowozin) 2018 Luo et al. 2020) all require several evaluations of the objective.
Given that MFG and their theoretical connections to other dimensionality reduction approaches like probabilistic PCA (Rolinek et al. 2019 Dai et al.; Lucas et al., 2019). J. it is important to characterize the training pathologies of MFG- VAE, well the concrete connections between these pathologies and down- stream tasks. More importantly, this characterization will help clarify for which tasks and datasets a MFG-VAI suffices and for which the computational tradeoffs are worth it.
remain popular today due to the ease of their implementation, speed of training.
2

---

Under review as a conference paper at ICLR 2021
3 BACKGROUND
Unsupervised VA (Kingma & Welling, 2013 VAE assumes the following generative process:
p(2) = N/(0,) I), (o(2))) _V(fo()).
(1)
where x in IR 2 E RK a latent variable and fo is a neural network parametrized by 8. We learn the likelihood parameters 0 while jointly approximating the posterior pe (2(ar) with go(z|z))
p0 (2) ELBO(0, (=(a)))
0 (x) [log pe (z)) 2 max Ep(x)) 0,6
(x
log
where p(z) is the true data distribution, po is the learned data distribution, and (=(zz) is a MFG with mean and variance (z) (r)), parameterized by neural network with parameters Φ. The VAE ELBO can alternately be written as sum of two objectives the "MLE objective" (MLEO), which maximizes the (2)).. and the "posterior matching objective" (PMO), which encourages variational posteriors to match posteriors of the generative model. That is, we can write:
argming,o ELBO(O, 6) argmin, (DLL)p(<)) (p (x)] + Ep(x) [Dulgo (=l=))
(3)
MLEO
PMO
This decomposition allows for more intuitive interpretation of VAE training and illustrates the tension between approximating the true posteriors and approximating p(æ).
Semi-Supervised VAEs We extend the VAE model and inference to incorporate partial labels, allowing for some supervision of the latent space dimensions. For this, we use the semi-supervise model introduced by Kingma et al. (2014) as the "M2 model", which assumes the generative process,
NN,, I); (00 T), y y~p(y), sly, fo +
where is observed only a portion of the time. The inference objective for this model is typically writ- ten as sum of three objectives, lower bound for the likelihood of M labeled observations, lower bound for the likelihood for N unlabeled observations, and term encouraging the discriminative powers of the variational posterior:
N J(0. 0) = Σ U(zzii0,0 + Σ L(mm, Ym 0, - a Σ TF
A
M
log go (ym (mm)
(5)
m==1
where the U and lower bound po (2) and (2x y). respectively (see Appendix A): the last term in the sum is included to explicitly increase discriminative power of the posteriors ( (..) (Kingma jet al.] (2014) and (Siddharth et al. (2017)); a, y controls the relative weights of the last two terms. Note that [0, Φ) is only a lower bound of the observed data log-likelihood only when 7 = 1, a but in practice, 7, a are tuned as syper-parameters. Following Kingma et al. (2014). we assume a MFG variational family for each of the unlabeled and labeled objectives.
WHEN VAES FAIL: PATHOLOGIES OF THE VAE OBJECTIVE
Our first set of contributions characterizes when MFG-VAEs fail. In Section 4. identify two pathological properties of the VAE training objective. In Section 4.2 we introduce new synthetic benchmarks that demonstrate these pathologies and can be used test future inference algorithms.
4.1 FORMALIZING PATHOLOGIES OF THE VAE OBJECTIVE
We fix set of realizable likelihood functions F. implied by our choice of the generative model network architecture. We assume that is expressive enough to contain any smooth function, including the ground truth generating function.
When VAEs Fail I: The ELBO trades off generative model quality for simple posteriors Intu- itively, global optima of the ELBO correspond to incorrect generative models under two conditions: (1) the true posterior is difficult to approximate by a MFG for a large portion of x 's, and (2) there does not exist likelihood function fo in F with simpler posterior that approximates p(a)) well.
3

---

the learned of will not equal the ground truth noise variance if qo (2(xx, are poor Under review as a conference paper at ICLR 2021
We formalize these intuitive conditions (1) and (2) in the theorem below. First, recall the decompo- sition the negative ELBO in Equation 3 In this discussion, we always set ó to be optimal for our choice of 0. Assuming that p(z) is continuous, then for any 1y R. we can decompose the PMO as:
Bp(z) [Dxula<()/<]]lpo (=|x=]] = -rr[ Exl [
(6)
+ Pr[[Kin (0) Ep(=)) [Dxr (=x_lloa((l))lllp(l11))
where [g. (=x]lpo(/=)]]pp(((()) S on (0), DKL. (g (2 (x)) [[]xx] > 77 on (0), with x;(0) C
X: where Ex(=)l,, is the expectation over p(r) restricted (0) and renormalized, and Pr[A[] is
the probability of (0) under p(ar). Let us denote the expectation first term on the right hand side of Equation 6 as (o) and the expectation in the second term as (ni(().
Let foor E F be the ground truth likelihood function, for which we may assume that the MLE objective (MLEO) term is zero. Now conditions (1) and (2) above may be rewritten more formally as:
Theorem 1. Suppose that there is an E R such that Pr[MHi (fcr) Dui(err is greater than Pr[ALo ((GT) DLo Suppose the following two conditions: (1) [True posterior often difficult]
there exist an fo E F with Dr.o (BcT) 2 (8) and
PH[ (OGT) (DH(OGT) - -Lo((CCT) > Pr[KH((O)] (Hi + Dxulp(=)
and (2) INo good, simpler alternative] that for no such fo E F is the MLEO DRL zero. Then the global minima (8", .²) of the negative ELBO, the MLEO will be non-zero.
equal
Theorem 1 shows that under conditions (1) and (2) the ELBO can prefer learning likelihood functions So that reconstruct p(x)) poorly, even when learning the ground truth likelihood is possible! While the proof is straightforward (Appendix BB.1) it is non-trivial to show that the conditions (1) and (2) are satisfied on actual datasets- we do this in Section 4.2
When VAES Fail II:
The ELBO biases learning of the observation noise variance In practice,
the noise variance of the dataset is unknown and it common to estimate the variance as a hyper- parameter. Here, we show that learning the variance of either via apper-prrameter search or via direct optimization of the ELBO can be biased.
Theorem 2. For an observation set of size N, we have thut
N
argmin ELBO(A, (1)) ELBO( d602
Σ n=1
[0000 - -
N
Proof in Appendix B.2 The above theorem shows that the variance of that minimizes the negative ELBO depends on the approximate posterior go (2|z), and thus even when the generative model is set the ground-truie approximations of the true posterior. Certainly, when the learned generative model does not capture p(x) (say, due to the conditions in Theorem then the learned a² will again be biased.
While the pathologies in Theorems they are actually artifacts of the ELBO. There are objectives that use MFG familics to obtain unbiased estimates of the ground truth generative model and o² (Finke & Thiery 2019
Remark:
seem to depend on our choice of variational family,
4.2 BENCHMARKS TO DEMONSTRATE OF VAE TRAINING PATHOLOGIES
We provide novel benchmarks to demonstrate the pathologies described in Theorems ! and 2 These benchmarks not only provide intuition for these failure modes, they can also serve as benchmarks for testing future VAE objectives and inference. Finally, while examples here are synthetic order to provide intuition for general failures on down-stream tasks, in Section 6 we describe how each example corresponds to a class of real datasets on which VAEs can exhibit training pathologies.
Experimental setup To show that an observed failure is due to pathologies identified in Theorem we verify that: (A) the learned models have simple posteriors for high mass regions where the ground truth models do not, (B) training with IWAE (complex variational families) results in generally superior generative models and (C) the VAE training cannot be improved meaningfully by methods designed to escape bad local optima, i.e. Lagging Inference Networks (LIN) (He et al. 2019). We
4

---

Under review as a conference paper at ICLR 2021
ao a
(a) Figure-8. Left: true, Mid: IWAE, Right: VAE.
(b) Circle. L: true, both.
Clusters. Left: true, Mid: IWAE, Right: VAE. (d) Absolute-Van Valuue L: true, R: both.
Figure I: Comparison of true data distributions versus the corresponding learned distributions of VAE and IWAE. When conditions of Theorem are satisfied, examples (a) and (c), VAE training approximates p(x) poorly and IWAE performs better. When one of the conditions is not met, in examples (b) and (d), then the VAE can learn p(x) as well as IWAE.
easily control the complexity of the implied variational family using the number of importance samples S, and in doing so we encompass other types of variational families. We ensure that IWAE does not suffer from training issues (e.g. low signal-to-noise ratio) by ensuring < 20 and by constructing all toy examples with ID latent spaces.
specifically chose IWAE baseline since
We fix flexible architecture (one that is significantly more expressive than needed to capture S so that our realizable set F is diverse enough to include likelihoods with simpler posteriors. We train each model to reach global optima as follows: for each method and arper-parameter settings, we train 5 random initialized randomly, and 5 random with the decoder and encoder initialized to ground truth values. We select the restart with the best value of the objective function. Details in Appendix C
approximation of p(z) is poor when Conditions (1) and (2) of Theorem both hold. Consider the "Figure-8" Example Figure Ta (described Appendix F.T For this dataset, values of in [-o -3.0] U [3.0, oo] map similar values of near (0, 0), where p(x:) is high. We verify that, near æ = (0, 0), the posteriors Pear (z lax:) are multi-modal, satisfying condition (1). We verify condition (2) is satisfied by considering all continuous parametrizations of the "Figure-8" curve: any such paramctrization will result a function fe for which distant values of map to similar values near (0, 0) and thus the posterior matching objective (PMO) will be high. As predicted by Theorem I the learned generative model approximates p(x) poorly, learning posteriors that are simpler than those of the ground truth model.
Benchmark:
To show that these issues occur because the MFG variational family over-regularizes the generative model, compare VAE with LIN and IWAE. As expected, IWAE learns p(x)) better than LIN, which outperforms the VAE (Figure Tap. Like the VAE, LIN compromises learning the data distribution in order learn simpler posteriors, since also uses MFG variational family. In contrast, IWAE is able to learn more complex posteriors and thus compromises p(z) far less. However, note that with 20 importance samples, IWAE still does not learn p(x) perfectly. See Appendix [D.1 for full quantitative analysis as well as more data-sets for which Theorem I holds. Next, we show that both conditions of Theorem are necessary for the VAE training pathology to occur.
approximation of p(x)) may be fine when only condition (2) holds. What happens the observations with highly -an-Gaussian posterior were few in number? Consider the "Circle" Example in Figure lb (described in Appendix F22. Here, the regions that have non-Gaussian posteriors are near r Rs (1.0, 0.0). since z E [-00, 3.00 U [3.0, oo map to points near (1.0, 0.0). However, since the overall number of such points is small, the VAE objective does not trade-off capturing p(z) for easy posterior approximation. Indeed, we see that VAE training capable of recovering p(æ), regardless of whether training initialized randomly at the ground truth.
Benchmark:
Benchmark: approximation of p(z) may be fine when only condition (1) holds. We now study the case where the true posterior has high PMO for large portion of z's, but there exist fo in
5

---

et al.. Under review as a conference paper at ICLR 2021
realizable set F that approximates p(z) well and has simple posteriors. Consider the "Absolute- Value" Example visualized in Figure Id] Although the posteriors under the ground truth generative model complex, there is an alternative likelihood f=(2) that models p(z) equally well and has simpler posteriors, and this is the model selected by the VAE objective, regardless of whether training was initialized randomly at the ground truth. Details in Appendix FF3
Benchmark: Theorem 2 implies that the ELBO biases noise variance estimates. Consider the "Spiral Dots" Example in Appendix F.5 We perform two experiments. In the first, we fix the noise variance ground-truth (o: = 0.01), we initialize and train 0, e following the experimental setup above, and finally, we recompute of that maximizes the ELBO for the learned 8, o. In the second experiment, we do the same, but train the ELBO jointly over o² 0 and Φ. Using these two methods of learning the noise, we get 0.014 + 0.001 and 0.020 + 0.003, respectively. The ELBO therefore over-estimates the noise variance by 50% and 100%, respectively.
5 IMPACT OF PATHOLOGIES ON DOWNSTREAM TASKS
In Section 4 we described when VAE pathologies occur, both theoretically and through new bench- marks. Now, we describe how how these pathologies can negatively impact specific downstream tasks. On unsupervised tasks, we show that IWAE can avoid the negative effects associated with the MFG variational family over-regularizing the generative model, while LIN cannot. However, surprisingly, IWAE cannot outperform the VAE on our semi-supervised tasks as its complex variational family allows the generative model to overfit.
Experiment Setup On unsupervised tasks, we consider only synthetic data, since existing work shows that on real data IWAE learns generative models with higher log data likelihood (Kingma 2017). For our semi-supervised tasks, we consider both synthetic data as well as 3 UCI sets: Diabetic Retinopathy Debrecen (Antal & Hajdu 2014). Contraceptive Method Choice (Alcala-Fdez et al. 2010; Dua & Graff 2017) and the Titanic (Alcala-Fde
al. 2016 Cremer et al.
2010, [Simonoff] 1997) datasets. In these, we treat the outcome partially observed label (observed 10% of the time). These datasets are selected because their classification is hard, and as we will show here, this is the regime in which we expect semi-supervised VAE training to struggle.
5.1 EFFECTS ON UNSUPERVISED DOWNSTREAM TASKS
Failure due to Theorem 1 In disentangled representation learning, we suppose that each dimension of the latent space corresponds to task-meaningful concept (Ridgeway) 2016 Chen et al. 2018). Our goal is to infer these meaningful ground truth latent dimensions. It's been noted in literature that this inference problem is ill-posed that is, there are an infinite number of likelihood functions (and hence latent codes) that can capture p(x) equally well (Locatello et al., 2018). In Appendix D.2, we show that, more problematically, the VAE objective can prefer learning the representations that entangles the ground-trus truth latent dimensions due to the pathology in Theorem [
Disentangled Representations:
Compressed Representations: Failure due to Theorem 1 In practice, if the task does not require specific latent space dimensionality, K, one chooses a K that maximizes the log po (æ). In Appendix D.3 we show that using larger K and a smaller o² means we can capture the data distribution with a simpler function fz(z) and hence get simpler posteriors. That is, increasing K alleviates the need compromise the generative model in order to improve the inference model and leads to better approximation of p(z). Thus, the ELBO will favor model mismatch (K larger than the ground truth) and prevent us from learning highly compressed representations when they are available. We provide two examples where the BI. prefers models with larger K over the ground truth model (K = 1), and that as K increases, the average informativeness of each latent code decreases, since the latent space learns to generate the observation noise e. We confirm that the posteriors become simpler K increases, lessening the incentive for the VAE to compromise on approximating p(x). Also, while LIN also shows preference for higher K"s,, IWAE does not.
Defenses against adversarial perturbations: Failure due to Theorems and 2 Manifold based defenses against adversarial attacks (e.g. Dalal et al. (2017); Meng & Chen (017) Samangouei et al. (2018); (Hwang et al. (2019); : Jang et al. (2200) require both accurate estimates of the noise as well as of the intrinsic dimensionality of the data (i.e. the ground truth latent dimensionality): however,

---

Under review as a conference paper at ICLR 2021
Theorem 2 shows that the ELBO is unable to identify the correct o at correct latent dimensionality K, and incorrect compression (above, due to Theorem I may further result in incorrect noise estimates due to incorrect ground truth latent space dimensionality. See Appendix E for full analysis.
(a) True p(x)
(b) True vs. Learned foly, 2). Left: true, Mid: IWAE, Right: VAE.
Figure 2: Discrete Semi-Circle. Comparison of VAE and IWAE for semi-supervision. The ground truth likelihood shows two distinct functions, for y 0, 1. The VAE likelihood is over-regularized by simple variational family and learns two nearly identical functions ("functional collapse"). The IWAE likelihood function is unregularized and learns two distinct but overfitted functions.
5.2 EFFECTS ON SEMI-SUPERVISHI DOWNSTREAM TASKS
In semi-supervised VAEs, we assume that some observations x have an additional label (Kingma al. [0144. These VAEs have been used for tasks such as generating synthetic cohorts (sampling from p(z|y- = 1). p(zz [zly = 0) respectively), and for generating counterfactuals (generating synthetic data x' with label y = 0 that is similar to a real observation æ with y = 1). For these tasks, is important to accurately model the data conditional p(xly). Surprisingly, Theorem leads to negative effect specific to semi supervsse tasks, a phenomenon we call "functional collapse": the model ignores the partial labels given by the semi-supervision, causing the learned conditionals to collapse onto a single distribution, po (x) Rs po (xly = 0) s po(xly = 1). While one might expect methods like IWAE fix this issue, IWAE actually overfits the few partial labels due its rich variational family, causing it to perform no better than a MFG VAE. In this section, we characterize when functional collapse occurs, well as how it impacts semi-supervised downstream tasks.
Due to Functional Collapse, VAES trade-off between generating realistic data and realistic counterfactuals In real datasets, we often have samples from multiple cohorts of the population. General characteristics of the population hold for all cohorts, but each cohort may have different distributions of these characteristics (Klys et al. 2018). Formally, this means that all of the cohorts to lie on a shared manifold but each has a different distribution on that manifold (that is xryy 0 and æly = 1 lie on the same manifold but p(rly = 0) - p(rly = 1)). On such data, the ground truth model's posterior for the unlabeled data pear (z|xx) = Jy Peer ylr)dy will be multi-modal, since for each value of y there are number of different likely z's, each from different cohort. As such, using a MFG variational family for the unlabeled portion of the semi-supervised objective (l in Equation 5) will encourage inference to either compromise learning the data-distribution in order to better approximate the posterior, or to learn the data distribution well but approx the posterior poorly, depending on our prioritization of the two objectives (indicated by our the choice of the hyperparameter y in Equation 5) In the first case, data generation will be compromised because the model will overfit the partial labels; however the model will at least be able generate from two distinct data conditionals p(zly = 0) and p(zly 1). In contrast, in the latter case the learned model will be able to generate realistic data but not realistic cohorts since the model will over over-regularize the likelihood function fo(2, y) to collapse to the same function for all values of (functional collapse). thereby collapsing the data conditionals po (ar yy) R p(r).. That is, p(xly) will generates identical looking cohort regardless of our choice of y.
We empirically demonstrate the trade-off between realistic data and realistic counterfactuals gen- eration on the "Discrete Semi-Circle" Example in Figure 2a (full details in Appendix G.I.. In this data-set, we show that the MFG-VAE is able to learn the data manifold and distribution well. However, the training objectives leams a model with simple posterior (in comparison to the true posterior), causing the learned fo(2, y) to collapse to the same function for all values of y (Figure 2b). As result, (o(v))) R p1() under the learned model. As expected, functional collapse occurs
7

---

Under review as a conference paper at ICLR 2021
when training with LIN as well. In contrast, IWAE is able to learn two distinct data conditionals po(yyy = 0), po (zly = 1). but it does so at a cost. IWAE does not regularize the generative model, and thus overfits to the few partial labels (Figure 2b). Lastly, IWAE learns p(z) considerably worse than the VAF, while learning p(zly) significantly better. In Appendix D.4 we provide a full quanti- tative and qualitative analysis of the above synthetic and real datasets. We also demonstrate an additional pathology for the case where is continuous, in which a naive choice of the discriminator go go(ylar ()) causes functional collapse even in IWAE.
IMPLICATIONS FOR PRACTICE
In this paper, we have presented two sets of contributions that advance our fundamental understanding of VAEs: first, we described the theory of when pathologies occur and introduced benchmarks to expose them, second, we described the impact of these pathologies on common downstream tasks. Now, we connect these insights with implications for using VAEs in practice.
How MFG-VAE training pathologies manifest in many real datasets. Our previous empirical demonstration of the pathology in Theorem ! are synthetic; here we describe how the conditions of Theorem D can manifest in real datasets. The "Figure-8" Example in Figure Ta generalizes to any data manifold with high curvature (e.g. images from videos showing continuous physical transformations of objects). i.c. where the Euclidean distance between two points in a high density region on manifold is (A) less than the length of the geodesic connecting these points and (B) within 2 standard deviation of observation noise. The "Clusters" Example in Figure Ic generalizes to cases where we learning -wwdimensioonl representations of multimodal data distributions (e.g. popular image datasets where similar images lie in clusters). On these datasets, the VAE training objective would prefer compromising the quality of the generative model for posteriors that are easy to approximate. For the pathology noted in Theorem 2 we expect that the ELBO yields biased estimates of the observation noise whenever the learned model approximates p(z) poorly. Finally, we expect the MFG-VAE's performance on semi-supervised downstream tasks to suffer when a neural network is unable to accurately predict y|ær. Difficulty in training a good classifier indicates that aly = 0 and r lie roughly on the same manifold, and are difficult to distinguish, leading to the pathologies described in Section 5.2
How to make inference choices We make three simple guidelines for practitioners when using VAEs in order to avoid the pathologies described in this work. While the guidelines are simple, our work provides formal rationales for why these practices matter (and we note that these best practices are not always used - c.g. it is common to set o² without examining the data-set, or to learn by optimizing jointly with model parameters (Lucas et al. 2019)). Finally, others have noted (Finke & Thiery 2019). a single methodological innovation is unlikely to fix all issues - each innovation makes specific tradeoff; thus, improvements will need to be task/data specific. Our guidelines are: (1) Set the noise variance o² using domain expertise, or by hyper-paranee low-variance log-likelihood estimator. (2) Investigate the topology of the data (e.g. using topological data analysis, dimensionality reduction) before choosing a variational family. If the data lies on a manifold in distorted Euclidean space (e.g. "Figure-8" Example), or if the data is clustered (c.g. "Clusters" Example), use a rich variational family if you need to learn a very low-dimensional latent space. (3) Whenever using a rich variational family, apply regularization (e.g. f2) to the decoder network weights to prevent overfitting. Lastly, (4) on semi-supervised tasks, use a rich variational family for go (2(2x) in the unlabeled data objective (ll in Equation when a simple neural network predicts y| je with low balanced-accuracy (and use a MFG variational family otherwise).
selection with an unbiased
7 CONCLUSION
In this work we characterize conditions under which global optima of the MFG-VA VAE objective exhibit pathologies and connect these failure modes to undesirable effects on specific downstream tasks. We find that while performing inference with richer variational families (which increases training time) can alleviate these issue on unsupervised tasks, the use of complex variational families introduce unexpected new pathologies in semi-supervised settings. Finally, we provide a set of synthetic datasets on which MFG- -VE exhibits pathologies. We hope that these examples contribute to a benchmarking dataset of "edge-cases" to test future VAE models and inference methods.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jesus Alcala-Fdez, Alberto Fernández, Julián Luengo, J. Derrac, S Garcia, Luciano Sanchez, and Francisco Herrera. Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. Journal of Multiple- Valued Logic and Soft Computing, 17: 235-287, 01 2010.
Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy. Fixing Broken ELBO, arXiv e-prints, art. arXiv:1711.0000 100444, November 2017.
Balint Antal and Andras Hajdu. An ensemble-based system for automatic screening of diabetic retinopathy. arXiv e-prints, art. arXiv:1410.8576, October 2014.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance Weighted Autoencoders.
arXiv: Seti::1509000510 00519 [cs, stat], November 2016. URL fttp:///axiv.orr//bbb3..
00519
arXiv: 1509.00519.
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating Sources of Disentan- glement in Variational Autoencoders. arXiv e-prints, art. 14X44:180..00942, February 2018.
Chris Cremer, Quaid Morris, and David Duvenaud. Reinterpreting Importance- Au-
toencoders. arXiv:1704.000 02916 [stat], August 2017. http:::ppaa....rgppppppoapp
02996 arXiv: 1704.02916.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference Suboptimality in Variational Autoen- coders. I8X1 103558 [cs, stat], May 2018. URL. http::/arxiv. wwtp:xiwwwww..../ooo 03558
arXiv: 1801.03558,
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Connections with Robust PCA and the Role of Emergent Sparsity in Variational Autoencoder Models. pp. 42.
Bin Dai, Ziyu Wang, and David Wipf. The Usual Suspects? Reassessing Blame for VAE Posterior Collapse. arXiv e-prints, art. 11Xiv:1912. 10702, December 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository. 2017. URL http::/archive wes.uci.eoo
Axel Finke and Alexandre H. Thiery. On inppoaancee antonccceer.. arXiv:/907. 104977 [cs, stat], September 2019. URL (tt.:tttp//aarriirrr///////00000 arXiv: 990770447
version: 2.
Partha Ghosh, Arpan Losalka, and Michael J Black. Resisting Adversarial Attacks using Gaussian Mixture Variational Autoencoders. arXiv e-prints, art. 1XX::1806.0005: May 2018.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wier- stra. Towards conceptual compression. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3549-3557. Curran Associates, Inc., 2016. URL htt.:htpp//// pttt:://papers.. nips.npppicoc..coo www.cc/ppppp..... 6542 towattowaddd-ooo
pdf
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatriek. Lagging Inference Networks and Posterior Collapse in Variational Autoencoders, arXiv:1901.055:: [cs, stat], January 2019. URLatt:://aaae
org. / abs/1901. 05534 arXiv: 1901.05534.
Uiwon Hwang, Jaewoo Park. Hyemi Jang. Sungroh Yoon, and Nam Ik Cho. PuVAE: A Variational March 2019.
Autoencoder to Purify Adversarial Examples. arXiv e-prints, art.
Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G. Dimakis. The Robust Manifold Defense: Adversarial Training using Generative Models. arXiv e-prints, art. arXiv:1112.099 December 2017.
Uycong Jang, Somesh Jha, and Susmit Jha. ON THE NEED FOR TOPOLOGY-AWARD GENERA- TIVE MODELS FOR MANIFOLD-BASED DEFENSES. pp. 24, 2020.

---

Under review as a conference paper at ICLR 2021
Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. Towards realistic individual recourse and actionable explanations in black-box box decision making systems. arXiv preprint arXiv: 110.v::997.09911.1 2019.
Yoon Kim, Sam Wiseman, Andrew C. Miller, David Sontag, and Alexander M. Rush. Semi-Amortized Variational Autoencoders. arXiv e-prints, art. arXiv:1802.02550 February 2018.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolu- tions. httv:/807.0000 [cs, July 2018. UKL/ttts///aaxiiv.rr//aa//////// 000. 03039
arXiv: 1807.03039.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv e-prints, art. arXiv:1312.6114 December 2013.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling, Suni--uperviied Learning with Deep Generative Models, arXiv e-prints, art.
June 2014.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving Variational Inference with Inverse Autoregressive Flow. arXiv e-prints, art. arXiv: (0066666644444 June 2016.
Jack Klys, Jake Snell, and Richard Zemel. Learning Latent Subspaces in Variational Autoencoders. arXiv e-prints, art. arXiv: 21111119..,,,.,.11111111 December 2018.
Jack Klys, Jake Snell, and Richard Zemel. Learingg latent subspaces in variational autoencoders. In Advances in Neural Information Processing Systems, pp. (444-6454, 2018.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disen- tangled Representations. arXiv e-prints, art. utXiv:1811.12333 November 2018.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Don't Blame the ELBO! Linear VAE Perspective on Posterior Collapse. arXiv e-prints, art. arXiv:1911 1..2499,,,, November 2019.
Yi Luo and Henry Pfister. Adversarial Defense of Image Classification Using a Variational Auto- Encoder. arXiv e-prints, art. arXiv: 1tt::1812.02200 812.02891, December 2018.
Yucen Luo, Alex Beatson, Mohammad Norouzi, Jun Zhu, David Duvenaud, Ryan Adams, and Ricky T. Q Chen. SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models. arXiv:2004. 00353 [cs, stat], April 2020. URL httt:://aaxii/ http://axxvvv..r/aab.s.e org/abs /2004 00353
arXiv: 2004.00353.
Dongyu Meng and Hao Chen. MagNet: Two-Pronged Defense against Adversarial Examples. arXiv e-prints, art.
May 2017.
Yishu Miao and Phil Blunsom. Language as a Latent Variable: Discrete Generative Models for Sentence Compression. arXiv e-prints, art. arXiv:1609.07317, September 2016.
Sebastian Nowozin. DEBIASING EVIDENCE APPROXIMATIONS: ON IMPORTANCE. WEIGHTED AUTOENCODERS AND JACKKNIFE VARIATIONAL INFERENCE. pp. 16, 2018.
Stephen Pfohl, Tony Duan, Daisy Yi Ding, and Nigam H Shah. Counterfactual reasoning for fair clinical risk prediction. arXiv preprint arXivv1907.066: 1907,0620.0 2019.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, la- bels and captions. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2352- 2360. Curran Associates, Inc., 2016. URL 6528 1228variatiioall -ww.-aacccee--eaaaaaaeae for -eor-deep-leade orgrroode--------eeee--aaaa-o---
niws. wwt.:iiiiss..cceeeeo
-abeees--aablss-e and-captions.
pdf
10

---

Under review as a conference paper at ICLR 2021
Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter Variational Bounds are Not Necessarily Better. arXiv: 04537 [cs, stat], March 2019. URL/tttp://axxte org abs www.b///88..cc0 04537
arXiv: 1802.04537.
Karl Ridgeway. A Survey of Inductive Biases for Factorial Representation-Lcamming arXiv e-prints, December 2016.
art.
Geoffrey Roeder, Yuhuai Wu, and David Duvenaud. Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference. arXiv e-prints, art. 1rXiv:1703.09199 March 2017.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational Autoencoders Pursue PCA Directions (by Accident). arXiv: 12....122...6.. 06775 [cs, star], April 2019. /tt/httpp://arxl..orr/
abs/1812.06773
arXiv: 1812.06775.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models, arXiv e-prints, art. arXiv:1805.c00 880500600, May 2018.
Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer, and Stefano Ermon. Amortized Inference Regularization.
Bengio, Wallach, H. Larochelle, Grauman, Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 4393-4402. Curran Associates, Inc., 2018. URL
N. Siddharth, Brooks Paige, Jan-Willem de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, and Philip H. S. Torr. Learning Disentangled Representations with Semi-Supervised Deep Generative Models. arXiv e-prints, art. MiXiv:1706.004000 June 2017.
N. Siddharth, Brooks Paige, Wan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, and Philip H. S. Torr. Learning Disentangled Representations with Semi-Supervised Deep Generative Models. arXiv:1706.. 00400 [cs, stat]. November 2017. URL
http:://axxiv httpgg/ab//77..0 00400
arXiv: 1706.00400.
Jeffrey Simonoff. The "unusual episode" and second statistics course. Journal of Statistics Education, 5, 03 1997. doi: 10. 1080/16616 (96699999819999 199.1997..111109
Jakub M. Tomczak and Max Welling. VAE with VampPrior, arXiv e-prints, art. arXiv: 1uXiv:1705.00 170.00710.0 May 2017.
George Tucker, Dieterich Lawson, Shixiang Gu, and Chris Maddison. Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives. arXiv:1810.044 [cs, stat], November 2018 URL http::ttaaxxi..ooooo org. abs/ 04152
arXiv: 1810.04152.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. arXiv e-prints, art, arXiv:1711.00993 November 2017.
Gregory P Way and Casey Greene. Extracting biologically relevant latent space from cancer transcriptomes with variational autoencoders. BioRxiv, pp. 174474, 2017.
Yaniv Yacoby, Weiwei Pan, and Finale Velei-Velez. Characterizing and Avoiding Problematic Global Optima Variational Autoencoders. arXiv e-prints, art. arXiv:2003.07756, March 2020.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards Deeper Understanding of Variational Autoencoding Models. arXiv: 00...702.000. 08658 (cs, stat), February 2017. URL http:/ttt/// /ttt::///axiv
brq//bs//700. 08658
arXiv: 1702.08658.
11


Reviews:
[{"review_id": "kof5q9Qhxx0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents a characterization of failure modes of Gaussian VAEs. It is known that Gaussian VAEs can fail to produce good models either by failing to match the data distribution or by learning latent variables that are uninformative. The paper builds upon prior work that suggests that the VAE objective can cause the inference model to over-regularize the generative model. The paper characterizes the conditions under which this over-regularization occurs with corresponding. Furthermore, the paper examines the affect of VAE pathologies on downstream tasks a number of unsupervised and supervised downstream tasks.\n\n----------------------------\n\nThe paper is clearly written and the authors have attempted cover all cases of the theoretical reasons they have identified for VAE failures with corresponding experiments. The attempt to cover a number of downstream tasks is also a positive since this is a main goal of unsupervised learning.\n\nAs for the core of the paper the authors have identified two types of VAE failures: 1) when the generative quality is traded off for simple posteriors and 2) when the learning of output variance is biased. The first case is presented as occuring when the true posterior is difficult to estimate and there is no good likelhood function with a simple posterior. An expression of this is given as theorem 1.\n\nThis brings me to my main concern: Theorem 1 seems to say that if 1) the model cannot match the true posterior (Gaussian assumption is false) and 2) the model cannot match p(x) without matching the true posterior (decoder is not too powerful) then the model will not match p(x). If this is the case, then I would suggest that the statement does not present any new insight into VAE failures since a good VAE model of p(x) would either match the true simple posterior or it will model p(x) without matching the posterior.\n\nAnother concern is that although the authors mention prior work on posterior collapse, they do not consider this when characterizing the conditions that cause pathological behvaviour in VAEs. In particular it seems that the \"No good, simpler alternative\" condition explicitly excludes the case of a strong decoder that is able to ignore the latent code to model p(x) directly. This seems to be at odds with the authors' claim that they consider \"fully flexible generative and inference models\" when comparing with the prior work of Yacoby et al. Also, the case of strong decoder able to ignore the latent code is difficult to see as a case of the inference model over-regularizing the generative model.\n\nIf my understanding above is correct, then I suggest that the authors revise their claim of considering fully flexible models. In particular, they should state whether or not the case of posterior collapse is handled by their characterization.\n\nThe authors also present theorem 2 which suggests that using the ELBO to choose output variance results in a biased estimate. As far as I can tell this is new and of interest.\n\nThe authors present a number of experiments to show VAE failures: when the true posterior is far from Gaussian, when the true posterior is Gaussian and the decoder is weak, when the posterior is far from Gaussian and but the decoder is strong enough. They also show experiments to show that the output noise is overestimated when the ELBO is used to compute it. One limitation is that the authors have chosen very simple datasets with 1-d latent spaces for illustration. This leads to another question of interest of whether there are problems with VAEs that only appear with higher dimensional latent spaces?\n\nOverall I found the paper to be interesting but in light of the questions mentioned above I do not recommend acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gM5zKauTA7j", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper exposes the pathologies of VAEs and characterizes them with concrete conditions. The synthetic data experiments well summarize the conditions that we might meet in real-world applications. The authors also analyze the corresponding effects for specific downstream tasks and give insightful suggestions to avoid these problems.\n\nThe trade-off between the generative distribution and inference distribution in VAEs has been studied for a long time and has been revealed from several perspectives such as information theory, etc. It is good to see in this paper that the two conditions in theorem 1 summarize well why VAEs work well or poorly.  \n\nQuestions:\nFor theorem 1, when only condition 1 holds, we know $p(x)$ might still be approximated but we may get an unwanted generative distribution since there exist several generative models that fit the data equally well. Is there any solution to avoid the duplicated solution?\n\nIn the paper, IWAE is applied to avoid the issues mentioned in VAEs. However, in Figure 9, the learned generative model seems to be different than the ground truth, and the posterior is thus simpler than the true posterior. Is there any possible explanation for this multi-modal case? \n\n---\n\nIn general, this is an interesting paper that well analyzes the training issues in VAEs and provides insightful guidance to the VAE studies. The paper is well written (the figure labels are too small to read) and easy to follow. \n\n\n----\nUpdate after the discussion stage\n\nI appreciate the authors' responses to address my questions in the experiments. However, I agree with the concerns of the other reviewers, especially the redundancy of Theorem 1 raised by Reviewer3. After reading the reviews and the discussion, the authors seem not to provide convincing responses to this part and this raises my concerns for this paper. Thus I change my rating below the borderline.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_LCQFE6rblS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents two analysis: (1) Characterization of when the training of VAEs using the ELBO leads to suboptimal generative models (biased towards ones with simple posteriors); and (2) How this suboptimality may affect downstream tasks that use the learned models. Specifically, the work focuses on VAEs using mean-field Gaussians as variational distributions, and explores how their limited modeling capacity affects the final generative model learned. They present some theoretical results and simple and illustrative scenarios. In addition, they present an analysis regarding how the suboptimal models learned affect other tasks, such as learning disentangled or compressed representations.\n\n\nPros:\n- The paper is very clearly written.\n- The theoretical results are novel, although based on well known ideas and maybe not very relevant from a practical perspective (see \"cons\").\n- It presents an interesting exploration of how the known failures of VAEs affect subsequent tasks that use the suboptimal models. I think this is quite relevant, and often does not receive as much attention as new training methods for VAEs.\n\n\nCons:\n- As mentioned in the paper, the failure modes of VAEs are known. The paper's first contribution is a characterization of when they happen. While the theorems give precise conditions and expressions, it is not clear to me whether they are useful in practice for real scenarios. I think most of the analysis of downstream tasks do not really require the precise conditions given in the theorems, but the ideas behind them, which were already known. For instance, the analysis regarding disentangled representations states that given several models with equal likelihood, optimization will choose the one with lower KL divergence (which may be disentangled or not, depending on the scenario). This does not require the conditions from theorem 1, but knowing that for a given log p(x), decreasing the KL divergence increases the ELBO (which is known, given that the tightness of the ELBO bound is exactly the KL divergence).\n- Something similar happens with the analysis regarding compressed representations. Here, it is mentioned that increasing K (dimensionality of latent space) leads to simpler functions and thus to simpler posteriors, and is therefore preferred when using mean-field Gaussians approximating distributions. Again, the precise conditions from the theorems do not seem to be used here. (To the authors, please let me know if I am wrong; that is, if the precise results from the theorems are actually used in these analysis and I missed it.)\n- The relevance of the result in theorem 2 is not clear to me. The fact that the optimal noise parameters is not recovered is expected (it is known that optimizing the ELBO leads to biases in the parameters of the generative model), albeit the expression that precisely quantifies the bias in terms of the approximating distribution was not known. However, this exact expression is not used in the subsequent analysis, only the fact that it is biased/suboptimal. Thus the relevance of this result is not clear to me.\n- I think that some parts of the second analysis could use further justification. For instance, consider the compressed representations part. It is not clear to me why increasing K leads to generative models with simpler posteriors. I saw the empirical analysis in the Appendix, but did not find an analysis justifying the claim. Am I missing something simple here?\n\n\nAll in all, I do not find the first part of the analysis to be very relevant to the community. This is not the case find the second part of the analysis; I consider that studying how these suboptimalities affect other downstream tasks is important. Despite the fact that I believe the analysis could be expanded, I think it represents a first step in this direction. Therefore, I'm inclined to recommend acceptance. (I updated the score after the discussion.)\n\n\nOne last comment, I would suggest using larger fonts in all plots.", "extra_scores": {}, "extra_text": {}}, {"review_id": "B3A2cuPoz2i", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper looks to investigate when VAEs fail to learn the maximum marginal likelihood (MML) model and some of the implications this can have for downstream tasks.  In particular, it introduces a theorem (Theorem 1) that provides assumptions under which MML will not be found, and then performs experiments to assess behavior in scenarios where the authors believe these assumptions are satisfied or not.\n\nThough the topic of the paper is interesting and the long term aims are a good line of research, I believe the actual approach taken is rather misguided and that the arguments and conclusions are not properly supported.  In short, I do not believe that the key claimed contribution of describing *when* pathologies occur in VAEs is actually accurate or that the paper adds notable additional insight on this compared to previous work.  As such, I do not believe it is suitable for publication at ICLR in its present state.\n\n*Strengths*\n- The problem the work is trying to tackle is important.\n- The supplement is very comprehensive and a lot of experiments have been run.\n- The paper is generally well written (though the hand-waviness of some of the arguments and reliance on the supplement do significantly detract from the clarity in the latter sections).\n- The work is mostly well referenced.  One important missing reference that should be added is  https://arxiv.org/abs/2006.10102 which already makes important related arguments about trade-offs in M2-style semi-supervised VAEs (in particular, they discuss the fact this type of semi-supervision imposes a mismatch between the marginal posterior q(z) and the prior p(z), which closely relates to this paper's discussion of mismatch between p(z|x) and q(z|x) but without needing to assume a ground truth posterior).  Direct discussion about the fact that a VAE need not learn a \"ground truth likelihood\" to perfectly match the data distribution should also be added (see e.g. http://ruishu.io/2017/01/14/one-bit/).\n- Though I am not sure I agree with all the associated conclusions or assumptions being made, it is clear that the authors have put noticeable effort into trying to link the findings of the paper to practical implications.\n\n*Weaknesses*\n- I believe the core result of the paper (Theorem 1) is redundant (see below).\n- The work makes lots of unnecessary and restrictive assumptions about a \"ground truth\" model that do not match up to typical real-world situations where such a model does not generally exist (in particular, the latent variables are generally an arbitrary construction such that the concept of a ground truth is actually meaningless).  Moreover, it is already well established that VAEs are not, in general, capable of uncovering a ground truth model even when this does exist (due to equivalence classes amongst other things); the paper offers little beyond this already well-known fact.  In fact, it is written in a way that often implies that we would expect such a ground truth to be uncovered even though various works have explained why this is not a reasonable expectation (e.g. Locatello et al 2019).  \n- Most of the paper is extremely hand-wavy and imprecise.  For example, various claims are made about the assumptions in Theorem 1 holding or not for different experiments, but the justifications for these are never really properly explained, let alone formally demonstrated.  Furthermore, I actually do not necessarily agree that the assertions made are always correct.  At the very least, it is certainly not the case that all the conclusions of the work have been fully demonstrated.\n- The paper constantly implicitly implies that a failure to satisfy the assumptions of Theorem 1 means that the associated pathologies will be avoided.  However, this logic does not hold as the assumptions in the theorem are sufficient not necessary conditions.  Moreover, I show later in the review that actually the result holds under far weaker assumptions, strongly imply that these inverse assertion that the paper relies on are likely to be false.\n- The paper is written in a way that incorrectly implies various well-known behaviors are surprising insights (e.g. \"the ELBO can prefer learning likelihood functions $f_{\\theta}$ that reconstruct $p(x)$ poorly, even when learning the ground truth likelihood is possible!\").\n- Too many of the experimental results are partially relegated to the appendices.  The paper would be stronger for being more focused on certain aspects of the experiments gone through carefully (with the results themselves actually in the paper!) and some completely relegated to the appendices.  At the moment the paper feels almost unfinished because it is not at all self-contained without the supplement, which is not really acceptable in this format.\n- The paper often talks in overly general terms that are not properly justified.  For example, the abstract just talks about generic \"pathologies\" whereas the topic of the paper is about some quite specific issues rather than a general analysis of different VAE pathologies.\n- The figures are scruffy and rather difficult to read (in particular their font size is ridiculously small).\n\n*Redundency of Theorem 1*\n\nI believe that the core result, Theorem 1, is a rather convoluted way of making somewhat simpler points (see below) and does not provide any particular insights; its first assumption is very much reversed engineered rather than a natural starting point.  As such, it is difficult to take intuitions from it or to understand what is required to satisfy it.  Moreover, the experiments seem to demonstrate that the conditions cannot formally be demonstrated in practice as only very hand-wavy explanations are given rather than concrete demonstrations. \n\nEven more problematically, I believe that the Theorem itself is actually a vacuous result in light of simpler, well-known, ideas.  In short, it is obvious and well-known that we will not achieve maximum marginal likelihood (MML) solutions by maximizing the ELBO if the expected KL cannot be driven to zero at the MML value of theta (except in the bizarre edge case where the expected KL does not vary with theta).  It is also straightforward and well-known that this will give a non-zero KL(p(x)||p_{theta}(x)) except in the special case where there is an alternative likelihood that gives the same p_{\\theta}(x) while allowing an expected KL of zero (interestingly, this special case may well occur with infinite data though as this allows the encoder variance to tend to zero).   As such, the result of the Theorem is somewhat obvious even without the first assumption holding: this core assumption is much stronger than it needs to be (it is only a sufficient condition, not a necessary one), which not only makes the theorem predominantly redundant, it also undermines most of the subsequent conclusions the paper derives from this result (e.g. the suggestion that pathologies will not occur if this assumption does not hold).\n\nTo demonstrate my misgivings more concretely, and verify that the result is indeed vacuous, I developed the following, which I believe to be a stronger and more intuitive alternative result (that also has the key advantage of not making any assumptions about some ground truth model)\n\n*Alternative Theorem*\n\nConsider a VAE with a fixed prior $p(z)$. Define the sets\n$$\\Theta_{MML} = argmin_{\\theta} KL(p(x)||p_{\\theta}(x)),$$\nand\n$$\\Theta_{opt} = argmin_{\\theta\\in\\Theta_{MML}} \\min_{\\phi} E[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))],$$\nsuch that $\\Theta_{opt} \\subseteq \\Theta_{MML}$ (note both of these will often be the same single point). If the following assumptions hold:\n1. $\\nexists \\theta \\in \\Theta_{MML} : E_{p(x)}[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))]=0$, i.e. the encoder cannot match the posterior (this assumption is not strictly necessary but is included for intuition because the next assumption cannot hold if this one does not, while it will almost always hold in practice when this assumption does hold),\n2. $\\exists \\theta \\in \\Theta_{opt} : \\lVert \\nabla_{\\theta} \\min_{\\phi} E[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))] \\rVert >0$ and $\\nabla_{\\theta} \\min_{\\phi} E[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))]$ is locally absolutely continuous for this $\\theta$ (see discussion of this assumption below)\n\nthen the global minima of the negative ELBO, {$\\theta',\\phi'$}, is non-optimal for the marginal maximum likelihood in the sense that $\\theta'\\notin\\Theta_{MML}$ and thus:\n$$KL(p(x)||p_{\\theta'}(x))>KL(p(x)||p_{\\theta_{MML}}(x))\\ge0$$\nwhere $\\theta_{MML}$ be an arbitrary element in $\\Theta_{MML}$.\n\n*Proof*:\nThe proof follows by considering an arbitrary $\\theta$ for which the second assumption is satisfied.  Here $\\nabla_{\\theta} KL(p(x)||p_{\\theta}(x))=0$ but $\\nabla_{\\theta} \\min_{\\phi} E[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))] \\neq 0$ so, by our continuity assumption, it must be possible to improve the ELBO by moving $\\theta$ in the direction $-\\nabla_{\\theta} \\min_{\\phi} E[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))]$.  However, because we are improving $\\min_{\\phi} E[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))]$, it impossible for this change to produce a $\\theta$ that remains in the set $\\Theta_{MML}$  (as this would imply our original point was not in $\\Theta_{opt}$).  As we have a $\\theta$ that improves the ELBO but is no longer in $\\Theta_{MML},$ we can conclude that the optimum of the ELBO is no longer optimal from the perspective of the maximum marginal likelihood. ☐\n\nHere that the second assumption in my Theorem above is very weak whenever the first assumption holds because it effectively equates to saying that the global optima for the MML are not all also local optima for the attainable expected KL divergence between the encoder and posterior.  For example, one would usually not expect the $\\theta \\in \\Theta_{MML}$ to be connected, in which case the assumption can only be violated if the gradient of the attainable expected KL divergence coincidentally happens to be zero for every MML optimal $\\theta$.  In light of this, my suggested theorem is making far weaker and (arguably) more intuitive assumptions than Theorem 1 in the paper, while it is also a slightly stronger final result as it does not require there to be a \"ground truth\" (which is a massive assumption to be able to drop).  \n\nNow, of course, the existence of an alternative theorem does not undermine the contributions of the work in itself, but the problem here is that my result above shows that Theorem 1 in the paper is vacuous because its result effectively always holds in practice if the encoder cannot exactly match the posterior (or more precisely, $\\nexists \\theta \\in \\Theta_{MML} : E_{p(x)}[KL(q_{\\phi}(z|x) || p_{\\theta}(z|x))]=0$).  As such, the complex first assumption in Theorem 1 is generally not necessary and offers little insight.  Moreover, because it is obvious that optimizing the ELBO does produce the MML parameters if the encoder can match the posterior, we see that Theorem 1 provides very little insight other than this already well-known fact.  In my opinion, this severely undermines the contribution of the work.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates various pathologies that occur when training VAE models. There was quite a bit of discussion (including "private" discussion between the reviewers) about the theory presented. Particular concerns included: For Theorem 1, while the required conditions formalise the setting in which the learned likelihoods are poor, it's unclear whether these particular conditions they are useful in practice or provided deep insight; for Theorem 2, its relevance and importance was not necessarily clear. In general the results in these two theorems are closely related to known challenges (e.g. that using the ELBO to optimise parameters may lead to bias), without necessarily providing as much new insight as one might hope.

I would note that all the reviewers included positive feedback as to the quality of the experiments, showing the impact of these pathologies on downstream tasks. However, as written much of the paper focuses on the theory — too many of the (very interesting!) figures and experimental results are relegated to the appendix.
Decision: Reject