VAE Approximation Error: ELBO and Exponential Families
Authors: Alexander Shekhovtsov
Dmitrij Schlesinger
Boris Flach
Venue: ICLR 2022 Spotlight (2022)

Abstract: The importance of Variational Autoencoders reaches far beyond standalone generative models -- the approach is also used for learning latent representations and can be generalized to semi-supervised learning. This requires a thorough analysis of their commonly known shortcomings: posterior collapse and approximation errors. This paper analyzes VAE approximation errors caused by the combination of the ELBO objective and encoder models from conditional exponential families, including, but not limited to, commonly used conditionally independent discrete and continuous models.
We characterize subclasses of generative models consistent with these encoder families. We show that the ELBO optimizer is pulled away from the likelihood optimizer towards the consistent subset and study this effect experimentally. Importantly, this subset can not be enlarged, and the respective error cannot be decreased, by considering deeper encoder/decoder networks.

Published as a conference paper at ICLR 2022
VAE APPROXIMATION ERROR: ELBO AND EXPONENTIAL FAMILIES
Alexander Shekhovtsov
Dmitrij Schlesinger
Czech Technical University in Prague shekhol eê fel. cvut.ez
Dresden University of Technology Dmytro Saarrr.hhhle wtt..:.hss.....geeet-d-rrreeeeee de
Boris Flach
Czech Technical University in Prague flachbor@fel. cvut.
ABSTRACT
The importance of Variational Autoencoders reaches far beyond standalone gen- erative models - the approach is also used for learning latent representations and can be generalized to semi-supervised learning. This requires thorough analy- sis of their commonly known shortcomings: posterior collapse and approximation errors. This paper analyzes VAE approximation errors caused by the combination of the ELBO objective and encoder models from conditional exponential families, including, but not limited to, commonly used conditionally independent discrete and continuous models. We characterize subclasses of generative models consis- tent with these encoder families. We show that the ELBO optimizer is pulled away from the likelihood optimizer towards the consistent subset and study this effect experimentally. Importantly, this subset can not be enlarged, and the respective error cannot be decreased, by considering deeper encoder/decoder networks.
INTRODUCTION
Variational autoencoders (VAE, Kingma & Welling, 2014; Rezende et al., 2014) strive at learning complex data distributions Pa(z). x E X' in a generative way. They introduce latent variables (=(2==))({ where p(2) is simple distribution which
Z and model the joint distribution is usually assumed to be known. The conditional distribution po z). called decoder, is modeled in terms of deep network parametrized by 0 O. Models defined in this way allow to sample from po (o = (x) z) easily, however at the price that computing the posterior po(x) x:) pa(x 2) 2))(2)/pol (2))/pp(G (x) is usually intractable. To handle this problem. VAE approximates the posterior Pe(2) 2) by amortized inference encoder (=(2|z) parametrized by E Φ. Given the empirical data distribution (,) the model is learned by maximizing the evidence lower bound (EIBO) of
the data log- likelihood T((0) Ep, log (o()). It can be expressed in the following two equivalent
forms:
(n Φ) = Epa Egg log pog] 2) DKr. (ps (=|2r) [[)()
(la)
= L(0)= Ep, [Drr. (gx 1) pa (=|=))]-
(Ib)
The first form allows for stochastic optimization of ELBO while the second form shows that the gap between log- likeliiolooo and ELBO is exactly the mismatch between the encoder and the posterior.
VAEs constitute powerful deep learning extension of the expectationnmaxxiiiza to handle latent variables. They are useful not only generative models but also, egg., in semi- supervised learning (Kingma et al., 2014; Mattei & Frellsen, 2019). Furthermore the encoder part constructs an efficient embedding of the data in the latent space, useful in many applications. The outreach of the VAE approach requires therefore a careful empirical and theoretical analysis of the are (i) posterior collapse (He et al., 2019; Lucas et al., 2019; Dai et al., 2018; Dai & Wipf, 2019; Dai et al., 2020) and (ii) approximation errors caused by inappropriate choice of the encoder family.
(EM) approach
problems and trade offs involved. The most important

---

Published as a conference paper at ICLR 2022
Consistent VAEs
Best Consistent
O
Posterior Collapse
Best VAE
F(O) minE,, (Daalo [[po(z]=))
ML Decoder
Diagram of the VAE trade-off. The optimal solution 0YAE is "in between" the maximum likelihood solution ML, and the best solution in the class O of consistent VAEs, where the posterior approximation error function F()) vanishes. We give an explicit characterization of this consistent set. At OVAR there is a balance between the gradient of - (blue arrow) and the gradient of the data log ligeliilloood (black arrow).
Figure 1:
The VAE approximation error has been studied (e.g., Cremer et al. 2018; Hjelm et 2016; Kim et al. 2018) so far mainly empirically. The problem also occurs and is well-recognized in the context of variational inference and variational Bayesian inference, where the target posterior distribution is expected to be complex. It is commonly understood, that the mean field approximation of (o by (0(2) æ) in (1b) significantly limits variational Bayesian inference. decoder may adopt to compensate for the chosen encoder family. The effect of this coupling, believe, is not fully understood. The phenomenon of decoder adopting to the posterior was experi- mentally observed, e-g., by Cremer al. (2018, Section 5.4), noting that the approximation error is often dominated by the amortization error. Turner & Sahani (2011, I, Sec. 1.4) analytically show for linear state space models that simpler variational approximations (such mean- (reannfield) can lead less bias in parameter estimation than more complicated structured approximations. Similarly, Shu et al. (2018) view the VAE objective as providing a regularization and show that making the amortized inference model smoother, while increasing the amortization gap, leads to a better generalization.
contrast, VAEs, the
The common (empirical) understanding of the importance of the gap between the approximate and the true posterior has led to many generalizations of standard VAEs, which achieve impressive practi- results, notably, tighter bounds using importance weighting (Burda al., 2016; Nowozin, 2018), encoders employing normalizing flows (Rezende & Mohamed, 2015; Kingma et al., 2016), hicrar- chical and autoregressive encoders (Vahdat & Kautz, 2020; Sønderby et al., 2016; Ranganath et al., 2016), MRF encoders (Vahdat al., 2020) and more. While these extensions mitigate the posterior mismatch problem, they often come at a price of a more difficult training and more expensive in- ference. Furthermore, simpler encoders may be of practical interest. Burda al. (2016, Appendix C) illustrates that IWAE approximate posteriors are less regular and more spread out. In contrast, factorized encoders provide simple embeddings useful for downstream tasks such as semantic hash- ing (Chaidaroon & Fang, 2017).
The aim of this paper is to study the approximation error of VAEs and its impact on the learned decoder. We consider a setting that generalizes many common VAEs, in particular popular models where encoder and decoder are conditionally independent Bernoulli or Gaussian distributions: we assume that both decoder and encoder are conditional exponential families. We identify the subclass of generative models where the encoder can model the posterior exactly, referred to as consistent VAEs. We give a characterization of consistent VAEs revealing that this set in fact does not depend on the complexity of the involved neural networks. We further show that the ELBO optimizer is pulled towards this set away from the likelihood optimizer. Specializing the characterization to several common VAE models, we show that the respective consistent models turn out to be RBM- like in many cases. We experimentally investigate the detrimental effect in one case and show that simpler but more consistent VAE can perform better in the other.
2 PROBLEM STATEMENT
We adopt the following notion of approximation error. Consider generative model class Pe = (po( 0 O). the encoder class Q. {2<()) Φ $) and the data distribution pa(x). The maximum likelihood generative model is given by OM.. E argmaxgee Epe(e) (o)))(<)). For decoder with parameters we define its approximation error as the likelihood difference L(8xa)) -
2

---

Published as a conference paper at ICLR 2022
L(#). Respectively, the VAE approximation ervor is defined for given A as: L( - max LB (,, Φ) L(0ML) L(0).
(2)
In order for this error to become zero, two conditions are necessary and sufficient:
Parameters (0, Φ) must be optimal for the ELBO objective.
ELBO must be tight at (0, Φ), i.e., (8, 6) L(0).
Assuming that the optimality can be achieved, we study the non-tightness gap L(O) L0(0, Φ). From (1b) it expresses Ep [PKI (Dx1((0(=11)) po(=|=))] It follows that ELBO is tight at (0, Φ) iff (=(=|z) = pe(z] z). Hence, we define the consistent set O2 S e as the subset of distributions pe(x,2) 2) whose posteriors are in 24. i.e..
4 = ( 36 po({=))
The KL-divergence in the ELBO objective (Ib) can vanish only 0 E Og. If the likelihood max- imizer OMI is not contained in O4. then this KL-divergence pulls the optimizer towards O4. and away from 0ML as illustrated in Fig. 1.
We characterize the consistent set ④4, on which the bound is tight, and show that this set is quite narrow and does depend on the complexity of the encoder and decoder networks beyond simple I-layer linear mappings of sufficient statistics.
3 THEORETICAL ANALYSIS
We consider a general class of VAEs, where both encoder and decoder are defined as exponential families. This class includes many common models, in particular Gaussian VAEs and Bernoulli VAEs with conditional independence assumptions, but also more complex ones, e.g., where the encoder a conditional random field (Vahdat et al., 2020)'.
Assumption (Exponential family VAE). Let X and 2 be sets of observations and latent variables, respectively. We consider VAE models defined by
pa z) h(()) exp
fo(=)) A(fo(=))
(4a)
9 x)) (2) cxp (1b()) go
B(go(=))),
(4b)
where X Th' and 1h: Z m are fixed sufficient statisties of dimensionality and m;; fe: Z + Rn and gs: X + R. are the decoder, resp., encoder, networks with learnable parameters resp. d; h: X R4, h': + 1 are strictly positive base measures and A, denote the respective log- neg-partition functions.
Notice that this assumption imposes no restrictions on the nature of random variables and They can be discrete or continuous, univariate or multivariate. Similarly, it imposes no restrictions on the complexity of the decoder and encoder networks fo(z) and (z).
Characterization of the consistent set. In the first step of our analysis, we investigate the conditions under which the approximation error of exponential family VAE can made exactly zero. As discussed above, a tight VAE (6, Φ) must satisfy V(2, 2) q (=(xx) po(z) which leads to the following theorem.
Theorem 1. The consistent sei O4 of an fami VAE is given by decoders of the form p(z| 2) = h(x) exp (((x), Wxb()) + (v(x), - A(z)],
(5)
where W a n m matrix and u E RR.. Moreover, the corresponding encoders have the form
[(2) h h2()) exp (x)22, WT v(=)) (1(2). B(x)),
(6)
where R
Notice, however, that this class docs not include VAFs with advanced encoder families like normalizing flows, hicrarchical and autoregressive encoders.

---

Published as a conference paper at ICLR 2022
This is direct consequence of a theorem by Amold & Strauss (1991) (see Appendix A.1 for more details). For a tight VAE. Theorem I states that the decoder and encoder are generalized linear models (GLMs) (5) and (6) with the interaction between and & parametrized by matrix W and two vectors u, v instead of the (complex) neural networks with parameters 0, c. The corresponding joint probability distribution takes the form of an EF Harmonium (Welling al., 2005):
p(a, 2) = (z) exp (v((), W (z)))) + (v(z). + (1(2), A).
(7)
The subset O₂ of consistent models can not be enlarged by considering more complex encoder networks g(z), provided that the affine family W v(z) can already be represented.
Corollary 1.
Let the decoder network family be affine in ((2), i... f(2) Wab(2) + a and let the encoder network family g(x) include at least all affine maps Vv(œ) + b. Then any global optimum of ELBO attains a zero approximation error.
Corollary 2.
VAE can escape consistency when it degenerates to flow. In practice, VAE models with rich de- coders are almost never tight. It is therefore natural to ask, whether a small VAE posterior mismatch error implies closeness of the optimal decoder to some decoder in the consistent set.
Definition 1. A VAE (po, qa) is e-tight for some & > 0 if Epa(s) (Ps))))P Il po(== S.
It turns out that this definition allows VAE approach tightness while not approaching consis- tency. In the continuous case an example satisfying e-tightness with non- non-linear decoder follows from Dai & Wipf (2019, Theorem 2). They show, for class of Gaussian VAEs with general neural networks fe- 9s- that it is possible to build a sequence of network parameters Oer d with the follow- ing properties: i) the target distribution is approximated arbitrary well, ii) the posterior mismatch (g1(90. (z 2x) Pe, (z] æ)) approaches zero and iii) both the encoder and decoder approach deter- ministic mappings. The VAE thus approaches flow model (or invertible neural network) between the data manifold and a subspace of the latent space (Dai & Wipf, 2019). Clearly, in a general case the flow must be non-linear. similar case can be made for discrete variables, see Example A.
Non-deterministic nearly-tight VAEs approach consistency. We would however argue that the mode where the decoder and encoder are ceariy-deterministic is not natural VAE solution. By making additional assumptions, excluding such deterministic solutions, and restricting ourselves to the finite space in order to simplify the analysis, we can show that an e-tight VAE does indeed approach EF-marmon..m.
Theorem 2. Let (po, 4o) be an exponential family VAE (Assumption 1) on a discrete space x Z with encoder q0(2|00 x) and decoder posterior po[2] x) both bounded from below by O. If the VAE is E E-tight, then there exists a matrix W E R. and vectors u E R", v E R" such that the joins model implied by the decoder po(x, 2) po(2) 2))(z) can be approximated by an unnormalized EF Harmonium
p(x, -
Wx((2) + (v(x), u) + (U, (2)) +c)
(8)
with the error bound
E Epa(z) (log z) log p(1, =))
2 o(s) V
The proof is given in Appendix A.3. In this theorem the function p(x,2) is non-negative but does not necessarily satisfy the normalization constraint of density. Re-normalizing it by adjusting c in (8) may break the approximation guarantee. Nevertheless, if & is small enough and, e.g., the data distribution is non-negative on the whole X', we expect it to approach a density, in particular to recover the result in Theorem in the limit. Note that the theorem does not make any assumptions about optimality of (0, Φ), i.e., it describes all models in the vicinity of the consistent set in Fig. 1.
3.1 CASES ANALYSIS
This subsection gives a detailed analysis of consistent VAE models in several concrete cases of practical interest.
Diagonal Gaussian VAE Let us consider a Gaussian VAE, as commonly applied to image gen- eration (e.g., Dai & Wipf (2019)). Let X
RD. Z Rm, p(x]2) [
4

---

Published as a conference paper at ICLR 2022
g[2]2) pe (z))) where pd, He and oe neural networks and du is common
pixel observation noise parameter. The decoder has minimal sufficient statistics x(x) = x and base
measure h(z)) (2 00 o. The encoder has minimal sufficient statistics (2) = (2, 2²), where
the square is wise.. Theorem I implies that a tight optimal VAE has the joint model
p(a, 2) x (x,((,,))) Wz + V + a) + (b, z) + (c,
(10)
for some matrices W. V and vectors a, c. Furthermore, the integral over 2 must be finite for all x and therefore x V + < O must hold for all x E R". This is possible only if V = 0 and c < O. The joint distribution therefore multivariate Gaussian and the same holds for its marginal p(z). The neural network sa((2) must degenerate to ma(2))
03 (Wz + a) and the two neural networks for the encoder to (z) = -1 22c and [e(z)) = - (W x + b)/2c, where divisions are coordinate-wise. VAEs with such simplified. linear Gaussian encoder-decode pairs, called "linear VAEs" (Lucas et al., 2019) are known to be consistent and to match the probabilistic PCA model (Dai et al., 2018; Lucas et al., 2019). In this context, our Corollary 2 is generalization of (Lueas et al., 2019, Lemma 1) showing consistency of linear VAEs, to decoders in any exponential family with natural parameters being a linear mapping of any fixed lifted latent representation e()).
We argue that joint Gaussian model is too simplistic to generate complex data such as realistic images and that in this case the VAE error is detrimental. In Section 4.2 we experimentally confirm that optimizing ELBO for a general decoder network pud causes qualitative and quantitative degrada- tion relative to the MI. decoder. Note that if we allowed ag to be dependent on z, the resulting joint statistics in @ to would include terms x²2 x222 The joint distribution would not be Gaussian and may be in fact multi-modal (see Anil Bhattacharayya's 's distribution in Arnold et al. 2001).
Bernoulli-MRF VAE Vahdat et al. (2020) proposed to consider encoders in the Markov Random Field (MRF) family, in particular encoders of the form g(> a:) εx exp((2², ((z)=²) + (b* (a:), + (b)) 22)). where =1, x² are two groups of latent variables and interaction weights V. b2, 12 are computed by the encoder network. In this case g(2| is itself (conditional) RBM. While evalu- ating q2| æ) is difficult, MCMC sampling is efficient. We assume binary observations x and a con- ditionally independent Bernoulli decoder family as above. The decoder thus has sufficient statistics v = x and the encoder has to = (z', ? 2² @ 2²). Introducing homogeneous constant components To = =0 z 1, the family of consistent joint distributions can be compactly described as
p(x exp (E.,,, W Zi 2) =2]
(11)
where the summation in all indices starts from 0 and - Wa. is the log ogg-patition function. This joint model is a higher order MRF with the highest order potentials given by cubic monomials.
Standard Bernoulli VAEs are a special case of the Bernoulli-MRF model, obtained when the inter- action weights V are zero. The joint distribution of such tight optimal VAEs takes the form
p(a, =) = _ exp(x W= + uTx x + +.Z2),
(12)
which is a restricted Boltzmann machine (RBM). Since RBMs are well known for being useful in many applications (dimensionality reduction, collaborative filtering, feature learning, topic model- ing), we hypothesize that they can make a good baseline for Bernoulli VAEs and furthermore that the effect of pulling the VAE solution towards an RBM may be benign in case of insufficient data. For example IWAE test likelihood in (Burda et al., 2016) is worse than that of an RBM (Burda al., 2015) on the Omniglot dataset. Furthermore, debiasing of IWAE (Nowozin, 2018) does not improve test likelihood in many cases.
Bernoulli VAE for Semantic Hashing One important application of Bernoulli VAEs is the seman- tic hashing problem, initially proposed and modeled with RBMs (Salakhutdinov & Hinton, 2009). The problem is assign to each document/ / image compact binary latent code that can be used for quick retrieval by the nearest neighbor search. We will detail now a more recent VAE model for text documents (Chaidaroon & Fang, 2017; Shen al., 2018) and show that it can be tight only in a full posterior collapse. We correct the encoder so as to allow a larger consistent set and observe that the resulting consistent joint distribution forms a multinomial Bermoulll RBM.
Let x E KK be word counts in a document with words from dictionary of size K. Let z E (0, 1] be a binary latent code. Let Σg 2g denote the document's 's length. We assume that the document
5

---

Published as a conference paper at ICLR 2022
length is independent of the latent topic and its distribution p(() can be learned separately (e.g., log- aognnomaal distribution is a good fit). The decoder is defined using the multinomial distribution model (words in the document are drawn from the same categorical distribution corresponding to its topic):
Ma,422 - p(())((r())) (x((e)))))))))))()))
(13)
where f(2) is a neural network mapping the latent code the logits of word occurrence probabil- ities, A(n) = log E. exp(nx) and h(x: 1) [Σx T6 ===]]]]) IIx 3r2!) is the base measure?. The sufficient statistics are the word counts The prior p(z) is assumed uniform Bernoulli.
The encoder the conditionally independent Bernoulli model, expressed as
q(2) x2 x exp(g(x)"2) 2),
(14)
where (xx) is the encoder network. Chaidaroon & Fang (2017) experimented with the encoder and decoder design and recommended using TFIDF features instead of raw counts. First, we note that the inverse document frequency (IDF) is not relevant, since it can be learned by the first linear transform in the encoder. Effectively, the term frequency (TF), given by x/1, is used. This choice is adopted in later works (Shen al., 2018; Zamani Dadaneh al., 2020; Nanculef et al., 2020). It might seem reasonable that the latent code modeling the document topic should not depend on the document length, only on the distribution of words the document. However, we will argue that this rationale is misleading for stochastic encoders.
We apply Theorem to two groups of variables: observed (z, and latent 2 with h(z,1) h(x ())( (Ix = x and (2)) z. It follows that the consistent joint family is
p(x, 1. 2) = 1(x I) cxp(x W2 + aTx x bT cc.
(15)
This however implies that g(x) = Wx + b, iee. the encoder network must be linear in x. Conse- quently, it cannot match function of word frequencies x:/l (as chosen by design) unless W = 0, i.e. a completely trivial model with an encoder not depending on æ. Such an encoder would imply full posterior collapse. The corresponding consistent set O& coincides with the set of collapsed VAEs where the decoder does not depend on the latent variable in Fig. 1. We conjecture that the inherent inconsistency of this VAE has detrimental effect on learning.
If instead, we let the encoder network to access word counts x directly, we obtain that g(x) = Wa + b can form consistent VAF. Inspecting this encoder model in more detail, we that builds up topic confidence in proportion to the evidence (total word counts), as the true pos- terior would. Indeed, the true posterior p(= ar, I) satisfies the factorization by Bayes's theorem: p(z] xx, 1) = p(z| z, ()p(z)/((z] (). The prior p(z) is constant by design, p(z| () does not vary with and p(a z, () factors over all word instances according to (13). In other words, the coupling between and in log p(2| xr 1) is linear in xr
In Section 4 we study the proposed correction experimentally and show that it enables learning better models under variety of settings.
4 EXPERIMENTS
4.1 ARTIFICIAL EXAMPLE
To start with, we illustrate our findings on a toy example. We consider simple Gaussian mixture model for which can casily generate samples and compute all necessary quantities including the ELBO objective. We define the ground truth model to be p" (z, 2) p²(z) (z)pp* (x 2). with = e R² p" (x] Nxxx uu=),, I²)), L.e. mixture of four 2D Gaussians. Fig. 2(a) shows the color-coded posterior distribution p" (2 x). We assign a color to each component and represent p" (z) 2²) for each pixel R² by the corresponding mixture of the component colors. For better interpretability, we illustrate further results by decision maps arg max p(2 x). Fig. 2(b) shows the decision map for p² (z) x))
{1...4, p" (2) = 0.25,
PPrior work (Chaidaroon & Fang, 2017; Shen et al., 2018) omits the base measure as it has no trainable parameters.

---

Published as a conference paper at ICLR 2022
17
100
[00 j01
(a)
(b)
(c)
(d)
(f)
Figure 2: Artificial example. (a) Color-coded posterior distribution p (z (z). (b-e): Decision maps max,) of: (b) true posterior p" (=|x), (c) factorized encoder g0(2) 2)) after joint learning, (d) model posterior po(z] æ) after joint learning, (e) RBM trained on the same data. Gaussian centers u(=) shown as black dots. (f) Probability simplex of distributions over the four binary config- urations. The vertices correspond to pure (deterministic) binary states represented by the code and its respective color. The surface shows the manifold of factorized distributions realizable by g(> ær). Notice that the two edges (00, 11) and (01, 10) are not in the manifold because they correspond to switching of two bits simultaneously in correlated way. The factorized approximation cannot model transitions between these states. Hence, when learning VAE, these pairs of states are repulsed in the decision maps (c), (d).
The aim of the experiment to learn a VAE and to study the influence of the factorization assump- tion the results. We use the decoder architecture as in the ground truth model Gaussian distribution Po (ar = N2x 0 I² where Zoh is the one-hot (categorical) representation of z, and ( is 2x4 matrix that maps the four latent codes to 2D centers at general locations. Note that the ground truth model is contained in the chosen decoder family. Hence, the ML solution is the ground truth decoder pz (ar 2).. We restrict the encoder to factor over the binary representation of the code 26 E {0, 1 and define it as go(2z) 2 x) xx exp (go(2), Ra), where (x(x) is implemented as feed-forward network with two hidden layers, each with 64 units and ReLU activations.
First, we pre- traitrain our factorized encoder by optimizing ELBO and keeping the ground truth decoder fixed. The next step is to jointly train the encoder and decoder by maximizing ELBO, Since we are interested how the ELBO objective distorts the likelihood solution, we start with the ground truth decoder and the pre-trained encoder from the previous step. The ELBO-optimal decoder has to match not only the training data, but also the inexact, factorizing encoder. The resulting {=(2| x) is shown in Fig. 2(c) and the learned model posterior (=(2) æ) x (x in Fig. 2(d). Note that they match each other pretty well, but differ substantially from the ground truth posterior shown in Fig. 2(b). The impact of the factorization is clearly visible one can see two decision boundaries (one for each bit of 2,), which together partition the z-space into four regions, approximating the true posterior. For comparison, Fig. 2(e) shows the posterior of an RBM trained on the same data. It is clearly seen that the ELBO optimizer is pulled away from the likelihood optimizer towards an RBM solution. Notice also the explanation given in Fig. 2(f). Summarizing, this simple toy example clearly shows the VAE approximation error caused by the combination ELBO objective and the factorization assumption for the encoder. While the numerical difference between ELBO and likelihoooo is small (see details in Appendix C.1), the qualitative difference in Fig. 2 appears substantial.
4.2 GAUSSIAN VAES FOR CELEBA IMAGES
The goal and design of this experiment similar to the previous one. We first define a ground truth decoder which is used to generate training images. Then we pre-train encoder by ELBO keeping the ground truth decoder fixed. Finally, we train both model parts starting from the ground truth decoder and the pre-trained encoder.
The ground truth generative model is obtained by training a convolutional Generative Adversarial network (GAN) using code of Inkawhich (2017) on the CelebA dataset Liu et al. (2015). We scale and crop all images to 64 x66 pixels. In order to get stochastic decoder, we equip the GAN gener- R 100 x E 664x64x3 with image noise aa The ground truth generative model thus defined as p² (x, 2) = p" [x]p((([2) where p" (z) N(2 I), (x 2) [x pa(2), d) and o is common noise variance for all pixels and color channels. We chose aı = 0.05 (the color values are normalized to [-1, 1).. This corresponds to an image noise level, which is just visible, but does not disturb visual perception essentially.
ator x d(z).

---

Published as a conference paper at ICLR 2022
Figure 3: Results for the encoder learned by su- pervised conditional likelihood. Top row: train- ing samples & p" (zz). Second row: the cor- responding reconstructions from mean values of i.e. x ~ p" (xr Mra (4)). Third row: reconstruc- tions from sampled z, i.e. 2 ~ (1), followed by ~ p" (x] ().
Figure 4: Visual comparison images drawn from the original/learned models. Each col- umn corresponds to a particular value of (O, I). Top row: the ground truth model, mid- dle row: learned decoder with fixed aas bottom row: decoder with learned
The decoder family of the considered VAE consists of networks with the same architecture as d(z). This ensures that the ground truth decoder is likelihood maximizer of the VAE model. The encoder defined as [z(2]a) æ) = [x uue (z), diag(o" where jle, Oe E 11000 are two outputs a convolutional neural network with architecture similar the architecture of the discriminator used for training the GAN (except the output layer), iee, (0(2) x) is a multivariate Gaussian with diagonal covariance matrix whose parameters depend on
We pre-train the encoder fully supervised by maximizing its conditional log- ligeliillhood (x-(z.=) log 9 (=(xx) examples drawn from the ground truth generating model (2 2).² The results of pre-training are shown in Fig. 3. Then we jointly learn the encoder and decoder by max- imizing ELBO ground truth decoder (x] z) and the encoder obtained in the previous step. Two variants are con- sidered for this training: (i) keeping the image noise aa fixed and (ii) learning it along with other model parameters. We evaluate the results quantitatively by computing the Frechet Inception Dis- tances (FID) between the ground truth model p" (æ 2) and the obtained decoders pe (z) 2) using the code of Seitzer (2020). For this we generate 200k images from each model. The obtained values are given Tab. Fig. shows images generated by the ground truth model and the two learned models.
z-samples drawn from the ground truth model. We start the learning with the
To conclude, ELBO optimization harms the decoder considerably as clearly seen both from FID- scores and the generated images. Models with higher ELBO values have worse FID-scores and produce less realistic images.
Optimizing the ELBO starting from the ML solution degrades the FID-score. The first row corresponds to the pre-trained encoder for the ground truth decoder, its FID-score therefore compares two image sets, both generated by the ground truth model.
Table 1:
Experiment
ELBO
FID
optimize encoder (conditional likelihood) optimize encoder and decoder (ELBO, fixed σι)) optimize encoder and decoder (ELBO, learned o.))
0.13
-5898.94
77.10
9035.69
117.87
4.3 BERNOULLII VAE FOR TEXT DOCUMENTS
This experiment compares training of the VAE model for semantic hashing discussed in Section 3.1 with and without our proposed correction on the 20Newsgroups dataset (Lang & Rennie, 2008). We describe the dataset, preprocessing and optimization details in Appendix C.2.
We compare three encoders: el: linear encoder on word counts (the proposed correction), e2: deep (2 hidden layers) encoder using word frequencies and e3: linear encoder frequencies. The
In contrast to the previous experiment we optimize the "forward" -iveerrrg,,, i.e. the conditional like- lihood, instead of the "reverse" Kll-divergence used in EI.BO for simplicity.
8

---

Published as a conference paper at ICLR 2022
encoders are compared across different numbers of latent Berouuli variables (bits) and different decoder depths. The decoder depth denotes the number of fully connected hidden ReLU layers (0- 2). In both the encoder and decoder use 512 units in hidden layers. The prior work mainly used linear decoders following the ablation study of Shen et al. (2018). Our experiments also suggest that using deep decoders in combination with longer bit-length leads significant overfitting When the decoder is linear, the posterior distribution is tractable and is linear as well, i.e., the VAE model equivalent to a Multinomial-Bemmne word counts el indeed works better in this case. However, perhaps more surprisingly, we also find out that it works better even for non-lineat decoders.
RBM. We experimentally verify that a linear encoder on
Table 2 show the achieved training and test Negative ELBO (NELBO) values. We observe across all settings that the simple encoder el is consistently better than the more complex encoder e2 which in turn is significantly better than the linear encoder on frequencies e3. We conclude that the use of VAEs with deep encoders based on word frequencies (Chaidaroon & Fang, 2017; Shen et al., 2018; Zamani Dadaneh et al., 2020; Nanculef et al., 2020) is sub-optimal for this dataset. We also observe that linear decoders generalize better under 32 and 64 bits compared to more complex decoders, which suffer from overfitting. This implies that in these cases the best encoder-decoder combination is linear, i.e. the basic RBM model. This evidence agrees with previously observed worse reconstruction error with deep architecture (Dai et al., 2020), however did not observe (a more severe) posterior collapse with deeper models amongst the depths we report.
Table 2: Training and test NELBO values for Text- Text-VAR with different configurations of bits, de- coder and encoder. Bold highlights the best encoder choice and underlined bold values are the best decoder-encoder combinations for each bit-length.
TRAINING
TEST
dhidden==
1hidden=1
dhidden=2
dhidden=0
dhidden=1
dhidden==2
Bits
Bits
el
e2
e3
el
e2
e3
el
e2
e3
el
e2
e3
el
e2
e3
el
e2
e3
8
419
429
439
321
390
415
325
370
421
8
423
429
435
413
421
424
418
423
427
16
382
398
419
201
329
407
164
269
413
16
409
417
421
404
422
420
410
416
422
32
337
412
165
189
403
132
159
411
396
413
416
399
413
418
406
416
421
64
296
324
407
171
189
398
134
149
409
64
392
411
414
398
413
417
406
417
422
5
CONCLUSIONS
We have analyzed the approximation error of VAES in a general setting, when both the decoder and encoder are exponential families. This includes commonly used VAE variants as, e.g., Gaussian VAEs and Bernoulli VAEs. We have shown that the subset of generative models consistent with the encoder class is quite restricted: it coincides with the set of log-bilinear models on the sufficient statistics of both decoder and encoder, i.e., RBM-like models. This consistent subset can not be enlarged by using more complex encoder networks as long as encoder's sufficient statisties remain unchanged. In combination with the ELBO objective, this causes an approximation error - the ELBO optimizer is pulled away from the data likelihood optimizer towards this subset. Moreover, we proved theoretically that close-to-tight EF VAES must be close to RBMs in a certain sense.
We have shown that the error is detrimental when the consistent subset too restrictive. In the cases where lot of data is available and high quality generative model is of the primary interest, such as in the CelebA experiment, more expressive encoder families are required addition to large networks. On the other hand the VAE approximation error may result in a useful regularization when the respective RBM is good baseline model. In this case we can speak of binning inductive bias towards RBM, such as in our text- teX experiment. Furthermore, simple encoders can be desired when the learned representations are of interest, in particular they appear to facilitate similarity in Hamming distance, useful in the semantic hashing problem.
Further connections to related work and discussion can be found Appendix

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENT
D.S. was supported by the German Federal Ministry of Education and Research (BMBF, 01/S (022A-FF by funding the competence center for Big Data and AI "ScaDS.AI Dres- den/Leipzig". A.S and B.F gratefully acknowledge support by the Czech OP VVV project "Re- search Center for Informatics" Czech Science Foundation, grant 19-09967S. The authors gratefully acknowledge the Center for Information Services and HPC (ZIH) at TU Dresden for providing computing time. We thank the anonymous reviewers for many helpful links and suggestions.
B.F. was also supported by
REFERENCES
B. C. Amold and J. Strauss. Bivariate distributions with conditionals in prescribed exponential families. Journal of the Royal Statistical Society Series B (Methodological), 53(2):365-3755 1991.
Barry C. Arood, Enrique Castillo, and Jose Maria Sarabia. Conditionally specified distributions: An introduction. Statistical Science, 16(3):249 274, 2001.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Accurate and conservative estimates of MRF log-likeldhood using reverse annealing. In AISTATS, volume 38, pp. 102-110, 09-10 May 2015.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR, 2016.
Suthee Chaidaroon and Yi Fang. Variational deep semantic hashing for text documents. In SIGIR Conference on Research and Development in Information Retrieval, pp. 75-84, 2017.
Chris Cremer, Xuechen Li. and David Duvenaud. Inference suboptimality in variational autoen- coders. In ICML, volume 80, pp. 1078-1086, 2018.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. ICLR, 2019.
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Connections with robust PCA and the role of emergent sparsity in variational autoencoder models. Journal of Machine Learning Research, 1-441):1-42,
42, 2018.
Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? Reassessing blame for VAE posterior collapse. In ICML, 2020.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatriie Lagging inference networks and posterior collapse in variational autoencoders. In ICLR, 2019.
Devon Hjelm, Russ R Salakhutdinov, Kyunghyun Cho, Nebojsa Jojic, Vince Calhoun, and Junyoung Chung. Iterative refinement of the approximate posterior for directed belief networks. In Advances in Neural Information Processing Systems, volume 29, pp. 4691 -4999, 2016.
Nathan Inkawhich. DCGAN tutorial, 2017. URL https: https://pytorch.. wwwp:eercc..gggtttttaaaaloo beginner/dcgar faces_tutoriall html.
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. Semi-amortized variational autoencoders. ICML, volume 80, pp. 2678-2687, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, pp. 3581-3589, Cambridge, MA, USA, 2014. MIT Press.
Diederik Kingma, Tim Salimans, Rafal Jozefowiez, Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In NeurIPS, volume 29, pp- 4743-4751, 2016.
10

---

Published as a conference paper at ICLR 2022
Ken Lang and Jason Rennie. The 20 newsgroups dataset, 2008. 20Newsqroups/.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Don't blame the ELBO! linear VAE perspective on posterior collapse. In NeurlPS, volume 32, pp.
2019.
Lars Maalø Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. In NewrIPS, volume 32, 2019.
Pierre-Alexandre Mattei and Jes Frellsen. MIWAE: Deep generative modelling and imputation of incomplete data sets. In ICML, volume 97, pp. 4413-4423, 09-15 Jun 2019.
Ricardo Nanculef, Francisco Alejandro Mena, Antonio Macaluso, Stefano Lodi, and Clau- dio Sartori. Self-supervised Bernoulli autoencoders for semi-supervised hashing. CoRR, 00s//007.08799, 2020.
Sebastian Nowozin. Debiasing evidence approximations: On ieportance-weigheed autoencoders and jackknife variational inference. In International Conference Learning Representations, 2018.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In ICML, vol- ume 48, pp. 324-333, 20-22 Jun 2016.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows ICML,, volume 37, pp. 1530-1538, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In ICML, 2014.
Ruslan Salakhutdinov and Geoffrey Hinton. Semantic hashing, Inl. J. Approx. Reasoning, 50(7): 969-978, July 2009.
Maximilian Seitzer. pytorch-fid: fid:: FID Score for PyTorch, https://github.r wttp://www..u.ccooo/eeeeeeeeeeee pytorch-fid, August 2020. Version 0.1.1.
Dinghan Shen, Qinliang Su, Paidamoyo Chapfuwa, Wenlin Wang, Guoyin Wang, Ricardo Henao, and Lawrence Carin. NASH: Toward end end-to-end neural architecture for generative semantic hash- ing. In Annual Meeting of the Association for Computational Linguistics, pp. 2041-2050, jul 2018.
Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochenderfer,, and Stefano Ermon. Amortized inference regularization. In NeurIPS, volume 31, 2018.
Robert Sicks, Ralf Korn, and Stefanie Sehwaar. generalised linear model framework for 3-- variational autoencoders based on exponential dispersion families. CoRR, 2021.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems, volume 29, pp. 9738-3746, 2016.
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Student- t variational autoencoder for robust density estimation. In IJCAI, pp. 2696-2702. 7 2018.
R. E. Turner and M. Sahani. Two problems with variational expectation maximisation for series models. In Bayesian Time series models, chapter pp. 109-130. 2011.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. In NewrIPS, volume 33, pp. 19667-19679, 2020.
Arash Vahdat, Evgeny Andriyash, and William Macready. Undirected graphical models approxi- mate posteriors. In ICML, volume 119, pp. 9680-9689, 13-18 Jul 2020.
11

---

Published as a conference paper at ICLR 2022
Max Welling, Michal Rosen-Zvi, and Geoffrey E Hinton. Exponential family harmoniums with an application to information retrieval. In NeurlPS, volume 17, 2005.
Mingzhang Yin and Mingyuan Zhou, ARM: Augnemm-EEINFRRCE-nnrg gradient for stochastic
binary networks. In ICLR, 2019.
Siamak Zamani Dadaneh, Shahin Boluki, Mingzhang Yin, Mingyuan Zhou, and Xiaoning Qian. Pairwise supervised hashing with Bernoulli variational auto-encoder and self-control gradient estimator, In UAL, volume 124, pp. 540-549, 2020.
12

---

Published as a conference paper at ICLR 2022
Appendix
A PROOFS
A.1 PROOF OF THEOREM 1
The proof directly follows from the characterization of conditionally specified joint distributions in the exponential family given by Arnold & Strauss (1991), see (Amold al., 2001, Theorem 3):
Theorem A.1 (Amold & Strauss 1991). Let x & X and 2 & Z be random variables with a strictly positive joint distribution such that both conditional distributions are exponential families with den- sities
p(al 2) h22) exp (v(z), A=z)]
(16a)
p(= x) (=)=)) exp (())= - B(x)),
(16b)
Z R are minimal sufficient statistics, Z - R" and g : + RT are any mappings, h((x) and (z) are base measures, and and B denote the respective log log-partition functions*.
where X R" and vb
Then there exists a matrix W E Minn + 1,m + 1), such that the density of the joint distribution can be represented
p(a, 2) = h(=) exp (x))))
(17)
where Ve- vhe denote the statistics vectors extended with additional component 1.
A.2 EXAMPLE OF DISCRETE VAE APPROACHING A FLOW
Example A.1. In this example we construct a VAE that can be arbitrary close to tight one, but where the decoder network does not approach linear map. Let X' {-1,1) Z {-1, and let p(z) be uniform. Let the decoder be conditionally independent
p 2) = exp[8 (z, (()) A())]
(18)
where T=)) denotes the invertible mapping of to X' given by
I 21
(19)
Σ2 = 21 22
(20)
If the parameter 8 is sufficiently large, the distribution p(x|2) approaches the deterministic dis-
tribution (g= Its posterior therefore also approaches the deterministic distribution 8 8== (x)
(notice that π = T). Let the encoder be the conditionally independent model <(2) x)
exp [3(2 म (2)) - A(x)]. This decoder by design approaches ōy=m(=) as well and thus the VAE (p, q) achieves e-tightness for sufficiently large 3. At the same time the deviation between loga-
rithms of probabilities log g(= (x) and log p(g((())) (z) grows with B.
A.3 PROOF OF THEOREM? 2
The idea of the proof is to bound the difference between p(= x) and log (()[xr), which is done by Proposition A.2 and then in the space of log-probabilities approximate the non-linear mapping fe(z) by a linear one as detailed in Proposition A. By carefully choosing the norms and the approximation we obtain a bound on the error for the joint model, which despite the discreteness assumption of the observation space X' in Theorem 2 does not depend on its cardinality.
For finite set X C. X let H. be the |X dimensionnll vector space with the inner product (u, u)ps Erex Pa(œ)u (z)) assuming that Pa(x) > 0 for all x E X. The respective norm will be denoted as
pa-
'The log-partition function is usually defined as function of the (natural) parameter. In this context we consider h.h', f. 9 to be fixed and consider the dependence x, only.
13

---

Published as a conference paper at ICLR 2022
Proposition A.1. Under model Assumption 1, for any finite X X there exists matrix W Minn + 1, 1) such that joint distribution implied by the decoder p(a,2) = p(z[2)p(2) can be
approximated by unnormalized EF Harmonium
P(x, 2) h(x=)h(()))) (x))
(21)
with the error bound
(Vz) Enex EmxxPa(x) log p(z, z) log =) S Emex Pa(z)) log - logp(=|>>) (22)
The function p(z, 2) is non-negative but does not necessarily satisfy the normalization constraint of density.
Proof. For clarity, we will omit the dependence of the decoder and encoder on their parameters 8, resp. c. Throughout the proof will also assume that single 2 is fixed. First, we expand
log 1og x²)
- - + log
(23a)
log ar) logp(x| ((=) - A(z)) + log h(x) + logp(=) logp(x))
- p(x)
(23b)
where A(z) A(f(z) and B(x) B(g(z)). We can therefore represent
log g(= x) [(gp(> g(x)), (xx) + logp(x) log h(x)
(24)
(v(z), f2)) + logp(2) log h'(z) A(z))
(25)
(Wa()) (u.(z),
(26)
where
(=(2) = (v(z), 1);
(27)
(x(x) = (v(x), 1);
(28)
9. = (g(x), logp(x) B(xx log h(x));
(29)
fe(2) (f(z), logp(z) A(z) log h1 (2)).
(30)
With this representation we have:
EnEx pa(z) (e(()) Pa(x) log [2) x)) logp(2| =: 2
22
(31)
(32)
Let V be the matrix with rows (x) for all x e X. Let be the matrix with rows (x(x) for all
E X. We can rewrite the condition (31) in the form
{ = [=(=)) Gt)aaz), || = 2
(33a)
(33b)
where & IR X1 is the vector of residuals. Let P the orthogonal projection onto the range of V in the space H. Multiplying (33a) by P on the left, we obtain
PV PG PP.
(34)
Because PV we obtain
Vf((2) (G) PE.
(35)
We therefore can consider the decoder network approximation fz(2) Wee(2), where W is the solution to the consistent system of linear equations VW -GG and is independent of z. We can therefore express
[v) f=)())) || S || A,
(36)
where the inequality holds because P is orthogonal projection in H.
14

---

Published as a conference paper at ICLR 2022
We obtained that the decoder network fz) can be approximated by linear mapping W th (2) such that
(xxxaee))(( (v,(s),, (,)) A
(37)
Expressing back
(v(()),((()) =v((x), ((=)) + logp(=) - A(=) log (z) logp(æ> =) + logp(z) log h(2))()))))))r))) log h (z) logp(z, 2) logh(z) log h/ (z)2
(38a)
(38b)
(38c)
we obtain that
Exex Pa(x) logp(a, log 2) s A?
(39)
where
logā(z, 2) = log h(x) log (=)) (V)(x), Weboz)).
(40)
Proposition A.2. Under model Assumption 1, X and 2 be discrete (finite) sets and let E be chosen. If q(] z) 2 and p(=[2) 2 for all X, where X X and
(41)
then
ExEX (e( - log ()) S a + o(e).
(42)
Proof.
Let us denote s(x) = DKI.(a(=1 [| Pinsker's inequality assures for each x
sup
|P (S) (()= S <(x)/2.
(43)
scz
Substituting S {=} we obtain
|x2] g(>1)) s c(x)/2.
(44)
By taking expectation in pa(z) on both sides we obtain a variant of Pinsker's inequality:
(p((=))((<)( (= < Bps(n)=(2/22 1/2 pp)) [x ) p(2[x))] S E//2.
(45)
Notice that the LHS depends on the given z. Because summands are neg-negetive follows that
VX X
ExEx Pa(x))p(=1 - <<=l e/2.
(46)
This inequality will be used in several places below.
Consider E X such that p(=|2=) > (==1xr). Then
log m²) log (=)((2)))22=-) log² p(=|s)
(47a)
q()|))
= =log² (1 + p(*1 -(=1=))
(47b)
< log" (1 + p(al p(a1))))(s1=)
(47c)
where the inequality holds because log² is monotonously inereasing for arguments greater equal than 1, which is ensured. Let us now consider x E X such that p(2| x)) < q(>l Then
g(s(=) 1o æ) log ( log"
(48a)
p(=]=)
log (1 + g(al d(+l=)-p(ela)
(48b)
p x)
< log (1 + q(z -(=1=))-(=1a) a
(48c)
In total, we obtain
logg(=>>>> - log 22 < log" (I
Ip(a -(]
(49)
15

---

Published as a conference paper at ICLR 2022
Let us denote u(x) (M())))-((((a)) and partition the set X into two parts:
Xi { X
a u(x) mo
(50a)
X₂ = {2 E X
u(x) 2 uo
(50b)
where chose uo [Ve-1, 1,5] for reasons to be clarified below. For both parts, i.e. k 1, have
Exex, Pa(z)] log g(= x)) log s E.Ex, Pa(x) log (1 + l=(e|=)=((z1a)
(51)
For 11 (51) can be further bounded
S Exex Pa(a:) log(1 + |p(z|a)_gle )
(52a)
log Exex, Pa(x)
[n(x] -
(52b)
log (pa(Xi)+ 2/2 Ezex, pa(=))p(z1=) - {<212))²))
(52c)
log (1+
2 ) 2a² o(E),
(52d)
2a²
where the first inequality holds for uo 5, because in this case (1 u) S log(1 +u²) holds, the second inequality is the Jensen's inequality for log and the last inequality uses (46) under monotone log.
For X2 we have the following. Let = pa((2) = ExEx, pa(x). We can express
Exex, log (1 + (
(53a)
-V(Ewex Pa( log² (1 (=(2)=))((z(a))
(53b)
a
Using that bound (53b)
< u² on X2 and that log + u) is monotone for a positive argument, we can
<V <v(Erexs Pa(x) X2) log? (1+ [
(54)
Further, using that log + v) is concave on X2 for 2 e 1, we have
< V log²
Exex, X2) Ip(a| |ع(يا -
(55a)
= Vlogg*
(1 +1=EzEx, pa(x| (
(55b)
S Vlogg
(55c)
where in the last step we used (46).
Next show that V itself bounded above by 2
follows from
v Σ pa(x)uf < Σ pa(x) ((=l=))((=(2)])
Σ
pa(x) -(=1==)-((+(
(56)
2
zEX;
xxX²
mEX
where the last inequality is again (46).
Now, letting r Z. 2 u > we can bound (55c) as follows
Vlog² (1 zá:v) of log? +++)
(57a)
< - sup,>1
log² (1+r) S L 0.64.
(57b)
Theorem follows by Proposition A.2 and Proposition A.I with the choice X = X.
16

---

Published as a conference paper at ICLR 2022
B DISCUSSION
In this section we discuss further connections to related work and some open questions.
Cremer al. (2018) showed experimentally that using more expressive class of models for the encoder reduces not only the posterior family mismatch but also the amortization error. This obser- vation is compatible with our results: increasing the expressive power of the encoder admits tight VAEs with more complex dependence of on z. Indeed, increasing the expressive power of the encoder in our setting means extending its sufficient statisties (2)2 by new components. While keeps the simple linear dependence g(x) more expressive GL. p(=| Conversely, our results suggest that increasing the complexity of the encoder network in order to reduce the amortization gap is only useful for models that are far from the consistent set. Furthermore, there could be negative impact from increasing the model depth in practice: (Dai et al., 2020) demonstrated that the risk of the learning converging to suboptimal solution (in particular leading to more collapsed latent dimensions) increases with decoder depth.
WTv(x)) characterizing tight VAEs, it does lead to a
Lucas et al. (2019) showed that any spurious local minima in linear Gaussian VAEs are entirely due to the marginal log likelihood and that the ELBO does not introduce any new local minima. It is good question² whether something similar can be said about the EF VAE generalization. To our best knowledge this is not straightforward in such a general setting. The result of Lucas et al. (2019) possible thanks to the fact that for linear Gaussian models ELBO is analytically tractable and its stationary point conditions can be written down and analyzed. In the general EF setup, which includes, e.g., MRF VAEs, this does not appear possible. On the other hand, for any consistent EF VAE, the decoder posterior must be in the EF of the encoder. Therefore the optimal encoder could be easier to find analytically or numerically using forward KL divergence and not the reverse KI. divergence used in ELBO, thus circumventing the question about local optima of ELBO. Since ELBO at the optimal encoder is tight for a consistent VAE, this could be an alternative way to find the global maximum.
A recent work by Sicks et al. (2021) extends the result of Lucas Lucas et al. (2019) in that they develop an analytical local approximation to ELBO, which is exact in the Gaussian linear model case and is a lower bound on ELBO for Binomial observation model. These results allow to ana- lyze ELBO (and in particular the posterior collapse problem) locally under the assumption that the decoder's 's mapping f(2) is (locally) an affine mapping of 2. Our Theorem implics it must be so globally for tight VAES in several special cases (e.g., Bernoulli model), while in general it is an affine mapping of (z).. These connections indicate that a better understanding of VAEs can be reached in the setting where either decoder or encoder or both are consistent with the joint model (7).
We restricted this study to exponential families. While in richer models discussed in the introduction, the approximation error still exists, it is made small by design, and is less relevant and harder to analyze it theoretically. One possible open direction where such analysis would make sense is to consider fully factorized non-exxpnential cases, e.g., Student-t VAEs (Takahashi al., 2018) or models satisfying hierarchical or partial factorization (Maalø e et al., 2019).
C DETAILS OF EXPERIMENTAL SETUP
C.I ARTIFICIAL EXAMPLE
We give here the achieved likelihood and ELBO values for this experiment. The negative entropy Σx p= (a:) log, p" (z)) of the ground truth model, i.e., the best reachable data log-likelihood, is -3.65.
The ELBO of the pre-trained VAE is -3.74. During the second step of training, i.e., the joint learning of the encoder and decoder by ELBO maximization, the ELBO value increases from -3.74 to -3.70. The data log-likelliie likelihood Σ, p" (a:) log (I) drops at the same time from -3.65 to -3.68. The approximation error (2) in the decoder caused by using the factorized encoder is only 0.03 nats, but the qualitative difference between the ground truth model and the BLBO optimizer model shown in Fig. appears detrimental.
"pointed out by reviewers
17

---

Published as a conference paper at ICLR 2022
C.2 BERNOULLI VAE FOR TEXT DOCUMENTS
Dataset In this experiment we used the version of the 20Newsgroups data set (Lang & Rennie, 2008) denoted as "processed" by the authors. The dataset contains bag-of-words representations of documents and is split into a training set with 11269 documents and a test set with 7505 documents. We keep only the 10000 most frequent words in the training set, which is a common pre-processing (each of the omitted words occurs not more than in 10 documents).
Optimization train VAE we used the state-of-the- art unbiased gradient estimator ARM (Yin & Zhou, 2019) and Adam optimizer with learning rate 0.001. We did not the test set for parameter selection. We train for 1000 epochs using 1-sample ARM and then for 500 more epochs using 10 samples for computing each gradient estimate with ARM. We report the lowest negative ELBO (NELBO) values for the training set and test set during all epochs.
Models
The decoder model (13) describes words as independent draws from a categorical distri- bution specified by the neural network f(z). This network respectively has structure
Linear -> (ReLU + Linear)
>
Logsoftmax.
xdhidden
The input dimension equals to the number of latent bits, the output dimension equals the number of words the dictionary, 10000. For decoders with dhidden 1, 2 the hidden layers contained 512 units.
The encoder networks e2, e3 take on the input word frequencies x/ Σn Th, the encoder network el takes on input word counts For the deep encoder e2 we used 2 hidden ReLU layers with 512 units each.
18


Reviews:
[{"review_id": "qJzqkiCXTnZ", "reviewer": "Reviewer_vibV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies the VAE approximation error, when the encoder and decoder distributions are from conditional exponential families (EFs). Using the definition of a consistent set Theta_{Phi} as above eq 3 (i.e. there exists a phi s.t. q_{phi}(z|x) \\equiv p_{theta}(z|x)), Theorem 1 characterizes the general form of the joint probability (an EF-Harmonium as per eq 7) which satisfies the consistent set condition for exponential family VAEs. It is interesting that this is an *undirected* model, while VAEs are usually thought of a directed z->x latent variable model. This result is interesting, but as it is a direct consequence of a theorem by Arnold and Strauss (1991) [as cited] it clearly has less novelty than if it was derived by the authors.\n\nIt is shown that a small VAE posterior mismatch does not have to imply closeness of the optimal decoder to some decoder in the consistent set, as per the counterexample below Defn 1. However, by prohibiting nearly deterministic encoders and decoders, Theorem 2 can be proved where a small VAE posterior mismatch does imply closeness of the optimal decoder to some decoder in the consistent set.\n\nSec 3.1 presents specific examples of such EF-VAEs, namely the diagonal Gaussian VAE, the Bernoulli-MRF VAE, and the Bernoulli VAE for semantic hashing. In this last case, the form derived from the EF-VAE differs somewhat from the \"standard\" version, by using word counts rather than frequencies.\n\nSec 4 give some experimental results. Sec 4.1 shows how a semantic hashing representation of the posterior (i.e. two +1/-1 bits as opposed to a 4-dim one hot code) does induce an approximation error in the model.\n\nSec 4.2 carries out an interesting experiment, where data is generated from a GAN trained on the CelebA dataset (plus Gaussian noise in image space). Given the (z,x) pairs a decoder can be readily learned for this model. What is interesting is that optimization of the decoder and encoder parameters to optimize the ELBO degrades the FID performance. This is a nice demonstration of how optimizing the ELBO for such a model can harm the quality of the generator.\n\nSec 4.3 show show the proposed correction (using word counts rather than frequencies) obtains better ELBO scores.\n\nOverall, I find that this paper provides a useful extension to knowledge around VAEs. It provides a new connection between tight EF-VAEs and RBM-like models (EF-Harmoniums), and shows that the consistent subset cannot be enlarged by using a more complex encoder network if the encoder's sufficient statistics remain unchanged.  The experiments in sec 4.2 show interesting results about the \"unlearning\" of the true generator when the ELBO is optimized, and sec 4.3 shows that a more careful modeling choice (using word counts rather than frequencies) inspired by Theorem 1 leads to better ELBO performance.\n\n* Other points\n\np 3 footnote -- punctuate this footnote better.\n\np 4 \"It turns out that the KL divergence has a pinhole ..\" -- \"pinhole\" is a bit of an odd choice of word here -- maybe \"leak\"?\n\np 6 Fig 2 -- to my eyes it was very hard to see the difference between the pink and orange colors -- please choose different colors to make this clearer.", "summary_of_the_paper": "The paper studies the VAE approximation error, when the encoder and decoder distributions are from conditional exponential families (EFs). Theorem 1 characterizes the form of the joint probability (an EF-Harmonium) which is consistent, i.e. where q_{phi}(z|x) \\equiv p_{theta}(z|x). Sec 4.2 shows a v interesting example of Gaussian VAEs for CelebA images, where it is shown that a VAE \"unlearns\" the ground truth solution (being pulled away from the likelihood optimizer towards the consistent subset). Sec 4.3 shows another interesting example, where a careful formulation of the model identifies that it should use word counts rather than frequencies, and this is demonstrated empirically (Table 2).", "summary_of_the_review": "Overall, I find that this paper provides a useful extension to knowledge around VAEs. It provides a new connection between tight EF-VAEs and RBM-like models (EF-Harmoniums), and shows that the consistent subset cannot be enlarged by using a more complex encoder network if the encoder's sufficient statistics remain unchanged.  The experiments in sec 4.2 show interesting results about the \"unlearning\" of the true generator when the ELBO is optimized, and sec 4.3 shows that a more careful modeling choice (using word counts rather than frequencies) inspired by Theorem 1 leads to better ELBO performance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XrmzhxV20oS", "reviewer": "Reviewer_GwqU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths.\nThe paper identifies and analyzes the approximation error in a general and realistic instance of the VAE with exponential family encoder and decoder and have backed it up with real examples from the literature including VAEs with RBM encoders and have used the theory to suggest improvements in the example of semantic hashing.\n\nThe theoretical justification of the suggestion that with exponential family encoder/decoder VAEs increasing depth does not help when the model is consistent or close to consistent is quite interesting. Some past work has shown empirical evidence that suggests that in many restricted cases adding depth does not help. I am not, however, aware of a clear theoretical justification of this empirical fact in the literature for the exponential family.\n\nFurthermore, it has been shown in (Dai and Wipf, Diagnosing and Enhancing VAE models, 2019) that with Gaussian output models one can construct VAE decoders as bijections when the manifold dimension is equal to the data dimension with the consequence that the decoder is not linear. This paper shows that with some limits on the densities to exclude deterministic solutions, the VAE solution remains close to the consistent solution which matches the posterior perfectly. Taken together the results imply that VAE solutions are either consistent or close to consistent when used with exponential family distributions under some constraints.\n\nWeaknesses/Improvements/Questions\n\nAre the sub-optimal results shown in the experiments a result of the factorization assumption or are they a general problem with the ELBO objective? One might interpret the results in table 1 as suggesting that the problem might lie with the ELBO as well since increasing the ELBO lowers the FID. My question is whether the poor FID results from ELBO pathologies such as unbounded likelihood when modeling with continuous models or is it something inherent to the ELBO? Or is it a combination of the factorized encoder plus the ELBO? I don't think this point is as well addressed in the paper (even though the ELBO is identified as the culprit in the paragraph above table 1, for instance).\n\nLucas et al (2019) suggest for the case of linear Gaussian models that any spurious local minima are entirely due to the marginal log likelihood and that the ELBO does not introduce any new local minima. Can something similar be said for the generalization to exponential family distributions?\n\nSimilarly the toy example explains quite well the limitations of the factorization assumption but I don't think it says much about the limitations of the ELBO. Perhaps the authors can comment on this.\n\nAs a minor point, it would have been nice to see implications of theorems 1 and 2 explored in the experiments to see that exponential family VAEs remain close to linear parameterized exponential models.\n\nClarity.\nThe paper is generally clear and the claims are easy to follow.\n\nCorrectness.\nThe arguments in the paper appear correct to me, but I have not checked the proof in the appendix very carefully.\n\nRelation to Previous Work.\nThe main portion of the paper can be seen as an extension of the work on linear VAEs (Lucas et al) to exponential families. The paper presents a theoretical analysis of the empirically observed approximation error previously observed in the literature. The idea of poor approximation is not novel (e.g., Cremer et al, 2018, and acknowledged in the paper) but the theoretical explanation is of some interest.", "summary_of_the_paper": "Summary.\nThis paper presents an analysis of the approximation error of VAE models when the encoder and decoder are from exponential families. They show that when the model is consistent (i.e., the encoder is able to match the posterior), the encoder and decoder distributions are exponential family distributions that are linearly parameterized. The paper also shows that barring pathological cases, when the model is tight the decoder distribution is close to a linearly parameterized exponential family distribution. In particular, this shows that in this case additional depth is not useful.\n\nDespite the restriction to exponential family distributions, the paper shows a number of real instances of models that lie in this class including Gaussian VAEs and VAEs with RBM encoders. For Bernoulli VAEs for semantic hashing, the paper proposes and verifies improvements to a model presented in the literature.\n", "summary_of_the_review": "Overall, I think this is a well-written paper with an argument for the limitations of an important class of models. I did not find some of the experiments as convincing as the rest of the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U0tW88jrV4W", "reviewer": "Reviewer_3pHE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very insightful paper with strong theoretical and empirical results and good writing. When first reading the paper my expectations were low because the problem of VAE posterior collapse and degeneracy is an old and tired one, however, the solution proposed by the paper is very interesting and to my knowledge completely novel. \n\nTheory: \n\nThe theoretical contribution is very insightful. This paper observes that when the encoder and decoder distributions belong to the exponential family, only general linear models satisfy ‘consistency’ or even approximate consistency. Even though Thm 1 is not new, bringing up this result in the context of VAE is a valuable contribution. I think Thm 2 is a very interesting new result. This theorem somewhat conclusively puts a rest to the debate of whether flexible encoder distribution is necessary when learning with ELBO, by showing that unless the VAE is a flow or if the decoder is very simple, a flexible encoder distribution is needed. \n\nThe paper also shows in Corollary 1 that a flexible encoder network cannot make up for the lack of flexibility in encoder distribution. I think this is an interesting result that provides a lot of insight in the discussion of VAE mode collapse. \n\nAs the authors acknowledge, the analysis has a short-coming in that VAE models that approach a flow are exempt from the impossibility results, but I think that this exception does not detract from the interest of the general result. \n\nExperiments: \n\nThe experiments are an insightful addition to the theoretical analysis. In particular, I really like the experiment where even with pretrained near-optimal decoders and encoders, further training on the ELBO degrades the decoder. I think this is a very clear experiment to show that failure of VAEs is often not caused by optimization issues, but rather the inherent limitations of ELBO learning with inflexible encoder distribution. \n\nOther comments:\n\nThe literature review is somewhat lackluster, as there are probably tens of refereed papers discussing similar issues. For example, the following paper also provides perspectives on posterior collapse. \n\nInfoVAE: Balancing Learning and Inference in Variational Autoencoders \n\nThe Usual Suspects? Reassessing Blame for VAE Posterior Collapse\n\nThat being said, to my knowledge the perspective in this paper is novel, but a good related work discussion on the debate around this topic can significantly strengthen the paper. \n\nSmall issues:\n\nIn Eq.(4a,4b) the partition function should also depend on theta and phi, also small typo in (4b) \n", "summary_of_the_paper": "This paper proposes theoretical and empirical results on why VAEs tend to fail when the encoder distribution is not flexible enough. ", "summary_of_the_review": "Even though the problem is an old one that has been well studied, the theoretical results provide a very refreshing and novel perspective. The experimental results are also precisely the right experiments to complement the theory. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nfljm-FpaY2", "reviewer": "Reviewer_8hFF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**presentation & related work**\n\nAuthors do a nice job in this aspect. \n\n**novelty & insights**\n \nThe interpretation and insight provided in theorem 1 is an interesting addition to the literature on analyzing VAEs, and is novel to the best of my knowledge. The empirical novelty, on the other hand, seems limited, given that the main experiments are demonstrating how an inferior variational family may lead the generative model astray even when the generative distribution are optimally initialized. \n\n**experiments & empirical results**\n\nThe paper scores adequately in this regard.\n\n**technical correctness**\n\nI read through the proof of Theorem 1 and proof of the cited reference. These results are correct. ", "summary_of_the_paper": "Author studies VAEs under the assumption that the decoding distribution p(x|z) and approximate posterior distribution q(z|x) are in the exponential family. They show that the consistent set for the decoding distribution -- the set of p(x|z) which results in p(z|x) = q(z|x) for all z -- has a log-bilinear form in x and z. Consequently, optimizing an ELBO with such forms of exponential family p(x|z) and q(z|x) could result in a deviation from the maximum likelihood solution theta for p_theta(x). ", "summary_of_the_review": "The paper is overall of good quality. While the empirical novelty is somewhat limited, theorem 1 is insightful. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Wide agreement from the reviewers.  Interesting theorems.  Empirical work illustrates the theory.
Claim and insight: failure of VAEs is caused by the inherent limitations of ELBO learning with inflexible encoder distribution.
Good discussion pointed out related work and insights from the experiments.
Decision: Accept (Spotlight)