Year: 2021

Abstract:
The importance of Variational Autoencoders reaches far beyond standalone generative models — the approach is also used for learning latent representations and can be generalized to semi-supervised learning. This requires a thorough analysis of their commonly known shortcomings: posterior collapse and approximation errors. This paper analyzes VAE approximation errors caused by the combination of the ELBO objective and encoder models from conditional exponential families, including, but not limited to, commonly used conditionally independent discrete and continuous models. We characterize subclasses of generative models consistent with these encoder families. We show that the ELBO optimizer is pulled away from the likelihood optimizer towards the consistent subset and study this effect experimentally. Importantly, this subset can not be enlarged, and the respective error cannot be decreased, by considering deeper encoder/decoder networks.

heading: 1 INTRODUCTION
text: Variational autoencoders proposed by Kingma & Welling (2014) strive at learning complex data distributions pd(x), x ∈ X in a generative way. They introduce latent variables z ∈ Z and model the joint distribution as pθ(x | z)p(z), where p(z) is a simple distribution that is usually assumed to be known. The conditional distribution pθ(x | z), called decoder, is modeled in terms of a deep network parametrized by θ ∈ Θ. Models defined in this way allow to sample from pθ(x) = Ep(z)pθ(x | z) easily, however at the price that computing the posterior pθ(z |x) = pθ(x | z)p(z)/pθ(x) is usually intractable. To handle this problem, VAE approximates the posterior pθ(z |x) by an amortized inference encoder model qφ(z |x) parametrized by φ ∈ Φ. Given the empirical data distribution pd(x), the model is learned by maximizing the evidence lower bound (ELBO) of the data loglikelihood L(θ) = Epd log pθ(x). It can be expressed in the following two equivalent forms:
LB(θ, φ) = Epd [ Eqφ log pθ(x | z)−DKL(qφ(z |x) ‖ p(z)) ] (1a)
= L(θ)− Epd [ DKL(qφ(z |x) ‖ pθ(z |x)) ] . (1b)
The first form allows for stochastic optimization of ELBO while the second form shows that the gap between log-likelihood and ELBO is exactly the mismatch between the encoder and the posterior.
VAEs constitute a powerful deep learning extension of the expectation maximization approach to handle latent variables. They are useful not only as generative models but also, e.g., in semisupervised learning (Kingma et al., 2014; Mattei & Frellsen, 2019). Furthermore the encoder part constructs an efficient embedding of the data in the latent space, useful in many applications. The outreach of the VAE approach requires therefore a careful empirical and theoretical analysis of the problems and trade offs involved. The most important ones are (i) posterior collapse (He et al., 2019; Lucas et al., 2019; Dai et al., 2018; Dai & Wipf, 2019) and (ii) approximation errors caused by an inappropriate choice of the encoder family.
The VAE approximation error has been studied (e.g., Cremer et al. 2018; Hjelm et al. 2016; Kim et al. 2018) so far mainly empirically. The problem also occurs and is well-recognized in the context of variational inference and variational Bayesian inference, where the target posterior distribution is expected to be complex. It is commonly understood, that the mean field approximation of pθ(z |x) by qφ(z |x) in (1b) significantly limits variational Bayesian inference. In contrast, in VAEs, the decoder may adopt to compensate for the chosen encoder family. The effect of this coupling, we
believe, is not fully understood. The phenomenon of decoder adopting to the posterior was experimentally observed, e.g., by Cremer et al. (2018, Section 5.4), noting that the approximation error is often dominated by the amortization error. Turner & Sahani (2011, Sec. 1.4) analytically show for linear state space models that simpler variational approximations (such a mean-field) can lead to less bias in parameter estimation than more complicated structured approximations. Similarly, Shu et al. (2018) view the VAE objective as providing a regularization and show that making the amortized inference model smoother, while increasing the amortization gap, leads to a better generalization.
The common (empirical) understanding of the importance of the gap between the approximate and the true posterior has led to many generalizations of standard VAEs, which achieve impressive practical results, notably, tighter bounds using importance weighting (Burda et al., 2016; Nowozin, 2018), encoders employing normalizing flows (Rezende & Mohamed, 2015; Kingma et al., 2016), hierarchical and autoregressive encoders (Vahdat & Kautz, 2020; Sønderby et al., 2016; Ranganath et al., 2016), MRF encoders (Vahdat et al., 2020) and more. While these extensions mitigate the posterior mismatch problem, they often come at a price of a more difficult training and more expensive inference. Furthermore, simpler encoders may be of practical interest. Burda et al. (2016, Appendix C) illustrates that IWAE approximate posteriors are less regular and more spread out. In contrast, factorized encoders provide simple embeddings useful for downstream tasks such as semantic hashing (Chaidaroon & Fang, 2017).
The aim of this paper is to study the approximation error of VAEs and its impact on the learned decoder. We consider a setting that generalizes many common VAEs, in particular popular models where encoder and decoder are conditionally independent Bernoulli or Gaussian distributions: we assume that both decoder and encoder are conditional exponential families. We identify the subclass of generative models where the encoder can model the posterior exactly, referred to as consistent VAEs. We give a characterization of consistent VAEs revealing that this set in fact does not depend on the complexity of the involved neural networks. We further show that the ELBO optimizer is pulled towards this set away from the likelihood optimizer. Specializing the characterization to several common VAE models, we show that the respective consistent models turn out to be RBMlike in many cases. We experimentally investigate the detrimental effect in one case and show that a simpler but more consistent VAE can perform better in the other.

heading: 2 PROBLEM STATEMENT
text: We adopt the following notion of approximation error. Consider a generative model class PΘ = {pθ(x, z) | θ∈Θ}, the encoder class QΦ = {qφ(z |x) | φ∈Φ} and the data distribution pd(x). The maximum likelihood generative model is given by θML ∈ argmaxθ∈Θ Epd(x) log pθ(x). For a decoder with parameters θ we define its approximation error as the likelihood difference L(θML)− L(θ). Respectively, the VAE approximation error is defined for a given θ as:
L(θML)−maxφ LB(θ, φ) ≥ L(θML)− L(θ). (2)
In order for this error to become zero, two conditions are necessary and sufficient:
• Parameters (θ, φ) must be optimal for the ELBO objective.
• ELBO must be tight at (θ, φ), i.e., LB(θ, φ) = L(θ). Assuming that the optimality can be achieved, we study the non-tightness gap L(θ) − LB(θ, φ). From (1b) it expresses as Epd [ DKL(qφ(z |x) ‖ pθ(z |x)) ] . It follows that ELBO is tight at (θ, φ) iff qφ(z |x) ≡ pθ(z |x). Hence, we define the consistent set ΘΦ ⊆ Θ as the subset of distributions pθ(x, z) whose posteriors are in QΦ, i.e.,
ΘΦ = { θ ∈ Θ ∣∣ ∃φ ∈ Φ : qφ(z |x) ≡ pθ(z |x)}. (3) The KL-divergence in the ELBO objective (1b) can vanish only if θ ∈ ΘΦ. If the likelihood maximizer θML is not contained in ΘΦ, then this KL-divergence pulls the optimizer towards ΘΦ and away from θML as illustrated in Fig. 1.
We characterize the consistent set ΘΦ, on which the bound is tight, and show that this set is quite narrow and does not depend on the complexity of the encoder and decoder networks beyond simple 1-layer linear mappings of sufficient statistics.

heading: 3 THEORETICAL ANALYSIS
text: We consider a general class of VAEs, where both encoder and decoder are defined as exponential families. This class includes many common models, in particular Gaussian VAEs and Bernoulli VAEs with conditional independence assumptions, but also more complex ones, e.g., where the encoder is a conditional random field (Vahdat et al., 2020)1.
Assumption 1 (Exponential family VAE). LetX andZ be sets of observations and latent variables, respectively. We consider VAE models defined by
pθ(x | z) = h(x) exp [ 〈ν(x), fθ(z)〉 −A(fθ(z))] (4a)
qφ(z |x) = h′(z) exp [ 〈ψ(z), gφ(x)〉 −B(gφ(x))], (4b)
where ν : X → Rn and ψ : Z → Rm are fixed sufficient statistics of dimensionality n and m; fθ : Z → Rn and gφ : X → Rm are the the decoder, resp., encoder networks, depending on learnable parameters θ, resp. φ; h : X → R+, h′ : Z → R+ are strictly positive base measures and A, B denote the respective log-partition functions.
Notice that this assumption imposes no restrictions on the nature of random variables x and z. They can be discrete or continuous, univariate or multivariate. Similarly, it imposes no restrictions on the complexity of the decoder and encoder networks fθ(z) and gφ(x).
Characterization of the consistent set. In the first step of our analysis, we investigate the conditions under which the approximation error of an exponential family VAE can be made exactly zero. As discussed above, a tight VAE (θ, φ) must satisfy ∀(x, z) qφ(z |x) = pθ(z |x), which leads to the following theorem.
Theorem 1. The consistent set ΘΦ of an exponential family VAE is given by decoders of the form p(x | z) = h(x) exp [ 〈ν(x),Wψ(z)〉+ 〈ν(x), u〉 −A(z) ] , (5)
where W is a n×m matrix and u ∈ Rn. Moreover, the corresponding encoders have the form q(z |x) = h′(z) exp [〈 ψ(z),WT ν(x) 〉 + 〈ψ(z), v〉 −B(x)], (6)
where v ∈ Rm.
This is a direct consequence of a theorem by Arnold & Strauss (1991) (see Appendix A.1 for more details). For a tight VAE, Theorem 1 states that the decoder and encoder take the form (5) and (6) with the interaction between x and z parametrized by a single matrix W instead of the complex neural networks with parameters θ, φ. Thus they turn out to be generalized linear models (GLMs). The corresponding joint probability distribution is an EF Harmonium (Welling et al., 2005):
p(x, z) = h(x)h′(z) exp ( 〈ν(x),Wψ(z)〉+ 〈ν(x), u〉+ 〈ψ(z), v〉 −A ) . (7)
1Notice, however, that this class does not include VAEs with advanced encoder families like normalizing flows, hierarchical and autoregressive encoders.
Corollary 1. The subset ΘΦ of consistent models can not be enlarged by considering more complex encoder networks g(x), provided that the affine family WTν(x) can already be represented.
Corollary 2. Let the decoder network family be affine in ψ(z), i.e., f(z) = Wψ(z) + a and let the encoder network family g(x) include at least all affine maps V ν(x) + b. Then any global optimum of ELBO attains a zero approximation error.
VAEs can escape consistency when they degenerate to an invertible flow. In practice, VAE models are almost never tight. It is therefore natural to ask, whether a small VAE posterior mismatch error implies closeness of the optimal decoder to some decoder in the consistent set.
Definition 1. A VAE (pθ, qφ) is ε-tight for some ε > 0 if Epd(x)[DKL(qφ(z |x) ‖ pθ(z |x))] ≤ ε.
It turns out that the the KL divergence has a leak, which allows a VAE to approach tightness while not approaching consistency. In the continuous case an example satisfying ε-tightness with non-linear decoder follows from Dai & Wipf (2019, Theorem 2). They show for a class of Gaussian VAEs with general neural networks fθ, gφ, that it is possible to build a sequence of network parameters θt, φt with the following properties: i) the target distribution is approximated arbitrary well, ii) the posterior mismatch DKL(qφt(z |x) ‖ pθt(z |x)) approaches zero and iii) both the encoder and decoder approach deterministic mappings. The VAE thus approaches a flow model (or invertible neural network) between the data manifold and a subspace of the latent space (Dai & Wipf, 2019). Clearly, in a general case the flow must be non-linear. A similar case can be made for discrete variables, see Example A.1.
Non-deterministic nearly-tight VAEs approach consistency. We would however argue that the mode where the decoder and encoder are nearly-deterministic is not a natural VAE solution. By making additional assumptions, excluding such deterministic solutions, and restricting ourselves to the finite space in order to simplify the analysis, we can show that the decoder of an ε-tight VAE does indeed approach a GLM.
Theorem 2. Let (pθ, qφ) be an exponential family VAE (Assumption 1) on a discrete space X × Z with strictly positive encoder qφ(z |x) and decoder posterior pθ(z |x), both bounded from below by α > 0. If the VAE is ε-tight, then there exists W ∈ Rn,m such that the decoder can be approximated by a GLM of the form p̃(x | z) = h̃(x) exp [ 〈ν(x),Wψ(z)〉 − Ã(z) ] with the error
Epd(x) [( log pθ(x | z)− log p̃(x | z) )2] ≤ ε2α2 + o(ε) ∀z ∈ Z. (8)
The proof is given in Appendix A.3.

heading: 3.1 CASES ANALYSIS
text: This subsection gives a detailed analysis of consistent VAE models in several concrete cases of practical interest.
Diagonal Gaussian VAE Let us consider a Gaussian VAE, as commonly applied to image generation (e.g., Dai & Wipf (2019)). Let X = Rn, Z = Rm, p(x | z) = N (x |µd(z), σ2dI), q(z |x) = N (z |µe(x),diag(σ2e(x))), where µd, µe and σe are neural networks and σd is a common pixel observation noise parameter. The decoder has minimal sufficient statistics ν(x) = x and the base measure h(x) = N (x | 0, σ2dI). The encoder has minimal sufficient statistics ψ(z) = (z, z2), where the square is coordinate-wise. Theorem 1 implies that a tight optimal VAE has the joint model
p(x, z) ∝ h(x) exp [〈 x,Wz + V z2 + a 〉 + 〈b, z〉+ 〈 c, z2 〉] (9)
for some matrices W , V and vectors a, b, c. Furthermore, the integral over z must be finite for all x and therefore xTV + c < 0 must hold for all x ∈ Rn. This is possible only if V = 0 and c < 0. The joint distribution is therefore a multivariate Gaussian and the same holds for its marginal p(x). The neural network µd(z) must degenerate to µd(z) = σ2d · (Wz + a) and the two neural networks for the encoder to σ2e(x) = −1/2c and µe(x) = −(WTx + b)/2c, where divisions are coordinate-wise. VAEs with such simplified, linear Gaussian encoder-decoder pairs, called "linear VAEs" (Lucas et al., 2019) are known to be consistent and to match the probabilistic PCA model (Dai et al., 2018; Lucas et al., 2019). In this context, our Corollary 2 is a generalization of (Lucas et al.,
2019, Lemma 1) showing consistency of linear VAEs, to decoders in any exponential family with natural parameters being a linear mapping of any fixed lifted latent representation ψ(z).
We argue that a joint Gaussian model is too simplistic to generate complex data such as realistic images and that in this case the VAE error is detrimental. In Section 4.2 we experimentally confirm that optimizing ELBO for a general decoder network µd causes qualitative and quantitative degradation relative to the ML decoder. Note that if we allowed σd to be dependent on z, the resulting joint statistics in ν ⊗ψ would include terms x2z, x2z2. The joint distribution would not be Gaussian and may be in fact multi-modal (see Anil Bhattacharayya’s distribution in Arnold et al. 2001).
Bernoulli-MRF VAE Vahdat et al. (2020) proposed to consider encoders in the Markov Random Field (MRF) family, in particular encoders of the form q(z|x) ∝ exp(〈z1, V (x)z2〉+ 〈b1(x), z1〉+ 〈b2(x), z2〉), where z1, z2 are two groups of latent variables and interaction weights V , b1, b2 are computed by the encoder network. In this case q(z|x) is itself a (conditional) RBM. While evaluating q(z|x) is difficult, MCMC sampling is efficient. We assume binary observations x and a conditionally independent Bernoulli decoder family as above. The decoder thus has sufficient statistics νe = (x) and the encoder has ψ = (z1, z2, z1⊗z2). Introducing homogeneous constant components x0 = z 1 0 = z 2 0 = 1, the family of consistent joint distributions can be compactly described as
p(x, z) = exp [∑
i,j,kWi,j,k xi z 1 j z 2 k
] , (10)
where the summation in all indices starts from 0 and −W0,0,0 is the log-partition function. This joint model is a higher order MRF with the highest order potentials given by cubic monomials.
Standard Bernoulli VAEs are a special case of the Bernoulli-MRF model, obtained when the interaction weights V are zero. The joint distribution of such tight optimal VAEs takes the form
p(x, z) = 1c exp(x TWz + uTx+ vTz), (11)
which is a restricted Boltzmann machine (RBM). Since RBMs are well known for being useful in many applications (dimensionality reduction, collaborative filtering, feature learning, topic modeling), we hypothesize that they can make a good baseline for Bernoulli VAEs and furthermore that the effect of pulling the VAE solution towards an RBM may be benign in case of insufficient data. For example IWAE test likelihood in (Burda et al., 2016) is worse than that of an RBM (Burda et al., 2015) on the Omniglot dataset. Further debiasing of the IWAE may not improve test likelihood (Nowozin, 2018).
Bernoulli VAE for Semantic Hashing One important application of Bernoulli VAEs is the semantic hashing problem, initially proposed and modeled with RBMs (Salakhutdinov & Hinton, 2009). The problem is to assign to each document / image a compact binary latent code that can be used for quick retrieval by the nearest neighbor search. We will detail now a more recent VAE model for text documents (Chaidaroon & Fang, 2017; Shen et al., 2018) and show that it can be tight only in a full posterior collapse. We correct the encoder so as to allow a larger consistent set and observe that the resulting consistent joint distribution forms a multinomial-Bernoulli RBM.
Let x ∈ NK be word counts in a document with words from a dictionary of sizeK. Let z ∈ {0, 1}m be a binary latent code. Let l = ∑ k xk denote the document’s length. We assume that the document length is independent of the latent topic and its distribution p(l) can be learned separately (e.g., a log-normal distribution is a good fit). The decoder is defined using the multinomial distribution model (words in the document are drawn from the same categorical distribution corresponding to its topic):
p(x, l | z) = p(l)h(x | l) exp(f(z)Tx− lA(f(z))), (12) where f(z) is a neural network mapping the latent code to the logits of word occurrence probabilities, A(η) = log ∑ k exp(ηk) and h(x | l) = [[ ∑ k xk=l]] ( l!/ ∏ k xk! ) is the base measure2. The sufficient statistics are the word counts x. The prior p(z) is assumed uniform Bernoulli.
The encoder is the conditionally independent Bernoulli model, expressed as
q(z |x, l) ∝ exp(g(x)Tz), (13) 2Prior work (Chaidaroon & Fang, 2017; Shen et al., 2018) omits the base measure as it has no trainable
parameters.
where g(x) is the encoder network. Chaidaroon & Fang (2017) experimented with the encoder and decoder design and recommended using TFIDF features instead of raw counts. First, we note that the inverse document frequency (IDF) is not relevant, since it can be learned by the first linear transform in the encoder. Effectively, the term frequency (TF), given by x/l, is used. This choice is adopted in later works (Shen et al., 2018; Zamani Dadaneh et al., 2020; Ñanculef et al., 2020). It might seem reasonable that the latent code modeling the document topic should not depend on the document length, only on the distribution of words in the document. However, we will argue that this rationale is misleading for stochastic encoders.
We apply Theorem 1 to two groups of variables: observed (x, l) and latent z with h(x, l) = h(x | l)p(l), ν(x, l) = x and ψ(z) = z. It follows that the consistent joint family is
p(x, l, z) = h(x, l) exp(xTWz + aTx+ bTz + c). (14)
This however implies that g(x) = Wx + b, i.e. the encoder network must be linear in x. Consequently, it cannot match a function of word frequencies x/l (as chosen by design) unlessW = 0, i.e. a completely trivial model with an encoder not depending on x. Such an encoder would imply full posterior collapse. The corresponding consistent set ΘΦ coincides with the set of collapsed VAEs where the decoder does not depend on the latent variable z in Fig. 1. We conjecture that the inherent inconsistency of this VAE has a detrimental effect on learning.
If instead, we let the encoder network to access word counts x directly, we obtain that g(x) = Wx + b can form a consistent VAE. Inspecting this encoder model in more detail, we see that it builds up topic confidence in proportion to the evidence (total word counts), as the true posterior would. Indeed, the true posterior p(z |x, l) satisfies the factorization by Bayes’s theorem: p(z |x, l) = p(x | z, l)p(z)/p(x | l). The prior p(z) is constant by design, p(x | l) does not vary with z and p(x | z, l) factors over all word instances according to (12). In other words, the coupling between x and z in log p(z |x, l) is linear in x. In Section 4 we study the proposed correction experimentally and show that it enables learning better models under a variety of settings.

heading: 4 EXPERIMENTS

heading: 4.1 ARTIFICIAL EXAMPLE
text: To start with, we illustrate our findings on a toy example. We consider a simple Gaussian mixture model for which we can easily generate samples and compute all necessary quantities including the ELBO objective. We define the ground truth model to be p∗(x, z) = p∗(z)p∗(x | z), with z ∈ {1 . . . 4}, p∗(z) ≡ 0.25, x ∈ R2, p∗(x | z) = N (x |µ(z), σ2I), i.e. a mixture of four 2D Gaussians. Fig. 2(a) shows the color-coded posterior distribution p∗(z |x). We assign a color to each component and represent p∗(z |x) for each pixel x ∈ R2 by the corresponding mixture of the component colors.
For better interpretability, we illustrate further results by decision maps arg maxz p(z |x). Fig. 2(b) shows the decision map for p∗(z |x). The aim of the experiment is to learn a VAE and to study the influence of the factorization assumption on the results. We use the decoder architecture as in the ground truth model — a Gaussian distribution pθ(x | z) = N (x | θzoh, σ2I), where zoh is the one-hot (categorical) representation of z, and θ is a 2×4 matrix that maps the four latent codes to 2D centers at general locations. Note that the ground truth model is contained in the chosen decoder family. Hence, the ML solution is the ground truth decoder p∗(x | z). We restrict the encoder to factor over the binary representation of the code zb ∈ {0, 1}2 and define it as qφ(zb |x) ∝ exp 〈gφ(x), zb〉, where gφ(x) is implemented as a feed-forward network with two hidden layers, each with 64 units and ReLU activations.
First, we pre-train our factorized encoder by optimizing ELBO and keeping the ground truth decoder fixed. The next step is to jointly train the encoder and decoder by maximizing ELBO. Since we are interested how the ELBO objective distorts the likelihood solution, we start with the ground truth decoder and the pre-trained encoder from the previous step. The ELBO-optimal decoder has to match not only the training data, but also the inexact, factorizing encoder. The resulting qφ(z |x) is shown in Fig. 2(c) and the learned model posterior pθ(z |x) ∝ p(z)pθ(x | z) in Fig. 2(d). Note that they match each other pretty well, but differ substantially from the ground truth posterior shown in Fig. 2(b). The impact of the factorization is clearly visible – one can see two decision boundaries (one for each bit of zb), which together partition the x-space into four regions, approximating the true posterior. For comparison, Fig. 2(e) shows the posterior of an RBM trained on the same data. It is clearly seen that the ELBO optimizer is pulled away from the likelihood optimizer towards an RBM solution. Notice also the explanation given in Fig. 2(f).
The approximation error (2) in the decoder caused by using the factorized encoder is only 0.03 nats (see Appendix B.1), but the qualitative difference in Fig. 2 appears detrimental. Summarizing, this simple toy example clearly shows the VAE approximation error caused by the combination of ELBO objective and the factorization assumption for the encoder.

heading: 4.2 GAUSSIAN VAES FOR CELEBA IMAGES
text: The goal and design of this experiment is similar to the previous one. We first define a ground truth decoder which is used to generate training images. Then we pre-train an encoder by ELBO keeping the ground truth decoder fixed. Finally, we train both model parts starting from the ground truth decoder and the pre-trained encoder.
The ground truth generative model is obtained by training a convolutional Generative Adversarial network (GAN) using code of Inkawhich (2017) on the CelebA dataset Liu et al. (2015). We scale and crop all images to 64×64 pixels. In order to get a stochastic decoder, we equip the GAN generator x = d(z), z ∈ R100, x ∈ R64×64×3 with image noise σd. The ground truth generative model is thus defined as p∗(x, z) = p∗(z)p∗(x | z), where p∗(z) = N (z | 0, I), p∗(x | z) = N (x |µd(z), σ2dI), and σ2d is a common noise variance for all pixels and color channels. We chose σd = 0.05 (the color values are normalized to [−1, 1]). This corresponds to an image noise level, which is just visible, but does not disturb visual perception essentially.
The decoder family of the considered VAE consists of networks with the same architecture as d(z). This ensures that the ground truth decoder is a likelihood maximizer of the VAE model. The encoder is defined as qφ(z |x) = N (x |µe(x),diag(σ2e(x))), where µe, σe ∈ R100 are two outputs of a convolutional neural network with an architecture similar to the architecture of the discriminator used for training the GAN (except the output layer), i.e., qφ(z |x) is a multivariate Gaussian with diagonal covariance matrix whose parameters depend on x.
We pre-train the encoder fully supervised by maximizing its conditional log-likelihood Ep∗(x,z) log qφ(z |x) on examples drawn from the ground truth generating model p∗(x, z).3 The results of pre-training are shown in Fig. 3. Then we jointly learn the encoder and decoder by maximizing ELBO on x-samples drawn from the ground truth model. We start the learning with the ground truth decoder p∗(x | z) and the encoder obtained in the previous step. Two variants are considered for this training: (i) keeping the image noise σd fixed and (ii) learning it along with other
3In contrast to the previous experiment we optimize the “forward” KL-divergence, i.e. the conditional likelihood, instead of the “reverse” KL-divergence used in ELBO for simplicity.
model parameters. We evaluate the results quantitatively by computing the Frechet Inception Distances (FID) between the ground truth model p∗(x | z) and the obtained decoders pθ(x | z) using the code of Seitzer (2020). For this we generate 200k images from each model. The obtained values are given in Tab. 1. Fig. 4 shows images generated by the ground truth model and the two learned models.
To conclude, ELBO optimization harms the decoder considerably as clearly seen both from FIDscores and the generated images. Models with higher ELBO values have worse FID-scores and produce less realistic images.

heading: 4.3 BERNOULLI VAE FOR TEXT DOCUMENTS
text: This experiment compares training of the VAE model for semantic hashing discussed in Section 3.1 with and without our proposed correction on the 20Newsgroups dataset (Lang & Rennie, 2008). We describe the dataset, preprocessing and optimization details in Appendix B.2.
We compare three encoders: e1: linear encoder on word counts (the proposed correction), e2: deep (2 hidden layers) encoder using word frequencies and e3: a linear encoder on frequencies. The encoders are compared across different numbers of latent Bernoulli variables (bits) and different decoder depths. The decoder depth denotes the number of fully connected hidden ReLU layers (0- 2). In both the encoder and decoder we use 512 units in hidden layers. The prior work mainly used linear decoders following the ablation study of Shen et al. (2018). Our experiments also suggest that using deep decoders in combination with longer bit-length leads to a significant overfitting. When the decoder is linear, the posterior distribution is tractable and is linear as well, i.e., the VAE model is equivalent to a Multinomial-Bernoulli RBM. We experimentally verify that a linear decoder indeed works better in this case. However, perhaps more surprisingly, we also find out that it works better even for non-linear decoders.
Table 2 show the achieved training and test Negative ELBO (NELBO) values. We observe across all settings that the simple encoder e1 is consistently better than the more complex encoder e2 which in turn is significantly better than the linear encoder on frequencies e3. We conclude that the use of VAEs with deep encoders based on word frequencies (Chaidaroon & Fang, 2017; Shen et al., 2018; Zamani Dadaneh et al., 2020; Ñanculef et al., 2020) is sub-optimal for this dataset. We do not
expect additional regularization techniques proposed in these works to change this conclusion. We also observe that linear decoders generalize better under 32 and 64 bits compared to more complex decoders, which suffer from overfitting. This implies that in these cases the best encoder-decoder combination is linear, i.e. the basic RBM model.

heading: 5 CONCLUSIONS
text: We have analyzed the approximation error of VAEs in a general setting, when both the decoder and encoder are exponential families. This includes commonly used VAE variants as, e.g., Gaussian VAEs and Bernoulli VAEs. We have shown that the subset of generative models consistent with the encoder class is quite restricted: we concisely characterize it as bilinear models on the sufficient statistics of both decoder and encoder, i.e., RBM-like models. This connection between tight VAEs and RBMs / EF Harmoniums was not previously known. The consistent subset can not be enlarged by using more complex encoder networks as long as encoder’s sufficient statistics remain unchanged. In combination with the ELBO objective, this causes an approximation error — the ELBO optimizer is pulled away from the data likelihood optimizer towards this subset. Moreover, we proved theoretically that close-to-tight VAEs have decoders close to GLMs.
We have shown that the error is detrimental when the consistent subset is too restrictive. On the other hand it may result in a useful regularization when it contains more expressive models. In the cases where a lot of data is available and a high quality generative model is of the primary interest, more expressive encoder families are required in addition to large networks. However, in cases when the learned representations are of interest, models with simple encoders can be used such as in the case of the Text VAE model in our example and their performance can be improved by considering consistency, i.e., adjusting the encoder and decoder so that the consistent set is large enough.
We restricted this study to exponential families. While in richer models discussed in the introduction, the approximation error still exists, it is made small by design, and it is less relevant and harder to analyze it theoretically. One possible open direction where such analysis would make sense is to consider fully factorized non-exponential cases, e.g., Student-t VAEs (Takahashi et al., 2018) or models satisfying hierarchical or partial factorization (Maalø e et al., 2019).

heading: A PROOFS
text: A.1 PROOF OF THEOREM 1
The proof directly follows from the characterization of conditionally specified joint distributions in the exponential family given by Arnold & Strauss (1991), see (Arnold et al., 2001, Theorem 3):
Theorem A.1 (Arnold & Strauss 1991). Let x ∈ X and z ∈ Z be random variables with a strictly positive joint distribution such that both conditional distributions are exponential families with densities
p(x | z) = h(x) exp [ 〈ν(x), f(z)〉 −A(z)] (15a)
p(z |x) = h′(z) exp [ 〈ψ(z), g(x)〉 −B(x)], (15b)
where ν : X → Rn and ψ : Z → Rm are minimal sufficient statistics, f : Z → Rn and g : X → Rm are any mappings, h(x) and h′(z) are base measures, and A and B denote the respective log-partition functions4.
Then there exists a matrix W ∈M(n+ 1,m+ 1), such that the density of the joint distribution can be represented as
p(x, z) = h(x)h′(z) exp 〈νe(x),Wψe(z)〉 , (16)
where νe, ψe denote the statistics vectors extended with an additional component 1.
A.2 EXAMPLE OF A DISCRETE VAE APPROACHING A FLOW
Example A.1. In this example we construct a VAE that can be arbitrary close to tight one, but where the decoder network does not approach a linear map. Let X = {−1, 1}2, Z = {−1, 1}2 and let p(z) be uniform. Let the decoder be conditionally independent
p(x|z) = exp [ β 〈x, π(z)〉 −A(βπ(z)) ] (17)
where π(z) denotes the invertible mapping of Z to X given by
x1 = z1 (18) x2 = z1z2. (19)
If the parameter β is sufficiently large, the distribution p(x | z) approaches the deterministic distribution δx=π(z). Its posterior therefore also approaches the deterministic distribution δz=π−1(x) (notice that π−1 = π). Let the encoder be the conditionally independent model q(z |x) = exp [ β〈z, π−1(x)〉 − A(βπ−1(x)) ] . This decoder by design approaches δx=π(z) as well and thus the VAE (p, q) achieves ε-tightness for sufficiently large β. At the same time the deviation between logarithms of probabilities log q(z|x) and log p(z|x) grows with β.
A.3 PROOF OF THEOREM 2
The idea of the proof is to bound the difference between log p(z|x) and log q(z|x), which is done by Proposition A.2 and then in the space of log-probabilities to approximate the non-linear mapping fe(z) by a linear one as detailed in Proposition A.1. Choosing the norms and the approximation appropriately allows to construct a bound, which despite the discreteness assumption about the observation space X in Theorem 2 does not depend on its cardinality. For a finite set X ⊂ X letH be the |X|-dimensional vector space with the inner product 〈u, v〉pd =∑ x∈X pd(x)u(x)v(x), assuming that pd(x) > 0 for all x ∈ X . The respective norm will be denoted as ‖ · ‖pd . 4The log-partition function is usually defined as a function of the (natural) parameter. In this context we consider h,h′, f , g to be fixed and consider the dependence on x, z only.
Proposition A.1. Under model Assumption 1, for any finiteX ⊆ X there exists a matrixW ∈ Rn,m and a GLM
p̃(x | z) = h̃(x) exp(〈ν(x),Wψ(z)〉 − Ã(z)). (20) such that the following inequality holds for all z ∈ Z:∑
x∈X pd(x) ∣∣ log p(x | z)− log p̃(x | z)∣∣2 ≤∑x∈X pd(x)∣∣ log q(z |x)− log p(z |x)∣∣2. (21)
That is, if the RHS is small, the decoder network f(z) can be effectively replaced by a linear model Wψ(z).
Proof. For clarity, we will omit the dependence of the decoder and encoder on their parameters θ, resp. φ. Throughout the proof we will also assume that a single z ∈ Z is fixed. First, we expand
log q(z|x) = 〈ψ(z), g(x)〉 −B(x) + log h′(z); (22a) log p(z|x) = log p(x|z) + log p(z)− log p(x)
= 〈ν(x), f(z)〉 −A(z) + log h(x) + log p(z)− log p(x), (22b) where A(z) = A(f(z)) and B(x) = B(g(x)). We can therefore represent
log q(z|x)− log p(z|x) = 〈ψ(z), g(x)〉 −B(x) + log p(x)− log h(x) (23) − ( 〈ν(x), f(z)〉+ log p(z)− log h′(z)−A(z) ) (24)
= 〈ψe(z), ge(x)〉 − 〈νe(x), fe(z)〉 , (25) where
ψe(z) = (ψ(z), 1); (26) νe(x) = (ν(x), 1); (27) ge(x) = (g(x), log p(x)−B(x)− log h(x)); (28) fe(z) = (f(z), log p(z)−A(z)− log h′(z)). (29)
With this representation we have:∑ x∈X pd(x) ∣∣ 〈νe(x), fe(z)〉 − 〈ψe(z), ge(x)〉 ∣∣2 (30) = ∑ x∈X pd(x)
∣∣ log q(z |x)− log p(z |x)∣∣2 =: ∆2. (31) Let V be the matrix with rows νe(x) for all x ∈ X . Let G be the matrix with rows ge(x) for all x ∈ X . We can rewrite the condition (30) in the form
ξ = V fe(z)−Gψe(z), (32a) ‖ξ‖2pd = ∆ 2, (32b)
where ξ ∈ R|X| is the vector of residuals. Let P be the orthogonal projection onto the range of V in the spaceH. Multiplying (32a) by P on the left, we obtain
PV fe(z)− PGψe(z) = Pξ. (33) Because PV = V we obtain
V fe(z)− PGψe(z) = Pξ. (34)
We therefore can consider the decoder network approximation f̃e(z) = W̃ψe(z), where W̃ is the solution to the consistent system of linear equations V W̃ = PG and is independent of z. We can therefore express
‖V fe(z)− V f̄e(z)‖pd = ‖Pξ‖pd ≤ ‖ξ‖pd = ∆, (35) where the inequality holds because P is an orthogonal projection inH.
We obtained that the decoder network fe(z) can be approximated by a linear mapping W̃ψe(z) such that ∑
x∈X pd(x) ∣∣〈νe(x), fe(z)〉 − 〈νe(x), W̃ψe(z)〉∣∣2 ≤ ∆2. (36)
Expressing back
〈νe(x), fe(z)〉 = 〈ν(x), f(z)〉+ log p(z)−A(z)− log h′(z) (37) = log p(x|z) + log p(z)− log h(x)− log h′(z) (38)
We obtain that ∑ x∈X pd(x) ∣∣ log p(x|z)− log p̃(x|z)∣∣2 ≤ ∆2, (39) where
log p̃(x|z) = log h̃(x) + 〈ν(x),Wψ(z)〉 − Ã(z); (40a) log h̃(x) = log h(x) + 〈ν(x), W̃1:n,n+1〉; (40b)
Ã(z) = A(z)− log p(z) + log h′(z) + 〈W̃n+1,1:n, ψ(z)〉+ W̃n+1,n+1; (40c) W = W̃1:n,1:n. (40d)
Proposition A.2. Under model Assumption 1, let X and Z be discrete (finite) sets and let z ∈ Z be chosen. If q(z |x) ≥ α and p(z|x) ≥ α for all x ∈ X , where X ⊆ X and
Epd(x)[DKL(q(z |x) ‖ p(z |x))] ≤ ε, (41)
then ∑ x∈X pd(x) ( log p(z |x)− log q(z |x) )2 ≤ εα2 + o(ε). (42) Proof. Let us denote ε(x) = DKL(q(z |x) ‖ p(z |x)). Pinsker’s inequality assures for each x
sup S⊂Z |Pq(z | x)(S)− Pp(z | x)(S)|2 ≤ ε(x)/2. (43)
Substituting S = {z} we obtain
|p(z |x)− q(z |x)|2 ≤ ε(x)/2. (44)
By taking expectation in pd(x) on both sides we obtain a variant of Pinsker’s inequality:
Epd(x)|p(z |x)− q(z |x)|2 ≤ Epd(x)ε(x)/2 = 12Epd(x)[DKL(q(z |x) ‖ p(z |x))] ≤ ε/2. (45)
Notice that the LHS depends on the given z. Because all summands are non-negative it follows that ∀X ′ ⊆ X ∑ x∈X′ pd(x)|p(z |x)− q(z |x)|2 ≤ ε/2. (46)
This inequality will be used in several places below.
Consider x ∈ X such that p(z |x) > q(z |x). Then
| log p(z |x)− log q(z |x)|2 = log2 p(z | x)q(z | x) (47a)
= log2(1 + p(z | x)−q(z | x)q(z | x) ) (47b)
≤ log2(1 + p(z | x)−q(z | x)α ), (47c)
where the inequality holds because log2 is monotonously increasing for arguments greater equal than 1, which is ensured. Let us now consider x ∈ X such that p(z |x) < q(z |x). Then
| log p(z |x)− log q(z |x)|2 = log2 q(z | x)p(z | x) (48a)
= log2(1 + q(z | x)−p(z | x)p(z | x) ) (48b)
≤ log2(1 + q(z | x)−p(z | x)α ). (48c)
In total, we obtain
| log p(z |x)− log q(z |x)|2 ≤ log2(1 + |p(z | x)−q(z | x)|α ). (49)
Let us denote u(x) = |p(z | x)−q(z | x)|α and partition the set X into two parts:
X1 = {x ∈ X | u(x) < u0} (50a) X2 = {x ∈ X | u(x) ≥ u0}, (50b)
where we chose u0 ∈ [ √ e− 1, 5] for reasons to be clarified below. For both parts, i.e. k = 1, 2, we have∑ x∈Xk pd(x)| log q(z |x)− log p(z |x)| 2 ≤ ∑ x∈Xk pd(x) log 2(1 + |p(z | x)−q(z | x)|α ). (51)
For X1, (51) can be further bounded as ≤ ∑ x∈X1 pd(x) log(1 + |p(z | x)−q(z | x)|2 α2 ) (52a)
≤ log ∑ x∈X1 pd(x) ( 1 + |p(z | x)−q(z | x)| 2 α2 ) (52b)
= log ( pd(X1) + 1 α2 ∑ x∈X1 pd(x)|p(z |x)− q(z |x)| 2 )
(52c) ≤ log (
1 + ε2α2 ) = ε2α2 + o(ε), (52d)
where the first inequality holds for u0 ≤ 5, because in this case log2(1+u) ≤ log(1+u2) holds, the second inequality is the Jensen’s inequality for log and the last inequality uses (46) under monotone log.
For X2 we have the following. Let V = pd(X2) = ∑ x∈X2 pd(x). We can express∑
x∈X2 pd(x) log 2 ( 1 + |p(z | x)−q(z | x)|α ) (53a)
=V (∑ x∈X2 pd(x |X2) log 2 ( 1 + |p(z | x)−q(z | x)|α )) (53b)
Using that u < u2 on X2 and that log2(1 + u) is monotone for a positive argument, we can bound (53b) as
≤ V (∑ x∈X2 pd(x |X2) log 2 ( 1 + |p(z | x)−q(z | x)| 2 α2 )) . (54)
Further, using that log2(1 + v) is concave on X2 for v ≥ e− 1, we have ≤ V log2 (∑ x∈X2 pd(x |X2) ( 1 + |p(z | x)−q(z | x)| 2 α2 )) (55a)
= V log2 ( 1 + ∑ x∈X2 pd(x |X2) |p(z | x)−q(z | x)|2 α2 ) (55b)
≤ V log2 (
1 + ε2α2V
) , (55c)
where in the last step we used (46).
Next we show that V itself is bounded above by ε 2u20α 2 . It follows from
u20V = ∑ x∈X2 pd(x)u 2 0 ≤ ∑ x∈X2 pd(x) |p(z | x)−q(z | x)|2 α2 ≤ ∑ x∈X pd(x) |p(z | x)−q(z | x)|2 α2 ≤ ε 2α2 , (56)
where the last inequality is again (46).
Now, letting r = ε2α2V ≥ u 2 0 ≥ 1 we can bound (55c) as follows V log2 (
1 + ε2α2V ) = ε2α2 1 r log 2 ( 1 + r )
(57a)
≤ ε2α2 supr≥1 1 r log
2 ( 1 + r ) ≤ ε2α2 0.64. (57b)
Theorem 2 follows by Proposition A.2 and Proposition A.1 with the choice X = X .

heading: B DETAILS OF EXPERIMENTAL SETUP
text: B.1 ARTIFICIAL EXAMPLE We give here the achieved likelihood and ELBO values for this experiment. The negative entropy∑ x p ∗(x) log p∗(x) of the ground truth model, i.e., the best reachable data log-likelihood, is −3.65. The ELBO of the pre-trained VAE is −3.74. During the second step of training, i.e., the joint learning of the encoder and decoder by ELBO maximization, the ELBO value increases from−3.74 to−3.70. The data log-likelihood ∑ x p ∗(x) log pθ(x) drops at the same time from−3.65 to−3.68. The approximation error (2) in the decoder caused by using the factorized encoder is only 0.03 nats, but the qualitative difference between the ground truth model and the ELBO optimizer model shown in Fig. 2 appears detrimental.
B.2 BERNOULLI VAE FOR TEXT DOCUMENTS
Dataset In this experiment we used the version of the 20Newsgroups data set (Lang & Rennie, 2008) denoted as “processed” by the authors. The dataset contains bag-of-words representations of documents and is split into a training set with 11269 documents and a test set with 7505 documents. We keep only the 10000 most frequent words in the training set, which is a common pre-processing (each of the omitted words occurs not more than in 10 documents).
Optimization To train VAE we used the state-of-the-art unbiased gradient estimator ARM (Yin & Zhou, 2019) and Adam optimizer with learning rate 0.001. We did not use the test set for parameter selection. We train for 1000 epochs using 1-sample ARM and then for 500 more epochs using 10 samples for computing each gradient estimate with ARM. We report the lowest negative ELBO (NELBO) values for the training set and test set during all epochs.
Models The decoder model (12) describes words as independent draws from a categorical distribution specified by the neural network f(z). This network respectively has a structure
Linear→ (ReLU→ Linear)︸ ︷︷ ︸ ×dhidden → Logsoftmax.
The input dimension equals to the number of latent bits, the output dimension equals the number of words in the dictionary, 10000. For decoders with dhidden = 1, 2 the hidden layers contained 512 units.
The encoder networks e2, e3 take on the input word frequencies x/ ∑ k xk, the encoder network e1 takes on input word counts x. For the deep encoder e2 we used 2 hidden ReLU layers with 512 units each.

Review summaries:
Summary. This paper presents an analysis of the approximation error of VAE models when the encoder and decoder are from exponential families. They show that when the model is consistent (i.e., the encoder is able to match the posterior), the encoder and decoder distributions are exponential family distributions that are linearly parameterized. The paper also shows that barring pathological cases, when the model is tight the decoder distribution is close to a linearly parameterized exponential family distribution. In particular, this shows that in this case additional depth is not useful.  Despite the restriction to exponential family distributions, the paper shows a number of real instances of models that lie in this class including Gaussian VAEs and VAEs with RBM encoders. For Bernoulli VAEs for semantic hashing, the paper proposes and verifies improvements to a model presented in the literature. 

The paper studies the VAE approximation error, when the encoder and decoder distributions are from conditional exponential families (EFs). Theorem 1 characterizes the form of the joint probability (an EF-Harmonium) which is consistent, i.e. where q_{phi}(z|x) \equiv p_{theta}(z|x). Sec 4.2 shows a v interesting example of Gaussian VAEs for CelebA images, where it is shown that a VAE "unlearns" the ground truth solution (being pulled away from the likelihood optimizer towards the consistent subset). Sec 4.3 shows another interesting example, where a careful formulation of the model identifies that it should use word counts rather than frequencies, and this is demonstrated empirically (Table 2).