A Sampling-Free Approximation of Gaussian Variational Auto-Encoders
Authors: Felix Petersen
Christian Borgelt
Hilde Kuehne
Oliver Deussen
Venue: ICLR 2022 Submitted (2022)

Abstract: We propose a sampling-free approximate formulation of Gaussian variational auto-encoders. Instead of computing the loss via stochastic sampling, we propagate the Gaussian distributions from the latent space into the output space. As computing the exact likelihood probability is intractable, we propose to locally approximate the decoder network by its Taylor series. We demonstrate that this approximation allows us to approximate the Gaussian variational auto-encoder training objective in closed form. We evaluate the proposed method on the CelebA, the 3D Chairs, and the MNIST data sets. We find that our sampling-free approximation performs better than its sampling counterpart on the Frechet inception distance and on par on the estimated marginal likelihood.

Under review as a conference paper at ICLR 2022
A SAMPLING-FREE APPROXIMATION OF GAUSSIAN VARIATIONAL AUTO ENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose sampling-fres free approximate formulation of Gaussian variational auto- encoders. Instead of computing the loss via stochastic sampling, we propagate the Gaussian distributions from the latent space into the output space. As computing the exact probability is intractable, we propose to locally approximate the decoder network by its Taylor series. We demonstrate that this approximation allows us to approximate the Gaussian variational auto-encoder training objective in closed form. We evaluate the proposed method on the MNIST, the Omniglot, the CelebA, and the 3D Chairs data sets. We find that our sampling-fue free approximation performs better than its sampling counterpart the Frechet inception distance and par on the estimated marginal likelihood.
INTRODUCTION
Variational Eito-Encoders (VAE) allow generating data, mapping data into a latent space, and modifying data by perturbing it in a controlled manner in the latent space [1. 2]. This enables them to generate and modify images [3, 4]. speech [5], and molecular structures [6], or detect outliers in distributions [7]. VAES extend the concept of regular auto-encoders by modeling the latent space as distribution and with distributional constraint that allows for a distributional latent space. Thus, they enforce all samples drawn from the prior be representative for the data set. For this, VAEs predict a distribution through the encoder, sample from this distribution, and propagate the sample through the decoder. During training, they constrain the predicted distribution with respect to the prior, and require that the reconstructions produced by the decoder are correct. To allow for random sampling in the context of gradient descent, Gaussian VAEs employ the sampling reparameterizatioo trick [8], which enables backpropagation through the sampling operation. In fact, the sampling reparameterization trick and VAEs have received a lot of attention in research [3, 9-11].
Recently, research has also started to look into techniques to compute the posterior and marginal distribution in an exact analytical solution without the need for sampling. In a sampling-free formula- tion, during training, the loss is not computed via stochastic sampling but derived by propagating the distributions from the latent space to the output space. However, computing the exact likelihood is in many cases intractable. Two approaches have recently been proposed address this challenging problem. Balestriero et al. [12] propose to compute the likelihood for small neural networks by inte- gration over piecewise continuous polytopes. Lucas al. [13] formulate an analytical sampling-free linear VAE, which can be efficiently computed in closed form.
In this context, the following work proposes sampling-free approximate formulation for VABs that allows sampling-fie freee training of VAES. To this end, we approach the problem of intractability by approximating the probability distributions instead of computing them exactly. This approximation is done by local Taylor series expansion of locally affine (ReL (Rattivcciv))ee)) neural networks. This corresponds to the local linearization of the respective neural network and allows for local lineariza- tion of the decoder of the Gaussian VAE. We propose to use the local linearization to estimate the (co-)variances for each data point, while the median of the output distribution is computed exactly. Locally approximating neural networks by their Taylor series around each individual data point enables us to express the covariance matrix of the probability distribution in terms of the Jacobian of the neural network. Furthermore, computing the covariance matrix explicitly is very expensive, the proposed formulation allows computing the likelihood without explicitly computing the covariance matrix nor its inverse. This allows us compute the training objective without sampling a reason-

---

Under review as a conference paper at ICLR 2022
able additional cost for standard architectures, specifically, at the cost of evaluating the Jacobian of the decoder. Note that the local Taylor series expansion or local linearization of the neural network decoder function does neither correspond to a global linearization of the neural network decoder nor a global approximation via its Maclaurin series.
We evaluate the proposed method on the CelebA [14], the 3D Chairs [15), and the MNIST [16] data sets. When considering the number of samples necessary to estimate a covariance matrix of equívalent quality wrt. the Frobenius norm via stochastic sampling, our approximation is (on average) equivalent to more than 235 samples for a deep network on the CelebA data set. Overall, the proposed approach achieves competitive results compared to sampling Gaussian VAEs. Especially on the Frechet Inception Distance (FID) [17]. the sampling- Prreei VAE performs better than the sampling VAF, while the estimated marginal likelihoods between both methods are on par. This demonstrates that local linearization is good approximation and does not damage performance in comparison to a regular sampling VAE. Following the setup by Lucas et al. [13], the sampling-fres VAE leads to smaller (e, - collapse than its sampling counterpart.
We summarize the contributions of this work as follows:
We present a novel sampling-free approximation for training Gaussian VAEs. In contrast to exact computation, requires only a small overhead compared to its sampling counterpart, and reaches a less noisy approximation of the training objective than sampling.
We show that the sampling-free Gaussian VAE approximation achieves competitive performance on the MNIST, Omniglot, CelebA, and 3D Chairs data sets compared sampling VAE methods. Also, the sampling-fre -ree formulation also achieves a better FID on CelebA when provided with the same computational budget as its sampling counterpart.
We find that sampling-free VAEs more robust against posterior collapse than sampling VAEs.
The full implementation as well as experiments will be publicly available upon publication.
2 RELATED WORK
Balestriero al. [12] present an approach compute the exact likelihood for small yet deep neural networks using the continuous piecewise affinity of respective neural networks to compute the exact likelihood by partitioning the network and integrating over the polytopes. Based on this, an analytical ponectttiionMxiiizzzaee
algorithm is derived that enables gradient-free learning. The method is empirically validated by training a three layer deep generative network with 8 and 16 hidden units and with one latent dimension on the number 4 from the MNIST data set, where they demonstrate that an exact sampling-free VAE is superior to the stochastic sampling VAE. However, as the exact analytical formulation of deep neural networks has a computational complexity that is exponential in the size of the neural network, is intractable for larger networks. Lucas et al. [13] examine sampling- compling-free free linear VAEs, which by the nature of their linearity allow for efficient exact computation of the likelihood and are able to show that the linear VAE recovers probabilistic principal component analysis (pPCA) [18]. They employ the sampling-fie linear VAE for understanding posterior collapse in VAEs and apply it to the MNIST data set.
Ghosh et al. [19] propose regularized deterministic auto-encoders, learn smooth latent space without sampling via different regularization schemes. They regularize their auto-encoders through weight decay, the gradient norm of the decoder and/or spectral normalization, which all bound the Lipschitz continuity of the decoder. Our work (implicitly) also includes a kind of gradient regularization of the decoder. However, in our work. this arises from approximating the marginal log-likelihood via local Taylor approximations of the decoder.
Tolstikhin et al. [20] propose Wasserstein auco-Encoders (WAEs), which minimize the Wasserstein distance between the model distribution and the target distribution. Kolouri et al. [21] propose Sliced Wasserstein Auto-Encoders (SWAEs), which regularize the auto-encoders using the sliced / marginal Wasserstein distance between the distribution of the encoded training samples and a predefined sampleable distribution. Note that both WAES and SWAEs require sampling [20-22]. Knop er al. [22] propose Cramer-Woo Auto-Encoders (CWAEs), sampling-fre distributionally regularized Auto-Encodee Encoder. They introduce the Cramer- Wold distance, which has a simple analytical formula for computing normality of high-dimensional samples. By regularizing their auto-encoders via the
Crader-Wold
2

---

Under review as a conference paper at ICLR 2022
Cramer- Wold distance, they can evaluate their training objective in closed form. Note that CWAEs are not VAEs as, for VAEs, the distance to the true distribution is measured using KL divergence under the latent variable model [22]. Instead, the CWAE is different kind of generative model that was designed to have a tractable sampling-free objective.
Burda al. [23] introduce importance weighted sampling for VAES by using multiple samples to have a tighter variational lower bound. Huang et al. [24] propose hierarchical importance weighted auto-encoders reducing the number of redundant samples and introducing hierarchical structure induce correlation. Dieng el al. [25] use importance weighted VAEs using an EM-based algorithm employing moment matching. Roeder al. [26] propose a low- variinca gradient estimator for VAEs. Park et al. [27] enrich the posterior by applying the Laplace approximation to VAEs, which enables modeling a more expressive full-covariance Gaussian posterior. Tomezak et al. [28] also produce an improved posterior distribution through a Householder transformation-
volume-preserving flow. Nielsen al. [29] bridge the gap between normalizing flows [30] and VAEs by preseting SurVAE Flows, which use surjective transformation which are deterministic in one direction and stochastic/ / sampling in the reverse direction. Morrow et al. [31] present VAEs with normalizing flow decoders.
Taylor series approximations [32] are ubiquitous tool in variational inference (VI), e.g., in the delta or the Laplace method [33, 34J. VI methods, such as the delta method or the Laplace method require solving optimization problem obtain an approximation of output distribution. In contrast, our method uses the Taylor approximation to produce a closed-form estimate of the training objective and the only optimization is the training of the VAE.
phenomenon frequently appearing in VAES is posterior collapse, i.e., the posterior distribution (produced by an encoder) is close the prior distribution. This leads reduced expressivity of the VAE's latent space. To tackle this, Kingma el al. [8] proposed to constrain the KL divergence such that it is only active if the KL divergence is above threshold. Other works used KL annealing [35, 36] or constrained the posterior to have minimum distance to the prior [37]. Lucas et al. [13] defined the (e, 8) -posterii collapse order reliably quantify posterior collapse VAEs.
3 METHOD
We begin this section by introducing the conventional data generating process of VAEs. After that, we briefly motivate and derive the evidence lower bound (ELBO) and present our sampling-fre local approximation of the ELBO including efficient way of computing it. Following this, discuss method for stabilizing training and investigate the balance between reconstruction quality and variance of reconstructions.
3.1 GAUSSIAN VARIATIONAL ACTO-ENCODERS
We begin by stating the assumptions about the data generating process. Note that this process does not differ from the conventional data generating process of VAEs [1]. For easier reference, we follow the notation of Kingma and Welling [9]. The goal of VAEs to approximate the true distribution of the data p² (x) with a distribution parameterized via model (x ~ p" (x)). Here,
p(z) = N2z;0,
(1)
(x(x|z) Nx; fo(z), ed)
for some e
(x(x)= (x)) dz.
denotes m-ddimnniional vector referring to the hidden latent space, x denotes n-dimensional vector referring to the input / output space, fo is the decoder generator network, and captures the observation uncertainty. A denotes multivariate Gaussian distribution and I, the x identity matrix. To approximate the intractable pa (z]x), VAEs include an encoder network Iø and network that computes the encoder's uncertainty he (in the form of diagonal covariance matrix) such that:
(z(x)) z px(x
where (z(z)) go hs (x)).
3

---

Under review as a conference paper at ICLR 2022
The training objective is to maximize the lig-liielhhoo of our data log pe (x) by optimizing @ and ø:
log po(x12)p(z))
g0
log pa (x) = E (ellx)
+ LE (oll))
log
(cf. [9])
(5)
20(2|x)
po(z)x)
log [0(2)xx
log g (x)
= (xfx)) (log Po (x/z)) Bg (m/xx)
+ (xl())
(6)
p(z)
pa (z)x)
= (xlx)) Pn Po (x/z)) DKL (2/xx, p(z)) + (KI.(99 (z/x), Po (z|xx))
))+
(7)
ELRO
not computable but non-negative
The last term is not computable, because (z(xx) x) is intractable, and hence the log-liielihood log (x) cannot be computed exactly. However, since a KL--divergence is non-negative, the first two terms yield lower bound for log (x), the so-called evidence lower bound (ELBO) [1, 9, 38). The ELBO is the substitute objective that allows training VAEs.
ELBO log po (x) DRL. DRt. (gx(2)x))) (o()))) S px))
(8)
The KL-Divergence (go (z)xx) p(z)) can be computed in closed form. However, the computation of (x))) (log px(x)))) is generally intractable and is therefore usually approximated via stochastic sampling from the latent distribution 4s (z) xx). In the stochastic sampling approach, data point is propagated through IS and has an element z is sampled from from.N (Nzz; 9o (x), (x)). ). and propagated through fo. which approximates the output distribution (including the observation uncertainty)
((x; fo(2). cf) where z ~ N (z; go (x), (x)). Since the observation uncertainty is an isotropic normal distribution, the sample log-likelihood log po (x|z) reduces to
lo
2c (ll x 1²)
n log((2r4) (2Tcc
(9)
2
which is, apart from the normalization constant, a mean squared error [1].
3.2 SAMPLING-FREE APPROXIMATION
At this point, we introduce the proposed sampling-free approach. In the sampling-free approach, data point x is propagated through J& and he. and then the obtained distribution N (zz go has (x)) (without sampling) approximately propagated through fe. which yields the output distribution N (x; fo(z), cf) where z ~ '' (z; go (x), (x). The problem is that computing the exact transfor- mation of the distribution A (z: go (x). h (x)) by fo is intractable (for non-trivial fo) because it can have an exponential complexity. The largest analytical sampling-fre free approach so far was able to operate on 24 hidden neurons [12]. This because, for the exact computation, every linear region of So has to be computed separately. To simplify notation, in the following, we drop subscripts A and for f,g,
To make this problem tractable, we approximate the neural network decoder via local Taylor series expansion, which simplifies computing the output distribution. For that, we define the Taylor series of the decoder f around g(x) for each data point x as fg(x). We display the Taylor series of an example neural network function in Figure A. in the supplementary material. Notably, the function and its Taylor series are equal near this point g(x), which is also the point of largest density of the distribution N (z; g(x), h(x)). Due to the continuous wisce-wise affinity of f, we only need to compute the first two terms of the Taylor expansion, as the remaining terms are zero. Thus,
(olo)((() + ((((x)) + dff((()) dg(x)
f(((x)) + r(0()) e
(10)
where J (x)) is the Jacobian of ff at point (x),, in the following abbreviated as J. This approxima- tion is exact if is small enough, such that it stays inside the linear region of f in which g(x) is located. In Section 1. we empirically demonstrate that the Taylor series delivers a good approximation for the neural network decoders in VAEs. For neural networks that are not piece-wise affine, like the
tanh networks used in the experiments in Section 4.3, we also use Equation 10, which in this case the first-order Taylor expansion. Using the Taylor series for each point fg(x) we can approximate the output distribution as follows:
(2) R f(z)
where
(1I)
4

---

Under review as a conference paper at ICLR 2022
As the family of Gaussian distributions is closed under affine transformations and summation [39]:
fatx) (z) ~ N f(g())), J(x)
where z ~ ~N(z (xx), h(x))
(12)
and including the observation uncertainty
ftt (x) o - Jh(x) JJ + +c)
where a o~N(o;0,,cl
(13)
Ex
This leads sampling
Prreeing-free free approximation of the log likelihood
(as(sli)) log po [x|z)= log (2x)* $/2 det [x) exp (-2(f(((x)) - x) (f(g((( - -x))
n2 log (2T) - 1 log det [xx) {(((((x)) - -x) x)_ Xx' - -x) (14)
2
At this point, it is necessary to resolve Σ and det det(
Lemma (Woodbury matrix identity). [40]. The inverse of a rank-m correction of a matrix can be computed in terms of the inverse of the original matrix as
(A (A+ = A- A (' VA VA
(15)
Since Σx c + h JT, and h(x) is a diagonal matrix, we can set UUT UV =J JT. As the computation of the covariance via the Jacobian directly produces a low-rank decomposition of the covariance matrix and because m & n, this allows for fast and space-efficient computation. Thus,
/h(xx such that
Σx'
(cI + Jh(x)JT)
=cc =(cI+UUT)-
11 - - I1 + + UUU =
= 1 +
(16)
= - (I- + +UOTO)-UT
(I- U
which only requires inverting an m x m matrix (innermost parentheses).
Lemma 2 (Matrix determinant lemma). [41]. The determinant of rank-m correction of matrix can be computed in terms of the inverse and determinant of the original matrix as
det(A A +U) det(Im + +VA- det(A)
(17)
With U defined as above, we can see that
(ex) det (el + JJ(())J
det (cl + UUT)
(18)
= det (Im + +UT! +UT!IU) TU) det(eT) = det (Im U)
As we can now compute Σx1 and det(Ex) very efficiently, we can extend the log- ooglliieldhood
(x)( log po (x(z))
- タ log (2i) log det(Ex) (f(((()) - x) (f((((( -x)
= 2タ- log (2iT- log det dI + J U) _/2 log(c)
(f(((x)) x) (I - UU + +U -UU)-- (f(g(x)) -x) (19)
(x) x) + + ((f(a(()))x U) U) 1.(UT (f(x(x))
Here, as the inverse of the covariance matrix (Σx' of size n) does not need to be computed explicitly and the factorization of the covariance can be multiplied with the error, only operations on small matrices need be performed. Specifically, it requires one multiplication of vector of size with matrix of size m x n, the inversion of an m X m matrix, and the multiplication of vectors from both sides with an m m matrix. This significantly reduces the computation cost.
Based on these insights, we can write the ELBO as
ELBO = Eg (x|x) (x|()) DKL (2))
(20)
うい log (2mc)- log de - U) (f((())) -x) (f(((x)))) -x)
2e ((f(g(x)) - -x) -x) TU) (cIm +U +U U) (U (f(g(x) -x)) Das(gs(z) (z)x())),p()) p(z))
Since the training objective is maximize the ELBO. the corresponding loss is L F.L.O.
5

---

Under review as a conference paper at ICLR 2022
3.3 STABILIZING TRAINING AND REDUCING BIAS BY REGULARIZATION
In practice, for covariance matrices with large variances, the ratio between the largest and smallest eigenvalue can become extremely large even with Tikhonov regularization (e.g. 10%). Thus, training is not feasible because the gradients in some or most directions are vanishingly small. To account for this, we propose a regularization that works independently of the ratio between eigenvalues. For this, we use mixture between the approximated log-likelihood as described above and the log- likelihood of a conventional auto anto-encoder (i.e., only considering observation-nont noise). Specifically, we replace Ex1 by 7: Σx + (1 (1-7)) (cf)- This regularization allows for an efficient and stable training, even if training with Σx² alone would be vanishingly slow or diverges. This means, we calculate
E Σxy ~ -.--7: " UY-
instead of Σx = I UU U U' (21)
where 7 quantifies the degree to which the vari- ation produced by the decoder is considered. In our experiments, we found that y ~ 0.99 avoids vanishingly slow training and divergence due to numerical instabilities. Empirically, we found that reducing y can reduce the bias of our sampling-free approximation. In Figure 1, we train a model using the proposed sampling-free objective with and evaluate it using the unbi- ased sampling ELBO. We find that with 000955, the model converges an ELBO of -8000. By reducing 7, i.e., increasing the regularization, we can increase the ELBO that the model con- verges to, which means that we improve the model. Setting 7-0.67, the ELBO already con- verges to - 7500, and after setting 0=0.5 it con- verges to 7400. We find that ? 0.5 works well overall and, thus, we use it as a canonical choice in our experiments.
-7500
7750
-5750
4-10.5 355
1-413
1945
-9000
0-10f
3- 106
2. 100
3- 10
100
5-10"
6-16
Namber or tteatioss
Figure 1: By scheduling 7 we can observe that increasing the regularization (i.e., reducing ?) re- duces the approximation bias and increases the point of convergence, which improves the model. The model is trained with the biased sampling-free VAE objective on the CelebA data set and evalu- ated using the unbiased sampling ELBO estimate.
4 EXPERIMENTS
4.1 QUALITY OF THE TAYLOR SERIES APPROXIMATION
To evaluate the quality of the Taylor series approximation in VAEs, we consider the precision to which variances and covari- ances are estimated and compare this to different numbers of sam- ples. For this, we use sampling VAES trained on the CelebA and 3D Chairs data sets as described in more detail in the next section and with a latent dimensionality of 16. We consider two metries:
Table 1: Quality of the Taylor series approximation. For both data sets, we trained a (sampling) VAE and evaluate the Pearson correlation between the true and estimated variances. For the CelebA data set, we also evaluate the Frobenius norm between the true and estimated covariance matrices. We evaluate our approximation and compare it to different numbers of samples.
Method
Taylor
Sampling
#Points
1
5
10
25
70
100
235
CelebA
Var. Corr.
.967
.680
.816
.912
.955
.966
.976
.988
Cov. Frob.
297
17 446
7779
2951
1 1455
1 033
737
302
The Pearson correlation coeffi- eient of the (pixel-wise) output
3D Chairs
Var. Corr.
.975
.686
.819
.914
.933
.966
.976
.988
variances and the Frobenius norms of output covariance matrices. Here, we compute the true output (co- vo-rincassss via a large number of samples (15 000) and compare it to the (co- no-)avarssnes produced by our approximation or the (co-)variances produced by certain number of samples. Results are shown in Table 1. We find that, even in large deep neural networks, we achieve Pearson correlations of 96.77 - 97.5 .55% for estimating the variances, which would require 70- 100 samples in the latent space when relying on sampling. On the Frobenius norm of covariance matrices, our approximation is (on average) equivalent to more than 235 stochastic samples for the CelebA data set. Because computing the true covariance matrix is intractable, we estimate it to a precision of 1% of the reported Frobenius norms. For the 3D Chairs data set, we omit estimating the true covariance matrices due to

---

Under review as a conference paper at ICLR 2022
computational feasibility.. In addition, we analyzed the ratio between the variances produced by sampling-free propagation and the sampling approach and found that across data sets this ratio is around 1.2. This means that the variances are over-estimated by factor of 1.2 by our approximation. We attribute this to the fact that non-linearities such as ReI.U can become constant which is not anticipated by the Taylor approximation.
4.2 MNIST, CALEBA, AND 3D CHAIRS
3 3 3
2 2 2
7
We evaluate the sampling- free VAE the CelebA data set [14], on the 3D Chairs data set [15]. as well on the MNIST data set [16]. For the CelebA and 3D Chairs experiments, we use the same network architecture as Higgins et al. [3] and train it with the Adam optimizer [42] at a learning
1
7
111111
1/11
///2222
222222
2
22 22222.2
2 2 2.
rate of 10-4 for 10 iterations. We use 12 64 latent dimensions for CelebA and 8 - 32 latent dimensions for 3D Chairs. As in previous works on VA we set 0.5 which corresponds to a standard deviation of the observation uncertainty of o v0.5 for all methods. We use a model for the encoder and decoder with 5 convolutional and fully connected layers each. This is also the architecture that was used in many recent works on sampling VAEs [3, 11] as well as similar to most public implementations. Further details can be found in the supplementary material.
Figure 2: Learned MNIST manifolds. Left: sampling-fres VAE. Right: sampling VAE. The latent space is 2 dimensional and the displayed range is [-5, 5j. Training details are in the sup- plementary material.
h
R
a
h
Qualitative Evaluation In Figure 2, we show the manifold of a sampling-free VAE in compar- ison to a sampling VAF. Figure 3 displays qual- itative results for CelebA and 3D Chairs. Here, we display reconstruction results as well as traver- sals in the latent space for images and 2 latent dimensions. In the supplementary material, we qualitatively compare the sampling-free VAE to the sampling VAE, the (sampling) 3 VAV as well as conventional auto-encoder.
Figure 3 shows that the sampling-free VAE can successfully reconstruct and that latent traversals produce meaningful images. For the CelebA data set, we show the traversal of the latent dimensions corresponding to, 'smiling', 'color temperature', 'brightness of background', and 'hair color'. For the 3D Chairs data set, the latent dimensions corre- spond to 'chair<-arm chair' and 'color', 'wheels', and "thickness of legs and back'. For the auto- encoder on the 3D Chairs data set, even among a training always con- verges to the plain white image as can be seen in the supplementary material.
Figure Reconstructions and traversals for the CelebA (left) and 3D Chairs (right) data sets. The traversals are in the range of [-3, 3]. An exhaustive display of all latent dimensions as well as a comparison to sampling VAEs and AEs is provided in the supplementary material.
variety of
Quantitative Evaluation To quantitatively evaluate the sampling- free approximate VAE, we com- pare the Frechet Inception Distance (FID) as well as the estimated marginal likelihood (ML) between conventional sampling VAE and sampling-free VAE as shown in Figure 4. The FID quantifies how close the images produced by the VAE's data generating process are to the original data set as well as the quality of the produced images. For the FID, the distribution of images sampled from the VAE is compared to the distribution of images in the data set by utilizing embeddings produced by an Inception neural network. The MI. quantifies the quality of VAE by quantifying how close
7

---

Under review as a conference paper at ICLR 2022
110
Sampling VAE Sampling VAE
105
-7220
100
0 95
-7230
90
-7240
-1250
Sampling-Free
-7260
Sampling VAEE
10 20
30
400
50
50
10
20
50
40
50
60
latent dimensions.
lattent dimensions
Figure 4: Quantitative evaluation of FID (left) and log marginal likelihood (right) on the CelebA data for different sizes of the latent space. On FID, the sampling- free Gaussian VAE significantly outperforms the sampling Gaussian VAE and performs on par regarding the log marginal likelihood. Results average over runs and minimum maximum are marked.
posteriors are to the priors as well as considering the reconstruction error under sampling. For the ML, for each image, the marginal likelihood is estimated via many samples (50 (50 000). We found that for CelebA, for each size of the latent space, the FID of the sampling-free VAE is significantly better than the FID of the sampling VAE. Furthermore, we found that the ML of the sampling- srppiieegeeeeee VAEs very close to the ML of the sampling VAEs, for some latent dimensionalities, the sampling-fres free VAE is better, while for others the sampling VAE is better. Plots for the 3D Chairs data set can be found in the supplementary material. We emphasize that evaluated the sampling-free VAE also via sampling such that there is no bias in the estimates.
To evaluate the models based compu- tational budget, we additionally report re- sults for the sampling-free VAE match the computational budget of the sampling VAE's full training. Specifically, we evalu- ate the sampling-free variant after 650 000 training steps, while the sampling counter- part trains for 1 000 000 steps. The results are shown Table Even in this setting, the sampling-fie frree formulation achieves better on FID than the sampling VAE on the CelebA data set, however, the esti- mated log marginal likelihood suffers.
Table 2: Quantitative evaluation for the sampling- VAE, where include the performance of the sampling- free VAE with shorter training such that the total training budget matches the total full training time of the sam- pling VAE. Results are reported for a 16 dimensional latent space and averaged over runs.
Method
CelehA
Chairs
FID
log ML
FID
log ML
Sampling-Fieee Free Gaussian VAR
95.79
7244.9
51.96
-7078.8
Sampling Gaussian
72245.
-7078.9
Sampling- Free (matching budget)
98.00
-7247.1
54.21
-7079.6
4.3 COMPARISON TO OTHER GRADIENT ESTIMATE VARIANCE REDUCTION METHODS
We compare our method to the Sticking Landingg (Stick) method by Roeder et al. [26] and to the importance weighted auto-encodes encoders (IWAE) by Burda et al. [23]. For this, we train VAE on the MNIST [16] and Omniglot [43] data sets and for the sampling methods (vanilla sampling VAE, Stick, IWAE) vary the number of samples. The encoder and decoder of the architecture each have two hidden layers with tanh activation functions and 200 neurons per layer. We train the model at a batch size of 20 using the Adam optimizer [42]. This follows the setup by Roeder al. [26]. However, difference is that we model the image space with Gaussians in contrast to the Bernoulli outputs used ([26)) readily supports Gaussian outputs, we use it with Gaussian outputs. As tanh is not piece-wise linear, we use the first-order Taylor approximation in this case.
by Roeder al. [26]; as the method
Table 3: Evaluation MNIST (left) and Omniglot (right) with different methods and numbers of samples. The metric the NLL on the test set (smaller is better) averaged over runs.
Samples
Ours
vanilla VAE
Stick
IWAE
Samples
Ours
vanilla VAE
Stick
IWAB
26.18
27.61
26.79
27.62
30.14
31.49
30.99
31.49
26.98
26.50
26.44
31.15
30.75
30.91
20
26.62
26.48
26.11
20
30.88
30.73
30.51
50
26.64
26.59
26.08
50
30.83
30.67
30.36
100
26.58
26.79
25.99
100
30.76
30.73
30.34
200
26.70
26.70
25.95
200
30.72
30.70
30.30
8

---

Under review as a conference paper at ICLR 2022
Table 4: Posterior collapse experiment: VAEs trained on MNIST with 200-dimensional latent space. Full batch training on subset of 1 000 MNIST images for a controlled setting where for the sampling-fiee free VAE no stochastic effects are present, and for the sampling approach only the stochasticity of the reparameterization is present. Results are averaged over 10 runs.
Method
Metric
(e, 6) posterioo collapse
Reconstruction Error
KL- Diveereegeeee
Epochs
100
100
100
1 1000
100
1000
Sampling VAE 99.0% (+1.5%)
60.0%
(+15..%) 7%)
(73.5
46,2 (=12.4)
0.1 (#0.1)
(.8(+1.3)
Sampling- FFree VAE
28.4% (+4.2%)
0.6% (+0.6%)
38.7 (+0.5)
9.8 (=:.4)
2.9 (+0.1)
(#2.1))1)
We report the results in Table 3. We can see that on both data sets, the sampling-fres method achieves competitive performance. The sampling VAE and the the same NLL even with 200 samples. Only the IWAE can outperform the sampling- -ree method on MNIST with at least 20 samples. On Omniglot, our method also performs better than IWAE with 200 samples. Overall, the results show that the sampling-free free approximation performs very well, even when compared to training with large numbers of samples.
method do not achieve
4.4 POSTERIOR COLLAPSE
We analyze the effect of a sampling- approximate formulation of deep VAEs on the problem of posterior collapse. For this, we use the notion of an (e, -collapes of a dimension introduced by Lucas et al. [13]. It is defined as follows: A latent dimension i has coolaasel if
Ex-p (DEL (2), < 2
(22)
We follow Lucas et al. [13] in setting 5 0.01. We replicate their setting of training a VAE with 2 hidden layers and 200 latent dimensions for 000 steps with full batch training subset of 1 000 MNIST images. We report the fraction of dimensions that have (e, 6) colllaped where 0.05, 5 = 0.01, the reconstruction error, well as the KL divergence after 100 and 000 training steps. We report these results averaged over 10 runs in Table 4.
We find that, for the sampling-free VAE, the reconstruction error is smaller than the reconstruction error for the sampling VAE. While the sampling VAE has a posterior collapse in 60% of the latent dimensions, the sampling-free VAE suffers from posterior collapse in only 0.6% of the latent dimensions. Simultancously. the KL-Divergence is lower for the sampling-free free VAE. These results indicate that the sampling-fire free VAE more robust against (e, than the sampling VAE.
4.5 RUNTIME ANALYSIS
Finally, we discuss the runtime of the proposed approach and compare it to its sampling counterpart. For this analysis, we compare the training times of both methods on the CelebA data set for 106 steps at a batch size of 64 and 16 latent dimensions on single Quadro RTX 6000 GPU. For the sampling VAE we measure an average training time of 9.7 hours, while the proposed sampling-fre free formulation takes on average 14.5 hours on the same GPU. Note that, in case of the sampling-fre free VAE, the computationally most expensive component is computing the Jacobian. Thus, if the hidden dimension would be very large, the computational overhead would scale linearly (O(m) with the dimensionality of the hidden space m. Note that the overhead of our method is small in comparison to the (in the number of neurons) exponential overhead of computing the exact sampling-free ELBO.
5 DISCUSSION & CONCLUSION
We proposed a novel approximate sumpling-free approximation of Gaussian VAEs. Our formulation is reasonably fast to compute, i.e., can be computed linear time in the number of latent dimensions, contrasting prior art which had an exponential complexity for computing the exact objective. We found that the proposed formulation achieves a significantly lower variance estimating the objectives than the sampling approach. However, while the sampling approach is unbiased, our lower-variance approximation is not unbiased. Nevertheless, our approximation learns better Gaussian VAEs than its sampling counterpart and can even compete when the computational budget is fixed. Furthermore, observe that the sampling-free free VAE is more robust against posterior collapse than the sampling VAE. We hope have paved the way for future research analyze and understand the behavior of VAEs.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
In this work, we only use public data sets. We implemented all experiments in PyTorch [44] and our implementation will be made publicly available upon publication. Network architectures and training hyperparameters are specified in the supplementary material.
REFERENCES
[ D. P. Kingma and M. Welling, "Anuticncoding variational bayes," in International Conference on Learning Representations (ICLR), 2014.
[2] D. Rezende and F. Viola, "Taming vaes," arXiv preprint v:Xiv:1810000599 2018
[3] I. Higgins, I.. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, "Beta-vae: Learning hasic visual concepts with constrained variational framework," in International
Conference on Learning Representations (ICLR), 2017.
[4] T. Inoue, Choudhury, G. De Magistris, and Dasgupta. "Transfer learning from synthetic to real images using variational autoencoders for precise position detection," " 2018 25th IEFE International Conference on Image Processing (ICIP), IEFE, 2018, pp. 2725-2729.
[5] M. Blaauw and J. Bonada, "Modeling and transforming speech using variational autocncoders," Inter- speech 2016; 2016 Sep 8- 12; San Francisco, CA., 2016.
[6] Q. Liu, M. Allamanis, M. Brockschmidt, and A. L. Gaunt, "Constrained graph variational autoencoders
for molecule design." in Proc. Neural Information Processing Systems (NIPS), 2018.
[7] S. Eduardo, A. Nazábal, C. K. Williams, and C. Sutton, "Robust variational autocncoders for outlier detection and repair of mixed-type data," in International Conference on Artificial Intelligence and Statistics, PMLR, 2020. pp. 4056-4066.
[8] D. P. Kingma, T. Salimans, R. Jozefowicz X. Chen, I. Sutskever, and M. Welling, "Improving variational inference with inverse autoregressive flow," in Proc. Neural Information Processing Systems (NIPS), 2016.
[9] Kingma and M. Welling, "An introduction to variational autoencoders," arXiv preprint 1rXiv:1906.0 02691, 2019.
[10] B. and Wipf. "Diagnosing and enhancing vae models," International Conference on Learning Representations (ICLR), 2019.
[11] C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner, "Understanding disentangling in beta-vae," in Proc. Neural Information Processing Systems (NIPS), 2017.
[12] R. Balestriero, Paris, and R. G. Baraniuk, "Analytical Probability Distributions and EM-Learning for Deep Generative Networks," in Proc. Neural Information Processing Systems (NIPS), 2020.
[13] J. Lucas, G. Tucker, R. B. Grosse, and M. Norouzi, "Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse," in Proc. Neural Information Processing Systems (NIPS), 2019, pp. 9408 9918.
[14] Z. Liu, P. Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in Proc. International Conference Computer Vision (ICCV), 2015.
[15] M. Aubry, D. Maturana, A. A. Efros, B. C. Russell, and J. Sivic, "Secing 3d chairs: Exemplar part-based 2d-3d alignment using large dataset of cad models," in Proc. International Conference Computer Vision and Pattern Recognition (CVPR), 2014.
[16] Y. LeCun, C. Cortes, and C. Burges. "Mnist handwritten digit database." ATT Labs, vol. 2, 2010. [Online]. Available: http: /yann.aa..... Lecun. com/ exdb /mnist.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by two time-scale 2017.
update rule converge to local nash equilibrium."" " arXiv preprint
[18] M. Tipping and C. M. Bishop. "Probabilistic principal component analysis." Journal of the Royal Statistical Society. Series (Statistical Methodology), vol. 61, no. 3, pp. 1-622, 1999.
[19] P. Ghosh, S. Sajjadi, A. Vergari, Black, and Scholkopf, "From variational to deterministic autocncoders," in International Conference on Learning Representations (ICLR), 2020.
[20] Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf, "Wasserstein auto-encoders," in International
Conference on Learning Representations (JCLR), 2018.
[21] S. Kolouri, P. F. Pope, C. F. Martin, and G. K. Rohde, SSliced-wassersted autoencoder: An embar- rassingly simple generative model," in International Conference on Learning Representations (ICLR), 2019.
[22] S. Knop, P. Spurek, J. Tabor, 1. Podolak, M. Mazur, and S. Jastrzębski, "Cramer-wol, auto-encoder." Journal of Machine Learning Research, vol. 21, 2020.
[23] Y. Burda, R. Grosse, and R. Salakhutdinov, "Importance weighted autoencoders," in International Conference Learning Representations (ICLR), 2016.
10

---

Under review as a conference paper at ICLR 2022
[24] C.-W. Huang, K. Sankaran, E. Dhekane, A. Lacoste, and Courville, "Hierarchical importance weighted autoencoders," " International Conference Machine Learning (ICML), PMLR, 2019, pp. 2869 2878.
[25]
A. B. Dieng and Paisley. "Reweighted expectation maximization." arXiv preprini arXiv: 0900099000000...
2019.
[26] G. Roeder, Y. Wu, and D. Duvenaud, "Sticking the landing:
Simple, lower-variance gradient estimators for variational inference," " Proc. Neural Information Procesinng Systems (NIPS), 2017.
[27] Y. Park, C. Kim, and G. Kim, "Variational laplace autoencoders," in International Conference on Machine Learning (ICML), PMLR, 2019, pp. 500255030000 5041.
[28] M. Tomezak and M. Welling. "Improving variational auto-encoders using householder flow." arXiv
preprint 161 000630, 2016.
[29] D. Nielsen, P. Jaini, E. Hoogeboom, O. Winther, and M. Welling, "Survac flows:
Surjections to bridge the gap between vacs and flows," in Proc. Neural Information Processing Systems (NIPS), 2020. [30] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. LLkshminarayannn, "Normalizing flows for probabilistic modeling and inference." Journal of Machine Learning Research, vol. 22, no. 57, pp. 1-64, 2021.
[31] R. Morrow and W.-. Chiu, "Variational autoencoders with normalizing flow decoders." arXiv preprint
arXiv:2004006617 2020.
[32] B. Taylor, Methodus incrementorum directa et inversa. Londini: Typis Pearsonianis prostant apud Gul. Innys ad Insignia Principis in Coemeterio Paulino, 1715.
[33] C. Wang and D. M. Blei, "Variational inference in nonconjugate models.." Journal of Machine Learning Research, vol. 14, no. 4, 2013.
[34] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, "Variational inference: A review for statisticians."
Journal of the American statistical Association, vol. 112, no. 518. pp. 859-877, 2017. 2017.
[35] S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, Jozefowicz, and Bengio, "Generating sentences from continuous space," in Proceedings of The 20th SIGNLI. Conference on Computational Natural Language Learning, Association for Computational Linguistics. 2016.
[36] C. K. Sønderby, T. Raiko, L. Maalóe, S. K. Sønderby, and O. Winther, "Ladder variational autoencoders." in Proc. Neural Information Processing Systems (NIPS), 2016.
[37] A. Razavi, A. V. Oord, B. Poole, and O. Vinyals, "Preventing postcrior collapse with delta-vacs," in International Conference on Learning Representations (ICLR), 2019.
[38] M. I. Jordan, Z. Chahramani, T. S. Jaakkola, and L. K. Saul, "An introduction to variational methods for graphical models," " Machine learning. vol. 37, no. 2. pp. 183-233 1999.
[39] M. P. Deisemroth, A. A. Faisal, and C. S. Ong, Mathematics for machine learning. Cambridge University Press, 2020.
[40]
M. A. Woodbury, "Inverting modified matrices," Memorandum report, vol. 42, no. 106, p. 336, 1950.
[41] D. Harville, "Matrix algebra from a statistician's perspective springer," New York, 1997.
[42]
Kingma and J. Ba, " "Adam: method for stochastic optimization," in International Conference on Learning Representations (ICLR), 2015.
[43] B. M. Lake, R. Salakhutdinov, and B. Tenenbaum, "Human- concept learning through probabilistic
program induction." Science, vol. 350, no. 6266, pp. 1332- -138,, 2015.
[44] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," Proc. Neural Information Processing Systems (NIPS), 2019, pp. 8024 8035.

---

Under review as a conference paper at ICLR 2022
A
TAYLOR SERIES VISUALIZATION
8(x)
Pe(s)
Example of Taylor series expansion of a locally affine neural network point g(x). (gray) is the original function / neural network, and fo(z) (orange) is the Taylor series of f point g(x). As g(r) is the mean of the latent normal distribution that is transformed by f. the Taylor series at point (2x) gives a good approximation in the vicinity of g(x).
Figure A.1:
B IMPLEMENTATION DETAILS
In the following, we discuss implementation details including network architectures.
Throughout the experiments in this work, we 0.5, which coincides with many public implementations (which omit the prefactor of 2c set this to 1).
B. CELEBA AND 3D CHAIRS
Network Architecture For the CelebA and 3D Chairs experiments,
the architecture by
Higgins et al. [3]. The encoder consists of five convolutional layers with kernel size 4 x 4. a stride of 2, and (32, 32, 64, 64, 256) latent dimensions with ReLU activations. Following that, a fully-connected layer mapping to the latent space. The explicit architecture is: Conv 32x4x4 (stride 2), ReLU, Conv 32x4x4 (stride 2), ReLU, Conv 64x4x4 (stride 2), ReLU, Conv 64x4x4 (stride 2), ReLU, Conv 256x4x4 (stride 1), ReLU, FC 256-1latent_dim
The decoder is the deconvolutional reverse of the encoder. In the qualitative evaluation, we use latent dimension of 32 for CelebA, and a latent dimension of 16 for 3D Chairs. In the quantitative evaluation, we use the range of 12 64 for CelebA, and the range of 16 for 3D Chairs.
Training We trained the CelebA and 3D Chairs models for 1 iterations with the Adam opti- mizer [42] at learning rate 10 10-4
B.2 MNIST
Network Architecture Here, use fully connected network architecture. The encoder consists of three fully connected layers with hidden dimensions (512, 256) and with ReLU activations. The decoder is the reverse of the encoder. For the 2D traversal experiment, we use latent dimension of and for the experiments examining posterior collapse latent dimension of 200.
Training We trained the MNIST model for the 2D traversal for 10 iterations with the Adam optimizer [42] at learning rate of 10-4, For the experiments examining posterior collapse, we trained for 10 iterations epochs at full batch training on reduced data set of 1 000 random samples and with learning rate of 10
C ADDITIONAL RESULTS
C.1 QUANTITATIVE EVALUATION FOR 3D CHAIRS
See Figure C.I for the quantitative performance the sampling-fre VAE compared to a sampling VAE for different latent dimensionalities.
12

---

Under review as a conference paper at ICLR 2022
Sampling-Free VAE
-7078.4
Sampling
-7078.6
0
60
7078.8
-7079.0
77073.4
Sampling Free VAE
50
7079.6
Sampling
lafent dimensions
# of latent dimersions
Figure C.I: Quantitative evaluation of FID (left) and log marginal likelihood (right) on the 3D Chairs data set for different sizes of the latent space. On FID (lower is better), the sampling-free VAE significantly outperforms the sampling VAE and performs on par / better regarding the log marginal likelihood (larger is better). Results average over 3 runs and minimum / maximum marked.
C.2 STANDARD DEVIATIONS
Table 5: Table but with standard deviations.
Method
CelebA
3D Chairs
FID
log ML
FID
log ML
Sampling- Free Gaussian VAE
95.79 + 0.33
7244.9 + 0.6
51.96 1.75
-7078.8 + 0.1
Sampling Gaussian VAE
99.66 + 0.65
-7245.3 0.1
53.53 - 1.05
-7078.9 + 0.0
Sampling-Fre (matching budget)
98.00 + 1.79
-7247.1 0.4
54.21
1.63
-7079.6 0.1
C.3
GAP BETWEEN UNBIASED ELBO AND BIASED ELBO
7500
8000
Sumpling- ELBO
Sampling ELBO
8500
7.996
7-0.9
7-0.8
0-077
0.5 0.5
9000
U- 10"
1 1-300
2: 102
3 10
4 10
5 10
6 60
Number Iterations
Gap between unbiased ELBO (Sampling ELBO) and biased ELBO (Sampling-Free ELBO) for different ys. Both lines are of the same model, which is trained with the biased ELBO and evaluated with both methods. Same training setting as Figure 1.
Figure C.2:
C.4 FURTHER EXAMPLES, FULL LATENT TRAVERSALS, AND COMPARISON TO OTHER METHODS
In Figures C.3-C.4, we show reconstructions and full latent traversals for 3D Chairs for the sampling- free VAE, the sampling VAE, the sampling 3-E,E, and an auto-encodere. In Figures C.5-C.8, show the same for CelebA. For the auto-encoder on the 3D Chairs data set, even among variety of hyper-parameters,, training always converges to the plain white image.
13

---

Under review as a conference paper at ICLR 2022
Analytical VAE
Sampling
VAE
Sampling VAE
Auto-Ern Outo-Encoder
%
F
h
h
4
%
4
R
4
R
R
b
&
%
%
&
Figure C.3: Additional reconstructions and all traversals for the 3D Chairs data set (part 1).
14

---

Under review as a conference paper at ICLR 2022
Analytical VAE
Sampling
VAE
Sampling VAE
Muto-Encoder
M
h
R
+
R
+
R
A
R
%
%
4
4
4
&
4
R
R
R,
R
Figure C.4: Additional reconstructions and all traversals for the 3D Chairs data set (part 2).
15

---

Under review as a conference paper at ICLR 2022
Analytical VAE
Sampling B - VAE
Sampling VAE
Auto- Auto-Encoder
Figure C.5: Reconstructions and traversals for the CelebA data set with 32 latent dimensions (part 1).
16

---

Under review as a conference paper at ICLR 2022
Analytical VAE
Sampling g - VAE
Sampling VAE
Auto-Encoode Encoder
a
Figure C.6: Reconstructions and traversals for the CelebA data set with 32 latent dimensions (part 2).
17

---

Under review as a conference paper at ICLR 2022
Analytical VAE
Sampling B - VAE
Sampling VAE
Auto-Encoder
Figure C.7: Reconstructions and traversals for the CelebA data set with 32 latent dimensions (part 3).
18

---

Under review as a conference paper at ICLR 2022
Analytical VAE
Sampling B - VAE
Sampling VAE
Auto-Encoder
Figure C.8: Reconstructions and traversals for the CelebA data set with 32 latent dimensions (part 4).
19


Reviews:
[{"review_id": "tvYV8SO8aJi", "reviewer": "Reviewer_smp3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Edit: Thank you for the rebuttal. The authors clarify some of my concerns and include some minor improvements in the experimental comparisons, which is why I slightly increase the score (3 $\\to$ 5). However, the overall contribution still remains minor in my opinion which is why I can't recommend acceptance._\n__________\n\n\n### Strengths\n- The paper is overall well written, and the approach is evaluated on several data sets.\n- The computational cost introduced by the approach is explicitly discussed and not hidden in an appendix. \n\n### Weaknesses\n- The contribution itself is very minor, as Taylor approximations to solve expectations are a common approach in the literature. (See, e.g. Wang and Blei (2013); Blei et al. (2017) and the references therein for further pointers, or the keyword delta method.)\n- The sampling-free method aims to offer a better approximation to the ELBO. However, the ELBO in itself is only of limited use and mainly serves as a gradient signal. While the experiments seem to show that the approach improves upon a naive sampling, the paper lacks a comparison (and a discussion) to other methods that aim to stabilize the gradient e, e.g. Roeder et al. (2017) and similar methods. \n- Similarly, a comparison with other approaches of similar network depths to properly place, e.g. its FID into relation with the current literature, is missing.\n\n### Minor points & Questions\n- How to read Figure 1 and the $\\gamma$ discussion? Sec 3.3 first mentions the exploration of three $\\gamma$ values (0.5,0.9,0.99), while Figure 1 explores a different set of values, which influence only the orange sampling ELBO, not the blue sampling-free ELBO. Is the legend broken here? Similarly, what does the number of iterations refer to, as the blue line seems to have converged from the beginning?\n- Figures 7,8 in the appendix seem to be broken for the Autoencoder. Is that an error in the figure or due to problems with the underlying model? \n- Sec 4.1 mentions the \"true output... via a large number of samples\". What is large here? _(I might have overlooked the hyperparameter in the appendix)_\n- What is the precise architecture used in the experiments? Sec 4.2 first claims to follow Burgess et al.'s architecture, then later mentions a 5 conv layers + 1 fully connected layer structure. Appendix B.1 switches to claiming to follow Higgins et al. instead but mentions again five convolutional layers, which is not what Higgins et al. report to be using in their appendix (they claim four convolutional layers). (Burgess et al. also use four convolutional layers but a different number of channels compared to Higgins.)\n- Are the minimum/maximum missing from the right plot in Figure 4, or so close to the mean to not be visible?\n- Table 2 gives the average over multiple runs. What is the standard deviation for the results?\n\n\n__________\nWang and Blei, Variational Inference in Nonconjugate Models, JMLR 2013  \nBlei et al., Variational Inference: A Review for Statisticians, JASA, 2017  \nRoeder et al., Sticking the landing: Simple, lower-variance gradient estimators for variational inference, NeuRIPS 2017\n", "summary_of_the_paper": "The paper proposes a sampling-free approximation to the ELBO of a variational auto-encoder with Gaussian likelihood and mean-field Gaussian variational posterior. The approach relies on a Taylor series around the mean of the posterior, which allows for an evaluation of the ELBO's expectation, rather than relying on Monte Carlo Sampling.\n", "summary_of_the_review": "The paper demonstrates the usefulness of a Taylor approximation compared to a sampling-based approach. However, that in itself is of limited novelty, and the paper lacks a proper comparison to other methods and the broader literature to classify it properly.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "enkaahz30b4", "reviewer": "Reviewer_AwHy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nStrengths \n\n1. The closed-form approximation reduces the computation cost for computing the ELBO of exact Gaussian VAE. \n2. The paper is well organized and clearly written. \n\n\nWeakness.\n\n1. The proposed method is similar to the well-known moment matching (MM) methods. However,  a discussion of the relationship between the proposed method and MM is missing. In addition, it is better to include a comparison with MM to justify the benefit of the proposed method. \n\n2. The proposed method is limited to Gaussian VAE.  It may not apply to VAE with other distribution. \n\n3.  In experiment 4.1,  how to set the hyper-parameter $\\gamma$ in Eq.(21).  It is better to set $\\gamma=1$ to justify the theoretical approximation quality. ", "summary_of_the_paper": "\nIn this paper, the authors propose a closed-form approximation for the Gaussian decoder distribution in VAE.  The closed-form is achieved by matching the mean and covariance matrix with the Taylor expansion of the nonlinear mapping. ", "summary_of_the_review": "\nOverall,  I think the proposed method is similar to the moment matching method. It is marginal novel and limited to the Gaussian case. Thus, in my opinion,  this paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UQwMinaoWbc", "reviewer": "Reviewer_Zwr6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-rebuttal: Thank you for the rebuttal. The authors addressed some of my concerns, added experimental comparison and updated the text. However, the overall novelty and contributions are still not adequate for ICLR in my opinion. I will not change the score.\n\n######\n\nPros: \n \n+ The analytical approximation would help with reducing the variance and computational expense of the ELBO. \n \n+ The proposed method achieves similar or better performance than its sampling counterpart.\n\n+ The proposed method is found alleviated the issue of posterior collapse that the sampling VAEs suffers.\n\n\nCons: \n\n- The writing should be substantially improved. The words and sentences are not so fluent. There are obvious grammar errors and typos. I believe that it is critical for the readers to correctly understand and evaluate the contributions. The figures are labeled in a random order.\n\n- The advantages in performance looks marginal comparing to the sampling VAE. The proposed method beats sampling VAE for small number of samples. I don't mean to criticize this point since it's an approximation while the sampling ELBO is an unbiased estimate. Instead, the claimed advantages are not well demonstrated. For example, the authors could show the wall time and variance of ELBO. Sec 4.4 is qualitative.\n\n- The proposed approximation is limited to Gaussian encoders and ReLU decoder networks. The Taylor expansion itself should be applicable to other types of networks, but the performance then needs further evaluations. \n\n- Posterior collapse part could be more elaborated. Can the authors address more how this approximation could solve the issue?\n\nConcerns:\n\n* What's its connection to the Laplace method in variational inference.\n* Strictly speaking, the approximation is not the exact ELBO using in the sampling VAE and thus different objective in fact. Futhermore, it's not necessarily a lower bound to the marginal likelihood. What is the gap between the approximation and the ELBO or true marginal likelihood?\n* The covariance of reconstructions evaluated in the paper is not clearly defined? For approximate VAE and sampling VAE, the output is different. The former is a distribution but the latter is a sample.\n* Sec 3.2: Why is the intractability due to exponential complexity rather than integrability?\n\nMinor:\n\n* The computional complexity better be in form of big O notion.\n* The layout of Eq. 16 better be topdown.\n* Figure 5 ref before Figure 1", "summary_of_the_paper": "The paper proposes an approximation that frees the calculation of ELBOs of Gaussian variational autoencoders from sampling. To achieve this, it utilizes Taylor expansion on the decoder networks. The proposed method was evaluated on three typical datasets. The authors also found that the approximation alleviated the issue of posterior collapse to the sampling VAEs.", "summary_of_the_review": "Overall, I vote for rejecting. I like the idea that analytical approximate ELBO would help the computation. The finding on posterior collapse is also interesting. However, the method and evaluation are not adequately demonstrated. The writing can be largely improved too.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ULJed9RFWjV", "reviewer": "Reviewer_SKjV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Update after rebuttal* I have read the authors’ response and other reviews. I would like to thank the authors for their response. They have addressed my concerns - limited experiments and poor presentation. Although, the related work section could still use some work in terms of writing (for example, to avoid presentation in a form of a list “A did this, B did that”, but it has been improved. I am therefore increasing my score.\n\n————————————————————\n\nStrong points:\n1. Simple yet novel approach for approximating ELBO that allows analytical solution\n2. Sampling-free training of Gaussian VAEs that scales to large networks\n3. Rather thorough experiments in terms of covering the question from different perspectives, i.e. assessment of Taylor approximation, quantitative performance, posterior collapse discussion\n\nWeak points:\n1. Limited experiments in terms of considering other baselines. Related works cover a lot of different VAEs and their training methods, yet almost all the experiments are on contrasting with sampling VAE (without the proper specification which specific sampling VAE is used)\n2. Motivation for the sampling free training for VAE in general as a concept is not very well provided considering that it comes with the computational overhead. The text implicitly assumes that sampling free training is obviously better and only additionally support it with better performance\n3. Some writing issues. The related work section in particular could see a lot of improvement. Considering the context of the paper, one would expect the related work section to focus on the discussion of the training methods for VAEs and on contrasting sampling vs sampling free methods. Instead a reader finds a generic overview of different VAEs, with some works that do not seem to be really related to the submission apart from being devoted to VAEs, e.g. Tolstikhin et al, or Rezende et al.\n\nI am voting for a weak acceptance of the paper as I found the general idea of the paper to be appealing and worth attention of the community, however the above mentioned weaknesses stop me from higher score. I believe the paper can be made very strong with some work as thorough empirical support and better focussed presentation.\n\nSpecific comments/suggestions (not necessarily important for assessment, but points to improve the paper):\n1. Mix of reference styles is used with Author and [Number] mixed\n2. Related work, first paragraph. 3 layer network, specific number of hidden units and latent dimension, MNIST dataset - these may be too much details for related work description. \n3. Related work, last sentence - Not very clear what this is for in this section\n4. f_{\\theta} is not defined\n5. I_m is not defined\n6. Figure 1. Not clear what sampling and sampling-free mean here as it seems that sampling one should not depend on \\gamma, a hyperparameter of the proposed sampling-free method\n7. Figure 4. Why there is no shaded area for log ML plot? \n8. It would be interesting to see a comparison between the proposed approximated sampling free method and the exact sampling free method, even though it would mean to rely on a small scale experiment\n\nMinor: \n1. Section 3.3. \"that works independentLY\"\n\n", "summary_of_the_paper": "The paper proposes a learning method for Gaussian variational auto-encoders based on Taylor expansion approximation. This allows an analytical formula for ELBO and therefore avoid the requirement of sampling during training. ", "summary_of_the_review": "The paper seems to provide an interesting idea for training a VAE that allows an analytical solution and therefore does not need sampling, but the empirical evaluation of the method is somewhat limited and presentation of the paper could be significantly improved. Therefore, I am voting for the weak acceptance as I believe the idea is worth to be known by the community but the existing drawbacks stop me from giving a higher score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a way to train Gaussian variational autoencoders that does not require the computation of empirical expectations but instead approximates the decoder network by its Taylor series. Results on 3 datasets show the competitiveness of the approach.

Based on the limited novelty of the approach, three (out of 4) knowledgeable reviewers recommend rejection and I agree. Variational autoencoders are simply doing variational inference in a specific model and, as one of the reviewers has pointed out, these types of approximations have been exploited in the inference world (before the popularization of the reparameterization trick)  for many years. Methods, where we replace a term in the joint distribution with a simpler function, are known in the variational inference world as local variational approximations, see, e.g. Murphy’s book (Machine Learning: A Probabilistic Perspective, 2012, Sec. 21.8) as a reference. The community has departed from such approaches as using the re-parameterization trick is unbiased, more general (e.g., not limited to Gaussian encoders) and allows for highly automated methods (no need to do derivations on a case-by-case basis).  Nevertheless, I encourage the authors to thoroughly explore the literature on variational inference with regards to these types of approximations. It may well be the case that, in the future, we revert back to these methods if they perform well in practice with modern architectures. For this, more comprehensive evaluations and comparisons are needed.
Decision: Reject