Variance Reduction in Hierarchical Variational Autoencoders
Authors: Adeel Pervez
Efstratios Gavves
Venue:  (2021)

Abstract: Variational autoencoders with deep hierarchies of stochastic layers have been known to suffer from the problem of posterior collapse, where the top layers fall back to the prior and become independent of input.
We suggest that the hierarchical VAE objective explicitly includes the variance of the function parameterizing the mean and variance of the latent Gaussian distribution which itself is often a high variance function.
Building on this we generalize VAE neural networks by incorporating a smoothing parameter motivated by Gaussian analysis to reduce variance in parameterizing functions and show that this can help to solve the problem of posterior collapse.
We further show that under such smoothing the VAE loss exhibits a phase transition, where the top layer KL divergence sharply drops to zero at a critical value of the smoothing parameter.
We validate the phenomenon across model configurations and datasets.

Under review as a conference paper at ICLR 2021
VARIANCE REDUCTION IN HIERARCHICAL VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Variational autoencoders with deep hierarchies of stochastic layers have been known to suffer from the problem of posterior collapse, where the top layers fall back to the prior and become independent of imput. We suggest that the hierarchical VAE objective explicitly includes the variance of the function parameterizing the mean and variance of the latent Gaussian distribution which itself is often a high variance function. Building on this we generalize VAE neural networks by incorporating smoothing parameter motivated by Gaussian analysis to reduce higher frequency components and consequently the variance in parameterizing functions and show that this can help to solve the problem of posterior collapse. We further show that under such smoothing the VAE loss exhibits a phase transition, where the top layer KI. divergence sharply drops to zero at a critical value of the smoothing parameter that is similar for the same model across datasets. We validate the phenomenon across model configurations and datasets.
INTRODUCTION
Variational autoencoders ( (VAE) [10] are popular latent variable model for unsupervised learning that simplifies learning by the introduction of a learned approximate posterior. Given data æ and latent variables x, we specify the conditional distribution p(z|z) by parameterizing the distribution parameters by a neural network. Since is difficult to learn such a model directly, another conditional distribution (==[x=) is introduced to approximate the posterior distribution. During learning the goal is to maximize the evidence lower bound (ELBO), which lower bounds the log likelihood, log p(x) 2 Eg(s|s) log p(x]=) + log p(z) log (<=[=²]. In their simplest form, the generative model p(x[(z) and the approximate posterior q(= (r) Gaussian distributions optimized in unison.
natural way to increase the modeling capacity of VAE is to incorporate a hierarchy of stochastic variables. Such models, however, tend to remain independent of input data problem termed posterior collapse. Posterior collapse in VAEs manifests itself by the latent distribution tending to fall back the prior. With hierarchical VAES the effect is found to be more pronounced in the top layers farther from the output. For the purpose of the paper and for clarity of exposition, we focus on the simplest extension of hierarchical variational autoencoders where stochastic layers are stacked serially on top of each other 12, 21]] p(z, = p(z)= ((())) 2L) I1 distributions in this model are commonly taken to be Gaussian distributions parameterized by neural network functions, so that p(2|2++1) = NV22 ((++1)) )). a (2)+1))), where m(z),a(( are neural networks computing the mean and variance of the Gaussian distribution. We refer to them as vanilla hierarchical variational autoencoders. For each stochastic layer in this model there corresponding
out be difficult to train and higher levels in the hierarchy
L-1 p(=]|z++1) and g(>[zz) 4(2) 21 |x: I 122==11z;). The intermediate
KI divergence term in the objective given by
(1)
As described later, expression can be easily decomposed to show an explicit dependence on the variance of the parameterizing functions (1(2)), a (2;) of the intermediate Gaussian distribution. We further show the KL divergence term to be closely related to the harmonies of the parameterizing function. For complex parameterizing functions the KL divergence term has large high frequency components (and thus high variance) which leads to unstable training causing posterior collapse.

---

Under review as a conference paper at ICLR 2021
Building this, we suggest method for training the simplest hierarchical extension of VAE that avoids the problem of posterior collapse without introducing further architectural complexity [3] 21]. Given hierarchical variational autocncoder,
training method incorporates smoothing parameter (we denote this by p) in the neural network functions used to parameterize the intermediate latent distributions. The smoothing is done such that expected values preserved, the higher frequencies are attenuated and the variance is reduced. Next, the gradients computed with the smooth functions are used to train the original hierarchical variational autoencoder.
For the construction of the smoothing transformations for VAEs with Gaussian latent spaces we make use of ideas from the analysis of Gaussian spaces. We analyze the stochastic functions in vanilla hierarchical VAEs as Hermite expansions on Gaussian spaces [9]. The Saatein-Uhlenbeck (OU) semigroup from Gaussian analysis is set of operators that we show to smoothly interpolate between random variable and its expectation. The OU semigroup provides the appropriate set of smoothing operators which enable us to control variance and avoid posterior collapse.
We further show that by smoothing the intermediate parameterizing functions ((=), in the proposed manner, the KL divergence of the top layer sees a sudden sharp drop toward zero as the amount of smoothing is decreased. This behaviour is retained when we evaluate the KL divergence on the original unsmoothed variational autoencoder model. This behaviour is reminiscent of phase transi- tions from statistical mechanics and we adopt the same terminology to describe the phenomenon. Our experiments suggest that the phenomenon is general across datasets and commonly used architectures. Furthermore, the critical value of the smoothing parameter p which the transition occurs is fixed for a given model configuration and varies with stochastic depth and width.
We make the following contributions. First, we establish connection between higher harmonics, variance, posterior collapse and phase transitions in hierarchical VAEs. Second, we show that by using the Ornstein- -heeenbek semigroup of operators on the generative stochastic functions in VAEs we reduce higher frequencies and consequently variance to mitigate posterior collpase. We corroborate our findings experimentally and further obtain in CIFAR- 10 likelihoods competitive with more complex architectural solutions alongside reduction in model size. We refer to the proposed family of models as Hermite variational autoencoders (HVAE).
2
HERMITE VARIATIONAL. AUTOENCODERS
2.1 ANALYSIS ON GAUSSIAN SPACES
The analysis of Gaussian spaces studies functions of Gaussian random variables. These are real- valued functions defined on IR" endowed with the Gaussian measure. Many functions employed in machine learning are instances of such functions: decoders for variational autoencoders, as the case in this work, and generators for generative adversarial networks being two examples.
By way of summary. the main facts we use from this field are that function on a Gaussian space can be expanded in an orthonormal basis, where the basis functions are the Hermite polynomials. This orthonormal expansion is akin to a Fourier transform in this space. The second fact is that the coefficients of such an expansion can be modified in way to reduce the variance of the expanded function by applying an operator from the Ornstein- Uhlenbeck semigroup of operators. Next, we give brief introduction. For further details on Gaussian analysis we refer to [9].
Gaussian Spaces: Let LIR 7) be the space of square integrable functions, f I + R, with the Gaussian measure 722) II, (2,10, 1). Given functions f.g in this space, the inner product is given by (f.g) = (f
Basis functions for T.2 TT, Taking the space of univariate functions I,2 (R, 7). it is known that the polynomial functions (=(2)) z* are basis for this space. By process of orthonormalization we obtain the normalized Hermite polynomial basis for this space. The first few Hermite polynomials
the following: ho = 1, (2) 2, h2
Basis functions for L (R", ,7): Letting a E No be a multi-index, the basis functions for L (2R" .7) obtained by multiplying the univariate basis functions across dimension, ha (z) = II. Na, (2i)-
2

---

Under review as a conference paper at ICLR 2021
Hermite expansion: A function in (2( can be expressed as Ecenr f(a)ha, where f(a) are the Hermite coefficients of f and are computed as f(a) = ha) = (=) Plancherel's s theorem is the following relation between the norm of and which follows from orthnormality of the basis functions.
Un = -
Orastein-Uhlenbect (OU) Semigroup: Given parameter P E [0, 1] and a Gaussian variable construct correlated variable z' as 2' pz /1 -p where zn (O, 1) is random standard Gaussian sample. The OU semigroup is set of operators, denoted Up and parameterized by p E [0, 1]. The action of Un on f at is average the function values on correlated z's around
(zf Ez1=If(=') = E.. (p + 11-p2zu))
(3)
The action of the Up operators on the Hermite expansion of function f(2) is to decay Hermite coefficients according their degree, (z](z) Emen. f(x)h, where |ax| Σi
If is reparameterized as σel + μ., the correlated OU sample is given by z' o (pE) /1 p2e2) μ4, where e standard Gaussian variables. This also be expressed in terms of as
2' = pz + (1 - p)p 1 200 -
(4)
2.2 HERMITE EXPANSIONS FOR VAES
Our proposed method is a new training procedure for the vanilla hierarchical variational autoencoder that builds upon Hermite expansions of Gaussian functions and properties of the OU semigroup.
In the context of hierarchical variational autoencoders, the Gaussian function f is the generative model ya (2(41) and (2(2++1) that receives inputs the latent variable 2i+1 return the Gaussian latent variable of the next layer, 2 ~ ((p(((+)) (2(++))))
We make of the following properties of the OU semigroup construct Gaussian functions of lower variance. The first property we employ is that the OU semigroup of operators interpolates between a random variable (p 1) and its expectation (p 0), where the parameter p controls the extent of the interpolation.
Proposition 1 The operators Up retuin the expected value of the operated function, = BiJ]=B[Um[]]
Proposition 2 The operators Up interpolate between a random variable and its expectation. In particular, as p 1, U,. and
and as p O. U,J [[]]
The second property we exploit is that the new random variable Upf(z) has lower variance compared with original variable (z) and is in general a smoother function than fz)). The smoothing properties of the operator Up can be understood by examining the Hermite expansion of Upff. First we note that we can express the expectation and variance of a function f in terms of its Hermite coefficients, which
specifically [e[]] = f() and Var(f) = [[ [[ = [E( - f(0)) = Eacalpo
follows from Plancherel's 's theorem (equation 2
Replacing with U, and using the Hermite expansion of U,J from equation 3 the mean remains the same, B.U,f]] = -0 f(0)= = f(0), and variance reduces like
Var[U,,1] [F(T,]]][[([AFFF [[/)) E[(S f(0)) Σ
(5)
a:/a/30
The last equation indicates that the contribution to the variance by f(a) decays by an amount pa1ol when p E (0, 1). This, in turn, leads to decrease in variance.
3

---

Under review as a conference paper at ICLR 2021
Algorithm. In essence, Hermite variational autoencoders are similar to variational autoencoders, save for applying the OU semigroup to the latent distributions p( 2 2241) that comprise the generator to compute gradients during training only. Specifically, apply these operators to the functions parameterizing the mean and variance of the latent Gaussian distributions. For each distribution p(2)|2++1) we substitute (z)])4(22+)) (x(2++)) with ( ((=](Up/(((4+)) (, The new functions result in latent distributions with parameters that have lower variance but the same expected value relative the conditional input latent distribution.
In an alternative parameterization we apply the OU semigroup to the ratio of the mean and variance functions: Up Hs (21+1) (see next section for a justification of this). The OU semigroup operators can also be applied on approximate posterior functions, but we observe little benefit. In practice, we compute (1 and Upo,(2++1) by Monte Carlo averaging. As for a function f. Unf E=)=] where z' are the correlated samples, we estimate the expectation by Monte Carlo averaging over z'. Experiments show that 10 samples suffice.
is important to emphasize that the substitution of the lower variance functions for parameterizing the distributions is only done when computing gradients during training. All evaluations, training test, are still done on the original hierarchical variational autoencoder model. Thus, the new training procedure has an additional computational cost only for the intermediate distributions in the generator, proportional to the number of correlated samples during training.
Complexity. In Hermite VAE the OU sampling operation is only applied in the intermediate stochastic layers in the generator network. In particular, it is not applied in the inference network or in the last layer of the decoder. The fact that OU sampling is not applied in the final stochastic layer computing p(æ 21) is especially important for deep VAEs for images since feature maps are upsampled to match image dimensions in this layer. Thus, for 5 OU samples, the added computational and activation memory complexity is significantly less than 5 times the total cost of the base VAE model, and is 5 times the cost in the higher decoder layers only in the base model. An empirical comparison of maximum memory usage of various models can be found in table 6
3 KL DIVERGENCE ANALYSIS
In this section we justify our approach as affecting a bias-variance trade-off in the KL divergence terms of the hierarchical VAE objective. The bias-variance trade-off arises from the fact that the KI. divergence term can be written so that it contains the variance of the functions p(z), (=)) parameterizing the intermediate distributions. This variance related to spectral complexity and reducing spectral complexity leads to a reduction in variance.
Consider the following expression from the ELBO as part of the KL divergence term.
Eg(= (log p(z1 22] 1e
22)
and q(=1 (a:) .V (2) (x)) For the purpose of analysis we assume that the standard deviation for is fixed and independent of 22, (22) = ap. Furthermore for ease of analysis and without loss of generality we think of 2 scalar. The general result for multivariate 21 follows from summing for all dimensions.
where
(22) = (N
For Gaussian p write equation 6 as =2 log / 2 2 - (=( From the inner term +2)) (21 (p we focus on the quadratic 21p2222 which is expanded as
E (4 (z2(2) 20 Re (x) ((L + Var(up)]
2
By Plancherel's theorem (equation we have
2 gei .z (1) (12)
1
1 (a(al)) 10)2
E
Σ ug(a)²
(8)
20
a:/a|>0
This shows that for o, independent of 22 the KL divergence term in the ELBO contains the variance of the parameterizing function (z2))
4

---

Under review as a conference paper at ICLR 2021
In our proposal we replace (4p(22) by (22) and the right side of equation becomes
1 E (=(= (x) p Var(Up(lp) 20
gt (x) 12)(0)2
p (ip(a)²
(9)
20,
ac cc/a/>0
since E[U,f] B[ The new variance is of order O(p²). Comparing this objective with the original VAE objective, for the second term we get a bias proportional to the difference of the variance
1
bias
(Var(up) Var(UpHp))
20
Comparing equations and we see the bias to be
The analysis above assumed op independent of Zg. For op dependent on 22 we can make similar argument for the variance of (21 - ((22))/o(22) or for the ratio ((22)/o(22) by expanding the square and considering the terms separately.
4 RELATED WORK
To increase the stochastic depth of VAEs [10 19). (1] proposes the Ladder VAEs. With architec- ture that shares a top-down dependency between the encoder and the decoder, Ladder VAEs allow for interactions between the bottom-up and top-down signals and enable training with several layers VAEs, adding skip
deep VAEs. Extending Ladder VAEs, [ proposed the bidirectional-infernee connections in the generative model and bidirectional stochastic path in the inference model.
observed that the latent distribution collapses to the prior in deep stochastic hierarchies a phenomenon now called posterior collapse. Posterior collapse appears different contexts including images text, and is strongly associated with the presence of powerful decoders, be it LSTMs (I] for text strong autoregressive models for images (16). where although the model may produce good reconstructions, it does not learn a meaningful generative distribution. A prevalent hypothesis behind posterior collapse that when the decoder is strong enough to generate very low cross entropy losses, the optimization may find it easier to simply set the KL divergence term to O to minimize the ELBO [. Making an association with probabilistic PCA, [12] hypothesize that posterior collapse is caused by local optima in the optimization landscape due to high variance, even without powerful decoders. High variance was identified as a potential culprit also by [21] for posterior collapse.
Given the breadth of the problem, many tried to address posterior collapse. 8 21 anneal the KL divergence between the approximate posterior to the prior from to Unfortunately. this solution does not optimize the original ELBO formulation and is shown (27 especially with large datasets and complex decoders. [11 introduce the concept of free bits ignoring the gradient if significant enough. [17 proposed 6-VAEs, which constrain the latent distribution ? monitor the mutual information between the latent and the observed variable aggressively optimize the inference model before every model update. suggest that using a tighter multi-sample ELBO can help alleviate collapse to some extent.
cause instabilities,
to have a minimum distance to the prior.
While many [ 21 suggested a connection between posterior collapse and variance reduction, no real solutions using variance reduction has been proposed. One reason may be the low variance the reparameterizatioo trick 10 [9 already offers. Empirically, while the reparametrization successful with producing low variance forward and backward propagations in shallow models, it has not been enough for deeper and wider ones. Another reason suggested by the approach of this paper that variance appears as side effect of spectral complexity.
To reduce the variance reparameterization gradients, 20 suggest removing a mean zero score function term from the total derivative while [15] build control variate using linear approximation for variance reduction. [2) propose importance weighted gradients and (24] extend [15] to multiple samples to obtain an estimator with improved signal-to-noise ratio. Other approaches increase the power of VAE models include normalizing flows [18), better posterior or prior distributions [2] adding autoregressive components [6] combination of both [1]].
Here, we theoretically argue and empirically validate that damping higher frequency components, thus lowering variance, allows for training deeper latent hierarchies while addressing posterior collapse.
5

---

Under review as a conference paper at ICLR 2021
25
60-
Divergence
KL Divergence
20
2.-
40-
10
K2
MNIST
0.80
0.85
0.90
1.00
0.2
P
Figure 1: VAE KL divergence undergoes phase transition the model decrease the amount of smoothing to recover the original model at 1. The figure shows top layer KL divergence vs. where each point different model after 100000 steps of training with fixed p. The plots show the KL divergence values for the original unsmoothed model after training on the smoothed version. The left figure shows this for same 4 stochastic (40 units per layer) layer MIPP model on MNIST and OMNIGLOT. It can be seen that the critical value where the KL. divergence drops toward 0 is the same for both datasets on this model. The next two figures show phase transition for 3 stochastic layer (200 units per layer) MLP model OMNIGLOT (middle) and convolutional model (8x7x7 latent dimension) on MNIST (right) showing that the phase transition becomes especially pronounced with the width of the latent layer.
We rely tools from the field of analysis on Gaussian spaces, amenable the analysis of stochastic processes 9
5 EXPERIMENTS
In the following experiments we first test our method's ability to prevent posterior collapse and compare with other methods designed for the same end. We then present observation that the top layer KI. divergence undergoes a phase transition as we decrease the amount smoothing. Finally, compare performance commonly used benchmarks against other methods from the literature.
We validate Hermite variational autoencoders on binary MNIST, OMNIGLOT and CIFAR-10 with various ResNet and MLP architectures. Validation ELBOS are evaluated using importance-weightel samples with 100 and L3000 denoting evaluation with 100 and 5000 samples.
5.1 INVESTIGATING POSTERIOR COLLAPSE
Table Test ELBO on statically binarized MNIST with MILP
We test Hermite VAEs for their ability to prevent poste- rior collapse on basic MLP network architectures. We compare static and dynamically binarized MNIST against various standard methods for mitigating pos- terior collapse. For dynamic MNIST choose models: the first is a 4 stochastic layer model with 64, latent variables. The deterministic layers have two layers of 512,256, 128,,,,,,26666... 64 units respectively, going from bottom to top. The second model has 4 stochastic layers VAE with 40 units per stochastic layer and layers of 200 units per stochastic layer. For static MNIST we only use the second model described above. All models have a simple stacked architecture with no skip connections.
Model
ELBO
VAE (L=2)
-86.05
VAE (L=I)+NF 18
-85.10
IWAE (L=2) 2
-85.32
VampPrior (L=2)
23
-83.19
HVAE (L=4), L0000
-83.42
Table 2: Test ELBO on dynamically bina- rized MNIST with ML.P
Model
ELBO
Ladder VAE (L.55)
-81.7
VampPrior (I=2)
2
-81.24
First we compare with a standard VAE on static MNIST. We trained the VAE with the standard training method and our method with p E (.... We show the validation curves and the KL divergence of the top stochastic layer K (L(((()))))))))) (o, 1) in figure 2 The standard training collapses the posterior immediately while method avoids posterior collapse and yields better validation ELBO.
HVAE (L=4), Ls0on
-81.2
HVAE (L=5), Ls000
-81.1

---

Under review as a conference paper at ICLR 2021
Table 3: Posterior collapse dynamic MNIST. The table shows top layer KL divergence and active units (top-to-bottom) for various method on the stochastic layer models. The 64-32-16- 8 latent dimension models have two layers of 512, 256, 128, 64 hidden units in each stochastic layer respectively. The 40-40-40-40 latent dimension models have two layers of 200 units per stochastic layer. '+KL' indicates KL annealing. All models were trained for 1M steps with the same hyperparameters.
Model
ELBO (C100)
Reconstruction
KLD
Top KLD
Active Units
IWAE (64-32-16
84.46
-65.3
23,98
4,88
3,15,30,64
IWAE (40-40-40-40)
84.63
-65.5
23.98
1.18
0,4,37,40
VAE+KL (64-32-16-8)
-8.6
-60.3
28.8
6.2
6,11,25,49
VAE+KL (40-40-40-40)
-84.7
-60.8
28.07
1.13
1,6,15,40
VAE+Frechits (64-32-16-8)
-85.5
-64.2
25.1
3.8
2,4,9,21
VAE+Frecbits (40-40-40-40)
-86.0
-65.8
23.6
2.46
1,2,8,18
HVAE (p- 0.95) (64-32-16-8)
-81.6
-59.78
26.1
8,99
8,16,32,54
HVAE (p = 0.9) (4-32-11-8)
-81.7
-60.0
25.6
9.56
8,16,32,43
HVAE (p = 0.95) (40-40-40-40)
-84.4
-65.7
23.7
9.34
40,40,40,40
Table 4: Posterior collapse on static MNIST. The table shows top layer KL divergence and active units (top-to-boite -ot-to-bottom) for various method on the same 4 stochastic layer model with 40 units each, *KLL' indicates KL annealing. All models were trained for 1M steps with the same hyperparameters.
Model
V. ELBO (C100)
Top KLDD
Active Units
VAE
-86.4
0.82
1,3,9,17
IWAE (5 samples)
-8499
1,005
1,3,9,24
VAE+KL
84.5
1.2
2,3,11,37
VAE+Freehits
-87
1.3
1,1,8,17
HVAE (p 0.85)
-85.2
20.6
40,40,40,24
HVAE (p = 0.9)
-8.55
19.5
40,40,40,40
HVAE+KL (p = 0.95)
-84.2
23.5
38,34,17,40
Next we compare against other methods designed to mitigate posterior collapse including KL annealing, free bits [1] and importance weighted objectives [2]. For KL annealing the annealing coefficient is set to 0 for the first 10,000 steps and is linearly annealed to 1 over the next 500,000 steps. For free bits, we apply the same free bits value to each stochastic layer. The free bits values are chosen from {0.5, 1.0, 2.0, 3.03. We find that training slows down considerably when using free bits. Values of free bits of 4.0 or more caused training to become unstable.
For this comparison we use the 40-40-40-40 4-layer architecture used for all methods for static MNIST and both the 40-40-40-40 and 64-32-16- architectures for dynamic MNIST. We show the results in tables and 3 where we include the total KL divergence, top layer KL divergence as well the number of active units in each of the layers. Compared to the baseline methods, method is able to maintain significant activity across the layers and a considerably higher KL divergence in the top layer. For dynamic MNIST the 64,32,16,8 model has successively smaller networks higher in the hierarchy and the other methods show better activity in the latent units than the 40-40-40-40 model. This suggests that training dynamics and the architecture affect the extent of posterior collapse with standard methods and have less of an effect with our proposal.
Here also find that employing other posterior collapse mitigation techniques can sometimes help with HVAE as well. As shown in table 4, KL annealing with the 4 layer HVAE allows it to reach higher validation ELBO. However, we did not find KL annealing to be advantageous when used with HVAE on more complex architectures and in particular we do not see improvement in CIFAR-10 10 performance.
5.1.1 PHASE TRANSITIONS
We have given some evidence to show that attenuating the higher frequency components of parameter- izing functions by smoothing, thus also reducing variance, is a justifiable mitigation against posterior collapse.
7

---

Under review as a conference paper at ICLR 2021
Validation ELBO
22
KL Divergenos
02-
VAE
Ours 0.91
13-
Ours
400090 03K00 360003
1000000 120000 1400303 TSUACCO 1803000 ZA0A6S
5500000
Steps
Steps
Figure 2: Training 4 layer VAE models on static MNIST with validation ELBO the left and [ZL[g(z\\>)]]W 1)) on the right. The VAE shows posterior collapse, while HVAE avoids alongside improved validation ELBO.
Here we show that as the amount of smoothing is reduced (as approaches 1) to recover the original VAE gradients, the top level KL divergence shows a sudden decline a critical value smoothing parameter. The sharpness of this decline depends the model configuration (stochastic layers, latent dimension). On the other hand, our experiments suggest that the sharpness of the decline independent of dataset.
Table Comparing bits per dimension, parameter effi- ciency and model depths between hierarchical methods and other well performing methods on CIFAR-10. '+' indicates stochastic skip connections.
An example of this phenomenon can be seen in figure D and the appendix. Here we show the top layer KL divergence af- ter training for 100k steps with varying p for 3 different architectures. We see that
Model
BPD
Layers
Parameters
the phase transition becomes more pro- nounced with greater stochastic dimen- sion. Figure further shows how HVAE can prevent collapse across spectrum of varying stochastic depth and width while maintaining good validation per-
LVAE
13
3.60
15
72.36M
LVAE+
13
3.41
15
73.35M
LVAE+ 13
3,45
29
119.71M
BIVA
13
3.12
15
102.95M
Discrete VAE++ 126
3,38
NICE 4
4.48
formance.
RealNVP
3.49
NVAE 25
2.91
VAE+AF I1!
3.11
5.2 BENCHMARK COMPARISONS
HVAE, L1
3.5
3
9.95M
HVAE, Ci
3.46
4
12.4M
5.2. MNIST & OMNIGLOT
HVAE, L1
3.43
6
16.8M
HVAE+, L1 IVAE+ C100
3.42
3
9.95M
MLPs. For static MNIST we report test ELBO for MI.Ps with 4 stochastic layers with 40 units each. Before each stochastic layer we have two deterministic layers of 200 tanh units.
3.39
3
9.95M
We show the results in table D comparing with others that improve VAEs: IWAE 2), VAE with normalizing flow [18 and VampPrior 2x
The dynamic MNIST results are in table 2 The 5-layer HVAE model in this instance has latent dimensions 64, 32, 16, 8, 4 with layers of 512, 256, 128, 64, 32 units in the respective stochastic layers.
Residual ConvNets. Next, experiment with more complex ResNet architecture MNIST and OMNIGLOT with up to 4 stochastic layers and up to 5 ResNet blocks between stochastic layers (14 14 features maps). The networks have the same hierarchical structure as the MLP VAEs with convolutional latent layers. We do not employ stochastic skip connections between blocks.
We improve the MLP scores reaching -96.08 validation ELBO OMNIGLOT, compared to -97.65 and -97.56 for VAE(L=2) and VampPrior (L=2). The detailed results are in the appendix.
In both experiments obtain competitive scores despite the simple architecture.
8

---

Under review as a conference paper at ICLR 2021
Pancresage
VAE: direegence layer
Stechanti Degic
Soeciantcc Degice
00000000000000000000
332
-12
0000
00000000 011000000400000000
Stochatic Witth
Stcchsstic
Marnuta VAB: Pertentiago of active letect ractaban
Ntemst VAZ: divergwoen oft stackattic haver
Parmtt VAE Zvalince Lover Douad
0O00 W00.000COOOO
10000000000000000
-50
Stochaece Depee
Ste Lonaa Supet.
100
0.0
00
20
10 20
10
50
20
Stadheedi
Stacheedr:
Figure 3: VAEs (above) exhibit suddenly many inactive units when increasing the depth to 2 layers and the width beyond 20 neurons, while collapsing completely after 3 layers and 60 neurons. The proposed Hermite VAEs (below) retain for nearly all cases full latent activity, while maintaining similar ELBOs (not shown). Trained on MNIST for 100K steps for practical reasons (normal convergence if left training), with 2 deterministic, 200-neuron tanh layers before every stochastic layer.
5.2.2 CIFAR-10
We experiment with ResNets of up 6 stochastic layers, interwined with deterministic layers comprising 6 ResNet blocks. We used 100 feature maps for all deterministic layers. The stochastic layers have 8 feature maps of width 16 x 16. We also experiment with skip connections between stochastic layers. We report results in table We obtain an ELBO of 3.5 bpd without skip connections. After adding skip connections, to an ELBO of 3.39 when evaluated with 100 importance samples. This result is on par with the 15-layer Ladder VAE (LVAE) [13. and comparable to the 15-layer LVAE+ adds skip connections to LVAE.
improve 3.42 bpd on the 3-layer architecture. This corresponds
3 architecture that
Table 6: Comparing maximum memory usage for various models. We report the memory usage 4 and 6-layer models CIFAR-10, using 5 deterministic layers per stochastic layer with 100 units per layer and 5 OU or IWAE samples. The memory usage is the maximum amount of used mem- ory with batch size of 64. For LVAE and BIVA we use code from the BIVA
Note also that the improvement comes signification reduction of parameters. Compared Ladder VAE the layer Hermite VAE uses about 7 times fewer parameters.
1
PyTorch repository.
CONCLUSION
Model
Layers
Memory
Parameters (M)
Training variational autoencoders with large hierarchies of stochastic layers has proven to be difficult. An often shared observation when moving to more com- plex variational autoencoders is posterior collapse where subset of the latent vari- ables falls back to the prior distribution.
VAE
4
2.12G
12.4
VAE
6
2.45G
16.8
IWAE
8,59G
12.4
IWAE
6
10.5G
16.8
LVAE
15
6.4G
59.5
BIVA
15
10.3G
103
HVAE
4
3.74G
12.4
HVAE
6
4.6G
16.8
For solution ysis of Gaussian functions and analyze intermediate VAE functions Hermite expansions. We argue that the Ornstein-Ullenbeck semigroup can be used to reduce variance and empirically show that its parameterizing variable can be used to precisely control phase transitions. We validate the analysis and solution on three datasets, MNIST, OMNIGLOT, CIFAR- 10, where we are able to avoid posterior collapse and obtain performance that competitive with more complex architectural solutions.
to the field of anal-

---

Under review as a conference paper at ICLR 2021
REFERENCES
[1] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. 2016. GGeneaating Sentences from Continuous Space In CoNLL.
[2] Yuri Burda, Roger B. Grosse, and Ruslan R. Salakhutdinov. 2016. Importance Weighted Autoencoders. CoRR, abs/1509.005199 ArXiv: 1509.00519.
[3] Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. Variational lossy autoencoder. ArXiv, abs/1611.00 1.02331.
[4] Laurent Dinh, David Krueger, and Yoshua Bengio. 2014. NICE: Non-linear Independent Components Estimation arXiv:1410.8516 [cs]. ArXiv: 1410.8516.
[5] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2017. Density estimation using Real NVP. 00X0000605008803 [cs, stat]. ArXiv: 1605.08803.
[6]
Ishaan Gulrajani, Kaushalendra Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez, and Aaron C. Courville. 2017. PixeIVAE: A Latent Variable Model for Natural Images. ArXiv, abs/1611 100//11110011. .05013.
[7] Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Barg-Kirkpatrick. 2019. Lagging Inference Networks and Posterior Collapse in Variational Autoencoders arXiv:1901.05530 [cs, stat]. ArXiv: 1901.05534.
[8] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. In ICLR.
[9] Svante Janson al. 1997. Gaussian hilbert spaces, volume 129, Cambridge university press.
[10] Diederik Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes.
[11] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved Variational Inference with Inverse Autoregressive Flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4743-4751.
Curran Associates, Inc.
[12] James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. 2019. Understanding Posterior Collapse in Generative Latent Variable Models. page 16.
[13] Lars Maalge, Marco Fraccaro, Valentin Liévin, and Ole Winther. 2019. BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling 190.v:2 [cs, stat), ArXiv: 1902.02102.
[14] Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. 2016. Auxiliary Deep Generative Models arXiv:1602.05473 [cs, stat]. ArXiv: 1602.05473.
[15] Andrew C. Miller, Nicholas Foti, Alexander D' Ammour and Ryan P. Adams. 2017. Reducing reparameterizztion gradient variance
[16] Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel Recurrent Neural Networks. In JCMI..
[17] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing Posterior Collapse with delta- delta-VAEs. arXiv:1901.03416 [cs, stat]. ArXiv: 1901.03416.
[18] Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In ICML.
[19] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic Backpropaga- tion and Approximate Inference in Deep Generative Models. In ICML.
[20] Geoffrey Roeder, Yuhuai Wu, and David Duvenaud. 2017. Sticking the landing: Simple lower-variance gradient estimators for variational inference
10

---

Under review as a conference paper at ICLR 2021
[21] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther.
2016. Ladder Variational Autoencoders. In NIPS.
[22] Jakub M. Tomczak and Max Welling. 2016. Improving variational auto-encoders using house- holder flow, arXiv preprint 1rX1v:1611.0633.
[23] Jakub M. Tomezak and Max Welling. 2017. VAE with VampPrior. arXiv preprint arXiv: 170.00700..
[24] George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J. Maddison. Doubly reparametenized gradient estimators for monte carlo objectives)
[25] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder
[26] Arash Vahdat, William G. Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny An- driyash. 2018 DVAE++: Discrete Variational Autoencoders with Overlapping Transformations arXiv:1802.0449 04920 [cs, stat). ArXiv: 1802.04920.
[27] Zichao Yang. Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. ICML.


Reviews:
[{"review_id": "xnb4AmXOM9z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-rebuttal update\n---------\n\nThank you for your response.  Now I understand that the algorithm works by smoothing the Gaussian parameters $\\mu_i,\\sigma_i$ w.r.t. the centered Gaussian rv (as described in my last reply, second part of bullet point (1)), so my original concern regarding the bias _in the Gaussian parameters_ does not hold.  However, I still cannot recommend acceptance at this point, because of a newly discovered issue in the theoretical analysis:\n\nThe analysis in Section 3 does not take into account the impact of smoothing on the ``downstream'' nonlinear layers.  The text only considers two layers of stochastic latents and the KL part of ELBO, but in the deeper case, the smoothing of $\\mu_i(z_{i+1})$ will additionally have influence on the layers below $i$, through the nonlinear functions $\\mu_{i'},\\sigma_{i'}$ for $i'<i$.  More concretely, consider the following scenario: $\\mu_i(z_{i+1})\\equiv z_{i+1}, \\sigma_i(z_{i+1})\\equiv \\epsilon$ which is very small. Further assume that $z_{i+1}$ is high-dimensional and approximately follows $\\mathcal{N}(0,I)$, so $\\\\|z_i\\\\|_2 = \\\\|\\mu_i(z_{i+1})+\\sigma_i\\varepsilon_i\\\\|_2 \\approx \\\\| z_{i+1} \\\\|_2 > 100$ with probability $1-\\epsilon_1$, where $\\epsilon_1$ is also very small. In this case, it is possible to achieve a low KL in the original ELBO, by using a $\\mu_{i-1}$ which only has sensible values in the region $B := \\\\{z_i: \\\\|z_i\\\\|>100\\\\}$; in the complement set $B^c$, $\\mu_{i-1}$ can be \"arbitrarily\" bad so long as its impact on the ELBO does not outweigh $\\epsilon_1$, the probability its input falls there. However, in the smoothed estimator with $\\rho=0$, the input to $\\mu_{i-1}$ only have norm $O_p(\\sigma_i(z_{i+1}))=O_p(\\epsilon)$, so the value of $\\mu_{i-1}$ on $B^c$ will have a far higher impact, easily exceeding the original by $O(1/\\epsilon_1)$.  To summarize, *it is possible to construct models where the ELBO has a reasonable value, but the smoothed objective behaves catastrophically*.  Moreover, even in the shallow case, $z_i$ will be fed into a final decoder block to generate the reconstruction image, so a similar issue exists, although it will be in the reconstruction likelihood part of the ELBO as opposed to the KL part.\n\nA less important issue is that parts of the analysis are written in a confusing way.  Apart from the abuse of notation $U_\\rho$ which leads to my original confusion, in Section 3 the $\\hat{\\mu}_p$'s should have a suffix of $z_1$, to signify the fact that they are coefficients of a function that depends on $z_1$ (see the last response from he authors).  Also it is unclear to me why there is no mention of $\\mu_p^4$, in the analysis of the variance of an estimator for $\\mu_p^2$.  But given the aforementioned issue, I don't think it is necessary to look further into this case.\n\n\nSummary\n-------\n\nThis work proposes to smooth the mean and variance parameters in the decoder of hierarchical VAEs with the O-U process. It is shown that the smoothing procedure reduces variance of the ELBO, alleviates posterior collapse, and improves on model likelihood on CIFAR-10 under a fixed number-of-parameter budget.\n\nThe idea to investigate the impact of ELBO variance in hierarchical VAE performance is sensible, and the experiments seem to show improvements. However, I have concerns regarding the theoretical claims, and the empirical results also seem to need clarification.\n\nMajor Concerns\n--------------\n\n- The claim that the smoothing doesn't change the expectation (of functions acting on the latents) doesn't seem correct. Prop.1 and 2 only holds when the expectation is taken wrt the standard normal distribution, while all but the top-level latents (i.e., $z_i$ for $i<L$) come from a mixture of Gaussian. Intuitively it also seems incorrect: what if $\\rho=0$?\n\n- The variance analysis works by assuming $\\sigma_q$, the decoder variance, is constant. This ignores the problem of unbounded likelihood [1], where posterior variance goes to zero, thus driving the ELBO and its variance to infinity. It would be helpful to include a plot of the decoder variances in the most realistic model, to see if this issue is relevant in modern hierarchical VAEs (and thus whether the analysis here provides a complete picture).\n\n- The conclusion of the analysis does not seem helpful: the bias is $O(1-\\rho^2)$ and the variance is $O(\\rho^2)$, so it is unclear from the bound whether there will be a $\\rho$ that decreases the overall MSE.\n\nMinor \n--------------\n\n- It is worth mentioning that there are several types of posterior collapse and not all of them are undesirable [2]: sometimes it is superfluous units rightfully pruned [3, 4]. This also implies that the number of active units is not a good measure of model quality; it is helpful to include reconstruction error in Section 5.1.\n\n- The observed phase transition of KLD connects to the fact that ELBO-trained VAE acts like a thresholding operator; see [2].\n\n- Why didn't Table 3 mention NVAE [5] and IAF-VAE [6], both of which have better BPD values? Seeing where those models are on the #parameters-BPD curve helps to put the results here in perspective.\n\nReferences\n----------\n\n[1]: Mattei and Frellsen, Leveraging the Exact Likelihood of Deep Latent Variable Models, in NeurIPS 18.\n\n[2]: Dai et al, The Usual Suspects? Reassessing Blame for VAE Posterior Collapse, in ICML 20.\n\n[3]: Lucas et al, Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse, in NeurIPS 19.\n\n[4]: Dai and Wipf, Diagnosing and Enhancing VAE Models, in ICLR 19.\n\n[5]: Vahdat and Kautz, NVAE: A Deep Hierarchical Variational Autoencoder, in NeurIPS 20.\n\n[6]: Kingma et al, Improved variational inference with inverse autoregressive flow, in ICLR 16.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fyaJSYIPXll", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Hermite variational auto-encoder which use Ornstein Uhlenbeck Semi-group to p(z_i|z_i+1) which i denotes the latent layer number. It has clear theoretical inspiration and had solid analysis on variance reduction. \n\nPros:\nQuality: The paper's generic theoretical motivation and analysis is with high quality. \nClarify: The paper's presentation is clear. \nOriginality: This paper provides a new perspective and used mathematically tools of Hermite expansion etc to inspire and proposed new method for variance reduction which prevent dying unity problem in Hierarchical VAE. Although motived by advanced tools, but the application of the method in vanilla version of hierarchical VAE (in term of implementation) seems very simple. Thus, the method looks easy to adopt. \n\nCons and questions:\nSignificance:\n1) Does the method work for vanilla VAE with only one level of z? It seems that it is only applicable to the hierarchical version as the operator is applied in  p(z_i|z_i+1) and if it is one level, it lost the point due to single Normal prior.  This may limit the application impact as VAE is much more widely adapted in different applications comparing to hierarchical VAE. \n2) I am not sure making unit not dying at all is desired (such as shown in last column of table one or Figure 3. Being Bayesian with the prior, there is a natural model selection behavior (implicit Occam's Razor) , thus, behavor such as the method with active unints (40,40,40,24) in table one may not be desired and rather a bit weird as only the last layer have dying units. Behavior such as VAE+KL (2,3, 11, 37) looks more natural to me as simpler model is needed in high hierarchy. \n3) Experiments only compared to Ladder VAE in number of dying unit but not in term of ELBO for performance. As LVAE is one of the most known work in this domain and also mentioned first in the related work section), this is weird. In LVAE paper, the MNIST performance is reported as -81-82 while in the paper it is about -85 which is significantly worse. Although there is a chance due to minior setting differences, I doubt the method's performance can match LVAE.  (Again with 2), I don't think that puring comparing the number of units without reporting performance makes sense). \n4) in term of performance of ELBO, most of the time, it does not match simple KL annealing either. \n5) there are more highly related work anaylsis the variance-bias trade off such as Tighter Variational Bounds are Not Necessarily Better are not discussed in the paper. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1dRlUdwNsmb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\nThis paper studies the training of deep hierarchical VAEs and focuses on the problem of posterior collapse. It is argued that reducing the variance of the gradient estimate may help to overcome posterior collapse. The authors focus on reducing the variance of the functions parameterizing the variational distribution of each layer using a layer-wise smoothing operator based on the Ornstein-Uhlenbeck semigroup (parameterized by a parameter $\\rho$). The operator requires additional Monte-Carlo samples. The authors provide an analytical analysis of bias and variance. Last they train multiple VAEs models, measure the posterior collapse and observe a phase transition behaviour depending on the parameter $\\rho$.\n\n2. a Strong Points\nThis paper introduced a theoretically grounded solution to the problem of posterior collapse. In particular, it is discussed that variance may be an issue. Great efforts were invested to study the behaviour of the Hermite VAE in theoretical terms and the authors provide analytical results on the Bias and Variance for this estimator. \n\n2. b Weak Points\n* Complexity \nIn the main text, it is written that \"*experiments show that 5 or 10 samples suffice*\". This is a major drawback for Hermite VAEs and the complexity of the algorithm is not discussed, nor it is studied empirically. Given 5 MC samples, I interpret that HVAE is 5 times more expensive than other approaches -- please clarify this point. \n* Empirical study of the variance \nThe problem of the variance is discussed in the paper but left apart in the experimental section. I would expect the authors to measure the variance (and/or SNR) of the gradients for the HVAE objective, the VAE, for advanced estimators such as STL and DReG. A study is required to corroborate the claim that reducing variance overcomes posterior collapse.\n* Experiments on posterior collapse\nI am surprised to see that none of the existing methods (KL warmup and freebits) allows overcoming posterior collapse (Figure 1). At least using the right amount of freebits should improve the results (the number of freebits is not reported). Furthermore, the authors should report the KL divergence in the benchmark experiment.\n* Experimental protocol \nI don't understand why VAE models trained in section 5 only have 2 layers whereas HVAE uses 4 layers: this is not a fair comparison. Furthermore, LVAE should be studied on the basis of posterior collapse -- not only in terms of likelihood.\n\n3. Recommendation\nUnfortunately, based on the current form of the paper, I recommend rejecting this paper.\n\n4. Recommendation Arguments \nDespite the good theoretical contributions, I do not find the experimental section to be strong enough to support the claims. In particular, the cost induced by the additional MC samples is not discussed and methods are hence not compared on the same basis. \n\n5. Questions to the Author\n- What is the complexity of HVAE? Do the VAE models use multiple MC samples as well?\n- Why using only 2 layers for the VAE models?\n- How are the freebits and KL-warmup applied in figure 1? \n\n6. Feedback \nYour work is very relevant and the theoretical insights are very interesting, this work would greatly benefit from an improved experimental section.\n\nIn the first page, two typos:\n-  you defined $q(z | x)$ and not $q(x, z)$\n- The KL divergence in equation 1 should depend on $q(z_i | z_{i-1})$ and $p(z_i | z_{i+1})$\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a smoothing procedure to avoid the problem of posterior collapse in VAEs. The method is interesting and novel, the experiments are well executed, and the authors answered satisfactorily to most of the reviewers' concerns. However, there is one remaining issue that would require additional discussion. As identified by Reviewer 1, the analysis in Section 3 is only valid when the number of layers is 2. Above that value, "it is possible to construct models where the ELBO has a reasonable value, but the smoothed objective behaves catastrophically". Thus, the scope of the analysis in Section 3 deserves further discussion. Given the large number of ICLR submissions, this paper unfortunately does not meet the acceptance bar. That said, I encourage the authors to address this point and resubmit the paper to another (or the same) venue.
Decision: Reject