AR-ELBO: Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE
Authors: Yuhta Takida
Wei-Hsiang Liao
Toshimitsu Uesaka
Shusuke Takahashi
Yuki Mitsufuji
Venue:  (2021)
Keywords: Generative model
variational autoencoders
posterior collapse
regularization

Abstract: Variational autoencoders (VAEs) often suffer from posterior collapse, which is a phenomenon that the learned latent space becomes uninformative. This is related to local optima introduced by a fixed hyperparameter resembling the data variance in the objective function. We suggest that this variance parameter regularizes the VAE and affects its smoothness, which is the magnitude of its gradient. An inappropriate choice of this parameter causes oversmoothness and leads to posterior collapse. This is shown theoretically by analysis on the linear approximated objective function and empirically in general cases. We propose AR-ELBO, which stands for adaptively regularized ELBO~(Evidence Lower BOund). It controls the strength of regularization by adapting the variance parameter, and thus avoids oversmoothing the model. Generation models trained by proposed objectives show improved Fréchet inception distance~(FID) of images generated from the MNIST and CelebA datasets.

Under review as a conference paper at ICLR 2021
AR-ELBO: PREVENTING POSTERIOR COLLAPSE INDUCED BY OVERSMOOTHING IN GAUSSIAN VAE
Anonymous authors Paper under double-blind review
ABSTRACT
Variational autoencoders (VAEs) often suffer from posterior collapse, which is a phenomenon that the learned latent space becomes uninformative. This is re- lated to local optima of the objective function that are often introduced by fixed hyperparameter resembling the data variance. We suggest that this variance pa- rameter regularizes the VAE and affects its smoothness, which is the magnitude of its gradient. An inappropriate choice of this parameter causes oversmoothness and leads to posterior collapse. This is shown theoretically by analysis on the linear approximated objective function and empirically in general cases. We pro- pose AR-ELBO, which stands for adaptively regularized ELBO (Evidence Lower BOund). It controls the strength of regularization by adapting the variance pa- rameter, and thus avoids oversmoothing the model. Generation models trained by proposed objectives show improved Fréchet inception distance (FID) of images generated from the MNIST and CelebA datasets.
INTRODUCTION
The variational autoencoder (VAE) framework (Kingma & Welling 2014 Higgins al.. 2017 Zhao et al. 2019) is a popular approach to achieve generative modeling in the field of machine learning. In this framework, a model that approximates the true posterior of observation data, is learned by joint training of encoder and decoder, which creates a stochastic mapping between the observation data and the learned deep latent space. The latent space is assumed to follow a prior distribution. The generation of new data sample can be done by sampling the latent space and passing the sample through the decoder. of the observation data follow a Gaussian distribution. This setup is also known as the Gaussian VAE. In this case, the variance of the decoder output usually modeled as an isotropic matrix of with scalar parameter o 2 O. Furthermore, in order to deal with the intractable log- log-likelihood of the true posterior, the evidence lower bound (ELBO) (Jordan et al. 1999) is adopted as the objective function instead.
common to assume that both the prior on the latent space and the posterior
While VAE-based generative models are usually considered to be more stable and easier to train than generative adversarial networks (Goodfellow et al. 2014). they often suffer from the problem of posterior collapse (Bowman et al.
2015 Sønderby et al. 2016 Alemi et al. 2017 Xu & Durrett 2019a Ma et al. 2019). in which the latent space has little information of the input data. The phenomenon is generally mentioned as "the posterior collapses the prior in the latent space" (Razavi et al., 2019a). Recently, several works have suggested that the variance parameter a in the ELBO is strongly related to posterior collapse. For example, Lucas et al. (2019) analyzed posterior collapse through the analysis on a linear VAE. It revealed that an inappropriate choice of o will introduce sub-optima local optima and cause posterior collapse. Moreover, Lueas et al. (2019) reveals that contrary to the popular belief, these local optima are not introduced by replacing the log-likelihood with the ELBO, but by excessively large o3. On the other hand, can be shown that fixing of to an excessively small value leads to rndul-regularizzaton of the decoder, which can cause overfitting. However, in most implementations of Gaussian VAE, the variance parameter o is a fixed constant regardless of the input data and is usually 1.0. In another work. Dai & Wipf 2019) proposed two-stage VAE and treated of as training parameter. Besides the inappropriate choice of the variance parameter, posterior collapse can also induced by other causes. For example, Dai et al. (2020) found that, small nonlinear perturbation introduced in
2018
He et al. 2019 Razavi et al.

---

Under review as a conference paper at ICLR 2021
the network architecture can also result into extra sub-optimal local minima. However, in this work we will keep our focus on the variance parameter.
We suggest that of affects the strength of regulation call the expected gradient magnitude the smoothness throughout this paper. The smaller the gradient magnitude, the smoother the model. In partieular, the model, which is the smoothness evaluated within the neighborhood of the encoded latent variable of the observation data. Thus, we begin with the following hypothesis:
the gradient magnitude of the decoder. We
would like focus on the local smoothness of
Main Hypothesis. The value of o controls the regularization strength of the smoothness of the decoder. Therefore, an excessively large o causes oversmoothness, which results in posterior col- lapse.
Following the hypothesis, the estimation of o² should be related to properties of the approximated posterior of the latent space, such its local smoothness. We will start with analyzing how of regularizes the local smoothness of the stochastic decoder and then propose new objective functions that inherently determine of via maximum likelihood estimation (MLE). This proposed objective function is named AR-ELBO (Adaptively Regularized ELBO), which controls the regularization strength via o². Furthermore, several variations are derived for different parameterizations of vari- ance parameters.
Our main contributions are listed as follows:
1. We show that our main hypothesis holds for linear approximated ELBO and empirically holds in the general case in Section 3 This also suggests that the variance parameter o should be estimated from properties of the approximated posterior instead of being treated as a hyperparameter.
2. We propose the AR-ELBO, an ELBO-based objective function that adaptively regularizes the smoothness of the decoder by MLE of the variance parameter o² in Section 4 Vari- ations of AR-ELBO for several variance parameterizations of posterior distributions are also derived. AR-ELBO prevents the model from the posterior collapse induced by over- smoothing and improves the quality of generation, which shown in Section S
The organization of this paper is as follows. In Section 2 we propose mathematical definition of posterior collapse in the form of mutual information. This also includes the conventional definition: "the posterior collapses to the prior in the latent space". In Section 3] the theoretical analysis and empirical support of the main hypothesis are given. We perform an analysis showing that o affects the smoothness of the decoder via variance parameters of the latent space learned by the encoder. In Section 4 we propose new AR-ELBO objective functions for various variance parameterizations of posterior distributions, which can relieve the decoder from being oversmoothed in the training and prevent posterior collapse. These objective functions no longer include any hyperparameters and can adaptively estimate the variance parameter of from the observation data. It should be noted that if we adaptively determine o² with the proposed AR-ELBO, the strength of regularization of the decoder smoothness will gradually decrease as training progresses. In Section 5 we conduct an experiment the MNIST and CelebA datasets, which shows that utilizing the proposed AR-ELBO with the standard Gaussian VAE can be competitive with many other variations of VAE models in most situations.
Throughout this paper, we use a, a and A for a scalar, a column vector and a matrix, and In and log denote the natural logarithm and common logarithm, respectively. Our code is available from the following URI! []]
2 POSTERIOR COLLAPSE IN GAUSSIAN VAE
We begin with the standard formulation of the Gaussian VAE, which is the foundation of our re- search. definition of posterior collapse is proposed by using mutual information (MI).
URR hidden due to blind review.
2

---

Under review as a conference paper at ICLR 2021
2.1 GAUSSIAN VAE
Consider data space X' C Rd and a sample set (x:)i=1 C X, where x; ~ (x). The empirical
distribution Pdana (x) X' can be evaluated by (x)ta()) N En= 5(x xn), where 8(-) denotes the Dirac delta function. In the standard VAE framework, latent space 2 C Rd is learned and the sampled latent variables z E 2 are used to generate data samples x' E X. Let go (z|x) x) and pa 2)) denote the stochastic encoder and decoder, respectively. Trainable parameters of the networks are denoted as 6 and O. The decoder generates data samples by px(x) == Ep(z) [ps (x|z)], where p(z) is the prior distribution on 2. The encoder and decoder are jointly trained by minimizing the following objective function:
neural
L - pp) [x:]]] + (x) DKL. (z)xx || po(x)))) + E (x) (In Pdataa (x)]
[InPaata
= DRL. (Pama(x) ] (x)) + Exsz (x) DKL (z)()))) po(z)x)).
(1)
This objective function was derived in Zhao et al. (2019) which represents everything in the form of Kullback- divergence, and is equivalent to ELBO maximization up an additive constant.
In the context of the Gaussian VAE, the encoder and decoder are assumed to satisfy
(+(2(x) (Z (x),, diag(o (x)) and
Po [x|2) 2)) N (x())xo(z) I
The prior p(z) is also assumed to be the Gaussian distribution as p(z) = /V (z Substituting (2)
into [ while omitting terms independent of 0 and leads to the following objective:
Jaz (0, 6) E (x)
(xx
E 20
1
(z) + D ll p(z))
which can be interpreted as the sum of the expected values of the reconstruction loss and the reg- ularization term. In the case of Gaussian prior and posterior, the regularization term is equal to Dd. (x)x(x) + (x.;(x)))))))) - log o2 (x) -).
2.2 POSTERIOR COLLAPSE
Posterior collapse is major problem, where the encoder learns map inputs to the latent space while ignoring the data distribution. In this phenomenon, the MI between input data and recon- structed data through the encoder-decoder path reduced because the latent space has less informa- tion about the data distribution. Here, we suggest the following definition of posterior collapse.
Definition 1. Posterior collapse is defined as the MI xxx:x') becoming nearly zero, where x' == μn (z) with 96 (z)x).
In many works (Bowman et al. 2015 Sønderby al. 2016 Alemi et al. 2017 He et al. 2019
Razavi et al. 2019a). the phenomenon denoted as posterior collapse has been mathematically rep- resented E (x) D || p(z)) - 0, which we hereafter refer as KL collapse (Xu & Durrett 2018). However, posterior collapse is not always caused by the diminished KL divergence, ) of 0 can occur. The proposed definition of posterior collapse includes KL collapse from the following theorem, which is proven in Appendix
i.e., posterior collapse with (x) (prl(qa (z) x) || p(z))
Theorem 2. I(x; x') > 0 as DxL
p(z)) - holds for any pe (x|z).
In Appendix E
demonstrated that posterior collapse can happen even if the KL divergence is nonzero when the posterior variance is fixed in Z.
3 VARIANCE PARAMETER of AND THE LOCAL SMOOTHNESS
In this section, we provide mathematical and empirical support of the main hypothesis. Throughout this section, we adopt the following parameterization for the encoder for simplicity: (z|x) = (2 (z2)4s (x),, II),, where the variance is parameterized as an isotropic matrix unlike the conventional VAE. A similar analysis showing that the choice of o² affects the convergence point of o². which is the variance parameter of the latent space. Then, show that of acts the weight of the gradient penalty, which is
the conventional VAE can be found in Appendix B It begins with
3

---

Under review as a conference paper at ICLR 2021
implicitly included in B3. This supports the main hypothesis that the over-regulation imposed by large of via of causes the oversmoothness of the decoder and leads to posterior collapse. is also empirically supported by observing the tendencies of the convergence point of o the smoothness and the MI I(x, x'). Ultimately, these items of evidence motivated us to develop a method that adapts of to prevent oversmoothing of the decoder.
3.1 REGULARIZATION EFFECT OF of IN LINEAR APPROXIMATED ELBO
The effect of of the convergence point of the variance parameter of can be observed from Iwo extreme cases, of + 0+ and of + co. In the first case, Jog reduces to (ana(x) (x/xxxl [=)2)]13), and it becomes zero only if o² = 0. the second case,
Jog reduces to (x) Pru(1 (z|x) || p(z)), and of becomes at the minimum point, from DKL(go.03 (z|x)) || p(z)) = 1x (o? logo? - + (x) This shows that small of makes of converge to a value around 0, while large of makes of converge value around
If -f is sufficiently small, as the training progresses to a certain extent, the perturbed decoding process (z + Eg) around z (x) with E, (Ne= I. can be approximated as linear function. The ELBO can be approximated as follows by using the linear approximation of μ4 (-) and omitting terms independent of 0 and Φ:
(0, o. 02)
1
E (xm [|xx ( (x)((s(x))]1(++ + l (x))] (4)
=
20²
In the approximation above, || ||F is the Frobenius norm and o is treated as a function parameter.
The derivation of the above approximation can be found in Appendix B Equation 4 decomposes the objective function into three terms: a reconstruction error term, gradient penalty term and L2 see from (4p, of regularizes the smoothness of the decoder by
regularization term. As penalizing its gradient norm in training. Although the linear approximation above is derived for the simplified VAE parameterization, we also provide the linear approximation of the ELBO for the standard VAE parameterization (2) in Appendix B where the second term in (4) becomes a weighted gradient penalty.
Summarizing the above observations shows that o² affects the smoothness via o². while o² di- rectly regularizes the smoothness. This means that if o is excessively large, it will cause over- regularization of the decoder and suppress I(zzx')( collapse. To avoid such over-regularization, of and o should be determined appropriately. In ad- dition, an experiment shows that posterior collapse can be triggered by directly manipulating 03. discussed in Appendix
Z(x.x') x')), which finally leads to posterior
3.2 EMPIRICAL STUDY ON SMOOTHNESS OF DECODER IN THE GENERAL CASE
Section 3.1 shows the impact of o on the regularization of the decoder smoothness through the linear approximated objective function. To support the main hypothesis in the general case, an ex- periment on the MNIST dataset (LeCun et al. 1998) is conducted. Several criteria are accessed to provide evidence for the regularization effect of of the decoder smoothness and its conse- quential effect on MI Z(x, To confirm that of affects the smoothness via o², we conduct the experiment for two cases: stochastic encoding and deterministic encoding. While the stochastic encoder 4g.0? (z|x) is used in the former case, a VAF equipped with a deterministic encoder, i.e., o3 is fixed to zero during the training, is investigated in the latter case. Observing the difference between the two cases provides empirical support for Section 3.1 To investigate the relation be- tween of and the smoothness of the decoder clearly, common generalization techniques such as batch normalization (Toffe & Szegedy 2015 Santurkar al. 2018) and weight decay are not used.
Criteria We used several criteria to observe the impact of a² 2 in the experiment, such as the recon- struction error (MSE), KI. divergence value (x / (1
|| p(z)) and the final converged value o3. In addition, we also estimate the local smoothness of the decoder and the MI between x and x', denoted as Σ(x, x'). To access the local smoothness, the expected local smoothness (ELS) introduced, which is the lower bound of the Lipschitz constant of the decoder. Consider sam- ple that is decoded with perturbation oo (x)((x) + Ez), where the perturbation follows zero-mean
4

---

Under review as a conference paper at ICLR 2021
Table I: Evaluation of various criteria for different o²: the expected value of ||x' x12 (MSE), KL divergence, the converged value of o the upper bound of the MI I(x'; z), the expected gap (perturbation variance s are set to 10-2 and 10-3) and expected local smoothness (ELS).
Stochastic encoding
Deterministic encodding
log o
Expected RAD
Expected gap
MSE
KL
M1
2
ELS
MSE
MI
2
10
ELS
52.74
6.35e
3.97e-4
25.3
7.43++2
0.1
18:03
9.39
9.56c-2
9.7
1.05
0,108
6.76
5.69
14.7
69. 1
22.5
6.82c+2
0.2
15.15
10.93
6.48o-2
12.5
1.30
0.135
8.34
5.38
17.9
20.7
6.37c+2
0.3
13.08
12.54
4.36e-2
16.0
1.51
0.157
9.72
5.37
21.4
58.1
17.9
5,78c+2
0.4
11.38
14.13
3.0le:2
20.6
1.77
0,184
1.14e+1
5,31
25.8
58.2
15.5
5.40e+2
0.5
10.18
15.30
2.14e-2
26.3
1.99
0.208
1.28ce
5.26
30.6
53.1
12.9
4.74c+2
-0.6
9.16
16.72
1.35c-2
33.2
2.16
0.227
1 1.40+1
5.14
38.9
48.9
11.8
4,42c+2
8.25
1.1le-2
2.31
1.50c=
46.1
3.98a+2
-0.8
7.72
19.27
8.21c-3
52.9
2,40
0.254
I 1.56c+1
5.06
58.0
43.2
9.15
3.71e42
0.9
7.13
20.55
5.976-3
64.9
2,43
0,257
1,58c+1
4.98
71.9
39.2
7.83
3.29c42
-1.0
6.70
21.75
4.45e:
82.3
2.57
0.272
1.67e+1
5.01
89.1
35.5
6.61
2.89c+2
Guassian distribution with variance s². 6, We define the expected gap A (s))
~ (O, I).
Let and cl be i.i.d. random variables.
A with A Ex, e = ue (Ho + + ) - pa + é
62, 6g)
(5)
As s² decereases, the ratio A (s (2s converges and becomes an indicator of Epan. (oo)))) which regularized by o² 4 Therefore, we can now define the ELS as
(x (Ms
Further details of the ELS are described in Appendix C As reference, estimate the upper bound I x'), which is [(x;; ; z). by Monte Carlo estimation.
Results Table summarizes the results for different o². In the stochastic encoding case, larger of consistently leads to a larger o². This results in a smaller expected gap, a smaller ELS and lower upper bound of MI. This supports the main hypothesis that larger of makes the decoder smoother. In the case of o2 1.0, all the criteria except MSE become nearly zero, where KI. collapse and posterior collapse both oceur due the over-regularizztion of the smoothness of the latent space. On the other hand, in the deterministic encoding case, the ELS increases with o² This is because of does not directly regularize the decoder via the gradient penalty as in 44p. As result, the MI upper bound does not shrink to zero even if of = 1.0, where posterior collapse occurs in the case of stochastic encoding. The difference in the results between the two cases clearly suggests that large of triggers the oversmoothness via of. which is consistent with the discussion in Section 3.1 These results provide empirical support of the main hypothesis as well as the discussion Section 3.1 Further details and examples of images are shown in Appendix D
3.3 DIFFICULTY OF CHOOSING APPROPRIATE of
According to previous sections, a large of will cause oversmoothness. Therefore, we consider the case of fixing of a sufficiently small value to avoid the problem. We arrive the following theorem, whose proof can be found in Appendix
Theorem 3. Consider the global optimum of 00, w.r. given of If o + 0, then o² 0.
In Theorem [op d, ) is optimized on the basis of the true data distribution instead of the empirical data distribution. According to the theorem, of converges zero as of approaches zero, which leads to zero gradient penalty for the decoder as the VAE training progresses. practice we have no access to Pdata (x), but we have access to the empirical distribution pdata (x). Theorem 3 satisfied even when Pdnta (x) is replaced with Paaaa (x). In this case, where o chosen be small, the optimization process of Jog will fit po. (x) to the empirical distribution Paata (x). which usually results in overfitting.
5

---

Under review as a conference paper at ICLR 2021
As shown above, choosing appropriate o that avoids both oversmoothness and overfitting is nontrivial. Moreover, it is likely that o² should be adapted depending on the status of training. Therefore, it is intuitive to adapt o² instead of fixing it, which will be described in the next section.
4 ADAPTIVELY REGULARIZED ELBO
A modified ELBO-based objective function is proposed, which can be interpreted as an implicit update scheme that simultaneously updates of and the rest of the parameters. We also derive corre- sponding objective functions for models with different variance parameterizations.
4.1 ELBO WITH ADAPTIVE of
In this subsection, we newly optimize the VAE objective function (I ing a which is usually fixed in existing implementations. Following the process of establishing (3) but keeping the terms related to oe we arrive at:
w.r.t. all the parameters includ-
Fan (x)
E (e)x) 20
||x (2) + DEL (1s(2)x) p(2))
+
de
In In (7)
2
From the partial derivative of J w.r.t of. the MLE of of denoted as o2, can be evaluated with the other parameters fixed. Then, the ordinary network parameters and can be updated by optimizing (7) with the variance of fixed. This combination of MLE and the alternative update between (0,0) and o² guarantees that (i) if 0 and & are fixed, then there exists 02 such that [(, Φ, o S [(0, Φ. o.) and (ii) for the o² obtained in the previous step, there exist & and o such that [(, o, 6.) I [(, o, 2...). In this respect, the convergence of the optimization is assured and the parameter 02 is kept as the MLE during the whole training stage. This inspired us to develop weight scheduling scheme for d² leading to modified ELBO-based objective function. Consider the trainable network parameters (0, Φ) and the variance parameter of The update of the objective [(, o, of)) is divided as
20+11 E (x) F. Begses (z|x) ||x Macc (2)
(8a)
dg
g(++1) ((+1) arg min I 2,20113 (), 0.0
(8b)
where t is the iteration index. The step updating () is the same as that in the standard VAE; the step updating o2 in (Ba) can be interpreted determining an appropriate balance between the reconstruction error and the KI. term in Joz (0, Φ). As the learning progresses, the parameter a² will decrease along with the MSE Exin((x) Eyy x ar (z) [[2], which is consistent with the discussion in Dai & Wipf (2019).
Proposed objective function (AR-ELBO) The update scheme above can be further simplified by substituting (8a) into (7). which converts J(0, 0,0)) into
d.p In E. [|x no (z) (x) DKL (z)() p(z))
TAR Φ) 2
where all constant terms w.r.t. the parameters are omitted. Optimizing 9 also makes o² remain as the MLE during the VAE training. Moreover, ( is equivalent to the standard Gaussian VAE plus weight balancing with (8b). This relieves the VAE from the problem of imbalance between the KL divergence term and the reconstruction loss. Also, as stated by Theorem decreasing of also decreases 03. This gradually relieves the regularization of the ELS (6. which can be observed from 4.. However, this eventually diminishes the gradient penalty; therefore, we suggest using early- stopping and learning rate scheduling to deal with this situation, which can give the decoder both appropriate smoothness and generalization capability.
4.2 OBJECTIVES FOR VARIOUS PARAMETERIZATTIONS
In the standard VAE given by (2), the variance of the decoded distribution on X. denoted as Egs is modeled as an identity matrix, I.c., Σ,, = I²I.. In this case, o2 is simply a scalar value and the re- construction objective is the same as conventional MSE and is minimized as in (9) However, out of

---

Under review as a conference paper at ICLR 2021
Table 2: Parameterizations of posterior variance in X' and corresponding reconstruction objectives.
Variance model (Ex)
Reconstruction objective (Tro((0, Φ, Ex))
(Iso-1)
du In (un yo (a/bx) [|| M4
2
(Iso-D)
pan (a x) [In ||x Me (z)
(Diag-I)
diag(o?))
==1 In LnEE B (x) (xpo)) - M8, (2))
(Diag-D)
diag(o
dx == Bxan(() B Ryo x) [I((r (o,;
curiosity, we would like to explore three other variance parameterizations in addition to and de- rive corresponding reconstruction objectives for these cases, in which the reconstruction objectives are no longer equal to MSE.
In fact, the variance Eg can not only be parameterized by an isotropic/diagonal matrix but also be chosen to be independent or dependent on 2. We denote these in Table [2 as Iso-I (Isotropic- Diag-I Dbiagonal-Independee
Independent), Iso-D
and Diag-D (Diagonal- parameterizations, the corresponding objectives may be summarized
Dependent). The first case, Iso-I, corresponds to the standard variance model Σ, I For these
[A,(H, o, Ex) = Jnc ,0,40 Φ, Ex) + Ersn DKL (g(z) x) || p(z))
(10a)
Jrc 05 Ex) 2 Bxou()) (x) (o(xlz)) |trace| (E= (0-'xx-ool po(z)) mo(z)) + In [E
(10b)
By evaluating the partial derivative of Tre w.r.t. E,, i.e., using its MLE, the reconstruction loss Tree corresponding to each case can be derived. All the derivations can be found in Appendix G The final reconstruction objectives with different parameterizations interesting note that the reconstruction error for each dimension in the data space has to be calculated separately for Diag-D; meanwhile, only MSE for the whole minibatch is needed in Iso-I.
Eg are listed in Table 2
Considering the optimization stability in practical situations, we suggest adding a small constant, e.g., 10-6, before taking the logarithm except for in the case of Iso-I.
Although the proposed objective functions are capable of determining Eg appropriately, it should be noted that a gap still exists between the prior p(z) and the aggregated posterior (z( obtained by the proposed methods. The cause can be observed from the reformulated [T]
(see Appendix HD:
[ DEL (x) po())) + DEL (Pamu(x) (z(x) || (x(2))(()))) + DKL (go(z) |l p(z))
(11)
The first two terms in [T] can eventually become dominant in the VAE training. As a consequence, generation through sampling latent variables from the prior can cause off-distribution samples to be generated. To overcome this prior-posterion mismatch, at least two approaches can be adopted:
(i) conduct another posterior estimation after the ordinary VAE training (van den Oord al. 2017
Razavi et al. 2019b Dai & Wipf 2019 Ghosh et al. 2000 Morrow & Chiu 2020) or (ii) add
another regularizing term the objective function (Makhzaní al. 2013 Tolstikhin et 2018
Zhao et al. 2019). The former approach is adopted in our work since it is effective (Ghosh et al.
2020) and is applicable any VAE variation.
To summarize, the proposed AR-ELBO regularizes the Gaussian VAE with appropriate weighting for the gradient penalty without the need for an extra hyperparameter. Moreover, the remaining mismatch between the prior and posterior is mitigated by an extra pass of posterior estimation. A detailed discussion comparing the proposed objective function with previous works can be found in Appendix
EXPERIMENTS
We compare the proposed methods with the following models: VAE, RAE (Ghosh et al. 2020 WAE-MMD (Tolstikhin et al., (018)) and plain autoencoder (AE). The quality generated im- ages is evaluated using Fréchet Inception Distance (FID) (Heusel al. 2017) the MNIST and
7

---

Under review as a conference paper at ICLR 2021
Table 3: Numerical evaluation on MNIST and CelebA. The MSE of sample reconstruction evalu- ated on the test set. The quality of generated samples is measured by FID for three cases: sampling the latent variables from the prior and from the estimated posterior by 2nd VAE and GMM.
MNIST
CelebA
FID
FID
MSE
Prior
VAE
GMM-10
Prior
2AL
GMM-10
GMM-100
VAE ( 1.0)
20.25
55.85
182.64
58.96
121.91
55.46
139,32
54.66
53.94
WAB-MMD
4.34
22.76
15.00
13,70
62.66
52.89
51.32
43.57
41.88
AE
4.31
20.66
13.20
61.44
62.34
46.47
43.41
RAE
4.28
18.54
13.68
61,49
57.26
46.50
43.89
RAE-GP
4,30
18.89
13.71
61.48
54.54
43.63
41.10
Is0-I my trainable !
4,28
20.93
14.83-
13,39
61.42
63.21
63.12
51,40
49.61
Diug. w/ trainahle Σ
5.70
26.08
18:44
15.72
62.65
59.66
54,75
45,86
43,50
Iso-D w/ trainable X
4.45
27.40
13.67
195.58
53.13
50.09
47.31
Diug-D D w/ trainable >>1
5.33
146.43
26.02
27.90
85,33"
354,20"
217.06"
188,
188.05
Ours [15o-] w/ AR-ELBO)
4.40
22.78
15.77
12.31
62.02
82.20
52.48
42,82
41.03
Ours (Diag-1 w/ AR-ELBO)
5.35
24.15
17.18
13.38
63.51
87.44
53,60
45.85
42.83
Ours (Iso-D w/ AR-ELBO)
4.31
22.94
17:57
12.89
61.38
78.24
49,97
43.27
40.39
Ours (Diag-D w/ AR-ELBO)
6.80
16.64
10.49
10.05
70.75
64.44
55,30
46.63
45,27
Σg is learned as trainable parameter like in Dai & Wipf 2019))
However, the work does not include Diag-I,
Iso-D and Diag-D parameterizations.
VAE is the second-stage VAE proposed in Dai & Wipf 2019) after the main VAE.
The training of Diag-D with a trainable Σg does not converge feasible local optima. Moreover, the trend of MSE diverges with that of the loss function. Therefore, the result which achieves the hest MSE on the test is reported here. On the other hand, Diag-D with the proposed AR-EL BOO does not suffer from this issue.
CelebA (Liu et al. 2015) datasets with the default train/test split. Regarding the mismatch, three approaches are tested on all the models. The first approach is the conventional case, which samples the latent variables from the prior. The other two approaches are applied after the ordinary training. The second approach forms an aggregated posterior qa(2) (2) by a second-stage VAE (Dai & Wipf (201)). The third approach uses Gaussian mixture model (GMM) with 10 to 100 components (Ghosh et al. (020) to fit the posterior. The baseline is the standard VAE, and two methods of choosing of are tested: (i) of fixed 1.0 as in general implementations: and (ii) of learned with (7) by an optimizer as an usual trainable parameter (Dai & Wipi 2019). In RAE, its objective function is the sum of the reconstruction error, the regularization of the decoder and the Ta regularization of the latent space. In RAE-GP, gradient penalty is used as the regularization of the decoder. The objective function of RAE-GP is equivalent parameters are determined manually. For WAE-MMD, inverse multi-quadratic kernels with seven scales are used as proposed by Tolstikhin et al. (0118). Note that both WAE and RAE have hy- perparameters in their objective functions. In contrast, the other methods, as well the proposed objective functions, include no hyperparameters. However, regarding the major factor that affects the smoothness of the decoder, the proposed methods and VAE control smoothness by regularizing terms in their objective functions, while WAE and AE rely only on the network architecture and generalization techniques.
except for that the weighting
The latent space dimensions for MNIST and CelebA were set to dN = 16 and 64, respectively, consistent with Ghosh et al. (2020). A common network architecture, which is adopted from Chen itt al. (2016) and described in Appendix [ is used for all models. In Table we report the evaluation result of each method as: (i) the MSE of the reconstructed test data and (ii) the FID of generated images. The proposed methods (Diag-L, Iso-D and Diag-D) D) except for Iso-I do not necessarily achieve the best MSE, since the MSE is no longer the reconstruction loss for them. In the case of sampling E 2 from the prior, a low FID is achieved by WAE due to the relatively strong regularization of the aggregated posterior with MMD. The images generated by the proposed method achieve the best FID score on MNIST, and is competitive on CelebA. It should be noted that the learned o2 values on MNIST and CelebA from Iso-I are 0.0056 and 0.0050, respectively, which are much smaller than 1.0. Examples of reconstructed and generated images shown in Appendix K
As shown in Table 3 different parameterizations of variances can affect the FID greatly. In order to clearly observe the advantage of estimating Σg by MLE rather than estimating it as an usual trainable parameter, we examined the two approaches on all the four parameterizations (Iso-I, Iso- D, D and Diag-D): (i) solve (10b) using MILE
(9) (BR: (ii) simply treat Σ,
8

---

Under review as a conference paper at ICLR 2021
Table Evaluation of FID scores of interpolated images on MNIST and CelebA. The interpolation ratio for each image pair is designated as (i) the mid-point; and (ii) a random-poin between the two.
MNIST
ColebA
Mid-pponil
Random point
Mid-ppoinl
Randomn point
VAE (O) 1.0) WAF-MMD
62.18
63.86
57.87
55.54
17.27
12.41
41.93
38,80
AE
18.49
12.81
50.35
45.01
RAE GP
17.98
12.96
45,22
40.58
Iso- w/ trainalle Σg
15.66
12.73
52.01
48.83
Diag-T trainable E,
17.34
14.77
44.22
41.23
Iso-D w/ trainable Ex
17.03
13.33
48,51
43.97
Diag-D trainable
Ours (Iso-] w/ AR-ELBO)
14.77
11.27
42.59
39.19
Ours (Diag-1 w/ AR-ELBO)
17.24
13.08
45.55
41.92
Ours (Iso: D AR-ARREBEE ELBO)
15.30
12.01
42.49
38.93
Ours (Diag-D w/ AR-ELBO)
11.96
8.68
46.86
44.15
trainable parameter Dai & Wipf 2019). The comparative result can be obtained from the bottom eight lines of Table 3 shows that applying AR-ELBO improves FID scores in most of the cases.
Furthermore, in order to examine the feasibility these learned latent spaces, we also evaluated the FID scores of the images generated by interpolating with all the models above. The proposed method still shows the best performance in terms of FID on MNIST and competitive on CelebA. This result suggests the feasibility of proposed method on downstream tasks. The detail and image samples can be found in Appendix
latent variables with ratios between [0, 1]
CONCLUSION
We analyzed the posterior collapse phenomenon on the Gaussian VAE and investigated how strongly the variance parameter impacts the local smoothness of the decoder. The relation between the variance parameter and the local smoothness examined both theoretically and empirically. We proposed optimization schemes to regulate the local smoothness appropriately, which leads to the prevention posterior collapse due to oversmoothness. The proposed AR-ELBO implicitly opti- mizes the variance parameter avoid over-lrgulaiizing of the smoothness. In addition, we proposed several parameterizations (Iso-D, Diag-I, Diag-D) of posterior variances, which are the extensions of the conventional VAE (Iso-I). The corresponding AR-ELBOS for these parameterizations also derived. Our experiments show that the Gaussian VAE equipped with the proposed objective functions competitive with other state-of-the-arr models terms of FID for both generated and interpolated images. Moreover, the proposed method remains stable in the most complicated pa- rameterization (Diag-D). In this work, the prior prior-posterin posterior mismatch was covered by extra posterior estimation methods; however, would like to seek thorough solution for this in the future.
REFERENCES
Alexander A Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken ELBO. arXiv preprint 1711 1.00644, 2017.
Samuel R Bowman, Luke Vilmis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from a continuous space. arXiv preprint arXiv: 1rX0v:1571.06349, 2015.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 2172-2180, 2016.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In Proc. International Conference on Learning Representation (ICLR), 2019.
Bin Dai, Ziyu Wang, and David Wipf. The usual suspects? Reassessing blame for VAE posterior collapse. In Proc. International Conference Machine Learning (ICMI., 2020.

---

Under review as a conference paper at ICLR 2021
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: simple approach to mitigating KL vanishing. arXiv preprint arXiv:1903.10145, 2019.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Berhhard Schölkopf. From variational to deterministic autoencoders. In Proc. International Conference Learning Repre- sentation (ICLR), 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of Wasserstein GANs. Proc. Advances in Neural Information Processing Sys- tems (NeurIPS), pp. 5767-57777 2017.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- King-Kirkpattick. Lagging inference networks and posterior collapse in variational autoencoders. In Proc. International Conference on Learning Reprexentation (ICLR), 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
GANs trained by a two time-scale update rule converge local Nash equilibrium. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 6626-6637, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAB: Learning basic visual concepts with constrained variational framework. In Proc. International Conference Learning Representation (ICLR), 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. International Conference on Machine Learning (ICML), 2015.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Interna- tional Conference on Learning Representation (ICLR), 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes. In Proc. International Conference on Learning Representation (ICLR), 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, (111)))2228222444 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3730-3738, 2015.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the ELBO! A linear VAE perspective on posterior collapse. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 9403-9413, 2019.
Xuezhe Ma, Chunting Zhou, and Eduard Hovy. MAE: Mutual posterior-divergence regulariza- tion for variational autocncoders. In Proc. International Conference on Learning Representation (ICLR), 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. In Proc. International Conference on Learning Representation (ICLR) Workshop, 2015.
10

---

Under review as a conference paper at ICLR 2021
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. Proc. International Conference on Learning Representation (ICLR), 2018.
Rogan Morrow and Wei-Chen Chiu. Variational autoencoders with normalizing flow decoders. arXiv preprint wwXiv:200.005617, 2020.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta- VAEs. In Proc. International Conference on Learning Representation (ICLR), 2019a.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VAE-2. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 14866- 14876, 2019b.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Proc. Advances in Neural Information Processing Systems (NeurIPS). pp. 2483-2493, 2018.
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controllable variational autoencoder. Proc. International Conference on Machine Learning (ICML), 2020.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Lad- der variational autoencoders. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 3738-7738777 3346,, 2016.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. In Proc. International Conference on Learning Representation (ICLR), pp. 5885-5892, 2018.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Proc. Advances Neural Information Processing Systems (NeurIPS), pp. 6306-6315, 2017.
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Info Balancing learning and inference in variational autoencoders. In Proc. AAAI Conference Artificial Intelligence (AAAI), pp. 5885- 5892, 2019.
A PROOF OF THEOREM
Let x the imput sample. We denote its corresponding latent space vector as z and the reconstructed sample x'. We have the following relation:
I(x: 2) 2 xx;x'',,
(12)
which can be proved similarly to the proof of Lemma in Appendix can be evaluated by using the definition of the MI
On the other hand, I(x; z)
I(x; z) = DKL(Pama (z) (z(xx) || Psaa (z)
Expu(t) E (x)x)) [ln go (z|x) In (z))
Ex DKLlqs (z|x) |l p(z)) Dku(qo(z) ll p(z))
(13)
I Exan(te) DRilqs p(z)),
(14)
where I(x;2), z), (z) p(z)) and (zI(go(Z) p(z)) are all non-negative. Inequalities 12
and (1] lead the proof.
11

---

Under review as a conference paper at ICLR 2021
B LINEAR APPROXIMATION OF THE ELBO-BASED OBJECTIVE Tog
We start with parameterizing the encoder while following the assumption in ( Given sufficiently small perturbation with p(e,) (z(0, (x)))) the linear approximation of (a(()) at (x) can be represented
no ( Ex) = no (x)) Jus (x)((
(15)
where JoH (M (x)) represents the Jacobian matrix of pe at mx(x). Substituting leads to
[5] into
E [x](=/x)]]x (
= =[Ntaloodiac(o [1xx (Mo(()) + Ino (x))
= =||x po (x) + (NCelodding( [6] Jus (x))()) Juo (x)(E))
+ diielo.di [(x (o(pa(x))) Jus (Ha((
(16)
where the last term is under the assumption that the perturbation is sufficiently small. The expectation in the second right-hand-side term can be evaluated as
Ex (id [c] Jus (x)( Juy (pu(())
trace [EN [ExE]] Juo (Ho(()) Jus (pu(()))) )
trace (diag(o (M)(X))
Jus (Hoo()))
de d.j
(2)))(2)
(x)
(17)
Ozj
===x((x)
which can be interpreted as the gradient penalty for the decoder weighted by (x)). By substituting the above result into (3), its linear approximation can be obtained as
Toz 0)
(x) ||x - us (oo
20
2 2 (x)
+ 20 [x)
(18)
Ozy
(x=))(())
In the case the simplified parameterization described Section 3 the second right-hand-side In the simplified case, the perturbation
term in (18) can be further reduced to follows multivariate i.i.d. Gaussian distribution, E₂ ~ O(z) Under this assumption, we have
Ep(c.) Je (HQ((x)) Jus (Mo (x))xx = Ep(c.) c. Σ (x)(()) Es
ds Auui(x) (p
20 Ai
(19)
where the ith eigenvalue of Jus (Ho(()) ( T. which a symmetrical positive definite
matrix, and the corresponding eigenvectors are (u;(x)) Following the simplified assumption, the second right-hand-side term in (16) now becomes Ex(200000) [c] (Ho())) Jno (x) Combining
19 and the fact that Ed= A
trace (no
12

---

Under review as a conference paper at ICLR 2021
(x)
finally obtain the following linear approximation for the simplified pa-
rameterization:
20( [c] JHH (H)(()) Jno (Ho((())
[xv(loo0o1) |trace (Juo (m)( (uo (Ho
d. -
@po,(z))
Ozy
==)a(x)
(20)
C EXPECTED LOCAL SMOOTHNESS OF DECODER
Here, deseribe the relation between the expected local smoothness (xs (o and the expected gap A (s First, consider the relation
A (x, E2, e')) || no (x)(x) + 6x) ( < = (x) e 6,112
(21)
with the perturbation 6, following the Gaussian CittsribbutiooNNee_...... 1) Applying the expectation operator to (1 leads to
X, E E¹r) = Eplen.e)) e
(22)
I Bptente)) (x( c2)² Ep(en)t) [||E= - c
(23)
=: (Mo s s
(24)
where N I. c.
-(c,-6,1,, 2s² and (H =
Ep))..e)) (Ko(u)(x), es, Note that (3). we assume that (z)x(x))) is independent of e, and éL. Consider the case that the variance s is sufficiently small to approximate (z(2) linearly around (x(x), which is perturbed with variance s². In such case, (x) C2, c() is
independent e, and e, which fits the assumption in 23 Under this local linearity assumption, K3 (x)) is bounded
(x) 22) S KT
(25)
where Kg denotes the Lipschitz constant of the decoder.
Following the assumption, ()))((x), can be formulated by invoking 15 as
K3 (x) Epic. <x) Mo (x) + +ee) ( + 1
E Ep(es,) llle.
Palle e; et) e%)) no (x)) Juo (x)(())((E EG)]
2d
trace (Eples,;))(Es (E= - T (Mg)()) Jup (x))
2d
trace (Juo (Mx(x)) T JuuO (cap (x)))
(26)
Applying the expectation operator to 266
leads to
(s²) == Ejiu
K3 (1 (HL (x))(x), (x),,3?
22)
E (xx(()) (trace (J (x)( Juo (m)())))
d₂
E Paza
no (uo
(27)
d=
Finally, combining (44 and (27 yields the following connection between the expected gap and the expected local smoothness:
2(s2) 21E
(x)o) s?.
(28)
13

---

Under review as a conference paper at ICLR 2021
D EXPERIMENTAL DETAILS FOR SECTION 3.2
D.1 EXPERIMENTAL SETUP
In the experiment, the model is trained with the Adam optimizer with a learning rate 10-3, The dimension of the latent space is set to 8. We run 200 epochs with minibatch size of 64 for all og We use the following DNN architectures for the encoder and decoder, respectively:
T. E R 28x28
Conves + ReLU
size of (64, 14, 14) size of (128, 7,
Conv12s ReLU Reshape 7)
Flatten + FC1024 + ReLU
FC16:
E
IR
16
FC1024 ReLU
FC128x7x7 > ReLU
size of (128,7, 7, size of (64, 14, 14) size of (1, 28).
ConvTes T61 + ReLU
+ ConvT, + Sigmoid
Here, FCk, Convi, COnvTR and ReLU denote the fully connected layer mapping to R² the con- volutional layer mapping to & channels, the transpose convolutional layer mapping to k channels and the rectified linear units (ReLU), respectively. The 3-tuple (channels, height, width) in the right column represents the output shape of each layer. In all the Conv, and ConvT; layers, 4 x convolutional filters are used with a common stride of (2.
Regarding the evaluation of criteria, MSE and KL evaluated the training set because the aim of the experiment is to validate the relation between of and the smoothness of the decoder. The upper bound of the MI obtained by calculating
d₂ 2
Enjun (x) Bag (xix) In E (x)) exp
2 n (x((x)13
013
(29)
for each minibatch and then taking their mean, where the batch size is 10, 000 for all the evaluations.
D.2 SAMPLES OF GENERATED IMAGES AND T-SNE VISUALIZATION OF LATENT SPACES
Figure D shows several images decoded from (x) + Es with Eg N(ee, 0, I for the cases with - 1.0 and 0.1. Posterior collapse can observed from these blurry images decoded from the stochastic encoding case with o = 1.0. This is due to the removal of batch normalization, which makes o² 1.0 become an inappropriate choice. However, if o is determined or adapted appropriately such by using the proposed method, posterior collapse will not happen. In the other settings, the tendency of how the image changes with the perturbation is similar, as shown in Table
Stochastic encoding
Deterministic encoding
10
10
10
10
o 1.0
9
a
9
0 0 9
0330444 4
3
3
2
2
0
2
5
8
Figure 1: Images in red boxes are the original images sampled from the MNIST dataset. The images in blue boxes are reconstructed by ns (x)x The other images are decoded from neighbor points of (x),, which perturbed by cg -N(e, 0, (I)
The latent spaces also visualized via t-SNE (Maaten & Hinton 2008) in Figure The dots with different colors represent the latent vectors encoded from images of different labels (numbers), and the pink dots are the sampling points generated from the prior p(z). As mentioned earlier, to observe the effect of o² clearly, we remove batch normalization, which usually helps prevent posterior collapse to certain extent. As a result, the latent space with o² 1.0 completely collapses and (2) approaches p(z) as shown in Figures [ and (). In this case, both KL collapse and posterior collapse occur.
14

---

Under review as a conference paper at ICLR 2021
(a) log 0 = 0.0
(b) log of -0.2
(c) log 0
(d) log 0 -0.6
(e) log 0
-.88
(f) log -1.0
Figure 2: Visualization of latent space via t-SNE. Pink dots are sampling points generated from the prior p(z).
Table 5: Evaluation of various criteria for different o². These criteria are the expected value of ||x' x (MSE), KL divergence, the upper bound of LI Mi z). the expected gap (the perturbation variance sf is set to 10 and 10 3) and expected local smoothness (ELS).
Expected gap
log ?
MSE
KL.
MI
ELS
10
52.74
8.0e-3
6.202-6
6.156-7
3.81c-5
0.9
52.74
19.48
6.8e-3
7.19e-6
7.1l
4.42e-5
0.8
134.83
8.5c+1
2.03--2
2.036-3
1.27c-1
0.7
20.50
139.09
1.0c42
2.37e-2
2.37e-3
1.48e-1
0.6
19.30
132.61
1.2e+2
2.87e:2
2.87e
1.80e
0.5
17.38
132.05
1.7c+2
3.44a-2
3.44o-3
2.16e-1
0.4
15.91
128.96
1.9c42
3.940-2
3.94e-3
2.47e11
125.97
2.3c+2
0.2
13.50
122.07
2.8c+2
5.16
5.1663
3.23e-
0.1
12.40
118.89
3.1c+2
5.830-2
5.810-3
3.67e-1
0.D
11.71
112.50
3.6e+2
6.79e-2
6.80e-3
4.26e-1
0.1
10.97
107.32
4.0e+2
7.53e.2
7.55e-3
4,74e
-0.2
10.23
103.87
4.2c+2
8.69--2
8.70s-3
5.46e1
-0.3
9.63
98.48
4.6c+2
9,842-2
9.88o-3
6.1ec-1
-0.5
8.71
88.35
5.2c+2
1.25g-1
1.26x-2
7,88c-1
-0.6
8.26
83.68
5.9e42
1.42e-1
1.436-2
8.9e-1
-0.7
7,82
79.70
6.6c+2
1.62e-1
1.62e
1.02
0..8
7.55
74.75
7.le+2
1.80c-1
1.06-2
1.13
7.26
7.3c42
2.04a-1
2.056-2
1.0
7.05
66.14
7.6e42
2.28c-1
2,300-2
1.45
E FIXING THE POSTERIOR VARIANCE OF LATENT SPACE
From the previous sections, we know that o² affects the smoothness via o². However, it would be interesting to see what will happen if o² is fixed while o² is optimized. In this experiment, the vari- ance parameter of is fixed while o² is optimized with the AR-ELBO 9 under the parameterization in Section 3 The other settings remain the same as those in Section 3.2 evaluate the numerical results for different of with the criteria listed in Section 3.2 According to Table 5 the tendencies of the expected gap and ELS show that large of makes the decoder smoother, which is consistent with the discussion in Section 3.1 However, the tendency of the KL divergence is different from that Section 3.2 Although a larger o? consistently leads to a smaller MI, and eventually the MI collapses zero; the KI. divergence still remains far from zero, which means that posterior collapse
15

---

Under review as a conference paper at ICLR 2021
(a) log 02 1.0
(b) log 0 0.6
(c) log o
(d) log 0 -.2
(e) log 0
0.6
(f) log -1.0
Figure 3: Visualization of latent space via t-SNE. Pink dots are sampling points generated from the prior p(z).
can happen without KL collapse. This phenomenon can be visually confirmed by observing the t-SNE plot in Figure The cause of this phenomenon can be roughly reasoned from the linear ap- proximated ELBO (4). which o² directly affects the gradient penalty and causes oversmoothness. It should be pointed out that the strength of L2 regularization in (4) is gradually decreased with decreasing o2: therefore, it does not dominate the whole objective function. As a result, the mean of
the approximated posterior (a(2) (2) is far from the mean of the prior p(2) (which is 0), and therefore (xI(fo(=) || p(=)) 13) does not diminish to zero.
F PROOF OF THEOREM 3
According to Theorem 4 in Dai & Wipf (2019), know that
lim E (x) E By (x(x) [||x (2) [ 0,
(30)
o²
which also leads to 02
0 (o? + 0). Here, 02 is estimated through MLE and is given by
03 E F (x) F (z d=
x [||x [z))))]
(31)
To prove Theorem 3 we need the following auxiliary theorem:
Theorem 4. In the training stage of VAE, we have o² 0 (o² + 0).
First, state the two lemmas with proofs.
Lemma 5. a VAE, I(x.x') & I(z, 2,) always holds, where z is the encoded latent variable 2. (x) with x Pdaaa(x).
Proof. The data processing flow of the VAE is x - 2.
x (x). (x). 2 + c,, and
x' (2),, where 6, O, The MI [(x; can be represented as I(x;2,x)) T(x; x') + T(x; x/x')
(32)
-(x; I(xxx(zz
(33)
Since and x' are conditionally independent on the given 2, follows that I(x; (z) 0. From the negaatiit of MI, we have I(x; 2) 2 I(x; x'). Repeating the same procedure for [(x; z, leads to the proof.
16

---

Under review as a conference paper at ICLR 2021
Lemma 6. The MI between and x' diverges to positive infinity o - 0. where x' is obtained from X ~ Pdasaa (x) as x' = polps Ez).
Proof. A lower bound I(x; x') is I(x; x')
DKL. (pemna oo (x' (x) Pdamal (x (x) (x)))
pxx (xx) (xx [In pa (x' | In Pa.a (x')
H [po.o (x')]
E purs (x) H [pa, (x'(xxxxxx)
H (po.o (x')] E Pup (x) (T
(34)
where (x x) = (xg (po [2]] and px.o(x') E8 (x (x((x))2)] Here, de- note the differential entropy the Gaussian with variance I as
I In(2Teed2d
(35)
0 as of 0, Ho (x')] Therefore, I(x:x') + as 0 + O.
Since a.
H Hlyaaaa (x)) and H( -o in the inequality of 34
Now we prove Theorem 3
The MI I(2; 2e) satisfies
I(z;2,) = DKL (gol [o ) loz (z|zz) ll 2o (z)
(x Bu (2/2.) (o (z)zz) In [0)0
H [ (z)] (z.) 7 (g03(2lze))
S 11 (2),))))) (0
dg In
det det(E).o?)
(36)
2
o²
where Eg.0 denotes the variance of gg.0? (z). Invoking Lemma and (36) leads to
In d=
det (let(Eg.og)
I(x;x') S
(37)
2
o²
Now, consider of / as L 0. According Lemma 6 it follows that det Σ x..0)) which contradicts the fact that g0.0z p(z). Thus, we must have of as of converges to zero.
+o0,
G DERIVATION OF PROPOSED OBJECTIVES
Here, we derive the objectives listed in Table The MLE of Σ,, Σr, can be obtained by
Consider an arbitrary Σn without any condition.
ΣS Bann (x) (x/x) [xx (oo()) (xoo))))))(()
(38)
From the partial derivative of Jo. o. Ex) w.r.t. En, we have
aJrc tb Σz)
1
+Σ, 2 (Eran (xx()) [(x (x(2))(x (()o)))(x)1((2))) +
(39)
The MLE of E.p and the objectives for the different parameterizations are described in the following.
G.1 Iso-I
For Iso-I, the MIE of oz can be given
dz Epan (x) By (x/s)) [|x (=(
0²
(40)
17

---

Under review as a conference paper at ICLR 2021
Substituting (40) into 7 leads to
d
(o Ø, σ²2)
E Bpen(()) (x)
B 20 (x
||| p0(2)]12]+
Dxl(go(z)x) p(z))
+ ln
2
2
dx
E (x) (95(2/x) p(z))
+ dæ In
||x us
dg
In dz
(41)
2
where the first and fourth terms are constants and thus omitted in (9).
G.2 ISO-D
First, substitute Σg = 02 into (10b)
22)(() ||x mo(z))l + de In (2)
Tre 20 Ex) Eis (x) (x/xx)
(42)
Also, we know that the MLE of oz(2) is
(2
(
(43)
dz
Substituting (43 into (42 leads the reconstruction objective of Iso-D:
dz In (2(22
Tree (0, 0, Ex) Exiaa (x) (x)x))
||x - (z)(2)1
(44)
(z)
d2
+ dx E (x
dx
[in ||x 1=(
dg.
(45)
2 2
2
G.3 DIAG-I
First, substitute Σ. diag(o?) into (00b:
d
de
(0, o, Ex) (x))
1
Bg (e/xc) [1Tz (o.(())))
In
0
(46)
20
2
Also, we know that the MLE of 02 is
E (xx) Byg
(1r: (z))
(47)
Substituting 44 into (46) leads to the reconstruction objective for Diag-I:
de
de + -
Jnc (0, Φ. Σx) = E (x)
E Egg (xlx) (1:; - (o.:(2))3]
In
22
(48)
20
2
1=1
d
1
do
(xp (o(()))) [(oe (2,
+
In
(49)
2
2
i=1
G.4 DIAG-D
First, substitute Σg diag(o?(z)) into 10b):
du
(xr - 1o,(2)))² + In 0 (2) }
1
(o, Φb Ex) Expaa(x) Ex((x(x)
(50)
20 2 (z)
Also, we know that the MLE of 2 (z) is
02 (2) (x. (z))
(51)
18

---

Under review as a conference paper at ICLR 2021
Substituting [51] into (50) leads to the reconstruction objective for Diag- D:
de
)
10 0, Ex)
Bien xx) B (2//xc)
20 (z) (2t (o,x(z))²
+ 2
In 0 02
(52)
du
E (x) (x) [In (x; (o,:(2))
(53)
2
H DERIVATION OF
The KL divergence terms can be represented as
Ean DKLlao (z/x) || po(z)))) =
E yx (x) (m/ qo (2/x) - In (2
(54)
Pdann (x) (2) xx]
E Pazs (x/x))
In
and
(x)g x) [l 10mp(z)p] = DkL(1((2) || p(z)).
(55)
Substituting the two equations above into I then L can be reformulated into
I RELATED WORKS
To the best of our knowledge, Lucas et al. (2019) were among the first to suggest that posterior collapse may be caused by a sub-optimal og- In the past, one of the common approaches for dealing with posterior collapse was to anneal the weight of the KL term in the ELBO. The first such attempt was KL annealing (Bowman al. 2015). Bowman et al. (2015) introduced weighting coefficient on the KI. term in the cost function during training. The weighting scheduling is determined in advance, e-g., the weight increases monotonically (Bowman al. 2015) Sønderby et al., 2016) or changes cyclically (Fu et al., 2019) as the training progresses.
The weighting coefficient also appears in Higgins et al. (2017), and is interpreted as a hyperpa- rameter that controls the information capacity of the latent space. The suggested value for such hyperparameter larger than 1. This analogous to setting o larger than the MLE value a² for 7 and (8a), which enforces stronger smoothness in exchange of better latent space disentanglement. The differences between Higgins et al. (0117) and the proposed method are: (i) o² changes between every minibatch; and (ii) the estimation of of is aimed prevent the oversmoothness.
Shao et al. (020)) proposed Con,rolVAE, which combined control theory with the VAE, and applied PI/PID control to determine the weight the KL term. Although applying control theory to the weighting of the KL term makes it possible to reflect the status of the optimization, Control needs extra hyperparameters be tuned in advance. On the other hand, our method be inter- preted as automatic KL annealing that estimates of through MLE without the need of tuning an extra hyperparameter.
Ghosh et al. (2020) interpreted the stochastic autoencoder with the reparameterization trick as noise injection process and proposed replacing such mechanism with an explicit regularized autoencoder (RAE). RAE regularizes its decoder in several ways: Lg regularization, gradient penalty (Gulrajani al. 2017) and spectral normalization (Miyato et al. 2018). As discussed Section 3.1 if is sufficiently small, the ELBO can also be approximately represented as sum of three losses which correspond to the terms included in the basic RAE objective funetion. The approximated objective function 4 can be obtained when RAE with gradient penalty (RAE-GP) is used and tuned appropriately.
Dai & Wipf (2019) optimized of using optimizer, which is the most similar approach to Iso-I model in that (7) is used as an objective function. Our proposed method provides simplified objective functions that enable the variance parameter to be optimized automatically and guarantee that o decreases as the reconstruction loss decreases, enabling the gradient penalty to be gradually weakened.
19

---

Under review as a conference paper at ICLR 2021
Comparing our work with these works showed that the proposed AR-ELBO and its variations also capable of regularizing the Gaussian VAE via weighting of the gradient penalty. In the standard parameterization such as (2), the second term in ④ is a weighted gradient penalty, which makes it possible to regularize each dimension of the latent space differently according to the property of the input data. In addition, the combination of the proposed objective functions together with standard Gaussian VAE allows implicit gradient regularization on the decoder with lower computational cost than that of explicitly adding the gradient penalty to the objective function, such in RAE.
J DETAILS OF EXPERIMENTAL SETUP IN SECTION 5
In this experiment, the Adam optimizer (Kingma & Ba 2015) is used and the maximum number of epochs set to 100 for MNIST and 70 for CelebA. The learning rates are 0.001 for MNIST and 0.0002 for CelebA. A minibatch size of 64 is used. All the FIL values are evaluated with 10, 000 generated samples.
For the posterior estimation by the second-stage VAE, we adopt the same networks for the encoder and decoder as those in Dai & Wipf (2019). For GMM fitting, we use the same settings as those in Ghosh et al. (2020). Experimental details including the network architectures for each dataset described in the following.
J.1 MNIST
We construct the encoder and decoder for the MNIST dataset using the architecture in Chen et al. 2016). The encoder constructed
E 1228x2
Conves ReLU
size of (64, 14, 14) size of (128 7,
+ Conv12s + ReLU + Reshape(128, 7,
Flatten FC1021
BatchNorm ReLU
FC16x2
The decoder is constructed as
E R
FC1024 + BatchNorm + ReLU
FC128x7x7 + BatchNorm ReLU
size of (128, 7, size of (64, 14, 14) size of (1, 28, 28).
ConvTEs + BatchNorm - ReLU
+ ConvT, + Sigmoid
In all the Conv layers and all the ConvTT layers except for the last, 5 5 convolutional filters with
stride (2, 2) are used. The difference between this architecture and those used in Appendix D. is whether batch normalization is applied or not. Although the original work of Chen et (2016).
the discriminator used Icaky ReLU (IReLU), we adopt ReLU for the encoder part, which improves the performance for all the models evenly.
J.2 CELEBA
The CelebA images are preprocessed with center cropping of 140 X 140, then resized described in Tolstikhin et al. (2018) and Ghosh al. (2020). It should be noted that the size of cropping differs among the previous works, and it markedly affects the FID score. We choose the above cropping size as is the largest among the related works and seems be the most difficult case for image generation. Moreover, Tolstikhin et al.] (2018) and [Ghoh et al.) (2020) also used this cropping size. Similarly in the previous section, the encoder and decoder are constructed on the basis of the discriminator and generator for CelebA used in Chen et al. (016)) The encoder is
x 64
We used the PyTorch version of the FID implementation from httpt:ttpssstttt /ttb///.tttt/bgggg/ com cwm/mseiizer. pytorch-ff fid for all the models. However, the result may slightly differ from that obtained with the Tensor- Flow implementation https::////.........ttt th https////gtttttccccc///ooo//////
20

---

Under review as a conference paper at ICLR 2021
constructed
Convi2s ReLU + Conv256 + BatchNorm + ReLU
size of (128, 32, 32)
size of (256, 16, 16)
Conv512 Normh ReLU
size of (512, 8, 8)
Convio24 + BatchNorm + ReLU
size of (1024, 4, 4)
Flatten + FC61x2-
The decoder is constructed as
2 E IR 64
7 FCsx8x1021
ConvTT112 + ReLU
size of (512, 16, 16)
+ Conv'T250 + BatchNorm - ReLU
size of (256, 32, 32)
ConvTi2s + BatchNorm + ReLU
size of (128, 64, 64)
7 ConvTT
size of (3, 64, 64).
In all the Conv& layers and all the Conv'I's layers except for the last, 5 x 5 convolutional filters with stride (2, are used. We use ReLU instead of leaky ReLU due to the performance consideration described in the previous subsection. To fit the size of the input images in experiment, one extra convolutional layer added for the encoder and the channel size twice as large as that in Chen al. 2016).
K
EXAMPLES OF RECONSTRUCTED AND GENERATED IMAGES IN SECTION 5
We show examples of reconstructed images and images generated by sampling the learned approxi- mated posterior from the proposed method and other works in Figures 4 and
Random samples
Reconstructions
2 VAE
7 2 / O
VAE =100
7 2 / 0 7 2 / 6 7 2 / 6 7 2 / O 7 2 / 6 7 2 / 6
7 7 3
8 3 8 8
6 9 8 7
WAE-MMD
Z 9 4
9
2 0 3
4 3 8
4
AB
S 3 2
B 8
2
RAE
8 6 2
3 9 4 9
RAE-GP
0 3 3 7
8 4 5 5
Isp-1 w/ trainable E,
7 0 2- 8
0 / 2 Ø
5 S 6 9
Diag-I w/ trainable E,
7 2 / 6
5 0
3 5 5 6
4 8 2
Iso-D w/ trainable E,
7 2 / 6
3 d 9 7
8 / 1
6 2 6
Diag-D w/ trainalle E.
7 2 6
8
% C
0 0 9
5 3 1
Ours (Iso-] w/ AR-ELBO)
7 2 6
8 8 ?
7 7
4 3 9
(Diug-I AR-ELBO)
7 2 / 6
3 5 O 7
O 3 8
5 3 7 2
Ours (Iso-D w/ AR-ELBO)
7 2 / 6
i 9 3 9
2 / 5
7 4 S 3
Ours (Diag-D w/ AR-ELBB)
7 2 6
6 3
S
6
6
7 9 J
5
3
Figure 4: Reconstructed images and examples of images generated from the prior and the estimated posterior on MNIST. "GT" stands for ground truth.
L INTERPOLATION OF LATENT VARIABLES
This section aims to investigate the feasibility of the learned latent space of the methods mentioned in Section If high quality images can be generated by interpolating the latent variables in latent
21

---

Under review as a conference paper at ICLR 2021
Random samples
Reconstrictions
Prior
GMM-10
GMM-100
GT
VAB (o2 1.0)
WAE-MMD
AE
RAE
RAE-GP
Iso-1 trainable E,
Diag-I w/ trainable E,
Isp-D triinable
Diag-D w/ tminnale X,
Ours (Iso-I w/ AR-ELBOO BO)
Ours (Diag-I w/ AR-ELBO)
Ours w/ AR-ELBO)
Ours (Ding-D AR-ELBO)
Figure 5:
Reconstructed images and examples of images generated from the prior and the estimated posterior on CelebA.
space, the corresponding latent space is more likely be feasible for other downstream tasks. There- fore, this section, we evaluate the FID scores images generated by latent variable interpolation for various models mentioned in section 5
First, we choose 10,000 random pairs of images from both MNIST and CelebA datasets. The interpolation is done by applying spherical interpolation (Ghosh et al. 2020 in latent spaces and then generate the interpolated images with the decoders. In the end, we evaluate the FID these interpolated images.
Furthermore, the experiment has been proceeded with two different setups of mixing ratios: (i) fixed ratio of 0.5, i.c., the mid-point of two latent variables; and (ii) uniformly distributed ran- dom ratio between [0, 1] for each image pair. The result is shown in Table 4 where the proposed method achieved the best score on MNIST and is competitive on CelebA. This suggests that a gen- erative model with proper smoothness achieved via the proposed method is also feasible for other applications such as interpolation and possibly applicable for other semantic controls.
22

---

Under review as a conference paper at ICLR 2021
(o) 110)
4 9 7
7
7
9
9
WAE-MMD
4 4 4 7
7
7
7
8
AE
H 7
7
7
7 7
7
8
7 7 7 4 4 4
7 8
RAE-GP
4 4 4
Y
7 7
7
Iso-] w/ trainable Xx
L 4 4
7 7 8
Diag-l w/ trainable E,
4 4 4 4
7
7
7 7
trainable E,
14 44 4
7
7 7 8
Diag. D w/ truinable Σ,
4 4
7 7
7
7
7
7
8
Ouns (Iso-1 w/ AR-ELBO)
4
4
7
7
7
7
Y
Y
8
Ours ( I w/ AR-ELBO)
7
7
7 7
7
7
7
7
7 7 7
Ours (Iso-D AR-ELBO)
4f
4
7
7
7
7
7
7
(Diag-D AR-EL
7
7 7 7 7 Y 8
Figure 6: Examples of interpolated images.
23


Reviews:
[{"review_id": "pdUcyXpUqON", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers Gaussian VAEs and their tendency to suffer from posterior collapse. In particular, the authors analyse the impact of the usually fixed covariance $\\sigma_x$ of the decoder Gaussian on the learned encoder variance. They show that the former can be seen as a regulariser for the latter and therefore impacts the \"smoothness\" of the encoder. The authors hypothesize that a large value of $\\sigma_x$ causes posterior collapse as a consequence.\n\nAs a remedy, the authors propose to consider the decoder covariance as a learnable parameter. To achieve this, they propose to optimize the ELBO objective by a block-coordinate descent approach where the parameters of the decoder covariance are considered as a separate block. The possibly remaining prior-posterior mismatch in the latent distributions is mitigated by an second stage VAE as e.g. in Dai and Wipf, 2019. Experiments compare the new approach (on MNIST and CelebA) with existing approaches in terms of MSE on the training data and Frechet inception distance for the generator and show that it is least on par or outperforming them.\n\nThe paper is well written and technically correct. All necessary concepts are concisely introduced. Its novelty is in my view the following:\n(1) the new, stronger definition of posterior collapse\n(2) the analysis of the regularising impact of the decoder variance\n(3) the proposed method for leaning the parameters of decoder variance\n\nOn the downside, I am not convinced that posterior collapse is caused solely by over-smoothness of the encoder. The reasons are the following.\n(1) The analysis in Dai and Wipf, 2019, Theorem 2 presumes that the encoder covariance is not required to be diagonal.\n(2) Posterior collapse is also observed in Bernoulli VAEs, where the latent variables are binary valued vectors. In these models, usually all(!) parameters of the encoder/decoder are learned by optimising the ELBO objective.\n\n*Further comments*\nPlease describe the relation of your objective (9) to $\\beta$-VAEs.\n\nThe authors have tried to answer all issues and questions raised in the reviews. At least, I can say so for the questions and issues raised by me. I therefore tend to keep my positive opinion on this paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "l858E7WN7Xx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the Gaussian VAE and figures out that the decoder variance regularizes the VAE and affects the model smoothness, and an inappropriate estimation of this parameter would raise posterior collapse, which is supported by theoretical analysis and empirical demonstrations. Hence, this paper then proposes an ELBO with adaptive decoder variance to avoid oversmoothing the model. Overall, the idea is interesting and provides some new insights for our community. The major concerns regarding this paper are listed as below.\n\nSince the variance parameter affects the performance of VAE, this paper proposes an adaptive training strategy using alternative updating of variance and the remaining parameters. In comparison to the strategy that directing treating the variance as a trainable parameter, though the following numerical experiments showcase that the proposed training strategy achieves higher FID scores, the reviewer is still unclear about the main difference between the two training strategies. The authors are suggested to make more explanations.\n\nFrom the comparison results in Table 3, it seems that the parameterizations of variance have a large impact on the performance of the proposed VAE. As for VAE (sigma^2_x: optimizer) and Ours (Iso-I), their structures are the same with the only difference in training strategy. It is observed that VAE (sigma^2_x: optimizer) outperforms Ours (Iso-I) on the MNIST dataset.\nHence, the authors are suggested to test the conventional VAE using all the studied four parameterizations of variance in order to provide a comprehensive comparison.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hagi4lRhM4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses the \"posterior collapse\" phenomenon observed in training latent variable models (in particular Variational Auto-Encoders), and propose a new training objective to remedy the problem. The theoretical analysis of the authors suggests that the posterior collapse is induced by an inappropriate choice of variance in the decoder distribution. The new objective they propose, the AR-ELBO, jointly optimises this variance along with the usual network parameters. The authors demonstrate that their objective yields relatively good results on image modelling, compared to other standard VAE methods.\n\nI think the paper is generally well-written, with sufficient clarity around the major hypotheses and results. The motivation for the paper is also well established, with a sensible exposition of the problem of posterior collapse given through mutual information, which seems intuitive. Additionally, the theoretical analysis is straightforward to follow and explained relatively well, which is appreciated. The empirical results do also seem promising, with the metrics given demonstrating the proposed objective seems to be competitive with other widely used objectives for equivalent models.\n\nI think the proposed objective is simple, which is a good thing. The variance of the output distribution is substituted by the current expected MSE of the reconstructions, which is its optimal value at any point in the optimisation. This results in the variance parameter becoming implicit. Table 1 shows the trade-off between MSE and KL nicely, and we can see the “sweet spot” of the choice of variance (to minimise the overall objective). There is an empirical comparison to the more obvious approach of just optimizing the variance parameter using gradient descent along with the neural network parameters. Something I would be interested in seeing discussed is why the AR-ELBO approach is a better method. I suspect that there may be something more concrete that can be said, in fact the substitution to eliminate the variance from the objective is reminiscent of the Rao-Blackwell theorem, which essentially states that it is best to integrate out variables in an estimator if you can.\n\nI do however, have a number of problems with the paper in its current form. One major issue is the seemingly limited setting in which the analysis is provided. The analysis is only defined on a continuous data domain, with Gaussian output distributions. And in fact the core of the hypothesis and results centers on the premise that the width of this Gaussian is a crucial component of the optimisation. However, it is in practice quite rare to actually have continuous data. Almost all data we encounter is discrete, and even the limited datasets the authors assess their objective on is discrete - MNIST and CelebA - since image data is discrete. It is common (and surely best) practice with such image data to use a discrete output distribution p(x|z), for example a discretised mixture of logistic distributions, which has been demonstrated to be very successful. The authors only provide any results or analysis for the continuous data case, with Gaussian output distributions, and in fact do not even mention that the discrete case is the usual case, or that the data that they seek to model is actually discrete. This seems to be misleading, or at the very least very restrictive. Indeed, it seems that in the discrete output distribution case, the analysis the authors provide and the objective they propose does not hold, since it relies on an algebraic manipulation of the Gaussian pdf (or derivative of).\n\nAnother issue I have is that the experimental verification of the AR-ELBO deals only with the sample quality (of either reconstructions or true samples), but this is not the full picture of a VAEs performance. We also care about the structure of the latent space itself, for example to perform latent interpolations or downstream tasks with the latents. We also care about the approximate density function we have learned - to perform anomaly detection etc. These are not mentioned in the results at all, which I think is an omission.\n\nOverall, I quite like the simple and well-presented nature of the paper, but the limited scope as raised above means I think the paper should be rejected in its current form. I would be willing to increase my score if the authors addressed some of the concerns I have raised.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4_eJn3fvhX7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "AR-ELBO: Preventing posterior collapse induced by oversmoothing in Gaussian VAE\n\nSummary:\n\nThe paper discusses the situation where the posterior approximation in the Gaussian variational autoencoder collapses during training. It is argued that a major contributing factor is a mismatched output variance parameter. Several analyses provide evidence for and intuition about this behavior. A simple procedure, where a maximum likelihood estimate of the noise is employed, is proposed. Experiments on some standard datasets provide an empirical evaluation of the method.\n\nPositive:\n\n1. The paper is in general well written and fairly easy to follow, at least for readers familiar with the Gaussian VAE.\n2. The analyses are interesting and provide some good insights.\n3. The proposed methods are presented in sufficient mathematical detail, including details for different variants of method.\n\nNegative:\n\n1. I encourage the authors to make the code available during the review process. Without access to the code, it is difficult to assess how well the results can be reproduced.\n2. Since this is not the first work to address estimating the noise variance in a VAE setting, I would like to have seen a more direct comparison with competing methods. This could include results on how precisely the variance is estimated and how the procedure influences the convergence of other parameters.\n3. The novelty of the proposed method is fairly limited - I would expect that learning the noise is common practice in applied work.\n\nRecommendation:\n\nWeak accept\n\nFurther comments:\n\n\"Contrary to popular belief...\" could you provide a reference?\n\n\"However, in most implementations ... constant of 1.0.\" is this really true? I would assume most people would either choose the noise variance based on prior knowledge or fit it (with maximum likelihood or a variational approximation) long with the other model parameters. If it really is common practice to not set the noise in accordance with the data, then the procedure presented in this work is definitely a much better default.\n\nThe paper uses the term \"adaptive\" - I would prefer to phrase this as maximum likelihood.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is about learning the output noise variance of a VAE and its effect on the generated image quality as measured by FID. The paper argues that the output variance parameter plays an important role and proposes a simple procedure, where a maximum likelihood estimate of the noise variance is estimated. Experiments on some standard datasets are provided.
Overall, the paper is well written and has been perceived positively by the reviewers. However, the effect of observation variance has been in detail analysed by earlier work, in particular Dai and Wipf 2019. The novelty of the current paper is somewhat limited in scope. The paper is somewhat borderline in these respect; a much stronger experimental section would have been helpful.

One key contribution of the work is empirical comparison of alternative parametrizations of the output noise. Overall, the paper would be stronger if this aspect is analysed more in detail, possibly with careful comparisons with competing methods. Inclusion of controlled experiments (e.g. by adding extra noise to data) to show how precise the noise variance estimation and how the procedure influences the convergence of other parameters would have made the paper much more impactful.

Decision: Reject