Shallow VAEs with RealNVP Prior Can Perform as Well as Deep Hierarchical VAEs
Authors: Haowen Xu
Wenxiao Chen
Jinlin Lai
Zhihan Li
Youjian Zhao
Dan Pei
Venue:  (2020)
Keywords: Variational Auto-encoder
RealNVP
learnable prior

Abstract: Using powerful posterior distributions is a popular technique in variational inference.  However, recent works showed that the aggregated posterior may fail to match unit Gaussian prior, even with expressive posteriors, thus learning the prior becomes an alternative way to improve the variational lower-bound.  We show that using learned RealNVP prior and just one latent variable in VAE, we can achieve test NLL comparable to very deep state-of-the-art hierarchical VAE, outperforming many previous works with complex hierarchical VAE architectures.  We hypothesize that, when coupled with Gaussian posteriors, the learned prior can encourage appropriate posterior overlapping, which is likely to improve reconstruction loss and lower-bound, supported by our experimental results.  We demonstrate that, with learned RealNVP prior, ß-VAE can have better rate-distortion curve than using fixed Gaussian prior.

Under review conference paper ICLR 2020
SHALLOW VAES WITH REALNVP PRIOR CAN PER- FORM AS WELL AS DEEP HIERARCHICAL VAES
Anonymous authors
Paper under double-blind review
ABSTRACT
Using powerful posterior distributions a popular technique in variational infer- ence. However, recent works showed that the aggregated posterior may fail to match unit Gaussian prior, even with expressive posteriors, thus learning the prior becomes an alternative way to improve the variational lower-bound. We show that using learned ReaLNVP prior and just one latent variable in VAE, we can achieve test NLL comparable to very deep ofte-oo--the-ar hierarchical VAE, outperform- ing many previous works with complex hierarchical VAE architectures. We hy- pothesize that, when coupled with Gaussian posteriors, the learned prior can en- courage appropriate posterior overlapping, which is likely to improve reconstruc- tion loss and lower-bound, supported by our experimental results. We demonstrate that, with learned RealNVP prior, 8 B-V can have better rale-distortion curve than using fixed Gaussian prior.
INTRODUCTION
Variational auto-encoder (VAE) (Kingma & Welling 2014 Rezende et al. 2014) is a powerful deep generative model. The use of amortized variational inference makes VAE scalable deep neural networks and large amount of data. Variational inference demands the intractable true posterior be approximated by tractable distribution. The original VAE used factorized Gaussian for both
the prior and the variational posterior (Kingma & Welling 2014 Rezende et al.)
2014)) Since then,
lots of more expressive variational posteriors have been proposed (Tran et al. 2016 Rezende
Mohamed, 2013; (Salimans el al., 2015) Nalisnick et al. 2016] Kingma el al., 2016;
Mescheder
al. [017]] van den Berg al. 2018). However, recent work suggested that even with powerful
posteriors, VAE may still fail to match aggregated posterior to unit Gaussian prior (Rosca et al., 2018), indicating there is still a gap between the approximated and the true posterior.
To improve the variational cower-bound one alternative way to using powerful posterior distribu- tions learn the prior, idea initially suggested by [offmmn & Johnson (2016). Later on, Huang et al. (2017) applied RealNVP (Dinh al.] (017)) to learn the prior. Tomezak & Welling (2018) proved the optimal prior is the aggregated posterior, which they approximate by assembling mixture of the posteriors with set of learned pseudo-inputs. |Bueer & Mnih| (2019) constructed rich prior by multiplying a simple prior with a learned acceptance function. (Takahashi et al. (2019) introduced the kernel density trick to estimate the KL divergence in ELBO, for their implicit prior.
Despite the achievements of these previous works on posteriors and priors, the state-of-the-art VAE models with continuous latent variables all rely on deep hierarchical latent variables"] although some of them might have used complicated costeriors/priors as components in their architectures. Most latent variables in such deep hierarchical VAEs have no clear semantic meanings, just technique for reaching good lower-bounds. This is sharp contrast with GANs (Goodfellow et al., 2014; Arjovsky et al.) (017) and flow-based models (Ho et al. 2018), where most of them can reach state-of-the-art results with only one latent variable, with clear semantic meanings. We thus raise and answer question: with the help of learned priors, can shallow VAEs achieve performance comparable or better than deep hierarchical VAEs? This question is important because a shallow
'The term "hierarchical latent variables" refers multiple layers of latent variables, formulated p(x) Eppt, y Exina (ms) ... Expa iny .)) p(x|zı,.... xx) )}: while "one latent variahle" refers to just one z standard VAES. Both z₁ and multi-dimensional tensors. Also, "deep" refers many hierarchical latent variables, while "shallow" refers few latent variables.

---

Under review conference paper ICLR 2020
VAE would be much more promising to scale more complicated datasets than deep hierarchical VAEs. To answer this question, we conduct comprehensive experiments on several datasets with learned RealNVP priors and just one latent variable, which even shows advantage over some deep hierarchical VAEs with powerful posteriors. We also propose hypothesis on why learning the prior can lead to such great improvement in model performance, supported by our experimental results. In summary, our contributions are:
We conduct comprehensive experiments four binarized datasets with four different work architectures. Our results show that VAE with RealNVP prior consistently outper- forms standard VAE and RealNVP posterior.
We are the first to show that using learned RealNVP prior with just one latent variable in VAE, it is possible to achieve test negative log-likelihoods (NLLs) comparable to very deep offte-of-the-at hierarchical VAE on these four datasets, outperforming many previous works using complex hierarchical VAE equipped with rich priots/posteriors.
We hypothesize that, when coupled with Gaussian posteriors, the learned prior can encour- age appropriate posterior overlapping. which is likely improve reconstruction loss and lower-bound, supported by our experimental results.
We demonstrate that, with learned RealNVP prior, 3-VAE can have better rate-distortion curve (Alemi et al. 2018 than using fixed Gaussian prior.
2 PRELIMINARIES
2.1 VARIATIONAL AUTO-ENCODER
Variational auto-encoder (VAE) (Kingma & Welling 2014 Rezende et al. 2014) is deep proba- bilistic model. It uses a latent vanable z with prior Px (z) and a conditional distribution pe (x(z)), to
model the observed variable x. po(x) xx) is defined as Po (x (z) px (z( (z)dz, dz, where (x(2))) is derived by
a neural network with parameter 0. log po(x) (x) is bounded below by evidence lower-bound (ELBO):
(x) 2 L(x: 0, o) (x)x) (x + (z) log (z|x)]
(1)
(x)x) (o DRL (go (x)x)))))
(2)
where go (z(xx) is the variational posterior to approximate po (z)x), derived by a neural network with parameter co In Eq. (2). the first term is the reconstruction loss of x, and the second term is the
Kullback Leibler (KI.) divergence between (x(x)x) and (z()). Optimizing q6 (z/xx and (x(22) w.r.f. empirical distribution p" (x) can be achieved by maximizing the expected ELBO w.r.f. p" (x):
C(A, 0, o) = (x)((() X, 0, +)] = (x) (xxx) (x)x))(x(z) + log Px log go (z/x)] (3)
A hyper-parameter 33 can be added to the training objective, in order to control the tradc-off between
reconstruction loss and KL divergence, known 3-VAE (Higgins al. 2017 Alemi et al. 2018):
Cx, Φ) (x)(x) (xo(x)x) [log pa (x z)) + / (log Px (z) log g0 (z(x))
(4)
The aggregated posterior gz(z) [x (z( p (x) dx, should be equal (z),., if VAE is per- fectly trained (Hoffman & Johnson| 2016). However, even with powerful posteriors, the aggregated posterior may still not match a unit Gaussian prior (Rosca et al. (018). To investigate this problem, Hoffman & Johnson (2066) suggested to decompose Eq.
C(A 0, Ex) (x)())x) [log (x/z)) D -
(5)
(=(x) dx is the mutual information. Since (x(z) is only in ② ELBO can be further enlarged if px (z) is trained to match (z).
where 1,|2;X] Jf
JJ (0(2, x) log
(a(xx))
2.2 REALNVP
Re Dinh et al., 2017) is deep probabilistic model. We denote its observed variable by z and latent variable by w, with marginal distribution (z) and prior ps (w). Unlike VAE, ReaINVP

---

Under review conference paper ICLR 2020
relates z and w by an invertible mapping w f2(2) instead a conditional distribution, as follows:
Px(z) = -Pe(w) det (0f((z) )
z 1=
(6)
Oz
where del (0fx(x)/0x) is the Jacobian determinant of [x- In RealNVP, f/ is composed of K invertible mappings, where fx(z) = (fk ..... fi (z). and each fx is invertible. fx must be carefully designed to ensure that the determinant can be computed efficiently. The original paper of RealNVP introduced the affine coupling layer as fi. Kingma & Dhariwal (208)) further introduced actnorm and invertible 1xl convolution. Details can be found their papers.
3 LEARNING THE PRIOR WITH REALNVP
It is straightforward obtain rich prior (x(z) from a simple (i.e., with constant parameters) one with ReaINVP. Denote the simple prior as Pe (w(,), while the RealNVP mapping as w = fz(... We
then obtain Eq. (6) as our prior (2(2). Substitute Eq. 66 into 3 we get to the training objective:
C(), Φ) (x) E (a)o)) logpo(x(z) + (og)e(f((z)) + log det (DS((a) ){ Oz
log go (z(2)))
(7)
We mainly use joint training (Tomezak Welling 2018 Bauer Mnih 2019) (where px(z) is jointly trained along with qp(z) (z(x) x) and Pa (x z by directly maximizing Eq. 7 but we also consider
other two strategies, Appendices B.3 and B.9 for discussions.
our initial experiments, training VAEs with deep RealNVP priors is unstable. The HVAEs (see Appendix [B.13) with RealNVP priors are even more unstable, where the training procedure can barely finish. However, tackle this problem by the following two techniques: (1) using gradient clip, as the original RealNVP (Dinh al.) (2017) does; and (2) clipping the std of Gaussian posterior go (z) x) of VAE (or [o[2g|X] (x) and qo [z[2i]x) (x) of HVAE) by a minimum value of e11 x 1.67 x 10-5.
4 APPROPRIATE POSTERIOR OVERLAPPING WITH LEARNED PRIOR
As we shall see in Section 5.4 RealNVP prior can result in a test negative log- likelihood (NLL) substantially better than RealNVP posterior, with the same flow depth. Also, using both RealNVP posterior and prior shows no significant advantage over using RealNVP prior only, although the total flow depth of the former variant is twice as large as the latter one. To investigate why VAE with RealNVP prior can lead to substantially better model performance than the other two variants, we take the particular architecture of our models into consideration (i.e., Bernoulli p=(x. Gaussian go (z) x). and unit Gaussian or flow prior px(z)), and start with the following proposition:
Proposition 1. Given finite number of discrete training duta, i.e., p*(x) 1/ En ö(x - x) po Bernoullif (pg(2)), where the Bernoulli mean pz(z)) produced by the decoder, and 0 < (z) < f.r each of its k-thh dimensional output, then the optimal decoder (z) is:
90 (2((
pz(2)= Σ (2) (0
where we (z) = E, (2((())
and Σ ()((2) =1
(8)
Proof. See Appendix A
Proposition I Jsuggssts that gu(z)x) for different overlap, then even the center of qo (z|x(() of
training point x0)), the optimal decoder will be average of both ( and other training points x6), weighted by (z) and (z).). Rezende & Viola 2018) have shown that weighted average like
this is likely cause poor reconstruction loss (0) Eq. (5)) and "blurry reconstruction". Besides poor reconstructions, when x digeeiilesiional, the reconstruction loss would typically dominate ELBO, suggesting that good reconstruction loss is necessary for obtaining good ELBO.
To enlarge @. it is a crucial goal to appropriately reduce the overlapping of go (z)x) for different x. For Gaussian posterior, this seems to involve shrinking the standard deviations (std) of most (z x).

---

Under review conference paper ICLR 2020
since when the mean of every 9s (z(2)x) is determined, smaller stds for the majority of go(x)x) would effectively result in less overlapping.
When the prior px (z) is fixed (such as unit Gaussian), however, smaller stds for (z x) are likely induce "holes" on the aggregated posterior, causing g.(z) (z) to be more dissimilar with the prior, enlarging @ in Eq. 45). Also, less overlapping among the Gaussian posteriors qa (z|x) would cor- respond to higher mutual information (@ in Eq. (5) (Hoffman & Johnson] 20T6; Mathicu et al. 2019). As a result, with fixed prior, there should exist trade-off between ① and
( In fact,
such a trade-off has already been discussed the context of 3- B-VAE (Alemi et al. 2018).
should a desirable property that the overlapping among da (z)x) would be large for similar x. and conversely, small for dissimilar x. We shall use the term "appropriate overlapping" to denote such property. We believe that the trade-off between the reconstruction loss (0) and the mutual information (3) should be one important ingredient for reaching the "appropriate overlapping". as implied by their respective meanings. However, the effect of @ is not so clear. We suspect that, in many cases, this term may exhibit rverreegularization, resulting too large overlapping, and further unsatisfactory reconstruction loss. In this paper, we choose to use learned RealNVP prior to reduce the effect from @, such that the trade-off would occur mainly between ! and 3. We hope this will bring us closer to the "appropriate overlapping". than using the fixed unit Gaussian prior.
conclusion, when pe(x|z) Bernoulli, we hypothesize that learning the prior can encourage "appropriate overlapping" among Gaussian posteriors, which is vital for good reconstructions and NLLs. Rezende & Viola (018)) has proved when (x 2) = =N (N) o I with fixed constant the optimal decoder is also Myy (z) = Σx (z) x thus our analysis also holds in such situation.
EXPERIMENTS
5.1 SETUP
Datasets We use four datasets in our experiments: statically and dynamically binarized MNIST (Larochelle & Murray 2011 Salakhutdinov & Murray 2008) (denoted as StaticMNIST
and MNIST in our paper, respectively), FashionMNIST (Xiao et al. 2017) and Omniglot (Lake
et al.
2015). Details of these datasets can be found in Appendix B.
We perform systematically controlled experiments, using the following VAE variants: (1) VAnseVAF, with dense layers; (2) CAE, with convolutional layers; (3) Res. with ResNet layers (Zagoruyko & Komodakis (016); and (4) Vix (Gulrajani et al. 2017). with several PixeICNN layers top of the ResnetVAE decoder. For RealNVP priors and posteriors, we use K blocks of invertible mappings (K is called flow depth hereafter), while each block contains an invertible dense, a dense coupling layer, and an actnorm (Dinh et al., 2017 Kingma & Dhariwal 2018). The dimensionality of z 40 for StaticMNIST and MNIST, while 64 for FashionMNIST and Omniglot. More details are Appendix B.2
Models
We also conduct experiments with RAs and PixelHVAE (Tomezak & Welling 2018 sec Appendix B.13
Training and evaluation Unless specified, all experiments are repeated for 3 times to report met- ric means. We Adam (Kingma & Ba 2015) and adopt warm up (KI. annealing) (Bowman et al.] 2016) to train all models. We perform early-stopping using negative log- likelikkihooo (NLL) on vali- dation set, to prevent over-fitting on StaticMNIST and on all datasets with PixeIVAE. We use 1,000 samples to estimate NLL and other metrics on test set. See Appendix B.3 for more details.
5.2 QUANTITATIVE RESULTS OF VAE WITH REALNVP PRIOR AND ONE LATENT VARIABLE COMPARED DEEP IIERACCICCLL VAES AND OTHER PREVIOUS WORKS
Tables ① andd] we compare ResnetVAE and VAX with RealNVP prior to other approaches
on Stat and MNIST. See Tables BB I0 [or results on Omniglot and FashionMNIST,
which have similar trend. All models except ours and that of Huang al. (2017) used at least 2 latent variables. For comparison between our model and Huang et al. (017)) see Appendix (B.7
Our ResnetVAE with RealNVP prior, K = 50 is second only to BIVA among all models without PixeICNN decoder, and ranks the first among all models with PixeICNN decoder. On MNIST, the NI.L of our model is very close BIVA, while the latter used 6 latent variables and very compli-

---

Under review conference paper ICLR 2020
Table Test NLL on StaticMNIST. "t" indi- cates a hierarchical model with 2 latent variables, while "3" indicates at least 3 latent variables.
Table Test NLL on MNIST. and "t" has the same meaning as Table
Model
NLL
Model
NLL
Models without PixelCNN decoder
Models without PixelCNN decoder
ConvHVAE Lars prior! (Bauer &
81.70
ConvHVAE + Lars priort (Buuer &
80.30
Mnih
2019)
Mnih 2019
Conv VAE
VampPriorf Tomezak
81.09
Conv VAE VampPriorf
Tomezak
79.75
& Welling 2018)
& Welling 2018)
RosConv RealNVP prior Huang
81.44 VAE + IA (Kingma et al.
2016
79.10
ct al.
2017)
BIVA (Maalee et al. 2019)
78.41
VAE LAFt (Kingma et al.
2016
79.88
ConvVAE + RNVP (
(
K
50
78.61
BIVA= (Mallee et al. 2019
78.59
ResnetVAE + RNVP ((z). K
78.49
Conv VAT RNVP (z).
50
80.09
Models with PixelCNN decoder
ResnetVAE RNVP p(z). K 50
79.84
VLAE
(Chen et al. 2017)
78.53
Models with PixelCNN decoder
PixeIVAET (Culrajani et a..
2017
79.02
VLAE* (Chen al. 2017
79.03
PiXelHVAE VampPrior''
Tomezak
78.45
PixeIHVAE + VampPries Prior Tomezak
79.78
&Welling 2018)
& Welling 2018)
Pix RNVP p(z).
50
78.12
Pixel VAE RNVP (zz),
K
50
79.01
7
2
6
3
0
2
9
8
3
3
7
7
17
9
9
2
7
2
2
6
4
2
(
1
2
6
1
3
3
8
8
8
0
4
8 8
9
4
4
4
7
3
0
5
0
8
9
4
5
6
0
2
2
2
2
4
4
2
2
2
4
2
0
9
5
2
2
1
7
7
1 9
5
5
S
5
5
5
5
5
5
5
5
5
5
5
9
/
9
8
9
9
0
4
4
9
9
J
9
2
2
2
2
2
2
4 0
P 6
5
9
6
2
9
3
3
3
3
8
8
3
3
3
8
8
standard
RealNVP (2(
log
40
-30
-20
-10
standard
RealNVP Px(z)
Figure 1: Sample means from (z( of Resnet- VAE with: (left) Gaussian prior; (right) Real- NVP prior. The last column of each 6x6 grid shows the training set images, most similar to the second-to-lan column in pixel-wise 1.2 distance.
Figure 2: Interpolations of from ResnetVAF, between the centers of qo(2[x) of two training points, and heatmaps of log (z).. The left- and right-most columns are the training points.
cated architecture. Mcanwhile, our ConvVAE with RealNVP prior. K 50 has lower test NLL than ConvHVAE with Lars prior and VampPrior. Since ConvVAE is undoubtedly simpler ar- chitecture than ConvHVAE, (which has latent variables), is likely that improvement comes from the RealNVP prior rather than the different architecture. We also conduct experiments on Ci (Krizhevsky] 2009). Although not as good as state-of-the-an deep VAEs, our ResnetVAE using RealNVP prior with just latent variable still shows great improvement prior, in both test bpd and sampling quality. See Appendix B.5 for more details.
the Gaussian
Tables and 2 show that using RealNVP prior with just one latent variable, it is possible to achieve LL comparable to very deep state-of-the-art VAE (BIVA), ourperforming many pre- vious works (including works on priors, and works of complicated hierarchical VAE equipped with rich posteriors like VAE + IAF). This discovery shows that shallow VAEss with learned prior and small number of latent variables is a promising direction.
5.3 QUALITATIVE RESULTS
samples images from ResnetVAE with/without RealNVP prior. Compared to standard Resnet VAE, ResnetVAE with RealNVP prior produces fewer digits that are hard interpret. The
Figure I

---

Under review conference paper ICLR 2020
Table 3: Average test NLL (lower is better) of different models, with Gaussian prior & Gaussian posterior ("standard"), Gaussian prior & RealNVP posterior ("RNVP g(=lx)")), and RealNVP prior
Gaussian posterior ("RNVP p(z)"). Flow depth 20.
DAn
ResnetVAE
PixeIVAE
RNVP
RNVP
RNVP ((=[>)
RNVP
RNVP
RNVP
Datasets
standard
standard
standard
((=lær)
p(z)
p(z)
q(z[>))
p(z)
StaticMNIST
88.84
86.07
84.87
82.95
80.97
79.99
79.47
79.09
78.92
MNIST 84.48
82.53
80.43
81.07
79.53
78.58
78.64
78.41
78.15
PaShionMNIST
228.60
227.79
226.11
226.17
225.02
224.09
224.22
223.81
223.40
Omniglot 106.42
102.97
102.19
96.99
94.30
93.61
89.83
89.69
89.61
Table 4: Test NLL of ResnetVAB MNIST, with ReaINVP posterior ("q(=lx)"),, RealNVP prior ("p(z)"), and RealNVP prior & posterior ("both"), Flow depth K is 2g for the posterior the prior "g(=|x)" and "²"""", while Ky for both the pos-
Table 5: Average number of active units of ResnetVAF, with standard prior & poste- rior ("standard"), RealNVP posterior ("RNVP ((=lz)"") and RealNVP prior ("RNVP p(=)").
ResnetVAE
terior and the prior in "both".
RNVP (<[xx)
RNVP
Kg
Datasets
standard
p(z)
ResnetVAE with
5
10
20
StaticMNIST
30
40
40
<<2[x), K = 2Ko
80.29
79.68
79.53
79.49
MNIST
25.3
40
40
both, K Ko
79.85
79.01
78.71
78.56
FashionMNIST
27
64
64
p(z), 2Ko
79.58
78.75
78.58
78.51
Omniglot 59.3
64
64
last column of each 6x6 grid shows the training set images, most similar to the second-to-lass column pixel-wise distance. There are differences between the last two columns, indicating our model is not just memorizing the training data. More samples are in Appendix B.I
5.4 ABLATION STUDY
Re prior leads to substantially lower NLLS than standard VAE and RealNVP posterior Table 3 shows the NLLs of VAEE ResnetVAE and PixeIVAE with flow depth K 20. The NLLS of VAE (with similar trends ResnetVAE) and the standard deviations of all models are reported in Table B.12 while the NLLs of ResnetVAE with different K' (range from to 50) can be found in Table B.15 We can see that RealNVP prior consistently outperforms standard VAE and RealNVP posterior in test NLL, with as large improvement about 2 nats (compared standard Resnet or nat (compared to ResnetVAE with RealNVP posterior) on ResnetVAE, and even larger improvement on Dense The improvement is not so significant on PixeIVAB likely due the less information encoded in the latent variable of ViX (Gulrajani et al. 2017).
Using RealNVP prior only has better NLL than using both RealNVP prior and posterior, or using Re posterior only, under the same model complexity. as shown in Table
Active units Table 5 counts the active units (Burda et al. 2016) of different Resnet which quantifies the number of latent dimensions used for encoding information from input data. We can see that, both RealNVP prior and posterior can make all units a ResnetVAE to be active (which in sharp contrast to standard VAE), However, RealNVP prior can in fact result in substantially better reconstruction loss than RealNVP posterior (see Table B.18) This indicates that, the good regular- ization effect, "a learned RealNVP prior can lead to more active units than fixed prior" (Tomczak) & Welling 2018: Bauurr & Mnih 2019), is not the main cause of the huge improvement in NLLs, especially for the improvement of RealNVP prior over RealNVP posterior.
5.5 RECONSTRUCTION LOSS AND POSTERIOR OVERLAPPING
Better reconstruction loss, but larger KL divergence In Table 6 ELBO and reconstruction loss ("recons", which is ① in Bq. (5) of ResnetVAR with RealNVP prior are substantially higher

---

Under review conference paper ICLR 2020
Table 6: Average test ELBO ("elbo"), reconstruction loss ("recons"), Exy (x) DEL (x(2)()))pp ("kl"), and Ex (x) (RIl(s(22 (z)xx) ("Ailaj" ") of Resnet with different priors.
standard
RealNVP p(z)
Datasets
elbo
recons
kl
klaj
elbo
recons
kl
k/u/1
StaticMNIST
-87.61
-60.09
27.52
4.67
-82.85
-54.32
28.54
2.87
MNIST
-84.62
-58.70
25.92
3.55
-80.34
-53.64
26.70
1.76
FashionMNIST
-228.91
-20..94
19.96
2.74
-225.97
-204.66
21.31
1.88
Omniglot
-104.87
-66.98
37.89
7.88
-99.60
-61.21
38.39
5.99
Standard
RealNVP p(z)
0.00 0.25 0.50
0.75
1.00
1
2
3
4
5
5
10
15
20
25
Figure 3: Histograms of: (left) per-dimensional stds go x): (middle) distances between closest pairs of 4o (z|x);; and (right) normalized distances. See Appendix B.4 for formulation.
than standard ResnetVAE, just as the trend of test likeeiill (LL) in Table 3 Metrics of other models are in Tables B.16 to B.19 On the contrary, Ep- DKL. (9x (z) (z(x)]]px (z)) ("'I",, which is ② + ) are larger. Since ELBO equals to 1 @@ + @). this suggests that in our experiments, the improvement in ELBO (and also NLL) of ResnetVAE with RealNVP prior all comes from the improved reconstruction loss. This serves as foundation for the analysis in Section
can happen to be smaller when recons gets larger (e.g., our DenseVAA Table B.16 and VAE with dense layers of (Bauer & Mnih 2019, Table 2)). Nevertheless, all our experiments with Real-
NVP prior show improved reconstruction loss, which validates the analysis of Section 4
Smaller standard deviation of Gaussian posterior with ReaINVP prior In Fig. we plot the histograms of per-dimensional stds of go (z|x),, as well as the distances and normalized distances (which is roughly distance/std) between cach closest pair of qo (z)x) (see Appendix B.4 for for- mulations). The stds of (z(x) with RealNVP prior substantially smaller, and the normalized distances are larger. Larger normalized distances indicate less density go x) to overlapping. This fact is direct evidence of the analysis in Section
The test NLL of the model and the learned std of go (z|x) is insensitive to the minimum clipping value under a certain threshold Sce Appendix [B66 for experiment results.
Appropriate overlapping among (z(x))) with learned prior go (z)x) with RealNVP prior are reduced according to the dissimilarity between rather than being reduced equally (i.e., (z(z)x) exhibits "appropriate overlapping"), we plot the interpolations of
demonstrate that stds
between the centers of go (z) x) two training points, and log (x( of these interpolations in Fig. 2 We visualize px (z), because it is trained to match (z), and can be computed much more reliable than (z); and because the density of (2(z) between corresponding
points be in- dicator of how (2(2) x) overlap between them. The learned RealNVP pa (z) scores interpolations of z between the centers of qo (z|x) of two training points, giving low likelihoods to hard-to-interpree interpolated samples between two dissimilar (the first three rows), while giving high likelihoods good quality samples between two similar x (the last three rows). In contrast, the unit Gaussian prior assigns high likelihoods to all interpolations, even to hard-to-inten to--tterrntrtttttt ones. This suggests that the posterior overlapping more appropriate with RealNVP prior than with unit Gaussian prior.
Learned prior inffuences the trade-off between reconstruction loss and KL divergence, suggested by Section 4 We plot the rate-distortion curve (RD curve) (Alemi et al. 2018) 3- ResnetVAE trained with different 3 and prior flow depth K in Fig. 4 Rate is ( (z))), while distortion is negative reconstruction loss. Each connected curve with the same shape points in Fig. correspond to the models with the same K, but different B. We can see that the curves K 1 is closer to the boundary formed by the green line and the & axes than K 0, while K 20 & 50 even closer. According to Alemi et al. 2018), points the RD curve being closer

---

Under review conference paper ICLR 2020
X
K-0
-4-K-1
K--2
K-50
K=0
+
K=l
-
K=20
K=50
150
10.0
125
100
2.0
nm diada
Distotion
30-
0.2
0-
125
180
022
133
05
1.0
2.0
5.0
10.0
Rate (KL)
beta
Figure 4: Rate (Dx))(a(()))))ll))) (z)) and dis- tortion (-reconstruction loss) of B-ResnetVAE trained with different and prior flow depth K.
Figure 5: Average normalized distance of 3- ResnetVAE trained with different B and prior flow depth K.
to the boundary suggests that the corresponding models are closer to the theoretical optimal models a particular dataset, when traded between reconstruction loss and KL divergence. Given this, we conclude that learned prior can lead to a "better" trade-off from the perspective of RD curve.
We also plotted the average normalized distance of (--ResetVAAE trained with different 3 and prior flow depth K in Fig. 5 The average std of ga(z x) can be found at Fig. B.6 Learned prior can encourage less posterior overlapping than unit Gaussian prior for various not only for 8
6 RELATED WORK
Learned priors, natural choice for the conditional priors of intermediate variables, have long been unintentionally used in hierarchical VABs (Rezende al. 2014 Sønderby
2016 Kingma
ct al. 2016 Maaløe et al. 2019). few works were proposed to enrich the prior, e.g., Gaussian mixture priors (Nalismick et al. 2016 Dilokthanakul et al. 2016 Bayesian non con-paramttric pri- Goyal al. (017), and regresssgee priors (Gulrajani al., 2017] CCen et al. 2017), without the awareness of its relationship with the aggregated posterior, until the analysis made by Hoffman & Johnson (2016). Since then, attempts have been made in matching the prior to aggreguted posterior, by using RealNVP (Huang al. 2017), variational mixture posteriors (Tomezak & Welling 2018), learned accept/reject sampling (Bauer Mnih (019), and kernel density trick (Takahashi et al. [2019). However, none of these works recognized the improved reconstruction loss induced by learned prior. Furthermore, they did not show that learned prior with just one latent variable can achieve comparable results to those many deep hicrarchical VAEs.
ors (Nalisnick & Smyth
The trade-off between reconstruction loss and KL divergence was discussed in the context of 3- VAE (Higgins et al.] [0]]]201)]||err) Alemi al.] [2018] Mathieu et al. (201)). however, they did not further discuss the impact of learned prior on this trade-off. |Mathieu et al. (2019) also discussed the posterior overlapping, but only within the 3- 3-VAE framework. thus was only able to control the degree of overlapping globally, without considering the local dissimilarity between
7 CONCLUSION
this paper, using learned RealNVP prior with just one latent variable in VAE, we managed to achieve test NLLs comparable very deep state-of-the-art hierarchical VAE, outperforming many previous works of complex hierarchical VAΣs equipped with rich priors/posteriors. We hypothesized that, when coupled with Gaussian posteriors, the learned prior can encourage appropriate posterior overlapping, which is likely to benefit the reconstruction loss and lower-bound, supported by our experimental results. We showed that with learned RealNVP prior, 3-VAE can have better rate- distortion curve (Alemi et al., (018) than with fixed Gaussian prior. We believe this paper is an important step towards shallow VAEs with leamed prior and a small number of latent variables, which potentially be more scalable large datasets than those deep hierarchical VAEs.

---

Under review conference paper ICLR 2020
REFERENCES
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy, Fixing a Broken ELBO. In International Conference on Machine Leurning, pp. 159-168, July 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701 07875, 2017.
Wasserstein gan. arXiv preprint
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 66-75, 2019.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. Generating Sentences from a Contimuous Space. In Proceedings of the Twentieth Conference on Computational Natural Language Learning (CoNLL),, 2016.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In Sth Interaational Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings, 2017.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv:1611.02644 2016.
Laurent Dinh, Jascha Sohl- Sohh-Dickstein, and Samy Bengio. Density estimation using real NVP, In Sth International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings, 2017.
Tan Goodfellow, Jean Fouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric P. Xing. Nonparametric vari- ational auto-encoders for hierarchical representation learning. In Proceedings of the IEEE Inter- national Conference on Computer Vision, pp. 5094-5102, 2017.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez, and Aaron C. Courville, Pixelvae: A latent variable model for natural images. In 5th Intema- fional Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, volume 3, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas. Yan Duan, and Pieter Abbeel. Flow++: Improving Flow- Based Generative Models with Variational Dequantization and Architecture Design. September 2018.
Matthew D. Hoffman and Matthew Johnson. Elbo surgery: Yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference. NIPS, 2016.
Chin-Wes Wei Huang, Ahmed Touati, Laurent Dinh, Michal Drozdzal, Mohammad Havaei, Laurent Charlin, and Aaron Courville. Learnable Explicit Density for Continuous Latent Space and Vari- ational Inference. arXiv:1710.02248 (cs, stat], October 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.

---

Under review conference paper ICLR 2020
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In Proceedings of the International Conference on Learning Represeniations, 2014.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Informa- tion Processing Systems, pp. 4743 4751, 2016.
Durk Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1xl Convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215-10224. Curran Associates, Inc., 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through prob- abilistic program induction. Science, 350(6266): 1338, December 2015. ISSN 0036-8075, 1095-9203. doi: 10.1 /26/ccience.aab3550
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 29-37, 2011.
Lars Maalee, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling. arXiv: [cx, stat), February 2019.
Emile Mathieu, Tom Rainforth, N. Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 4402 4412, 2019.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th Inter- national Conference Machine Learning-Voole Volume 70, pp. 2391-2400. JMLR. org, 2017.
Eric Nalisnick, Lars Hertel, and Padhraic Smyth. Approximate inference for deep latent gaussian mixtures. In NIPS Workshop on Bayesian Deep Learning, volume 2016.
Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders. Sth Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Danilo Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Pro- ceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1530-1538, 2015.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. arXiv: 1810.00597 [cs, stat], October 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning Volume 32, ICML'114, pp. 11-822, Beijing, China, 2014. JMLR.org.
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution Matching in Varia- tional Inference. arXiv preprint 1oXXiv/882.06883 2018.
Ruslan Salakhutdinov and lain Murray. On the quantitative analysis of deep belief networks. In Proceedings of the 25th International Conference on Machine Learning, pp. 879. ACM, 2008
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1218-1226, 2015.
10

---

Under review conference paper ICLR 2020
Casper Kaae Sønderby, Tapani Raiko, Lars Maalde, Søren Kaae Sønderby, and Ole Winther, Ladder variational autoencoders. In Advances in Neural Information Processing Systems, pp. 3738-3748 2016.
Hiroshi Takahashi, Tomoharu Twata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Vari- ational autoencoder with implicit optimal priors. In The Thirty-Third AAAI Conference Ar- tificial Intelligence, AAAI 2019, Honolula, Hawaii, USA, January 27 February 1, 2019., pp. 5066-5073, 2019.
Jakub Tomezak and Max Welling. VAE with VampPrior. In International Conference on Artificial Intelligence and Statistics, pp. 1214-1223, 2018.
Dustin Tran, Rajesh Ranganath, and David M. Blei. Variational gaussian process. In 4th Interna tional Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling. Sylvester nor- malizing flows for variational inference. Proceedings of the Thirty-Fourth Conference on Un- certainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 393-402, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: Novel Image Dataset for Bench- marking Machine Learning Algorithms. arXiv:1708.07747 [cs, stat), August 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016, 2016.

---

Under review conference paper ICLR 2020
A PROOF FOR PROPOSITION
We first re-state Proposition
!
Proposition. Given finite number of discrete training data, i.e, p²()x) + 0(x 6(x- x())
if Pa (xiz) = (eemoull((pg(2) where the Bernoulli mean po(z) is produced by the decoder, and 0 < (z) < 1 for each of its k-th dimensional output, then the optimal decoder Mu(z) is:
where (z) 40 (o(z)x(1)
(z(2)=
Σ w;(2)x() x
and E wx(2) =1
9,90 (0(2))))
We shall need the Euler's equation for the calculus of variations (see Gelfand Silverman (2000 page 15, 24, and 35)): Theorem A.1. Let [[f be a functional of the form:
[lfl LLf=-fa F(. f. dr
defined the set of functions f(x) which satisfies the boundary conditions y(a) A. y(b) B,
then a necessary condition for [( to have an extremum for a given function f(x) is that. f(x)
sutisfy Euler's equation (Geljand & Silverman) 2000 page 15);
OF
OF dx 0(0.f)
(A.1)
0f
where we use Ogf to denote
4f de
For multi-variate function f(11, 322) and the functional:
f[f)= = J/ F( x2, f, dz, f. O2g da dx2
where R. is a closed area, the Fuler's equation becomes (Gelfand & Silverman 2000 page 24):
OF
OF dr f( Orz 0(0,,f)
OF
(A.2)
0f
For vector value function f(x) if (x),
(x)) f and the functional:
[( = [ F(a,J" 8",, Sm, 8 dx
a necessary condition for f(z) to be an extremal of [( is that, every (x) satisfies Euler's equa- lion (Gelfand & Silverman 2000 page 35).
With some straightforward extensions, can also get to the Euler's equation for multi-variate vector value functions.
The original Fuler's equation requires integral to be definite, whereas a neural network is defined on the whole I²" space. However, in practice, the input values (i.e., the training and test data) are usually within limited range, and the norm of the neural network parameters not too large. As result, there should exist sufficiently large number R, such that intermediate outputs are within the closed area [- R, RJ", In this sense, the definite and indefinite integral would not have too much difference. We shall always the indefinite integral notations in the following proof.
Finally, we get the proof of Proposition]
|
Proof. To apply calculus variations, we need to substitute the parameterized, bounded (z)(2) with a non-parameterized, unbounded mapping. Since 0 < (²(x)) < 1, and (z(()) is produced by neural network. which ensures (z(z)) (z) is continuous mapping. then V0, there exists unbounded t(z).
exp((*(z)
(z(2) + (x)
(*(2) log (z)(2) log(1 (z))
12

---

Under review conference paper ICLR 2020
and for all continuous mapping t(z), there also exists (z(z), satisfying the above equations. fact, this substitution is also adopted in the actual implementation of our models.
The probability of pe (x(z) is given by:
(=(x ((rno II (1ā(z)))* (1 nb (1-ak)
Then we have:
log (x(()))
Σ nu log (b(()) -1- (1- (-(()(2)) }
Σ ah (x(2) xs log [I - exp(t (x))] - (1 log [I + (x)({ }
(xaa(()))) (ifa)-loog log [1 cx)(((
The training objective & can be then formulated as a functional on t(z):
Po(z)
[] Ey (x) Foo(a(x) loggo(x(2) + log g x)
If p² (xx go(2)x) (10gpo(x(2) + log po(z)
dz dx
90 (
/ F(z, tz
where F(z, t) is:
F(2,t) = / px (x)
Σ (z) log (x)((((2))] log po(2) dx
(z(x)))
According to Buler's equation (Gelfand Silverman 2000 page 14 and 35), the necessary condition for C[t] to have an extremum for given t(z) is that, t(z) satisfies aF = 0, vk. Thus we have:
OF
/ (x(x) 1+ exp(tt (f((())
(xp((+(2))
Ot = 0
dx = 0
/ [x(x))((z)1) 10X -nā(xi) dx 0
Exlexx(") (2 - nă ud(z))
E 216)) = -(_ (o)())))))) u mg(z)
E, (2| 2
mz(
E, (66(2(x(1))
That is say, (z)())
> (x)
E, x((()x(1)
2, no (x/pet)))
Rezende Viola 2018 has proved that when po (x/2) = N ((g(())) where a a global fixed constant, the optimal decoder (z( = Σ, wx(z) x(1), which is exactly the same as our conclu- sion. Rosca et al. (2018) has proved that the gradient of Bernoulli (1gii(4ggl (z)) is the same as that NV (z), o when o 1, but they did not calculate out the optimal decoder. We push forward both these works.
13

---

Under review conference paper ICLR 2020
B EXPERIMENTAL DETAILS
B.I DATASETS
MNIST MNIST is 28x28 grayscale image dataset of hand-written digits, with 60,000 data points for training and 10,000 for testing. When validation is required for early-stopping, we randomly split the training data into 50,000 for training and 10,000 for validation.
Since we use Bernoulli pa (x|z) to model these images in VAE, we binarize these images by the method in (Salakhutdinov & Murray, 2008): each pixel value is randomly set to I in propertion its pixel intensity. The training and validation images are re-binarized at each epoch. However, the test images are binarized beforehand for all experiments. We binarize each test image 10 times, and all these 10 binarized data points in evaluation. This method results in 10 times larger test set, but we believe this can help us to obtain a more objective evaluation result.
StaticMNIST StaticMNIST (Larochelle & Murray, 2011 is pre- pre-binarized MNIST image dataset, with the original 60,000 training data already splitted into 50,000 for training and 10,000 for validation. We always use validation set for early-stopping on StaticMNIST. Meanwhile, since StaticMNIST has already been binarized, the test set is used as-is without 10x enlargement.
MNSSS FaShionMNIST (Xiao et al. 2017) is a recently proposed image datasct of grayscale fashion products, with the same specification as MNIST. We thus use the same training- validation split and the same binarization method just as MNIST.
Omniglot Omniglot (Lake et al. 2015) a 28x28 grayscale image dataset of hand-written char- acters. We use the preprocessed data from (Burda et al. (016), with 24,345 data points for training and 8.070 for testing. When validation required, we randomly split the training data into 20,345 for training and 4,000 for validation. We use dynamic binarization on Omniglot just as MNIST.
B.2 NETWORK ARCHITECTURES
Notations In order to describe the detailed architecture of our models, we will introduce auxil- iary functions denote network components. A function ha(x) should denote sub-network in go (o(2) x), with a subset of 6 as its own learnable parameters. For example, if we write go (2(2))) go (z) (x)) - (ha o (x)(x)) I), it means that the the posterior go(z x)) is Gaussian, whose mean and standard deviation are derived by one shared sub-network hep (x) and two separate sub-networks (-) and (-)., respectively.
The structure a network is described by composition of elementary neural network layers.
Lincar(k) [t] indicates a linear dense layer with outputs. Dense[k) indicates a non-linear dense layer. + b indicates composition of and b, egg. Dense[m) - Dense(n) indicates two successive dense layers, where the first layer has m outputs and the second layer has outputs. These two dense layers can also be abbreviated as Dense -> rr].
Conv H x W CJ denotes a non- non-linear convolution layer, whose output shape is H x W x C, where H is the height, W is the width and C is the channel size. As an abbreviation, Conv H, wW xC1 + H2 x W₂ x C2 denotes two successive non-linear convolution layers. Resnet[-] denotes non-linear resnet layer(s) (Zagoruyko & Komodakis (016). All Conv and Resnet layers by default use 3x3 kernels, unless the kernel size specified subscript (e.g., Convix1 denotes Ixl convolution layer). The strides Conv and Resnet. layers are automatically determined by the input and output shapes, which is 2 in most cases. LinearConv [:] denotes linear convolution layer(s).
DcConv[] and DeResnet(f [-] denote deconvolution and deconvolutional resnet layers, respectively.
PixcICNN[]] || is PixelCNN layer proposed by Salimans et al. (2017). It uses resnet layers, instead of convolution layers. Details can be found its original paper.
Flatten indicates to reshape the input 3-d tensor into a vector, while UnFlatten H x W x C) indicates to reshape the input vector into a 3-d tensor of shape H x W x C. Concat(a, b) indicates concat the output of and along the last axis.
14

---

Under review conference paper ICLR 2020
CouplingLayer and ActNorm are components of RealNVP, proposed by Dinh et al. (2017): Kingma & Dhariwal (2018). InvertibleDense is component modified from invertible Ixl con- volution (Kingma & Dhariwal 2018). We shall only introduce the details of CouplingLayer since it contains sub-networks, while the rest two are just simple components.
General configurations All non-linear layers use leaky relu (Maas et al. 2013) activation.
The ohserved variable (ie. the input image) is always tensor, shape xW xC, whether or not the model is convolutional. The latent variable is vector, whose number of dimensions chosen be 40 MNIST and StaticMNIST. while 64 on FashionMNIST and Omniglot. This is because we think the latter two datasets are conceptually more complicated than the former two, thus requiring higher dimensional latent variables.
The Gaussian posterior 4o (z|x) is derived as:
(x(z)x) = N (hs ( (x)) hx(x) Linear [Dim(z)] log a (x) hx(x) Linear [Zim(z)]
Note make the network produce log or (x)) instead of directly producing (x)))())).
(x) is the hidden layers, varying among different models.
For binarized images, we use Bernoulli conditional distribution po z), derived as:
(x(x = Bernoull((4,
For Dense VAE:
ng (ho (z))z))))))))))))))) hz(z) - Linear[784 + UnFlatten(28 x 28 x
log
(z))
And for others (f.e., VAE, ResnetVAE and PixeIVAE):
log (o(h)())) ho(2) + LinearConvixi x 28 x []] (ho()))))))
Note we make the network to produce log (e(heiz)]
(4)1)))))) the logits of Bernoulli distribution, instead of producing the Bernoulli mean (h)(())) directly.
Dense VAE hx(x) and hz(2) of Dense VAE are composed of dense layers, formulated as:
(x) Flatten Dense[500 500]
ho(z) z Dense[500 -> 500]
Conv/ResnetVAE (x) and (z) of Conv are composed (de)convolutional layers, while those of ResnetVAE consist of (deconvolutional) resnet layers. We only describe the architecture
of Resnet here. The structure of CAE can be casily obtained by replacing all (deconvolu- tional) resnet layers with (de)convolution layers:
(x(x)= = x + Resnet[28 x
28 x
32 +
28 x 28 x 32
X 64 + 14 x 14 64
64 7 x x 16)
Flatten
ha(z) = Dense [84] + UnFlatten [7 x x 16)
- Deltesnel/7 x x 64 + 14 x 14 x 64
+ 14 x 14 x 64 - 28 x 28 x 32
28 x 28 32]
15

---

Under review conference paper ICLR 2020
PixeIVAE (x) PixelVAE is exactly the same as ResnetVAE, while (z) is derived as:
hz(z) Concat xx (z))
- PixelCNN 28 x 28 33 - x 33
- 28 x 28 x 33)
ho(z) 2 + Dense 784 + UnFlatten 7 x 7 x 16]
DeReemett 7
64 + 14 14 64
14 64
32
x 28 32]
As Salimans et al. 20177) we use dropout in PixeICNN layers, with rate 0.5.
RealNVP The RealNVP consists of K blocks, while each block consists of an invertible dense, a coupling layer, and an actnorm. The RealNVP mapping for prior, i.e. fx [x),, can be formulated as:
[x(2) = + Ji (h,) - ...... (bx) Sb(hg) = h2 - Invertiblel Dense > Coupling - ActNorm
CouplingLayer(() Concat ur, u, @ Sigmoid (sx(hx,(())) tx (x(hx,(ui))
hx,((u) in Dense|250
(x,)(u) - Linear (u,)
ta (1 hx.x h + Linear[Dim(u,))
where U. [ii(((u])) is the left half of u, and (Dim(u)/2|--inniii is the right half.
The RealNVP posterior, derived from the original Gaussian posterior Go (w(x), is denoted 90.n (2)x). and formulated as:
(w)())
Ca.n (z)x) = 26 (w/x))
(B.1)
Ow
z= (w)
where structure of the RealNVP mapping (w) for posterior exactly the same as [x(2) for prior. The ELBO for VAE with RealNVP posterior is then simply:
C(, 0, (x)(x) (w/x))
(w/x)
logpo(x(f((w)) + log Px (fr()))
(55(w)
-o + log det
(B.2)
dw
B.3 ADDITIONAL DETAILS OF TRAINING AND EVALUATION
General methodology All the mathematical expressions of expectations are computed by Monte Carlo integration. For example, (xs()))) [f x)] is estimated by:
distributions
B [f x)] R
f(2(0,
where (4) is sample from (x)x))
Training use Adam (Kingma Ba 2015) to train our models 2,400 epochs. The batch size is 128 for VAn, VAT and Omniglot, set the learning rate to be 10 in the first 800 epochs. 10 10-4 in the next 800 epochs, and 10-5 the last 800 epochs. On StaticMNIST, we set the learning rate be 10-4 in the first 1,600 epochs, and 10-5 in the last 800 epochs.
VAE and ResneiVAF and 64 for PixeIVAF. On MNIST, FaST
L2 regularization with factor 10 is applied on weights of all non linein hidden layers, i.e., kernels of non-linear dense layers and convolutional layers, in (x),, ho ho()), (x(z) and f, (w).
16

---

Under review conference paper ICLR 2020
The ELBO is estimated by z sample for each x in training. We adopt warm-up (KL anneal-
ing) (Bowman et al. 2016). The ELBO using warm-up is formulated as:
[(x: A, 8, Ø) (x|x)) [log pa (x( (x + 3 (logPx(z) - log (x(x))))
3 is increased from 0.01 to linearly in the first 100 epochs, and it remains afterwards. The
warm-up ELBO for VAEs with RealNVP priors and posteriors can be obtained by replacing px(z)
and qe (2|x) of the above equation by Eq. 6 and Eq. (B.I). respectively.
We adopt carly-stopping using NLL on validation set, to prevent over-fitting on StaticMNIST and PixeIVAE. The validation NLL is estimated using 100 samples for each x, every 20 epochs.
Training strategies for (z) We consider three training strategies for optimizing Eq. 7
Post-hoe training (Bauer & Mnih, 2019): qa (z|x) and po (x z.) are firstly trained w.z.f the unit Gaussian prior, then go (z) x) and po (z) are fixed and px(z) is in turn optimized. This is the most intuitive training method according to Eq. (5). however, it does not work as well as joint training in terms of test negative log-likelihood (NLL), which is observed both in our experiments (Table B.6) and by Bauer & Mnih (2019).
Joint training (Tomczak & Welling 2018 Bauer & Mnih 2019): : px(z) is jointly trained
along with (o( and (x()))) by directly maximizing Eq. 77
Iterative training: Proposed by us, we alternate between training pe (x(x(z) & qo (z(2))) and training px (z), for multiple iterations. The first iteration to train pa (x|z) & 9a (z(x)) should use the unit Gaussian prior. Early-stopping should be performed during the whole process if necessary. See Algorithm [or detailed procedure of this strategy. We adopt this method mainly for investigating why post-hoc training does not work as well as joint training.
To train (x) with post-hoc strategy, we start from a trained VAE, adding RealNVP prior onto it, and optimizing the RealNVP fx(z) for 3,200 epochs, with learning rate set to 10-3 in the first 1,600 epochs, and 10-1 in the final 1,600 epochs.
Algorithm B.1 Pseudocode for iterative training.
Iteration 1a: Train go (z) x) and px(1)z), with (z)z) (0,
Iteration 1b: Train (z) for 2M epochs, with fixed q6 x) and (x(2)))
fori=2.... I do
Iteration ia: Train go (z(x))) and po for M epochs, with fixed pa (z). Iteration ib: Train (z( for M epochs, with fixed qo (z)x) and pa (x (((2).
end for
Algorithm B.I the pseudocode of iterative training strategy. For Iteration la, all
are the same with training standard VAE, where particular, the number of training epochs set to 2.400. For Iteration 1b, the learning rate is 10-3 for the first M epochs, and is 10 for the next M epochs. For all the next iterations, learning rate is always 10-1 The number of iterations I is chosen to be 16, and the number of epochs M is chosen to be 100, for MNIST, FashionMNIST and Omniglot. For StaticMNIST, we find it overfits after only a few iterations, thus we choose I be 4, and M to be 400. With these hyper-parametesd qo (z(xx) pa(x z)) and px (z( are iteratively trained for totally 3,200 epochs on all datasets (starting from Iteration lb), after the pre-training step (Iteration la).
Regularization term for (z) In order to compare some metrics (e.g. the active units) of VAE+ + RealNVP prior with those of standard VAE, we introduce additional regularization term for go (z). such that (z( of VAE using RealNVP prior would have roughly zero mean and unit variance, just as a standard VAE with unit Gaussian prior. The regularization term for (z) (denoted as Reg (z)) and the final training objective augmented with the regularization term (denoted [xx; X, 0, Φ)) are:
Dim(x) Σ
Reg [zs( Dim
(Mean(zu)) + (Var(zu) 1)²]
1=1
Ż(A, 0, (x) L(x; X, 0 Reg (2)(
17

---

Under review conference paper ICLR 2020
where 24 is the k-th dimension of z, (im(z) is the number of dimensions, Mcan(zs) (x) (x [=x] and Var(za) = Bpr (x)) (abb)) (** - Mcca(=x)) are the mean and variance each dimension.
Table B.1: Avg. M2an(z+) and Var(z+) of pagllarizeduun-regulaaiaze ResnetVAE with RealNVP prior.
regularized
un-regularized
Datasets
Avg. Me)
Avg. Var(ea)
Avg. (ea
Avg. Var(za)
StaticMNIST
-0.02 = 0.03
0.93 + 0.01
0.05 + 0.37
1.50 + 0.02
MNIST
0.00 + 0.00
0.98 + 0.01
-0.01 + 0,02
0.78 + 0.04
MNIST
0.00 + 0.00
0.98 0.00
0.03 + 0.03
0.83 + 0.04
Omniglot
0.00 = 0.02
0.95 = 0.00
0.00 + 0.01
0.24 - 0.01
Table B.1 shows the average Mean(za) and Var(=x) of ResnetVAE with ReaINVP prior, computed on test data. Average Mean(zz) is defined as Dim(z) DD Mean(ax); while average Var(zz)
_bim(() Var(zu). The means and standard deviations of the above table are com- defined as Dim(z) _k=1
puted w.t.f. repeated experiments. Using the regularization term for qo (z( makes the Mean(2y and Var(z:) of z samples close to (o0, D). which is in sharp contrast with the un-regularized case. For
fair comparison in Table ( and Fig. 3 it is crucial to have Mean(zy, (z.)) and Var(*k) close to (O, I).
Test NLLs are not reported here, because we find no significant difference between regularized and un-regularized models in terms of test NLL.s
Evaluation
The negative log-likelihood (NLL), the reconstruction loss and the KL divergence
Exp(s) DKL (z)(x))xx)()) are estimated with 1,000 samples from (z(x((()) for each x(6) from test data:
NLL
1 N
ExgM
1000
[og
+ (z (z(())
log (z (z(3))) x())
N 1090
Reconstruction Loss
Σ
log (x [11)))]
1000.V
==1
N
1000
(x)) (wl(go(2))l))))
Σ
log (2
log (2(60)))))
1000N
j=1
where each ( is one sample from (z(x)x(())
Ex [f(x(0), [1:)]]] is defined as:
LogMeanExpy'z1 f(x!") z ](:0)]] fmax log L Σ /exp [fxx(0, fmaa))
where fmou = max, [(x(1), (1.)))
We use 1,000 samples, because the two major previous works of us (i.e., Tomezak & Welling 2018); Bauer & Mnih (2019)) both used this number samples. It is true that some previous works may have used substantially larger number of samples (e.g., [Chen et al. (2017) used 4,096 samples), and according to (Grosse al., 2015), the Monte Carlo estimator for NLL is a stochastic lower- bound, such that larger number
samples typically would lead to lower (yet better) estimated
NLLs. However, in order to have fair comparison with (Tomezak & Welling 2018 Bauer & Mnih 2019), decide to stick to 1,000 samples.
Active units active unity BBrrda al. 2016) is defined the number of latent dimensions whose variance is larger than 0.01. The variance the k-th dimension is formulated as:
Varg (x0) (xle) [=e])
18

---

Under review conference paper ICLR 2020
where by is the k-th dimension of compute Var, on the training data, while the inner expec- tation By (xpp) [zu] is estimated by drawing 1,000 samples of for each x.
B.4 FORMULATION OF CLOSEST PAIRS OF (0(x)x) AND OTHERS
Closest pairs of (a(z)x) For each x(1 from training data, find the training point x(),, whose posterior (z(x)(()) is the closest neighbor to (z(z)x())
j= a-arg arngmn ( M (x(()))
where (x) is the mean of (z(x))), and || || the L2 norm. These two posteriors (z(z)x()) and 2o (1) are called a closest pair of go (z)x).
Distance of pair of go(2(x) The distance di of a closest pair go(2|x(0) and qo (z(x((())) is:
d = (x(((()) (x(x(())
d, = ||d
Normalized distance of pair of (z(x))) For each closest pair (x(2|x(1) and (o(2|x(1)), we
compute its normalized distance dij by:
2dg
des Std [i j;
(B.3)
Sud[i;]] is formulated as:
di
Std[i: Var an (x/be(1))
(z 10--alls Mo(x(())
(B.4)
deg
where (z (f(z)) is the variance of (z) w.r.t. (z)x)(()) We use 1,000 samples to estimate each Std[i; j]. Roughly speaking, the normalized distance diy can be viewed as "distance/std" along the direction of dij. which indicates the scale of the "hole" between go (z(x)(()) x()) and go g0 (z))
B.5 EXPERIMENT RESULTS ON CIFAR10
We conduct experiments Cifarl0 Krizhevsky 2009). Although the results on Cifar 10 are good as the state-of-the-art deep hicrarchical VAEs, using RealNVP prior also shows great improve- ment on test NL..s and sampling quality over Gaussian prior, with just one latent variable.
Architectures of ResnetVAE for Cifar10 The latent variable is 3-d tensor, with shape 8 x 8 32. The Gaussian posterior 9o (2(x) is derived as:
40 (2(x) = A (H e (h (x)) (x))
(x)( (x) + LinearConv
32]
log p (hx(x)) (x(x) + LinearConvix [ 8 32]
hx(x)= = + Resnet x 32 96 32
x 32
32
+ x 16 192 + 16 x 16 x 192 - 16 x 16 x 192
+ 8 x 8 256 + 8 8 8 256]
The generative distribution Pe (x(2) is derived as: (x[2)) = Discretizedllogistiic (14, (ho( (2)) So (z)( An + LinearConvixl [32 x 32 3]
log (b)( (z) + LinearConv 1x1 [32 32 3]
he (z) z + DcResnet(S 8 256
256
+ 16 x 16 192 - 16 16 x 192 - 16 x 16 x 192
+ 32 x 32 96 + 32 x 32 x 96]
where DiscretizedLogistic is the discretized logistic distribution proposed by Kingma et al. (2016).
Note also clip log 8o (z)(x)) by -1 just the for the log-std of (x|xx)
19

---

Under review conference paper ICLR 2020
Table B.2: Test bpd on Cifar 10.
Model
bpd
Models without PixelCNN decoder
VAE IAF (Kingma 2016)
3.11
BIVA (Maalee et al. 2019
3.08
Resnet VAT
3,98
ResnetVAE RealNVP p(z). K 50
3.54
Models with PixeICNN decoder
VLAB (Chn et al. 2017)
2.95
(a) Gaussian p(z)
(b) RealNVP p(z)
Figure B.I: Samples of: (a) ResnetVAE with Gaussian p(=); and (b) ResnetVAE with RealNVP p(z). Both models use just one latent variable of shape 8 x x 32.
RealNVP prior Cifar consists of blocks, while each block consists of an invertible /xl conv (Kingma & Dhariwal 2018), coupling layer, and an actormm which can be formulated
RealNVP prior
fx(z)= = fi (hr) + ..... fx (bx) fa(hg) = h2 - InvertibleConv - Couplinglayer > ActNorm CouplingLayer(1) = Concat(u, [ur, u, O Sigmoid (hx, (x)x)
hx,(uu) u, Resnet(8 x 8 64 x 8 64]
sx(hxx(u)) hx,((u) - LincarConv 1x1 [8 (x)))(())) (u.)(u)) + LinearConv 1x1 [8 x x 32]
32]
where uy 0, (Dmm(u)))) is the left half channels of and (2n is the
right half channels.
Experiment results Table B.2 shows the test bpd of our ResnetVAE on Cifar10. Although the test bpd of our model is still far from those of deep hierarchical models, it is worth mention that BIVA used 15 latent variables to reach the state-of-the-art. Our experiments show that, adding a deep RealNVP prior can make the test bpd much better, even with just one latent variable. This suggests that complicated prior can be very effective for reaching good test bpd, and is potentially helpful for reducing the depth of hierarchical VAEs.
20

---

Under review conference paper ICLR 2020
Re
Gaussian ----min
std
2
1
0
NLL
150
100
min
logstd
The average learned std Gaussian ga x) and the test NLL ResnetVAE with Real- NVP and Gaussian p(=) on MNIST. We also plot "min std" as reference, the minimum attainable std clipped by the configured mininum logstd.
Figure B.2:
The most surprising fact is that, in our experiments, it is not possible to obtain any meaningful samples using ResnetVAE with Gaussian p(=). In contrary, the samples obtained using ResnetVAE with RealNVP p(z) is much more natural images.
B.6 IMPACT THE MINIMUM CLIPPING VALUE FOR THE STD OF (z(x))
We have mentioned in Section 5.1 that clip the std Gaussian posterior (z(xx)) minimum value of e-11 2 1.67 10- so to avoid unstable training. The clipping is actually done by directly clipping the linear activation of log-std of go(2 x) by 11, and then obtain std, instead of clipping the std itself. Formally, the clipped log-std and the std are derived by:
logstd maximum (o=(Se(2)), 111
std exp (logstd)
where (2) is the hidden feature network for (z(x))). and (.(-) is a linear fully-connected layer. We the clipped log-std directly instead of the std whenever possible in computing the log- likelihood of Gaussian qo (2)x). which can be formulated as:
log qo (2)x)) = log(2r) logstd 2 exp (llogsts)) (x (o((o(2))))
where (z(z) a linear fully conte-co layer, producing the mean go (z)x). Std clipping is widely used in many open-source implementations of VATs. However, this trick is seldom discussed in published papers, which might suggest more theoretical work should be done on this topic.
The clipping value -11 our experiments not chosen for better model performance. The term exp (-2logstd) will have maximum value of approximately 3.58 x 10" with this value, which we think is safe enough for not having NaNs in training, considering that mainly use float32 in neural network implementation.
In fact, we find that the test NLL the model and the learned std go (2(x) is insensitive to the clipping value under certain threshold, which is demonstrated in Fig. B.2 and Table B.3 As long the minimum std small enough, both the RealNVP prior and the Gaussian prior could induce adaptive learned std of Gaussian posterior go (z x). One surprising fact is that, there a turning point where the clipping value is not small enough yet (i.e., min logstd being -3 3 for RealNVP p(2), and -I for Gaussian p(z)), the final learned std smaller than the learned std with smaller clipping values. The cause of this phenomenon is not analyzed in our paper, and we leave it as future work.
21

---

Under review conference paper ICLR 2020
Table B.3: The minimum std ("min_steé average learned std of Gaussian g. (z(x)) ("std"), " the ratio
minimum std with respect to the learned std ("min std"), test NLL and the active units ("au") of ResnetVAE with RealNVP p(=) and Gaussian p(2) on MNIST.
Resnet with
min logstd
min std
std
min std / std
NLL
RealNVP p(z)
2.7183
2.7183
1.0000
78.61
40
1.0000
1.0000
1.0000
78.57
40
-1
0,3679
0.3679
1.0000
78.58
40
-2
0.1353
0.1353
1.0000
78.57
40
-3
0.0498
0.0500
0,9967
78.59
37
-5
0.0067
0.1796
0.0375
78.56
40
-8
0.0003
0.1662
0.0020
78.56
40
-11
0.0000
0.1750
0,0001
78.60
40
-14
0.0000
0.1791
0.0000
78.56
40
-x
0.0000
0.1693
0.0000
78.58
40
Normal
1
2.7183
2.7183
1.0000
184.16
40
0
1.0000
1,0000
1.0000
94.46
40
0.3679
0.5890
0.6246
84.03
-2 0.1353
0.6839
0.1979
81.75
25
-3
0.0498
0.6877
0.0724
81.07
25
0.0067
0.6882
0.0098
81.05
26
-8
0.0003
0.6871
0.0005
81.04
25
-11
0.0000
0.6860
0.0000
81.03
26
-14 0.0000
0.6896
0.0000
81.06
-x
0.0000
0.6873
0.0000
81.08
26
B.7 COMPARISON OF OUR WORK THE PREVIOUS PAPER USING REALNVP PRIOR
The arperpparameters of Huang
2017 are quite different from ours: they only used layers of ResNet blocks in their qe (z and 4 layers in their Po (x/z). Also, they trained their ResConv VAE StaticMNIST only for 300 epochs, using initial learning rate of 0.0005 without learning In addition, we adopt invertible dense and actnorm our RealNVP flow, while they just used the original RealNVP flow. Most parts of our Resnet with ReaINVP prior have larger complexity than the ResConv VAE of Huang et al. (2017), expect that they used hidden layers in each coupling layer, while we just used
rate annealing, and without validation
Table B.4: Test NLL of our ResnetVAE with low complexity and ResConv VAE (Huang al. 2017) on StaticMNIST. "pure" refers the original RealNVP flow, without invertible dense and actnorm.
p(z) flow
(=(x)) flow
Early stopping
Model
Epochs
Initial Ir
NLL
Our ResnetVAE Low Complexity
pure, K 12
300
0.0005
81.05
pure, K
pure, K
300
0.0005
no 80.88
K 12
300
0.0005
no 80.67
K =
K =
300
0.0005
80.66
ResConv VAE
pure, K = 12 pure, K
300
0,0005
81.44
Huang al. 2017)
pure, K
300
0.0005
no
80.81
By removing the invertible dense and actnorm components from the RealNVP prior, enforcing the encoder and decoder to have similar model complexity, using the same number training epochs and learning rate, and not using validation set to do carly-stopping, we obtained similar results as of "Ours"). Adding invertible dense and actnorm (the third
Huang al. (2017) (the first fourth rows of "Ours") will improve the test NLL, suggesting that the invertible dense actnorm can indeed bring benefit to the model performance.
According to the above results. it is reasonable to believe that, it is the deeper RealNVP prior, more complicated model architecture, and longer training time, that give better test NL.I than Huang

---

Under review conference paper ICLR 2020
al. (2017) conclusion, there may not too many differences between our model and that of Huang et al. (2017). However, we tune the hyper-parameters much better, and demonstrate that Resnet with RealNVP prior and just one latent variable can work as well as many deep hierar- chical VAEs on MNIST-like datasets. The additional analysis should also be our novel contribution, when compared to the work of Huang et al. (2017).
B.8 DISCUSSIONS ABOUT (k..(ss<))llx(()))
The KL divergence between the aggregated posterior and the prior, i.e. DKL. (2) (z)). ). is first
analyzed by Hofiman & Johnson (016) as one component of the ELBO decomposition (5). Since (z) (z(2 (x) dx and (2 = ps(z/x) (x) dx. Rosca al. (2018) used this KL divergence a metric to quantify the approximation quality of both po(x) to p (x), and (z|x) to pa(z)x). As we are focusing on learning the prior, DxI((a. (z)(z))pxx22) can in turn be a metric for quantifying whether px (x(2) is close enough to the aggregated posterior (z(..
The evaluation of (g [z))), however, not an easy task. One way to estimate the KL divergence two arbitrary distributions the density ratio trick (Sugiyama et al. 2012 Mcscheder al. 2017). where Dxc(ge (z)llpx (z)) is estimated by separately trained neural network classifier. However, Rooca al. (2018) revealed that such approach can under-estimate (gx((((2) [px (z)), and the training may even diverge when z has hundreds of or more dimensions.
Another approach is to directly estimate (g.((gz()) (P)))x by Monte Carlo integration. The KL divergence can be rewritten into the following form:
(Ku(go(z) (p)xx)) (o(2) log (z) dz
(2(
= (allzx))p((x dx
log (0(22(x)) p² (x') dx'
dz
(x(z)
f (x) ʃ (o(xxxx)oo (qa(z)x')p"<x p" dx' dz
(2(2) (x) (e)x)) [log Exe las (z)x'] log)x(z)]
(B.5)
Rosca et al. 2018) has already proposed Monte Carlo based algorithm estimate DKI. (z)(z))px z)). in case that the training data is statically binarized. Since we mainly use dy- namically binarized datasets, we slightly modified the algorithm according to Eq. (B.5). to allow sampling multiple x from each image. Our algorithm is:
Surprisingly, we find that increasing n. will cause the estimated (za())) (x (z)) decrease on MNIST, see Table B.3 Given that our v2, = 10 for all nz, the number of our sampled z (even when R2 1) is 10x x 60, 000 = 6 10S, which should not be too small, since Rosca et al. 10" in their experiments. When n2. 8, the number of is Rince/Rosca
2018 only used
10 x 60, 000 is 4.8x larger than Rosca et al. (2018). not to mention the inner expectation Ep (go (z/x² x')] is estimated with 8x larger number of x There should be in total 38 larger number of log q (z|x') computed for estimating DKL go (z) (x))))(()) than Rosca et al. (2018), which has already costed about 3 days on 4 GTX 1080 Ti graphical cards. We helieve such a large number of Monte Carlo samples should be sufficient for any algorithm with good behavior.
4.8 10 which
Table B.5: (g) (z)) of a ResnetVAE with RealNVP prior trained on MNIST, estimated by Algorithm B.2 n2 = 10 for all ng.. We only tried n, for up to due to the growing computation time of (nn)).
R2
1
2
DKL (xx (Px
15.279
14.623
14.254
13.796
13.392
According to the above observation, we suspect there must be some flaw Algorithm B.2 of this, we do not adopt Algorithm B.2 to estimate (xx.(99(2)]lxxx(
Because
Since no mature method has been published to estimate DKL (as (z)(2) |I (z)). we decide not to use DRa. (gs (x)))xx(z)) to measure how our learned px (x) approximates the aggregated posterior (z))..
23

---

Under review conference paper ICLR 2020
Algorithm B.2 Pseudocode for estimating DxL. (20(2)]pp (2)) (denoted as marginal kl) on dynam- ically binarized dataset, where ja is the pixel intensities of each original image.
x_samples =
for μs in training dataset do for i ... R2 do sample x from Bernoulli(p)) append x_samples
end for end for marginal.kl=
for x samples
fori= n, do
sample from (z(2(x) posterior list []
for x' in x_samples do
append log (o(2(x)) posterior_list end for log (z(z) = LogMeenExppposterror Jist)
marginal.kl marginal log (z) log (z)
end for end for marginal kl munginalLk//len(xxxsamppp x n2)
B.9 ITERATIVE TRAINING CAN LEAD IMPROVED RECONSTRUCTION LOSS AND INCREASED ACTIVE UNITS
Table B.6 shows the NLLs of iterative training and post-hoc training with ResnetVAB (see also Although still comparable joint training, both methods can bring large im- provement in NLLs over standard VAE. Also, iterative training even further outperforms post-hoc training by large margin.
Appendix B.3
Table B.7 shows that, compared with post-hoc training, ilerative training can lead to larger recon- struction loss and increased number active units. Section 5.3 suggests that the learned prior can encourage better trade-off between reconstruction loss and the KL divergence, thus well-trained prior is essential for obtaining good (o(z) (x)x) and pa (x(z)) However, the prior is in turn determined by (z x) and zx()zz), according Eq. (5). Thus, it important alternate berween training Po (x(z) & go x) and training PA (z). until they catch up wtih each other and reach an equilibrium.
Considering these reasons, is not surprising why iterative training can result larger (yet better) reconstruction loss than post-hoc training. It is also clear why joint training works the best: because Po (x|22 & go (z) x), and px (2), are trained to catch up with each other every step. thus the training is more sufficient when using joint training than using the other two techniques.
24

---

Under review conference paper ICLR 2020
Table B.6: Test NLL of ResnetVAE, with prior trained by: joint training, ilerative training, post-hoc training, and standard VAE ("none") reference. Flow depth K 20.
Resnet
Datasets
joint
iterative
post-hoc
none
StaticMNIST
79.99
80.63
80.86
82.95
MNIST
78.58
79.61
79.90
81.07
FNSST
224.09
224.88
225.22
226.17
Omniglot 93.61
94.43
94.87
96.99
Table B.7: Average reconstruction loss ("recons"), DKL (z) (pxx))pppxx1 (z)) ("kF") and active units ("au") of ResnetVAE with iteratively trained and post-hoe trained RealNVP priors.
iterative
post-hoe
Datasets
recons
kl
au
recoRS
a
StaticMNIST
-58.0
26.4
38.7
-60.1
25.3
30
DynamicMNIST
-57.2
25.1
40
-58.7
24.7
25.3
FNsSIonMNIST
-207.8
19.4
64
-208.9
19.0
27
Omniglot
-63.3
37.9
64
-67.0
35.8
59.3
B.10
ADDITIONAL QUANTITATIVE RESULTS
Additional quantitative results are shown in Table B.8 to B.20
Table B.8: Test NLL on StaSTC and has the same meaning as Table
Model
NLL
Models without PixelCNN decoder
ConyHVAE + Lars prior!
(Bauer & Mnih
2019)
81.70
ConvHVAE VampPrior'
Tomezak & Welling 2018
81.09
ResConv + RealN'VP prior
Huang et al.
2017)
81.44
VAE IAFF (Kingma al.
2016)
79.88
BIVA MMallie et al.
2019
78.59
VAE+ RNVP pz)) K 50
80.09 + 0.01
ResnetVAE RNVP p22).
50
79.84 0.04
Models with PixelCNN decoder
VLAE (Chen et al. 2017)
79.03
PixelHVAE + VampPrion Prior" Tomezak Welling 2018
79.78
PixelVAE + RNVP (z)),
50
79.01 + 0.03
25

---

Under review conference paper ICLR 2020
Table B.9: Test NLL on MNIST. "t" and "I" has the same meaning Table
Model
NLI
Models without PixelCNN decoder
ConvHVAE Lars prior!
(Bauer & Mnih 2019)
80.30
ConvHVAE + VampPrior'
Tomezak & Welling 2018
79.75
VAB IAFF (Kingma al.
2016)
79.10 + 0.07
BIVA (Maalec et al. 2019
78.41
VOnY RNVI pz)) K
50
78.61 + 0.01
ResnetVAE RNVP p(z).
50
78.49 0.01
Models with PixelCNN decoder
VLABt (Chen et al. 2017))
78.53
PixeIVAF! (Gulaajani
2017)
79.02
PixelHVAE + VampPrior Tomezak & Welling 2018
78.45
PixelVAE + RNVP p(z).
50
78.12 0.04
Table B.10: Test NLL on Omniglot. "f" and "t" has the same meaning Table 1
Model
NLL
Models withou PixelCNN decoder
ConvHVAE + Lars priort (Bauer & Mnih 2019
97.08
ConvHVAE VampPrior" Tomezak & Welling 2018
97.56
BIVA= Maaloe et al. 2019
91.34
ConvVAE RNVI pz)). . K 50
93.62 + 0.02
ResnetVAE + RNVP p(z). K 50
93.52 0.02
Models with PixelCNN decoder
VLAB (Chen et al. 2017)
89.83
PixelHVAE + VampPrior (Toezakk Welling 2018)
89.76
PixelVAE + RNVP p(z),
50
89.60 + 0.01
Table 1.11: Test NLL on FashionMNIST. "I" and wt" has the same meaning as Table
Model
NLL
Models without PixelCNN decoder
ConvHVAE Lars priort (Bauer & Mnih 2019
225.92
Conv VAE RNVP p(z), R 50
224.64 + 0.01
ResnetVAR RNVP pz)) = 50
224.07 0.01
Models with PixelCNN decoder
PixeiVAE + RNVP pzz), K 50
223.36 0.06
B.11 ADDITIONAL QUALITATIVE RESULTS
Additional qualitative results are shown in Fig. B.3 to B.5
26

---

Under review conference paper ICLR 2020
9
4
9
7
4
3
7
7
0 2
5
4
(
7
9
5
5
0
6
9
8
2 2 2
)
7
7
3
2
Q
3
9
3
6
/
/
5 8 4
8
/
7
5
2
2
8
4
7
3
/
2
8
2
2
1
7
9
2
7
8
4 7 2 2 5 3 9 6
4
4
3
9
7
4
7
3
3
5
7
9
8 9 3 5
1
/
/
2
0
7
<
6
8
8
7
3
S
9 3 5 1 7 9
4
3
6
P
9
5
2
8
44
/ 0
/
459
/
4
7 5 9 D 4 7
M
3
0
5
/
0
1
2
a
0
2
3
0
0
U
b 7 5 8
/ p
a
8
4
1
2
9
0
0
5
6
/
6
2
3
0
1
4
7
3
3
6
9
0
g
6 5 9
1
5
5
4
5
5
4
2 8
3 9
9
2 9
3
4
/
4
3 4
3
3
2 8
7 4
9
8
1
4.
1
2
6
6
4
9
9 7
86
2
9
6 9
9
O
2
4
5
4
3
8
8
2
8
3
(a) standard
(b) RealNVP ((=[z)
(c) RealNVP p(=)
Figure B.3: Samples from VAS trained MNIST. The last column each 10x grid shows the training set images, most similar to the second-to to-sast column in pixel-wise L2 distance.
(a) standard
(b) RealNVP ((=(=)
(c) RealNVP p(z)
Figure B.4: Samples from Resnet VAE trained on FashionMNIST. The last column of cach 10x 10
grid shows the training set images, most similar to the second-to-last column pixel-wise L2 dis- tance.
H
M
9 ? 0
16
E
P
4
A
u/
4
K
11
P
{
K K
dP
B
3
H
P
E
F 3
8 19
9
W
8
3
3
3
m
H F
9
6
3
6
+
6
E
F8
13
4
R
5
2
(a) standard
(b) ReaINVP ((x(<)
(c) RealNVP p(z)
Figure B.5: Samples from ResnetVAE trained on Omniglot. The last column of each 10x 10 grid shows training set images, most similar second-to-last column pixel-wise distance.
B.12 ADDITIONAL RESULTS: ABLATION STUDY AND POSTERIOR OVERLAPPING
Additional results about ablation study and posterior overlapping are shown in Table B. B.19

---

Under review conference paper ICLR 2020
Table 1212: Average test NLL (lower is better) of different models, with Gaussian prior & Gaussian posterior ("standard"), Gaussian prior & RealNVP posterior ("RNVP g(=[x)")) and RealNVP prior & Gaussian posterior ("RNVP p(z)"). Flow depth K' 20.
Datasets
Models
StaticMNIST
MNIST
FashionMNIST
Omniglot
standard
88.84 + 0.05
84.48 = 0.03
228.60 = 0.03
106.42 = 0.14
VAn
RNVP ((=(>))
86.07 + 0.11
82.53 0.00
227.79 + 0.01
102.97 0.06
RNVP p(=)
84.87 + 0.05
0.01 80.43
226.11 + 0.02
102.19 0.12
standard
83.63 = 0.01
82.14
0.01
227.51 = 0.08
97.87
0.02
VAn
RNVP q(((=)
81.11 = 0.03
80.09 0.01
226.03 0.00
94.90 0.03
RNVP p(=)
80.06 + 0.07
0.01 78.67
224.65 + 0.01
93.68 0.01
standard
82.95 + 0.09
81.07
0.03
226. + 0.05
96,99
0.04
ResnetVAE
RNVP (((
80.97 = 0.05
79.53 0.03
225.02 = 0,01
94.30 0.02
RNVP p(z)
79.99 = 0.02
0.01 78.58
224.09 = 0.01
0.04 93.61
standard
79.47 + 0.02
78.64 0.02
224.22 + 0.06
89.83 0.04
PixeIVAE
RNVP (2(22)
79.09 = 0.01
78.41 0.01
223.81 0.00
89.69
0.01
RNVP p(z)
78.92 + 0.02
78.15 0.04
223.40 = 0.07
89.61 0.03
Table B.13: Test NLL of ResnetVAE on MNIST, with RealNVP posterior (" RealNVP prior ("p(=)")) and RealNVP prior & posterior ("both"). Flow depth K is 2 for the posterior the prior in "g(z[z)" and "p(z)", while Ko for both the posterior and the prior "both".
Kg
Resnet with
2
5
10
20
((222), K 2Kg
80.29
0.03
80.00
0.00
79.68
0.01
79.53
0.02
79.49
0.02
both, Kg
79.85 + 0.01
79.42
+ 0.02
79.01 + 0.02
78.71
0.00
78.56 + 0.01
p(2). K 2Ko
79.58 + 0.02
79.21 + 0.01
78.75 + 0.01
78.58 + 0.01
78.51 + 0.02
Table 1.14: Test reconstruction loss of ResnetVAB MNIST, with RealNVP posterior ("g(=lz)"), RealNVP prior ("p(=)"), and RealNVP prior & posterior ("both""). Flow depth K is 2Kg for the posterior or the prior in "q(=|z>"" and ")(""", while Ko for both the posterior and the prior in "both".
Ko
ResnetVAB with
2
5
10
20
g(=[x), 3, K- 2Kg
-57.20
0.05
-56.82
0.01
-56.37
0.04
-56.15
0.05
-55.98
0.05
both, K Kg
-56.34
0.01
-55.64
+ 0.02
-54.68
+ 0.04
-54.04
+ 0.04
-53.76
+ 0.02
p(2), 2Kg
-55.75 0.04
-54.90
0.03
-54.00
0.04
-53.64 0.02
-53.49 0.03
Table 1.15: Test NLL of ResnetVAF, with RealNVP prior of different flow depth K. Even K RealNVP prior improves NLLs by about nat. There is no over-fitting for up to 50. However, we think over-fitting may happen with sufficiently powerful prior, 50-blocks of RealNVP may simply not be deep enough.
Datasets
Flow depth
StaticMNIST
MNIST
FashionMNIST
Omniglot
0
82.95 + 0.09
81.07 = 0.03
226.17 = 0.05
96.99 0.04
81.76 4: 0.04
80.02 0.02
225.27 + 0.03
96.20 0.06
2
81.30 + 0.02
79.58 + 0.02
224.78 + 0.02
95.35
0.06
5
80.64
79.09 0.02
224.37 0.01
94.47 0.01
10
80.26 00.26:0.000
78.75 1755000 0.01
224.18 - 0.01
93.92 0.02
20
79.99 0.02
78.58 0.01
224.09 = 0.01
93.61 0.04
30
79.90 + 0.05
78.52 0.01
224.07 0.01
93.53
0.02
50
79.84
0.04
78.49 0.01
224.07 + 0.01
93.52 0.02
28

---

Under review conference paper ICLR 2020
Table Eg (x) DKT
1.16:
Average test
ELBO ("elbo"), reconstruction loss ("recons"),
("K(" and Ey Deu.(olltxx//loolxxx)) ("klatæ") of various
Dense VAE, Flow depth K 20.
standard
Datasets
elbo
kl
kl
StaticMNIST
-94.53
-65.54
29.00
5.69
MNIST
-88.19
-62.04
26.16
3.71
MNSSI
-230.81
-211.71
19.11
2.22
Omniglot
-113.59
-78.25
35.33
7.17
RealNVP ((]lar)
Datasets
elbo
recons
kl
kl je
StaticMNIST
-90.78
-62.55
28.23
4.71
MNIST
-85.15
-58.83
26.31
2.62
FaShionMNIST
-229.53
-210.35
19.18
1.75
Omniglot
-108.34
-73.71
34.83
5.57
RealNVP p(2)
Datasets
elbo
kl
kl
StaticMNIST
-88.87
-63.35
25.52
4.00
MNIST
-82.57
-55.99
26.58
2.14
FashionMNIST
-227.72
-208.13
19.59
1.61
Omniglot
-107.99
-73.10
34.89
5.80
ELBO ("elbo"), reconstruction loss ("recons""), (xpr DKL (go (ppx)))) ("k(") and (x (ealaaaaaxlxl)lllaaaaaiii ("klaaa")) of various ConvVAE Flow depth K 20.
Table B.17:
Average test
standard
Datasets
elbo
recons
kl
kl
StaticMNIST
-88.15
-60.34
27.81
4.51
MNIST
-85.89
-58.97
26.92
3.75
NNST
-230.43
-210.24
20.18
Omniglot
-104.70
-66.00
38.70
6.83
RealNVP (((xz)
Datasets
elbo
recons
kl
kl
StaticMNIST
-83.92
-56.77
27.25
2.82
MNIST
-82.46
-56.08
26.38
2.37
FashionMNIST
-228.20
-207.33
20.88
2.18
Omniglot
-99.92
-62.48
37.44
5.02
ReaINVP p(=)
Datasets
elbo
recons
kl
kl
StaticMNIST
-82.91
-54.01
28.90
2.86
MNIST
-80.42
-53.33
27.09
1.75
NNST
-226.65
-204.93
21.72
2.00
Omniglot
-98.59
-59.07
39.51
4.91
29

---

Under review conference paper ICLR 2020
Table 1.18:
Average test
ELBO ("elbo"), reconstruction loss ("recons"), Deu.lastttx///oo(zxx)) ("klatæ") of various
(x) (KT, (z|x)]]ppx ("k(") and Ey (x) ResnetVAE. Flow depth K = 20.
standard
Datasets
elbo
kl
kl
StaticMNIST
-87.61
-60.09
27.52
4.67
MNIST
-84.62
-58.70
25.92
3.55
MNSSI
-228.91
-208.94
19.96
2.74
Omniglot
-104.87
-66.98
37.89
7.88
RealNVP (((arr)
Datasets
elbo
recons
kl
kl oe
StaticMNIST
-84.72
-58.10
26.63
3.75
MNIST
-81.95
-56.15
25.80
2.42
FaShionMNIST
-227.16
-206.61
20.54
2.14
Omniglot
-100.30
-63.34
36.96
6.00
RealNVP p())
Datasets
elbo
kl
kl
StaticMNIST
-82.85
-54.32
28.54
2.87
MNIST
-80.34
-53.64
26.70
1.76
FashionMNIST
-225.97
-204.66
21.31
1.88
Omniglot
-99.60
-61.21
38.39
5.99
Table B.19:
Average test ELBO ("elbo"), reconstruction loss ("recons""),
Exycsr DKL (gs (2|x))]pxxxx) ("k(") and (x (aalaaaalax)x) ("klaaa") of various PixeIVAE. Flow depth K 20.
standard
Datasets
elbo
recons
kl
kl
StaticMNIST
-81.06
-69.02
12.03
1.59
MNIST
-79.88
-68.73
11.15
1.24
NNSH
-225.60
-214.15
11.45
1.38
Omniglot
-91.58
-82.80
8.78
1.75
RealNVP (((xz)
Datasets
elbo
recons
kl
StaticMNIST
-80.65
-67.91
12.75
1.57
MNIST
-79.59
-68.17
11.42
1.18
FashionMNIST
-225.05
-213.41
11.64
1.24
Omniglot
-91.26
-83.17
8.10
1.57
ReaINVP p(=)
Datasets
elbo
recons
kl
kl
StaticMNIST
-80.60
-62.22
18.38
1.68
MNIST
-79.28
-64.41
14.87
1.13
NNST
-224.65
-210.16
14.49
1.26
Omniglot
-91.78
-79.70
12.07
2.16
30

---

Under review conference paper ICLR 2020
Table B.20: Test NLL of ResnetVAE, with prior trained by: joint training, iterative training, post- hoc training, and standard VAE ("none") as reference. Flow depth K = 20.
ResnetVAE
Datasets
joint
iterative
post-hoc
none
StaticMNIST
79.99 + 0.02
80.63 = 0.02
80.86 + 0.04
82.95 + 0.09
MNIST
78.58 0.01
79.61 = 0.01
79.90 0.04
81.07 = 0.03
MNSST
224.09
224.88 0.02
225.22
0.01
226.17 0.05
Omniglot
93.61 0.04
94,43 = 0.11
94.87 0.05
96.99 + 0.04
X K=0
+ K=1
K=20
K=50
0.8
0.6
std
0.4
0.2
0.0
0.1
0.2
0.3
0.5
0.7
1.0
2.0
5.0
10.0
beta
Figure B.6: Avg. std of (x(x) of (-Resnet VAE trained with different 3 and prior flow depth K.

---

Under review conference paper ICLR 2020
B.13 ADDITIONAL RESULTS: TWO LEVEL HIERARCHICAL VAE WITH FLOW PRIOR
In addition to the comprehensive experiments based on VAE, we also conduct experiments using two-level ResnetHVAE (Tomczak & Welling 2018), just as the two major previous works (Tomczak
& Welling 2018 Bauer& & Mnih 2019))
Architectures of ResnetHVAE for binarized datasets The observed variable a 3-d tensor, just the same as VAEs. The latent variables 21 and &2 are vectors with the same number of dimen- sions, which is 40 on MNIST and StaticMNIST, and 64 on FashionMNIST and Omniglot. We use (z) to denote this dimensionality.
The overall factorization for HVAE is:
(x(x, 22, 2, po (0 21 22) Po (22) P> (22)
g0 [2, 21 xx - go (z) [22, x) Uo (22 x)
where (x(2) either flow prior or a unit Gaussian prior, depending on the context.
The shared feature network (x) for go (22lx) and go (2) |x) is:
h₂(x)
Resnet [28 28 32 28 28 32
+ 14 14
64
+ 7 x 7 x
64 4 16]
Flatten
The Gaussian posterior (0/2 x) is derived as:
(o(22|x) = N (N (x)), (h
(x)) = (x)x)) + Linear[Dimie
log (ho(()) lx) + Lincar Lincar(Dimz)) [D)
The Gaussian posterior qo (x 22) is derived
(0(21) x, x2) = (N (x, 22)) a (go 12))
Ho (go 22)) gs(x, 22) Linear| [im()))
log a (go (x, 22)) = 96 22) - Linear Dim (z))
Io [x 22) Concat(t, us (Zz)) - Dense|300]
(x) hx(x) + Dense [500 + 150]
u (zz) = 2 + Dense[150 + 150]
The Gaussian prior (e(2) (2) is derived as:
pa(zi x2) = N (1g (tz(2)))) 3 (tx I)
(n(()(2z)) tz(22) + Lincar Dim [z)
log or (to((z)) tt((2) Linear [Dim [z]
to(22) = 02 > Dense[150 - 150]
For binarized images, the Bernoulli conditional distribution (x x2) is derived as:
Po ( za) Bernoull(puogooz. x2)]]
(go 1 22))
log
(o(2) 22) + Linear CaneaarCovvxx1/2 [28 x 28 x I]
M4 (go (z, zz))
9o (21, 22) Concal (21, 22] hz(
hz(z) Dense 784] + UnFlatten x 7 16]
+ DeResnet(7 x 7 64 + 14 x 14 x 64
14 x 14 x 64 + 28 x 28 x 32
28 28 32]
Note the shared feature network (x(x) and the decoder network (z)z) have exactly the same archi- tecture VAT (see Appendix B.2)).
32

---

Under review conference paper ICLR 2020
Table B.21: Test negative log- log-iikeldhod of ResnetHVAA,, compared with ResnetVAE, Flow depth
K 20 for (x( ResnetVAT. For ResnetHVAA,, (2 flow prior with K 20, and
Px [21 x2) is a Gaussian prior.
Datasets
Models
StaticMNIST
MNIST
FashionMNIST
Omniglot
standard
82.95 = 0.09
81.07 0.03
226.17 + 0.05
96.99 = 0.04
ResnetVAT:
RNVP p(z)
79.99 = 0.02
78.58 + 0.01
224.09 + 0.01
93.61 = 0.04
standard
81,58 = 0.00
80.09 0.03
225.18 + 0.02
95.80 = 0.03
ResnetHVAE
RNVP p(22)
79.95 0.01
78.56 0.01
224.11 + 0.01
93.47 0.01
Table B.22: Average number of active units of ResnetHVAE of 22 and E1, with unit Gaussian p(=2) and p(2) ("standard"), and RealNVP p(z2) and Gaussian p(a)) ("RNVP p(2)"").
Datasets
StaticMNIST
MNIST
FashionMNIST
Omniglot
ResnetH VAR
21
22
21
22
21
22
21
2/2
standard
11.3
25
8.7
18.3
16.3
16.7
12.3
57.7
RealNVP p(22)
1.3
40
40
0.3
64
0
64
Experimental results Table [a.2 shows the test negative log-likelihood (NLL) of ResnetH with or without RealNVP p(zz), compared with ResnetVAE, with or without RealNVP p(z). We that, ResnetHVAE without flow prior has significantly better test NLLs than ResnetVAE without flow prior. However, ResnetHVAE with RealNVP p(z2) only has roughly the same test NLLs as Resnet with RealNVP p(z).
Further investigation on the active units of ResnetHVAE (Table B.22 reveals the cause of such ex- perimental results: when coupled with RealNVP p(zz), the second latent variable z in ResnetH degenerates, such that no dimension is active in the case of MNIST, FashionMNIST and Omniglot, while almost dimension is active in the case of StaticMNIST. We also added RealNVP to Zı, but the results showed no difference.
However, we find such degeneration may not always happen. We conducted pilot experiment ResnetHVAE with RealNVP p(z,) and Gaussian p(z,) on Cifar-10. The shape of Z2 and z are both 8 x 8 16. The number of active units of Zg 1024 (i.e.. all dimensions the number
active), while active units of z is 768 (specifically, these 768 dimensions consist 12 channels 8 x 8 feature maps). The test bpd of this experiment is 3.96. This suggests that the zero active units of z in Table B.22 is not caused by the RealNVP prior. The resnet network architecture we used in our experiments (on the binarized datasets) may have limited the model capacity, such that the ResnetVAT with RealNVP prior is already very close to the limitation of test NLLs in such condition, where adding auxiliary latent variable will not help.
We also report the reconstruction loss, the KL divergence (gs,( zi (z2, zi)) and DKL. (go 22, Zi |x] (po (Z2, zı (x)) of ResnetHVAE with or without RealNVP p (Zz)) which have the same trend as ResnetVAE (Table Gp.
33

---

Under review conference paper ICLR 2020
Table B.23:
Average
test
ELBO ("elbo"), reconstruction loss ("recons") ") ("k("), and Dga(1o(22- 2, (22 x)) ("kla/a") of of
DKL (ga(gs (Z2, 21 x)] (z2, 21)
ResnetHVAEs with different prior.
standard
RealNVP p(22)
Datasets
elbo
recons
klgja
elbo
recons
k
StaticMNIST
-85.67
-5.88
26.78
4.08
-82.82
-54.11
28.71
2.87
MNIST
-82.98
-57.19
25.79
2.90
-80.30
-3.55
26.76
1.74
MNIST
-227.56
-207.46
20.10
2.38
-225.99
-204.48
21.51
1.88
Omniglot -103.05
-65.04
38.01
7.25
-99.37
-61.00
38.37
5.90
B.14 ADDITIONAL RESULTS: LEARNED PRIOR ON LOW POSTERIOR SAMPLES
Rosca et al. 2018) observed that some the samples from unit Gaussian prior can have low (z(z) and low visual quality, and they name these samples the "low posterior samples"s Unlike this paper, they tried various approaches to match qo (z) unit Gaussian px(z). including adversarial train- ing methods, but still found low posterior samples scattering across the whole prior. We shall see that learning the prior can avoid having high (z) on low posterior samples, and thus we suggest adopting learned prior a cheap solution to this problem.
We shall first introduce the algorithm proposed by Rosca al. (2018), which can used to obtain low posterior samples (z samples which have low likelihoods on (z)) from a trained VAE. Their algorithm first samples large number of from the prior px (z), then uses Monte Carlo estimator evaluate (z) on these z samples, and finally chooses a certain number z with the lowest (z) likelihoods as the low posterior samples. They also sampled one x from po (x (z) for each low posterior sample and plotted the sample means of these and the histograms of ELBO these x. Although we have found their Monte Carlo estimator for DKL (gs(())))) (z))) vulnerable (see Appendix B.8 their low posterior samples algorithm only ranks (z(z) for cach z, and the visual results of their algorithm seem plausible. Thus we think this algorithm should be still convincing enough, and we also use to obtain low posterior samples.
To compare the learned prior with unit Gaussian prior on such low posterior samples, we first train a RAsnetVAE with unit Gaussian prior (denoted as standard ResnetVAE), and then add post-hoc trained RealNVP prior upon this original ResnetVAE (denoted as post-hoc trained ResnetVAE). We then obtain 10,000 samples from the standard Resnet VAE, and choose 100 z with the lowest (z) among these 10,000 samples, evaluated on standard ResnetVAE. Fixing these 100 samples, we plot the histograms of log px (z) w.E.. standard ResnetVAF. and post-hoc trained ResnetVAF. also obtain one x sample from pa (x|z) (2) for each z, and plot the histograms ELBO for each x (w.e.. the two models) and their sample means. See Fig. B.7
The x samples of post-hoc trained ResnetVAE (bottom right) the same those of standard ResnetVAE (top right), since post-hoc trained ResnetVAE has exactly the same p+ (x|zz) and go (z as standard ResnetVAE. However, the post-hoc trained prior successfully assigns much lower pxg()(z) (bottom left) than unit Gaussian prior (top left) on the low posterior samples, which sug- gests that post-hoc trained prior can avoid granting high likelihoods to these samples the latent space. Note post-hoc Iruined ResnetVAE also assigns slightly lower ELBO (bottom middle) than standard ResnetVAE. (top middle) to samples corresponding to these low posterior samples.
To verify whether a learned prior avoid obtaining low posterior samples in the first place, we obtained low posterior samples from a ResnetVAE with jointly trained prior (denoted as jointly trained RAsnetVAE), see Fig. B.8 Compared with Fig. B.7 we can see that log px()) of these low posterior samples and ELBO of the corresponding samples are indeed substantially higher than those of standard ResnetVAE and post-hoc truined ResnetVAE. However, the visual quality is not perfect, indicating there is still room for improvement.

---

Under review conference paper ICLR 2020
train samples
jow_posterior samples
0 b249 9409900000
Standard Resnet VAB
7 76b
3 D9DD006AS
0610A39541
D DA59068041
2 9/5069745
974
6 203633600
-90
-75
-60
-5
240
-100 120 60
Post-hoc Trained Resnet VAE
29 07
O4 9999900000
7 76b309
D9D
0650A39501
D D139068047
8P50697850
07403261
603658600
-150
-120
900
60
30
-240 1880-180-10 120
60
log PA (z)
elbo
X Po (x|z)
Figure B.7: (left) Histograms of log px) of the low posterior samples, (middle) histograms of ELBO of samples corresponding each low posterior sample, and (right) the means of these on standard ResnetVAE and post-hoc trained ResnetVAE.
train samples
low posterior samples
7 7 P:
9 3 4 9
Jointly Truined Resnet
/ F- 01.
AQ49045111
4 ) 3 446353
4 4736402043
28 P
GTODEASYOF
037332098
-80
-40
-20
240 180 120
-60
log PA (z)
elbo
(x
Figure B.8: (left) Histograms of log (2) of the low posterior samples, (middle) histograms of ELBO of samples corresponding each low posterior sample, and (right) the means of these on jointly trained ResnetVAF.
B.15 WALL-CLOCK TIMES
Hour
Standard
RealNVP q(z()
RealNVP
20
10
DenseVAE
COnVVAE
ResnetVAE
PixelVAE
Figure B.9: Average training time of various models on MNIST, flow depth K 20. Black sticks are standard deviation bars. For PixeIVAE, training mostly terminates in half way due to early- stopping.

---

Under review conference paper ICLR 2020
NLL
NLL
Time
Hour
81
40
80
30
79
20
20
50
Flow Depth
Figure 1.10: Average training time and test negative log-likelihood (NLL) of ResnetVAE with Re- aINVP prior of different flow depth. Vertical sticks are standard deviation bars.
report the average training time various models trained on MNIST, see Figs. B.9 and B.10 Each experiment runs on one GTX 1080 Ti graphical card. In Fig. B.9 we can see that the compu- tational cost of RealNVP prior is independent with the architecture of other parts of model. For complicated architectures like ResnetVAE and PixeIVAE, the cost of adding RealNVP prior fairly acceptable, since it can bring large improvement. In Fig. [0 we can see that for K > 50, there is likely to be little gain in test NLL, but the computation time will grow even larger. That's why we do not try larger K in our experiments.
REFERENCES
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 66-75, 2019.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowiez, and Samy Bengio. Generating Sentences from a Continuous Space. In Proceedings of the Twentieth Conference on Computational Natural Language Learning (CoNLL.)., 2016.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Hya Sutskever, and Pieter Abheel. Variational lossy autoencoder. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
I Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017. Conference Track Proceedings, 2017.
Izrail Moiseevitch Gelfand and Richard A. Silverman. Calculus of Variations. Courier Corporation, 2000.
Roger B. Grosse, Zoubin Ghahramani, and Ryan P. Adams. Sandwiching the marginal likelihood using bidirectional Monte Carlo. arXiv:1511.022. (cs, star), November 2015.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez, and Aaron C. Courville. Pixelvae: A latent variable model for natural images. In 5th Interna- tional Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Matthew D. Hoffman and Matthew Johnson. Elbo surgery: Yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016.
Chin- Wei Huang. Ahmed Touati, Laurent Dinh, Michal Drozdzal, Mohammad Havaci, Laurent Charlin, and Aaron Courville. Learnable Explicit Density for Continuous Latent Space and Vari- ational Inference. arXiv:1710.02228 [es, stat], October 2017.
36

---

Under review conference paper ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: method stochastic optimization. 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9. 2015, Conference Track Proceedings, 2015.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Informa- tion Processing Systems, pp. 4743-4751, 2016.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1xl Convolutions. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215-10224, Curran Associates, Inc., 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
B. M. Lake, R. Salakhutdinov, and B. Tenenbaum. Human-level concept learning through prob- abilistic program induction. Science, 330(6266):3332-1333333 1338,, December 2015. ISSN 0036-8075, 1095-9203. doi: 10. 126/science..ab30500
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 29-37, 2011.
Lars Maalee, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: Latent Variables for Generative Modeling. UrXii:/902.022 [cs, stat), February 2019.
A Very Deep Hierarchy
Andrew Maas, Awni Y. Hannum, and Andrew Y. Ng. Rectifier nonlincarities improve neural network acoustic models. In Proc. Ieml, volume 30, pp. 3, 2013.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th Inter- national Conference on Machine Learning- Volume 70, pp. 2391-2400. JMLR, org, 2017.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. arXiv: 18100007 [cs, stat), October 2018.
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution Matching in Varia- tional Inference. arXiv preprint arXiv: (tff:::80200644 2018.
Ruslan Salakhutdinov and lain Murray. On the quantitative analysis of deep belief networks. In Proceedings of the 25th International Conference on Machine Learning, pp. 872-879, ACM, 2008.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012.
Jakub Tomezak and Max Welling. VAE with VampPrior. In International Conference on Actificial Intelligence and Statistics, pp. 1214-1223, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Rushion-MNISTY A Novel Image Dataset for Bench- marking Machine Learning Algorithms. arXiv: [cs, stat), August 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK September 19-22, 2016, 2016.
37


Reviews:
[{"review_id": "ryxeWqalqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to use learned RealNVP with a shallow VAE instead of deep hierarchical VAE, which will not hurt the performance. The authors conduct thorough experiments to backup their claims and hypotheses.\n\nCons:\n1. The writing needs to be improved. The four contributions listed do not have a clear logic flow.\n2. The proposed method seems like a combination of previous studies, making the paper more like a technical report.\n3. One advantage claimed by the authors is that only one latent variable has clear semantic meanings, which is not explicitly supported by the experiments.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ryeDf7PxFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission shows that using learned autoregressive priors (real NVP) allows shallow VAEs to achieve comparable log-likelihood performances compared to more complex deep VAE architectures.\n\nI found this paper an enjoyable read, and its results quite intriguing. While most of VAE research focuses on building more powerful encoder and decoder architectures, these results show that focusing on a learned prior distribution is as important. \n\nThe models introduced in this paper are not novel, but the authors introduce some tricks (gradient and std clipping) that allow them to achieve nearly SOTA results with relatively simple architectures. \nI think these tricks should have been demonstrated more in detail in the experiments, for example:\n* why is the std clipped exactly at e^-11? What happens if I increase/decrease this number?\n* Can the authors clarify the differences between their model and that of Huang et al, which also uses real NVP priors? If I took the exact architecture of Huang et al and used the clipping trick would it perform similarly to your model?\n* Would other more complex SOTA models also benefit from your clipping tricks?\n\nA very interesting addition to the paper would be running some experiments on more complex data distributions such as the natural images of celebA or CIFAR10, to understand whether your model could achieve:\n(1) similar improvements in terms of  ELBO \n(2) more importantly, a quality of the generated samples comparable to deep VAE models such as VAE+IAF or BIVA.\n\nOverall I liked the paper so I am voting towards acceptance. However, while there are a considerable number of experiments in this paper, for me to increase the score I would like to see at least some of the experiments suggested above, since they could help better understand the behavior of VAEs with learned priors and make this an even more impactful paper.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rkxnJ9o0YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper claims that learning prior from the data could achieve superior performance than using a standard unit Gaussian prior. Experimental results further show that the proposed method could achieve a lower or comparable negative log-likelihood compared to other VAE variants using a complex hierarchical architecture.\n\nI have the following concerns about the paper:\n\nIt is widely accepted that the prior serves as the regularization for the Bayesian inference. Using a data-dependent prior is promising to derive a lower NNL than a data-independent prior. However, it would easily lead to an overfitting model with bad generalization, especially for a noisy dataset. \n\nIt is claimed in the abstract and conclusion that one latent variable is used for learning the RealNVP prior while the authors use the words \"shallow\" (refers to few latent variables) in the experiments. So how many latent variables are used in the experiment?\n\nIt is listed in the related work that \"Huang et al. (2017) applied RealNVP (Dinh et al., 2017) to learn the prior\", which means the idea using the learned RealNVP prior is not a new idea. Therefore, what is the contribution of this paper? A detailed discussion is needed to elaborate on the differences from previous methods using a prior learned from the data.\n\nThe equation of the aggregated posterior in page 2 after Eq.4 is wrong. The aggregated posterior should be the integration over x instead of z.\n\nThe drawn conclusion \"using both RealNVP posterior and prior shows no significant advantage over using RealNVP prior only, although the total flow depth of the former variant is twice as large as the latter one\" is quite unprofessional. Only one experiment set was conducted with k=20.  One obvious reason is that the current setting with k=20 makes that the model over-parameterized. More comparisons are needed for smaller k. \n\nAs claimed in the paper that the clipping would be a navie method to promote overlapping among the posterior. A comparison with a navie baseline using clipping is needed before drawing a conclusion that the learned RealNVP prior is the reason for enhancing the overlapping.\n\nIs the likelihood function p(x|z) a Bernoulli or Gaussian?\n\n\"Although BIVA has a much lower NLL on StaticMNIST, in contrast to our paper, the BIVA paper (Maaløe et al., 2019) ...... attributed to having fewer training data\". The author should confirm with the authors instead of giving a conjecture in a scientific paper. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an interesting insight into the fitting of variational autoencoders.  While much of the recent literature focuses on training ever more expressive models, the authors demonstrate that learning a flexible prior can provide an equally strong model.  Unfortunately one review is somewhat terse.  Among the other reviews, one reviewer found the paper very interesting and compelling but did not feel comfortable raising their score to "accept" in the discussion phase citing a lack of compelling empirical results in compared to baselines.  Both reviewers were concerned about novelty in light of Huang et al., in which a RealNVP prior is also learned in a VAE.  AnonReviewer3 also felt that the experiments were not thorough enough to back up the claims in the paper.  Unfortunately, for these reasons the recommendation is to reject.  More compelling empirical results with carefully chosen baselines to back up the claims of the paper and comparison to existing literature (Huang et al) would make this paper much stronger.


Decision: Reject