Title: SHALLOW VAES WITH REALNVP PRIOR CAN PER-FORM AS WELL AS DEEP HIERARCHICAL VAES

Abstract: Using powerful posterior distributions is a popular technique in variational inference. However, recent works showed that the aggregated posterior may fail to match unit Gaussian prior, even with expressive posteriors, thus learning the prior becomes an alternative way to improve the variational lower-bound. We show that using learned RealNVP prior and just one latent variable in VAE, we can achieve test NLL comparable to very deep state-of-the-art hierarchical VAE, outperforming many previous works with complex hierarchical VAE architectures. We hypothesize that, when coupled with Gaussian posteriors, the learned prior can encourage appropriate posterior overlapping, which is likely to improve reconstruction loss and lower-bound, supported by our experimental results. We demonstrate that, with learned RealNVP prior, β-VAE can have better rate-distortion curve than using fixed Gaussian prior.

1. INTRODUCTION:

Variational auto-encoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) is a powerful deep generative model. The use of amortized variational inference makes VAE scalable to deep neural networks and large amount of data. Variational inference demands the intractable true posterior to be approximated by a tractable distribution. The original VAE used factorized Gaussian for both the prior and the variational posterior (Kingma & Welling, 2014; Rezende et al., 2014) . Since then, lots of more expressive variational posteriors have been proposed (Tran et al., 2016; Rezende & Mohamed, 2015; Salimans et al., 2015; Nalisnick et al., 2016; Kingma et al., 2016; Mescheder et al., 2017; van den Berg et al., 2018) . However, recent work suggested that even with powerful posteriors, VAE may still fail to match aggregated posterior to unit Gaussian prior (Rosca et al., 2018) , indicating there is still a gap between the approximated and the true posterior.

To improve the variational lower-bound, one alternative way to using powerful posterior distributions is to learn the prior, an idea initially suggested by Hoffman & Johnson (2016) . Later on, Huang et al. (2017) applied RealNVP (Dinh et al., 2017) to learn the prior. Tomczak & Welling (2018) proved the optimal prior is the aggregated posterior, which they approximate by assembling a mixture of the posteriors with a set of learned pseudo-inputs. Bauer & Mnih (2019) constructed a rich prior by multiplying a simple prior with a learned acceptance function. Takahashi et al. (2019) introduced the kernel density trick to estimate the KL divergence in ELBO, for their implicit prior.

Despite the achievements of these previous works on posteriors and priors, the state-of-the-art VAE models with continuous latent variables all rely on deep hierarchical latent variables1 , although some of them might have used complicated posteriors/priors as components in their architectures. Most latent variables in such deep hierarchical VAEs have no clear semantic meanings, just a technique for reaching good lower-bounds. This is in sharp contrast with GANs (Goodfellow et al., 2014; Arjovsky et al., 2017) and flow-based models (Ho et al., 2018) , where most of them can reach state-of-the-art results with only one latent variable, with clear semantic meanings. We thus raise and answer a question: with the help of learned priors, can shallow VAEs achieve performance comparable or better than deep hierarchical VAEs? This question is important because a shallow VAE would be much more promising to scale to more complicated datasets than deep hierarchical VAEs. To answer this question, we conduct comprehensive experiments on several datasets with learned RealNVP priors and just one latent variable, which even shows advantage over some deep hierarchical VAEs with powerful posteriors. We also propose a hypothesis on why learning the prior can lead to such great improvement in model performance, supported by our experimental results. In summary, our contributions are:

• We conduct comprehensive experiments on four binarized datasets with four different network architectures. Our results show that VAE with RealNVP prior consistently outperforms standard VAE and RealNVP posterior. • We are the first to show that using learned RealNVP prior with just one latent variable in VAE, it is possible to achieve test negative log-likelihoods (NLLs) comparable to very deep state-of-the-art hierarchical VAE on these four datasets, outperforming many previous works using complex hierarchical VAE equipped with rich priors/posteriors. • We hypothesize that, when coupled with Gaussian posteriors, the learned prior can encourage appropriate posterior overlapping, which is likely to improve reconstruction loss and lower-bound, supported by our experimental results. • We demonstrate that, with learned RealNVP prior, β-VAE can have better rate-distortion curve (Alemi et al., 2018) than using fixed Gaussian prior.

2. PRELIMINARIES 2.1 VARIATIONAL AUTO-ENCODER:

Variational auto-encoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014 ) is a deep probabilistic model. It uses a latent variable z with prior p λ (z), and a conditional distribution p θ (x|z), to model the observed variable x. p θ (x) is defined as Z p θ (x|z) p λ (z) dz, where p θ (x|z) is derived by a neural network with parameter θ. log p θ (x) is bounded below by evidence lower-bound (ELBO):

log p θ (x) ≥ L(x; λ, θ, φ) = E q φ (z|x) [log p θ (x|z) + log p λ (z)log q φ (z|x)]

(1) = E q φ (z|x) [log p θ (x|z)] -D KL (q φ (z|x) p λ (z))

(2)

where q φ (z|x) is the variational posterior to approximate p θ (z|x), derived by a neural network with parameter φ. In Eq. ( 2), the first term is the reconstruction loss of x, and the second term is the Kullback Leibler (KL) divergence between q φ (z|x) and p λ (z). Optimizing q φ (z|x) and p θ (x|z) w.r.t. empirical distribution p (x) can be achieved by maximizing the expected ELBO w.r.t. p (x):

L(λ, θ, φ) = E p (x) [L(x; λ, θ, φ) ] = E p (x) E q φ (z|x) [log p θ (x|z) + log p λ (z)log q φ (z|x)] (3)

A hyper-parameter β can be added to the training objective, in order to control the trade-off between reconstruction loss and KL divergence, known as β-VAE (Higgins et al., 2017; Alemi et al., 2018) :

EQUATION

The aggregated posterior q φ (z) = X q φ (z|x) p (x) dx, should be equal to p λ (z), if VAE is perfectly trained (Hoffman & Johnson, 2016) . However, even with powerful posteriors, the aggregated posterior may still not match a unit Gaussian prior (Rosca et al., 2018) . To investigate this problem, Hoffman & Johnson (2016) suggested to decompose Eq. (3) as:

EQUATION

)

where I φ [Z; X] = q φ (z, x) log q φ (z,x) q φ (z) p (x) dz dx is the mutual information. Since p λ (z) is only in 2 , ELBO can be further enlarged if p λ (z) is trained to match q φ (z).

2.2. REALNVP:

RealNVP (Dinh et al., 2017) is a deep probabilistic model. We denote its observed variable by z and latent variable by w, with marginal distribution p λ (z) and prior p ξ (w). Unlike VAE, RealNVP relates z and w by an invertible mapping w = f λ (z) instead of a conditional distribution, as follows:

EQUATION

where det (∂f λ (z)/∂z) is the Jacobian determinant of f λ . In RealNVP, f λ is composed of K invertible mappings, where

f λ (z) = (f K • • • • • f 1 )(z)

, and each f k is invertible. f k must be carefully designed to ensure that the determinant can be computed efficiently. The original paper of RealNVP introduced the affine coupling layer as f k . Kingma & Dhariwal (2018) further introduced actnorm and invertible 1x1 convolution. Details can be found in their papers.

3. LEARNING THE PRIOR WITH REALNVP:

It is straightforward to obtain a rich prior p λ (z) from a simple (i.e., with constant parameters) one with RealNVP. Denote the simple prior as p ξ (w), while the RealNVP mapping as w = f λ (z). We then obtain Eq. ( 6) as our prior p λ (z). Substitute Eq. ( 6) into (3), we get to the training objective:

L(λ, θ, φ) = E p (x) E q φ (z|x) log p θ (x|z) + log p ξ (f λ (z)) + log det ∂f λ (z) ∂z log q φ (z|x)

We mainly use joint training (Tomczak & Welling, 2018; Bauer & Mnih, 2019) (where p λ (z) is jointly trained along with q φ (z|x) and p θ (x|z) by directly maximizing Eq. ( 7)), but we also consider other two strategies, see Appendices B.3 and B.9 for discussions.

In our initial experiments, training VAEs with deep RealNVP priors is unstable. The HVAEs (see Appendix B.13) with RealNVP priors are even more unstable, where the training procedure can barely finish. However, we tackle this problem by the following two techniques: (1) using gradient clip, as the original RealNVP (Dinh et al., 2017) does; and (2) clipping the std of Gaussian posterior q φ (z|x) of VAE (or q φ (z 2 |x) and q φ (z 1 |x) of HVAE) by a minimum value of e -11 ≈ 1.67 × 10 -5 .

4. APPROPRIATE POSTERIOR OVERLAPPING WITH LEARNED PRIOR:

As we shall see in Section 5.4, RealNVP prior can result in a test negative log-likelihood (NLL) substantially better than RealNVP posterior, with the same flow depth. Also, using both RealNVP posterior and prior shows no significant advantage over using RealNVP prior only, although the total flow depth of the former variant is twice as large as the latter one. To investigate why VAE with RealNVP prior can lead to substantially better model performance than the other two variants, we take the particular architecture of our models into consideration (i.e., Bernoulli p θ (x|z), Gaussian q φ (z|x), and unit Gaussian or flow prior p λ (z)), and start with the following proposition:

Proposition 1. Given a finite number of discrete training data, i.e., p

(x) = 1 N N i=1 δ(x -x (i) ), if p θ (x|z) = Bernoulli(µ θ (z))

, where the Bernoulli mean µ θ (z) is produced by the decoder, and 0 < µ k θ (z) < 1 for each of its k-th dimensional output, then the optimal decoder µ θ (z) is: i) , where w i (z) = q φ (z|x (i) ) j q φ (z|x (j) )

µ θ (z) = i w i (z) x (

and i w i (z) = 1 (8) Proof. See Appendix A.

Proposition 1 suggests that if q φ (z|x) for different x overlap, then even at the center of q φ (z|x (i) ) of one training point x (i) , the optimal decoder will be an average of both x (i) and other training points x (j) , weighted by w i (z) and w j (z). Rezende & Viola (2018) have shown that weighted average like this is likely to cause poor reconstruction loss ( 1 in Eq. ( 5)) and "blurry reconstruction". Besides poor reconstructions, when x is high-dimensional, the reconstruction loss would typically dominate ELBO, suggesting that good reconstruction loss is necessary for obtaining good ELBO.

To enlarge 1 , it is a crucial goal to appropriately reduce the overlapping of q φ (z|x) for different x.

For Gaussian posterior, this seems to involve shrinking the standard deviations (std) of most q φ (z|x), since when the mean of every q φ (z|x) is determined, smaller stds for the majority of q φ (z|x) would effectively result in less overlapping.

When the prior p λ (z) is fixed (such as unit Gaussian), however, smaller stds for q φ (z|x) are likely to induce "holes" on the aggregated posterior, causing q φ (z) to be more dissimilar with the prior, enlarging 2 in Eq. ( 5). Also, less overlapping among the Gaussian posteriors q φ (z|x) would correspond to higher mutual information ( 3 in Eq. ( 5)) (Hoffman & Johnson, 2016; Mathieu et al., 2019) . As a result, with a fixed prior, there should exist a trade-off between 1 and 2 + 3 . In fact, such a trade-off has already been discussed in the context of β-VAE (Alemi et al., 2018) .

It should be a desirable property that the overlapping among q φ (z|x) would be large for similar x, and conversely, small for dissimilar x. We shall use the term "appropriate overlapping" to denote such property. We believe that the trade-off between the reconstruction loss ( 1 ) and the mutual information ( 3 ) should be one important ingredient for reaching the "appropriate overlapping", as implied by their respective meanings. However, the effect of 2 is not so clear. We suspect that, in many cases, this term may exhibit over-regularization, resulting in too large overlapping, and further unsatisfactory reconstruction loss. In this paper, we choose to use learned RealNVP prior to reduce the effect from 2 , such that the trade-off would occur mainly between 1 and 3 . We hope this will bring us closer to the "appropriate overlapping", than using the fixed unit Gaussian prior.

In conclusion, when p θ (x|z) is Bernoulli, we hypothesize that learning the prior can encourage "appropriate overlapping" among Gaussian posteriors, which is vital for good reconstructions and NLLs. Rezende & Viola (2018) has proved when p θ (x|z) = N (µ θ (z), σ 2 I) with fixed constant σ, the optimal decoder is also µ θ (z) = i w i (z) x (i) , thus our analysis also holds in such situation.

5. EXPERIMENTS:



5.1. SETUP:

Datasets We use four datasets in our experiments: statically and dynamically binarized MNIST (Larochelle & Murray, 2011; Salakhutdinov & Murray, 2008) (denoted as StaticMNIST and MNIST in our paper, respectively), FashionMNIST (Xiao et al., 2017) and Omniglot (Lake et al., 2015) . Details of these datasets can be found in Appendix B.1.

Models:

We perform systematically controlled experiments, using the following VAE variants:

(1) DenseVAE, with dense layers; (2) ConvVAE, with convolutional layers; (3) ResnetVAE, with ResNet layers (Zagoruyko & Komodakis, 2016) ; and (4) PixelVAE (Gulrajani et al., 2017) , with several PixelCNN layers on top of the ResnetVAE decoder. For RealNVP priors and posteriors, we use K blocks of invertible mappings (K is called flow depth hereafter), while each block contains an invertible dense, a dense coupling layer, and an actnorm (Dinh et al., 2017; Kingma & Dhariwal, 2018) . Training and evaluation Unless specified, all experiments are repeated for 3 times to report metric means. We use Adam (Kingma & Ba, 2015) and adopt warm up (KL annealing) (Bowman et al., 2016) to train all models. We perform early-stopping using negative log-likelihood (NLL) on validation set, to prevent over-fitting on StaticMNIST and on all datasets with PixelVAE. We use 1,000 samples to estimate NLL and other metrics on test set. See Appendix B.3 for more details.

5.2. QUANTITATIVE RESULTS OF VAE WITH REALNVP PRIOR AND ONE LATENT VARIABLE COMPARED TO DEEP HIERARCHICAL VAES AND OTHER PREVIOUS WORKS:

In Tables 1 and 2 Since ConvVAE is undoubtedly a simpler architecture than ConvHVAE (which has 2 latent variables), it is likely that our improvement comes from the RealNVP prior rather than the different architecture. We also conduct experiments on Cifar10 (Krizhevsky, 2009) . Although not as good as state-of-the-art deep VAEs, our ResnetVAE using RealNVP prior with just one latent variable still shows great improvement over the Gaussian prior, in both test bpd and sampling quality. See Appendix B.5 for more details.

Tables 1 and 2 show that using RealNVP prior with just one latent variable, it is possible to achieve NLLs comparable to very deep state-of-the-art VAE (BIVA), ourperforming many previous works (including works on priors, and works of complicated hierarchical VAE equipped with rich posteriors like VAE + IAF). This discovery shows that shallow VAEs with learned prior and a small number of latent variables is a promising direction.

5.3. QUALITATIVE RESULTS:

Figure (Gulrajani et al., 2017) .

Using RealNVP prior only has better NLL than using both RealNVP prior and posterior, or using RealNVP posterior only, under the same model complexity, as shown in Table 4 .

Active units Table 5 counts the active units (Burda et al., 2016) of different ResnetVAEs, which quantifies the number of latent dimensions used for encoding information from input data. We can see that, both RealNVP prior and posterior can make all units of a ResnetVAE to be active (which is in sharp contrast to standard VAE). However, RealNVP prior can in fact result in substantially better reconstruction loss than RealNVP posterior (see Table B .18) . This indicates that, the good regularization effect, "a learned RealNVP prior can lead to more active units than a fixed prior" (Tomczak & Welling, 2018; Bauer & Mnih, 2019) , is not the main cause of the huge improvement in NLLs, especially for the improvement of RealNVP prior over RealNVP posterior.

5.5. RECONSTRUCTION LOSS AND POSTERIOR OVERLAPPING:

Better reconstruction loss, but larger KL divergence In Table 6 , ELBO and reconstruction loss ("recons", which is 1 in Eq. ( 5)) of ResnetVAE with RealNVP prior are substantially higher than standard ResnetVAE, just as the trend of test log-likelihood (LL) in Table 3 . Metrics of other models are in Tables B.16 to B.19. On the contrary, E p (x) D KL (q φ (z|x) p λ (z)) ("kl", which is 2 + 3 ) are larger. Since ELBO equals to 1 -( 2 + 3 ), this suggests that in our experiments, the improvement in ELBO (and also NLL) of ResnetVAE with RealNVP prior all comes from the improved reconstruction loss. This serves as a foundation for the analysis in Section 4.

kl can happen to be smaller when recons gets larger (e.g., our DenseVAE in Table B .16, and VAE with dense layers of (Bauer & Mnih, 2019 , Table 2 )). Nevertheless, all our experiments with Real-NVP prior show improved reconstruction loss, which validates the analysis of Section 4.

Smaller standard deviation of Gaussian posterior with RealNVP prior In Fig. 3 , we plot the histograms of per-dimensional stds of q φ (z|x), as well as the distances and normalized distances (which is roughly distance/std) between each closest pair of q φ (z|x) (see Appendix B.4 for formulations). The stds of q φ (z|x) with RealNVP prior are substantially smaller, and the normalized distances are larger. Larger normalized distances indicate less density of q φ (z|x) to be overlapping. This fact is a direct evidence of the analysis in Section 4.

The test NLL of the model and the learned std of q φ (z|x) is insensitive to the minimum clipping value under a certain threshold See Appendix B.6 for experiment results.

Appropriate overlapping among q φ (z|x) with learned prior To demonstrate that the stds of q φ (z|x) with RealNVP prior are reduced according to the dissimilarity between x rather than being reduced equally (i.e., q φ (z|x) exhibits "appropriate overlapping"), we plot the interpolations of z between the centers of q φ (z|x) of two training points, and log p λ (z) of these interpolations in Fig. 2 , We visualize p λ (z), because it is trained to match q φ (z), and can be computed much more reliable than q φ (z); and because the density of q φ (z) between z corresponding to two x points can be an indicator of how q φ (z|x) overlap between them. The learned RealNVP p λ (z) scores the interpolations of z between the centers of q φ (z|x) of two training points, giving low likelihoods to hard-to-interpret interpolated samples between two dissimilar x (the first three rows), while giving high likelihoods to good quality samples between two similar x (the last three rows). In contrast, the unit Gaussian prior assigns high likelihoods to all interpolations, even to hard-to-interpret ones. This suggests that the posterior overlapping is more appropriate with RealNVP prior than with unit Gaussian prior.

Learned prior influences the trade-off between reconstruction loss and KL divergence, as suggested by Section 4. We plot the rate-distortion curve (RD curve) (Alemi et al., 2018) of β-ResnetVAE trained with different β and prior flow depth K in Fig. 4 . Rate is D KL (q φ (z|x) p θ (z)), while distortion is negative reconstruction loss. Each connected curve with the same shape of points in Fig. 4 correspond to the models with the same K, but different β. We can see that the curves of K = 1 is closer to the boundary formed by the green line and the x & y axes than K = 0, while K = 20 & 50 are even closer. According to Alemi et al. (2018) , points on the RD curve being closer to the boundary suggests that the corresponding models are closer to the theoretical optimal models on a particular dataset, when traded between reconstruction loss and KL divergence. Given this, we conclude that learned prior can lead to a "better" trade-off from the perspective of RD curve.

We also plotted the average normalized distance of β-ResnetVAE trained with different β and prior flow depth K in Fig. 5 . The average std of q φ (z|x) can be found at Fig. B.6. Learned prior can encourage less posterior overlapping than unit Gaussian prior for various β, not only for β = 1.

6. RELATED WORK:

Learned priors, as a natural choice for the conditional priors of intermediate variables, have long been unintentionally used in hierarchical VAEs (Rezende et al., 2014; Sønderby et al., 2016; Kingma et al., 2016; Maaløe et al., 2019) . A few works were proposed to enrich the prior, e.g., Gaussian mixture priors (Nalisnick et al., 2016; Dilokthanakul et al., 2016) , Bayesian non-parametric priors (Nalisnick & Smyth, 2017; Goyal et al., 2017) , and auto-regressive priors (Gulrajani et al., 2017; Chen et al., 2017) , without the awareness of its relationship with the aggregated posterior, until the analysis made by Hoffman & Johnson (2016) . Since then, attempts have been made in matching the prior to aggregated posterior, by using RealNVP (Huang et al., 2017) , variational mixture of posteriors (Tomczak & Welling, 2018) , learned accept/reject sampling (Bauer & Mnih, 2019) , and kernel density trick (Takahashi et al., 2019) . However, none of these works recognized the improved reconstruction loss induced by learned prior. Furthermore, they did not show that learned prior with just one latent variable can achieve comparable results to those of many deep hierarchical VAEs.

The trade-off between reconstruction loss and KL divergence was discussed in the context of β-VAE (Higgins et al., 2017; Alemi et al., 2018; Mathieu et al., 2019) , however, they did not further discuss the impact of a learned prior on this trade-off. Mathieu et al. (2019) also discussed the posterior overlapping, but only within the β-VAE framework, thus was only able to control the degree of overlapping globally, without considering the local dissimilarity between x.

7. CONCLUSION:

In this paper, using learned RealNVP prior with just one latent variable in VAE, we managed to achieve test NLLs comparable to very deep state-of-the-art hierarchical VAE, outperforming many previous works of complex hierarchical VAEs equipped with rich priors/posteriors. We hypothesized that, when coupled with Gaussian posteriors, the learned prior can encourage appropriate posterior overlapping, which is likely to benefit the reconstruction loss and lower-bound, supported by our experimental results. We showed that with learned RealNVP prior, β-VAE can have better ratedistortion curve (Alemi et al., 2018) than with fixed Gaussian prior. We believe this paper is an important step towards shallow VAEs with learned prior and a small number of latent variables, which potentially can be more scalable to large datasets than those deep hierarchical VAEs.

A PROOF FOR PROPOSITION 1:

We first re-state Proposition 1:

Proposition. Given a finite number of discrete training data, i.e., p

(x) = 1 N N i=1 δ(x -x (i) ), if p θ (x|z) = Bernoulli(µ θ (z))

, where the Bernoulli mean µ θ (z) is produced by the decoder, and 0 < µ k θ (z) < 1 for each of its k-th dimensional output, then the optimal decoder µ θ (z) is: i) , where w i (z) = q φ (z|x (i) ) j q φ (z|x (j) )

µ θ (z) = i w i (z) x (

and i w i (z) = 1

We shall need the Euler's equation for the calculus of variations (see Gelfand & Silverman (2000, page 15, 24, and 35)): Theorem A.1. Let L[f ] be a functional of the form: 

L[f ] = b a F (x, f, ∂ x f ) dx

∂F ∂f - d dx ∂F ∂(∂ x f ) = 0 (A.1)

where we use ∂ x f to denote df dx . For multi-variate function f (x 1 , x 2 ) and the functional:

L[f ] = R F (x 1 , x 2 , f, ∂ x1 f, ∂ x2 f ) dx 1 dx 2

where R is a closed area, the Euler's equation becomes (Gelfand & Silverman, 2000, page 24) :

∂F ∂f - ∂ ∂x 1 ∂F ∂(∂ x1 f ) - ∂ ∂x 2 ∂F ∂(∂ x2 f ) = 0 (A.2)

For vector value function f (x) = [f 1 (x), . . . , f n (x)] and the functional:

L[f ] = b a F (x, f 1 , . . . , f n , ∂ x f 1 , . . . , ∂ x f n ) dx

a necessary condition for f (x) to be an extremal of L[f ] is that, every f i (x) satisfies Euler's equation (Gelfand & Silverman, 2000, page 35) .

With some straightforward extensions, we can also get to the Euler's equation for multi-variate vector value functions.

The original Euler's equation requires the integral to be definite, whereas a neural network is defined on the whole R n space. However, in practice, the input values (i.e., the training and test data) are usually within limited range, and the norm of the neural network parameters are not too large. As a result, there should exist a sufficiently large number R, such that all intermediate outputs are within the closed area [-R, R] n . In this sense, the definite and indefinite integral would not have too much difference. We shall always use the indefinite integral notations in the following proof.

Finally, we get to the proof of Proposition 1:

Proof. To apply calculus of variations, we need to substitute the parameterized, bounded µ θ (z) with a non-parameterized, unbounded mapping. Since 0 < µ k θ (z) < 1, and µ θ (z) is produced by neural network, which ensures µ θ (z) is a continuous mapping, then ∀θ, there exists unbounded t(z), s.t.

µ k θ (z) = exp t k (z) 1 + exp(t k (z)) t k (z) = log µ k θ (z) -log 1 -µ k θ (z)

and for all continuous mapping t(z), there also exists µ θ (z), satisfying the above equations. In fact, this substitution is also adopted in the actual implementation of our models.

The probability of p θ (x|z) is given by:

p θ (x|z) = Bernoulli(µ θ (z)) = k µ k θ (z) x k 1 -µ k θ (z) (1-x k )

Then we have:

log p θ (x|z) = k x k log µ k θ (z) + (1 -x k ) log 1 -µ k θ (z) = k x k t k (z) -x k log 1 + exp t k (z) -(1 -x k ) log 1 + exp t k (z) = k x k t k (z) -log 1 + exp t k (z)

The training objective L can be then formulated as a functional on t(z):

L[t] = E p (x) E q φ (z|x) log p θ (x|z) + log p θ (z) q φ (z|x) = p (x) q φ (z|x) • log p θ (x|z) + log p θ (z) q φ (z|x) dz dx = F (z, t) dz

where F (z, t) is:

F (z, t) = p (x) q φ (z|x) k x k t k (z) -log 1 + exp t k (z) + log p θ (z) q φ (z|x) dx

According to Euler's equation (Gelfand & Silverman, 2000, page 14 and 35), the necessary condition for L[t] to have an extremum for a given t(z) is that, t(z) satisfies ∂F/∂t k = 0, ∀k. Thus we have:

∂F ∂t k = 0 =⇒ p (x) q φ (z|x) x k - exp t k (z) 1 + exp(t k (z)) dx = 0 =⇒ p (x) q φ (z|x) x k -µ k θ (z) dx = 0 =⇒ i q φ (z|x (i) ) x (i) k -µ k θ (z) = 0 =⇒ i q φ (z|x (i) ) x (i) k = i q φ (z|x (i) ) µ k θ (z) =⇒ µ k θ (z) = i q φ (z|x (i) ) x (i) k j q φ (z|x (j) ) That is to say, µ θ (z) = i q φ (z|x (i) ) x (i) j q φ (z|x (j) ) = i w i (z) x (i) .

Rezende & Viola (2018) has proved that when p θ (x|z) = N (µ θ (z), σ 2 I), where σ is a global fixed constant, the optimal decoder µ θ (z) = i w i (z) x (i) , which is exactly the same as our conclusion. Rosca et al. (2018) has proved that the gradient of Bernoulli(µ θ (z)) is the same as that of N (µ θ (z), σ 2 I) when σ = 1, but they did not calculate out the optimal decoder. We push forward both these works.

B EXPERIMENTAL DETAILS B.1 DATASETS:

MNIST MNIST is a 28x28 grayscale image dataset of hand-written digits, with 60,000 data points for training and 10,000 for testing. When validation is required for early-stopping, we randomly split the training data into 50,000 for training and 10,000 for validation.

Since we use Bernoulli p θ (x|z) to model these images in VAE, we binarize these images by the method in (Salakhutdinov & Murray, 2008) : each pixel value is randomly set to 1 in proportion to its pixel intensity. The training and validation images are re-binarized at each epoch. However, the test images are binarized beforehand for all experiments. We binarize each test image 10 times, and use all these 10 binarized data points in evaluation. This method results in a 10 times larger test set, but we believe this can help us to obtain a more objective evaluation result.

StaticMNIST StaticMNIST (Larochelle & Murray, 2011 ) is a pre-binarized MNIST image dataset, with the original 60,000 training data already splitted into 50,000 for training and 10,000 for validation. We always use validation set for early-stopping on StaticMNIST. Meanwhile, since StaticMNIST has already been binarized, the test set is used as-is without 10x enlargement.

FashionMNIST FashionMNIST (Xiao et al., 2017 ) is a recently proposed image dataset of grayscale fashion products, with the same specification as MNIST. We thus use the same trainingvalidation split and the same binarization method just as MNIST.

Omniglot Omniglot (Lake et al., 2015) is a 28x28 grayscale image dataset of hand-written characters. We use the preprocessed data from (Burda et al., 2016) , with 24,345 data points for training and 8,070 for testing. When validation is required, we randomly split the training data into 20,345 for training and 4,000 for validation. We use dynamic binarization on Omniglot just as MNIST.

B.2 NETWORK ARCHITECTURES:

Notations In order to describe the detailed architecture of our models, we will introduce auxiliary functions to denote network components. A function h φ (x) should denote a sub-network in q φ (z|x), with a subset of φ as its own learnable parameters. For example, if we write q φ (z|x) = q φ (z|h φ (x)) = N (µ φ (h φ (x)), σ 2 φ (h φ (x)) I), it means that the the posterior q φ (z|x) is a Gaussian, whose mean and standard deviation are derived by one shared sub-network h φ (x) and two separate sub-networks µ φ (•) and σ φ (•), respectively.

The structure of a network is described by composition of elementary neural network layers. Conv[H ×W ×C] denotes a non-linear convolution layer, whose output shape is H ×W ×C, where H is the height, W is the width and C is the channel size. As an abbreviation, CouplingLayer and ActNorm are components of RealNVP, proposed by Dinh et al. (2017) ; Kingma & Dhariwal (2018) . InvertibleDense is a component modified from invertible 1x1 convolution (Kingma & Dhariwal, 2018) . We shall only introduce the details of CouplingLayer, since it contains sub-networks, while the rest two are just simple components.

Conv[H 1 ×W 1 ×C 1 → H 2 × W 2 × C 2 ]

General configurations All non-linear layers use leaky relu (Maas et al., 2013) activation.

The observed variable x (i.e., the input image) is always a 3-d tensor, with shape H ×W ×C, whether or not the model is convolutional. The latent variable z is a vector, whose number of dimensions is chosen to be 40 on MNIST and StaticMNIST, while 64 on FashionMNIST and Omniglot. This is because we think the latter two datasets are conceptually more complicated than the former two, thus requiring higher dimensional latent variables.

The Gaussian posterior q φ (z|x) is derived as:

q φ (z|x) = N (µ φ (h φ (x)), σ 2 φ (h φ (x)) I) µ φ (h φ (x)) = h φ (x) → Linear[Dim(z)] log σ φ (h φ (x)) = h φ (x) → Linear[Dim(z)]

Note we make the network to produce log σ φ (h φ (x)) instead of directly producing σ φ (h φ (x)). h φ (x) is the hidden layers, varying among different models.

For binarized images, we use Bernoulli conditional distribution p θ (x|z), derived as:

p θ (x|z) = Bernoulli[µ θ (h θ (z))]

For DenseVAE:

log µ θ (h θ (z)) 1 -µ θ (h θ (z)) = h θ (z) → Linear[784] → UnFlatten[28 × 28 × 1]

And for others (i.e., ConvVAE, ResnetVAE and PixelVAE):

log µ θ (h θ (z)) 1 -µ θ (h θ (z)) = h θ (z) → LinearConv 1×1 [28 × 28 × 1]

Note we make the network to produce log µ θ (h θ (z)) 1-µ θ (h θ (z)) , the logits of Bernoulli distribution, instead of producing the Bernoulli mean µ θ (h θ (z)) directly.

DenseVAE h φ (x) and h θ (z) of DenseVAE are composed of dense layers, formulated as:

h φ (x) = x → Flatten → Dense[500 → 500] h θ (z) = z → Dense[500 → 500]

Conv/ResnetVAE h φ (x) and h θ (z) of ConvVAE are composed of (de)convolutional layers, while those of ResnetVAE consist of (deconvolutional) resnet layers. We only describe the architecture of ResnetVAE here. The structure of ConvVAE can be easily obtained by replacing all (deconvolutional) resnet layers with (de)convolution layers:

h φ (x) = x → Resnet[28 × 28 × 32 → 28 × 28 × 32 → 14 × 14 × 64 → 14 × 14 × 64 → 7 × 7 × 64 → 7 × 7 × 16] → Flatten h θ (z) = z → Dense[784] → UnFlatten[7 × 7 × 16] → DeResnet[7 × 7 × 64 → 14 × 14 × 64 → 14 × 14 × 64 → 28 × 28 × 32 → 28 × 28 × 32] PixelVAE h φ (x)

of PixelVAE is exactly the same as ResnetVAE, while h θ (z) is derived as:

h θ (z) = Concat[x, hθ (z)] → PixelCNN[28 × 28 × 33 → 28 × 28 × 33 → 28 × 28 × 33] hθ (z) = z → Dense[784] → UnFlatten[7 × 7 × 16] → DeResnet[7 × 7 × 64 → 14 × 14 × 64 → 14 × 14 × 64 → 28 × 28 × 32 → 28 × 28 × 32]

As Salimans et al. (2017) , we use dropout in PixelCNN layers, with rate 0.5.

RealNVP The RealNVP consists of K blocks, while each block consists of an invertible dense, a coupling layer, and an actnorm. The RealNVP mapping for prior, i.e. f λ (z), can be formulated as:

f λ (z) = z → f 1 (h 1 ) → • • • → f K (h K ) f k (h k ) = h k → InvertibleDense → CouplingLayer → ActNorm CouplingLayer(u) = Concat u l , u r Sigmoid (s λ (h λ,k (u l ))) + t λ (h λ,k (u l )) h λ,k (u l ) = u l → Dense[256] s λ (h λ,k (u l )) = h λ,k (u l ) → Linear[Dim(u r )] t λ (h λ,k (u l )) = h λ,k (u l ) → Linear[Dim(u r )]

where u l = u 0: Dim(u)/2 is the left half of u, and u r = u Dim(u)/2 :Dim(u) is the right half.

The RealNVP posterior, derived from the original Gaussian posterior q φ (w|x), is denoted as q φ,η (z|x), and formulated as:

q φ,η (z|x) = q φ (w|x) ∂f η (w) ∂w -1 (B.1) z = f η (w)

where the structure of the RealNVP mapping f η (w) for posterior is exactly the same as f λ (z) for prior. The ELBO for VAE with RealNVP posterior is then simply:

L(λ, θ, φ, η) = E p (x) E q φ (w|x) log p θ (x|f η (w)) + log p λ (f η (w))

log q φ (w|x) + log det ∂f η (w) ∂w (B.2)

B.3 ADDITIONAL DETAILS OF TRAINING AND EVALUATION:

General methodology All the mathematical expressions of expectations w.r.t. some distributions are computed by Monte Carlo integration. For example, E q φ (z|x) [f (z, x)] is estimated by:

E q φ (z|x) [f (z, x)] ≈ 1 L L i=1 f (z (i) , x)

where z (i) is one sample from q φ (z|x).

Training We use Adam (Kingma & Ba, 2015) to train our models for 2,400 epochs. The batch size is 128 for DenseVAE, ConvVAE and ResnetVAE, and 64 for PixelVAE. On MNIST, FashionMNIST and Omniglot, we set the learning rate to be 10 -3 in the first 800 epochs, 10 -4 in the next 800 epochs, and 10 -5 in the last 800 epochs. On StaticMNIST, we set the learning rate to be 10 -4 in the first 1,600 epochs, and 10 -5 in the last 800 epochs.

L2 regularization with factor 10 -4 is applied on weights of all non-linear hidden layers, i.e., kernels of non-linear dense layers and convolutional layers, in h φ (x), h θ (z), f λ (z) and f η (w).

The ELBO is estimated by 1 z sample for each x in training. We adopt warm-up (KL annealing) (Bowman et al., 2016) . The ELBO using warm-up is formulated as: L(x; λ, θ, φ) = E q φ (z|x) log p θ (x|z) + β (log p λ (z)log q φ (z|x)) β is increased from 0.01 to 1 linearly in the first 100 epochs, and it remains 1 afterwards. The warm-up ELBO for VAEs with RealNVP priors and posteriors can be obtained by replacing p λ (z) and q φ (z|x) of the above equation by Eq. ( 6) and Eq. (B.1), respectively. We adopt early-stopping using NLL on validation set, to prevent over-fitting on StaticMNIST and PixelVAE. The validation NLL is estimated using 100 z samples for each x, every 20 epochs.

Training strategies for p λ (z) We consider three training strategies for optimizing Eq. ( 7):

• Post-hoc training (Bauer & Mnih, 2019) : q φ (z|x) and p θ (x|z) are firstly trained w.r.t. the unit Gaussian prior, then q φ (z|x) and p θ (x|z) are fixed and p λ (z) is in turn optimized. This is the most intuitive training method according to Eq. ( 5), however, it does not work as well as joint training in terms of test negative log-likelihood (NLL), which is observed both in our experiments (Table B .6) and by Bauer & Mnih (2019) . • Joint training (Tomczak & Welling, 2018; Bauer & Mnih, 2019) : p λ (z) is jointly trained along with q φ (z|x) and p θ (x|z), by directly maximizing Eq. ( 7). To train p λ (z) with post-hoc strategy, we start from a trained VAE, adding RealNVP prior onto it, and optimizing the RealNVP f λ (z) for 3,200 epochs, with learning rate set to 10 -3 in the first 1,600 epochs, and 10 -4 in the final 1,600 epochs.

Algorithm B.1 Pseudocode for iterative training. Iteration 1a: Train q φ (z|x) and p θ (x|z), with p λ (z) = N (0, I). Iteration 1b: Train p λ (z) for 2M epochs, with fixed q φ (z|x) and p θ (x|z). for i = 2 . . . I do Iteration ia: Train q φ (z|x) and p θ (x|z) for M epochs, with fixed p λ (z).

Iteration ib: Train p λ (z) for M epochs, with fixed q φ (z|x) and p θ (x|z). end for Algorithm B.1 is the pseudocode of iterative training strategy. For Iteration 1a, all hyper-parameters are the same with training a standard VAE, where in particular, the number of training epochs is set to 2,400. For Iteration 1b, the learning rate is 10 -3 for the first M epochs, and is 10 -4 for the next M epochs. For all the next iterations, learning rate is always 10 -4 . The number of iterations I is chosen to be 16, and the number of epochs M is chosen to be 100, for MNIST, FashionMNIST and Omniglot. For StaticMNIST, we find it overfits after only a few iterations, thus we choose I to be 4, and M to be 400. With these hyper-parameters, q φ (z|x), p θ (x|z) and p λ (z) are iteratively trained for totally 3,200 epochs on all datasets (starting from Iteration 1b), after the pre-training step (Iteration 1a).

Regularization term for q φ (z) In order to compare some metrics (e.g., the active units) of VAE + RealNVP prior with those of standard VAE, we introduce an additional regularization term for q φ (z), such that q φ (z) of VAE using RealNVP prior would have roughly zero mean and unit variance, just as a standard VAE with unit Gaussian prior. The regularization term for q φ (z) (denoted as Reg [q φ (z)]) and the final training objective augmented with the regularization term (denoted as L(x; λ, θ, φ)) are:

Reg [q φ (z)] = 1 Dim(z) Dim(z) k=1 (Mean(z k )) 2 + (Var(z k ) -1) 2 L(λ, θ, φ) = E p (x) L(x; λ, θ, φ) + Reg [q φ (z)]

where z k is the k-th dimension of z, Dim(z) is the number of dimensions, Mean(z k ) =

E p (x) E q φ (z|x) [z k ] and Var(z k ) = E p (x) E q φ (z|x) (z k -Mean(z k ))

2 are the mean and variance of each dimension. Var(z k ). The means and standard deviations of the above table are computed w.r.t. repeated experiments. Using the regularization term for q φ (z) makes the Mean(z k ) and Var(z k ) of z samples close to N (0, I), which is in sharp contrast with the un-regularized case. For fair comparison in Table 5 and Fig. 3 , it is crucial to have Mean(z k ) and Var(z k ) close to N (0, I). Test NLLs are not reported here, because we find no significant difference between regularized and un-regularized models in terms of test NLLs.

Evaluation The negative log-likelihood (NLL), the reconstruction loss and the KL divergence E p (x) D KL (q φ (z) p λ (z)) are estimated with 1,000 z samples from q φ (z|x (i) ) for each x (i) from test data:

NLL ≈ 1 N N i=1 LogMeanExp 1000 j=1 log p θ (x (i) |z (i,j) ) + log p λ (z (i,j) ) -log q φ (z (i,j) |x (i) ) Reconstruction Loss ≈ 1 1000N N i=1 1000 j=1 log p θ (x (i) |z (i,j) ) E p (x) D KL (q φ (z) p λ (z)) ≈ 1 1000N N i=1 1000 j=1 log q φ (z (i,j) |x (i) ) -log p λ (z (i,j) )

where each z (i,j) is one sample from q φ (z|x (i) ).

LogMeanExp L j=1 f (x (i) , z (i,j) ) is defined as:

LogMeanExp L j=1 f (x (i) , z (i,j) ) = f max + log 1 L L j=1 exp f (x (i) , z (i,j) ) -f max where f max = max j f (x (i) , z (i,j) ).

We use 1,000 samples, because the two major previous works of us (i.e., Tomczak & Welling (2018); Bauer & Mnih ( 2019)) both used this number of samples. It is true that some previous works may have used substantially larger number of samples (e.g., Chen et al. ( 2017) used 4,096 samples), and according to (Grosse et al., 2015) , the Monte Carlo estimator for NLL is a stochastic lowerbound, such that larger number of samples typically would lead to lower (yet better) estimated NLLs. However, in order to have fair comparison with (Tomczak & Welling, 2018; Bauer & Mnih, 2019) , we decide to stick to 1,000 samples.

Active units active units (Burda et al., 2016) is defined as the number of latent dimensions whose variance is larger than 0.01. The variance of the k-th dimension is formulated as:

Var k = Var p (x) E q φ (z|x) [z k ]

where z k is the k-th dimension of z. We compute Var k on the training data, while the inner expectation E q φ (z|x) [z k ] is estimated by drawing 1,000 samples of z for each x.

B.4 FORMULATION OF CLOSEST PAIRS OF q φ (z|x) AND OTHERS:

Closest pairs of q φ (z|x) For each x (i) from training data, we find the training point x (j) , whose posterior q φ (z|x (j) ) is the closest neighbor to q φ (z|x (i) ):

j = arg min j =i µ φ (x (j) ) -µ φ (x (i) )

where µ φ (x) is the mean of q φ (z|x), and • is the L2 norm. These two posteriors q φ (z|x (i) ) and q φ (z|x (j) ) are called a closest pair of q φ (z|x).

Distance of a pair of q φ (z|x) The distance d ij of a closest pair q φ (z|x (i) ) and q φ (z|x (j) ) is:

d ij = µ φ (x (j) ) -µ φ (x (i) ) d ij = d ij

Normalized distance of a pair of q φ (z|x) For each closest pair q φ (z|x (i) ) and q φ (z|x (j) ), we compute its normalized distance d ij by:

d ij = 2d ij Std[i; j] + Std[j; i] (B.3) Std[i; j] is formulated as: Std[i; j] = Var q φ (z|x (i) ) z -µ φ (x (i) ) • d ij d ij (B.4)

where Var q φ (z|x (i) ) (f (z)) is the variance of f (z) w.r.t. q φ (z|x (i) ). We use 1,000 samples to estimate each Std[i; j]. Roughly speaking, the normalized distance d ij can be viewed as "distance/std" along the direction of d ij , which indicates the scale of the "hole" between q φ (z|x (i) ) and q φ (z|x (j) ).

B.5 EXPERIMENT RESULTS ON CIFAR10:

We conduct experiments on Cifar10 (Krizhevsky, 2009) . Although the results on Cifar10 are not as good as the state-of-the-art deep hierarchical VAEs, using RealNVP prior also shows great improvement on test NLLs and sampling quality over Gaussian prior, with just one latent variable.

Architectures of ResnetVAE for Cifar10:

The latent variable z is a 3-d tensor, with shape 8 × 8 × 32. The Gaussian posterior q φ (z|x) is derived as:

q φ (z|x) = N (µ φ (h φ (x)), σ 2 φ (h φ (x)) I) µ φ (h φ (x)) = h φ (x) → LinearConv 1×1 [8 × 8 × 32] log σ φ (h φ (x)) = h φ (x) → LinearConv 1×1 [8 × 8 × 32] h φ (x) = x → Resnet[32 × 32 × 96 → 32 × 32 × 96 → 32 × 32 × 96 → 16 × 16 × 192 → 16 × 16 × 192 → 16 × 16 × 192 → 8 × 8 × 256 → 8 × 8 × 256]

The generative distribution p θ (x|z) is derived as:

p θ (x|z) = DiscretizedLogistic (µ θ (h θ (z)), s θ (h θ (z))) µ θ (h θ (z)) = h θ (z) → LinearConv 1×1 [32 × 32 × 3] log s θ (h θ (z)) = h θ (z) → LinearConv 1×1 [32 × 32 × 3] h θ (z) = z → DeResnet[8 × 8 × 256 → 8 × 8 × 256 → 16 × 16 × 192 → 16 × 16 × 192 → 16 × 16 × 192 → 32 × 32 × 96 → 32 × 32 × 96]

where DiscretizedLogistic is the discretized logistic distribution proposed by Kingma et al. (2016) . Note we also clip log s θ (h θ (z)) by -11, just as the for the log-std of q φ (z|x). RealNVP prior The RealNVP prior for Cifar10 consists of 50 blocks, while each block consists of an invertible 1x1 conv (Kingma & Dhariwal, 2018), a coupling layer, and an actnorm, which can be formulated as:

f λ (z) = z → f 1 (h 1 ) → • • • → f K (h K ) f k (h k ) = h k → InvertibleConv 1×1 → CouplingLayer → ActNorm CouplingLayer(u) = Concat u l , u r Sigmoid (s λ (h λ,k (u l ))) + t λ (h λ,k (u l )) h λ,k (u l ) = u l → Resnet[8 × 8 × 64 → 8 × 8 × 64] s λ (h λ,k (u l )) = h λ,k (u l ) → LinearConv 1×1 [8 × 8 × 32] t λ (h λ,k (u l )) = h λ,k (u l ) → LinearConv 1×1 [8 × 8 × 32]

where u l = u :,:,0: Dim(u)/2 is the left half channels of u, and u r = u :,:, Dim(u)/2 :Dim(u) is the right half channels.

Experiment results Table B .2 shows the test bpd of our ResnetVAE on Cifar10. Although the test bpd of our model is still far from those of deep hierarchical models, it is worth mention that BIVA used 15 latent variables to reach the state-of-the-art. Our experiments show that, adding a deep RealNVP prior can make the test bpd much better, even with just one latent variable. This suggests that a complicated prior can be very effective for reaching good test bpd, and is potentially helpful for reducing the depth of hierarchical VAEs. The average learned std of Gaussian q φ (z|x) and the test NLL of ResnetVAE with Real-NVP and Gaussian p(z) on MNIST. We also plot "min std" as a reference, the minimum attainable std clipped by the configured mininum logstd.

The most surprising fact is that, in our experiments, it is not possible to obtain any meaningful samples using ResnetVAE with Gaussian p(z). In contrary, the samples obtained using ResnetVAE with RealNVP p(z) is much more like natural images.

B.6 IMPACT OF THE MINIMUM CLIPPING VALUE FOR THE STD OF q φ (z|x)

We have mentioned in Section 5.1 that we clip the std of Gaussian posterior q φ (z|x) by a minimum value of e -11 ≈ 1.67 × 10 -5 , so as to avoid unstable training. The clipping is actually done by directly clipping the linear activation of log-std of q φ (z|x) by -11, and then obtain std, instead of clipping the std itself. Formally, the clipped log-std and the std are derived by:

logstd = maximum {σ φ (f φ (z)), -11} std = exp (logstd)

where f φ (z) is the hidden feature network for q φ (z|x), and σ φ (•) is a linear fully-connected layer.

We use the clipped log-std directly instead of the std whenever possible in computing the loglikelihood of Gaussian q φ (z|x), which can be formulated as:

log q φ (z|x) = - 1 2 log(2π) -logstd - 1 2 exp (-2logstd) • (x -µ φ (f φ (z))) 2

where µ φ (z) is a linear fully-connected layer, producing the mean of q φ (z|x). Std clipping is widely used in many open-source implementations of VAEs. However, this trick is seldom discussed in published papers, which might suggest more theoretical work should be done on this topic.

The clipping value -11 in our experiments is not chosen for better model performance. The term exp (-2logstd) will have a maximum value of approximately 3.58 × 10 9 with this value, which we think is safe enough for not having NaNs in training, considering that we mainly use float32 in our neural network implementation.

In fact, we find that the test NLL of the model and the learned std of q φ (z|x) is insensitive to the clipping value under a certain threshold, which is demonstrated in Fig. B .2 and Table B .3. As long as the minimum std is small enough, both the RealNVP prior and the Gaussian prior could induce adaptive learned std of Gaussian posterior q φ (z|x). One surprising fact is that, there is a turning point where the clipping value is not small enough yet (i.e., min logstd being -3 for RealNVP p(z), and -1 for Gaussian p(z)), the final learned std is smaller than the learned std with smaller clipping values. The cause of this phenomenon is not analyzed in our paper, and we leave it as a future work.

Table B .3: The minimum std ("min std"), average learned std of Gaussian q φ (z|x) ("std"), the ratio of minimum std with respect to the learned std ("min std / std"), the test NLL and the active units ("au") of ResnetVAE with RealNVP p(z) and Gaussian p(z) on MNIST. The hyper-parameters of Huang et al. (2017) are quite different from ours: they only used 3 layers of ResNet blocks in their q φ (z|x) and 4 layers in their p θ (x|z). Also, they trained their ResConv VAE on StaticMNIST only for 300 epochs, using an initial learning rate of 0.0005 without learning rate annealing, and without validation set. In addition, we adopt invertible dense and actnorm in our RealNVP flow, while they just used the original RealNVP flow. Most parts of our ResnetVAE with RealNVP prior have larger complexity than the ResConv VAE of Huang et al. (2017) , expect that they used 2 hidden layers in each coupling layer, while we just used 1. By removing the invertible dense and actnorm components from the RealNVP prior, enforcing the encoder and the decoder to have similar model complexity, using the same number of training epochs and learning rate, and not using validation set to do early-stopping, we obtained similar results as Huang et al. (2017) (the first two rows of "Ours"). Adding invertible dense and actnorm (the third and fourth rows of "Ours") will improve the test NLL, suggesting that the invertible dense and actnorm can indeed bring benefit to the model performance.

ResnetVAE:

According to the above results, it is reasonable to believe that, it is the deeper RealNVP prior, more complicated model architecture, and longer training time, that give us better test NLL than Huang et al. (2017) . In conclusion, there may not be too many differences between our model and that of Huang et al. (2017) . However, we tune the hyper-parameters much better, and demonstrate that ResnetVAE with RealNVP prior and just one latent variable can work as well as many deep hierarchical VAEs on MNIST-like datasets. The additional analysis should also be our novel contribution, when compared to the work of Huang et al. (2017) .

B.8 DISCUSSIONS ABOUT D KL (q φ (z) p λ (z))

The KL divergence between the aggregated posterior and the prior, i.e., D KL (q φ (z) p λ (z)), is first analyzed by Hoffman & Johnson (2016) as one component of the ELBO decomposition (5). Since q φ (z) = q φ (z|x) p (x) dx and p λ (z) = p θ (z|x) p θ (x) dx, Rosca et al. (2018) used this KL divergence as a metric to quantify the approximation quality of both p θ (x) to p (x), and q φ (z|x) to p θ (z|x). As we are focusing on learning the prior, D KL (q φ (z) p λ (z)) can in turn be a metric for quantifying whether p λ (z) is close enough to the aggregated posterior q φ (z).

The evaluation of D KL (q φ (z) p λ (z)), however, is not an easy task. One way to estimate the KL divergence of two arbitrary distributions is the density ratio trick (Sugiyama et al., 2012; Mescheder et al., 2017) , where D KL (q φ (z) p λ (z)) is estimated by a separately trained neural network classifier. However, Rosca et al. (2018) revealed that such approach can under-estimate D KL (q φ (z) p λ (z)), and the training may even diverge when z has hundreds of or more dimensions.

Another approach is to directly estimate D KL (q φ (z) p λ (z)) by Monte Carlo integration. The KL divergence can be rewritten into the following form: Rosca et al. (2018) has already proposed a Monte Carlo based algorithm to estimate D KL (q φ (z) p λ (z)), in case that the training data is statically binarized. Since we mainly use dynamically binarized datasets, we slightly modified the algorithm according to Eq. (B.5), to allow sampling multiple x from each image. Our algorithm is:

D KL (q φ (z) p λ (z)) = q φ (z) log q φ (z) p λ (z) dz = q φ (z|x) p (x) dx log q φ (z|x ) p (x ) dx p λ (z) dz = p (x) q φ (z|x) log q φ (z|x ) p (x ) dx p λ (z) dz dx = E p (x) E q φ (z|x) log E p (x ) [q φ (z|x )] -log p λ (z) (B.5)

Surprisingly, we find that increasing n x will cause the estimated D KL (q φ (z) p λ (z)) to decrease on MNIST, see Table B .5. Given that our n z = 10 for all n x , the number of our sampled z (even when n x = 1) is 10×60, 000 = 6×10 5 , which should not be too small, since Rosca et al. (2018) only used 10 6 z in their experiments. When n x = 8, the number of z is 8 × 10 × 60, 000 = 4.8 × 10 6 , which is 4.8x larger than Rosca et al. (2018) , not to mention the inner expectation E p (x ) [q φ (z|x )] is estimated with 8x larger number of x . There should be in total 38.4x larger number of log q φ (z|x ) computed for estimating D KL (q φ (z) p λ (z)) than Rosca et al. (2018) , which has already costed about 3 days on 4 GTX 1080 Ti graphical cards. We believe such a large number of Monte Carlo samples should be sufficient for any algorithm with good behavior.

Table B .5: D KL (q φ (z) p λ (z)) of a ResnetVAE with RealNVP prior trained on MNIST, estimated by Algorithm B.2. n z = 10 for all n x . We only tried n x for up to 8, due to the growing computation time of O n 2 x . n x 1 2 3 5 8 D KL (q φ (z) p λ (z)) 15.279 14.623 14.254 13.796 13.392

According to the above observation, we suspect there must be some flaw in Algorithm B.2. Because of this, we do not adopt Algorithm B.2 to estimate D KL (q φ (z) p λ (z)).

Since no mature method has been published to estimate D KL (q φ (z) p λ (z)), we decide not to use D KL (q φ (z) p λ (z)) to measure how our learned p λ (z) approximates the aggregated posterior q φ (z). , with RealNVP posterior ("q(z|x)"), RealNVP prior ("p(z)"), and RealNVP prior & posterior ("both"). Flow depth K is 2K 0 for the posterior or the prior in "q(z|x)" and "p(z)", while K 0 for both the posterior and the prior in "both". In addition to the comprehensive experiments based on VAE, we also conduct experiments using two-level ResnetHVAE (Tomczak & Welling, 2018) , just as the two major previous works (Tomczak & Welling, 2018; Bauer & Mnih, 2019) .

Architectures of ResnetHVAE for binarized datasets:

The observed variable x is a 3-d tensor, just the same as VAEs. The latent variables z 1 and z 2 are vectors with the same number of dimensions, which is 40 on MNIST and StaticMNIST, and 64 on FashionMNIST and Omniglot. We use Dim(z) to denote this dimensionality.

The overall factorization for HVAE is:

p θ (x, z 2 , z 1 ) = p θ (x|z 1 , z 2 ) p θ (z 1 |z 2 ) p λ (z 2 ) q φ (z 2 , z 1 |x) = q φ (z 1 |z 2 , x) q φ (z 2 |x)

where p λ (z 2 ) is either a flow prior or a unit Gaussian prior, depending on the context.

The shared feature network h φ (x) for q φ (z 2 |x) and q φ (z 1 |x) is:

h φ (x) = x → Resnet[28 × 28 × 32 → 28 × 28 × 32 → 14 × 14 × 64 → 14 × 14 × 64 → 7 × 7 × 64 → 7 × 7 × 16] → Flatten

The Gaussian posterior q φ (z 2 |x) is derived as:

q φ (z 2 |x) = N µ φ (h φ (x)), σ 2 φ (h φ (x)) I µ φ (h φ (x)) = h φ (x) → Linear[Dim(z)] log σ φ (h φ (x)) = h φ (x) → Linear[Dim(z)]

The Gaussian posterior q φ (z 1 |x, z 2 ) is derived as:

q φ (z 1 |x, z 2 ) = N (µ φ (g φ (x, z 2 )), σ 2 φ (g φ (x, z 2 ))I) µ φ (g φ (x, z 2 )) = g φ (x, z 2 ) → Linear[Dim(z)] log σ φ (g φ (x, z 2 )) = g φ (x, z 2 ) → Linear[Dim(z)] g φ (x, z 2 ) = Concat[t φ (x), u φ (z 2 )] → Dense[300] t φ (x) = h φ (x) → Dense[150 → 150] u φ (z 2 ) = z 2 → Dense[150 → 150]

The Gaussian prior p θ (z 1 |z 2 ) is derived as:

p θ (z 1 |z 2 ) = N µ θ (t θ (z 2 )), σ 2 θ (t θ (z 2 )) I µ θ (t θ (z 2 )) = t θ (z 2 ) → Linear[Dim(z)] log σ θ (t θ (z 2 )) = t θ (z 2 ) → Linear[Dim(z)] t θ (z 2 ) = z 2 → Dense[150 → 150]

For binarized images, the Bernoulli conditional distribution p θ (x|z 1 , z 2 ) is derived as:

p θ (x|z 1 , z 2 ) = Bernoulli[µ θ (g θ (z 1 , z 2 ))] log µ θ (g θ (z 1 , z 2 )) 1 -µ θ (g θ (z 1 , z 2 )) = g θ (z 1 , z 2 ) → LinearConv 1×1 [28 × 28 × 1] g θ (z 1 , z 2 ) = Concat[z 1 , z 2 ] → h θ (z) h θ (z) = Dense[784] → UnFlatten[7 × 7 × 16] → DeResnet[7 × 7 × 64 → 14 × 14 × 64 → 14 × 14 × 64 → 28 × 28 × 32 → 28 × 28 × 32]

Note the shared feature network h φ (x) and the decoder network h θ (z) have exactly the same architecture as VAE (see Appendix B.2). Further investigation on the active units of ResnetHVAE (Table B .22) reveals the cause of such experimental results: when coupled with RealNVP p(z 2 ), the second latent variable z 1 in ResnetHVAE degenerates, such that no dimension is active in the case of MNIST, FashionMNIST and Omniglot, while almost no dimension is active in the case of StaticMNIST. We also added RealNVP to z 1 , but the results showed no difference.

However, we find such degeneration may not always happen. We conducted a pilot experiment of ResnetHVAE with RealNVP p(z 2 ) and Gaussian p(z 1 ) on Cifar-10. The shape of z 2 and z 1 are both 8 × 8 × 16. The number of active units of z 2 is 1024 (i.e., all dimensions are active), while the number of active units of z 1 is 768 (specifically, these 768 dimensions consist 12 channels of 8 × 8 feature maps). The test bpd of this experiment is 3.96. This suggests that the zero active units of z 1 in Table B .22 is not caused by the RealNVP prior. The resnet network architecture we used in our experiments (on the binarized datasets) may have limited the model capacity, such that the ResnetVAE with RealNVP prior is already very close to the limitation of test NLLs in such condition, where adding an auxiliary latent variable will not help.

We also report the reconstruction loss, the KL divergence D KL (q φ (z 2 , z 1 |x) p θ (z 2 , z 1 )) and D KL (q φ (z 2 , z 1 |x) p θ (z 2 , z 1 |x)) of ResnetHVAE with or without RealNVP p(z 2 ), which have the same trend as ResnetVAE (Table 6 ).

Table B .23: Average test ELBO ("elbo"), reconstruction loss ("recons"), D KL (q φ (z 2 , z 1 |x) p θ (z 2 , z 1 )) ("kl"), and D KL (q φ (z 2 , z 1 |x) p θ (z 2 , z 1 |x)) ("kl z|x ") of of ResnetHVAEs with different prior. -227.56 -207.46 20.10 2.38 -225.99 -204.48 Rosca et al. (2018) observed that some of the samples from unit Gaussian prior can have low q φ (z) and low visual quality, and they name these samples the "low posterior samples". Unlike this paper, they tried various approaches to match q φ (z) to unit Gaussian p λ (z), including adversarial training methods, but still found low posterior samples scattering across the whole prior. We shall see that learning the prior can avoid having high p λ (z) on low posterior samples, and thus we suggest adopting learned prior as a cheap solution to this problem.

We shall first introduce the algorithm proposed by Rosca et al. (2018) , which can be used to obtain low posterior samples (z samples which have low likelihoods on q φ (z)) from a trained VAE. Their algorithm first samples a large number of z from the prior p λ (z), then uses Monte Carlo estimator to evaluate q φ (z) on these z samples, and finally chooses a certain number of z with the lowest q φ (z) likelihoods as the low posterior samples. They also sampled one x from p θ (x|z) for each low posterior sample z, and plotted the sample means of these x and the histograms of ELBO on these x. Although we have found their Monte Carlo estimator for D KL (q φ (z) p λ (z)) vulnerable (see Appendix B.8), their low posterior samples algorithm only ranks q φ (z) for each z, and the visual results of their algorithm seem plausible. Thus we think this algorithm should be still convincing enough, and we also use it to obtain low posterior samples.

To compare the learned prior with unit Gaussian prior on such low posterior samples, we first train a ResnetVAE with unit Gaussian prior (denoted as standard ResnetVAE), and then add a post-hoc trained RealNVP prior upon this original ResnetVAE (denoted as post-hoc trained ResnetVAE).

We then obtain 10,000 z samples from the standard ResnetVAE, and choose 100 z with the lowest q φ (z) among these 10,000 samples, evaluated on standard ResnetVAE. Fixing these 100 z samples, we plot the histograms of log p λ (z) w.r. there is likely to be little gain in test NLL, but the computation time will grow even larger. That's why we do not try larger K in our experiments.

:

The term "hierarchical latent variables" refers to multiple layers of latent variables, formulated as p(x) = E p(z 1 ) E p(z

|z 1 ) . . . E p(z K |z K-1 ,...,z 1 ) [p(x|z1, . . . , zK )]; while "one latent variable" refers to just one z in standard VAEs. Both z k and z are multi-dimensional tensors. Also, "deep" refers to many hierarchical latent variables, while "shallow" refers to few latent variables.

Figures and Tables:

Figure 1: Sample means from p λ (z) of Resnet-VAE with: (left) Gaussian prior; (right) Real-NVP prior. The last column of each 6x6 grid shows the training set images, most similar to the second-to-last column in pixel-wise L2 distance.

Figure 3: Histograms of: (left) per-dimensional stds of q φ (z|x); (middle) distances between closest pairs of q φ (z|x); and (right) normalized distances. See Appendix B.4 for formulation.

Figure 4: Rate (D KL (q φ (z|x) p θ (z)) and distortion (-reconstruction loss) of β-ResnetVAE trained with different β and prior flow depth K.

defined on the set of functions f (x) which satisfies the boundary conditions y(a) = A, y(b) = B, then a necessary condition for L[f ] to have an extremum for a given function f (x) is that, f (x) satisfy Euler's equation(Gelfand & Silverman, 2000, page 15):

Linear[k] indicates a linear dense layer with k outputs. Dense[k] indicates a non-linear dense layer. a → b indicates a composition of a and b, e.g., Dense[m] → Dense[n] indicates two successive dense layers, where the first layer has m outputs and the second layer has n outputs. These two dense layers can also be abbreviated as Dense[m → n].

denotes two successive non-linear convolution layers. Resnet[•] denotes non-linear resnet layer(s)(Zagoruyko & Komodakis, 2016). All Conv and Resnet layers by default use 3x3 kernels, unless the kernel size is specified as subscript (e.g., Conv 1×1 denotes a 1x1 convolution layer). The strides of Conv and Resnet layers are automatically determined by the input and output shapes, which is 2 in most cases. LinearConv[•] denotes linear convolution layer(s). DeConv[•] and DeResnet[•] denote deconvolution and deconvolutional resnet layers, respectively. PixelCNN[•] is a PixelCNN layer proposed by Salimans et al. (2017). It uses resnet layers, instead of convolution layers. Details can be found in its original paper. Flatten indicates to reshape the input 3-d tensor into a vector, while UnFlatten[H × W × C] indicates to reshape the input vector into a 3-d tensor of shape H × W × C. Concat[a, b] indicates to concat the output of a and b along the last axis.

Figure B.1: Samples of: (a) ResnetVAE with Gaussian p(z); and (b) ResnetVAE with RealNVP p(z). Both models use just one latent variable of shape 8 × 8 × 32.

Figure B.2:The average learned std of Gaussian q φ (z|x) and the test NLL of ResnetVAE with Real-NVP and Gaussian p(z) on MNIST. We also plot "min std" as a reference, the minimum attainable std clipped by the configured mininum logstd.

Figure B.3: Samples from ResnetVAE trained on MNIST. The last column of each 10x10 grid shows the training set images, most similar to the second-to-last column in pixel-wise L2 distance.

Figure B.4: Samples from ResnetVAE trained on FashionMNIST. The last column of each 10x10 grid shows the training set images, most similar to the second-to-last column in pixel-wise L2 distance.

Figure B.5: Samples from ResnetVAE trained on Omniglot. The last column of each 10x10 grid shows the training set images, most similar to the second-to-last column in pixel-wise L2 distance.

Figure B.7: (left) Histograms of log p λ (z) of the low posterior samples, (middle) histograms of ELBO of x samples corresponding to each low posterior sample, and (right) the means of these x, on standard ResnetVAE and post-hoc trained ResnetVAE.

, we compare ResnetVAE and PixelVAE with RealNVP prior to other approaches on StaticMNIST and MNIST. See Tables B.10 and B.11 for results on Omniglot and FashionMNIST, which have a similar trend. All models except ours and that of Huang et al. (2017) used at least 2 latent variables. For comparison between our model and Huang et al. (2017), see Appendix B.7. Test NLL on StaticMNIST. " † " indicates a hierarchical model with 2 latent variables, while " ‡ " indicates at least 3 latent variables.

Table content: <table><tr><td>Model</td><td>NLL</td></tr><tr><td>Models without PixelCNN decoder</td><td/></tr><tr><td>ConvHVAE + Lars prior  † (Bauer &amp;</td><td>81.70</td></tr><tr><td>Mnih, 2019)</td><td/></tr><tr><td>ConvHVAE + VampPrior  † (Tomczak</td><td>81.09</td></tr><tr><td>&amp; Welling, 2018)</td><td/></tr><tr><td>ResConv + RealNVP prior (Huang</td><td>81.44</td></tr><tr><td>et al., 2017)</td><td/></tr><tr><td>VAE + IAF  ‡ (Kingma et al., 2016)</td><td>79.88</td></tr><tr><td>BIVA  ‡ (Maaløe et al., 2019)</td><td>78.59</td></tr><tr><td>ConvVAE + RNVP p(z), K = 50</td><td>80.09</td></tr><tr><td colspan="2">ResnetVAE + RNVP p(z), K = 50 79.84</td></tr><tr><td>Models with PixelCNN decoder</td><td/></tr><tr><td>VLAE  ‡ (Chen et al., 2017)</td><td>79.03</td></tr><tr><td>PixelHVAE + VampPrior  † (Tomczak</td><td>79.78</td></tr><tr><td>&amp; Welling, 2018)</td><td/></tr><tr><td>PixelVAE + RNVP p(z), K = 50</td><td>79.01</td></tr></table>

Test NLL on MNIST. " † " and " ‡ " has the same meaning as Table1.

Table content: <table><tr><td>Model</td><td>NLL</td></tr><tr><td>Models without PixelCNN decoder</td><td/></tr><tr><td>ConvHVAE + Lars prior  † (Bauer &amp;</td><td>80.30</td></tr><tr><td>Mnih, 2019)</td><td/></tr><tr><td>ConvHVAE + VampPrior  † (Tomczak</td><td>79.75</td></tr><tr><td>&amp; Welling, 2018)</td><td/></tr><tr><td>VAE + IAF  ‡ (Kingma et al., 2016)</td><td>79.10</td></tr><tr><td>BIVA  ‡ (Maaløe et al., 2019)</td><td>78.41</td></tr><tr><td>ConvVAE + RNVP p(z), K = 50</td><td>78.61</td></tr><tr><td colspan="2">ResnetVAE + RNVP p(z), K = 50 78.49</td></tr><tr><td>Models with PixelCNN decoder</td><td/></tr><tr><td>VLAE  ‡ (Chen et al., 2017)</td><td>78.53</td></tr><tr><td>PixelVAE  † (Gulrajani et al., 2017)</td><td>79.02</td></tr><tr><td>PixelHVAE + VampPrior  † (Tomczak</td><td>78.45</td></tr><tr><td>&amp; Welling, 2018)</td><td/></tr></table>

1 samples images from ResnetVAE with/without RealNVP prior. Compared to standard ResnetVAE, ResnetVAE with RealNVP prior produces fewer digits that are hard to interpret. The Average test NLL (lower is better) of different models, with Gaussian prior & Gaussian posterior ("standard"), Gaussian prior & RealNVP posterior ("RNVP q(z|x)"), and RealNVP prior & Gaussian posterior ("RNVP p(z)"). Flow depth K = 20.

Table content: <table><tr><td/><td/><td colspan="2">DenseVAE</td><td/><td colspan="3">ResnetVAE</td><td colspan="2">PixelVAE</td></tr><tr><td>Datasets</td><td colspan="2">standard</td><td>RNVP q(z|x)</td><td>RNVP p(z)</td><td>standard</td><td>RNVP q(z|x)</td><td>RNVP p(z)</td><td>standard</td><td>RNVP q(z|x)</td><td>RNVP p(z)</td></tr><tr><td>StaticMNIST</td><td/><td colspan="9">88.84 86.07 84.87 82.95 80.97 79.99 79.47 79.09 78.92</td></tr><tr><td>MNIST</td><td/><td colspan="9">84.48 82.53 80.43 81.07 79.53 78.58 78.64 78.41 78.15</td></tr><tr><td colspan="11">FashionMNIST 228.60 227.79 226.11 226.17 225.02 224.09 224.22 223.81 223.40</td></tr><tr><td>Omniglot</td><td colspan="10">106.42 102.97 102.19 96.99 94.30 93.61 89.83 89.69 89.61</td></tr><tr><td colspan="6">Table 4: Test NLL of ResnetVAE on MNIST, with</td><td colspan="5">Table 5: Average number of active units</td></tr><tr><td colspan="6">RealNVP posterior ("q(z|x)"), RealNVP prior</td><td colspan="5">of ResnetVAE, with standard prior &amp; poste-</td></tr><tr><td colspan="6">("p(z)"), and RealNVP prior &amp; posterior ("both").</td><td colspan="5">rior ("standard"), RealNVP posterior ("RNVP</td></tr><tr><td colspan="6">Flow depth K is 2K 0 for the posterior or the prior</td><td colspan="5">q(z|x)"), and RealNVP prior ("RNVP p(z)").</td></tr><tr><td colspan="6">in "q(z|x)" and "p(z)", while K 0 for both the pos-terior and the prior in "both".</td><td/><td/><td colspan="3">ResnetVAE</td></tr><tr><td/><td/><td/><td colspan="2">K 0</td><td/><td colspan="2">Datasets</td><td>standard</td><td>RNVP q(z|x)</td><td>RNVP p(z)</td></tr><tr><td colspan="2">ResnetVAE with</td><td>1</td><td>5</td><td>10</td><td>20</td><td colspan="2">StaticMNIST</td><td>30</td><td>40</td><td>40</td></tr><tr><td colspan="6">q(z|x), K = 2K 0 80.29 79.68 79.53 79.49</td><td colspan="2">MNIST</td><td>25.3</td><td>40</td><td>40</td></tr><tr><td>both, K = K 0</td><td/><td colspan="4">79.85 79.01 78.71 78.56</td><td colspan="2">FashionMNIST</td><td>27</td><td>64</td><td>64</td></tr><tr><td colspan="6">p(z), K = 2K 0 79.58 78.75 78.58 78.51</td><td colspan="2">Omniglot</td><td>59.3</td><td>64</td><td>64</td></tr><tr><td colspan="11">last column of each 6x6 grid shows the training set images, most similar to the second-to-last column</td></tr><tr><td colspan="11">in pixel-wise L2 distance. There are differences between the last two columns, indicating our model</td></tr><tr><td colspan="9">is not just memorizing the training data. More samples are in Appendix B.11.</td><td/></tr><tr><td colspan="3">5.4 ABLATION STUDY</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="11">RealNVP prior leads to substantially lower NLLs than standard VAE and RealNVP posterior</td></tr><tr><td colspan="11">Table 3 shows the NLLs of DenseVAE, ResnetVAE and PixelVAE with flow depth K = 20. The</td></tr><tr><td colspan="11">NLLs of ConvVAE (with similar trends as ResnetVAE) and the standard deviations of all models</td></tr><tr><td colspan="11">are reported in Table B.12, while the NLLs of ResnetVAE with different K (range from 1 to 50) can</td></tr></table>

Average test ELBO ("elbo"), reconstruction loss ("recons"), E p (x) D KL (q φ (z|x) p λ (z)) ("kl"), and E p (x) D KL (q φ (z|x) p θ (z|x)) ("kl z|x ") of ResnetVAE with different priors.

Table content: <table><tr><td/><td/><td colspan="3">standard</td><td/><td/><td/><td colspan="2">RealNVP p(z)</td></tr><tr><td>Datasets</td><td>elbo</td><td colspan="2">recons</td><td>kl</td><td/><td>kl z|x</td><td>elbo</td><td>recons</td><td>kl</td><td>kl z|x</td></tr><tr><td>StaticMNIST</td><td>-87.61</td><td colspan="2">-60.09</td><td colspan="3">27.52 4.67</td><td>-82.85</td><td>-54.32</td><td>28.54 2.87</td></tr><tr><td>MNIST</td><td>-84.62</td><td colspan="2">-58.70</td><td colspan="3">25.92 3.55</td><td>-80.34</td><td>-53.64</td><td>26.70 1.76</td></tr><tr><td colspan="10">FashionMNIST -228.91 -208.94 19.96 2.74 -225.97 -204.66 21.31 1.88</td></tr><tr><td>Omniglot</td><td>-104.87</td><td colspan="2">-66.98</td><td colspan="3">37.89 7.88</td><td>-99.60</td><td>-61.21</td><td>38.39 5.99</td></tr><tr><td/><td/><td/><td colspan="2">Standard</td><td/><td colspan="2">RealNVP p(z)</td><td/></tr><tr><td colspan="3">0.00 0.25 0.50 0.75 1.00 0</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5 0</td><td colspan="2">5 10 15 20 25</td></tr></table>

Iterative training: Proposed by us, we alternate between training p θ (x|z) & q φ (z|x) and training p λ (z), for multiple iterations. The first iteration to train p θ (x|z) & q φ (z|x) should use the unit Gaussian prior. Early-stopping should be performed during the whole process if necessary. See Algorithm B.1 for detailed procedure of this strategy. We adopt this method mainly for investigating why post-hoc training does not work as well as joint training.

Table content: <table/>

1: Avg. Mean(z k ) and Var(z k ) of regularized/un-regularized ResnetVAE with RealNVP prior.

Table content: <table><tr><td/><td colspan="2">regularized</td><td colspan="2">un-regularized</td></tr><tr><td>Datasets</td><td colspan="2">Avg. Mean(z k ) Avg. Var(z k )</td><td colspan="2">Avg. Mean(z k ) Avg. Var(z k )</td></tr><tr><td>StaticMNIST MNIST FashionMNIST Omniglot</td><td>-0.02 ± 0.03 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.02</td><td>0.93 ± 0.01 0.98 ± 0.01 0.98 ± 0.00 0.95 ± 0.00</td><td>0.05 ± 0.37 -0.01 ± 0.02 0.03 ± 0.03 0.00 ± 0.01</td><td>1.50 ± 0.02 0.78 ± 0.04 0.83 ± 0.04 0.24 ± 0.01</td></tr></table>

1 shows the average Mean(z k ) and Var(z k ) of ResnetVAE with RealNVP prior, computed on test data. Average Mean(z k ) is defined as 1

Table content: <table><tr><td/><td>Dim(z)</td><td>Dim(z) k=1</td><td>Mean(z k ), while average Var(z k ) is</td></tr><tr><td>defined as 1 Dim(z)</td><td>Dim(z) k=1</td><td/></tr></table>

2: Test bpd on Cifar10.

Table content: <table><tr><td>Model</td><td>bpd</td></tr><tr><td>Models without PixelCNN decoder</td><td/></tr><tr><td>VAE + IAF (Kingma et al., 2016)</td><td>3.11</td></tr><tr><td>BIVA (Maaløe et al., 2019)</td><td>3.08</td></tr><tr><td>ResnetVAE</td><td>3.98</td></tr><tr><td>ResnetVAE + RealNVP p(z), K = 50</td><td>3.54</td></tr><tr><td>Models with PixelCNN decoder</td><td/></tr><tr><td>VLAE (Chen et al., 2017)</td><td>2.95</td></tr></table>

4: Test NLL of our ResnetVAE with low complexity and ResConv VAE(Huang et al., 2017)   on StaticMNIST. "pure" refers to the original RealNVP flow, without invertible dense and actnorm.

Table content: <table><tr><td>Model</td><td>p(z) flow</td><td>q(z|x) flow</td><td colspan="2">Epochs Initial lr</td><td>Early stopping</td><td>NLL</td></tr><tr><td>Our ResnetVAE</td><td>pure, K = 12</td><td>/</td><td>300</td><td>0.0005</td><td>no</td><td>81.05</td></tr><tr><td>Low Complexity</td><td colspan="2">pure, K = 8 pure, K = 8</td><td>300</td><td>0.0005</td><td>no</td><td>80.88</td></tr><tr><td/><td>K = 12</td><td>/</td><td>300</td><td>0.0005</td><td>no</td><td>80.67</td></tr><tr><td/><td>K = 8</td><td>K = 8</td><td>300</td><td>0.0005</td><td>no</td><td>80.66</td></tr><tr><td>ResConv VAE</td><td>pure, K = 12</td><td>/</td><td>300</td><td>0.0005</td><td>no</td><td>81.44</td></tr><tr><td>(Huang et al., 2017)</td><td colspan="2">pure, K = 8 pure, K = 8</td><td>300</td><td>0.0005</td><td>no</td><td>80.81</td></tr></table>

TableB.6: Test NLL of ResnetVAE, with prior trained by: joint training, iterative training, post-hoc training, and standard VAE ("none") as reference. Flow depth K = 20. Average reconstruction loss ("recons"), E p (x) D KL (q φ (z|x) p λ (z)) ("kl") and active units ("au") of ResnetVAE with iteratively trained and post-hoc trained RealNVP priors.Additional quantitative results are shown in TableB.8 to B.20. TableB.8: Test NLL on StaticMNIST. " † " and " ‡ " has the same meaning as Table1.TableB.9: Test NLL on MNIST. " † " and " ‡ " has the same meaning as Table1. 10: Test NLL on Omniglot. " † " and " ‡ " has the same meaning as Table1.

Table content: <table><tr><td>Model</td><td/><td/><td/><td/><td/><td/><td/><td/><td>NLL</td></tr><tr><td colspan="3">Models without PixelCNN decoder</td><td/><td colspan="2">ResnetVAE</td><td/><td/><td/></tr><tr><td colspan="7">Datasets ConvHVAE + Lars prior  † (Bauer &amp; Mnih, 2019) joint iterative post-hoc</td><td>none</td><td/><td>80.30</td></tr><tr><td colspan="5">StaticMNIST ConvHVAE + VampPrior  † (Tomczak &amp; Welling, 2018) 79.99 80.63 MNIST 78.58 79.61 FashionMNIST 224.09 224.88 VAE + IAF  ‡ (Kingma et al., 2016) BIVA  ‡ (Maaløe et al., 2019)</td><td>80.86 79.90 225.22</td><td/><td>82.95 81.07 226.17</td><td/><td>79.75 79.10 ± 0.07 78.41</td></tr><tr><td colspan="5">Omniglot ConvVAE + RNVP p(z), K = 50 ResnetVAE + RNVP p(z), K = 50 Models with PixelCNN decoder VLAE  ‡ (Chen et al., 2017) Table B.7: iterative 93.61 94.43 PixelVAE  † (Gulrajani et al., 2017) PixelHVAE + VampPrior  † (Tomczak &amp; Welling, 2018)</td><td>94.87</td><td colspan="2">96.99 post-hoc</td><td/><td>78.61 ± 0.01 78.49 ± 0.01 78.53 79.02 78.45</td></tr><tr><td colspan="3">Datasets PixelVAE + RNVP p(z), K = 50 recons</td><td>kl</td><td>au</td><td>recons</td><td/><td>kl</td><td>au</td><td>78.12 ± 0.04</td></tr><tr><td/><td>StaticMNIST</td><td>-58.0</td><td colspan="2">26.4 38.7</td><td>-60.1</td><td/><td>25.3</td><td>30</td></tr><tr><td/><td>DynamicMNIST</td><td>-57.2</td><td>25.1</td><td>40</td><td>-58.7</td><td/><td>24.7</td><td colspan="2">25.3</td></tr><tr><td/><td>FashionMNIST</td><td>-207.8</td><td>19.4</td><td>64</td><td>-208.9</td><td/><td>19.0</td><td>27</td></tr><tr><td>Model</td><td>Omniglot</td><td>-63.3</td><td>37.9</td><td>64</td><td>-67.0</td><td/><td>35.8</td><td colspan="2">59.3</td><td>NLL</td></tr><tr><td colspan="3">Models without PixelCNN decoder</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="4">ConvHVAE + Lars prior  † (Bauer &amp; Mnih, 2019)</td><td/><td/><td/><td/><td/><td>97.08</td></tr><tr><td colspan="5">B.10 ADDITIONAL QUANTITATIVE RESULTS ConvHVAE + VampPrior  † (Tomczak &amp; Welling, 2018)</td><td/><td/><td/><td/><td>97.56</td></tr><tr><td colspan="2">BIVA  ‡ (Maaløe et al., 2019)</td><td/><td/><td/><td/><td/><td/><td/><td>91.34</td></tr><tr><td colspan="3">ConvVAE + RNVP p(z), K = 50 ResnetVAE + RNVP p(z), K = 50 Models with PixelCNN decoder</td><td/><td/><td/><td/><td/><td/><td>93.62 ± 0.02 93.52 ± 0.02</td></tr><tr><td colspan="5">Model VLAE  ‡ (Chen et al., 2017) PixelHVAE + VampPrior  † (Tomczak &amp; Welling, 2018) Models without PixelCNN decoder ConvHVAE + Lars prior  † (Bauer &amp; Mnih, 2019) PixelVAE + RNVP p(z), K = 50</td><td/><td/><td/><td/><td>89.83 NLL 89.76 81.70 89.60 ± 0.01</td></tr><tr><td colspan="5">ConvHVAE + VampPrior  † (Tomczak &amp; Welling, 2018)</td><td/><td/><td/><td/><td>81.09</td></tr><tr><td colspan="4">ResConv + RealNVP prior (Huang et al., 2017)</td><td/><td/><td/><td/><td/><td>81.44</td></tr><tr><td colspan="3">VAE + IAF  ‡ (Kingma et al., 2016)</td><td/><td/><td/><td/><td/><td/><td>79.88</td></tr><tr><td colspan="2">BIVA  ‡ (Maaløe et al., 2019)</td><td/><td/><td/><td/><td/><td/><td/><td>78.59</td></tr><tr><td colspan="3">ConvVAE + RNVP p(z), K = 50 ResnetVAE + RNVP p(z), K = 50 Models with PixelCNN decoder</td><td/><td/><td/><td/><td/><td/><td>80.09 ± 0.01 79.84 ± 0.04</td></tr><tr><td colspan="2">VLAE  ‡ (Chen et al., 2017)</td><td/><td/><td/><td/><td/><td/><td/><td>79.03</td></tr><tr><td colspan="5">PixelHVAE + VampPrior  † (Tomczak &amp; Welling, 2018)</td><td/><td/><td/><td/><td>79.78</td></tr><tr><td colspan="3">PixelVAE + RNVP p(z), K = 50</td><td/><td/><td/><td/><td/><td/><td>79.01 ± 0.03</td></tr></table>

11: Test NLL on FashionMNIST. " † " and " ‡ " has the same meaning as Table1.

Table content: <table><tr><td>Model</td><td>NLL</td></tr><tr><td>Models without PixelCNN decoder</td><td/></tr><tr><td>ConvHVAE + Lars prior  † (Bauer &amp; Mnih, 2019)</td><td>225.92</td></tr><tr><td>ConvVAE + RNVP p(z), K = 50 ResnetVAE + RNVP p(z), K = 50 Models with PixelCNN decoder</td><td>224.64 ± 0.01 224.07 ± 0.01</td></tr><tr><td>PixelVAE + RNVP p(z), K = 50</td><td>223.36 ± 0.06</td></tr><tr><td>B.11 ADDITIONAL QUALITATIVE RESULTS</td><td/></tr><tr><td>Additional qualitative results are shown in Fig. B.3 to B.5.</td><td/></tr></table>

12: Average test NLL (lower is better) of different models, with Gaussian prior & Gaussian posterior ("standard"), Gaussian prior & RealNVP posterior ("RNVP q(z|x)"), and RealNVP prior & Gaussian posterior ("RNVP p(z)"). Flow depth K = 20.

Table content: <table><tr><td/><td/><td/><td/><td>Datasets</td></tr><tr><td colspan="2">Models</td><td>StaticMNIST</td><td>MNIST</td><td>FashionMNIST</td><td>Omniglot</td></tr><tr><td>DenseVAE ConvVAE ResnetVAE PixelVAE</td><td colspan="5">standard RNVP q(z|x) 86.07 ± 0.11 82.53 ± 0.00 227.79 ± 0.01 102.97 ± 0.06 88.84 ± 0.05 84.48 ± 0.03 228.60 ± 0.03 106.42 ± 0.14 RNVP p(z) 84.87 ± 0.05 80.43 ± 0.01 226.11 ± 0.02 102.19 ± 0.12 standard 83.63 ± 0.01 82.14 ± 0.01 227.51 ± 0.08 97.87 ± 0.02 RNVP q(z|x) 81.11 ± 0.03 80.09 ± 0.01 226.03 ± 0.00 94.90 ± 0.03 RNVP p(z) 80.06 ± 0.07 78.67 ± 0.01 224.65 ± 0.01 93.68 ± 0.01 standard 82.95 ± 0.09 81.07 ± 0.03 226.17 ± 0.05 96.99 ± 0.04 RNVP q(z|x) 80.97 ± 0.05 79.53 ± 0.03 225.02 ± 0.01 94.30 ± 0.02 RNVP p(z) 79.99 ± 0.02 78.58 ± 0.01 224.09 ± 0.01 93.61 ± 0.04 standard 79.47 ± 0.02 78.64 ± 0.02 224.22 ± 0.06 89.83 ± 0.04 RNVP q(z|x) 79.09 ± 0.01 78.41 ± 0.01 223.81 ± 0.00 89.69 ± 0.01 RNVP p(z) 78.92 ± 0.02 78.15 ± 0.04 223.40 ± 0.07 89.61 ± 0.03</td></tr></table>

13: Test NLL of ResnetVAE on MNIST, with RealNVP posterior ("q(z|x)"), RealNVP prior ("p(z)"), and RealNVP prior & posterior ("both"). Flow depth K is 2K 0 for the posterior or the prior in "q(z|x)" and "p(z)", while K 0 for both the posterior and the prior in "both". = 2K 0 80.29 ± 0.03 80.00 ± 0.00 79.68  ± 0.01 79.53 ± 0.02 79.49 ± 0.02 both, K = K 0 79.85 ± 0.01 79.42 ± 0.02 79.01 ± 0.02 78.71 ± 0.00 78.56 ± 0.01 p(z), K = 2K 0 79.58 ± 0.02 79.21 ± 0.01 78.75 ± 0.01 78.58 ± 0.01 78.51 ± 0.02 Table B.14: Test reconstruction loss of ResnetVAE on MNIST

Table content: <table><tr><td/><td/><td/><td>K 0</td><td/><td/></tr><tr><td>ResnetVAE with</td><td>1</td><td>2</td><td>5</td><td>10</td><td>20</td></tr><tr><td>q(z|x), K</td><td/><td/><td/><td/><td/></tr></table>

15: Test NLL of ResnetVAE, with RealNVP prior of different flow depth K. Even K = 1, RealNVP prior improves NLLs by about 1 nat. There is no over-fitting for K up to 50. However, we think over-fitting may happen with sufficiently powerful prior, 50-blocks of RealNVP may simply not be deep enough. D KL (q φ (z|x) p λ (z)) ("kl") and E p (x) D KL (q φ (z|x) p θ (z|x)) ("kl z|x ") of various DenseVAE. Flow depth K = 20. KL (q φ (z|x) p λ (z)) ("kl") and E p (x) D KL (q φ (z|x) p θ (z|x)) ("kl z|x ") of various ConvVAE. Flow depth K = 20. D KL (q φ (z|x) p λ (z)) ("kl") and E p (x) D KL (q φ (z|x) p θ (z|x)) ("kl z|x ") of various ResnetVAE. Flow depth K = 20. KL (q φ (z|x) p λ (z)) ("kl") and E p (x) D KL (q φ (z|x) p θ (z|x)) ("kl z|x ") of various PixelVAE. Flow depth K = 20.

Table content: <table><tr><td>Table B.16: Table B.18:</td><td colspan="3">Average test ELBO ("elbo"), Average test ELBO ("elbo"),</td><td colspan="2">reconstruction loss ("recons"), reconstruction loss ("recons"),</td></tr><tr><td colspan="5">E p (x) standard E p (x) standard</td></tr><tr><td/><td>Datasets Datasets</td><td>elbo elbo</td><td>recons recons</td><td/><td>kl kl</td><td>kl z|x kl z|x</td></tr><tr><td/><td>StaticMNIST StaticMNIST</td><td>-94.53 -87.61</td><td>-65.54 -60.09</td><td colspan="2">29.00 5.69 27.52 4.67</td></tr><tr><td/><td>MNIST MNIST</td><td>-88.19 -84.62</td><td>-62.04 -58.70</td><td colspan="2">26.16 3.71 25.92 3.55</td></tr><tr><td/><td colspan="5">FashionMNIST -230.81 -211.71 19.11 2.22 FashionMNIST -228.91 -208.94 19.96 2.74</td></tr><tr><td/><td>Omniglot Omniglot</td><td>-113.59 -104.87</td><td>-78.25 -66.98</td><td colspan="2">35.33 7.17 37.89 7.88</td></tr><tr><td/><td/><td/><td colspan="3">RealNVP q(z|x) RealNVP q(z|x)</td></tr><tr><td/><td>Datasets Datasets</td><td>elbo elbo</td><td>recons recons</td><td/><td>kl kl</td><td>kl z|x kl z|x</td></tr><tr><td/><td>StaticMNIST StaticMNIST</td><td>-90.78 -84.72</td><td>-62.55 -58.10</td><td colspan="2">28.23 4.71 26.63 3.75</td></tr><tr><td/><td>MNIST MNIST</td><td>-85.15 -81.95</td><td>-58.83 -56.15</td><td colspan="2">26.31 2.62 25.80 2.42</td></tr><tr><td/><td colspan="5">FashionMNIST -229.53 -210.35 19.18 1.75 FashionMNIST -227.16 -206.61 20.54 2.14</td></tr><tr><td/><td>Omniglot Omniglot</td><td>-108.54 -100.30</td><td>-73.71 -63.34</td><td colspan="2">34.83 5.57 36.96 6.00</td></tr><tr><td/><td/><td/><td colspan="3">RealNVP p(z) RealNVP p(z)</td></tr><tr><td/><td>Datasets Datasets</td><td>elbo elbo</td><td>recons recons</td><td/><td>kl kl</td><td>kl z|x kl z|x</td></tr><tr><td/><td>StaticMNIST StaticMNIST</td><td>-88.87 -82.85</td><td>-63.35 -54.32</td><td colspan="2">25.52 4.00 28.54 2.87</td></tr><tr><td/><td>MNIST MNIST</td><td>-82.57 -80.34</td><td>-55.99 -53.64</td><td colspan="2">26.58 2.14 26.70 1.76</td></tr><tr><td/><td colspan="5">FashionMNIST -227.72 -208.13 19.59 1.61 FashionMNIST -225.97 -204.66 21.31 1.88</td></tr><tr><td/><td>Omniglot Omniglot</td><td>-107.99 -99.60</td><td>-73.10 -61.21</td><td colspan="2">34.89 5.80 38.39 5.99</td></tr><tr><td>Table B.17: Table B.19:</td><td colspan="3">Average test ELBO ("elbo"), Average test ELBO ("elbo"),</td><td colspan="2">reconstruction loss ("recons"), reconstruction loss ("recons"),</td></tr><tr><td colspan="5">E p (x) D standard E p (x) D standard K 0 ResnetVAE with 1 2 5 Datasets elbo recons Datasets elbo recons</td><td>kl kl</td><td>kl z|x kl z|x</td><td>10</td><td>20</td></tr><tr><td colspan="6">StaticMNIST StaticMNIST q(z|x), K = 2K 0 -57.20 ± 0.05 -56.82 ± 0.01 -56.37 ± 0.04 -56.15 ± 0.05 -55.98 ± 0.05 -88.15 -60.34 27.81 4.51 -81.06 -69.02 12.03 1.59 both, K = K 0 MNIST -85.89 -58.97 26.92 3.75 MNIST -79.88 -68.73 11.15 1.24 -56.34 ± 0.01 -55.64 ± 0.02 -54.68 ± 0.04 -54.04 ± 0.04 -53.76 ± 0.02 FashionMNIST -230.43 -210.24 20.18 2.92 FashionMNIST -225.60 -214.15 11.45 1.38 p(z), K = 2K 0 -55.75 ± 0.04 -54.90 ± 0.03 -54.00 ± 0.04 -53.64 ± 0.02 -53.49 ± 0.03 Omniglot -104.70 -66.00 38.70 6.83 Omniglot -91.58 -82.80 8.78 1.75</td></tr><tr><td/><td/><td/><td colspan="3">RealNVP q(z|x) RealNVP q(z|x)</td></tr><tr><td/><td>Datasets Datasets</td><td>elbo elbo</td><td>recons recons</td><td/><td>kl kl</td><td>kl z|x kl z|x</td></tr><tr><td/><td>StaticMNIST StaticMNIST</td><td>-83.92 -80.65</td><td>-56.67 -67.91</td><td colspan="2">27.25 2.82 12.75 1.57</td></tr><tr><td/><td>MNIST MNIST</td><td>-82.46 -79.59</td><td>-56.08 -68.17</td><td colspan="2">26.38 2.37 11.42 1.18</td></tr><tr><td/><td colspan="5">Datasets FashionMNIST -228.20 -207.33 20.88 2.18 FashionMNIST -225.05 -213.41 11.64 1.24</td></tr><tr><td colspan="6">Flow depth StaticMNIST Omniglot Omniglot 0 82.95 ± 0.09 1 81.76 ± 0.04 Datasets Datasets 2 81.30 ± 0.02 5 80.64 ± 0.06 StaticMNIST StaticMNIST 10 80.26 ± 0.05 MNIST MNIST 20 79.99 ± 0.02 FashionMNIST -226.65 -204.93 21.72 2.00 MNIST FashionMNIST -99.92 -62.48 37.44 5.02 -91.26 -83.17 8.10 1.57 81.07 ± 0.03 226.17 ± 0.05 RealNVP p(z) RealNVP p(z) 80.02 ± 0.02 225.27 ± 0.03 elbo recons kl kl z|x elbo recons kl kl z|x 79.58 ± 0.02 224.78 ± 0.02 79.09 ± 0.02 224.37 ± 0.01 -82.91 -54.01 28.90 2.86 -80.60 -62.22 18.38 1.68 78.75 ± 0.01 224.18 ± 0.01 -80.42 -53.33 27.09 1.75 -79.28 -64.41 14.87 1.13 78.58 ± 0.01 224.09 ± 0.01 FashionMNIST -224.65 -210.16 14.49 1.26 30 79.90 ± 0.05 78.52 ± 0.01 224.07 ± 0.01 50 79.84 ± 0.04 78.49 ± 0.01 224.07 ± 0.01 Omniglot -98.59 -59.07 39.51 4.91 Omniglot -91.78 -79.70 12.07 2.16</td><td>Omniglot 96.99 ± 0.04 96.20 ± 0.06 95.35 ± 0.06 94.47 ± 0.01 93.92 ± 0.02 93.61 ± 0.04 93.52 ± 0.02 93.53 ± 0.02</td></tr></table>

20: Test NLL of ResnetVAE, with prior trained by: joint training, iterative training, posthoc training, and standard VAE ("none") as reference. Flow depth K = 20. Avg. std of q φ (z|x) of trained with different β and prior flow depth K.B.13 ADDITIONAL RESULTS: TWO LEVEL HIERARCHICAL VAE WITH FLOW PRIOR

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="4">ResnetVAE</td></tr><tr><td>Datasets</td><td colspan="2">joint</td><td/><td colspan="2">iterative</td><td/><td colspan="3">post-hoc</td><td>none</td></tr><tr><td colspan="10">StaticMNIST MNIST FashionMNIST 224.09 ± 0.01 224.88 ± 0.02 225.22 0.01 226.17 ± 0.05 79.99 ± 0.02 80.63 ± 0.02 80.86 ± 0.04 82.95 ± 0.09 78.58 ± 0.01 79.61 ± 0.01 79.90 ± 0.04 81.07 ± 0.03 Omniglot 93.61 ± 0.04 94.43 ± 0.11 94.87 ± 0.05 96.99 ± 0.04</td></tr><tr><td/><td colspan="2">K=0</td><td/><td colspan="2">K=1</td><td colspan="2">K=20</td><td/><td>K=50</td></tr><tr><td/><td>0.8</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.6</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>std</td><td>0.4</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.2</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.0</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0 . 1</td><td>0 . 2</td><td>0 . 3</td><td>0 . 5</td><td>0 . 7</td><td>1 . 0</td><td>2 . 0</td><td>5 . 0</td><td>1 0 . 0</td></tr><tr><td/><td/><td/><td/><td/><td colspan="2">beta</td><td/><td/></tr><tr><td>Figure B.6:</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

21: Test negative log-likelihood of ResnetHVAE, compared with ResnetVAE. Flow depth K = 20 for p λ (z) in ResnetVAE. For ResnetHVAE, p λ (z 2 ) is a flow prior with K = 20, and p λ (z 1 |z 2 ) is a Gaussian prior. ) 79.95 ± 0.01 78.56 ± 0.01 224.11 ± 0.01 93.47 ± 0.01 Table B.22: Average number of active units of ResnetHVAE of z 2 and z 1 , with unit Gaussian p(z 2 ) and p(z 1 ) ("standard"), and RealNVP p(z 2 ) and Gaussian p(z 1 ) ("RNVP p(z 2 )"). Table B.21 shows the test negative log-likelihood (NLL) of ResnetHVAE, with or without RealNVP p(z 2 ), compared with ResnetVAE, with or without RealNVP p(z). We can see that, ResnetHVAE without flow prior has significantly better test NLLs than ResnetVAE without flow prior. However, ResnetHVAE with RealNVP p(z 2 ) only has roughly the same test NLLs as ResnetVAE with RealNVP p(z).

Table content: <table><tr><td/><td/><td/><td/><td/><td>Datasets</td><td/><td/><td/></tr><tr><td colspan="2">Models</td><td/><td>StaticMNIST</td><td>MNIST</td><td colspan="2">FashionMNIST</td><td/><td>Omniglot</td></tr><tr><td>ResnetVAE ResnetHVAE</td><td colspan="8">standard RNVP p(z) 79.99 ± 0.02 78.58 ± 0.01 224.09 ± 0.01 93.61 ± 0.04 82.95 ± 0.09 81.07 ± 0.03 226.17 ± 0.05 96.99 ± 0.04 standard 81.58 ± 0.00 80.09 ± 0.03 225.18 ± 0.02 95.80 ± 0.03 RNVP p(z 2 Datasets</td></tr><tr><td/><td colspan="2">StaticMNIST</td><td colspan="2">MNIST</td><td colspan="2">FashionMNIST</td><td colspan="2">Omniglot</td></tr><tr><td>ResnetHVAE</td><td>z 1</td><td>z 2</td><td>z 1</td><td>z 2</td><td>z 1</td><td>z 2</td><td>z 1</td><td>z 2</td></tr><tr><td>standard</td><td>11.3</td><td>25</td><td>8.7</td><td>18.3</td><td>16.3</td><td>16.7</td><td>12.3</td><td>57.7</td></tr><tr><td>RealNVP p(z 2 )</td><td>1.3</td><td>40</td><td>0</td><td>40</td><td>0.3</td><td>64</td><td>0</td><td>64</td></tr><tr><td colspan="2">Experimental results</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
This submission shows that using learned autoregressive priors (real NVP) allows shallow VAEs to achieve comparable log-likelihood performances compared to more complex deep VAE architectures. I found this paper an enjoyable read, and its results quite intriguing. While most of VAE research focuses on building more powerful encoder and decoder architectures, these results show that focusing on a learned prior distribution is as important. The models introduced in this paper are not novel, but the authors introduce some tricks (gradient and std clipping) that allow them to achieve nearly SOTA results with relatively simple architectures. I think these tricks should have been demonstrated more in detail in the experiments, for example: * why is the std clipped exactly at e^-11? What happens if I increase/decrease this number? * Can the authors clarify the differences between their model and that of Huang et al, which also uses real NVP priors? If I took the exact architecture of Huang et al and used the clipping trick would it perform similarly to your model? * Would other more complex SOTA models also benefit from your clipping tricks? A very interesting addition to the paper would be running some experiments on more complex data distributions such as the natural images of celebA or CIFAR10, to understand whether your model could achieve: (1) similar improvements in terms of ELBO (2) more importantly, a quality of the generated samples comparable to deep VAE models such as VAE+IAF or BIVA. Overall I liked the paper so I am voting towards acceptance. However, while there are a considerable number of experiments in this paper, for me to increase the score I would like to see at least some of the experiments suggested above, since they could help better understand the behavior of VAEs with learned priors and make this an even more impactful paper.