NCP-VAE: Variational Autoencoders with Noise Contrastive Priors
Authors: Jyoti Aneja
Alex Schwing
Jan Kautz
Arash Vahdat
Venue:  (2021)
Keywords: Variational Autoencoders
Noise Contrastive Estimation
Sampling

Abstract: Variational autoencoders (VAEs) are one of the powerful likelihood-based generative models with applications in various domains. However, they struggle to generate high-quality images, especially when samples are obtained from the prior without any tempering. One explanation for VAEs’ poor generative quality is the prior hole problem: the prior distribution fails to match the aggregate approximate posterior. Due to this mismatch, there exist areas in the latent space with high density under the prior that do not correspond to any encoded image. Samples from those areas are decoded to corrupted images. To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior. We train the reweighting factor by noise contrastive estimation, and we generalize it to hierarchical VAEs with many latent variable groups. Our experiments confirm that the proposed noise contrastive priors improve the generative performance of state-of-the-art VAEs by a large margin on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ 256 datasets.

Under review conference paper at ICLR 2021
NCP-VAE: VARIATIONAL AUTOENCODERS WITH NOISE CONTRASTIVE PRIORS
Anonymous authors Paper under double-bling review
ABSTRACT
Variational autoencoders (VAEs) one of the powerful likelihood-based generative mod- els with applications in various domains. However, they struggle to generate high-quality images, especially when samples obtained from the prior without any tempering. One explanation for VAEs' poor generative quality is the prior hole problem: the prior distribu- tion fails match the aggregate approximate posterior. Due to this mismatch, there exist areas in the latent space with high density under the prior that do not correspond to any encoded image. Samples from those areas are decoded to corrupted images. To tackle this issue, we propose energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior. We train the reweighting factor by noise contrastive estimation, and we generalize it to hierar- chical VAEs with many latent variable groups. Our experiments confirm that the proposed noise contrastive priors improve the generative performance of state-of-the-art VAEs by large margin on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ 256 datasets.
1
INTRODUCTION
((2)
Variational autoencoders (VAEs) Kinggma & Welling 2014; Rezende et al. 2014) are one of the powerful likelihood- generative models that have ap-
plications in image generation (Brock et al. 2018 Karras et al. 2019 Razavi
p(2)
ct al.
2019). music synthesis (Dhariwal et al. 2020), speech generation (Oord
et al.
2016
Ping.Ping et. 2020), image captioning (Aneja et al. 2019 Deshpande
at.
2019 Aneja et al. (018). semi-supervised learning (Kingma et al. 2014:
x(2
Izmailov et al. 2020). and representation learning (Van Den Oord et al.
2017
Fortuin al. (018)). Although there has been tremendous progress in improv- ing the expressivity of the approximate posterior, several studies have observed that VAE priors fail to match the aggregate (approximate) posterior (Rosca)
We propose an EBM prior using the prod- uct of a base prior p(z) and reweighting factor r(z). designed to bring the base prior closer to the aggre- gate posterior q(z).
Figure 1:
ct al., 2018 [Hffiman & Johnson 2016). This phenomenon is sometimes de- scribed as holes the prior, referring to regions in the latent space that are not decoded to data-like samples. Such regions often have a high density under the prior but have a low density under the aggregate approximate posterior.
The prior hole problem is commonly tackled by increasing the flexibility of the prior via hierarchical priors (Klushyn et al., 2019), autoregressive models (Gulrajani et al. (016), a mixture of approximate posteriors (Tomezak & Welling 2018), normalizing flows (Xu et al.] 2019: Chen et al. 2016), resampled priors (Bauer & Mnih, 2019). and energy-based models (Pang et al. 2020 Vahdat et al. 2018h a: 2020). Among them, energy-base based models (EBMs) (Du & Mordatch| (21paattch[[0]]]]]1 2019 Pang et al. (020) have shown promising results learning expressive priors. However, they require running iterative MCMC steps during training which is computationally expensive, especially when the energy function is represented by a

---

Under review conference paper at ICLR 2021
neural network. Moreover, they scale poorly to hierarchical models where an EBM defined on each group of latent variables.
Our key insight in this paper is that trainable prior is brought as close as possible to the aggregate posterior as result of training a VAE. The mismatch between the prior and the aggregate posterior can be reduced by simply reweighting the prior to re-adjust its likelihood in the area of mismatch with the aggregate posterior. To represent this reweighting mechanism, we formulate the prior using an EBM that is defined by the product of a reweighting factor and base trainable prior as shown in Fig. T using neural networks and the base prior using Normal distributions.
We represent the reweighting factor
Instead of expensive MCMC sampling, we use noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010) for training the EBM prior. We show that NCE naturally trains the reweighting factor in our prior by learning a binary classifier to distinguish samples from a target distribution (i.e., samples from the approxi- mate posterior) vs. samples from noise distribution (i.e., the base trainable prior). However, since NCE's s success depends on how close the noise distribution is to the target distribution, we first train the VAE with the base prior to bring it close to the aggregate posterior. And then, we train the EBM prior using NCE.
In this paper, we make the following contributions: i) We propose an EBM prior termed noise contrastive prior (NCP) which is trained by contrasting samples from the aggregate posterior to samples from a base prior. NCPs are learned as post-training mechanism to replace the original prior with more flexible prior, which can improve the generative performance of VAEs with any structure. ii) We also show how NCPs are trained on hierarchical VAEs with many latent variable groups. We show that training hierarchical NCPs scales easily many groups, as they are trained for each latent variable group in parallel. iii) Finally, we demonstrate that NCPs improve the generative quality of VAEs by a large margin across datasets.
2 BACKGROUND
We first review VAEs, their extension to hierarchical VAES, and the prior hole problem.
Variational Autoencoders: VAFs learn a generative distribution p(x, z) = (x)))(((z)) where p(z) is a prior distribution over the latent variable z and p(x|z) is a likelihood function that generates the data x given z. VAEs are trained by maximizing a variational lower bound on the log-likelihood log p(x):
log p(x) 2 Ex= p z)] (L(((Z))))l)(() == (xve
(1)
where (z(x)) is an approximate posterior and KL is the Bellback-Leibler divergence. The final training objective is formulated by (x)( (FvAFx))) where (x) is the data distribution (Kingma & Welling 2014).
Hierarchical VAFs (HVAFs): To increase the expressivity of both prior and approximate posterior, ear- lier work adapted a hierarchical latent variable structure (Vahdat & Kautz] [020]] Kingma et al.] 2016) Sønderby et al.] 2016;; Gregor et al. 2016). In HVAEs, the latent variable is divided into K separate groups, Z (......, }. The approximate posterior and the prior distributions are then defined by q(z|x)= = =IIk= ((zx) Zck. x) and p(z) = II p(zz/Z<k). Using these, the training objective becomes:
K
LHVAF(x)= == Eyga [logp(x
(xxea///xx...x)(e)ex/ Ey(easll) (KL(g(z) Zick x)(lp(za [zx))],
(2)
where q(z<x|x) I q(z,) iec. x) is the approximate posterior up the (Is 1)th grout !
The Prior Hole Problem: Let q(z) A Exp(() [xx(x))(((()x) [g(z(x)] denote the aggregate (approximate) posterior. In Appendix B.1
we show that maximizing (x)()) [xva with respect to the prior parameters corresponds
to bringing the prior as close possible to the aggregate posterior by minimizing (L(((z)) Ip(z)) w.r.. p(z). Formally, the prior hole problem refers to the phenomenon that p(z) fails to match q(z).
For &= 1, the expectation inside the summation simplified to (L (x)]((()))
2

---

Under review conference paper at ICLR 2021
3 NOISE CONTRASTIVE PRIORS (NCPS)
One of the main causes of the prior hole problem is the limited expressivity of the prior that prevents it from matching the aggregate posterior. Recently, energy-based models have shown promising results in representing complex distributions. Motivated by their success, we introduce the noise contrastive prior (NCP) PNCP (z) = (z)),(z), where p(z) is base prior distribution, e-g. a Normal, r(z) is a reweighting
factor, and Z = r( r())p(z)de is the normalization constant. The function IR" - IR+ maps the latent variable z E R" to a positive scalar, and can be implemented using neural nets.
The reweighting factor r(z) can be trained using MCMC sampling as discussed Appendix However, MCMC requires expensive sampling iterations that scale poorly to hierarchical VAEs. To address this, we describe a noise contrastive estimation based approach to train PNCP(z) without MCMC sampling.
3.1 LEARNING THE REWEIGHTING FACTOR WITH NOISE CONTRASTIVE ESTIMATION
Recall that training VAEs closes the gap between the prior and the aggregate posterior by minimizing (L({(z))lp(z)) with respect to prior. Assuming the base prior p(z) to be fixed, (L(g(z)) is zero when r(z) = q(z)/pzz). However, since we do not have the density function for q(2), cannot compute the ratio explicitly. Instead, in this paper, we propose to estimate r(z) using noise contrastive estimation (Gutmann & Hyvärinen (2010), also known as the likelihood ratio trick that has been popularized in machine learning by predictive coding (Oord et al. 2018)) and generative adversarial networks (GANs) (Goodfellww et al. 2014). Since, we can generate samples from both p(z) and q(2) we train binary classifier to distin- guish samples from (z) and samples from the base prior p(z) by minimizing the binary cross-entropy loss:
min EZza(z) (log ((z)) - Re- (log(1 D(z))]-
Here, D:R" + 1) 1) is a binary classifier that generates the classification prediction probabilities. Eq. (3) is minimized when D(z) p(x)+p(z) Denoting the classifier at optimality by D (*), we estimate the
((a)
reweighting factor r(z) = piz) 1-Dー The appealing advantage of this estimator is that it is obtained
g(x) (x)
by simply training a binary classifier rather than using expensive MCMC sampling.
3.2 TWO-STAGE TRAINING FOR NOISE CONTRASTIVE PRIORS
To properly learn the reweighting factor, NCE training requires the base prior distribution to be close the target distribution. Intuitively, if p(z) is very close to ((z) (i.e., p(z) RJ q(z)), the optimal classifier will have a large loss value in Eq. 3 and we will have r(z) = will easily learn to distinguish samples from the two distributions and it will not learn the likelihood ratios correctly. If p(z) is roughly close to qz).. then the binary classifier can learn the ratios.
If p(z) is instead far from (z), the binary classifier
To ensure that the base prior distribution is close to the target aggregate posterior distribution, we propose a two-stage training algorithm. In the first stage, we train the VAE with only the base prior p(z). From Appendix B.1 we know that at the end of training, p(z) is as close as possible to g(z). In the second stage, we freeze the VAE model including the approximate posterior g(z) x), the base prior p(z). and the likelihood p(x|z), and we only train the reweighting factor r(z) using Eq. (3) The second stage can be thought of as replacing the base distribution p(z) with a more expressive distribution of the form PNcp(z) x r(z)p(z). Hence, NCP matches the prior to the aggregate posterior qzz) by using the reweighting factors. Note that our proposed method is generic as it only assumes that we can draw samples from g(z) and p(z), which applies any VAE. Our training is illustrated in Fig. 2
We generate samples from the aggregate posterior ((z)) Epetx) [x())x) via ancestral sampling: draw data from the training set (x (x)) and then sample from x2)x).
3

---

Under review conference paper at ICLR 2021
Stage (VAE Truining)
Stage 2 (NCE Training)
Test Time Sampling
p(s)
g(fle)
(i(a) r(s)
Pace (+) p(s)r(s)
Daa. (x(el)) [(a)
D(+)
x(-))
Decoder
Decocir
Binary Classiter
p(2(2)
~ +
~ p(a)
Pictt (x)
Recondtrucion Loss
Figure 2: NCP-VAE trained in two stages. In the first stage, we train a VAE using the original VAE objective. In the second stage, we train the reweighting factor r(z) using noise contrastive estimation (NCE). NCE trains a classifier to distinguish samples from the prior and samples from the aggregate posterior. Our noise contrastive prior (NCP) is then constructed by the product of the base prior and the reweighting factor, formed via the classifier. At test time, we sample from NCP using SIR or LD. These samples are then passed the decoder generate output samples.
3.3 TEST TIME SAMPLING
To sample from VAE with an NCP, we first generate samples from the NCP before passing them the decoder to generate output samples (shown in Fig. D We propose two methods for sampling from NCPs.
(SIR): We first generate M samples from the base prior distribution (z(m)) 1M p(z). We then resample one of the M proposed samples using importance weights propor- tional to (m)) PNCP (m(m)) //p(z((m()))))))) (m)(m)). The benefit of this technique: both proposal generation and the evaluation of r on the samples are done in parallel.
Langevin Dynamics (LD):: Since NCP is an EBM, we can L.D for sampling. Denoting the energy function by Bzz) = - (z) - log p(z), we initialize sample zo by drawing from p(z) and update the sample iteratively using: Z = - 0.5 E(z) + Vλc, where e NO 1) and À is the step size. LD is run for finite number of iterations, and in contrast to SIR, it can be slow given its sequential form.
3.4 GENERALIZATION TO HIERARCHICAL VAES
The state-of-the-arn VAE Vahdat & Kautz, 2020) uses hierarchical g(z[x) and p(z). Here p(z) is cho- sen to be a Gaussian distribution. Appendix B.2 shows that training a HVAE encourages the prior to minimize Eytr))) [KL(q (<<)]lp((z [=]))]] for each conditional, where q(z<k) & Eps(s)]((ZzH) [x] is the aggregate posterior up to the (k - 1)th group, and ((z]/Z<4) Epa(x) [g(zx (z.zz x)] is the ag- gregate conditional for the 1tth group. Given this observation, we extend NCPs hierarchical mod- els to match each conditional in the prior with g(zz/Z<k). Formally, we define hierarchical NCPs by PNcP(z) 1 II r(zx) Rzck )(z)) (zek) where each factor is an EBM. (NCP(Z) resembles energy machines 2019).
with an autoregressive structure among groups (Nash & Durkan
In the first stage, we train the HVAE with prior I p(zz k<.) For the second stage, we use K binary classifiers, each for hierarchical group. Following Appendix C train each classifier via:
min E Eps(x)a((aesxx)) E Eg(o//een llog Dz(ze, c(Zzx))) Ep(ry (ne)) [log(1 (1 Dx(zx, clz c(ax))))] (4) D2
4

---

Under review as conference paper at ICLR 2021
where the outer expectation samples from groups up to the (k- group, and the inner expectations sample from approximate posterior and base prior for the Athh group, conditioned on the same Zzck The discriminator D2 classifies samples 2y while conditioning its prediction on Zek using a shared context feature c(zck).
Denoting the The NCE training in Eq. (4) is minimized when Dz(Z c(zck)) = g(z. saatee ([(a))((ee))
classifier at optimality by (z, c(z<k)) we obtain the reweighting factor r r(zk (<ck) & T- [
in the second stage. Given our hierarchical NCP, we use ancestral sampling to sample from the prior. For sampling from each group,
use SIR or LD as discussed before.
The context feature c(z<x) extracts a representation from Zzk- Instead of learning a new representation at stage two, we simply use the representation that is extracted from Zeck in the hierarchical prior, trained in the first stage. Note that the binary classifiers in the second stage are trained in parallel for all groups.
4 RELATED WORK
In this section, we review prior works related to the proposed method.
Energy Gnergy-based -based Models (EBMs): Early work on EBMs for generative learning goes back to 1980s (Ackley jtt al. 1985 Hinton al. 1986). Prior to the modern deep learning era, most attempts for building genera- tive models using EBMs were centered around Boltzmann machines (Hinton 2002] Hinton et al.] (2066 and their "deep" extensions (Salakhutdinov & Hinton 2009 Larochelle & Bengio 2008). Although the energy function in these models is restricted to simple bilinear functions, they have been proven effective for repre- senting the prior in discrete VAEs (Rolfe 2016; Vahdat et al., 2018a b 2020). Recently, EBMs with neural energy functions have gained popularity for representing complex data distribution (Du & Mordatch] 2019). Pang et al. (2220) have shown that neural EBMs can represent expressive prior distributions. However, this case, the prior is trained using MCMC sampling, and it has been limited to single group of latent variables. To eliminate MCMC sampling, NCE (Gutmann & Hyvärinen] 2010 has recently been used for training normalizing flow on data distributions (Gao et al. (020)). Moreover, Han et al. (2019) (020) have used divergence triangulation to sidesteps MCMC sampling. In contrast, we use NCE to train an EBM prior where noise distribution is easily available through pre- pre-trained VAE.
Similar to NCE, generative adversarial networks (GANs) (Goodfellow et al. 2014) also rely on discriminator to learn the likelihood ratio between noise and real images. However, GANs use the discriminator to update the generator, whereas in NCE, the noise generator is fixed. In spirit similar are recent works (Azadi et al.) 2018 Turner et al. 2019 Che et al. 2020) that link GANs, defined in the pixels space, to EBMs. We apply the likelihood ratio trick to the latent space of VAES. The main difference: the base prior and approximate posterior are trained with the VAE objective rather than the adversarial loss.
Adversarial Training:
Adversarial loss has been used for training implicit encoders in VAEs (Makhzani et al.) 2015 Mescheder
ct al. 2017 Engel et al., 2018). But, they have not been linked to energy-based priors as we do explicitly.
Prior Hole Problem: Among prior works on this problem, VampPrior (Tomezak& & Welling 2018) uses a mixture of encoders to represent the prior. However, this requires storing training data or pseudo-data to generate samples at test time. Takahashh et al. (019) use the likelihood ratio estimator to train a simple prior distribution. However at test time, the aggregate posterior is used for sampling in the latent space. Bauer& & Mnih Mnih[(019)) (2019) propose reweighting factor similar to ours, but it is trained via importance sampling. Recently, Lawson et al. (2019) introduced energy-inspiee
models (EIMs) that define distributions induced by the sampling processes used by (Bauer & Mnih 2019) as well as our SIR sampling. Although, EIMs have the advantage of end-to-end training and can be used as either prior or decoder in VAEs, they require multiple samples during training (up to 1K). This can make application of EIMs deep hierarchical models such as NVAES very challenging as these models are often very memory intensive and are trained with a few training samples per GPU.
5

---

Under review conference paper at ICLR 2021
4
7
0
6
6
5
8
3
2
6
0
6
6
7
7
/
7
8
9 4 8 7
8
2
5
4
1
0
7
5
4
7
2
2
0
2
9 R
8
9
4
8
3
0
4
0
7
9
7
3
6
7
2
y
0
4
6
6
/
(a)
MNIST 1.0)
(b) CIFAR-10 (t 0.5)
(c) CelebA 64x64 0.7)
(d) CelebA HQ 256 256 0.7)
Figure Randomly sampled images from NCP- VAE with the temperature t for the prior.
Two-stage VAEs: VQ-VAE (Van Den Oord et al. 2017 Razavi et al. 2019) first trains an autoencoder and then fits an autoregressive PixeICNN (Van Den Oord et al. (016) prior to the latent variables which is slow
to sample from. Two-stage VAE (2s- VAE) (Dai & Wipi 2018) trains a VAE on the data, and then, trains another VAE in the latent space. Regularized autoencoders (RAE) (Ghosh et al. (2200) train autoencoder, and subsequently a Gaussian mixture model on latent codes. In contrast, we train the model with the original VAE objective in the first stage, and improve the expressivity of the prior using EBM.
5 EXPERIMENTS
Our implementation of NCP-VAE builds upon NVAE (Vahdat & Kautz 2020). the state-of-the-art hier- archical VAE. We examine NCP- on four datasets including dynamically binarized MNIST (LeCun) 1998), CIFAR-10 (Krizheskyy et al. 2009). CelebA-64 (Liu et al. (015) and CelebA-HQ-256 (Karras et al. 2017). For CIFAR-I0 and CelebA-64, the model has 30 groups, and for CelebA-HQ-22 it has groups. We sample from NCP-VAE using SIR with 5K proposal samples at the test (image generation) time. On these datasets, we measure the sample quality using the Fréchet Inception Distance (FID) score (Heusel al. 2017) with 50,000 samples, computing the oiglliiel requires estimating the intractable nor- malization constant. To report log-likelihood results, we train an NVAE model with a small latent space on MNIST with 10 groups of 4 latent variables. We estimate log normalization constant in NCPs using 1000 importance weighted samples. We intentionally limit the latent space to ensure that we can estimate the normalization constant correctly (standard deviation of log 7 estimation < 0.23). Thus, this dataset, we report negative log-likeliioow (NLL). Implementation details are provided in Appendix E
5.1 QUANTITATIVE RESULTS
The quantitative results reported in Tab. 1 Tab. Tab. and Tab. On all four datasets, model improves upon state-of-the-art NVAE, and it reduces the gap with GANs by a large margin. On CelebA 64, improve NVAE from FID of 13.48 5.25, comparable to GANs. On CIFAR-10, VAE
6

---

Under review conference paper at ICLR 2021
Table 1: Generative performance on CelebA-64
Table 2: Generative performance on CIFAR-10
Model
FID4
Model
FID
NCP-VVE (ours)
5.25
NCP-VAE (ours)
24.08
NVAE Vahddtt & Kautz
2020)
13.48
NVAE (Vahdat & Kautz
2020
51.71
RAE (Ghosh et al 2020
40.95
RAE (Ghosh et al. 20200
74.16
2s-VAE Dai & Wipi
2018
44.4
2s- 2s-VAE (Dai & Wipt 201
72.9
WAE Tolstikhın et 2018
35
Perceptial AE (Zhang ct
2020
51.51
Percephaal Ala/hang
220
13.8
EBM (Du Mordatch 2019
40.58
Latent EBM (Pang et al.
2020)
37.87
Latent EBM (Pang et al.
2020
70.15
COCO-GAN (Lin ct al.
2019)
4.0
Style-GANv2 (Karras et al.
2020)
3.26
QA-GAN (Parimala & Channappayya 2019)
6.42
Denoising Diflusion Process HH et al.
2020
3.17
NVAE -eccnneeon (ahddtt Kautz 2020)
1.03
NVAE Recon (Vahdat Kautz
2020)
2.67
Table 3: Generative results on CelebA-HQ-256
Table 4: Likelihood results on MNIST in nats
Model
FID
Model
NLL
NCP-VAL PAP (ours)
24.79
NCP- VAE (ours)
78.10
NVAE Vahddat & Kautz 2020
40.26
NVAE-small (Vahdat Kautz 2020)
78.67
GLOW (Kingma & Dhariwal 2018)
68.93
BIVA (Maaloe et al. 2019)
78.41
Advers. LAE (Pidhorskyl et al.
2020)
19.21
DAVE+ Vahdat
2018b
78.49
PGGAN (Karaas ct al. 2011
8.03
IAF-VAE (Kingma el all
2016
79.10
NVAE- Vahddtt & Kautz 2020
0.45
VampPrior AR dec.
Tomezak & Welling
78.45
improves the NVAE FID of 51.71 to 24.08. On MNIST, although our latent space much smaller, our model outperforms previous VAEs. NVAE has reported 78.01 nats on this dataset with larger latent space.
Reconstruction: the last row of Tab. Tab. 2 and Tab. 3 we report the FID score for the reconstructed images of the NVAE baseline. Note how reconstruction FID is much lower than our FID, indicating that model is far from memorizing the training data. In Appendix we also provide nearest neighbours from training data for generated samples. In Appendix present NCP applied to vanilla VAEs.
QUALITATIVE RESULTS
We visualize samples generated by NCP-VAE in Fig. 3 without any manual intervention. We adopt the common practice of reducing the temperature of the base prior p(z) by scaling down the standard-deviate the conditional Normal distributions (Kingma & Dhariwal 2018 Brock et al. (2018); Vahdat & Kautz (2020) also observe that re-adjusting the nonelioormallzationnn (BN), given temperature applied to the prior, improves the generative quality, Similarly, we achieve diverse, high-quality images by re-adjusting the BN statistics described by Vahdat & Kautz (2020). Additional qualitative results are shown in Appendix H
5.3 ADDITIONAL. STUDIES
We perform additional experiments to study i) how hierarchical NCPs perform as the number of latent groups increases, ii) the impact of SIR and LD hyperparameters, and iii) what the classification loss in NCE training conveys about p(z) and q(z). All experiments this section are performed on CelebA-66 data.
'owering of the temperature is only used to obtain qualitative samples. It's not used when computing any of the quantitative results in Sec. [.
7

---

Under review conference paper at ICLR 2021
Table 6: Effect of SIR sample size and LD iterations. Time-N is the time used generate batch of N images.
SIR proposal samples
Time-1 (sec)
Time-10 10 (sec)
Memory (GB)
# LD iterations
Time-1 (scc)
Time- 10 (sec)
Memory (GB)
FID
FID
11.75
0.34
0.42
1.96
14.44
3.08
3.07
1.94
50
8.58
0.40
1.21
4.30
50
12.76
27.85
28.55
1,94
500
6.76
1,25
9.43
20.53
500
8.12
276.13
260.35
1.94
5000
5.25
10.11
95.67
23.43
1000
6.98
552
561.44
1.94
Binary Classifler Loss for ColobA 64
Effective Sample Size vs. Loss
Effective Sample Size
1,0
0.84
0.2
0.0
20
40
60
80
100
0.0
0.1
0.2
0.3
2.4
0.5
2.6 D.7
Training Epoch (a)
Binary Classifier Loss
(b)
Figure 4: (a) Classification loss for binary classifiers on latent variable groups. A larger final loss upon training indicates that (z) and p(z) are more similar. (b) The effective sample size vs. the final loss value at the end of training. Higher effective sample size implies similarity of two distributions.
Table # ggroup & gener- ative performance in FID..
Number of latent variable groups: Tab. shows the generative performance of hierarchical NCP with different amounts of latent variable groups. As we increase the number of groups, the FID score of both NVAE and our model improves. This demonstrates the efficacy of our NCPs, even with expressive hierarchical priors and in the presence of many groups.
# groups
NVAE NCP-VAE
33.18
18.68
15
14.96
5.96
30
13.48
5.25
SIR and LD parameters: The computationally complexity of SIR similar
to LD if we set the number of proposal samples in SIR equal to the number LD iterations. Tab. 0 reports the impact these parameters. We observe that increasing both the number of proposal samples in SIR and the LD iterations leads to noticeable improvement in FID score. For SIR, the proposal generation and the evaluation of r(z) are parallelizable. Hence, as shown in Tab. 6 image generation faster with SIR than with LD (LD iterations proposals, but not with the number of LD iterations. Interestingly, SIR, albeit simple, performs better than LD when using about the same compute.
sequential). However, GPU memory usage scales with the number of SIR
Classification loss in NCE: We can draw a direct connection between the classification loss in Eq. and the similarity of p(z) and (z). Denoting the classification loss in Eq. optimality by L* Goodfellow al. (2014) show that (SD(p(z) (()()))(((() between distributions. Fig. 4 a) plots the classification loss (Bq. 4 for each classifier for a 15-group NCP trained on the CelebA-64 dataset. Assume that the classifier loss at the end of training is a good approximation of L. We observe that 8 out of 15 groups have L 2 0.4. indicating a good overlap between p(z) and q(z) for those groups To further assess the impact of the distribution match on SIR sampling, in Fig. (b). we visualize the effective sample size (ESS SIR vs. L for the same group. We observe a strong correlation between L and the effective sample size. SIR is more reliable on the same 8 groups that have high classification loss. These groups are mostly at the top of the NVAE hierarchy which have been shown to control the global structure of generated samples (see B.6 in Vahdat & Kautz (2020)).
log 2 - 0.5 x L where JSD denotes the Jensen-Shannon divergence
*ESS measures reliability of SIR via 1/ Em (m))))) where ((m)) r(a(m))/ Eme Em(r(a(m)) Owen 2013
8

---

Under review conference paper at ICLR 2021
5.4 ADDITIONAL COMPARISON TO PRIOR ART
In our previous experiments, we rely on NVAE as the VAE backbone. Although this shows that our approach can be applied to large scale models successfully, this may make comparison against the prior art unfair as they are often applied to smaller models. To provide fair comparison, in this section, we apply NCP to several commonly used small VAE models.
Comparison against RAE, 2s- VA,, and WAE: In Tab. 7 we show the generative performance of our approach applied to the VAE architecture in RAE (Ghosh et al. 2020). Note that this VAE architecture has only one latent variable group. The same base architecture & Wipt (018) and WAE (Tolstikhin et al.] 2018). We borrow the training setup from Ghosh et al. (2020) the CelebA-64 dataset, and compare the baselines reported in this work. We apply our NCP-VAE on top of vanilla VAE with Gaussian prior as well as a 10-component Gaussian mixture model (GMM) prior that was proposed in RAEs. Our NCP-VAE improves the performance on the base VAE, improving the FID score to 41.28 from 48. 12. Additionally, when NCP is applied to the VAE with GMM prior (the RAE model), it improves its performance from 40.95 the FID score of 39.00.
used in the implementation of 2s- VAE (Dai
Comparison against LARS and SNIS: In Tab. 8 we compare NCP- VAE to LARS (Bauer & Mnih 2019) and SNIS (Lawson et al. 2019) priors on MNIST. We implement our NCP- VAE using the VAE and energy- function architectures that were used by] Lawson et al. (2019). We closely follow the training hyperparame- ters used by [awson et al. (2019) as well as their approach for obtaining a lower bound on the log likelihood. NCP-VAE obtains the negative log-likelihood (NLL) of 82.82, comparable to Lawson et al. (019)), while outperforming LARS (Bauer & Mnih 2019).
Table 7: Generative performance on CelebA-64 with the RAE (Ghosh et al.) 2020) architecture
Likelihood results on MNIST on sin- gle latent group model with architecture from LARS (Bauer & Mnih (219)) & SNIS (Lawson ct al. 2019) (results in nats)
Table 8:
Model
FID
VAE w/ Gaussian prior
48.1 12
2s- VAE DDai & Wipf 2018)
49.70
Model
NLL
WAE Tollstihhnn el al. 2018
42.73
VAE w/ Gaussian prior
84.82
RAE (Gosh el al. 2020)
40.95
VAE w/ LARS prior (Bauer & Mnih 2019)
83.03
NCP w/ Gaussian prior base
41.28
VAE w/ SNIS prior (Lawson 2019
82.52
NCP w/ GMM prior as base
39.00
NCP-VAE
82.82
6 CONCLUSIONS
The prior hole problem is one of the main reasons for VAEs' poor generative quality. In this paper, we tackled this problem by introducing the noise contrastive prior (NCP), defined by the product of reweighting factor and a base prior. We showed how the reweighting factor is trained by contrasting samples from the aggregate posterior with samples from the base prior. Our proposal is simple and can be applied to any VAE to increase its prior's expressivity. We also showed how NCP training scales to large hierarchical VAEs, as it can be done in parallel simultaneously for all the groups. Finally, we demonstrated that NCPs improve the generative performance of state-of-the-arl NVAEs by a large margin, closing the gap to GANs.
REFERENCES
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for Boltzmann ma- chines. Cognitive science, 91):1477169, 1985.
Jyoti Aneja, Aditya Deshpande, and Alexander G Schwing. Convolutional image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5561-5570, 2018.
9

---

Under review conference paper at ICLR 2021
Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. Sequential latent spaces for modeling the intention during diverse image captioning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4261 2661-477, 2019.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discriminator rejection sampling. In International Conference on Learning Representations, 2018.
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders, In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 66-75. PMLR, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint
2018.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.06060, 2020.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv: 1.02331, 2016.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. In International Conference Learning Representations, 2018.
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander Schwing, and David Forsyth. Fast, diverse and accurate image captioning guided by part-of-speech. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 10695-10704, 2019.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Juke- box: A generative model for music. arXiv preprint arXi::2005.00:: 000401 2020.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, pp. 3608-3618, 2019.
Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models. In International Conference on Learning Representations, 2018.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som-vae: Interpretable discrete representation learning on time series. In International Conference on Learning Representations, 2018 8.
Ruiqi Gao, Erik Nijkamp. Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow con- trastive estimation of based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7518-7528, 2020.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From varia- tional to deterministic autoencoders. In International Conference on Learning Representations, 2020.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. 1rXiv:1406.2661, June 2014.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards con- ceptual compression. In Advances In Neural Information Processing Systems, pp. 3549-3557, 2016.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. PixeIVAE: A latent variable model for natural images. arXiv preprint 1rXiv:16/1.05013 2016.
10

---

Under review conference paper at ICLR 2021
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for un- normalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu, Divergence triangle for joint training of generator model, energy- model, and inferential model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8670-8679, 2019.
Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7978-7987, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, pp. 6626-6637, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Geoffrey E Hinton, Terrence J Sejnowski, et al. Learning and relearning in boltzmann machines. Parallel distributed processing: Explorations the microstructure of cognition, 1(282-317):2 1986.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets.
Neural computation, 18(7):15271154, 2006.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint
arxiv:2006. / 1239,, 2020.
Matthew D Hoffman and Matthew Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances Approximate Bayesian Inference, NIPS, 2016.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-eeciiaatoo networks. Proceedings of the IEEE conference on computer vision and pailern recognition, pp. 7132-7141, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 4448-466, 2015.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. In ICML, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10190 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp- 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training gener- ative adversarial networks with limited data. arXiv preprint arXiv:2006.0660 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.0 6980, 2014.
arXiv preprint
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In The International Conference on Learning Representations (ICLR), 2014.
11

---

Under review conference paper at ICLR 2021
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learn- ing with deep generative models. In Advances in Neural Information Processing Systems, pp. 3581 3581-3589, 2014.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp. 4743-4751 2016.
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10236-10245, 2018.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchi- cal priors vaes. Advances in Neural Information Processing Systems, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Hugo Larochelle and Yoshua Bengio. Classification using discriminative restricted boltzmann machines. In Proceedings of the 25th international conference on Machine learning, pp. 536-543, 2008
John Lawson, George Tucker, Bo Dai, and Rajesh Ranganath. Energy-inspired models: Learning with sampler-induced distributions. In Advances in Neural Information Processing Systems, pp. 8501-8513,
2019.
Yann LeCun. The mnist database handwritten digits. http://yann. lecun. com/exdb/mnistl, 1998.
Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong Chen. Coco-gan: generation by parts via conditional coordinating. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4512-4521, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608.:160..03990 2016.
Lars Maalge, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: very deep hierarchy of latent variables for generative modeling. In Advances in neural information processing systems, pp. 6548-6558, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoen- coders. arXiv preprint arXiv:1511.05644 2015.
Lars Mescheder, Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational toencoders and generative adversarial networks. In 34th International Conference on Machine Learning (ICML), pp. 2391-2400. PMLR, 2017.
Charlie Nash and Conor Durkan. Autoregressive energy machines. arXiv preprint arXiv:1904.05622 2019.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 1rXiv:1609.03499 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint ariiv: 2018.
12

---

Waveflow: A compact flow-based model for raw Under review conference paper at ICLR 2021
Art B. Owen, Monte Carlo theory, methods and examples. 2013.
Bo Pang, Tian Han, Erik Nijkamp, Goog-Chun Zhu. and Ying Nian Wu. Learning latent space energy based prior model. arXiv preprint arXiv:2006.082005 08820, 2020.
Kancharla Parimala and Sumohana Channappayya. Quality aware generative adversarial networks. In Ad- vances in Neural Information Processing Systems, pp. 2948-2958, 2019.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14104-14113, 2020.
Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. audio. ICML., 2020.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.055941.000 2017.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14837-14847, 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.0059)) 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In International Conference on Machine Learning, pp. 1278-
1286, 2014.
Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint arXiv:1609002200 2016.
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in variational infer- ence. arXiv preprint arXiv:1802.06844 2018.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificiaa intelligence and statis- fics, pp. 4558, 2009.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder varia- tional autoencoders. In Advances in neural information processing systems, pp. 3738-3746, 2016.
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Variational autoencoder with implicit optimal volume 33, pp. 5066-5072 2019.
optimal priors. In Proceedings of the AAAI Conference on Artificial Intelligence.
Tollstikhin, Bousquet, S Gelly, and Schölkopf. Wasserstein auto-encoders. In International Conference on Learning Representations (ICLR 2018). OpenReview. net, 2018
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelli- gence and Statistics, pp. 1214-1223, 2018
Ryan Turner, Jane Hung, Eric Frank, Yunus Saatchi, and Jason Yosinski. Metropolis-hastings generative adversarial networks. In International Conference Machine Learning, pp. 6345-6353. PMLR, 2019.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
Arash Vahdat, Evgeny Andriyash, and William G Macready. DVAE#: Discrete variational autoencoders with relaxed Boltzmann priors. In Neural Information Processing Systems, 2018a.
13

---

Under review conference paper at ICLR 2021
Arash Vahdat, William G. Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny Andriyash. DVAE++: Discrete variational autoencoders with overlapping transformations. In International Conference on Ma- chine Learning (ICML), 2018b.
Arash Vahdat, Evgeny Andriyash, and William Macready. Undirected graphical models as approximate posteriors. In International Conference Machine Learning (ICML), 2020.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, pp- 1747-1756. JMLR. org, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances Neural Information Processing Systems, pp- 6306-6315, 2017.
Haowen Xu, Wenxiao Chen, Jinlin Lai, Zhihan Li, Youjian Zhao, and Dan Pei. On the necessity and 2019.
effectiveness of learning the prior of variational auto-encoder. arXiv preprint
Zijun Zhang, Ruixiang Zhang, Zongpeng Li, Yoshua Bengio, and Liam Paull, Perceptual generative autoen- coders. In International Conference on Machine Learning, 2020.
14

---

Under review conference paper at ICLR 2021
A TRAINING ENERGY-BASEN PRIORS USING MCMC
In this section, we show how a VAE with energy-based model in its prior can be trained. Assuming that the prior the form (2) z zr (2)) the variational bound is of the form:
E (x)( [vVAE] - Exx(xx (p(txx))oogg/ ( - (L( (z))]
(x)))) [ (logp(x k)) log ((x|x) + log r(z) + log log Z,
where the expectation term, similar to VAEs, can be trained using the reparameterization trick. The only problematic term is the log-normalization constant log Z, which captures the gradient with respect the parameters the prior (z). Denoting these parameters by 8, the gradient of log Z is obtained by:
00 log Z 2 ((((z)((z)) dz
p(z) d (og(r(z)p(z)) dz ErgaM (x)
@log(rZZ)p(z), (5)
00
Z
00
00
where the expectation can be estimated using MCMC sampling from the EBM prior.
B MAXIMIZING THE VARIATIONAL BOUND FROM THE PRIOR'S PERSPECTIVE
In this section, we discuss how maximizing the variational bound in VAEs from the prior's perspective corresponds to minimizing a KL divergence from the aggregate posterior to the prior. Note that this relation has been explored by Hoffman & Johnson (2016); Rezende & Viola (2018); Tomezak & Welling (2018) and we include it here for completeness.
B. VAE WITH A SINGLE GROUP OF LATENT VARIABLES
Denote the aggregate (approximate) posterior by ((z) Exa(x) [[(z)x)]. Here, we show that maximizing the (SvAee)) with respect to the prior parameters corresponds to learning the prior by minimizing (L(g(z)(l((z)) To see this, note that the prior pız) only participates in the KL term in LVae (Eq. D We
hence have:
arg Pxp( (VVAEX)] arg min E pa (KI.(g(2)))llx(x)) p(x)
v(*)
arg min E Exa ([((x)x))
p(z)
arg min -H((z)) Eg (ogg(()) p(*)
arg min (L(((2)]|p(()))
p(z)
where H.) denotes the entropy. Above, replaced the expected entropy ([9(2x) with H (g(())
the minimization with respect to the parameters of the prior p(z).
B.2 HIERARCHICAL VAES
Denote hierarchical approximate posterior and prior distributions by: g(z(x) I g(zz/zzk and p(z) IT p(zz (zc). The hierarchical VAE objective becomes:
K
CHVAE(X) = (lgg
By(neelse) [KL(g(zy Zzk: (z) [zx))]
(6)
15

---

Under review conference paper at ICLR 2021
where g(z<<|>)) II= g(zi|Zer, x) the approximate posterior up the (k - 1)th group. Denote the
aggregate posterior up to the (K 1hth group by ((<ck) (pa(x)]9(Z<<> x)] and the aggregate conditional
for the hthh group given the previous groups ((zx) Zec) E (xa(x) (g Zeck x)]-
Here, we show that maximizing (x) (CIVAE (x) with respect to the prior corresponds to learning the prior by minimizing By(zer) (<< |p( (z) (<ck) )) for each conditional:
arg max (x)) [x)] = arg min Bp((x) [Egtace (x) [KL [gL Zz p(a, (x.ce)
p(a, mcx)
arg min wwwp:x/llaaaaalllllll.........
p(a, mex)
arg min - -E ((n, plog p(xx zcx))
p(a, mex)
arg min - -gllac)) [Bglzr log ((zz)lze))
p(x, m(x)
arg min By(mee))
H [-H( lalaalaee [logp(z)/z<a)
p(a, (ic)
arg min Bo(nes) (KL(g(Z(ZZ<)) (p(Z+/=<=)) pla, mec))
C CONDITIONAL NCE FOR HIERARCHICAL VAES
In this section, we describe how we derive the NCE training objective for hierarchical VAEs given in Eq. (4). Our goal is learn the likelihood ratio between the aggregate conditional g(zz ((ck) and the prior p(Zz/Z<<k) We can define the NCE objective to train the discriminator D& (z Zick) that classifies Z given samples from the previous groups Z<k using:
min ( [log (zE z<x)] Epizs [log(1 (2(,,, VZck-
(8)
Since Zeck is in high dimensional space, we cannot apply the minimization VyŞk- Instead, we sample from Zek using the aggregate approximate posterior g(z<k) as done for the KI. in hierarchical model (Eq. (7):
min Ey Ey(ner)
Eyta, (m [log (2x Z k<k)] - Epexs [log(1 Di z<x)]]
(9)
D2
Since q( ( xx) = q(z,,, Zec) = (x) (2 (x 2 Zzck x)],
we have:
(xu [log D& (Zk, [])] Fp - Dy(ZE, zcx)]]
min E [p((x)g(xcx)x)) Dr
(10)
Finally, instead of passing all the samples from the previous latent variables groups D, we can pass the context feature c(zz<) that extracts a representation from all the previous groups:
min E (pp(x))(tccsx))) Eegnsolace bx)lll/eexxx))leoo [log DE(zk; c(Zcc))) - -Epres/zea))loge Du (((ex))( (11)
D₁
D NVAE BASED MODEL AND CONTEXT FEATURE
The base model NVAE (Vahdat & Kautz 2020) is hicrarchical. To encode the information from the lower levels of the hierarchy to the higher levels, during training of the binary classifiers, concatenate the context feature c(zck) ) to the samples from both p(z) and q(2). The context feature for each
Context Feature:
group is the output of the residual cell of the top-down model and encodes representation from Zick-
16

---

Under review conference paper at ICLR 2021
2C
Squeene Excitation
Squazie Eactorton
Cons 5x3 (sl. pl)
Conv 3x3 (s1pp)
Factorized Reductio (charne concreaatoo)
Ranch-Noms
Batch
2C
(4.
3x3 pl)
(+2, p!)
Swish
Batch Norm Swish
Batch-Norm Swish
Residual Block.A
Residual Block-B
Figure 5: Residual blocks used in the binary classifier. We s, P and C to refer to the stride parameter, the padding parameter and the number of channels the feature map, respectively.
Image Decoder p(x(z): The base NVAE Vahdat & Kautz 2020) uses a mixture of discretized logistic dis- tributions for all the datasets but MNIST, for which it uses Bernoulli distribution. In our model, we observe that replacing this with a Normal distribution for the RGB image datasets leads to significant improvements in the base model performance. This is also reflected in the gains of our approach.
IMPLEMENTATION DETAILS
The binary classifier is composed of two types residual blocks as Fig. The residual blocks use batch- normalization (loffe & Szegedy 2015). the Swish activation function (Ramachandran et al., 2017). and the guneezee-andEExcitain
(SE) block (Hu et al. 2018). SE performs squeeze operation (e.g. mean) to obtain a single value for each channel. An excitation operation linn-innerr transformation) is applied values to get per-channel weights. The Aesidual-Block. differs from Residual- -lockk-A in that it doubles the number of channels (C 2C), while down-sampling the other spatial dimensions. It therefore also includes a factorized reduction with x convolutions along the skip-connection. The complete architecture of the classifier is:
these
PERFORMANCE ON SINGLE GROUP VAE
To demonstrate the efficacy of our approach on any off-the -fef VAE, we, apply our NCE based approach to the VAE in (Gosh et al. 2020). Note we use the vanilla VAE with Normal prior provided by the authors. The FID for CelebA 39.1 19. Single group VAFs are known to perform poorly on the task of image geeration, which is reflected in the high FID value reconstruction.
improves from 48.1 41.28. Note that the FID for reconstruction reported as
G NEAREST NEIGHBORS FROM THE TRAINING DATASET
To highlight that hierarchical NCP generates unseen samples at test time rather than memorizing the training dataset, Figures 6 visualize samples from the model along with few training images that are most similar
17

---

Under review conference paper at ICLR 2021
Conv (s1, pl)
ReLU
Residual-Block-A
Residual-Block-A
Residual-Block-A
Residual-Block-B
Residual-Block
Briidual-Block-.
Residual-Block-A
Residual-Block-B
2D average pooling
Linear Sigmoid
Optimizer Learning Rate Batch size
Adam Kingma Ba 2014)
Initialize at 1e-3, ConaleAnnealingg (Loshchllov & Hutter 2016) to le-7
512 (MNIST, CIFAR-1O), 256 (C))ebA-64), 128 (CelehA HQ 256
Table 9: Hyper-parameterr for training the binary classifiers.
to them (nearest neighbors). To get the similarity score for pair of images, downsample to 64 64, center crop to 40 x 40 and compute Euclidean distance. The KD-tree algorithm used to fetch the nearest neighbors. We note that the generated samples are quite distinct from the training images.
18

---

Under review conference paper at ICLR 2021
Query Image
Nearest neighbors from the training dataset
2
25
Figure 6: Query images (left) and their nearest neighbors from the CelebA-HQ-2551 training dataset.
19

---

Under review conference paper at ICLR 2021
Query Image
Nearest neighbors from the training dataset
IND
OE
Figure 7: Query images (left) and their nearest neighbors from the CelebA-HQ-24 training dataset.
20

---

Under review as conference paper at ICLR 2021
H ADDITIONAL QUALITATIVE EXAMPLES
Figure 8: Additional samples from CelebA-64 at t = 0.7.
21

---

Under review conference paper at ICLR 2021
20
25
F
Figure 9: Additional samples from CelebA-HQ-256 at L = 0.7.
22

---

Under review conference paper at ICLR 2021
Figure 10: Selected good quality samples from CelebA-HQ-256.
23

---

Under review conference paper at ICLR 2021
I EXPERIMENT ON SYNTHETIC DATA
In Fig. I we demonstrate the efficacy of our approach the 25- S3-Gasssians dataset, that is generated by mixture 25 two-dimensional Gaussian distributions that are arranged on grid. The encoder and decoder of the VAE have 4 fully connected layers with 256 hidden units, with 20 dimensional latent variables. The discriminator has 4 fully connected layers with 256 hidden units. Note that the samples decoded from prior p(z) Fig. (T{b)) without the NCP approach generates many points from the the low density regions in the data distribution. These are removed by using our NCP approach (Fig. IT()).
Trx Didroadee Sanoko
frem
Roweishe Sende
(a) Samples from the true distribution
Samples from VAE
(c) Samples from NCP-VAE
Figure 11: Qualitative results on mixture of 25-Gaussians.
We 50k samples from the true distribution estimate the log-likelihood. Our NCP approach obtains an average log-likelihood of -0.954 nats compared to the log-likelihood obtained by sampling from the Gaussian prior, -.7553 nats. We 20k Monte Carlo samples estimate the log partition function for the calculation of log-likelihodd
24

---

Under review conference paper at ICLR 2021
J ADDITIONAL QUALITATIVE EXAMPLES
In Fig. 12 we show additional examples of images generated by NVAE (Vahdat & Kautz 2020) and our
NCP-VAT. We use temperature=0.7 for both. Visually corrupt images are highlighted with red square.
Random Samples from NVAE at 0.7
2
Random Samples from NCP-VAE at f = 0.7
Figure 12: Additional samples from CelebA-64 at 0.7.
25


Reviews:
[{"review_id": "jSQ_GJeMTur", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main concerns are,\n* The idea is somewhat novel but very similar to ideas developed in  [1,2] and (Bauer & Mnih, 2019). Note that, in [1] rejection sampling is performed on samples from the prior distribution based on the likelihood ratio obtained from a discriminator. While the results reported in [1] are not competitive to those reported in this work, it is probably be due to differences in model architecture. Therefore, a fair comparison to prior works [1,2] and (Bauer & Mnih, 2019) is required using a common model architecture. E.g. experiments using a simple DCGAN style architecture on CIFAR-10  would be sufficient to indicate efficacy.\n\n* Prior resampling vs more complex priors: While the proposed NCP sampling scheme is applied on a Gaussian base prior with the NVAE architecture, it is unclear whether a similar gain in performance can be obtained using a more complex prior e.g. flow [3] or autoregressive (Van Den Oord et al., 2017; Razavi et al., 2019), [4]. A simple experiment which could help here is: keeping the architecture constant compare NCP with Gaussian base vs a flow or autoregressive prior.\n\n* Additionally, no comparison to state of the art VAE based image synthesis approaches like VQ-VAE-2 (Razavi et al., 2019) are provided e.g. on ImageNet 256 × 256.  \n\n* No comparison to VAE-GAN based approaches [5,6], which achieves FIDs of  23.4 (vs 24.08 of this paper). It it also unclear whether the proposed approach can be applied on top of VAE-GAN based approaches.\n\n* The current work uses the proposed NCP sampling scheme on a Gaussian base prior. Can the proposed prior be applied to more complex priors e.g. flow or autoregressive priors?\n\n* No qualitative comparison of sample quality: Figure 3 only shows samples for the NCP approach. The paper should qualitative demonstrate that NCP leads to reduction in the number of \"corrupted\" images. To do this one possibility would be to demonstrate that the worst sample image in a batch from the NCP approach is on average significantly better than that from the plain NVAE approach. Alternatively, an even simpler experiment can be performed on the 2d Ring and Grid [7] datasets to demonstrate gains in sample quality.\n\n[1] Dual Rejection Sampling for Wasserstein Auto-Encoders, ECAI 2020.\n\n[2] Variational Rejection Sampling, AISTATS 2018.\n\n[3] Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings, ICLR 2020.\n\n[4] Normalizing Flows with Multi-scale Autoregressive Priors, CVPR 2020.\n\n[5] Variational Approaches for Auto-Encoding Generative Adversarial Networks, arXiv 2017.\n\n[6] \"Best-of-Many-Samples\" Distribution Matching, NeurIPS Workshop 2019.\n\n[7] GDPP: Learning Diverse Generations using Determinantal Point Process, ICML 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YqFM6Qn5Ok", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors highlight an important problem in VAE - the prior-hole problem - which is that the approximate posterior and the simple gaussian prior do not match in spite of the KL term in the ELBO which makes sampling an issue - leading to the prior putting probability mass on latents that are not decoded to high probability mass regions in data manifold. Prior approaches have overcome this problem by increasing the expressivity of the prior through autoregressive models, and/or using hierarchical latents, EBMs with MCMC sampling. This paper proposes a very simple two stage method - (1) train a regular VAE, (2) train a binary classifier in NCE style to distinguish samples from prior and approx. posterior; use the re-weighting term from the NCE score to sample from a better re-weighted prior - either through langevin dynamics or re-sampling. The authors combine this approach with the use of hierarchical latents and produce really good performing generative models on a host of benchmarks with good looking samples.\n\nPros:\n\nvery simple and neat approach that produces really good results.\nlooks easy enough to reimplement and adopt widely for future research in VAEs.\nsamples look great, FID scores are good.\nablations on LD and SIR are very useful.\nCons:\n\nwould have been nice not to have a 2-stage approach. figuring out a way to have an online way of trying to train the re-weighting classifier in a GAN-like manner would be much more preferred.\npaper can do a better job at citing related work - ex - should cite Variational Lossy Autoencoder - Chen et al 2016 - on using autoregressive priors and pointing out a connection to IAF posterior.\nwould be nice to show results even w/o hierarchical latents - how much improvement can be obtained over vanilla deep VAE with this two-stage approach\nreport results on log-likelihood benchmarks - CIFAR10, ImageNet-32, ImageNet-64 - compare to Flow models, Autoregressive Models, etc.\n\nPost Rebuttal:\nI believe the positioning of the paper could be improved but at the same time empirical results in the paper are strong. I am adjusting my score to 6 with a confidence of 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XwhaqeYiPAx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**GENERAL**\nThe goal of the paper is to model the marginal over latents in VAEs in such a way to minimize the mismatch with the aggregated posterior. The paper proposes a new class of marginal distributions over the latent space that is a product of two experts: the first expert is a non-trainable probability distribution, and the second expert is an unnormalized probability distribution parameterized using neural networks. Since training a product of experts requires to apply an approximate inference (e.g., MCMC sampling), the authors propose to use the likelihood ratio trick. Eventually, a VAE is trained in two stages. First, they assume the marginal over z's to be simply the non-trainable distribution, and the VAE is trained. At the second stage, they propose to train the second expert (i.e., the binary classifier that distinguishes z ~ q(z) and z ~ p(z)) in order to obtain the final NCP that better matches the aggregated posterior. Further, the idea is extended to hierarchical VAEs, and a separate binary classifier is trained per each stochastic level.\n\n**Strengths:**\nS1: The idea is very interesting and allows to enrich the marginal distribution over z's in the VAE framework.\n\nS2: The paper is well positioned in current trends in generative modeling. I find the combination of energy-based models and VAEs as a very appealing research direction.\n\nS3: The proposed two-stage learning procedure is very logical and seem to be efficient. Its simplicity increases its reproducibility.\n\nS4: Obviously, introducing an energy-based component results in intractability of the likelihood function due to the partition function. However, the FID scores and the quality of generated images are very convincing.\n\n**Deficiencies:**\nD1: In this review, I kept using \"a non-trainable expert\" or \"a non-trainable marginal\" for the base prior (the term used by the authors). However, I am not quite sure whether the base prior is non-trainable. I was unable to find any information about it in the paper. Therefore, I assumed it is a standard Gaussian after inspecting Figure 1. It would be easier for a reader, if this information was included in the text.\n\n\n**Remarks:**\nR1: The Appendix B.1 is closely related to the following papers:\n- Rezende, D. J., & Viola, F. (2018). Taming vaes. arXiv preprint arXiv:1810.00597.\n- Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016.\n- Jakub Tomczak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelli- gence and Statistics, pp. 1214–1223, 2018.\nIt is maybe worth to mention these papers there. Otherwise, the text may sound as a new contribution.\n\nR2: Section 3, first paragraph: The authors stated that: \"Recently, energy-based models have shown promising results in representing complex distributions\". This statement is very misleading, because the energy-based models (e.g., Boltzmann machines) have been used in ML for over 30 years, e.g.:\n- Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive science, 9(1), 147-169.\n- Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. Parallel distributed processing: Explorations in the microstructure of cognition, 1(282-317), 2.\n- Salakhutdinov, R., & Hinton, G. (2009, April). Deep boltzmann machines. In Artificial intelligence and statistics (pp. 448-455).\n- Larochelle, H., & Bengio, Y. (2008, July). Classification using discriminative restricted Boltzmann machines. In Proceedings of the 25th international conference on Machine learning (pp. 536-543).\n\nThe word \"recently\" suggests that this is a new invention that is simply not true.\n\nR3: The proposed prior (NCP) could be seen as a specific form of a product of experts (e.g., Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural computation, 14(8), 177). In my opinion, it is an interesting connection.\n\nR4: I wonder whether it is feasible to use some sort of importance sampling (e.g., Annealed Importance Sampling, Salakhutdinov, R., & Murray, I. (2008, July). On the quantitative analysis of deep belief networks. In Proceedings of the 25th international conference on Machine learning (pp. 872-879).) or other procedure (e.g., Perturb-and-MAP, Hazan, T., & Jaakkola, T. (2012, June). On the partition function and random maximum a-posteriori perturbations. In Proceedings of the 29th International Conference on International Conference on Machine Learning (pp. 1667-1674).) to estimate the partition function.\n\n**AFTER REBUTTAL**\nI would like to thank the authors for their rebuttal and all updates. The paper is related to other ideas in the literature, however, it constitutes an interesting contribution to the field. I appreciate all new results and discussions. I keep my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "9c6qSywGeH6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors approach the \"hole problem\" of variational autoencoders where the aggregate posterior fails to match the prior, causing some areas of the prior distribution to be left out. Consequently, the decoder is not trained properly to operate in such regions, and the whole generate models is then subject to suboptimal performance. To attack this problem authors introduce two changes:\n- Once a standard VAE is trained, a post-hoc learning of the prior distribution is performed. This is expected to move the prior closer to the aggregate posterior.\n- In order to increase prior's expressivity, authors employ Energy Based Models (EBMs). To sidestep the difficulties of the likelihood-based training of EBMs authors switch to Noise Contrastive Estimation (NCE) procedure. Two techniques are proposed to perform approximate sampling from the EBM.\n\n**Strong points**:\nThe paper is clearly written, technically correct and presents a new state-of-the-art method that significantly improves sample quality of Variational Autoencoders.\n\n**Weak points**:\nMy major problem with the paper is the seeming disparity between the models in the experimental comparison. In particular, authors build NCP-VAE upon NVAE [1], a state-of-the-art VAE architecture notorious for its large size and use of many stochastic layers (groups of latent variables in authors' terms). At the same time, for both multistage VAE baselines (*2 stage VAE* and *RAE*) the FID metrics were taken from the corresponding papers, which were based on much simpler architectures with a single group of latent variables.\n\nA more fair comparison would be to take the NVAE architecture and apply techniques from *RAE* [2] / *2 stage VAE* [3] to it. In the end of the sec. 4 authors say \"It is not clear how 2s-VAE or RAE are applied to state-of-the-art hierarchical models\". I believe this is a doable task. For example, consider (perhaps appropriately downscaled) NVAE's encoder (fig. 2a from [1]) and replace the input x with a random noise ε – the resulting latent variable model could be used as a second stage VAE [3]. For the *RAE* remove the red bottom-up path completely and make distribution over z1 a mixture. It might also be interesting to consider a mixture of size 1 -- a single gaussian distribution. In this case the 2nd stage training would be moving the prior towards the aggregate posterior. While in theory one might expect p(z) already be a decent approximation for q(z) (as shown in appendix B), in practice suboptimality of stochastic joint optimization might prevent p(z) from matching the aggregated q(z) to the fullest. The NVAE shares some weights between the prior and the encoder, which could either alleviate this problem by ensuring both encoder and prior change in a similar fashion, or hurt prior's expressivity since the top-down path has to be able to work in two somewhat different scenarios.\n\nFinally, the qualitative evaluation seems disconnected from the problem being solved. It is my understanding that lessening the prior hole problem should result in a fewer unappealing / blurry samples generated by the model. The additional samples in appendix H still contain some such samples. Unfortunately, no such samples are presented for the baseline models (in [1] the samples were curated). It's therefore hard to tell if the frequency of such samples was reduced.\n\n**Conclusion**: the results presented in the paper are impressive and the quality of samples is indeed very high. Unfortunately, I have to vote for rejection of this paper in its current form for its lack of rigorous experimental evaluation. It feels like the authors did overly focus on presenting a new method and have not given the prior work their best effort.\n\nMy suggestions to authors:\n- Evaluate two stage approach to the NVAE: 1) nested VAE prior as in [3], 2) mixture over z1 as in [2], 3) fine-tune NVAE's prior. The later will also disentangle the benefits of an energy-based prior from those of the two-stage training.\n- Perform qualitative evaluation based on frequency of visually unappealing samples from NCP-VAE versus that of the NVAE.\n\n[1]: Arash Vahdat and Jan Kautz.  NVAE: A deep hierarchical variational autoencoder. \n\n[2]: Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf.  From variational to deterministic autoencoders.\n\n[3]: Bin Dai and David Wipf.  Diagnosing and enhancing vae models.\n\n\n\n## Post-rebuttal update\n\nAuthors have addresses some of the issues outlined above, in particular the additional comparison in sec. 5.4 and table 7 is informative. However, the RAE numbers indicate that a simple 10-component Gaussian Mixture is superior to the complex model of NCP-VAE with a Gaussian prior on a small VAE, and the superiority of NCP with GMM prior base provides little information as RAE with 50-components GMM could have performed even better. It's also not clear how 2s-VAE, RAE and WAE would scale to bigger architectures such as NVAE, which raises the question if the NCP was the optimal one.\n\nIt's surprising to see the 2-stage VAE to perform worse than the standard VAE, whereas (Dai & Wipf, 2018) have shown a significant improvement in the original paper. I think, this result needs further elaboration and validation.\n\nRegarding the qualitative evaluation: figure 12 does show that the NCP-NVAE has fever poor samples, although it's hard to tell whether the difference is significant. Another issue is that the comparison does not include prior works.\n\nAs a result I bump by score to 5, but I think the paper still needs more work to make a sound argument for the proposed idea.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is rejected.

All of the reviewers found the empirical results strong. However, R3 and R4 pointed out concerns with the positioning of the work relative to prior work and that their approach is conceptually similar to previous work. The authors have tried to address these concerns in their rebuttal. Both reviewers appreciate the changes, but still have remaining concerns that I agree with. Based on these concerns, it is unclear if the strong empirical results are mostly due to using the NVAE architecture, rather than a methodological improvement over previous methods. The authors should work on positioning their paper in the context of prior work and the comparisons requested by R3 and R4 for a resubmission.

Decision: Reject