VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
Authors: Zhisheng Xiao
Karsten Kreis
Jan Kautz
Arash Vahdat
Venue: ICLR 2021 Spotlight (2021)
Keywords: Energy-based Models
Variational Auto-encoder
MCMC

Abstract: Energy-based models (EBMs) have recently been successful in representing complex distributions of small images. However, sampling from them requires expensive Markov chain Monte Carlo (MCMC) iterations that mix slowly in high dimensional pixel space. Unlike EBMs, variational autoencoders (VAEs) generate samples quickly and are equipped with a latent space that enables fast traversal of the data manifold. However, VAEs tend to assign high probability density to regions in data space outside the actual data distribution and often fail at generating sharp images. In this paper, we propose VAEBM, a symbiotic composition of a VAE and an EBM that offers the best of both worlds. VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples. Moreover, the VAE component in VAEBM allows us to speed up MCMC updates by reparameterizing them in the VAE's latent space. Our experimental results show that VAEBM outperforms state-of-the-art VAEs and EBMs in generative quality on several benchmark image datasets by a large margin. It can generate high-quality images as large as 256$\times$256 pixels with short MCMC chains. We also demonstrate that VAEBM provides complete mode coverage and performs well in out-of-distribution detection. 

Published as a conference paper at ICLR 2021
VAEBM: A SYMBIOSIS BETWEEN VARIATIONAL AU- TOENCODERS AND ENERGY-BASED MODELS
Zhisheng Xiao
Karsten Kreis, Jan Kautz, Arash Vahdat
Computational and Applied Mathematics The University of Chicago zx i www.@uuchccag..c.
NVIDIA (kkreis, 8, jkautz, aaadaaaaaaaaaeeeiiiia..ooom
ABSTRACT
Energy-based models (EBMs) have recently been successful in representing com- plex distributions of small images. However, sampling from them requires ex- pensive Markov chain Monte Carlo (MCMC) iterations that mix slowly in high dimensional pixel space. Unlike EBMs, variational autoencoders (VAEs) gener- ate samples quickly and are equipped with latent space that enables fast traversal of the data manifold. However, VAEs tend to assign high probability density to re- gions in data space outside the actual data distribution and often fail at generating sharp images. In this paper, propose VAEBM, symbiotic composition of a VAE and an EBM that offers the best of both worlds. VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples. Moreover, the VAE component in VAEBM al- lows us to speed up MCMC updates by reparameterizing them in the VAE's latent space. Our experimental results show that VAEBM outperforms state-of-the-ar VAES and EBMs in generative quality on several benchmark image datasets by a large margin. It can generate high-quality images as large as 256 x 256 pixels with short MCMC chains. We also demonstrate that VAEBM provides complete mode coverage and performs well in -uttr--distribution detection.
INTRODUCTION
Deep generative learning is central problem in machine learning. It has found diverse applications, ranging from image (Brock et al.) 2018 Karras et al. 2019 Razavi et al. 2019), music (Dhariwal
al. 2020) and speech (Ping et al. 2020 Oord et [006 6a) generation, distribution alignment
across domains (Zhu et al. [017]] Liu et al. 2017: T zeng et al. 2017) and semi-supervised learn-
ing (Kingma et al., 2014 Izmailov al. (020)) to 3D point cloud generation (Yang et al. 2019)
light-transpore simulation (uiller et al. (2019), molecular modeling (Sanchez- Lengeling & Aspuru-
Guzik 2018 Noé et al. 2019) and equivariant sampling in theoretical physics (Kanwar al. 2020).
Among competing frameworks, likelihood-baan based models include variational autoencoders (VAEs)
(Kingma & Welling [014] Rezende et al.) 2014), normalizing flows (Rezende & Mohamed] 2015
Dinh et al.| 2016)) autoregressive models (Oord al. 22066b), and energy-based models (EBMs)
(Lecun et al. 2006; Salakhutdinov et al. 2007). These models are trained by maximizing the data
likelihood under the model, and unlike generative adversarial networks (GANs) (Goodfellow al.,
2014), their training is usually stable and they cover modes in data more faithfully by construction.
Among seaiihood-basce models. EBMs model the unnormalized data density by assigning low en- regions in the data space (Xie et al.) 2016 Du & Mordatch] 2019). EBMs
ergy to appealing because they require almost restrictions on network architectures (unlike normal- izing flows) and are therefore potentially very expressive. They also exhibit better robustmess and out-of-distribution generalization (Du & Mordatch| 2019) because, during training, areas with high probability under the model but low probability under the data distribution are penalized explicitly. However, training and sampling EBMs usually requires MCMC, which can suffer from slow mode mixing and is computationally expensive when neural networks represent the energy function.
Work done during an internship NVIDIA

---

Published as a conference paper at ICLR 2021
On the other hand, VAES are computationally more efficient for sampling than EBMs, they do not require running expensive MCMC steps. VAEs also do not suffer from expressivity limitations that normalizing flows face (Dupont al. 2019; Kong & Chaudhuri, 2020), and in fact, they have recently shown state-of-the-art generative results among non-autoregressive fikelihood-based mod- els (Vahdat & Kautz, 2020). Moreover, VAES naturally come with latent embedding of data that allows fast traverse of the data manifold by moving in the latent space and mapping the movements to the data space. However, VAEs tend to assign high probability regions with low density under the data distribution. This often results in blurry or corrupted samples generated by VAEs. This also explains why VAEs often fail at out-of d--striibutiin detection (Nalisnick et al.) 2019))
In this paper, we propose novel generative model as a symbiotic composition of VAE and an EBM (VAEBM) that combines the best of both. VAEBM defines the generative distribution as the product of a VAE generator and an EBM component defined in pixel space. Intuitively, the VAE captures the majority of the mode structure in the data distribution. However, may still generate samples from low-probability regions in the data space. Thus, the energy function focuses on refining the details and reducing the likelihood of non-data-like regions, which leads to significantly improved samples.
Moreover, we show that training VAEBM by maximizing the data likelihood easily decomposes into training the VAE and the EBM component separately. The VAE is trained using the reparameteriza- tion trick, while the EBM component requires sampling from the joint energy-base model during training. We show that we can sidestep the difficulties of sampling from VAEBM, by reparametriz- ing the MCMC updates using VAE's latent variables. This allows MCMC chains to quickly traverse the model distribution and it speeds mixing. As a result, we only need to run short chains to obtain approximate samples from the model, accelerating both training and sampling at test time,
Experimental results show that our model outperforms previous EBMs and state-of-the-art VAEs on image generation benchmarks including CIFAR-10, CelebA 64, LSUN Church 64, and CelebA HQ 256 by a large margin, reducing the gap with GANs. We also show that our model covers the modes in the data distribution faithfully, while having less spurious modes for out-o--distribution data. To the best of knowledge, VAEBM is the first successful EBM applied to large images.
In summary, this paper makes the following contributions: i) We propose new generative model us- ing the product of a VAE generator and an EBM defined in the data space. ii) We show how training this model can be decomposed into training the VAE first, and then training the EBM component. iii) We show how MCMC sampling from VAEBM can be pushed to the VAE's latent space, accelerating sampling. iv) We demonstrate state-of-the-an image synthesis quality among likelihood- -ased mod- els, confirm complete mode coverage, and show strong -it----diitiiuttion detection performance.
2 BACKGROUND
Energy-based Models: An EBM assumes (x) to be a Gibbs distribution of the form py (xx exp (x), Zzp. where (x) is the energy function with parameters and Zg- Ix exp - (x)) dx is the normalization constant. There is no restriction on the particular form of (x). Given a set of samples drawn from the data distribution pa(x), the goal of maximum likelihood learning is to maximize the log-likelihood (( Ex-pa(x) [log Po which has the derivative (Woodford) 2006)::
T.b = Ex~pa(x) O Enp (x)] + (xxxps(x) Fnp Eng (x)]
(1)
For the first expectation, the positive phase, samples are drawn from the data distribution pa(x), and for the second expectation, the negative phase, samples are drawn from the model p (x) itself. However, sampling from (x) in the negative phase is itself intractable and approximate sam- ples usually drawn using MCMC. A commonly used MCMC algorithm is Langevin dynamics (LD) (Naa 1993). Given an initial sample Xy, Langevin dynamics iteratively updates it as:
X++1 X
(x))(xx) + /njw,, Wt ~ N I), 2
!
(2)
where 7) is the siep-size In practice, Eq. with invariant distribution approximately close the original target distribution.
2 is run for finite iterations, which yields a Markov chain
In principle one would require an accept/reject step make it rigorous MCMC algorithm, but for suffi- ciently small stepsizes this is not necessary in practice (Neal) 1993.
2

---

Published as a conference paper at ICLR 2021
VAE T5 (c2)
Z
VAE Decoder (z, 6x)
Energy Function E
Figure 1: Our VAEBM is composed of VAE generator (including the prior and decoder) and an energy function that operates on samples x generated by the VAE. The VAE component is trained first, using the standard VAE objective; then, the energy function is trained while the generator is fixed. Using the VAE generator, we can express the data variable x as deterministic function of white noise samples 6z and 6x- This allows us reparameterize sampling from VAEBM by sampling in the joint space of 6z and Ex. We use this in the negative training phase (see Sec. 3.1
Variational Autoencoders: VAEs define generative model of the form (x 2) pe (z pe (x z), where z is the latent variable with prior po po(z), (z). and po (x(xx) is a conditional distribution that models the likelihood of data given z. The goal of training is to maximize the marginal log-likelihood log pe (x) given a set of training examples. However since the marginalization intractable, instead, the variational lower bound on log po(x) (x) is maximized with 4s (z)x, x) the approximate posterior:
log Po (x) 2 E Ex [log (x(22] DKL (z(x)))ll (z)) == L (x, Φ))
The state-of-the-ar VAE, NVAE (Vahdat & Kautz 2020). increases the expressivity of both prior and approximate posterior using hierarchical latent variables (Kingma et al., 2016) where is de- composed into a set of disjoint groups, z = [21, 21..... }. and the prior po (z) Il po (o 2<1) and the approximate posterior (z(x)) = Il 96 (2gl2zz z x) are defined using autoregressive distri-
butions over the groups. We refer readers to Vahdat & Kautz (2020) for more details.
ENERGY-BASED VARIATIONAL AUTOENCODERS
One of the main problems VAES that they tend to assign high probability to regions in data space that have low probability under the data distribution. To tackle this issue, we propose VAEBM, generative model constructed by the product of a VAE generator and an EBM component defined in the data space. This formulation allows model to capture the main mode structure of the data distribution using the VAE. But when training the joint VAEBM, in the negative training phase sample from the model itself and discover non-data- data-like samples, whose likelihood is then reduced by the energy function explicitly. The energy function defined in the pixel space also shares similarities with discriminator in GANs, which can generate crisp and detailed images.
Formally, we define the generative model in VAEBM as hy,o z) Po z)e-Eeeeee (x) where po(x, z) = pe (x|xx) is a VAE generator and (x) is a neural network-band energy func- tion, operating only in the x space, and Z = f (x(ee FFu (x) is the normalization constant. VAEBM is visualized in Fig. [ Marginalizing out the latent variable gives
(x))(x)
1 Za.e
po(xx z)e (x(x)
1
po(x) (x)x)
(4)
Z
Given a training dataset, the parameters of VAEBM, vb, 8, are trained by maximizing the marginal log-likelihood on the training data:
log hy.e (x) px - (x) log Z
(5)
2 F Ex log po 1o0ox(1)) (x/2) DRA. (2(x)))lpppp)))
E.s (x) log Z
(6)
(x,
CeBM CEBM (0.
where we replace log Py (x) with the variational lower bound from Eq. 3 Eq 6 forms the objective function for training VAEBM. The first term corresponds to the VAE objective and the second term corresponds to training the EBM component. Next, we discuss how can optimize this objective.
3

---

Published as a conference paper at ICLR 2021
3.1 TRAINING
The (xAM(X, v5, 0) term in Eq. 6 is similar to the EBM training objective except that the log partition
function depends on both is and 0. We show in Appendix A that log Zo.o has the gradients
0p log Z (xx.x) Eu (x)] and 0g log zv.o Ex~hu,.
(0g log Pa (x))
The first gradient can be estimated easily by evaluating the gradient of the energy function at samples drawn from the VAEBM model 2 using MCMC. However, the second term involves com- puting the intractable log Jog.o((x) In Appendix A we show that estimating 300 log pø (x) requires sampling from the VAË's posterior distribution, given model samples x ~ f...0 (x, z). To avoid the computational complexity of estimating this term, for example with second round of MCMC, we propose a two-stage algorithm for training VAEBM. In the first stage, we train the VAE model in our VAEBM by maximizing the Lume 0, () term in Eq. 6 This term is identical to the VAE's objective, thus, the parameters 0 and ø are trained using the reparameterized trick in Sec. 2 In the second stage, we keep the VAE model fixed and only train the EBM component. Since is now fixed, we only require optimizing CEBM (x. 2/, 0) w.r.t. v. the parameters of the energy function. The gradient of L(b) = Excepa (EBM(Xx 15, 8)] w.r.t t6 is:
200
(v) = (x)) 1 B (x)] + Ex-hhg. (x.x. Eg (x)],
which decomposes into positive and negative phase, as discussed in Sec. 2
Reparametrized sampling in the negative phase: For gradient estimation in the negative phase, can draw samples from the model using MCMC. Naively, we can perform ancestral sampling, first sampling from the prior pz)2), then running MCMC for pe (x(2)) (x) in x-space. This is problematic, since pe (x|z) is often sharp and MCMC cannot mix when the conditioning z is fixed.
In this work, we instead run the MCMC iterations in the joint space of and x. Furthermore, we accelerate the sampling procedure using reparametrization for both x and the latent variables Recall that when sampling from the VAE, we first sample z ~ p(z) and then x (x(x))) This sampling scheme can be reparametrized by sampling from a fixed noise distribution (e.g., (ex, 6x) ~ Pe- (0, I)) and deterministic transformations TV such that
z = (E))).
x x= == (zz2), ex) = TT (T))
Here, T denotes the transformation defined by the prior that transforms noise e, into prior samples and T represents the decoder that transforms noise 6x into samples x, given prior samples z. We can apply the same reparameterization when sampling from ex, 2). This corresponds to sampling 62) from the "base distribution":
(9) and then transforming them and z via Eq. (see Appendix for derivation). Note that 62 and Ex have the same scale, as Pe (Ex,ez) is a standard Normal distribution, while the scales of x and z can be very different. Thus, running MCMC sampling with this reparameterization in the (ex; 6z) space has the benefit that we do not need to tune the sampling scheme (e.g., step size in LD) for each variable. This is particularly helpful when z itself has multiple groups, as in our case.
hw.0 (Exx Ex) (TT( Pe (Ex,62)
The advantages of two-stage training: Besides avoiding the difficulties of estimating the full gradient of log Ze..e two-stage training has additional advantages. As we discussed above, updating di is computationally expensive, each update requires an iterative MCMC procedure to draw sam- ples from the model. The first stage of our training minimizes the distance between the VAE model and the data distribution, and in the second stage, the EBM further reduce the mismatch between the model and the data distribution. As the pre-trained VAE po (x) provides a good approximation to Pa(x) already, we expect that a relatively small number of expensive updates for training & is needed. Moreover, the pre-trained VAE provides a latent space with an effectively lower dimension- ality and a smoother distribution than the data distribution, which facilitates more efficient MCMC.
Alternative extensions:
During the training of the energy function, we fix the VAE's parameters. we discuss a possible extension to our training objective that also updates the VAE.
In Appendix C
4 RELATED WORK
Early variants of EBMs include models whose energy is defined over both data and auxiliary latent variables (Salakhutdinov & Hinton| 2009; Hinton (012)), and models using only data variables (Hin)
ton 2002 Mnih & Hinton 2005). Their energy functions simple and they do not scale to high
4

---

Published as a conference paper at ICLR 2021
dimensional data. Recently, it was shown that EBMs with deep neural networks as energy function can successfully model complex data such as natural images (Du & Mordatch] 2019: Nijkamp et al.) 2019b/a). a). They are trained with maximum likelihood and only model the data variable. Joint EBMs [Grathwohl et al. 2020a I Liu & Abbeel 2020) model the joint distribution of data and labels. In
contrast, VAHBM models the joint distribution of data and general latent variables.
Besides fundamental maximum likelihood training, other techniques to train EBMs exist, such as minimizing F-divergence (Yu al., 2020a) or Stein discrepancy (Grathwohl et al.) 2020b), con- trastive estimation (Gutmann & Hyvärinen 2010; Gao et al. 2020 and denoising score matching el. 2019). Recently, noise contrastive score networks and diffusion models have demonstrated high quality image synthesis (Song & Ermon, 2019 [220]] Ho et al.) (200). These models are also based on denoising score matching (DSM) (Vincent 2011), but do not parameterize any explicit energy function and instead directly model the vector- valued score function. We view score-based models as alternatives to EBMs trained with maximum likelihood. Although they do not require iterative MCMC during training, they need very long sampling chains to anneal the noise when sampling from the model & 1000 steps). Therefore, sample generation is extremely slow.
VAEBM is EBM with VAE component, and it shares similarities with work that builds con- nections between EBMs and other generative models. Zhao et al. (2017): ) Che et al. (2020); [Song ct al. (0220); Arbel et al. (2020) formulate EBMs with GANs, and use the discriminator to assign an energy. Xiao et al. 2020); Nijkamp et al. (2020) use normalizing flows that transport complex data to latent variables to facilitate MCMC sampling (Hoffman al. 2019), and thus, their methods can be viewed as EBMs with flow component. However, due to their malizing flows cannot easily transport complex multimodal data, and their sample quality on images is limited. A few previous works combine VAEs and EBMs in different ways from ours. Pang el al. (2020) and Vahdat et al. (2018b a 2020) use EBMs for the prior distribution, and (Han et al. [020] 2019] jointly learn a VAE and an EBM with independent sets of parameters by an adversarial game.
nature, nor-
Finally, as we propose two-stage training, our work related to post training of VAEs. Previous work in this direction learns the latent structure of pre-trained VAEs (Dai & Wipi 2019 Xiao et al.)
2019 Ghosh et al. 2020), and sampling from learned latent distributions improves sample quality. These methods cannot easily be extended to VAEs with hierarchical latent variables, as it is difficult to the joint distribution of multiple groups of variables. Our purpose for two-stage training is fundamentally different: we post-train an energy function refine the distribution in data space.
EXPERIMENTS
In this section, evaluate our proposed VAEBM through comprehensive experiments. Specifically, we benchmark sample quality in Sec. 5. provide detailed ablation studies on training techniques in Sec. 5.2 and study mode coverage of our model and test for spurious modes in Sec. 5.3 We choose NVAE (Vahdat& & Kautz, 2020) as our VAE, which we pre-train, and use simple ResNet as energy function Egg, similar to Du & Mordatch (2019). We draw approximate samples both for training and testing by running short Langevin dynamics chains on the distribution in Eq. [ Note that NVAE, the prior distribution is a group-wise auto-regressive Gaussian, and the conditional pixel-wise distributions in x are also Gaussian. Therefore, the reparameterization corresponds to shift and scale transformations. For implementation details, please refer to Appendix
5.1 IMAGE GENERATION
In T we quantitatively compare the sample quality of VAEBM with different generative mod- els on (unconditional) CIFAR-10. We adopt Inception Score (IS) (Salimans et al. (016]) and FID (Heusel et al.] 2017) as quantitative metrics. Note that FID reflects the sample quality more faith- fully, as potential problems have been reported for IS on CIFAR- 10 (Barratt & Sharmal 2018).
We observe that our VAEBM outperforms previous EBMs and other explicit likelihood- mod- els by a large margin. Note that introducing persistent chains during training only leads to slight improvement, while Du & Mordatch (2019) rely on persistent chains with a sample replay buffer. This is likely due to the efficiency of sampling in latent space. Our model also produces significantly better samples than NVAE, the VAE component of our VAEBM, implying a significant impact of our proposed energy-based refinement. We also compare our model with otate-of-the-art GANs and
5

---

Published as a conference paper at ICLR 2021
Table 1: IS and FID scores for unconditional generation on CIFAR-10.
Model
1ST
FID.
VAEBM w/o persistent chain
8.21
12.26
Ours
VAERM persistent chain
8.43
12.19
IGEBM (Du & Mordatch 2019)
6.02
40.58
FBM with short-run MCMC
NNijkamp et al. 2019h
6.21
F-div EBM (Yu al.
2020a
8.61
30.86
EBMs
FlowCE (Gao ct al. (2020)
37.3
FlowEBM (Nikkmmp el al
2020
78. 12
GEBM (Arbel el al.
2020)
23.02
Divergence Trangle [Ha et al.
2020
30.1
Glow (Kingma Dhariwal
2018
3.92
48.9
Other
PIxeICNN (Oord el al.
2016b
4.60
65.93
Likelihood
NVAE (Vahdat & Kautz 2020)
5.51
51.67
Models
VAE with EBM prior Pang al.
2020
70.15
NCSN (Song Ermon 2019)
8.87
25.32
Score-based
NCSN (Song Ermoon 20200
31.75
Models
Multi-scale DSM (Li al. 2019
8.31
31.7
Denoising Diffusion (Ho ct al.
2020
9.46
3.17
SNGAN (Miyato al. 2018)
8.22
21.7
SNGAN+DDLS (Che
2020)
9,09
15.42
GAN-based
SNGAN+DCD (Song ct al
2020
9.11
16.24
Models
BigGAN (Brock et al.
2018)
9.22
14.73
StyleGANZ w/o w/o ADA Karras et al.
2020a
8.99
9.9
PixelIQN (Ostrovski et all 2018)
5.29
49.
Others
MoLM (Ravuri et al. 2018
7.90
18.9
recently proposed score-based models, and we obtain comparable or better results. Thus, we largely close the gap GANs and score-models, while maintaining the desirable properties of models trained with maximum likelihood, such as fast sampling and better mode coverage.
Qualitative samples generated by our model are shown in Fig. Za and intermediate samples along MCMC chains in Fig. 2b) We find that VAEBM generates good samples by running only a few MCMC steps. Initializing MCMC chains from the pre-trained VAE also helps quick equilibration.
(a)
Figure 2: (a) CIFAR-10 10 samples generated by VAEBM. (b) Visualizing MCMC sampling chains. Samples are generated by running 16 LD steps. Chains are initialized with pre-trained VAE. We show intermediate samples at every 2 steps. See Appendix for additional qualitative results.
We also train VAEBM on larger images, including CelebA 64, CelebA HQ 256 ei al. 2015) and LSUN Church 64 YYu al. 2015). We report the FID scores for CelebA 64 and CelebA HQ 256 in Tables and On CelebA 64, our model obtains results comparable with the best GANs. Although model obtains worse results than advanced GANs CelebA HQ 256, significantly

---

Published as a conference paper at ICLR 2021
(a) CelebA 64
(b) LSUN Church 64
(c) CelebA HQ 256
Figure 3: Qualitative results on CelebA 64, LSUN Church 64 and CelebA HQ 256. For CelebA HQ 256, we initialize the MCMC chains with low temperature NVAE samples (1 = 0.7) for better visual quality. On this dataset samples selected for diversity. See Appendix for additional qualitative results and uncurated CelebA HQ 256 samples obtained from higher temperature initializations. Note that the FID in Table computed with full temperature samples.
Table 2: Generative performance on CelebA 64
Table 3: Generative performance on CelebA HQ 256
Model
FIDL
VAEBM (ours)
5.31
Model
FID
NVAE (Vahdat Kautz)
14.74
VAEBM (ours)
20.38
Flow CE (Gao ct al.)
12.21
NVAE (Vahdat & Kautz)
45.11
Divergence Inangic (Han et al.
24.7
GLOW (Kingma & Dhariwal
68.93
NCSNv2 (Song & Ermon)
26.86
Advers. LAE (Pidhorskyi et al.)
19.21
COCO-GAN (Lin al.)
4.0
PGGAN (Krras et al.)
8.03
QA-GAN (Parmaaaa & Channappayya)
6.42
reduce the gap between likelihood based models and GANs on this dataset. On LSUN Church 64, we obtain FID 13.51, which significantly improves the NVAE baseline FID 41.3. We show qualitative samples in Fig. Appendix H contains additional samples and MCMC visualizations.
Our model can produce impressive samples by running very short MCMC chains, however, we find that when we run longer MCMC chains than training chains, most chains stay around the local mode without traversing between modes. We believe that the non-min non-mixing is due to the long mixing time of Langevin Dynamics Neal et al. (20IT). as Nijkamp et al. (201 a) also observe that models trained with short-run MCMC have non-mixing long-run chains. We conjecture that mixing can be improved by training and sampling with more advanced MCMC techniques that known to mix faster, such as HMC Neal et al. (2011), and this will be left for future work.
Table 4: Comparison for IS and FID on CIFAR- 10 between several related training methods.
Table 5: Mode coverage on StackedMNIST.
Model
Modest
KL.]
Model
ISt
FID
VEEGAN (Srivastava et al.)
761.8
2.173
NVAE Vahdat Kautz)
5.19
55.97
PacGAN (Lin et al.
992.0
0.277
on x (Du & Mordatch)
5,85
48.89
PresGAN (Dieng
999.6
0.115
EBM on x, MCMC int w/ NVAE
7.28
29.32
InclusiveGAN Yu
997
0.200
WGAN w/ NVAE decoder
7.41
20.39
StyleGAN2 (Karras et al.)
940
0.424
VAEBM (ours)
8.15
12.96
VAEBM (ours)
1000
0,087
7

---

Published as a conference paper at ICLR 2021
5.2 ABLATION STUDIES
In Table we compare VAEBM to several closely related baselines. All the experiments here are performed on CIFAR-10, and for simplicity, we use smaller models than those used in Table [] Appendix summarizes the experimental settings and Appendix provides qualitative samples.
Data space vs. augmented space: One key difference between VAEBM and previous work such as Du & Mordatch (2019) is that our model is defined on the augmented space (x, 2), while their EBM only involves x. Since we pre-train the VAE, one natural question is whether our strong results are due to good initial samples from the VAE, which are used to launch the MCMC chains. To address this, we train an EBM purely on x as done in Du Mordatch (2019). We also train another EBM only but we initialize the MCMC chains with samples from the pre-trained NVAE instead of noise. As shown in line 3 of Table 4 this initialization helps the EBM which is defined only on x. However, VAEBM in the augmented space outperforms the EBMs
only by large margin.
Adversarial training vs. sampling: The gradient for & in Eq. 7 is similar to the gradient updates of WGAN's discriminator (Arjovsky al., 2017). The key difference is that draw (approximate) samples from by by MCMC, while WGAN draws negative samples from a generator (Che et al., 2020). WGAN updates the generator by playing an adversarial game, while we only update the energy function Eg. We compare these two methods by training de and 0 with the WGAN objective and initializing 0 with the NVAE decoder. As shown in line of Table we significantly outperform the WGAN version of our model, implying the advantage of our method over adversarial training.
5.3 TEST FOR SPURIOUS OR MISSING MODES
We evaluate mode coverage on StackedMNIST. This dataset contains images generated by randomly choosing MNIST images and stacking them along the RGB channels. Hence, the data distribution has 1000 modes. Following [in et al. (2018), we report the number of covered modes and the KI. divergence from the categorical distribution over 1000 categories from generated samples to true data (Table VAEBM covers all modes and achieves the lowest KL divergence even compared to GANs that are specifically designed for this task. Hence, our model covers the modes more equally. We also plot the histogram of likelihoods for CIFAR- 10 train/test images (Fig, 6 Appendix (D) and present nearest neighbors of generated samples (Appendix []. We conclude that we do not overfit.
We evaluate spurious modes in model by assessing its performance (OOD) detection. Specifically, we use VAEBM trained on CIFAR-10, and estimate unnormalized log hx) on in-distribution samples (from CIFAR-10 test set) and OOD samples from several datasets. Following Nalisnick et al. (2019), we use area under the ROC curve (AUROC) as quan- titative metric, where high AUROC indicates that the model correctly assigns low density to OOD samples. In Table 6 we see that VAEBM has significantly higher AUROC than NVAE, justifying our argument that the energy function reduces the likelihood of non-data-lik, regions. VAEBM also performs better than IGEBM and JEM, while worse than HDGE. However, we note that JEM and HDGE are classifier-based models, known to be better for OOD detection (Liang et al. 2018).
out-of-distributie
Table 6: Table for AUROCT of log p(x) computed on several OOD datasets. In-distribution dataset is CIFAR-10. Interp. corresponds to linear interpolation between CIFAR-10 images.
SVHN
Interp.
CIFAR100
CelebA
NVAE (Vahdat & Kautz 2020
0.42
0.64
0.56
0.68
Glow (Knnmma & Dhariwal 2018)
0.05
0.51
0.55
0.57
Unsupervised Training
IGEBM (Du& & Mordatch 2019
0.63
0.7
0.5
0.7
Divergence Traingle (Han et al.
2020
0.68
0.56
VAEBM (ours)
0.83
0.7
0.62
0.77
Supervised
JEM (Grathwohl et al. 2020al
0.67
0.65
0.67
0.75
Training
HDGE (Liu Abbeel
2020)
0.96
0.82
0.91
0.8
5.4 EXACT LIKELIHOOD ESTIMATE ON 2D ToY DATA
VAEBM an explicit likelihood model with a parameterized density function. However, like other energy-based models, the estimation of the exact likelihood is difficult due to the intractable partition
8

---

Published as a conference paper at ICLR 2021
function log Z. One possible way to estimate the partition function is to Annealed Importance Sampling (AIS) (001), However, using AIS to estimate log Z in high-dimensional spaces is difficult. In fact, Du & Mordatch (2019) report that the estimation does not converge in days CIFAR-10. Furthermore, AIS gives a stochastic lower bound on log Z, and therefore the likelihood computed with this estimated log Z would be upper bound for the true likelihood. This makes the estimated likelihood hard to compare with the VAE's likelihood estimate, which is usually lower bound on the true likelihood (Burda al. 2015).
As result, to illustrate that our model corrects the distribution learned by the VAE and improves the test likelihood, we conduct additional experiments on a 2-D toy dataset. We use the 25-Gaussians dataset, which is generated by mixture of 25 two-dimensional isotropic Gaussian distributions arranged in grid. This dataset also studied in Che et al. (2020). The encoder and decoder of the VAE have 4 fully connected layers with 256 hidden units, and the dimension of the latent variables 20. Our energy function has 4 fully connected layers with 256 hidden units.
In the 2-D domain, the partition function log Z can be accurately estimated by numerical inte- gration scheme. For the VAE, we use the IWAE bound (Burda et al. 2015) with 10,000 posterior samples to estimate its likelihood. We use 100,000 test samples from the true distribution to evaluate the likelihood. Our VAEBM obtains the average log likelihood of -1.50 nats on test samples, which significantly improves the VAE, whose average test likelihood is -2.97 nats. As reference, also analytically compute the log likelihood test samples under the true distribution, and the result is -1.10 nats.
We show samples from the true distribution, VAE and VAEBM in Figure 4 VAEBM successfully corrects the distribution learned by the VAE and has better sample quality.
We observe that the
9
0
(a) Samples from the true distribution
(b) Samples from VAE
(c) Samples from VAEBM
Figure 4: Qualitative results on the 25-Gaussians dataset
5.5 SAMPLING EFFICIENCY
Despite their impressive sample quality, denoising score matching models (Song Ermon 2019 Ho et al. 2020) are slow at sampling. often requiring 1000 MCMC steps. Since VAEBM uses short MCMC chains, it takes only 7.79 seconds to generate 50 CIFAR-10 samples, whereas NCSN (Song
& Ermon 2019)
takes 107.9 seconds, which about 12x slower (see Appendix for details).
CONCLUSIONS
We propose VAEBM, energy-based generative model in which the data distribution is defined jointly by a VAE and an energy network, the EBM component of the model. In this joint model, the EBM and the VAE form symbiotic relationship: the EBM component refines the initial VAE- defined distribution, while the VAE's latent embedding space is used to accelerate sampling from the joint model and therefore enables efficient training of the energy function. We show that our model can be trained effectively in two stages with maximum likelihood objective and we can efficiently sample it by running short Langevin dynamics chains. Experimental results demonstrate strong generative performance on several image datasets. Future work includes further scaling up the model larger images, applying it to other domains, and using more advanced sampling algorithms.

---

Published as a conference paper at ICLR 2021
REFERENCES
Michael Arbel, L. Zhou, and A. Gretton. Generalized energy based models. arXiv preprint arXiv:2003.0510 050033, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. Proceedings of the 34th International Conference on Machine Learning, 2017.
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv: 18010097, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.1199 2018.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519, 2015.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003. 060600, 2020.
International Conference on Bin Dai and David Wipf. Diagnosing and enhancing vae models. Learning Reprexentations, 2019.
Prafulla Dhariwal, Heewoo Jun, Christine Payne. Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: generative model for music. arXiv preprint arXiv:2005.00341 2020.
Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed generative adversarial networks. arXiv preprint 1910:4490 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint ar 1005::60000880. 2016.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, pp. 3608-3618, 2019..
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural ODEs. In Advances in Neural Information Processing Systems, pp. 3 3140-3150, 2019.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7518-7528, 2020.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational deterministic autoencoders. In International Conference on Learning Representa- tions, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems, pp. 2672-2680, 2014.
Will Grathwohl, Kuan-Chieh Wang. Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2020a.
Will Grathwohl, Kuan-Chinn Chinh Wang, Jörn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy- based models without sampling. In ICML, 2020b.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pp. 297-304, 2010.
10

---

Published as a conference paper at ICLR 2021
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training generator model, energy-based model, and inferential model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8670-
8679, 2019.
Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7978-79877 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in nearal information processing systems, pp. 6626-6637, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
Geoffrey Hinton. A practical guide to training restricted boltzmann machines. In Neural networks: Tricks of the trade, pp. 599-619. Springer, 2012.
Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Matthew Hoffman, Pavel Sountsov, Joshua V Dillon, Ian Langmore, Dustin Tran, and Srinivas Vasudevan. Neutra-lizing bad geometry in hamiltonian monte carlo using neural transport. arXiv preprint
2019.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learn- ing with normalizing flows. In ICML, 2020.
Gurtej Kanwar, Michael S. Albergo, Denis Boyda, Kyle Cranmer, Daniel C. Hackett, Sébastien Racanière, Danilo Jimenez Rezende, and Phiala E. Shanahan. Equivariant flow-based sampling for lattice gauge theory. Phys. Rev. Lett., 125:12 Sep 2020. doi: 10.1 1e.ehhyseeeeee 125. 121601.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006. 206.:20000006676, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederík Kingma and Max Welling, Auto-encoding variational bayes. In The International Con- ference on Learning Representations (ICLR), 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions, In Advances in neural information processing systems, pp. 10215-10224, 2018.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances neural information processing systems, pp- 3581-3589, 2014.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751, 2016.

---

Published as a conference paper at ICLR 2021
Zhifeng Kong and Kamalika Chaudhuri. The expressive power of class of normalizing flow mod- els. arXiv preprint arXiv:2006. 00300, 2020.
Yann Lecun, Sumit Chopra, Raia Hadsell, Mare Aurelio Ranzato, and Fu Jie Huang. A tutorial on energy-based learning. In G. Bakir, T. Hofman, B. Scholkopt, A. Smola, and B. Taskar (eds.), Predicting structured data. MIT Press, 2006.
Zengyi Li, Yubei Chen, and Friedrich T Sommer. Annealed denoising score matching: Learning energy-based models in high-dimensional spaces, arXiv preprint arXiv:1910.0776. 2019.
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection neural networks. In International Conference Learning Representations, 2018.
Chich Hubert Lin, Chia- Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann- Tzong Chen. Coco-gan: generation by parts via conditional coordinating. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4512-4521, 2019.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power two samples in generative adversarial networks. In Advances in neural information processing systems, pp- 1498-1507, 2018.
Hao Liu and Pieter Abbeel. Hybrid discriminnaive--enerraiee training via contrastive learning. arXiv preprint 00000 2020.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Andriy Mnih and Geoffrey Hinton. Learning nonlinear constraints with contrastive backpropagation. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, IEEE, 2005.
pp.
Thomas Müller, Brian Mcwilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural Importance Sampling. ACM Trans. Graph., 38(5), October 2019. ISSN 0730-0301. doi: 10.1145/334115..
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji I Lakshminarayanan. deep generative models know what they don't know? In International Conference on Learning Representations, 2019.
Radford M Neal. Probubilistic inference using Markov chain Monte Carlo methods. 1993.
Radford M Neal. Annealed importance sampling. Statistics and computing, 11(2): 139, 2001.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme- based maximum likelihood learning of energy-based models. arXiv preprint /903. /2370, 2019a.
Erik Nijkamp. Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward ensrgy-based model. In Advances Neural Information Pro- cessing Systems, pp. 5232-5242, 2019b.
Erik Nijkamp, Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song- Sang-Chun Zhu, and Ying Nian Wu. Learning energy-based model with Bow-based backbone by neural transport meme. arXiv preprint arXiv:2006.008897 2020.
12

---

Published as a conference paper at ICLR 2021
Frank Noé, Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning. Science, 365(6457), 2019. ISSN 0036-8075.
doi: 10.1 www.cciencc..aw//.....o
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. ICML, 2016b.
Georg Ostrovski, Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. arXiv preprint arXiv:/806.05575. 2018.
Bo Pang. Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. arXiv preprini WXiv:2006.08203 2020.
Kancharla Parimala and Sumohana Channappayya. Quality aware generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 1944-29558 2019.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14104-14113, 2020.
Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. ICML, 2020.
Suman Ravuri, Shakir Mohamed, Mihaela Rosca, and Oriol Vinyals. Learning implicit generative models with the method of learned moments. arXiv preprint arXiv:1806.11 1006: 2018.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866-14876, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In
International Conference Machine Learning, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference on Machine Learn- ing. pp. 1286, 2014.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pp. 448-455, 2009.
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. collaborative filtering. In ICMI., 2007.
Restricted boltzmann machines for
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, pp. 901- 909, 2016.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-22422 2016.
Benjamin Brrchezz-engeling and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, 361(6400):30--66, 2018. ISSN 0036-8075. doi: 10. htt.sccience.at266...
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances Neural Information Processing Systems, pp. 11918-11930. 2019.
Yang Song and Stefano Enmon. Improved techniques for training score-based generative models. arXiv preprint 3rXiv:2006.090/1 2020.
13

---

Published as a conference paper at ICLR 2021
Yuxuan Song, Qiwei Ye, Minkai Xu, and Tie- Yan Liu. Discriminator contrastive divergence: Semi-amortized generative modeling by exploring energy of the discriminator. arXiv preprint arXiv:2004.01704, 2020.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Infor- mation Processing Systems, pp. 3308-3318. 2017.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7167-7176, 2017.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Infor- mation Processing Systems (NeurIPS), 2020.
Arash Vahdat, Evgeny Andriyash, and William Macready. DVAE#: Discrete variational autoen- coders with relaxed Boltzmann priors. In Neural Information Processing Systems, 2018a.
Arash Vahdat, William G. Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny Andriyash.
DVAE++: Discrete variational autoencoders with overlapping transformations. In International Conference Machine Learning (ICML), 2018b.
Arash Vahdat, Evgeny Andriyash, and William G Macready. Undirected graphical models as ap- proximate posteriors. In International Conference on Machine Learning (ICML), 2020.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1674, 2011.
Oliver Woodford. Notes on contrastive divergence. Department of Engineering Science, University of Oxford, Tech. Rep, 2006.
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow: A framework for non-adversarial image generation. arXiv preprint arXiv: 10485, 2019.
Zhisheng Xiao, Qing Yan, and Yali Amit. Exponential tilting of generative models: Improving
sample quality by training and sampling from latent energy. arXiv preprint urXiv:2006.08100, 2020.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644, 2016.
G. Yang, X. Huang, Z. Hao, M. Liu, S. Belongie, and B. Hariharan. PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4540-4549, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 100Xvv:5506.03666 2015.
Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with f-divergence minimization. ICML, 2020a.
Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive gan: Improving data and minority coverage in generative models. arXiv preprint crXiv:2004.03355, 2020b.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. ICLR, 2017.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2232, 2017.
14

---

Published as a conference paper at ICLR 2021
A DERIVING THE GRADIENT OF log z
Recall that z0 po(x)) (x) For the derivative of log v00 w.r.t. 8, we have:
log Z
① log (f [po(x)e (x0x)
1
/ @pp(x)
(x)
00
00
Z
00
1
/ po(x)e (x) (x)()
log po(xx) dx f 1x)
d log po(x)) dx
00
00
allo
-Ex
(10)
00
Similarly, it is easy to show that 8 log Z Ex (xc.x)
(x. the
Intuitively, both gradients
265
encourage reducing the likelihood of the samples generated by the VAEBM model. Since, h3.0 is an EBM, the expectation can be approximated using MCMC samples.
Note that Eq. 10 is further expanded to:
a
a log po (x. z'
log Zo 8 = Exchs, (xxx)) Eyg (ee 00
00
which can be approximated by first sampling from VAEBM using MCMC (i.e., ho (x, z)) and then sampling from the true posterior of the VAE (i.e., z' ~ po(z') x))). The gradient term can be easily computed given the samples. Two approaches
be used to draw approximate samples from pe(z' (x). i) We can replace pe (z' xx) with the approximate posterior (z However, the quality of this estimation depends
how well (z'((x) matches the true posterior on samples generated by (x. z), which can be very different from the real data samples. To bring (z closer to [o [x), we can maximize the variational bound (Eq. 3 on samples generated from heg (x, z) with respect to o, the encoder parameter However, this will add additional complexity training. ii) Alternatively, we can use MCMC sampling to sample z' (o))) To speed up MCMC, we can initialize the z' samples in MCMC with the original z samples that were drawn in the outer expectation (i.e., X, z ~ h (x, 2x))).. However, with this approach, the computational complexity of the gradient estimation for the negative phase is doubled, as we now require running MCMC twice, once for z ~ o z) and again for z' pz(22(x),
We can entirely avoid the additional computational complexity and the complications of estimat- ing a0 log Ze. e. if we assume that the VAE is held fixed when training the EBM component of our VAEBM. This way, we require running MCMC only to sample x ~ hw,0 z) to compute av log Za.0.
B REPARAMETRIZATIONN FOR EBM
Suppose we draw the re- re-parametrization variables (Ex: 6x) ~ pe(Ex, For convenience, we denote To(ex; = (T (Ex),
(TT)))) 6x), (Ex)) (x,z). 2).
(11)
Since Te deterministic and invertible transformation that maps (Ex, EZ) to (x, z), by the change of variables formula, we can write
po(x, 2) Pe 2)) det (Jr.- (x, (x)22)
(12)
where Jr. is the Jacobian of To Consider a Gaussian distribution a simple example: z ~ N(H,, Oz) and xiz ~ ((xx ax xx (z)), then
(Ex) = Mz Oz Ez
x T7 (Exx Ex) [x((z) ox(z) Ex,
and
J (x, 1xx(z)=1,
Maximizing ELBO with respect corresponds minimizing DXL (g)(z)x) [z(xx) while 0 is fixed.
15

---

Published as a conference paper at ICLR 2021
Recall that the generative model of our EBM is
(v,0(x,)) 2) C E. (x) (x
(13)
Z
We can apply the change of variable to hx, 2) in similar manner.
h (ex, Ex) = (To,(TT((Ex 6x)) Idet (Jro 6x)),
(14)
where Jrg is the Jacobian of Te. Since we have the relation
Jp-1 o ffrof=Jp JF
(15)
for invertible function f, have that
hg,0(Ex,6 Ex) = h (To(ex (2)()xxx22)) deet (Jr, (Ex, 6x))
(16)
Zo
1
e
(TTT(ex pT 62)) |det (Jr, (ex,
(17)
-
1
(.-ollb(eraa))) (T (x, 2)) det (Jr. (x.2) det (JTo (xx:
(18)
Z
1
e (-Eo(To(ex Pe (Tg (x, 2))
(19)
Zu,0
1
(To((((ea pel E2):
(20)
7
which is the distribution in Eq. 9
After we obtained samples (ex; 6x) from the distribution Eq. 20 we obtain (x, by applying the transformation Tp in Eq.
B. COMPARISON OF SAMPLING IN (EX, E2)-ACEE AND IN (x, SPACCE
Above we show that sampling from h z) is equivalent to sampling from ha. 62) and applying the appropriate variable transformation. Here, we further analyze the connections between sampling from these two distributions with Langevin dynamics. Since each component of x and z can be re-parametrzied with scaling and translation of standard Gaussian noise, without loss of generality, we assume variable с (c can be single latent variable in z or a single pixel in x) and write
+ ae.
Suppose we sample in the e space with energy function f on c and step size n. The update for is
c11 E, V Viu We (O,
7
Now plug EL+1 into the expression of while noting that Vef oVef We obtain
C++1 + 06 = -u (c:
" "V.. + V
2
0²1 V Vnoour vel
uu + 0e;
8 vef + VnpZun.
C; 2
Therefore, we see that running Langevin dynamics in (Ex, Ep)-s is equivalent to running Langevin dynamics in (x, z)-space with step size for each component of and x adjusted by its variance. However, considering the high dimensionality of x and z, the step size adjustment is difficult to implement.
The analysis above only considers a variable individually. More importantly, our latent variable in the prior follows bise auto-regressive Gaussian distributions, so the variance of each
16

---

Published as a conference paper at ICLR 2021
component in depends the value of Zzi- We foresee that because of this dependency, using a fixed step size per component of will not be effective, even when set differently for each component. In contrast, all the components in (ex: Ex) -space have unit variance. Hence, universal step size for all the variables in this space can be used.
To further provide empirical evidence that adjusting the step size for each variable is necessary, try sampling directly in (x, 2 -space without adjusting the step size (i.e., use a universal step size for all variables). Qualitative results are presented in Figure 5 We examine several choices for the step size and we cannot obtain high-quality samples.
(a) Step size 8c-4
(b) Step size 8c-5
(c) Step size 8c-6
Figure Qualitative samples obtained from sampling (x, s)-ace with different step sizes.
In conclusion, the re-parameterizzation provides an easy implementation adjust step size for each variable, and the adjustment is shown to be crucial to obtain good samples.
C EXTENSION TO TRAINING OBJECTIVE
In the first stage of training VAEBM, the VAE model is trained by maximizing the training data log-likelihood which corresponds minimizing upper bound on (xI(pa(x)]lo((S In the second stage, when we are training the EBM component, we use the VAE model to sample from the joint VAEBM by running the MCMC updates in the joint space of e, and 6x. Ideally, we may want to bring (x) closer to h (x) in the second stage, because when px(x) h we will not need the expensive updates for th. We can bring Pe(x) closer o by minimizing (po (x) hl/ (xx) with respect to 0 which was recently discussed in the context of an EBM- interpretation of GANs by Che al. (2020). To do so, we assume the target distribution o
w.r.t. 0.
fixed and create copy of 0, named 8, and we update 0' by the gradient:
V DKLLppLpepp (x)] = V (xx [E (x)]
(21)
In other words, one update step for 0' that minimizes DRL (x) w.r.t. O' can be easily
done by drawing samples from (x) and minimizing the mnergy-function w.r.t 8'. Note that this approach is similar to the generator update in training Wasserstein GANs (Arjovsky et al.] 2017). ?)
The above KL objective will encourage po (x) to model dominants modes in (x)) However, may cause (x( to drop modes.
17

---

Published as a conference paper at ICLR 2021
C.I DERIVATION
Our derivation largely follows Appendix A.2 of Che et al. (2020). Note that every time we update we are actually taking the gradient w.r.t 0', which can viewed as copy of 0 and initialized as 0. In particular, we should note that the 9 in (w.o(x) is fixed. Therefore, we have
Vor DKL (per (x) (x) (m4,0()))))) = V Per (x) po po (x) - log (x)) dx
/ V or po (x)] [log pa (x) log Hip (x) dx
/ pa (x) V log Var log hyp (x)) dx
(22)
V or po (x)) [log Pa (x) - log hyp (x)) dx.
(23)
where the second term in Eq. 22 is O because the log (x)) does not depend on 0' and the expec- tation of the score function is O:
Par Vx) log po (x)dx Ex (x) Vor or log pa (x)] 0.
Recall that 6' has the same value as before the update, so
po (x) >no(x)e (x(
log po (x) log hxx log
log Zw.0
Ei (x) llog Zw.o.
(24)
Plug Eq. 24 into Eq. 23 we have
/ Vorpor (x) [Ex (x) log Zv.o] dx
0r DKL (po (x)
= -or [x)],
(25)
since
/ Vorpo (x) log Zw.adx = V log Zuro / po (x))xx= Ver log Zo.0 = 0.
C.2 RESULTS
We train VAEBM with an additional loss term that updates the parameter 0 to minimize DKL (po (x))ll),e(x))) as explained above. Our experiment uses the same initial VAE as in Sec. 5.2 and details of the implementation are introduced in Appendix F We obtain FID 14.0 and 8.05, which is similar to the results of plain VAEBM (FID 12.96 and IS 8.1.). Therefore, we conclude does not improve the performance, and
that training the model by minimizing DKL (pe hx) updating the decoder is not necessary. This is likely because the initial VAE is pulled closely as possible to the data distribution already, which is also the target for the joint VAEBM (x))
D COMPARING LIKELIHOODS ON TRAIN AND TEST SET
In Figure 6 we plot a histogram of unnormalized log- legeliklliooos of 10k CIFAR- 10 train set and test set images. We see that our model assigns similar likelihoods both train and test set images. This indicates that VAEBM generalizes well to unseen data and covers modes in the training data well.
E IMPLEMENTATION DETAILS
In this section, introduce the details of training and sampling from VAEBM.
18

---

Published as a conference paper at ICLR 2021
test train
600
500
400
300
200
100
0
20000
40000
60000
80000
100000
Figure 6: Histogram of unnormalized log- log-likelihoods on 10k CIFAR-10 train and test set images
VAEBM uses NVAE as the (x) component in the model. We train the NVAE with its official implementation We largely follow the default settings, with one major difference that we use Gaussian decoder instead of discrete logistic mixture decoder as in Vahdat & Kautz (2020). The reason for this is that we can run Langevin dynamics only with continuous variables. The number of latent variable groups for CIFAR-10, CelebA 64, LSUN Church 64 and CelebA HQ 256 are 30, 15, and 20, respectively.
NVAE:
Table 7: Network structures for the energy function Eng. (x)
CelebA HQ
LSUN Church 64
3 3 conv2d, 64
CelcbA 64
3 x 3 conv2d, 64
ResBlock down 64
CIFAR-10
x conv2d, 64
ResBlock down 64
ResBlock 64
3 conv2d, 128
ResBlock down 64
ResBlock 64
ResBlock down 128
ResBlock down 128
ResBlock 64
ResBlock down 128
ResBlock 128
ResBlock 128
ResBlock down 128
ResBlock 128
ResBlock down 128
ResBlock down 256
ResBlock 128
ResBlock 128
ResBlock 128
ResBlock 256
ResBlock down 128
ResBlock down 128
ResBlock down 256
ResBlock down 256
ResBlock 256
ResBlock 256
ResBlock 256
ResBlock 256
ResBlock down 256
ResBlock 256
ResBlock down 256
Global Sum Pooling
ResBlock 256
ResBlock down 256
ResBlock 256
FC layer scalar
Global Sum Pooling
ResBlock 256
ResBlock down 512
FC layer scalar
Global Sum Pooling
ResBlock 512
FC layer + scalar
Global Pooling
FC layer + scalar
Network for energy function: We largely adopt the energy network structure for CIFAR-10 in Mordatchl (2019), and increase the depth of the network for larger images. There 2 major we replace we found it improves training stability, and 2. we do not use spectral normalization (Miyato et al. (2018); instead, we use weight normalization with date-dependent initialization (Salimans & Kingma 2016). The network structure for each dataset is presented in T
differences between our energy networks and the ones used in Du & (oordatch((0199)
the LeakyReLU activations with Swish activations,
Training of energy function: We train the energy function by minimizing the negative log likeli-
hood and an additional spectral regularization loss which penalizes the spectral norm of each con- volutional layer in Exg. The spectral regularization loss also used training NVAE, as we found
httts::/gitthiie wwm/No/////// aaas/NVA
19

---

Published as a conference paper at ICLR 2021
it helpful regularize the sharpness of the energy network and better stabilize training. We use coefficient 0.2 for the spectral regularization loss.
Table 8: Important haparepasameters for training VAEBM
Dataset
Learning rate
Batch size
Persistent
# of L.D steps
L.D Step size
CIFAR-I0 w/o persistent chain
4e-5
32
No
10
8e-5
CIFAR-10 w/ persistent chain
4e-5
Yes
6
6e
CelebA 64
5e-5
32
No
10
5e 6
LSUN Church 64
4e-5
32
Yes
10
4e-6 6
CelebA HQ 256
4e
16
Yes
6
3e
We summarize some key hyper- prper-paramettrr we used to train VAEBM in Table
On all datasets, we train VAEBM using the Adam optimizer Kingma& & Ba, 2015) and weight decay 3c-5. We use constant learning rates, shown in Table Following
Mordatch (2019), we clip training gradients that are more than 3 standard deviations from the 2nd-order Adam parameters.
While persistent sampling using a sample replay buffer has little effect on CIFAR-10 10, we found to be useful on large images such as CelebA HQ 256. When we do not use persistent sampling, we always initialize the LD chains with (cx,62). sampled from standard Gaussian. When we use persistent sampling in training, we keep sample replay buffer that only stores samples of ez. while cx is always initialized from a standard Gaussian. The size of the replay buffer is 10,000 for CIFAR- 10 and LSUN Church 64, and 8,000 for CelebA HQ 256. At every training iteration, the MCMC chains on &, by drawing & from the replay buffer with probability p and from standard Gaussian with probability 0 to 0.6 in 5,000 training iterations, and for CelebA HQ 256, we linearly increase from 0 to 0.6 in 3,000 training iterations. The settings of Langevin dynamics are presented in Table g
initialize
p. For CIFAR-10 and LSUN Church 64, we linearly increase from
We do not explicitly set the number of training iterations. Instead, we follow Du & Mordatch (2119) to train the energy network until we cannot generate realistic samples anymore. This happens when the model overfits the training data and hence energies of negative samples are much larger than energies of training data. Typically, training takes around 25,000 iterations (or 16 epochs) on CIFAR-10, 10, 20,000 iterations (or epochs) on CelebA 64, 20,000 iterations (or 5 epochs) on LSUN Church 64, and 9,000 iterations (or epochs) on CelebA HQ 256.
Test time sampling: After training the model, we generate samples for evaluation by running Langvin dynamics with (cx,C) initialized from standard Gaussian, regardless of whether persis- tent sampling is used in training not. We run slightly longer LD chains than training to obtain the best sample quality. In particular, our reported values are obtained from running 16 steps of LD for CIFAR-10 10, 20 steps of LD for CelebA64 and LSUN Church 64, and 24 steps for CelebA HQ 256. The step sizes are the same as training step sizes.
In CelebA HQ 256 dataset, optionally use low temperature initialization for better visual quality. To do this, we first draw samples from the VAE with low temperature and readjusted the BN statistics introduced by Vahdat & Kautz (2020), and then initialize the MCMC chain by (ex, Eg) obtained by encoding the low-temperature samples using VAE's encoder without readjusted BN statistics.
Evaluation metries:
We use the official implementations of FID and I We compute IS using 50k CIFAR 10 samples, and we compute FID between 50k generated samples and training images, except for CelebA HQ 256 where we use 30k training images (the CelebA HQ dataset contains only 30k samples).
F SETTINGS FOR ABLATION STUDY
In this section, we present the details of ablation experiments in Sec. 5.2 Throughout ablation experiments, we smaller NVAE with 20 groups of latent variables trained on CIFAR-10. We use the same network architectures for the energy network as Table with potentially different
http:://gitthbb..cmm/eeea///////
20

---

Published as a conference paper at ICLR 2021
normalization techniques discussed below. We spent significant efforts on improving each method we compare against, and we report the settings that led to the best results.
WGAN initialized with NVAE decoder: We initialize the generator with the pre-trained NVAE decoder, and the discriminator is initialized by a CIFAR-10 10 energy network with random weights. We use spectral normalization and batch normalization in the discriminator we found them neces- sary for convergence. We update the discriminator using the Adam optimizer with constant learning rate 5e-5, and update the generator using the Adam optimizer with initial learning rate 5e-6 and cosine decay schedule. We train the generator and discriminator for 40k iterations, and we reach convergence of sample quality towards the end of training.
EBM on X, w/ or w/o initializing MCMC with NVAE samples: We train two EBMs on data space similar to Du & Mordatch (2019), where for one of them, we use the pre-trained NVAE to initialize the MCMC chains that draw samples during training. The setting for training these two EBMs are the same except for the initialization of MCMC. We use spectral normalization in the energy network and energy regularization in the training objective as done in Du & Mordatch (2019) because we found these modifications to improve performance. We train the energy function using the Adam optimizer with constant learning rate le-4. We train for 100k iterations, and we reach convergence of sample quality towards the end of training. During training, we draw samples from the model following the MCMC settings in Du & Mordatch (2019). In particular, we use persistent sampling and sample from the sample replay buffer with probability 0.95. We run 60 steps of Langevin dynamics to generate negative samples and clip gradients to have individual value magnitudes of less than 0.01. We use a step size of 10 for each step of Langevin dynamics. For test time sampling, we generate samples by running 150 steps of LD with the same settings as during training.
VAEBM with DKL (px)))) loss: We use the same network structure for Esg as in VAEBM. We find persistent sampling significantly hurts the performance in this case, possibly due to the fact that the decoder is updated and hence the initial samples from the decoder change throughout training. Therefore, we do not use persistent training. We train the energy function using the Adam optimizer with constant learning rate 5. We draw negative samples by running 10 steps of LD with step size 8e-5. We update the decoder with the gradient in Eq, using the Adam optimizer with initial learning rate 5e-6 and cosine decay schedule. For test time sampling, we run 15 steps of LD with step size 5c-6.
EB:::: The training of VAEBM in this section largely follows the settings described in Appendix We use the same energy network as for CIFAR-10, and train using the Adam optimizer with constant learning rate 5e-5. Again, we found that the performance of VAEBM with or without persistent sampling is similar. We adopt persistent sampling in this section because it is faster. The setting for the buffer is the same as in Appendix E We run 5 steps of LD with step size 5 during training, and 15 steps of LD with the same step size in testing.
G QUALITATIVE RESULTS OF ABLATION STUDY
In Figure 7, we show qualitative samples from models corresponding to each item in Table 4 as well as samples generated by VAEBM with additional (po(()))hh loss.
H ADDITIONAL QUALITATIVE RESULTS
We present additional qualitative results in this section.
Additional samples and visualizations of MCMC on CIFAR-10 are in Figures 8 and respectively.
Additional samples on CelebA 64 are in Figure 10
Additional samples LSUN Church 64 are in Figure 11 We visualize the effect of running MCMC
by displaying sample pairs before and after MCMC in Figure 12
Additional samples CelebA HQ 256 generated by initializing VAE samples with temperature 0.7 are shown in Figure [3] Samples generated by initializing VAE samples with full temperature 1.0 are shown in Figure 14 We visualize the effect of running MCMC by displaying sample pairs
21

---

Published as a conference paper at ICLR 2021
a NVAE baseline
(b) WGAN, initialized with NVAE decoder
(c) EBM x, MCMC initialized with NVAE sam- ples
(d) VAEBM with DEC. (xx) (ex., loss
(c) VAEBM
Figure 7: Qualitative results of ablation study in Sec. 5.2 and Appendix C
before and after MCMC in Figure [5 Note that the samples used to visualize MCMC are generated by initializing MCMC chains with VAE samples with full temperature 1.0.
22

---

Published as a conference paper at ICLR 2021
Figure 8: Additional CIFAR-10 samples
NEAREST NEIGHBORS
We show nearest neighbors in the training with generated samples on CIFAR-10 (in Figure I6 and (7] and CelebA HQ 256 (in Figure 18 and 19). We observe that the nearest neighbors are sigmificantly different from the samples, suggesting that our models generalize well.
SETTINGS OF SAMPLING SPEED EXPERIMENT
We use the official implementation and checkpoints of NCSN at https:: /ttt:///gthbb com/ ermongroup/ /ongo/ssu///n// /nssn We run the experiments on computer with a Titan RTX GPU, We use PyTorch 1.5.0 and CUDA 10.2.
23

---

Published as a conference paper at ICLR 2021
Figure 9: Additional visualizations of MCMC chains when sampling from the model for CIFAR-10
24

---

Published as a conference paper at ICLR 2021
Figure 10: Additional CclebA 64 samples
25

---

Published as a conference paper at ICLR 2021
Figure 1I: Additional LSUN Church 64 samples
26

---

Published as a conference paper at ICLR 2021
Figure 12:
Visualizing the effect of MCMC sampling on LSUN Church 64 dataset. For each subfig- ure, the top row contains initial samples from the VAE, and the bottom row contains corresponding samples after MCMC. We observe that MCMC sampling fixes the corrupted initial samples and refines the details.
27

---

Published as a conference paper at ICLR 2021
Figure 13: Additional CelebA HQ 256 samples. Initial samples from VAE for MCMC initializations generated with temperature 0.7. Samples are uncurated.
28

---

Published as a conference paper at ICLR 2021
RT
Figure 14: Additional CelebA HQ 256 samples. Initial samples from VAE for MCMC initializations generated with full temperature 1.0. Samples are uncurated.
29

---

Published as a conference paper at ICLR 2021
Figure 15: Visualizing the effect of MCMC sampling on CelebA HQ 256 dataset. Samples generated by initializing MCMC with full temperature VAE samples. MCMC sampling fixes the artifacts of VAE samples, especially on hairs.
30

---

Published as a conference paper at ICLR 2021
Figure 16: CIFAR-10 nearest neighbors in pixel distance. Generated samples are in the leftmost column, and training set nearest neighbors are in the remaining columns.
CIFAR-10 nearest neighbors in Inception feature distance. Generated samples in the leftmost column, and training set nearest neighbors are in the remaining columns.
Figure 17:
31

---

Published as a conference paper at ICLR 2021
35
2A
CelebA HQ 256 nearest neighbors in pixel distance, computed on a 160 x 160 center crop to focus more on faces rather than backgrounds. Generated samples are in the lefumost column, and training set nearest neighbors are in the remaining columns.
Figure 18:
32

---

Published as a conference paper at ICLR 2021
Figure 19: CelebA HQ 256 nearest neighbors in Inception feature distance, computed on a 160 160 center crop. Generated samples are in the leftmost column, and training set nearest neighbors are in the remaining columns.
33


Reviews:
[{"review_id": "mbeMO6foEWF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\nThe paper provides a thorough overview of recent work towards training EBMs.  \nThe approach generates high quality image samples by combining EBMs and VAE based models. \nThe paper is well written and is easy to follow \nI find it quite interesting that a combination of both models leads to significant overall improved generative performance \nI also enjoyed the proposed change in the paper -- and it seems to elegantly solve several problem in EBM training. \n\nWeaknesses:\nMy most major concern is that since we are utilizing a maximum likelihood objective to train models, it would be good to evaluate  the overall likelihood of the trained model, even if only in the  2D domain. \nThe histogram of likelihoods of data points is a bit disappointing -- it falls a similar trend of other EBM models, but it would nicer if it followed a Gaussian distribution \nWhat happens when more Langevin sampling steps are applied to the model? (greater than the few used in training) \nI'm also curious on what sampling only the trained energy model looks like (without using the trained VAE parameterization) at evaluation time \nI would also be curious to see how the trained EBM, with the VAE generator  can compose together with other models. See for example [1].\n\n[1] Yilun Du, Shuang Li, Igor Mordatch. Compositional Visual Generation and Inference with Energy Based Models. NeurIPS 2020\n\n#### Post Rebuttal-Update\n\nI thank the authors for responding to my concerns. I enjoyed reading the paper and maintain my rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SqmBjHN2yWi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nthis method proposed to use VAE+EBM for generative modelling. Unlike other VAE+GAN/EBM-liked model, it added a EBM after VAE.\nOverall method is easy to understand and follow.\nTo accelerate the training, the authors also applied a buffer to store the previous examples for easy sampling.\n\nCons:\nIn the experiment, the authors compared other models with VAEBM, it is reasonable to compare the results with reported scores in other works, however, since the architecture is a fairly important factor (such that swish instead relu, resblock instead of cnn, weight norm instead of spectral norm), etc, is it also reasonable that the improvement is partially contributed by such design of architecture.\nSo I will suggest that the authors should use the same architecture design (choose other one or two models for all tasks), and test whether the proposed method can actually gain that much of improvement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PeAZ5Vopb9P", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model that corrects VAE by an energy-based model defined on image space. The model is learned in two phase. The first phase learns the VAE model, while the second phase learns the EBM correction term by MLE. Experimental results show that the proposed method outperforms pure EBM defined on image space and also pure VAE models by large margins. \n\n- pros: the paper is clear written and easy to follow. The ablation study shows clearly the advantage over baseline methods. Sampling from EBM on image space is hard. With VAE as a backbone, the sampling can be transferred to the latent space and the residual \\epsilon in the image space, which is much more friendly to MCMC sampling. \n\n- cons:\n1. The energy term is used to correct only on image space. Would be interesting to see if VAE can be corrected by a latent EBM where the energy function is defined on (x, z). \n2. After learning, would long-run MCMC sampling chain remain stable and mix well? It would be interesting to diagnose the long run chain behavior, and compare the difference of sampling in the space (\\epsilon_x, \\epsilon_z) and (x, z). \n3. For the synthesized results of CIFAR-10, it seems that some patterns appear repeatedly (e.g., the white dog face). Is the model suffered from mode collapsing problem? \n\nOverall, it is a good submission that proposes a principled method to combine VAE and EBM and demonstrates strong empirical results. I tend to accept this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ARQ0Ecwt8ci", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a generative model that is a combination (product) of a VAE and an EBM, where the goal of the EBM is to reduce the probability of out-of-manifold samples, which are typically generated by VAEs. The authors propose efficient training and sampling procedures, in which the VAE is trained first and during the EBM negative-phase, samples are drawn from the joint (x, z) VAE space using reparameterization. The method is shown to achieve high quality samples on several modern image datasets, good FID scores and mode coverage. Ablation studies show the contribution of the different elements.\n\nThis is, in my opinion, a very good work, which combines a novel and well-motivated idea with clear writing and extensive experimental evidence.\n\nSome comments and questions:\n- Does the separate twos-stage training enable the model to reach the optimal point that can be reached in joint training, or is it an approximation? If its an approximation, I think it should be discussed or perhaps bounded.\n- Does the combined model allow computing the likelihood? Can it be evaluated and compared to other models in terms of bits/dimension (e.g. as in VAE or NVAE)?\n- It might be interesting (not something that I think is mandatory) to measure the NVAE log-likelihood of samples generated by the combined model compared to samples generated just by the NVAE.\n\nTo summarize:\npros:\n- novelty\n- significance\n- experimental evidence\n- quality of writing\n\ncons:\n- combining two separately trained models - perhaps sub-optimal\n\n**Update: I thanks the authors for their answers and revised version and keep my positive rating.**\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a method to combine EBMs and VAEs in two stages. First, the VAE model is learned; second, an EBM-based correction term is learned via MLE. The methodology is novel and of interest to the ICLR community.
Decision: Accept (Spotlight)