Learning Energy-Based Models by Diffusion Recovery Likelihood
Authors: Ruiqi Gao
Yang Song
Ben Poole
Ying Nian Wu
Diederik P Kingma
Venue: ICLR 2021 Poster (2021)
Keywords: energy-based model
EBM
recovery likelihood
generative model
diffusion process
MCMC
Langevin dynamics
HMC

Abstract: While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset. Each EBM is trained with recovery likelihood,  which maximizes the conditional probability of the data at a certain noise level given their noisy versions at a higher noise level. Optimizing recovery likelihood is more tractable than marginal likelihood, as sampling from the conditional distributions is much easier than sampling from the marginal distributions. After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution and progressively samples the conditional distributions at decreasingly lower noise levels.  Our method generates high fidelity samples on various image datasets. On unconditional CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs. Moreover, we demonstrate that unlike previous work on EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us to accurately estimate the normalized density of data even for high-dimensional datasets. Our implementation is available at \url{https://github.com/ruiqigao/recovery_likelihood}.


Published as a conference paper at ICLR 2021
LEARNING ENERGY-BASED MODELS BY DIFFUSION RECOVERY LIKELIHOOD
Ruiqi Gao
Yang Song Stanford University
Ben Poole
UCLA
Google Brain
ruiqigacEucla.ed edu
yangsengees
edu stanford. stw.aangssssssssoo...oooo
poolebegoogle.. com
Ying Nian Wu
Diederik P. Kingma Google Brain
UCLA ywu@stat. www.stttt.clllee...
durk@google..cm com
ABSTRACT
While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on diffusion probabilistic models, we present a diffusion re- covery likelihood method to tractably learn and sample from sequence of EBMs trained on increasingly noisy versions of a dataset. Each EBM is trained with recovery likelihood, which maximizes the conditional probability of the data certain noise level given their noisy versions at higher noise level. Optimizing re- covery likelihood is more tractable than marginal likelihood. as sampling from the conditional distributions is much easier than sampling from the marginal distribu- tions. After training. synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution and progressively samples the conditional distributions at decreasingly lower noise levels. Our method gener- ates high fidelity samples on various image datasets. On unconditional CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs. Morcover, we demonstrate that unlike previous work EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us accurately estimate the normalized density of data even for high-dimensional datasets. Our implementation avail- able at htthtts::/// https:://gitttbb http:/.booomeeee httpm//riqigao/oree www.oeer.ree.e.........ooo.o Likeliiie hood
INTRODUCTION
EBMs LeCun et al. 2006 N Ngiam al. 2011 Kim & Bengio 2016,Zhao Zhao et al. 2016 Goyal al.
2017 Xie et al.] [016bb Finn et al. 2016 Gao et al. [2018] Kumar et al. 2019 Nij kamp et al.
2019b Du & Mordatch 2019 Grathwohl et al. 2019 Desjardins al. 2011 Gao
2020 Che
eet al. 2020; Grathwohl et al. 2020 Qiu et al. 2019 Rhodes et al. 2020) are an appealing class of
probabilistic models, which can be viewed as generative versions of discriminators (Jin al. 2017
Lazarow et al. 2017 Lee et al.) 2018 Grathwohl et al. 2020), yet can be learned from unlabeled data. Despite number of desirable properties, two challenges remain for training EBMs on high- dimensional datasets. First, learning EBMs by maximum likelihood requires Markov Chain Monte Carlo (MCMC) generate samples from the model, which can be extremely expensive. Second, as pointed out in Nijkamp et al. (2019a). the energy potentials learned with non-convergent MCMC do not have valid steady-state, the sense that samples from long-run Markov chains can differ greatly from observed samples, making it difficult to evaluate the learned energy potentials.
Another line of work, originating from Sohl-Dickstein et al. 2015). is to learn from a diffused version of the data, which are obtained from the original data via diffusion process that sequentially adds Gaussian white noise. From such diffusion data, one can learn the conditional model the data at certain noise level given their noisy versions at the higher noise level of the diffusion process. After learning the sequence of conditional models that invert the diffusion process, one can then generate synthesized images from Gaussian white noise images by ancestral sampling. Building on

---

Published as a conference paper at ICLR 2021
Figure 1: Generated samples on LSUN 128 church_outdoor (left), LSUN 128² bedroom (center) and CelebA 64 (right).
Sohl-Dickstein et al. (2015) Ho et al. (2020) further developed the method, obtaining strong image synthesis results.
Inspired by Sohl-Dickstenn lihood method to tackle the challenge of training EBMs directly on a dataset by instead learning a sequence of EBMs for the marginal distributions of the diffusion process. The sequence of marginal EBMs are learned with recovery likelihoods that are defined as the conditional distributions that in- vert the diffusion process. Compared standard maximum likelihood estimation (MLE) of EBMs, learning marginal EBMs by diffusion recovery likelihood only requires sampling from the condi- tional distributions, which much easier than sampling from the marginal distributions. After learn- ing the marginal EBMs, we can generate synthesized images by a sequence of conditional samples initialized from the Gaussian white noise distribution. Unlike Ho al. (2020) that approximates the reverse process by normal distributions, in our case the conditional distributions are derived from the marginal EBMs, which are more flexible. The framework of recovery likelihood was originally proposed in Bengio al. (2013). In our work, we adapt it to learning the sequence of marginal EBMs from the diffusion data.
al. (2015) and Ho ct al. (2020). propose diffusion recovery like-
Our work is also related to the denoising score matching method of Vincent (2011). which was further developed by Song & Ermon (2019 2020) for learning from diffusion data. The training ob- jective used for diffusion probabilisitic models weighted version of the denoising score matching objective, as revealed by Ho et al. (2020). These methods learn the score functions (the gradients of the energy functions) directly, instead of using the gradients of learned energy functions in EBMs. On the other hand, Saremi et al. (2018) parametrizes the score function as the gradient of a energy function, and Saremi & Hyvarinen (0199 further unifies denoising score matching and neural empirical Bayes
We demonstrate the efficacy of diffusion recovery likelihood on CIFAR-I0, CelebA and LSUN datasets. The generated samples are of high fidelity and comparable to GAN-based methods. On CIFAR-10, we achieve FID 9.58 and inception score 8.30, exceeding existing methods of learning explicit EBMs to large extent. We also demonstrate that diffusion recovery likelihood outperforms denoising score matching from diffusion data if we naively take the gradients explicit energy functions as the score functions. More interestingly, by using thousand diffusion time steps, we demonstrate that even very long MCMC chains from the sequence of conditional distributions pro- duce samples that represent realistic images. With the faithful long-run MCMC samples from the conditional distributions, we can accurately estimate the marginal partition function at zero noise level by importance sampling, and thus evaluate the normalized density of data under the EBM.
2

---

Published as a conference paper at ICLR 2021
t=0 0
t=1
t=2
t=3
t=4
t=
Synthesized samples
Estimated density
Figure 3: Illustration of diffusion recovery likelihood on 2D checkerboard example. Top: progres- sively generated samples. Bottom: estimated marginal densities.
BACKGROUND
Let Pdata (x) denote training example, and pe px(x) denote model's probability density func- tion that aims to approximates (x). An energy-based model (EBM) is defined as:
1 ex)(Jo(x)),
(x
Zo
where Zo = exp((f(() (x) is the partition function, which is analytically intractable for high- dimensional x. For images, we parameterize fo(x) with convolutional neural network with scalar output.
The mnergy-based model in equation I can, in principle, be learned through MLE. Specifically, suppose we observe samples Pdata (x) for = 1,2, 2...,, n. The log-likelihood function is
n log (x;) - Ex [og po()).
(
In MLE, we seek to maximize the log-likelihood function, where the gradient approximately fol- lows (Xie et al. 2016b)
a ( f0(())
DKL ( (Pdata lppe) Excepaaa
30 5o(3))
Ex-upo
30
The expectations can be approximated by averaging over the observed samples and the synthesized samples drawn from the model distribution po(x) respectively. Generating synthesized samples from pa (x) can be done with Markov Chain Monte Carlo (MCMC) such as Langevin dynamics (or Hamiltonian Monte Carlo (Girolami & Calderhead 201I)), which iterates
52 x = Vxfo(x") + "E",
(4)
where indexes the time, is the step size, and e" ~ N(O, I). The difficulty lies in the fact that for high- dimensional and multi-modal distributions, MCMC sam- pling can take long time to converge, and the sampling chains may have difficulty traversing modes. As demon- strated in Figure 2 training EBMs with synthesized sam- ples from non-convergent MCMC results in malformed energy landscapes (Nijkamp et 2019b), even if the sam- ples from the model look reasonable.
Data
Ours
Short-run
Qurs
Shart run
Synthesized samples
Estimated density
RECOVERY LIKELIHOOD
Figure 2:
Comparison of learning
EBMs by diffusion recovery likeli- hood (Ours) versus marginal likelihood (Short-run).
3.1 FROM MARGINAL TO CONDITIONAL.
Given the difficulty sampling from the marginal density
(x),,, following Bengio al. (003)), the recovery likelihood defined by the density of the
3

---

Published as a conference paper at ICLR 2021
observed sample conditional noisy sample perturbed by isotropic Gaussian noise. Specifically, let X = x + de be the noisy observation x, where < ~~ (0, I). Suppose pe is defined by the
EBM in equation [ then the conditional EBM can be derived
po(x =
(x) exp [fo(x)
20 x x((
(5)
where (x) f exp (fo(x) 20 | x x dx is the partition function of this conditional EBM. See Appendix A. for the derivation. Compared to px(x) (equation []) the extra quadratic term 27 | - x|| in pal x x) constrains the energy landscape to be localized around around x, making the latter less multi-modal and easier to sample from. As we will show later, when is small, pe (x X) is approximately a single mode Gaussian distribution, which greatly reduces the burden of MCMC.
A more general formulation is x + σe, where is positive constant. In that case, we can let y = ax, and treat y as the observed sample. Assume pe Py) = I % exp(foly)), then by change of variable, the density function of x can be derived as ye (x) = (ax).
3.2 MAXIMIZING RECOVERY LIKELIHOOD
With the conditional EBM, assume we have observed samples x; Pdata (x) and the corresponding perturbed samples X; = X; + oe; for i = 1,..., n. We define the recovery log-likelihood function as
[(0)
log (x,
The term recovery indicates that we attempt to recover the clean sample x; from the noisy sample X,. Thus, instead of maximizing [(0) in equation 2 are easier to sample from. Specifically, we generate synthesized samples by K steps of Langevin dynamics that iterates
we can maximize I(0), whose distributions
82 x =x -Vx((((x'))) (x + OE".
The model is then updated following the same learning gradients MLE (cquation 3). because the _ || - x|2 is not related to 0. Following the classical analysis of MLE, we can
quadratic term 2 show that the point estimate given by maximizing recovery likelihood is unbiased estimator of the true parameters, which means that given enough data, a rich enough model and exact synthesis, maximizing the recovery likelihood learns 0 such that Pdata (x) (x))). See Appendix A.2 for theoretical explanation.
3.3 NORMAL APPROXIMATION TO CONDITIONAL
When the variance of perturbed noise o² is small, (x) can be approximated by normal distri- bution via a first order Taylor expansion at &. Specifically, the negative conditional energy is
[x) = fo(x)
20 x - x|2
(8)
fx() + (Vxfo(x), x x)
20 |
x||2
(9)
20t ||x (x
2 Vxf +c
(10)
where e include terms irrelevant of x (see Appendix A.3 for a detailed derivation). In the above approximation, we do not perform second order Taylor expansion because o² is small, and || xl|2/2o0 will dominate all the second order terms from Taylor expansion. Thus we can approximate (x(x by a Gaussian approximation Po (x(x):
(x N (x:
V Vxfo(xx)
(11)
We can sample from this distribution using:
Xgen = x + V fo(() + OE,
(12)
where (0, 1). This resembles single step of Langevin dynamics, except that ae is replaced
by V2re in Langevin dynamics. This normal approximation has two traits: (1) it verifies the fact that the conditional density po (x(x) can be generally easier to sample from when or is small; (2)
provides hints of choosing the step size of Langevin dynamics, discussed in section 3.5
4

---

Published as a conference paper at ICLR 2021
3.4 CONNECTION TO VARIATIONAL INFERENCE AND SCORE MATCHING
The normal approximation to the conditional distribution leads to a natural connection to diffu-
sion probabilistic models (Sohl-Dickstein al., 2013 Ho al. 2020) and denoising score match-
ing (Vincent 201T, Song & Ermon 2019) 2020]] Saremi et al. 2018 Saremi & Hyvarinen 2019).
Specifically, diffusion probabilistic models, instead of modeling pe (4) as an energy-based model,
it recruits variational inference and directly models the conditional density as
pe (x ~ .V (x + sa (x))
(13)
which is in agreement with the normal approximation (equation with sp(x) Vxfa(x). On the other hand, the training objective of denoising score matching is to minimize
20 (x [||x (x + o s0(())]
(14)
where sx(x) is the score of the density of x. This objective is in agreement with the objective of maximizing log-likelihood of the normal approximation (equation 10). except that for normal ap- proximation, (-) .) is the score of density of x, instead of x. However, the difference between the scores of density of and & is of O(o²), which is negligible when is sufficiently small (see Ap- Lendix[A.4]fee details). We can further show that the learning gradient of maximizing oog-likeihhod of the normal approximation approximately the same the learning gradient of maximizing the original recovery log-likelihood with one step of Langevin dynamics (see Appendix (A.3). It indi- cates that the training process of maximizing recovery likelihood agrees with the one of diffusion probabilistic models and denoising score matching when a is small.
As the normal approximation is accurate only when small, requires many time steps in the diffusion process for this approximation to work well, which is also reported in Ho et al. (2020) and [ong & Ermon 22220). In contrast, the diffusion recovery likelihood framework can be more flexible in choosing the number of time steps and the magnitude of a.
3.5 DIFFUSION RECOVERY LIKELIHOOD
As we discuss, sampling from po (x x) becomes simple only when o is small. the extreme case when - oc, po (x{x) converges to the marginal distribution px(x), which is again highly multi- modal and difficult to sample from. To keep a small and meanwhile equip the model with the abil- ity to generate new samples initialized from white noise, inspired by SSohl-Dickstein et al. (2015) and Ho et al.] 2220), we propose to learn a sequence of recovery likelihoods, on gradually per- turbed observed data based on diffusion process. Specifically, assume sequence of perturbed observations Xg, X1...., Xr such that
Xo Pdata (x); Xi+1 V - 04411141, 0,
(15)
The scaling factor / 11-0241 ensures that the sequence is a spherical interpolation between the / 1241X,, and we assume sequence of observed sample and Gaussian white noise. Let conditional EBMs
1 exp (Telyr.1) 207-1
1 v.ll?) 1= 0,0,,,.,,TT (16)
Pa (y) (x.1()
|x++1
where fo(y.,t) is defined by a neural network conditioned on
A question is how to determine the step size schedule d, of Langevin dynamics. Inspired by the sampling procedure of the normal approximation (equation 12]. we set the step size &, = bo,, where b < is tuned hyperparameter. This schedule turns out to work well in practice. Thus the steps of Langevin dynamics iterates
We follow the learning algorithm in section 3.2
1ooo 2 (VyJo(yT,1) + o7 (x yE)) + bo,e".
y! y +
(17)
Algorithm summarizes the training procedure. After training, we initialize the MCMC sampling from Gaussian white noise, and the synthesized sample each time step serves initialize the MCMC that samples from the model of the previous time step. See algorithm To show the efficacy of our method, Figures and 2 display several 2D toy examples learned by diffusion recovery likelihood.
5

---

Published as a conference paper at ICLR 2021
Algorithm Training
Algorithm Progressive sampling
repeat
Sample XT (0, for + T - to do
Sample ~ Unif(
1)).
Sample pairs (yt, X++1).
Yt Xt+1
Set synthesized sample yi = X++1-
for T + 1 to K do
for < to K do
Update yt according to equation 17 end for
Update y, according to equation 17
end for
== 0241-
Update 0 following the gradients & 00 fo (yr,)) e fy.
end for return Xo-
until converged.
EXPERIMENTS
To show that diffusion recovery likelihood is flexible for diffusion process of various magnitudes of noise, test the method under two settings: (1) T 6, with K = 30 steps of Langevin dynamic per time step: (2) T 1000, with sampling from normal approximation. (2) resembles the noise schedule of [o al. 2020 and the magnitude of noise added each time step much smaller compared to (1). For both settings, we set of to increase linearly. The network structure of fe(x, t) based on Wide ResNet (Zagorykko & Komodakis (016)) and we remove weight normalization. is encoded by Transformer sinusoidal position embedding as in (Ho et al.) (2220). For (1), we find that adding another scaling factor e, to the step size 6, helps. Architecture and training details are in Appendix B Henceforth we simply refer the two settings as 76 and 77k.
4.1 IMAGE GENERATION
Figures D and 4 display uncurated samples generated from learned models on CIFAR-10, CelebA
64 64, LSUN 64 x 64 and 128 128 datasets under T6 setting. The samples are of high fidelity
and comparable to GAN-based methods. Appendix C.5 provides more generated samples. Tables
1 and 3 summarize the quantitative evaluations on CIFAR-TO and CelebA datasets, in terms of
Frechet Inception Distance (FID) (Heusel et al.) (017)) and inception scores (Salimans et al.) 20166]
On CIFAR-IO, our model achieves FID 9.58 and inception score 8.30, which outperforms existing methods of learning explicit energy-based models to a large extent, and is superior to majority of
GAN-based methods. On CelebA our model obtains results comparable with the state-of-the-ar
GAN-based methods, and outperforms based methods (Song & Ermon 2019 2020 Note that the score- based methods (Song & Ermon 2019: 2020) and diftusion probabilistic models (Ho)
al. 2020) directly parametrize and learn the of data distribution, whereas our goal is to learn explicit energy-based models.
Figure 4: Generated samples unconditional CIFAR-10 (left) and LSUN 642 church.outdoor (cen- ter) and LSUN 64 bedroom (right).

---

1282 (left) and CelebA 64 (right). With each block, Published as a conference paper at ICLR 2021
Table 1: FID and inception scores CIFAR-10.
Table 2: Ablation of training objectives, time steps T and sampling steps K on CIFAR-IO. K 0 indicates that we sam- ple from the normal approximation.
Model
FID, Inception
GAN-based
WGAN-GP (Gulrajani el al.
2017
36.4
7.86 1 .07
Setting Objective
FID
Inceptiont
SNGAN (Miyato et al. 2018
21.7
8.22
- .05
T= k 180
32.12
6.72 0.12
SNGAN- DDLS Che et al.
2020
15,42
9.09
= .10
T= 1000, 0
22.58
7.71 + 0.08
StyleGAN2-ADA Karras
2020
3.26
9.74
.05
T= 1000, K = 0 (DSM)
21.76
7.76 + 0.11
Score-based
T 6, K 10
= 6. K 30
9.58
8.30 0.11
NCSN (Song & Ermon 2019
25.32
8.87
+ .12
= 50
9.36
8.46 + 0.13
NCSN-v2 [Song Ermon 2020
10.87
8.40 + .07
DDPM Ho et al. 2020
3.17
9.46
Explicit EBM-conditional
Table 3: FID scores on CelebA 642.
CoopNets Xic ct al.
2019
7.30
Model
FID
EBM-IG (Du & Mordatch
2019
37.9
8.30
QA-GAN (Parimala & Channappayya
2019
6.42
JEM GGrthhohl et al.
38.4
8.76
COCO- GAN (Lin el al.
2019
4.0
Explicit EBM
NVAE Vahdat & Kautz 2020
14.74
Muli-grid
Gao al. 2018
40.01
6.56
NCSN Song & Ermon 2019
25.30
CoopNets
Xie et al. 20
33.61
6.55
NCSN-VZ Song & Ermon
2020
10.23
EBM-SR SR
Nijkamp et al
2019b
6.21
FBM-SR (Nijkamp al. 201 9h
23.02
EBM-IG IG Du & Mordatch
2019
38.2
6.78
FBM-Triangee Triangle (Han al
FBM-
2020
24.70
Ours (76)
9.58
8.30 + .11
Ours (76)
5.98
Figure S: Interpolation results between the leftmost and rightmost generated samples. For top to
bottom: LSUN church_outdon 128 LSUN bedroom 128 and CelebA 64².
Table 4: Test bits per dimension CIFAR- 10. indicates that we estimate the bit per dimension with the approximated log partition function instead of analytically computing it. See section 4.2
Model
BPD|
DDPM (Ho al. 2020
3.70
Glow (Kingma Dhariwal 2018
3.35
Flow4+ (Ho el al.
2019)
3.08
GPixeICNN Van den Oord et al
2016
3.03
Figure 6: Image inpainting on LSUN church.outdoor
Sparse Transformer (Child et al.
2019
2,80
DistAug Juun al.
2020
2.56
the top row mask images while the bottom are inpainted images.
Ours' (77k)
3.18
As shown in Figure 5 our model is capable of smooth interpolation between two generated samples. Specifically, for two samples (0) and x we do sphere interpolation between
Interpolation.

---

Published as a conference paper at ICLR 2021
(0) the initial white noise images XT. and XT and the noise terms of Langevin dynamics E,r and
(0)
(1)
for every sampling step at every time step. More interpolation results can be found in Appendix C.3
Image inpainting. A promising application of energy-based models is to use the learned model as a prior model for image processing, such image inpainting, denoising and super-resolution (Gao et al. 2018: Du & Mordatch 2019 Song & Ermon 2019). In Figure 6 we demonstrate that the learned models by maximizing recovery likelihoods are capable of realistic and semantically meaningful image inpainting. Specifically, given a masked image and the corresponding mask, we first obtain a sequence of perturbed masked images at different noise levels. The inpainting can be easily achieved by running Langevin dynamics progressively on the masked pixels while keeping the observed pixels fixed at decreasingly lower noise levels. Additional image inpainting results can be found in Appendix C.4
Ablation study. Table 2 summarizes the results of ablation study on CIFAR-10. We investigate the effect of changing the numbers of time steps T and sampling steps K. First, to show that it is beneficial to learn by diffusion recovery likelihood, we compare against a baseline approach use only one time step, that the recovery likelihood becomes
(T 1, K 180) where marginal likelihood. The approach is adopted by Nijjiaammp et al. (2019b) and & Mordatch (2019). For fair comparison, we equip the baseline method the same budget of MCMC sampling setting (i.e.. 180 sampling steps). Our method outperforms this baseline method by a large margin. Also the models are trained more efficiently as the number of sampling steps per iteration is reduced and amortized by time steps.
our T6
Next, we report the sample quality of setting Tik. We test two training objectives for this setting: (1) maximizing recovery likelihoods (T = 1000, K = 0) and (2) maximizing the approximated normal distributions (T=1000, K=0 (DSM)). As mentioned in section 3.4 (2) is equivalent to the training objectives of denoising score matching (Song & Ermon 2019 2020) and diffusion probabilistic model (Ho et al. 2020), except that the score functions are taken as the gradients of explicit energy functions. In practice, for direct comparison, (2) follows the same implementation as in Ho et al. (2020), except that the score function is parametrized as the gradients of the explicit energy function used in method. (I) and (2) achieve similar sample quality in terms of quantitative metries, where (2) results in a slightly better FID score yet a slightly worse inception score. This verifies the fact that the training objectives of (1) and (2) are consistent. Both (1) and (2) performs worse than setting T6. A possible explanation that the sampling error may accumulate with many time steps, so that more flexible schedule of time steps accompanied with certain amount of sampling steps is preferred.
Last, we examine the influence of varying the number of sampling steps while fixing the number of time steps. The training becomes unstable when the number of sampling steps are not enough (T' = 6, K 10), and more sampling steps lead to better sample quality. However, since K 50 does not gain significant improvement versus K = 30, yet of much higher computational cost, we keep K- = 30 for image generation on all datasets. See Appendix C. for a plot of FID scores over iterations.
4.2 LONG-RUN CHAIN ANALYSIS
Besides achieving high quality generation, a perhaps equally important aspect of learning EBMs is to obtain faithful energy potential. A principle way to check the validity of the learned potential is to perform long-run sampling chains and see if the samples still remain realistic. However, as pointed out in Nijkamp et al. (2019a). almost all existing methods of learning EBMs fail in getting realistic long-run chain samples. In this subsection, we demonstrate that by composing a thousand diffusion time steps (Tlk setting), we can form steady long-run MCMC chains for the conditional distributions.
First we prepare a faithful sampler for conducting long-run sampling. Specifically, after training the model under TlA setting by maximizing diffusion recovery likelihood, for each time step, we first sample from the normal approximation and count it as one sampling step. and then use Hamiltonian Monte Carlo (HMC) (Neal al. 2011) with 2 leapfrog steps to perform the consecutive sampling steps. To obtain a reasonable schedule of sampling step size, for each time step we adaptively adjust the step size of HMC to make the average acceptance rate range in [0.6, 0.9), which is computed
8

---

Published as a conference paper at ICLR 2021
3:45
5 0.80
0222
3333
2000
0.70
3289
0280
020
Time
Number
Figure 7: Left: Adjusted step size of HMC over time step. Center: Acceptance rate over time step. Right: Estimated log partition function over number of samples with different number of sampling steps per time step. The axis plotted in log scale.
over 1000 chains for 100 steps. Figure displays the adjusted step size (left) and acceptance rate (center) over time step. The adjusted step size increases logarithmically. With this step size schedule, generate long-run chains from the learned sequence of conditional distributions. As shown in Figure images remain realistic even 100k sampling steps total (i.e., 100 sampling steps per time step), resulting in FID 24.89. This score is close to the one computed samples generated by 1k steps (i.c., sampled from normal approximation), which is 25.12. As a further check, we recruit No-U-Tumn Sampler (Hoffman & Gelman 2014) with the same step size schedule as HMC to perform long-run sampling, where the samples also remain realistic. See Appendix C.2 for details.
More interestingly, given the faithful long-run MCMC sam- ples from the conditional distributions, we can estimate the log ratio of the partition functions of the marginal dis- tributions, and further estimate the partition function of (y()). The strategy is based on annealed importance sam- pling (Neal, (001). See Appendix |A66 for the implementa- tion details. The right subfigure of Figure 7 depicts the es- timated log partition function of po (yo) over the number of MCMC samples used. To verify the estimation strategy and again check the long-run chain samples, we conduct mul- tiple runs using samples generated with different numbers of HMC steps and display the estimation curves. All the curves saturate to values close to each other at the end, indicating the stability of long-run chain sam- ples and the effectiveness of the estimation strategy. With the estimated partition function, by change of variable, we can estimate the normalized density of data as (xo) = /1 ofpu( / of xo). We report test bits per dimension on CIFAR-10 in Table Note that the result should be taken with a grain of salt, because the partition function is estimated by samples and as shown in Appendix A.6 it is stochastic lower bound of the true value, that will converge to the true value when the number samples grows large.
Figure 8: Long-run chain samples from model-TIk with different total amount of HMC steps. From left to right: 1K steps, 10k steps and 100k: steps.
CONCLUSION
We propose to learn EBMs by diffusion recovery likelihood, variant of MLE applied to diffusion processes. We achieve high quality image synthesis, and with a thousand noise levels, we obtain faithful long-run MCMC samples that indicate the validity of the learned energy potentials. Since this method can learn EBMs efficiently with small budget of MCMC, we are also interested in scaling up to higher resolution images and investigating this method in other data modalities in the future.
ACKNOWLEDGEMENT
The work was done while Ruiqi Gao and Yang Song were interns at Google Brain during the summer of 2020. The work of Ying Nian Wu is supported by NSF DMS-2015577. We thank Alexander A. Alemi, Jonathan Ho, Tim Salimans and Kevin Murphy for their insightful discussions during the course of this project.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In Advances in neural information processing systems, pp. 899-907, 2013.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.0644 2020.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. Xrviv preprint 1904: 10509, 2019.
Guillaume Desjardins, Yoshua Bengio, and Aaron Courville. On tracking the partition function. In Advances in neural information processing systems, pp. 2501-2509, 2011.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy- -based models. arXiv preprint DeXivv9903 08688,, 2019.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between generative
adversarial networks, inverse reinforcement learning, and energy- baseg models. arXiv preprint 2016.
Ruiqi Gao, Yang Lu, Junpei Zhou, Cong-Chur Zhu, and Ying Nian Wu. Learning generative con- vnets via multi-grid modeling and sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9155-9164, 2018.
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai. and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7518-7528, 2020.
Mark Girolami and Ben Calderhead. Riemann manifold langevin and hamiltonian monte carlo methods. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2): 123-214, 2011.
Anirudh Goyal Alias Parth Goyal, Nan Rosemary Ke, Surya Ganguli, and Yoshua Bengio. Vari- ational walkback: Learning a transition operator as stochastic recurrent net. In Advances in Neural Information Processing Systems, pp. 4392 4402, 2017.
Will Grathwohl, Kuan-Chich Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky, Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv: 0233223, 2019.
Will Grathwohl, Kuan-Chieh Wang, Jorn-Henri Jacobsen, David Duvenaud, and Richard Zemel. Cutting out the min: Training and evaluating energy-based models without sampling. arXiv preprint arXiv:2002.05611 00516600..666...... 2020.
Roger B Grosse, Siddharth Ancha, and Daniel M Roy. Measuring the reliability of meme inference with bidirectional monte carlo. In Advances in Neural Information Processing Systems, pp. 2451- 2459, 2016
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.
Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7978-7987, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: based generative models with variational dequantization and architecture design. arXiv preprint arXiv: 1902. 2019.
Improving flow-
10

---

Published as a conference paper at ICLR 2021
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 11239, 2020.
Matthew D Hoffman and Andrew Gelman. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Much. Learn. Res., 15(1):1593-1623, 2014.
Long Jin, Justin Lazarow, and Zhuowen Tu. Introspective classification with convolutional nets. In Advances in Neural Information Processing Systems, pp. 823-833, 2017.
Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Distribution augmentation for generative modeling. In Proceedings of Machine Learn- ing and Systems 2020, pp. 10563-10576. 2020.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06677. 2020.
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probability estimation. arXiv preprint 1606: 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances Neural Information Processing Systems, pp. 10215-10224, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based based models. arXiv preprint arXiv:/90. /9001098000 2019.
Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2774-2783, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 10), 2006.
Kwonjoon Lee, Weijian Xu, Fan Fan, and Zhuowen Tu. Wasserstein introspective neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3702- 2711,2018.
Chich Hubert Lin, Chia-Chrr Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann- Tzong Chen. Coco-gan: generation by parts via conditional coordinating. In Proceedings of the IEEE International Conference Computer Vision, pp. 4512-4521 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba)
dataset. Retrieved August, 15:2018, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 180200597 2018.
Radford M Neal. Annealed importance sampling. Statistics and computing, 1(2)::25-139, 2001.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Jiquan Ngiam, Zhenghao Chen, Pang W Koh, and Andrew Y Ng. Learning deep energy models. In Proceedings of the 28th international conference on machine learning (ICML-II), pp. 1105- 1112, 2011.
Erik Nijkamp, Mitch Hill, Tian Han, Gong-Chun Zhu, and Ying Nian Wu. On the anatomy of meme- based maximum likelihood learning of energy-baud based models. arXiv preprint arXiv: 1903:1900.1233 2019a.
11

---

Published as a conference paper at ICLR 2021
Erik Nijkamp, Mitch Hill, Song-Chun run meme toward energy-based model. arXiv preprint wwXiv:1904.09770 2019b.
Chun Zhu, and Ying Nian Wu. On learning non-convergent short-
KANCHARLA Parimala and Sumohana and Channappayya. Quality aware generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 2948-2958, 2019.
Yixuan Qiu, Lingsong Zhang, and Xiao Wang. Unbiased contrastive divergence algorithm for train- ing energy-based latent variable models. In International Conference on Learning Representa- tions, 2019.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Ad- vances in Neural Information Processing Systems, 33, 2020.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Saeed Saremi and Aapo Hyvarinen. Neural empirical bayes. Journal of Machine Leurning Research, 20:1-23, 2019.
Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf. and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprint 1rXiv:1805.083366 2018.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: 503.03585 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-11930, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.0001 09011,, 2020.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et Con- ditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 2367):1661-100 2011.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. arXiv preprint arXiv:1609.09400 2016a.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. (635-2644, 2016b.
Jianwen Xie, Zilong Zheng, Xiaolin Fang, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of fast thinking initializer and slow thinking solver for multi-modal conditional learning. arXiv preprint ar Xrv: 1902.02822. 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint
03365,, 2015.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv: 1609:0160.003/26 2016.
12

---

Published as a conference paper at ICLR 2021
A EXTENDED DERIVATIONS
A.1 DERIVATION OF EQUATION 5
Let x = xx+ oe, where ((0, I). Given the marginal distribution of
1
(x(x)
(x)()o(x)),
(18)
7
We can derive the conditional distribution of given x as Po xx{x = po po(x)p(x(xx)/1(()
(19)
1
Ze (x))
(2TO2) $ exp(
| //(3)
(20)
20
cxp
fo(xx)
202 11 x1(
1
(21)
(x)
where we absorb all the terms that are irrelevant of as (x).
A.2 THEORETICAL UNDERSTANDING
In this subsection, we analyze the asymptotic behavior of maximizing the recovery log- ligllikeihhod.
For model class (x),x), v0}, suppose there exists 6* such that Pdata Pe- According to the classical theory of MLE, let d be the point estimate by MLE. Then we have 0 is an unbiased estimator of 8" with asymptotic normality:
vn((g - 0*) + (0, (N, Io)
(22)
where Io) = Ex-pul - v log [x is the Fisher information, and is the number of observed samples.
Let Ô be the point estimate given by maximizing recovery log-likelihood, we can derive a result in parallel to that of MLE:
Vn( - 0-) (N,
(23)
where I(0) (x.
V log (x|x)]. The relationship between 1o( and [() is that
Io( I(0) + (x,x)) V3 (o)))-
(24)
Thus there is loss of information, but Ô is still an unbiased estimator of 0" with asymptotic normality.
A.3 DETAILED DERIVATION OF NORMAL APPROXIMATION
-x((x fo(x) 202 - - x||
(25)
= fx()) + vx))((), x x)
20 | x
(26)
= 20
[||x|/2 - 2(xxx x) + +||119 + xo - ~Vxxfo()) x) + fo())
(27)
|||x|| -2(x + -²Vxf((x) x 2/22 |1:11 (V) + fo( (28)
202
20 [|xx (x 0² V +c,
(29)
A.4 DIFFERENCE BETWEEN THE SCORES OF p(x) AND p(x)
For notation clarity, with x + e, we let be the distribution of x, and be the distribution of x. Then for a smooth testing function with vanishing tails,
E(h(x)] = E[h(x + c)) = E[h(x) 11(()( + 12" (x)e22 /2] E[h(x)) (x)] /2.
(30)
(31)
(32)
13

---

Published as a conference paper at ICLR 2021
Integral by parts,
[x)"())] = h=xx))(x)dxx= - (x(x)xx(x)xxx== / (((x))((x)xx
(33)
Thus we have the heat equation
x(x) - p(x) + (x)
(34)
The score
Vx logg(x)
V 1og
Vx log(1 p +x"(())(((xx)) /2) Vx (x))xxxxxxx(ee) /2.
(35)
= V log p(x)
(36)
Thus the difference between the score of p and p is of the order o², which negligible when o² small.
A.S LEARNING GRADIENTS OF NORMAL APPROXIMATION AND ORIGINAL RECOVERY LIKELIHOOD
In this subsection we demonstrate that the learning gradient of maximizing likelihood the normal approximation is approximately the same the gradient of maximizing the original recovery likeli- hood with one step of Langevin sampling. Specifically, the gradient the normal approximation of recovery reg-likelhhood for observed Xohs is
V
Vof Xnbs (x o² of
(x+ 0² fx))).
(37)
20
On the other hand, to maximize the original recovery likelihood, suppose sample Xayu (x) then the gradient ascent of the original recovery log-likelihood is
Vofo - E fe (Xsyn)l = hy (Xobs) )- (Xayn)]
(38)
where hx(x) (x). Approximately, if we perform one step of Langevin dynamics from x to
obtain Xsyn; i.e., Tsyn 2 (x) + V2or, and assume fo(x) locally linear x, then
Volo((Xtt=) o (Xinit]]
(39)
ho(Xobs) E|ha & + o² 5% + oe)]
(40)
ho(()) + (x) - x)) - (x()) + ae)]
(41)
18((8)(Xoss (x+ 0² (x))))
(42)
(x) (x o² (x)).
(43)
Comparing equations 37 and 43 we see that the two gradients agree with each other.
A.6 ESTIMATING THE PARTITION FUNCTION
We can utilize the sequence of learned distributions of y: (= / 1x4 estimate the partition function. Specifically, the marginal distribution of yt is
x1)
1
po (y))) = Zo,t
exp (Jo(y.,
(44)
We can estimate ratio the partition functions two consecutive time steps using importance sampling
Zo.
E (po(y)4)) [exp(fo(y, fel 1))1
(45)
Zo.t+1
(exp(fo(ye415 foly 1))],
(46)
M
14

---

Published as a conference paper at ICLR 2021
where 11+1,i , samples generated by progressive sampling. Starting from t T, where (r) fol- lows Gaussian distribution, we can compute log Zo., along the reverse path of the diffusion process, until we reach t 0:
T-1
Zo.t
7a.0 = Zorr
II
(47)
Zo
(=0
In practice, since the ratio given by MCMC samples can vary across many orders magnitude, is more meaningful estimate
T-1
Zo
log Ze log Ze.r T
log
(48)
Zo.141
Unfortunately, although equation 46 an unbiased estimator of
the logarithm of this
estimator is generally stochastic lower bound of log( ZZ.,/ (Zo.4/) (Grosse et al. 2016). However, as we show below, this bound will gradually converge to an unbiased estimator log(Ze. (e( Zg,++1) as the number of samples becomes large. Specifically, let A be the estimator in equation 46 be the true value of Z8,111 We have E[A] = μ. then by second order Taylor expansion,
Ellog A] = E log μ + (A n1)
2n² (A (A))²
(49)
llgg μ 2n12
Var (A).
(50)
By law of large number, Var(A) + 0 M oo, and thus Ellog A] - log u. This is also consistent with the estimation curves in the right subfigure of Figure 7 since Var(A) 2 0, the estimation curve increases from below as the number of samples becomes larger. When the curve becomes stable, it indicates the convergence.
B EXPERIMENTAL DETAILS
Model architecture. Our network structure is based on Wide ResNet ZZgoruyko & Komodakis, 2016). Table lists the detailed network structures of various resolutions. The number of ResBlocks at every level N is hyperparameter that we sweep over. The values of N for various datasets are listed Table 6. Each ResBlock consists of two Conv2D layers. For the second Conv2D layer, zero initialization for the weights, and add trainable channel-wise scaling parameter to the output. We remove the weight normalization, and use leaky ReLU (slope = 0.2) as the activation function in ResBlocks. Spectral normalization (Miyato et al., (0188) is used to regularize parameters in Cony2D layer, ResBlocks and Dense layer. For encoding time step f, we follow the scheme in (Ho et al. 2020). Specifically, the time step / is first transformed into sinusoidal embedding, and then two Dense layers is added. The time embedding is added after the first Conv2D layer of each ResBlock.
Training. We use Adam (Kingma & Ba 2014) optimizer for all the experiments. We find that for high resolution images, using smaller 31 in Adam help stabilize training. We use learning rate 0.0001 for all the experiments, For the values of 3,. batch sizes and the number of training iterations for various datasets, see Table
Datasets. We use the following datasets in our experiments: CIFAR-10 Krizhevsky al. 2009), CelebA (Liu et al. 2018) and LSUN (Yu et al. (015). CIFAR-10 is of resolution 32 x 32, and contains 50, 000 training images and 10, 000 test images. CelebA contains 202,599 face images, of which 162,770 are training images and 19, 19,962 are test images. For processing, we first clip each image to 178 x 178 and then resize it to 64 64. For LSUN, we use
and bedroom categories, which contains 126,227 and 3,033,042 training images respectively. Both categories contain 300 test images. For processing, we first crop each image to a square image of the smaller size among the height and weight, and then resize it to 64 x 64 128 128. For resizing, set antialias to True. We apply horizontal random flip as data augmentation for all datasets during training.
15

---

Published as a conference paper at ICLR 2021
Evaluation metrics. We use FID and inception scores quantitative evaluation metrics of sample quality. On all the datasets, we calculate FID and inception scores on 50,000 samples using the original code from Salimans et al. (2016) and [Heusel et al. (2017).
Table 5: Model architectures of various solutions. N is hyperparameter that we sweep over.
Resolution 32 32
(b) Resolution 64 64
(c) Resolution 128 128
x 3 Conv2D, 128
x 3 Conv2D, 128
3 3 Conv2D, 128
ResBlocks, 128 Downsample x 2
N ResBlocks, 128 Downsample 2 x
N ResBlocks, 128 Downsample 2 2
ResBlocks, 256
N ResBlocks, 256 Downsample 2 x
N ResBlocks, 256 Downsample 2 2
Downsample x 2
N ResBlocks, 256 Downsample x 2
N ResBlocks, 256 Downsample 2 x
N ResBlocks, 256 Downsample 2 2
ResBlocks, 256
N ResBlocks, 256 Downsample 2 x
N ResBlocks, 256 Downsample 2 2
ReLU, global sum Dense
N ResBlocks, 512
N ResBlocks, 512 Downsample 2 2
ReLU, global sum Dense
N ResBlocks, 512
ReLUU global sum Dense
(d) Time embedding (temb)
(c) ResBlock
sinusoidal embedding
leakyReI.U, 3 x 3 Conv2D
Dense, leakyReI.U
+ Dense(leayyReLLUtenbe
Dense
leakyReLU, 3 x Conv2D
input
Table 6: Hyperparameters of various datasets.
Dataset
N
81 in Adam
Batch size
Training iterations
CIFAR-10
8
0.9
256
240k
CelebA
6
0.5
128
880k
LSUN church_outdove 6
2
0.9
128
960k
LSUN bedroom 64
2
0.9
128
760k
LSUN church outdoor 128
2
0.5
64
840k
LSUN bedroom 128
5
0.5
64
580k
C ADDITIONAL EXPERIMENTAL RESULTS
C.I FID SCORES OVER ITERATIONS
Figure 9
demonstrates FID scores computed on 2,500 samples every 15,000 iterations.
C.2 LONG-RUN CHAIN SAMPLING WITH NUTS
As further check, we use No-U-Tum Sampler (Hoffman & Gelman 2014) to perform the long- chain sampling, with the same step size schedule obtained for HMC sampler. Figure [0 displays samples with different number of sampling steps. The samples remain realistic after 100% sampling steps in total and the FID score remains stable.
16

---

Published as a conference paper at ICLR 2021
140
K=10
K=30
K=50
FID score
80-
200
Iteration (x103)
Figure FIDs for different number of Langevin steps.
(a) 1k steps, FID=24.78
(b) 10h steps, FID=23.89
(c) 100% steps, FID=25.08
Figure 10: Long run chain samples with different total number of NUTS steps.
C.3 ADDITIONAL INTERPOLATION RESULTS
Figures 12 and 13 display more examples of interpolation between two generated samples on
CelebA 64 LSUN church.outdoor 128 and LSUN bedroom 128
Figure 11:
Interpolation results between the leftmost and rightmost generated samples on CelebA
64 x 64.
17

---

Published as a conference paper at ICLR 2021
Figure 12: Interpolation results between the leftmost and rightmost generated samples on LSUN church_outdoor 128 x 128.
Figure 13: Interpolation results between the leftmost and rightmost generated samples on LSUN bedroom 128 128.
C.4 ADDITIONAL IMAGE INPAINTING RESULTS
Figures 14 and 15 show additional examples of image inpainting on CelebA 642 and LSUN
church outdoor 1282
C.5 ADDITIONAL UNCURATED SAMPLES
Figures 16 17
18
19
20 and
show uncurated samples from the learned models under 76
setting on CIFAR-10 10, CelebA 64
LSUN church.outdoor 128?, LSUN bedroom 128?, LSUN
church outdoor 64 and LSUN bedroom 64 datasets.
18

---

Published as a conference paper at ICLR 2021
YO
WI
Figure 14: Image inpainting results on CelebA 64 64. Top: masked images, bottom: inpainted images.
Figure 15: Image inpainting results on LSUN Muuch_outdoor 128 128. Top: masked images,
bottom: inpainted images.
19

---

Published as a conference paper at ICLR 2021
fae
26
Figure 16: Generated samples on CIFAR-10.
20

---

Published as a conference paper at ICLR 2021
2
24
2
L
CUR
2
PLR
Figure 17: Generated samples on CelebA 64 x 64.
21

---

Published as a conference paper at ICLR 2021
Figure 18: Generated samples on LSUN church.outdoor 128 x 128, FID-9.76
22

---

Published as a conference paper at ICLR 2021
Figure 19: Generated samples on LSUN bedroom 128 x 128. FID=11.27
23

---

Published as a conference paper at ICLR 2021
Figure 20: Generated samples on LSUN church.outdons 64 x 64. FID=7.02
24

---

Published as a conference paper at ICLR 2021
Figure 21: Generated samples on LSUN bedroom 64 x 64. FID=8.98
25


Reviews:
[{"review_id": "QhX_ysaJIKI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary of the paper :\nThe authors propose to learn the recovery likelihood of a sequence of Energy-Based Models (EBM) trained on increasingly noisy version of the datasets. The authors demonstrate that optimizing the recovery likelihood is more convenient than learning the actual marginal likelihood, as it leads to more stable MCMC sampling and provide high-fidelity samples. The authors show competitive generation performances on CIFAR-10, Celeb1 and LSUN datasets.\n\n#### Pros : \n* The problem of non-convergent MCMC sampling in EBM trained with marginal likelihood is well motivated and well illustrated.\n* The authors proposition is well grounded in theory, and comes with rigorous and strong mathematical derivation.\n* The paper is clear, well written, and well structured\n\n#### Cons:\n* The quantitative comparison (IS and FID) is performed only on the CIFAR-10 datasets (see detailed comments).\n* The experimental evidences are not convincing.\n\n#### Recommandation:\nGiven the strength of the theoretical derivation, I would tend to accept the paper (marginally above threshold). However, stronger experimental evidences / discussion concerning the comparison with [1] & [2] would be beneficial for a better recommandation.\n\n#### Detailed comments:\n* While the authors show qualitative sampling results on LSUN, CIFAR-10, and CelebA they are quantitatively comparing the proposed framework on CIFAR-10 only. Even if CIFAR is challenging for generative models, I think the quantitative comparison on other databases is missing. In particular the comparison with [1] and [2] is necessary as the main claim of the paper is to show improvement using recovery likelihood.\n\n* The strength of the authors proposition is to provide a framework that could leverage less diffusion time steps than [2] while keeping good generation performance. Even if the authors clearly state such a theoretical advantage in paragraph 3.4, the experimental evidence is not convincing. Less diffusion time step (e.g. the case T=6, K=50) results in degraded FID and IS compared to [2]. In addition, the comparison between  T=1000, K=0 (DSM) versus [2] show a significant advantage in favor of [2] (they also use 1000 time step in [2]). What is the reason ? It suggests that optimizing the recovery likelihood is less efficient than optimizing the marginal one in diffusion problem...\n\n* Comparable setup (i.e T=1000, K=0 (DSM) versus T=1000, K=0) lead also to degraded performance : Why score-matching approach leads to better performance than diffusion recovery ? The quality of the paper would be greatly improved with such a discussion.\n\n#### Typos and suggestions to improve the paper\n* The initials 'MLE' are not explicitly defined\n* The Eq 3 should come with a demonstration (in Annex) or at least with a reference .\n* Eq 9 should be an approximate sign as it is a Taylor expansion.\n* The transition from Eq.9 to Eq.10 should be demonstrated (in Annex)\n* Eq 13 : N —> \\matcal{N}\n* Algorithm 2, line 2: It seems that there is a typo : t <- T-1 to T? Is it : t <- T-1 to 0\n\n[1] : Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. On learning non-convergent short- run mcmc toward energy-based model. arXiv preprint arXiv:1904.09770, 2019b. \n\n[2]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GKH2A27tBMz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel method to train EBMs based on diffusion recovery likelihood. It constructs a sequence of noisy version of data and learn a conditional between consecutive noisy pairs. Compare to working with the likelihood directly, doing so makes the training much easier. Besides, even using a potentially non-convergent MCMC for gradient estimation, it still leads to a well-behaved energy potential, unlike EBMs trained via maximising the likelihood directly.\n\n## Pros\n\n1. The paper is well-written and easy to understand. \n2. The method is novel and solves the training difficulty issue for EBMs neatly.\n3. The analysis of the normal approximation and how it leads to choose step sizes in the Langevin dynamics is neat.\n4. The paper discusses related work well while motivating the method and also makes interesting connections to VI and score matching in Section 3.4.\n\n## Cons\n\n1. Only applications on image generations are conducted.\n\n## Questions\n\n1. Does the method work only with the specific architecture here, especially the Transformer sinusoidal position embedding?\n2. It says T = 6 and K = 10 leads to unstable training in Section 4.1. Does the optimization diverge or something? It would be good to see some trace plots for the training procedure.\n3. Why do you use the specific HMC setup for Section 4.2? Specifically, using a leapfrog step of 2 can lead to issues in which the MCMC only explore local modes. I would like to see how a long-run NUTS works here. A well-behaved potential should work in both settings.\n\n## Discussions\n\n1. I'm quite glad to see Section 4.2 that shows the long-run MCMC works on the trained EBMs, even with (potentially) biased gradients for training. Is it because we only need relative short-run MCMC for convergence when using recovery likelihood so that the bias is actually small?\n2. How does the method relate to amortized MCMC?\n\n## Related work\n\nThere are a few more related works that the authors may consider including.\n\n[1] proposed a similar idea of making the modelling task easier using a sequence of targets (for EBMs), but achieved by density ratio estimation.\n[2] uses coupled MCMC to resolve the biased gradient issue when using CD.\n\n[1] Rhodes, Benjamin, Kai Xu, and Michael U. Gutmann. \"Telescoping Density-Ratio Estimation.\" Advances in Neural Information Processing Systems. 2020.\n[2] Qiu, Yixuan, Lingsong Zhang, and Xiao Wang. \"Unbiased Contrastive Divergence Algorithm for Training Energy-Based Latent Variable Models.\" International Conference on Learning Representations. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0N7YHITYd0e", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes training a sequence of conditional EBMs (inspired by  Ho et al. (2020)) instead of training unconditional EBMs.  Each conditional energy describes the probability of recovering x, given its noisy version \\hat{x}. The noisy version of x can be described as a normal distribution centered at x, so the condition EBM has an additional term ||x - \\hat{x}||^2, which constrains the Langevin dynamics to remain in the vicinity of \\hat{x}, so it converges faster!\nThe inference is that starting at white noise x_0, it defines a conditional EBMs given x_0, runs Langaving dynamics to sample from P(x|x_0) defined by the conditional EBMs, and then use the sample as the evidence of another conditional EBMs.\n\nI assume the main advantage of this training over Ho et al. (2020) is having an energy-based model that can be used in other applications, so that would be nice to see the performance of trained EBMs on some applications other than image generation such as image inpainting and robust classification as discussed in Du and Mordatch (2019).\n\nIn general, I believe that this is an exciting paper and an important step towards training better EBMs.\n\nIn connection to score matching, Saremi et al. (2018) and Saremi and Hyvarinen (2019) should be cited as well.\n\nSaremi et al. (2018), Deep Energy Estimator Networks.\nSaremi and Hyvarinen (2019), Neural Empirical Bayes.\n\ntypo: \"score-based based methods\" in Section 4.1\n\nAlg2: for t \\gets T - 1 to 0 do", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting model to sample from the Gibbs distribution using diffusion based method. The theory is interesting and it is related well to the current research in the field. All reviewers agree that this is a noteworthy contribution to ICLR. 
Decision: Accept (Poster)