Learning Energy-Based Prior Model with Diffusion-Amortized MCMC
Authors: Peiyu Yu
Yaxuan Zhu
Sirui Xie
Xiaojian Ma
Ruiqi Gao
Song-Chun Zhu
Ying Nian Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: Energy-Based Model
Denoising Diffusion Probabilistic Model
MCMC
TL;DR: We propose a learned amortization of long-run MCMC to develop a better learning algorithm for latent space EBMs.

Abstract: Latent space EBMs, also known as energy-based priors, have drawn growing interests in the field of generative modeling due to its flexibility in the formulation and strong modeling power of the latent space. However, the common practice of learning latent space EBMs with non-convergent short-run MCMC for prior and posterior sampling is hindering the model from further progress; the degenerate MCMC sampling quality in practice often leads to degraded generation quality and instability in training, especially with highly multi-modal and/or high-dimensional target distributions. To remedy this sampling issue, in this paper we introduce a simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based on it. We provide theoretical evidence that the learned amortization of MCMC is a valid long-run MCMC sampler. Experiments on several image modeling benchmark datasets demonstrate the superior performance of our method compared with strong counterparts.

Learning Energy-Based Prior Model with Diflusion-Amortized MCMC
Peiyu Yu' yupeiyu980g, ucla. edu
Yaxuan Zhu' yaxuanzhuβg, ucla. edu
Sirui Xie² srxieOucla, edu
Xiaojian Ma 2.4
Ruiqi Gao ruiqigggoogle.com com
Song-Chun Zhu sczhu@stat.. .uw..ccll..oo edu
xiaojian ma@ucla.... edu
Ying Nian Wu' ywuêstat ucla. .ddu
UCLA Department of Statistics Google DeepMind "Beijing Institute for General Artificial Intelligence (BIGAI)
UCLA Department of Computer Science
Abstract
Latent space Energy-Based Models (EBMss, also known energy-based priors, have drawn growing interests the field generative modeling due to its flexi- bility in the formulation and strong modeling power of the latent space. However, the common practice of learning latent space EBMs with non-convergent short- run MCMC for prior and posterior sampling is hindering the model from further progress; the degenerate MCMC sampling quality in practice often leads to de- graded generation quality and instability in training, especially with highly multi- modal and/or high-dimensional target distributions. To remedy this sampling is- sue, in this paper introduce simple but effective diffusion-based amortization method for long-run MCMC sampling and develop a novel learning algorithm for the latent space EBM based it. We provide theoretical evidence that the learned amortization of MCMC a valid long-run MCMC sampler. Experiments on sev- eral image modeling benchmark datasets demonstrate the superior performance of our method compared with strong counterparts'.
Introduction
Generative modeling of data distributions has achieved impressive progress with the fast develop- ment of deep generative models recent years [1-9]. provides powerful framework that allows successful applications in synthesizing data of different modalities [10-15], extracting semantically meaningful data representation [16-18] as well other important domains of unsupervised or semi- supervised learning (19-21]. A fundamental and powerful branch of generative modeling is the Deep Latent Variable Model (DLVM). Typically, DLVM assumes that the observation (e.g., a piece of text images) is generated by its corresponding low-dimensional latent variables via top-down generator network [1-3]. The latent variables are often assumed follow non-informative prior distribution, such uniform isotropic Gaussian distribution. While one can directly learn deep top-down generator network to faithfully map the non con-iffrrmative prior distribution to the data distribution, learning an informative prior model in the latent space could further improve the ex-
Code and data available at http:///gitaaloooyypuooouootttts http:///iiiisiimmmrriia--a------
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

pressive power of the DLVM with significantly less parameters [22]. In this paper. we specifically consider learning an EBM in the latent space as an informative prior for the model.
Learning energy-based prior can be challenging. as it typically requires computationally expen- sive Markov Chain Monte Carlo (MCMC) sampling estimate learning gradients. The difficulty of MCMC-based sampling is non-negligible when the target distribution is highly multi-modal or high-dimensional. In these situations, MCMC sampling can take long time to converge and per- form poorly on traversing modes with limited iterations [23]. Consequently, training models with samples from convergeet short-run MCMC [24], which is common choice for learning latent space EBMs [22], often results in malformed energy landscapes [15, 24, 25] and biased estima- tion of the model parameter. One possible solution is to follow the variational learning scheme [1], which however requires non-trivial extra efforts on model design to deal with problems like posterior collapse [26-28] and limited expressivity induced by model assumptions [1, 29, 30].
To remedy this sampling issue and further unleash the expressive power of the prior model, we propose to shift attention to an economical compromise between unrealistically expensive long-run MCMC and biased short-run MCMC: we consider learning valid amortization of the potentially long-run MCMC for learning energy-based priors. Specifically, inspired by the connection between MCMC sampling and denoising diffusion process 17, 8, 31J, in this paper we propose a diffusion- based amortization method suitable for long-run MCMC sampling in learning latent space EBMs. The learning algorithm derived from it breaks the long-run chain into consecutive affordable short- rum segments that can be iteratively distilled by a diffusion-basee sampler. The core idea is simple and can be summarized by one-liner (Fig. 1). We provide theoretical and empirical evidence that the resulting sampler approximates the long-run chain (see the proof-of-concept toy examples in Appendix E.1), and brings significant performance improvement for learning latent space EBMs on several tasks. We believe that this proposal is notable attempt to address the learning issues of energy-based priors and is new to the best of our knowledge. We kindly refer to Section 5 for comprehensive discussion of the related work.
Contributions i) We propose a diffusion-basee amortization method for MCMC sampling and de- velop a novel learning algorithm for the latent space EBM. ii) We provide theoretical understanding that the learned amortization of MCMC is a valid long-run MCMC sampler. iii) Our experiments demonstrate empirically that the proposed method brings higher sampling quality, a better-learned model and stronger performance on several image modeling benchmark datasets.
2 Background
2.1 Energy-Based Prior Model
We assume that for the observed sample a e R.D, there exists E Rdd as its unobserved latent variable vector. The complete-data distribution
pe(z 2, x) == Pa (=))s(1))) Pa( = exp (fa (f)( po(z),
(I)
Zo
where (z) is the prior model with parameters a, (a) 2) is the top-down generation model with parameters (3, and 0 (ax,3) The prior model (=) can be formulated as an energy-based model, which we refer to as the Latent-space Energy-Basse
Model (LEBM) [22] throughout the paper. In this formulation, fa (x) is parameterized by neural network with scalar output, Za is the partition function, and Po(=) (z) standard normal as a reference distribution. The prior model in Eq. (1) can be interpreted as an energy- based correction or exponential tilting of the original prior distribution Po- The generation model follows P (ax 2) .N(gs(z) (z). o Ip). where ga is the generator network and o² takes a pre-specified value as in VAE [1]. This is equivalent using la error for reconstruction.
The parameters of LEBM and the generation model can be learned by Maximum Likelihood Esti- mation (MLE) [22]. To be specific, given the training data æ. the gradients for updating a, (3 are.
8 (x: := (p)() [V [x)] (p) [=)], 5g(x)= := Epp() [V log Pa [ 2=] (2)
In practice one may use the Monte-Carlo average to estimate the expectations in Eq. (2). This in- volves sampling from the prior Pa Pa(2) (z) and the posterior po (2|2)) distribution using MCMC, specifi- cally Langevin Dynamics (LD) [32], to estimate the expectations and hence the gradient. For target

---

@ px+1
k = 0 .... K -
x
a (2. a.)
lo.
20
Ss x
t++
KiTe
25
20
Zt
z1+1
Ph
Figure I: Learning the DAMC sampler. The training samples for updating the sampler to Φ8y1 is obtained by T' T'step short-run LD, initialized with the samples from the current learned sampler Φx. Best viewed in color.
distribution nt (z), the dynamics iterates
s² v. log (2.) + swe, t == 0, I,..,,TT- 1, w; ((O,
Z
(3)
2
where 8 is small step size. One can draw small step size 8, the distribution of z, will converge to x as oc [32]. However, is prohibitively expensive to run LD until convergence in most cases, for which LD for sampling in practice. This non-convergent short chain yields a moment-matching distribution close to the true (2) but is often biased, which was dubbed as short-run LD [15, 23-25].
NNO,, La) to initialize the chain. For sufficiently
may resort to limited iterations of
2.2 Denoising Diffusion Probabilistic Model
Closely related to EBMs are the Denoising Diffusion Probabilistic Models (DDPMs) [5, 7, 8). As pointed out in [5, 8), the sampling procedure of DDPM with e-prediction parametrization resembles LD; E (predicted noise) plays a similar role to the gradient of the log density [8].
In the formulation proposed by Kingma et al. [33], the DDPM parameterized by Φ is specified by noise schedule built upon A log[32/o3]] i.e., the log signal-to-noise-ratii that decreases monotonically with s. 8g and of are strictly positive scalar-valued functions of s. We use zo to denote training data in Rd. The forward-time diffusion process (==|=0) is defined as:
<<zo<z0) = NZzzi B, LaL,)
q(2'21=e) = N(2,; (8. La).
(4)
8 & S and 0;11 (1- eAw )o Noticing that the forward process can be reverted as q( 2 (z,,, Re,, za) (2zz (zs, 20), , 02 Ia), an ancestral sampler (x(z)(z<)) [8] that starts at æs ~ (O., can be derived accordingly [33]:
where 0
[zps(zz, 20) c²v (0./a0 2 + (1 ex - )a,z0 02 = -1-etr-An
(5)
2, = ". 20)
where is standard Gaussian noise, 2o is the prediction of #0 by the DDPM o, and y is hy- perparameter that controls the noise magnitude, following [34]. The goal of DDPM is to recover the distribution of zo from the given Gaussian noise distribution. Ez,x [||{(x)) - <I3], where ~ (=(O0, I2) and is drawn from a distribution of log noise-to-signal ratio p()) over uniformly sampled times s [0, S). This loss can be justified as a lower bound on the data log- likeelooooood [8. 33] or variant of denoising score matching [7, 35]. We will exploit in this paper the connection between DDPMs and LD sampling of EBMs, based upon which we achieve better sampling performance for LEBM compared with short-run LD.
be trained by optimizing
3 Method
In this section, we introduce the diffusion-based amortization method for long-run MCMC sampling learning LEBM Section 3. The learning algorithm of LEBM based on the proposed method and details about implementation are then presented in Section 3.2 and Section 3.3, respectively.

---

3.1 Amortizing MCMC with DDPM
Amortized MCMC Using the same notation as in Section 2, we denote the starting distribution of LD as TT(2), the distribution after t-th iteration as Ty (z) and the target distribution as т T()). The trajectory of LD in Eq. (3) is typically specified by its transition kernel K((2='). The process starts with drawing 2o from Tz(z) and iteratively sample 2; at the (-th iteration from the transition
kernel conditioned on 2t-1, i.e. [T(2)) (z), where [=)=1(z) := I {(())=)m
To where Kc, denotes the 1-step transition kernel. LD can therefore be viewed as approximating a fixed point update in a non parammereeti fashion since the target distribution 'r is stationary distribution T(z) == S [x(=)=")=<=')d Vz. This motivates several works for more general approximations of this update [36-38] with the help of neural samplers.
Recursively,
Inspired by [36], we propose to use the following framework for amortizing the LD in learning the LEBM. Formally, let = { be the set of amortized samplers parameterized by Φ. Given the transition kernel K. the goal is to find a sampler 9s* to closely approximate the target distribution т. This can be achieved by iteratively distill T step transitions of L.D into qø-
a - arg min D(ge, ll 9d, I.T == KTTØ, 1
L0 & To, == o, --1. 1.
(6)
where D[-II-] is the Kullbeck-Ieebber Divergence (KLD) measure between distributions. Concretely, Eq. (6) means that to recover the target distribution æ, instead of using long-run L.D, we can repeat the following steps: i) employ 7-step short-run L.D initialized with the current sampler lo. to approximate Kr as the target distribution of the current sampler, and ii) update the current sampler qoh 1 to -4.-- The correct convergence of qa to π with Eq. (6) is supported by the standard theory of Markov chains [39], which suggests that the update in Eq. (6) is monotonically decreasing in terms of KLD, D[g0. ||=] & P(g [|=]. We refer to Appendix A. for detailed explanation and discussion of this statement. In practice, one can apply gradient-based methods to minimize Dlas, T. [I and approximate Eq. (6) for the update from Lø, to qø. The above formulation provides a generic and flexible framework for amortizing the potentially long MCMC.
biffusion-based amortization To avoid clutter, we simply write - T as Qr. We can see that arg min D[gr = arg min-- H((rr) + H(qr, q4) = arg min- min-Einnee - [log qol 4+
(7) where H represents the entropy of distributions. The selection of the sampler qs is a matter of de- sign. According to Eq. (7), we may expect the following properties from q:: i) having analytically tractable expression of the exact value or lower bound of log-likelihood, ii) easy to draw samples from and iii) capable of close approximation to the given distribution {97). In practice, iii) is impor- tant for the convergence of Eq. (6). If qo is far away from @r in each iteration, then non-increasing KLD property Dig ||m] & Di9 [|=] might not hold, and the resulting amortized sampler would not converge to the true target distribution ((2).
Se
9o
For the choice of qs. let us consider distilling the gradient field of qr in each iteration, so that the resulting sampler is close to the gr distribution. This naturally points to the DDPMs [8]. To be specific, learning a DDPM with E-prediction parameterization is equivalent to fitting the finite-time marginal of a sampling chain resembling annealed Langevin dynamics [7, 8, 31]. Moreover, it also fulfills i) and ii) of the desired properties mentioned above. We can plug in the objective of DDPM (Section 2.2), which is a lower bound of log 9s.
9s. to obtain the gradient-based update rule for qΦ: Pk-1 - 2 E. [l]<(z))
(8) where e .V( À is drawn from a distribution of log noise-to-signal ratio p(A). 17 is the step size for the update, and M is the number of iterations needed in Eq. (8). In practice, we find that when amortizing the LD sampling chain, light-weight DDPM qo updated with very small M, iee. few steps of Eq. (8) iteration, approximates Eq. (6) well. We provide a possible explanation using the Fisher information by scoping the asymptotic behavior of this update rule in the Appendix A.2. We term the resulting sampler as Diffusion- -aaotizzed MCMC (DAMC).
<|2] $ < (M)) i 0, 1,.,, M 1-1
3.2 Approximate MLE with DAMC
In this section, we show how to integrate the DAMC sampler into the learning framework of LEBM and form a symbiosis between these models. Given set of N training samples [a:)][] inddpne- dently drawn from the unknown data distribution Pdata (x) the model pe (Section 2.1) can be trained

---

by maximizing the log-likeli likelihood over training samples [ 1
N -i=1 log pe (z;). Doing so typ-
ically requires computing the gradients of C(0), where for each æ; the learning gradient satisfies:
Ve log (x,) = (m.( Ve log pe (= ax)
E po(. [V Va (z:)) - T (=.)) [a))( B po (x)) [V3 Pa 22)])
áa (ax)
(m.))
(9)
approximate the distri- butions in Eq. (9) by drawing samples from [x:] ~ KT, 9sh, (z](z;), 2; ~ 90, (zi), to estimate the expectations and hence the learning gradient. Here we learn the DAMC samplers To, (2) and Te,,, (z;) for the posterior and prior sampling chain, respectively. Φr. represents the current sampler as in Section 3.1; Zr,,./ and KT.,,, are the transition kernels for posterior and prior sampling chain. Equivalently, this means to better estimate the learning gradients we can i) first draw approximate posterior and prior MCMC samples from the current Lá, model, and ii) update the approximation of the prior Pa (=) and posterior pe (2 (x) distributions with additional T'-step LD initialized with l samples. These updated samples are closer to po (2|2;;) and Pa<(z;) compared with short-run LD samples based on our discussion in Section 2.2. Consequently, the diffusion- amortized LD samples provide a generally better estimation of the learning gradients and lead to better performance, as we will show empirically in Section 4. After updating 0 = (a. 33 based on these approximate samples with Eq. (9), we can update ls, with Eq. (8) to distill the sampling chain into ( As shown in Fig. 1, we can see that the whole learning procedure iterates between the approximate MLE of pg and the amortization of MCMC with q4- We refer to Appendix A.3 for an extended discussion of this procedure.
Intuitively, based on the discussion in Section 2.1 and Section 3.1,
After learning the models, we can use either DAMC or LEBM for prior sampling. For DAMC, we may draw samples from (z())) with Eq. (5). Prior sampling with LEBM still requires short-run LD initialized from (0, Ia). For posterior sampling, we may sample from J,,,, l0, (z,(z,), ie., first draw samples from DAMC and then run few steps of LD obtain posterior samples.
Algorithm 1: Learning algorithm of DAMC.
Input: initial parameters (a, B. Φ): learning rate 17 = (Mec 71, no); observed examples (2(0) )- prob. of uncond. training Puncond for the DAMC sampler.
Output: (glK) (k) (K) (
1 for i = 0 K - do
2
Sample a minibatch of data (0)0)
Draw DAMC samples: For each (f), draw ( and 20 from qo, (z.(20;).
Prior LD update: For each (1), update (2) using Eq. (3), with T (2,) (a (z:);
Posterior LD update: For each (9))) update = using Eq. (3), with T(z,) = Pag / (2i(z;))
(4)
Update (*), Update (4) and (a()) using Monte- Carlo estimates (i.e., Monte- Carlo
average) of Eq. (9) with (2(4)) R and 1,
Update o (k); Update (k) using Eq. (8) with Puncond and (2(2), ),E, as the target.
3.3 Implementation
In order to efficiently model both Ls. (z][z;) and gor (zi). we follow the method of [40] to train a single network to parameterize both models, where qs, (z,(0;) can be viewed as a conditional DDPM with the embedding of x; produced by an encoder network as its condition, and go, (2,) an unconditional one. For 9s, (6, (z+), we can input a null token Ø as its condition when predicting the noise c. We jointly train both models by randomly nullifying the inputs with the probability Puncond 0.2. During training, we use samples from gs, (=:(æ;;) to initialize both prior and poste- rior updates for training stability. For the posterior and prior DAMC samplers, we set the number of diffusion steps to 100. The number of iterations in Eq. (8) is set to M throughout the experi- ments. The LD runs 30 and 60 iterations for posterior and prior updates during training with step
5

---

Table 1: MSE(() and FID()) obtained from models trained different datasets. The FID scores are com- puted based on S0k generated images and training images for the first three datasets and 5k images for the CelebA-HQ dataset. The MSEs are computed based on unseen testing images. We highlight our model results in gray color. The best and second best performances are marked in bold numbers and underlines, respectively; tables henceforth follow this format. "[41] uses prior model with 4x parameters compared with [22] and ours.
SVHN
Celeb
CIFAR-10
CelebA-HO
Model
MSE
FID
MSE
FID
MSE
FID
MSE
FID
VAE [1]
0.019
46.78
0.021
65.75
0.057
106.37
0.031
180.49
2s- VAE [48]
0,019
42.81
0.021
44.40
0.056
72.90
RAE [49]
0.014
40.02
0.018
40.95
0.027
74.16
NCP-VAE [50]
0.020
33.23
0.021
42.07
0.054
78.06
Adaptive CE* [41]
0.004
26.19
0.009
35.38
0.008
65.01
ABP [51]
49.71
51.50
0.018
90.30
0.025
160.21
SRI [24]
0.018
44.86
0.020
61.03
SRI (L==5) [24]
0.011
35.32
0.015
47.95
LEBM [22]
0.008
29.44
0.013
37.87
0.020
70.15
0.025
133.07
Ours-LEBM
21.17
35.67
60.89
89.54
0.002
0.005
0.015
0.023
Ours-DAMC
18.76
30.83
57.72
85.88
size of s 0.1. For test time sampling from 1x. 90. (z)(a)). T 10 for the additional LD. For fair comparison, we use the same LEBM and generator as in [22, 41] for all the experiments. We summarize the learning algorithm in Algorithm architecture and further training details, as well as the pytorch-style pseudocode of the algorithm.
Please see Appendices B and C for network
4 Experiments
In this section, we are interested in the following questions: (i) How does the proposed method compare with its previous counterparts (e.g., purely MCMC-based or variational methods)? (ii) How the scalability of this method? (iii) How are the time and parameter efficiencies? (iv) Does the proposed method provide desirable latent space? To answer these questions, we present series of experiments on benchmark datasets including MNIST [42], SVHN [43], CelebA64 [44], CIFAR- 10 [45], CelebAMask-HO [46], FFHQ [10] and LSUN-Tower [47]. As to be shown, the proposed method demonstrates consistently better performance various experimental settings compared with previous methods. We refer to Appendix D for details about the experiments.
4.1 Generation and Inference: Prior and Posterior Sampling
Generation and reconstruction We evaluate the quality of the generated and reconstructed im- ages to examine the sampling quality of DAMC. Specifically, we would like to check i) how well does DAMC fit the seen data, ii) does DAMC provide better MCMC samples for learning LEBM and iii) the generalizability of DAMC on unseen data. We check the goodness of fit of DAMC by evalu- ating the quality of the images generated with DAMC prior samples. If DAMC does provide better MCMC samples for learning LEBM, we would expect better fitting of data and hence an improved generation quality of LEBM. We evaluate the performance of posterior sampling given unseen test- ing images by examining the reconstruction error on testing data. We benchmark our model against variety of previous methods in two groups. The first group covers competing methods that adopt the variational learning scheme, including VAE [1]. as well as recent two-stage methods such as 2- stage VAE [48], RAE [49] and NCP-VAE [50], whose prior distributions are learned with posterior samples in a second stage after the generator is trained. The second group includes methods that adopt MCMC- based sampling. It includes Alternating Back- back-Propogation (ABP) [51], Short- In- ference (SRI) from [24] and the vanilla learning method of LEBM [22], which relies on short-run LD for both posterior and prior sampling. We also compare our method with the recently proposed Adaptive CE [41]. It learns series of LEBMs adaptively during training, while these LEBMS are sequentially updated by density ratio estimation instead of MLE To make fair comparisons, we follow the same evaluation protocol as in [22, 41].

---

2
82
5 3
8
16
22
6781
0C 36
2
1884
3
3
K
5
(a) SVHN
(b) CelebA
(c) CIFAR-10
(d) CelebA-HQ
Figure 2: Samples generated from the DAMC sampler and LEBM trained on SVHN, CelebA, CIFAR-I0 and CelebA-HQ datasets. In each sub-figure, the first four rows are generated by the DAMC sampler. The last four rows are generated by LEBM trained with the DAMC sampler.
For generation, we report the FID scores [52] in Table 1. We observe that i) the DAMC sampler, denoted as Ours-DAMC, provides superior generation performance compared to baseline models, and ii) the LEBM learned with samples from DAMC, denoted as Ours-LEBM, demonstrates sig- nificant performance improvement compared with the LEBM trained with short-run LD, denoted as LEBM. These results confirm that DAMC is a reliable sampler and indeed partly addresses the learning issue of LEBM caused by short-run LD. We would like to point out that the improvement is clearer the CelebAMask- HHQ dataset, where the input data is of higher dimension (256 256) and contains richer details compared with other datasets. This illustrates the superiority of DAMC sampler over short-run LD when the target distribution is potentially highly qualitative results of generated samples in Fig. 2, where we observe that our method can generate diverse, sharp and high-quality samples. For reconstruction, we compare our method with baseline methods in terms of MSE in Table 1. We observe that our method demonstrates competitive recon- struction error, if not better, than competing methods do. Additional qualitative results of generation and reconstruction are presented in Appendices E.2 and E.3.
We show
GAN inversion We have examined the scal- ability of our method on the CelebAMask-HH dataset. Next, we provide more results on high- dimensional and highly multi-modal data by performing GAN inversion [54] using the pro- posed method. Indeed, we may regard GAN inversion as an inference problem and a spe- cial case of posterior sampling. As a suitable testbed, the StyleGAN structure [10] is specif- ically considered as our generator in the exper- iments: [53] points out that to effectively infer the latent representation of a given image, the GAN inversion method needs to consider an ex- tended latent space of StyleGAN, consisting of 14 different 312-dimensional latent vectors. We attempt to use the DAMC sampler for GAN inver- sion. We benchmark our method against i) learning an encoder that maps given image to the latent space [16], which relates to the variational methods for posterior inference, ii) optimizing a random initial latent code by minimizing the reconstruction error and perceptual loss [53], which can be viewed as a variant of LD sampling, and iii) optimizing the latent code by minimizing both the ob- jectives used in ii) and the energy score provided by LEBM. We use the pretrained weights provided by [18] for the experiments. Both the DAMC sampler and the encoder-based method are augmented with 100 post-processing optimization iterations. We refer to Appendix D for more experiment de- tails. We test LEBM-based inversion with different optimization iterations. To be specific, 1x, 2x, and 4x represent 100, 200, and 400 iterations respectively. We can see in Table 2 that DAMC per- forms better than all the baseline methods on the unseen testing data, which supports the efficacy of our method in high-dimensional settings. We provide qualitative results in Fig. 3.
FFHQ
LSUN-T
Model
MSE
FID
MSE
FID
Opt. [53]
0.055
149.39
0.080
240.11
Enc. [16]
0.028
62.32
0.079
132.41
[22] w/ 1x
0.054
149.21
0.072
239.51
[22]
0.039
101.59
0.066
163.20
[22] w/ 4x
0.032
84.64
0.059
111.53
Ours
0.025
52.85
0.059
80.42
Table 2: MSE(L) and FID(L) for GAN inversion on different datasets. Opt. and Enc. denotes the optimization-based and encoder-hased methods.
Parameter efficiency and sampling time One potential disadvantage of our method its param- eter inefficiency for introducing an extra DDPM. Fortunately, our models are in the latent space
7

---

(a) Observation
Inversion
Figure 3: Qualitative results of StyleGAN inversion using the DAMC sampler. In each sub-figure, the left panel contain samples from the FFHQ dataset, and the right panel contains samples from the LSUN-T dataset.
(a) 200 steps
(b) 2500 steps
Figure 4: Transition of Markov chains initialized from (0 L.) towards pa (=). We present results by running LD for 200 and 2500 steps. In each sub-figure, the top panel displays the trajectory in the data space uniformly sampled along the chain. The bottom panel shows the energy score fa (z) over the iterations.
so the network is lightweight. To be specific, on CIFAR-I0 dataset the number of parameters in the DDPM only around 10% (excluding the encoder) of those the generator. The method has competitive time efficiency. With the batch size of 64, the DAMC prior sampling takes 0.3s, while 100 steps of short-run LD with LEBM takes 0.2s. The DAMC posterior sampling takes 1.0s, while LEBM takes 8.0s. Further discussions about the limitations can be found in the Appendix G.I.
4.2 Analysis of Latent Space
Long-run langevin transition In this section, we examine the energy landscape induced by the leameed LEBM. We expect that well-trained Pa (2(2) fueled by better prior and posterior samples from the DAMC sampler would lead to energy landscape with regular geometry. In Fig. 4, we visualize the transition of LD initialized from (0, La) towards (=) the model trained on the CelebA dataset. Additional visualization of transitions on SVHN and CIFAR-10 datasets can be found in the Appendix E.4. The LD iterates for 200 and 2500 steps, which is longer than the LD within each training iteration (60 steps). For the 200-step set-up, we can see that the generation quality quickly improves by exploring the local modes (demonstrating different facial features, e,g., hairstyle, facial expression and lighting). For the 2500-step long-run set-up, we can see that the LD produces consistently valid results without the oversaturating issue of the long-run chain samples [23]. These observations provide empirical evidence that the LEBM is well-trained.
Anomaly detection We further evaluate how the LEBM learned by method could benefit the anomaly detection task. With properly learned models, the posterior (2( could form a dis- criminative latent space that has separated probability densities for in-distribution (normal) and out- of-distribution (anomalous) data. Given the testing sample æ, we use un- un-normalized log joint density Pe.e (x) (ppe (xx, 2) (=(x))2)p (=))
as our decision function. This means that
we draw samples from KC (o (z) and compare the corresponding reconstruction errors and en- ergy scores. A higher value of log joint density indicates higher probability of the test sample being a normal sample. To make fair comparisons, we follow the experimental settings [22, 41, 55, 56] and train our model on MNIST with one class held out as an anomalous class. We consider the

---

Table 3: AUPRC(() scores for unsupervised anomaly detection on MNIST. Numbers taken from [41].
Results of our model are averaged over the last 10 trials to account for variance.
Heldout Digit
1
4
5
7
VAE [I]
0,063
0.337
0..225
0.148
0.104
ABP [51]
0.095 = 0.03
0.138 + 0.04
0.147 = 0.03
0.138 + 0.02
0.102
0.03
MEG [55]
0.281 + 0.04
0.401 + 0.06
0.402 + 0.06
0.290 + 0.04
0.342 + 0.03
BiGAN-o [56]
0.287 = 0.02
1.433:000
0.514 0.03
0.347
0.02
0.307
0.03
LEBM [22]
0.336 = 0.01
0.630
0.02
0.619 0.01
0.463
+ 0.01
0.413 +
0.01 0.01
Adaptive CE [41]
0.531 + 0.02
0.729 + 0.02
0.742 + 0.01
0.620 + 0.02
0.499 + 0.01
Ours
0.684 0.02
0.911 1 0.01
0.939 + 0.02
0.801 + 0.01
0.705 + 0.01
Table 4: Ablation study on CIFAR-10 dataset. VI denotes learning LEBM using variational methods. SR
denotes learning LEBM with short-run LD. DAMC-G replaces the LEBM in DAMC-LEBM with a standard Gaussian distribution. NALR denotes the non-amortized DDPM setting. For each set-up, provide results
using the vanilla sampling method, denoted as V, and the ones using the DAMC sampler, denoted D.
VI-LEBM
SR-LEBM
DAMC-G
NALR-LEBM DAMC-LEBM
Model
V.
D.
V.
D.
V.
D.
V.
D.
V.
D.
MSE
0.054
0.020
0,018
0.015
0.028
0.016
0.021
0.015
FID
78.06
70.15
90.30
66.93
68.52
64.38
60.89
57.72
baseline models that employ MCMC-based or variational inferential mechanisms. Table shows the results of AUPRC scores averaged over the last 10 trials. We observe significant improvements in our method over the previous counterparts.
4.3 Ablation Study
In this section, we conduct ablation study on several variants of the proposed method. Specifically, we would like to know: i) what is the difference between the proposed method and directly training DDPM in fixed latent space? ii) What is the role of LEBM in this learning scheme? iii) Does DAMC effectively amortize the sampling chain? We use CIFAR-10 10 dataset for the ablative experi- ments to empirically answer these questions. More ablation studies can be found in Appendix F.
Non-Amortized DDPM vs. DAMC We term directly training a DDPM in a fixed latent space as the non amortized DDPM. To analyze the difference between amortized DDPM and DAMC, we first train a LEBM model with persistent long-run chain sampling [57] and use the trained model to obtain persistent samples for learning the non-amortized DDPM. In short, the non-amortized DDPM can be viewed as directly distilling the long-run MCMC sampling process, instead of progressively amortizing the chain. We present the FID and MSE of the non-amortized model (NALR) in Table 4. We observe that directly amortizing the long-run chain leads to degraded performance compared with the proposed method. The results are consistently worse for both posterior and prior sampling and the learned LEBMs, which verify the effectiveness of the proposed iterative learning scheme.
The contribution of LEBM it might not be necessary to jointly learn LEBM in the latent space. To demonstrate the necessity of this joint learning scheme, we train a variant of DAMC by replacing the LEBM with a Gaussian prior. The results are presented in Table 4. We observe that models trained with non-informative Gaussian prior obtain significantly worse generation results. It suggests that LEBM involved in the learning iteration provides positive feedback to the DAMC sampler. Therefore, we believe that is crucial to jointly learn the DAMC and LEBM.
One may argue that since we have the DAMC as powerful sampler,
Vanilla sampling vs. DAMC We compare the vanilla sampling process of each model with DAMC. The vanilla sampling typically refers to short-run or long-run LD initialized with (0, Iz). We also provide results of learning LEBM using variational methods for comparison. We can see in Table 4 that sampling with DAMC shows significantly better scores of the listed models, com- pared with vanilla sampling. The result is even better than that of the persistent chain sampler (V. of NALR-LEBM), This indicates that DAMC effectively amortizes the sampling chain. Comparing DAMC sampler with the variational sampler also indicates that DAMC is different from general vari- ational approximation: it benefits from its connection with LD and shows better expressive power.

---

5 Related Work
Energy-based prior model EBMs [23, 24, 58-60) play an important role in generative model- ing. Pang et al. [22] propose to learn an EBM as a prior model in the latent space of DLVMs; it greatly improves the model expressivity over those with con-informative priors and brings strong performance on downstream tasks, e.g., image segmentation, text modeling, molecule generation, and trajectory prediction [12, 15, 61, 62]. However, learning EBMs or latent space EBMs requires MCMC sampling to estimate the learning gradients, which needs numerous iterations to converge when the target distributions are nigh-dimensional or highly multi-modal Typical choices of sam- pling with con-ernveg short-run MCMC [24] in practice can lead to poor generation quality, malformed energy landscapes [15, 24, 25], biased estimation of the model parameter and instability in training [23, 25, 59, 60]. In this work, we consider learning valid amortization of the long-run MCMC for -neegy-based priors; the proposed model shows reliable sampling quality in practice.
Denoising diffusion probabilistic model DDPMs [7, 8, 31J, originating from [5], learn the gen- erative process by recovering the observed data from a sequence of noise-perturbed versions of the data. The learning objective can be viewed as a variant of the denoising score matching objective [35]. As pointed out in [5, 8), the sampling procedure of DDPM with e-prediction parametrization resembles LD of an EBM; (predicted noise) plays similar role to the gradient of the log den- sity [8]. To be specific, learning a DDPM with e-prediction parameterization is equivalent to fitting the finite-time marginal of sampling chain resembling annealed Langevin dynamics [7. 8, 31]. Inspired by this connection, we propose to amortize the long-run MCMC in learning energy-based prior by iteratively distilling the short-run chain segments with DDPM-based sampler. We show empirically and theoretically that the learned sampler is valid for long-run chain sampling.
Amortized MCMC The amortized MCMC technique is formally brought up by Li et al. (36). which incorporates feedback from MCMC back to the parameters of the amortizer distribution 4ø- It is concurrently and independently proposed by Xie et al. [37] as the MCMC teaching framework. Methods under this umbrella term [36-38, 63-66] generally learns the amortizer by minimizing the divergence (typically the KLD) between the improved distribution and its initialization, i.c., [KCr9o,_1194) where KT represents Ttstep MCMC transition kernel and - represents the current amortizer. The diffusion- based amortization proposed in this work can be viewed as an in- stantiation of this framework, while our focus is learning the energy-based prior. Compared with previous methods, our method i) specifically exploits the connection between EBMs and DDPMs and is suitable for amortizing the prior and posterior sampling MCMC of energy-based prior, and ii) resides in the lower-dimensional latent space and enables faster sampling and better convergence.
More methods for learning EBM Several techniques other than short-run MCMC have been pro- posed to learn the EBM. In the seminal work, Hinton [67] proposes to initialize Markov chains using real data and run several steps of MCMC to obtain samples from the model distribution. Tieleman [57] proposes to start Markov chains from past samples in the previous sampling iteration, known as Persistent Contrastive Divergence (PCD) or persistent chain sampling, to mimic the long-run sam- pling chain. Nijkamp et al. [23] provide comprehensive discussions about tuning choices for LD such as the step size and sampling steps T to obtain stable long-run samples for persistent train- ing. [59, 60) employ a hybrid of persistent chain sampling and short-run sampling by maintaining a buffer of previous samples. The methods draw from the buffer or initialize the short-run chain with noise distribution with some pre-specified probability. Another branch of work, stemmed from [68], considers discriminative contrastive estimation to avoid MCMC sampling. Gao et al. [69] use a normalizing flow [4] as the base distribution for contrastive estimation. Aneja et al. [50] propose to estimate the energy- based prior model based the prior of pre-trained VAE [70] by noise con- trastive estimation. More recently, Xiao and Han [41] learn a sequence of EBMs in the latent space with adaptive multi-stage NCE to further improve the expressive power of the model.
6 Conclusion
In this paper, we propose the DAMC sampler and develop novel learning algorithm for LEBM based on it. We provide theoretical and empirical evidence for the effectiveness of our method. We notice that our method can be applied to amortizing MCMC sampling of unnormalized continuous densities in general. can also be applied to sampling posterior distributions of continuous latent variables in general latent variable models. We would like to explore these directions in future work.
10

---

Acknowledgements
Y. N. Wu was supported by NSF DMS-2015577. We would like thank the anonymous reviewers for their constructive comments.
References
[1] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[2] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of International Con- ference on Machine Learning (ICML), 2014.
[3] lan J Goodfellow, Jean Pouget-Abadie. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
[4] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows, In Inter- national conference on machine learning, pages 1530-1538. PMLR, 2015.
[5] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. In Proceedings of International Conference on Machine Learning (ICML), 2015.
[6] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fideling images with vq-vae-2, Advances in neural information processing systems, 2019.
[7] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data dis- tribution. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[8] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Ad- vances in Neural Information Processing Systems (NeurIPS), 2020.
(9] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2020.
[10] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 4401-4411 2019.
[11] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0976 2020.
[12] Peiyu Yu, Sirui Xie, Xiaojian Ma. Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Unsu- pervised foreground extraction via deep region competition. Advances in Neural Information Processing Systems, 2021.
[13] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[14] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
[15) Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruigi Gao, Yixin Zhu, Song-Chun Chuu Zhu, and Ying Nian Wu. Latent diffusion energy-based model for interpretable text modeling. arXiv preprint arXiv:2206.05895 2022.
[16] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros Generative visual manipulation on the natural image manifold. Proceedings of European Conference on Com- puter Vision (ECCV), 2016.

---

[17] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 2017.
[18] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference Computer Vision (ECCV), 2020.
[19] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models, Advances in Neural Information Processing Systems (NeurIPS), 2020.
[20] Will Grathwohl, Kuan-Chieh Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations (ICLR), 2020.
[21] Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[22] Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[23] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Chun-Chhn Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy- based models. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2020.
[24] Erik Nijkamp. Mitch Hill, Song- Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-baset model. In Advances Neural Information Processing Systems (NeurIPS), 2019.
[25] Ruiqi Gao. Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy- based models by diffusion recovery likelihood. In International Conference on Learning Rep- resentations (ICLR), 2020.
[26] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), 2016.
[27] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. arXiv preprint (001:0090000344 2019.
[28] James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the elbo! a linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[29] Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In Proceed- ings of International Conference on Machine Learning (ICML), 2016.
[30] Jakub Tomczak and Max Welling. Vae with a vampprior. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.
[31] Yang Song and Stefano Ermon. Improved techniques for training score- based generative mod- els. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[32] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of International Conference on Machine Learning (ICML). 201
[33] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[34] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuusio probabilistic models. In Proceedings of International Conference on Machine Learning (ICML), 2021.
[35] Pascal Vincent. connection between score matching and denoising autoencoders. Neural computation,
2011.
12

---

[36] Yingzhen Li, Richard E Turner, and Qiang Liu. Approximate inference with amortised meme. arXiv preprint arXiv: 1777.00843333 2017.
[37] Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy- -bssd model and latent variable model via meme teaching. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2018.
[38] Francisco Ruiz and Michalis Titsias. contrastive divergence for combining variational infer- ence and meme. In Proceedings of International Conference on Machine Learning (ICML), 2019.
[39] Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
[40] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.
[41] Zhisheng Xiao and Tian Han. Adaptive multi-stage density ratio estimation for learning latent space energy-based model. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[42] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
[43] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[44] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730- 3738, 2015.
[45] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[46] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and in- teractive facial image manipulation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[47] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506::5500003355 03365, 2015.
[48] Bin Dai and David Wipf. Diagmosing and enhancing vae models. International Conference on Learning Representations (ICLR), 2019.
[49] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. International Conference on Learning Repre- sentations (ICLR), 2020.
[50] Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[51] Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Alternating back pock-prrpagttion for generator network. In Proceedings of AAAI Conference Artificial Intelligence (AAAI), 2017.
[52] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[53] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of International Conference on Computer Vision (ICCV), 2019.
[54] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang- Gan inversion: survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
13

---

[55] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-basec based models. arXiv preprint wrXiv:1901008500 2019.
[56] Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chandrasekhar. Efficient gan-based anomaly detection. Proceedings of the 20th IEEE Inter- national Conference on Data Mining (ICDM), 2018, 2018.
[57] Tijmen Tieleman. Training restricted boltzmann machines using approximations to the like- lihood gradient. In Proceedings of International Conference on Machine Learning (ICML), 2008.
[58] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In Proceedings of International Conference Machine Leurning (ICML), 2016.
[59] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv: 1903. 1903.0889, 2019.
[60] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. In Proceedings of International Conference on Machine Learning (ICML), 2021.
[61] Bo Pang. Tian Han, and Ying Nian Wu. Learning latent space energy-based prior model for molecule generation. arXiv preprint vrXiv:2010.09351, 2020.
[62] Bo Pang, Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy-based model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[63] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Generative voxelnet: learning energy-based models for 3d shape synthesis and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2468-2484, 2020.
[64] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto- encoder as amortized sampler. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2021.
[65] Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. tale of two flows: cooperative learn- ing of langevin flow and normalizing flow toward energy-based model. arXiv preprint WxXiv:2005.06924, 2022.
[66] Yaxuan Zhu, Jianwen Xie, Yingnian Wu, and Ruiqi Gao. Learning energy-based models by cooperative diffusion recovery likelihood. arXiv preprint arXiv:2309.05/53 2023.
[67] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
[68] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation prin- ciple for unnormalized statistical models. In Proceedings of the thirteenth international con- ference on artificial intelligence and statistics, 2010.
[69] Ruiqi Gao, Erik Nijkamp. Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[70] Arash Vahdat and Jan Kautz. Nvac: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 2020.
[71] JM Bernardo, MJ Bayari, JO Berger, AP Dawid, D Heckerman, AFM Smith, M West, et al. The variational bayesian em algorithm for incomplete data: with application to scoring graph- ical model structures. Bayesian statistics, 7(453-464):210, 2003.
[72] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1551114444, 2015.
14

---

[73] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[74] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[75] Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. Advances in Neural Information Processing Systems, 2021.
[76] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint ar7.0v260...8802,, 2016.
[77] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[78] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[79] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprini 1.X00 2017.
[80] Shohei Taniguchi, Yusuke Iwasawa, Wataru Kumagai, and Yutaka Matsuo. Langevin autoen- coders for learning deep latent variable models. Advances in Neural Information Processing Systems, 35:13277-13289, 2022.
15

---

A Theoretical Discussion
A.1 Monotonically Decreasing KLD
We state in the main text that Pla || To show this, we first provide a proof of D[ < D[ where म, and T++ are the distributions of z at t-th and (t + -h iteration, respectively. This is a known result from [39], and we include it here for completeness.
Dla 1 ||m], where the stationary distribution.
[((2 T(z)) = (m(z))))()))lzz)))
Te (1(2))
D D[m[|<]] ||| = Bm)()))
log
log TT ((z
10' K (=)22))))+
E (2(1)) (scless))
log 13' (2()2222)))((111
(10)
D[m++a/|T] + Ex, ((E)11) D[K mep (ze (01]2+1)]| (2,
(4) D D(m=1||>),
where we denote the forward-time transition kernel as KC and the reverse-time kernel as KC. (1) holds because are just re-factorizing the joint density of [zt, ze+1] Te Tz(2))K(zz+11)) K' F (e 1)++ and ((z+1
= (/=(zzlz(1) (z(+1)) (ii) holds because the
KLD is non-negative. We
that D [ [[r] & [[ is direct result from Eq. (10), and
that π as f - oc under proper conditions [32].
In the main text, we describe the update rule of the sampler To as follows:
arg min Digoo_..rllo)) I. KI Aoo To, k O..,KKKK K- (11)
In the ideal case, can assume that the objective in Eq. (6) is properly optimized and that {go) is expressive enough to parameterize each - With qd, s xo and 9s, ~ Kr we can conclude that D[g0,,11 [r]] & D[ge,_,,ll [|=] for each h = 1.... K according to Eq. (10). We will discuss in the following section the scenario where we apply gradient-based methods to minimize I [ and approximate Eq. (6) for the update from q to 9
A.2 Discussion about Diffusion-Based Amortization
We can see that the statement in Appendix A. holds when Tø. is a close approximation of 4r KT This motivates choice of employing DDPM amortize the LD transition, sidering its capability of close approximation the given distribution {gr}. Based the derivation of DDPM learning objective in [1], we know that
arg min P[grll46b arg min - H(gT) + H(gT, go) arg min Egr [log qol
N
< arg min E,, [|(6(())) - -1 x arg min
(Z,,,)))
-
8
g
(12)
where 2) is draw from ==>\/=0) = Mizx; BxzO, La). from distribution of log noise-to-signal ratio p()). In practice, we use Monte-Carlo average to approximate the objective and employ a gradient-based update rule for qø:
4T NoO. L2) and À is drawn
N
(++1) 1i - V N
Σ
(M) t - ( 1= M 1.
(13)
This can be viewed M-estimation of &. Recall that gr KT we construct dx (0--1.)) to minimize the KLD, where & models the transition kernel Kr.. Therefore, initializing ls to be optimized with - effectively maximizing C(Φ) $ Σラ= logpz($)l2)
S1

---

where {zz} are from -6--1 and {2y} are from Kr Let ø be the result of M-estimation and be the true target parameter. Then based on the derivation in [25], asymptotically we have
VN ( - -2)
NN
(14)
V2 log pz is the Fisher information matrix. This interpretation tells us that i) when the sample size N large, the estimation Φ is asymptotically unbiased, and ii) we want to obtain the estimation Φ with few gradient- updates, then the eigenvalues of the Fisher information matrix would be relatively small but non-zero. ii) suggests that KTTØ,-1 should be significantly different from 9 which is confirmed by [38] and our preliminary experiments, but it should not be too far away because that would require more gradient-basee updates. We find that setting 30 and M 6 works well in the experiments.
where I (3" (Ø')-Fzz. Rg..x
A.3 Further Discussion about the Learning Algorithm
For completeness, we first derive the learning gradients for updating 0.
[ pa (2, z)
Vo (o))) = po(a)) Vo (ar z)dz =
Vo log po(x z)dz
(o(x))
(oo()))))) [Vo log po (2, 2:)
(15)
(p)(()
[V log Pa (z) (po(z(m) (olloga(<>>>))
(ppp(=))) (ala (x)(*) [VaJa(z)] po x)) [V log pog
6a
<g(#)
We can see that estimated in a MLE-by-EM style. The learning gradient is the same as that of directly maximizing the observed data likelihood, while need to approximate the expectations in Eq. (15). Estimating the expectations is like the E-step, and update with Eq. (15) is like the M-step in the EM algorithm. The proposed diffusion- amortization brings better estimation of the expectations in the E-step, and incorporate the feedback from the M-step by running prior and posterior sampling LD as follows
82 2.
2t+1 2, + VZz
fa(z)) 12 +sw
log NogPer(=:) (=:)
(16)
21+1 = 2t
+
Vz
x
gg( (zz1)
+ fa(z))
- +Sw
2
20²
log pe (z| pog
obtain training data. Here 0, 1, 2o (2(2)) for posterior sampling and 20 g 2) for prior sampling. WP ~ N(0 Ia). Note that we plug-in pe (æt z) for the target distribution of posterior sampling LD. This is because given the observed data æ. by Bayes' rule know that
[=( z) = Pa(æ 2x (z). The whole learning iteration can be viewed as a variant of the variational EM algorithm [71].
S2

---

B Network Architecture and Training Details
The energy network Sa uses simple fully-connected structure throughout the experiments. We describe the architecture in details in Table S1. The generator network has a simple deconvolution structure similar DCGAN [72] shown Table S2. The denoising diffusion model is implemented by light-weight MLP-based U-Net [73] structure (Table S3). The encoder network embed the observed images has a fully convolutional structure [74], as shown Table S4.
Architecture
network. LReLU denotes the Leaky ReLU activation func- tion. The slope Leaky ReLU set to 0.2. For SVHN CelebA datasets, we use nz=100. For CIFAR- 10 and CelebA-HI datasets, we use nz=128. We use nz=8 for anomaly detection on the MNIST dataset, and n2-7168 for GAN inversion. We use ndf=512 for GAN inversion and nd0=200 for the rest experiments.
Table S1: Network structures of the energy
Layers
Out Size
Input:
n {8, 100, 128, 7168)
Linear, LReLU
ndf E {200, 512]
Linear, LReLU ndf {200, 512]
Linear
Table Network structures generator networks used for SVHN, CelebA, CIFAR-10 10, CelebA- HQ and MNIST (from top to bottom) datasets. For GAN inversion, we use the StyleGAN [10] structure as our generator network. ConvT(() indicates a transposed convolutional operation with n output channels. We use ngf=64 for the SVHN dataset and ngf=128 for the rest. LReLU indicates the Leaky-Rel.. ReUU activation function. The slope Leaky ReLU is be 0.2.
Layers
Out Size
Stride
Input:
1x1x100
4x4
ConvT(nngf x 8), LReLU
4x4x x 8)
1
4x4
ConvT(nngf x 4), L.ReLU
8x8x(ngf x 4)
2
4x4
ConvT(ngf x 2). LReLU
16x16x(ngf
2
4x4 ConvT(3), Tanh
32x32x3
2
Layers
Out Size
Stride
Input:
1x1x100
4x4
ConvT((ng' x 8). LReLU
4x4x(ngf x 8)
1
4x4
ConvT(ngf x 4). LReLU
8x8x(ngf 4)
2
4x4
ConvT((ngf x 2), LReLU
16x16x(ngf
2)
2
4x4
ConvT(ngf x 1), LReLU
32x32x(ngf x
2
4x4 ConvT(3), Tanh
64x64x3
2
Layers
Out Size
Stride
Input: z
Ixlx128
8x8 ConvT(ngf x 8), LReLU
8x8x(ngf x 8)
1
4x4
ConyT(ngg x 4), LReLU
16x16x(ngf x 4)
2
4x4
ConvT(nnf x 2). LReLU
32x32x(ngf
2
3x3 Cony T(3), Tanh
32x32x3
1
Layers
Out Size
Stride
Input:
1x1x128
4x4
ConvT(nnf x 16), LReLU
4x4x(ngf x 16)
1
4x4
ConvT(nng x 8), LReLU
8x8x(ngf x 8)
2
4x4
ConvT(nnf x 4), LReLU
16x16x(ngf x 4)
2
4x4
ConvT(ngf x 4), LReLU
32x32x(ngf
2
4x4
ConvT(ngf x 2). LReLU
64x64x(ngf x 2)
2
4x4
ConvT(ngf x 1), LReLU
128x (28x(gxxx
2
4x4 ConyT(()), Tanh
256x256x3
2
Layers
Out Size
Stride
Input:
1xlx8
7x7 ConvT(ngf x 8). LReLU
7x7x(ngf x 8)
1
4x4 ConvT(ngf x 4), LReLU
14x14x(ngf
2
4x4 ConvT(nng x 2), LReLU
28x28x(ngf x 2)
2
3x3 ConvT(1), Tanh
28x28x1
1
S3

---

Table S3: Network structure of the denoising difflusion network. a) We use the sinusoidal embedding embed the time index as [25, 8j. b) We use the learned fourier feature module [75] to embed the input z. c) The merged time embedding and context embedding is used to produce a pair of bias and scale terms to shift and scale the embedding of input z. nz is the input dimension, as in Table SI. nemb is the dimension of image embedding in Table S4,
Layers
Out size
Note
Time Embedding
Input:
time index
Sin. emb." 128
Linear, SiL.U 128
Linear
128
Input Embedding
Input: 2
nz
Fr. emb."
2x(nz)
Fourier feature
Basic Block nzf, nemb, 128
Input: z, Zat,, 2: Cat, SiLUU
2, ctx. and I emb.
nemb 128
merge ctx. & emb.
Linear, SiL.U
Linear
nout
Input emb. scalc-shift z emb. w/ merged emb. skip connection from
Scale-shift"
nout
Add
nout
Denoising Diffusion Network
Input: Zas, nz. nemb, 128
Input Input & emb.
Embedding 2x(nz), 128
Basic Block
128
Encoding
Basic Block
256
Basic Block
256
Basic Block
256
Intermediate Cat & Decoding
Basic Block
256
Basic Block
128
Basic Block
nz
Output
Hyperparameters and training details As mentioned in the main text, for the posterior and prior DAMC samplers, we set the number of diffusion steps to 100. The number of iterations in Eq. (8) is set to MM = 6 for the experiments. The LD runs T = 30 and T = 60 iterations for posterior and prior updates during training with a step size of 0.1. For test time sampling from KT.,, |m. 9s, (2,), we set T = 10 for the additional LD. For test time prior sampling of LEBM with LD. we follow [22, 41] and set T 100. To further stabilize the training procedure, we i) perform gradient clipping by setting the maximal gradient norm as 100, ii) use separate target diffusion network which is the EMA of the current diffusion network to initialize the prior and posterior updates and iii) add noise- initialized prior samples for the prior updates. These set-ups are identical across different datasets.
The parameters of all the networks are initialized with the default pytorch methods [77]. We use the Adam optimizer [78] with 31 = 0.5 and 32 = 0.999 to train the generator network and the energy score network. We use the AdamW optimizer [79] with B3 0.5. B3 0.999 and geight_decayse 1erray-ee-44 e--4 to train the diffusion network. The initial learning rates of the generator and diffusion networks are 20-4. 4, and 1e-4 for the energy score network. The learning rates are decayed with factor of 0.99 every 1K training iterations, with a minimum learning rate of 1e-5. We run the experiments on a A6000 GPU with the hatch size of 128. For GAN inversion, we reduce the batch size to 64. Training typically converges within 200K iterations on all the datasets.
S4

---

Table S4: Network structures of the encoder networks used for the SVHN, CelebA, CIFAR-I0, CelebA-HQ and MNIST (from top bottom) datasets. For GAN inversion, the encoder network structure is the same [18]. Conv(reJNorm indicates a convolutional operation with oulput channels followed by the Instance Normalization [76]. We use nif-64 and nemb=1024 for all the datasets. LReLU indicates the Leaky-Rela activation function. The slope in Leaky ReLU is set to be 0.2.
Layers
Out Size
Stride
Input: a
32x32x3
3x3 Conv(nif 1)Norm, LReLU
32x32x(nif x 1)
4x4 Conv(nif 2)Norm, LReLU
16x16x(nif 2)
2
4x4 Conv(nif x 4)Norm, LReLU
8x8x(nif 4)
2
4x4
Conv(nif 8)Norm, LReLU
4x4x(nif 8)
2
4x4 Cono(nemb)Norm, LReLU
1xIx(nemb)
1
Layers
Out Size
Stride
Input: æ
64x64x3
3x3 Conv(nif 1)Norm, LRReUU
64x64x(nif x 1)
4x4 Conv(nif x 2)Norm, LReLUU
32x32x(nif x 2)
2
4x4 Conv(nif 4)Norm, LReLU
16x16x(nif x 4)
2
4x4 Conv(nif 8)Norm, LReLU
8x8x(nif 8)
2
4x4 Conv(nif x 8)rrm,, LReLU
4x4x(nif 8)
2
4x4 (ommbnn))ooom,, LReLU
IxIx(nemb)
Layers
Out Size
Stride
Input: æ
32x32x3
3x3 Conv(nif x 1)Norm, LReLU
32x32x(nif x 1)
1
4x4 Conv(nif 2)Norm, IReLUU
1x x 2)
2
4x4 Conv(nif x 4)Norm, LReLU
8x8x 4)
2
4x4 Conv(nif 8)Norm, LReLU
4x4x(nif 8)
2
4x4 (ommbneb))oomm, LReLU
IxIx(nemb)
1
Layers
Out Size
Stride
Input: x6
256x256x3
3x3 Conv(nif x 1)Norm, LReLU
256x256x(nif x 1)
1
4x4 Conv(nif 2)Norm, LReLU
128x128x(nif
2
4x4 Conv(nif x 4)Norm, LReLU
64x64x(nif x 4)
2
4x4 Conv(nif 4)Norm, LReLU
32x32x(nif x 4)
2
4x4 Conv(nif x 8)Norm, LReLU
16x1 x 8)
2
4x4 Conv(nif 8)Norm, LReLU
8x8x(nif 8)
2
4x4 Conv x 8)Norm, LReLU
4xix 8)
2
4x4 (om LReLU
Ixlx(nemb)
Layers
Out Size
Stride
Input: æ
28x28x3
3x3 Conv(nif 1)Norm, LReLU
28x28x(nif x 1)
1
4x4 Conv(nif x 2)Norm, LReLU
14x14x(nif x 2)
2
4x4 Conv(nif 4)Norm, LReLU
7x7x(nif 4)
2
4x4 Conv(nif x N)rmm, LReLU
3x3x 8)
2
3x3 Nonv(nembNNomm, LReLU
1xIx(nemb)
1
S5

---

C Pytorch-style Pseudocode
We provide pytorch-style pseudocode help understand the proposed method. We denote the gen- erator network as G, the energy score network as E and the diffusion network as Q. The first page sketches the prior and posterior sampling process. The second page outlines the learning procedure.
Listing 1: Prior and posterior LD sampling.
def sample_langevine langevin__proor (zz netE):
s = size
for in range(n_steps):
en netE (z). sum ()
z_norm 1.0 / 2.0 torch s
z_grad torch, autograd grad (en z_norm, 2)[0]
w torch randn_like((
# Prior LD Update
z data z. data - 0.5 (s ** 2) z_grad +
return 2. detach
def sample sample_langevin pesevvinposterior.coo z (z, x, netG, netE):
step_size size
sigma_inv 1.0 / (2.0 sigma ** 2)
for i in range (n_steps)
x hat netG
g g_log_lkhd lkhd = sigma_iny inv torch sum ((x (x_hat - x) ** 2)
z_n = 1.0 / 2.0 torch (2**)))
en netE (z).sum(
total en g l_llo______ lkhd en + z_n z_grad torch. autograd grad (total_en, 2)(0]
w torch randn_like
# Posterior LD Update
data 2. data 0.5 (s ** 2) z_grad +
return 2. detach ()
S6

---

Listing 2: Learning LEBM with DAMC.
for x in dataset:
# mask for unconditional learning of DAMC torch rand ((len(x),), device=x. device) 7_mask torch ones len (x), device=x. device)
z_ f_mask_prot
7_mask 2______a__p___ < 0.2] 0.0
7_mask 7_mask. unsqueeze
# draw DAMC samples z0 = Q(x)
zk zo_pos, zk neg
zk_neg 20. detach () clone O), z0. detach O) clone (
# prior and posterior updates
zk pos = sample_ saagevin_postervtee 2
zk_ neg = z=zk_pos, x=x, netG=G, netE=E)
z=torch cat(
[zk_neg torch randn like ( (zk_neg ) dim=0), netE=E)
# update
for
in range (6):
0_optimizer zeroeeeoeee _rad
Qlloss Q. calculateeeeaaee lossaee_los(( c
x=x, z=zk_pos, mask=7_mask) mean ()
Q loss backward
_optimizer. step
# update
G G_optimizer. zero_grad ()
x hat = -G(z_poss
g loss torch sum ((x hat x) ** 2, dim =[1 (1,2,,33 _2,3)) mean ()
llsss. backward
G G_optimizer step ()
# update B_optimizer. zero_grad () e_pos, e_neg E(zk_pos), B B(zk_neg) B_loss = e_pos, mean () e_neg mean ()
B_loss backward
E __optimizer.. step ()
S7

---

D Dataset and Experiment Settings
Datasets We include the following datasets study our method: SVHN (32 32 3), CIFAR- 10 (32 x 32 x 3), CelebA (64 x 64 x 3), CeleAMask-HO (256 x 256 3) and MNIST (28 x 28 1). Following Pang et al. [22], we use the full training set of SVHN (73,257) and CIFAR-10 (50,000), and take 40,000 samples of CelebA as the training data. We take 29,500 samples from CelebAMask-HQ dataset as the training data, and test the model on 500 held-out samples. For anomaly detection on MNIST dataset, we follow the experimental settings in [22, 41, 55, 56] and use 80% of the in-domain data to train the model. The images are scaled to [-1, and randomly horizontally flipped with prob. of .5 for training.
GAN inversion settings We attempt to use the DAMC sampler for GAN version on the FFHQ (256 x 256 x 3) and LSUN-Tower (256 x 256 x 3) datasets. We take 69,500 samples from the CelebAMask- dataset as the training data, and use the held-out 500 samples for testing. We follow the default data splits of the LSUN dataset.
For the LEBM- based inversion method, train LEBM in the 14 512 dim. latent space. During training. we add Iz-regularization on the energy score of LEBM to stabilize training, as suggested in [59]. For the DAMC sampler and the encoder-based inversion method [16], we generate initial posterior samples of the training data using [18] and train the DAMC sampler and the encoder- based method for 5K iterations using these samples
warm-up step. The encoder-based method is trained by minimizing the l2 distance between the encoder output and the target samples. After that, these methods trained with the default learning algorithms.
S8

---

E
Additional Qualitative Results
E.1
Toy Examples
Figure S1: The 2-arm pinwheel-shaped prior distribution used the toy example.
(a) Evolution of the learned posterior distributions
(b) GT
Figure S2: Evolution of the posterior distributions learned the DAMC sampler. In each row, display from left right evolution of learned posterior distributions from the DAMC sampler through training iterations. The last column shows the corresponding ground-truth posterior distribution obtained by running IK-3K steps of Langevin Dynamies until convergence for posterior sampling.
As the proof-of-concept toy examples, we implement the neural likelihood experiments following the same set-up mentioned in Section 5.1 and B. in [80]. We choose to more complex prior distribution, i.e., 2-arm pinwheel-shaper prior distribution (shown in Fig. S1) instead of a standard normal one to make sure that the true posterior distributions are multimodal. The ground-truth poste- rior distributions are obtained by performing long-run LD sampling until convergence. We visualize the convergence trajectory of the posterior distributions learned by model to the ground-truie ones. Fig. S2, we can see that our model can faithfully reproduce the ground-truth distributions with sufficient training iterations, which indicates that the learned sampler successfully amortizes the long-run sampling chain with the length of 1, 00003,03 000 iterations.
S9

---

E.2
Generation
We provide additional generated samples from our models trained on SVHN (Fig. S3), CelebA (Fig, S4), CIFAR-10 (Fig. S5) and CelebA-HQ (Fig. S6).
6
0
)
6
3
6
16
1
8
3
5
8
0
8
3 3
0
06
1
1
C.
I
4
10
6 1
4
5
0
7
9
3
8
11
1
01
4
4
0
2
666
00
S
L&SS
9
3
1
9
2
1
1100
6
0
12
8
5
2
18 233
2
1
57
0
4
X
6
8
4516
16
319
2 161
SL
6
1
22
6
1
C 8
3
0
5
15
1
2
S
9
6
3
2
8
3
44
S
18
(
1
0
2
4
8
(a) Ours-DAMC
(b) Ours-LEBM
Figure S3: Samples generated from the DAMC sampler and LEBM trained on the SVHN dataset.
(a) Ours-DAMC
(b) LE
Figure S4: Samples generated from the DAMC sampler and LEBM trained on CelebA dataset.
E.3 Reconstruction
We provide qualitative examples about the reconstruction results from our models trained on SVHN (Fig. S7), CelebA (Fig. S8), CIFAR-10 (Fig. S9) and CelebA-HQ (Fig. S10). Observed images are sampled from the testing set unseen during training.
S10

---

(a) Ours-DAMC
(b) Ours-LEBM
Figure S5: Samples generated the DAMC sampler LEBM trained on the CIFAR-10 dataset.
Ours-DAMC
Ours-LEEBN
Figure S6: Samples generated from the DAMC sampler LEBM trained on the CelebA-HQ dataset.
E.4 Visualization of Transitions
We provide additional visualization results of LD transitions initialized from A Ia) on SVHN (Fig. S11) and CIFAR-10 10 datasets (Fig. S12). For the 200-step set-up, we can see that the generation quality quickly improves by exploring the local modes with LD. For the 2500-step long-run set-up, we can see that the LD produces consistently valid results without the oversaturating issue of the long-run chain samples.
$11

---

5
2
0
1
9
1
5
2
0
1
9
1
18
1833
1
4
4
1833
6
5
1
4
4
6
344
2
1919
6
3
4
2:0
1919
2
5
4
14
2
8
2
5
4
14
9
600
1151
600
155p1
166
9
20
9
166
9
20
2
6559
1515
0
659
1515
9
8
8
51 5
2999
57
57
29
1
1
(a) Observation
(b) Reconstruction
Figure S7: Reconstructed samples from the posterior DAMC sampler trained on the SVHN dataset.
(a) Observation
(b) Reconstruction
Figure S8: Reconstructed samples from the posterior DAMC sampler trained on the CelchA dataset.
S12

---

(a) Observation
(b) Reconstruction
Figure $9; Reconstructed samples from the posterior DAMC sampler trained the CIFAR-10 datasct.
(a) Observation
(b) Reconstruction
Figure S10: Reconstructed samples from the posterior DAMC sampler trained the CelebA-IIQ dataset.
3
1 13
19
JR
2
2
8183
26
91
1 4
58 CX A:
13
(a) steps
(b) 2500 steps
Figure S11: Transition of Markov chains initialized from (0, 1.) towards (2) SVHN. We present
results by running LD for 200 and 2500 steps. each sub sub- figure, the top panel displays the trajectory in the data space uniformly sampled along the chain. The bottom panel shows the energy score (=) over the iterations.
$13

---

(a) 200 steps
(b) 2500 steps
Figure S12: Transition of Markoy chains initialized from ((0, towards (z) CIFAR-10. We present results by running LD for and 2500 steps. each sub-figure, the top panel displays the trajec- tory in the data space uniformly sampled along the chain. The bottom panel shows the energy score fa(z) over the iterations.
S14

---

F Additional Quantitative Results
F.1 Learning DAMC in the Latent Space of Other DLVMs
We have compared our method with directly training diffusion models in the latent space. In the NALR-LEBM column in Table we compare our method with learning diffusion model in the pre-trained energy-based latent space on CIFAR-10 dataset. We further add experiments of learning this model in the pre-trained VAE [1] and ABP [51] model latent spaces summarized Table S5, We can see that our method greatly outperforms these baselines using the same network architectures.
Table S5: Further baseline results for learning DAMC CIFAR-10 dataset. These models are imple- mented using the same encoder and decoder network architectures for fair comparison
VAE
ABP
NALR
Ours
FID
102.54
69.93
64.38
57.52
MSE
0.036
0.017
0.016
0.015
F.2 Further Ablation Studies
For prior LD We conduct ablation studies inspect the impact of some experiments for fair com- sampling, we follow [22] and set the sampling step as 60 throughout the parison. We further add experiments LD sampling steps T and with different capacities of the amortizer qá. summarized in Table S6. We observe that larger posterior sampling steps and larger model capacities for training brings marginal improvement compared with our default set-up. Less sampling steps and lower model capacities, however, may have negative impact on the performances.
CIFAR-10 for training our model with different posterior
(a) Posterior sampling steps T
(b) Model capacity of go
T=10
T=30
T=50
f=1/4
f=1/2
f=1
f=2
f=4
FID
74.20
57.72
57.03
FID
116.28
80.28
57.52
57.82
57.56
MSE
0.016
0.015
0.015
MSE
0.017
0.016
0.015
0.015
0.015
Table S6: Ablation Studies for the choice of Langevin steps and model capacity. We highlight the results of set-up reported in the main text. T the posterior sampling step. stands for the factor for the model capacity, egg., f-2 means 2x the size of the original model.
S15

---

G Further Discussion
G.1 Limitations
We mentioned in the main text that one potential disadvantage of our method is its parameter ineffi- ciency for introducing extra DDPM. Although fortunately, our models are the latent space so the network is lightweight. To be specific, on SVHN, CelebA, CIFAR-IO and CelebA-HQ datasets the number of parameters the diffusion network is around 10% of those the generator.
Another issue is the time efficiency. We mentioned in the main text that the time efficiency for sampling competitive. With the batch size of 64, these datasets the DAMC prior sampling takes 0.38, while 100 steps of short-run LD with LEBM takes 0.2s. The DAMC posterior sampling takes 1.0s, while LEBM takes 8.0s. However, during training need run 30 steps of posterior LD sampling and 60 steps of prior LD sampling in each training iteration. We observe that the proposed learning method takes 15.2 minutes per training epoch, while the short-run LD-based learning method takes 14.8 minutes per epoch. These methods are slower than the VAE-base method, which takes 5.5 minutes for an training epoch. We can see that the time efficiency for training is generally bottlenecked by the LD sampling process, and could be improved in future works.
G.2 Broader Impacts
Generative models could be misused for disinformation faking profiles. Our work focuses the learning algorithm of energy-based prior model. Though we consider our work to be foundational and not tied to particular applications or deployments, it is possible that more powerful energy- based generative models augmented with this method may be used maliciously. Work on the reliable detection of synthetic content could be important address such harms from generative models.
S16


Reviews:
[{"review_id": "E3JoziPdWQ", "reviewer": "Reviewer_x1ko", "summary": "The paper proposes a diffusion-based amortized MCMC method for sampling the prior and posterior in latent space energy-based models. The paper provides some theoretical evidence using directly the result from Li et al., 2017. The paper shows the effectiveness of the proposed method throughout an extensive campaign on a variety of tasks such as image generation and construction, anomaly detection.", "questions": "- There are a lot of hyper-parameters in the proposed methods. How do the authors choose $T$?\n- It would be great if the authors could ablate the amortization gap of the proposed method by considering different capacities of the $q_{\\phi}$", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper aims at tackling an important task in deep latent variable models, which is to improve energy-based prior model.\n- The paper is well-motivated and is well-written.\n- The experiments on image datasets are extensive and diverse.", "weaknesses": "- The idea of amortizing the short-run MCMC sampling of the prior and posterior distributions is already proposed in the original work of Pang et al., 2020. This paper aims to improve this amortization using a diffusion model. In addition, the theoretical contribution is weak, as the convergence property is already shown by Li et al., 2017.\n- Although the authors claim that the theoretical evidence that the learned amortization of MCMC is a valid long-run MCMC sampler, this result is taken directly from Li et al., 2017. As such, the authors should show that the proposed amortization scheme can converge to the true distributions, at least via toy examples. For example, the authors could use examples from this work [1].\n- The improvement is expectable as the authors use a very powerful diffusion model to amortize the prior and posterior. However, this increases the computational costs, as the authors discussed at the end of Section 4.1.\n- As, again, the authors employ a power diffusion model in latent space. To be fair, the authors should consider the baselines of using diffusion models in the latent space of latent variable models; for example [2], at the very least.\n\n[1] Taniguchi et al., Langevin Autoencoders for Learning Deep Latent Variable Models. NeurIPS 2022.\n\n[2] Pandey et al., DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents. TMLR 2022.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your detailed and insightful comments\nWe sincerely thank you for your time and constructive comments! Below, we provide point-to-point replies to your comments that hopefully would address the concerns you have.\n\n- > The idea of amortizing the short-run prior and posterior MCMC sampling is already proposed in [1].\n    - As mentioned in the 2.6 and A.11 sections of [1], they propose to amortize the posterior sampling process with the \n      network $q_{\\phi}(z|x)$ following the variational learning (VI) scheme introduced by [2]. However, the Gaussian (or other tractable density) assumption \n      of the posterior distribution made in this scheme could greatly limit the expressivity of $q_{\\phi}(z|x)$ [3,4]. In this work, we use a conditional diffusion model to address this issue. As shown in tab. 1, we have compared our model with the NCP-VAE baseline, which indeed trains the inference network through VI to learn the energy-based prior. We can see that our method significantly outperforms the NCP-VAE. On the CIFAR-10 dataset, the FID and MSE scores are 78.06 vs. 57.72, and 0.054 vs. 0.015.\n    - In the A.11 section, [1] mentioned that one can learn a synthesis network $q_{\\psi}(z)$ for prior sampling, but did not provide a concrete solution for this problem. Model design for amortizing the prior sampling requires non-trivial \n      extra efforts, since the latent space is ever-changing during training; delicate designs are needed to balance the learning of this amortized model and other jointly learned models for stable training. In this work, we follow the method of [5] to train a single neural network to parameterize both the prior and posterior sampling models to deal with this problem.\n\n- > The authors should show that the proposed this amortization scheme can converge to the true distributions, at least via toy examples in [4].\n    - Thank you for pointing us to this very interesting example! We have implemented the neural likelihood examples following the same set-up mentioned in sec. 5.1 and B.1 in [6]. We choose to use a more complex prior distribution, i.e., 2-arm pinwheel-shaped prior distribution instead of a standard normal one to make sure that the true posterior distributions are multimodal. We have attached the visualization of the convergence of the learned posterior distributions by our model to the ground-truth ones (please see the global response). We can see that our model could faithfully reproduce the ground-truth distributions. We will add these results in revision and make explicit connections to [4].\n\n- > The authors use a very powerful diffusion model to amortize the prior and posterior. However, this increases the computational costs.\n    - We would like to point out that we actually used a very light-weight MLP-based diffusion for all the experiments in this paper, since our models are in the low-dimensional latent space. As mentioned in sec 4.1 L257-258 in the main text and G.1 in the appendix, the number of parameters in the diffusion model is only around 10% of those in the generator.\n    - Consequently, the computational costs induced by this model is very marginal. To be specific, as mentioned in sec 4.1 L258-261, with the batch size of 64, our method takes ~0.3s for prior sampling, while 100 steps of short-run LD with LEBM takes 0.2s; posterior sampling with the proposed method takes ~1.0s, while posterior LD sampling takes ~8.0s because it requires back-propagations through a much heavier generator network.\n\n- > Should consider the baselines of using diffusion models in the latent space of latent variable models; for example [6].\n    - Thank you for pointing us to this very interesting work! However, we notice that DiffuseVAE actually trains a diffusion model in the pixel space to refine the output of a VAE model. This is essentially different from our set-up, where we learn the models in the latent space of generative models. We will dicuss the key differences between DiffuseVAE and our work in revision.\n    - We have compared our method with directly training diffusion models in the latent space. In the NALR-LEBM column in tab. 4 in sec 4.3 and L292-300, we compared our method with learning a diffusion model in the pre-trained energy-based latent space on CIFAR-10 dataset. We further add experiments of learning this model in the pre-trained VAE and ABP model latent spaces summarized below. We can see that our method greatly outperforms these baselines using the same network archs.\n||VAE|ABP|NALR|Ours \n|:--:|:--:|:--:|:--:|:--:|\n|FID|102.54|69.93|64.38|**57.52**|\n|MSE|0.036|0.017|0.016|**0.015**|\n\n- > How do the authors choose $T$? Would be great to consider different capacities of the $q_{\\phi}$.\n    - Thank you for bringing this matter to our attention! For prior sampling, we follow the set-up of [1] for fair comparison. We further add experiments on CIFAR-10 for different posterior steps and capacities of $q_{\\phi}$ summarized below. We used $T=30$ for training in this paper. We can see that larger $T$ and larger $q_{\\phi}$ brings very marginal improvement. \n||T=10|T=30|T=50|\n|:--:|:--:|:--:|:--:|\n|FID|74.20|**57.72**| 57.03|\n|MSE|0.016|**0.015**| 0.015|\n    - Here $f$ stands for the factor for the $q_{\\phi}$ capacity, e.g., $f=2$ means 2x the size of the original model. \n||f=1/4 |f=1/2 |f=1|f=2|f=4 \n|:--:|:--:|:--:|:--:|:--:|:--:|\n|FID|116.28| 80.28|**57.52**|57.82|57.56|\n|MSE|0.017| 0.016|**0.015**|0.015|0.015|\n\nThank you again for providing constructive and detailed feedback on our submission. If you have any additional questions/comments/concerns, please feel free to let us know here. **Otherwise, we would appreciate it if you would consider raising your rating of this submission**.\n\n[1] Pang et al. LEBM. NeurIPS 2020.\n\n[2] Kingma et al. VAE. 2013\n\n[3] Rajesh et al. HVAE. ICML 2016.\n\n[4] Taniguchi et al. LAE. NeurIPS 2022.\n\n[5] Ho et al. Classifier-Free Guidance. NeurIPS 2021 Workshop\n\n[6] Pandey et al., DiffuseVAE. TMLR 2022"}, {"review_id": "LwrXkUnGvZ", "reviewer": "Reviewer_ga25", "summary": "This paper proposed a diffusion-based amortised method to address the short-run MCMC samplers issues in the latent-space energy-based models. One interesting part is that it interleaves the distill T-steps of Langevin dynamics and KL divergence minimisation to sample from the target distribution $\\pi$. Regarding the choice of amortised sampler parameters $\\phi$, it uses the gradient of DDPM objective function to optimise them. The experimental results has verified the effectiveness of the DAMC. ", "questions": "Sorry I am not an expert in this topic. I do not have questions for the authors. ", "limitations": "I believe the authors has addressed the limitations. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has the following strengths:\n\n**1**, the usages of KL minimization and amortised MCMC transition is a well defined framework. Also, the usages of DDPM gradient for the long-run MCMC sampling in LEBMs is interesting. \n\n**2**, the experimental evaluation is sufficient and clearly verifies the advantages of DAMC.  ", "weaknesses": "Regarding the weakness, I think the paper may need another round of polish. Some notations are not well defined before use and some introductory statement is not consistent. For example:\n\n**1**, p_{uncond} may need more explanation, rather than a short word in the input of Algorithm. \n**2**, $z_+^{(i)}, z_-^{(i)}$ are a bit confusing. \n**3**, it may be a better idea if the notations of section 2.1 and 2.2 are consistent", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your constructive comments\nWe sincerely thank you for your kind words and thoughtful comments! Below, we provide point-to-point responses to hopefully address the concerns you have.\n\n- > p_{uncond} may need more explanation, rather than a short word in the input of Algorithm. $z_{+}$ and $z_{-}$ are a bit confusing.\n    - Thank you for pointing this out. We have introduced the meaning of p_{uncond} in sec 3.3 L175-180. $z_{+}$ and $z_{-}$ denotes the posterior and prior samples respectively. \n    - We agree that more explicit explanation of the hyperparameter and notations could make this draft more readable. We will make sure to revise our paper accordingly.\n\n- > It may be a better idea if the notations of section 2.1 and 2.2 are consistent\n    - Thank you for bringing this matter to our attention! We choose these notations to make sure that they are consistent with those used in Fig 1. We will revise these notations according to your kind suggestion.\n\nThank you again for providing constructive and detailed feedback on our submission! If you have any additional questions/comments/concerns, please feel free to let us know here."}, {"review_id": "r0fQRhCCgY", "reviewer": "Reviewer_NcyD", "summary": "The authors propose DAMC, an amortization of MCMC sampling, via a scheme based on diffusion models, as an alternative to pure MCMC sampling, which usually suffers from either long mixing time or from being short and biased,for  priors and posteriors in energy based models. The method is theoretically sound, and experimentally argued for.", "questions": "Are there some constraints that the latent space needs to satisfy for the proposed method to have an advantage over the vanilla MCMC counterpart ?", "limitations": "I see None for now.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The theoretical analysis is sound. The algorithm is clear. The experiments are convincing.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your insightful comments\nWe sincerely thank you for your kind words and thoughtful comments! Below, we provide point-to-point responses to hopefully address the concerns you have.\n\n- > Are there some constraints that the latent space needs to satisfy for the proposed method to have an advantage over the vanilla MCMC counterpart?\n    - Thank you for this very insightful input! For now, we see no obvious constraints on the latent space for the proposed method to outperform the vanilla short-run MCMC counterpart. We observed that our method shows a clearer advantage over the vanilla short-run MCMC method when the target distribution is highly multimodal or generally hard for short-run MCMC to fully explore.\n    - We also see no obvious issues for now on generalizing our method to learning amortized samplers for (unnormalized) densities or distributions other than learning energy-based models (including those arise in molecule dynamics and simulated annealing optimization). We are excited about these problem as a direction for future work.\n\nThank you again for providing constructive and thoughtful feedback on our submission. If you have any additional questions/comments/concerns, please feel free to let us know here."}, {"review_id": "uKcxB6Jzkc", "reviewer": "Reviewer_hz9t", "summary": "This paper proposes the DAMC sampler (Diffusion-Amortized MCMC) and develops a new learning algorithm for LEBM (Latent-space Energy-Based Model) based on it. Theoretical and empirical evidences are provided for the effectiveness of our method. \n", "questions": "see above.", "limitations": "see above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is generally well-written. The idea of amortizing the LD with DDPM in learning the LEBM seems to be new. \n", "weaknesses": "The basic idea of using an auxiliary model to amortize the LD in learning energy-based models has been well known and explored in the literatures, e.g. in [a,b] to name a few. Methodology connection and experiment comparison with those methods are needed.\n\nBy looking at FID on CIFAR-10 (Table 1), the performance of the proposed method (FID 57.72) is far behind previous methods [a,b] (33.61, 20.9).\n\nIn the current literature, long-run MCMC analysis is typically conducted over a range of 10,000 to 100,000 Langevian Dynamics iterations. However, in this paper (Figure 4), the authors perform a much shorter 2,500 Langevian dynamics update, which is considerably less extensive compared to other studies in the field.\n\n[a] Cooperative Training of Descriptor and Generator Networks, arXiv:1609.09408v3\n[b] Learning Neural Random Fields with Inclusive Auxiliary Generators, arXiv:1806.00271v4\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your insightful comments\nWe sincerely thank you for your time and thoughtful comments! Below, we provide point-to-point replies to your comments that hopefully would address the concerns you have.\n\n- > Methodology connection and experiment comparison with existing methods using an auxiliary model to amortize the LD, e.g., [a] and [b] are needed. The FID scores on CIFAR-10 are far behind [a] and [b]\n    - Thank you for pointing us to these interesting works! We have generally introduced the amortized MCMC methods including those using an auxiliary model in sec. A in the appendix. We will make sure to explicitly connect and discuss these methods including [a,b] in revision.\n    - We would like to point out that both [a] and [b] trains the auxiliary models and energy-based models in the pixel space, which is essentially different from our set-up. In this paper, we focus on learning the energy-based prior and its corresponding posterior and prior sampler in the latent space of generative models. One of the key differences is that model design for learning the latent space samplers requires non-trivial extra efforts, since the latent space is ever-changing during training unlike the data space; delicate designs are often needed to balance the learning of this amortized model and other jointly learned models for stable training, if we want to employ these pixel space methods. In this work, we follow the method of [c] to train a single neural network to parameterize both the prior and posterior sampling models to deal with this problem.\n    - In addition, as mentioned in sec. 3.3 L184, we use the same network architectures as in previous works [d,e] for fair comparison with these methods. To be specific, as shown in sec. C in appendix the generator network used in this work for the CIFAR-10 experiment uses 5 layers of transposed convolution, while the generator network in [b] uses a ResBlock-based deeper generator. The FID scores are therefore not directly comparable between ours and those reported in [a] and [b]. We hope to explore more advanced architectures to address these issues in future work.\n\n- > In the current literature, long-run MCMC analysis is typically conducted over a range of 10,000 to 100,000 Langevian Dynamics iterations.\n    - Thank you for bringing this matter to our attention! We checked the performance of our model for longer LD chain on CIFAR-10, SVHN and CelebA64 datasets by calculating the FID scores of long-run samples. The results are summarized below. We can see that our model could produce consist results with long-run chains.\n|Steps|100|200|2500|10000|100000\n|:--|:--:|:--:|:--:|:--:|:--:|\n|CIFAR-10|60.89|60.73|61.20|60.76|61.20|\n|SVHN|21.17|20.91|20.07|20.68|20.71|\n|CelebA64|35.67|35.39|35.40|35.29|35.17|\n\nThank you again for providing constructive and thoughtful feedback on our submission. If you have any additional questions/comments/concerns, please feel free to let us know here. **Otherwise, we would appreciate it if you would consider raising your rating of this submission**.\n\n[a] Cooperative Training of Descriptor and Generator Networks, arXiv:1609.09408v3\n\n[b] Learning Neural Random Fields with Inclusive Auxiliary Generators, arXiv:1806.00271v4\n\n[c] Classifier-Free Diffusion Guidance. NeurIPS 2021 Workshop.\n\n[d] Learning latent space energy-based prior model. NeurIPS 2020.\n\n[e] Adaptive multi-stage density ratio estimation for learning latent space energy-based model. NeurIPS 2022."}, {"review_id": "m7evbjODAk", "reviewer": "Reviewer_rkuE", "summary": "The paper presents a new training and sampling procedures for learning energy based generative models.  The method is compared to earlier (few-step) MCMC based approaches and to diffusion models.  The procedure is evaluated on computer vision tasks.", "questions": "No further questions.", "limitations": "Limitations are adequately addressed.", "rating": 3, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents an original algorithm, with a clear methodological description, and a large collection of benchmarking results.", "weaknesses": "The paper could make clearer for the general audience why additional energy based modeling methodology is needed.  As a researcher who has been working extensively on diffusion models for the past two years, but has not worked with (other) energy based models in approaches in recent years additional context in goals of this direction and the limitations of previous approaches would be helpful.\n\nFor example, the authors comment that EBMs are \"closely related to\" DDPMs.  Which do not suffer from the \"sampling issue\" associated with \"non-convergent short-run MCMC\" used for EBMs.   Is there something that can be accomplished with EBMs that is not solved by DDPMs that motivates their continued study?  \n\nThe paper is also unclear with respect to the claim for \"theoretical evidence that the learned amortization of MCMC is a valid long-run MCMC sampler\".  Could the authors make this theoretical claim more explicit (e.g. as a proposition or theorem)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your detailed comments\nWe sincerely thank you for your time and detailed comments. Below, we provide point-to-point responses to hopefully address the concerns you have.\n\n- > Additional context in goals of this direction and the limitations of previous approaches would be helpful. Is there something that can be accomplished with EBMs that is not solved by DDPMs that motivates their continued study?\n    - Thank you for bringing this matter to our attention. We have generally introduced the background and related works of this direction in sec. 1 and 2 in the main text and sec. A in the appendix. We will make sure to more explicitly discuss these related works in revision.\n    - An example of \"something that can be accomplished with EBMs that is not solved by DDPMs\" can be found in [1] with a brief but insightful discussion in sec. 4.2 of [1]. For tasks that requires explicit likelihood function, e.g., compositional generation, the energy-based parameterization (EBM) is preferred over epsilon-parameterization (DDPM) for its flexibility in modeling. This energy-based parametrization enables the use of more accurate samplers for greatly improved sample quality and convergence in the compositional generation task compared with DDPM. \n    - More broadly speaking, there are certain distributions where the epsilon-parameterization cannot generate decent samples [2]. With the standard epsilon-parameterization one can only utilize unadjusted samplers, which perform well in practice. But for target distributions such as those with lighter-than-Gaussian tails [2], the unadjusted sampler chain is transient and may not produce desired samples. Our method in this submission learns both the latent space EBM and its corresponding amortized sampler, thus combining the strengths of both sides, i.e., a well-learned explicit (unnormalized) density and a strong sampler that greatly mitigate the sampling issue.\n\n- > Could the authors make the theoretical claim more explicit (e.g. as a proposition or theorem)?\n    - Thank you for pointing this out. We will consider revising our paper accordingly.\n\nWe hope our answers could further address the reviewer’s concerns of this work. If you have any additional questions/comments/concerns, please feel free to let us know here.\n\n[1] Du et al. Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC. ICML 2023.\n\n[2] Roberts and Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, pp. 341–363, 1996.\n"}]

Author rebuttal:
### Summary of our response
We thank the reviewers for their insightful and constructive comments and careful reviews of our paper! We appreciate that the reviewers consider our submission "well-written and well-motivated", "clearly stated", "new" and "interesting" and provide "convincing", "diverse" and "extensive" experiment results. We have provided point-to-point replies to your comments that hopefully would address the remaining concerns you have. We summarize our response as follows:
- Clarification
    - Key differences between our method and the ideas mentioned in [1, 2] and [3, 4]. (Response for reviewer x1ko and hz9t)
    - Discussion for advantages of energy-based parametrization over epsilon-parameterization (Response for reviewer rkuE)
- Further Experiments
    - Toy example from [5] as a proof-of-concept to show that the proposed amortization scheme can converge to the true distributions. Please see the attached PDF file for the additional results. (Response for reviewer x1ko)
    - Adding more baselines including training DDPMs in the latent spaces of VAE and ABP models using the current network architectures. (Response for reviewer x1ko)
    - Adding ablation studies for steps of Langevin Dynamics $T$ and the model capacity of the amortizer $q_{\phi}$. (Response for reviewer x1ko)
    - Adding quantitative results for longer-run chain analysis. (Response for reviewer hz9t).
- Including missing references pointed out by reviewers.
- Polishing notations and writing in general as suggested by reviewers.

Thank you again for providing constructive and detailed feedback on our submission. If you have any additional questions/comments/concerns, please feel free to let us know.

[1] Pang et al. Learning Latent Space Energy-Based Prior Model. NeurIPS 2020.

[2] Pandey et al. DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents. TMLR 2022.

[3] Xie et al. Cooperative Training of Descriptor and Generator Networks. TPAMI 2018.

[4] Song and Ou. Learning Neural Random Fields with Inclusive Auxiliary Generators. arXiv:1806.00271.

[5] Taniguchi et al. Langevin Autoencoders for Learning Deep Latent Variable Models. NeurIPS 2022.

Meta-review: The paper offers an innovative approach to address the sampling issue found when learning latent space Energy-Based Models (EBMs). By using a diffusion-based amortized Markov Chain Monte Carlo (MCMC) method, the paper aims to enhance the sampling of the prior and posterior in such models.

The paper provides extensive simulations and convincingly shows the effectiveness of the proposed method, comparing favourable to other leading techniques. Furthermore, the application of Denoising Diffusion Probabilistic Models (DDPM) gradient for long-run MCMC sampling in Latent Energy-Based Models (LEBMs) is an interesting choice, indicating the integrated use of established methodologies for more efficient and effective sampling.

Two reviewers were leaning towards a rejection -- after careful examination of these particular reviews, it was concluded that the objections raised were based on an extremely superficial reading of the manuscript. It is demonstrated by the shortness of these particular reviews (and their content!), the low confidence of the reviewers, as well as the discussions with the authors during the rebuttal period.

Overall, the paper is recommended for acceptance as it successfully tackles a critical challenge in the domain of deep latent variable models, namely the design of more efficient methodologies for learning  energy-based prior models. The proposed is a promising contribution to the field of generative modeling.
Decision: Accept (poster)