Learning Energy-based Model via Dual-MCMC Teaching
Authors: Jiali Cui
Tian Han
Venue: NeurIPS 2023 poster (2023)
Keywords: Energy-based model
MCMC
Joint-training
Generator model

Abstract: This paper studies the fundamental learning problem of the energy-based model (EBM). Learning the EBM can be achieved using the maximum likelihood estimation (MLE), which typically involves the Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics. However, the noise-initialized Langevin dynamics can be challenging in practice and hard to mix. This motivates the exploration of joint training with the generator model where the generator model serves as a complementary model to bypass MCMC sampling. However, such a method can be less accurate than the MCMC and result in biased EBM learning. While the generator can also serve as an initializer model for better MCMC sampling, its learning can be biased since it only matches the EBM and has no access to empirical training examples. Such biased generator learning may limit the potential of learning the EBM. To address this issue, we present a joint learning framework that interweaves the maximum likelihood learning algorithm for both the EBM and the complementary generator model. In particular, the generator model is learned by MLE to match both the EBM and the empirical data distribution, making it a more informative initializer for MCMC sampling of EBM. Learning generator with observed examples typically requires inference of the generator posterior. To ensure accurate and efficient inference, we adopt the MCMC posterior sampling and introduce a complementary inference model to initialize such latent MCMC sampling. We show that three separate models can be seamlessly integrated into our joint framework through two (dual-) MCMC teaching, enabling effective and efficient EBM learning.

Learning Energy-based Model via Dual-MCMC Teaching
Jiali Cui, Tian Han
Department of Computer Science, Stevens Institute of Technology {jeui7, thann)odteevns..dd
Abstract
This paper studies the fundamental learning problem of the energy-based model (EBM). Learning the EBM can be achieved using the maximum likelihood esti- mation (MLE), which typically involves the Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics. However, the initiiintia Langevin dynamics can be challenging in practice and hard to mix. This motivates the explo- ration of joint training with the generator model where the generator model serves as a complementary model to bypass MCMC sampling. However, such method be less accurate than the MCMC and result in biased EBM learning. While the generator can also serve as an initializer model for better MCMC sampling, its learning can be biased since it only matches the EBM and has no access empirical training examples. Such biased generator learning may limit the potential of learning the EBM. To address this issue, we present joint learning framework that interweaves the maximum likelihood learning algorithm for both the EBM and the complementary generator model. In particular, the generator model learned by MLE to match both the EBM and the empirical data distribution, making it a more informative initializer for MCMC sampling of EBM. Learning generator with observed examples typically requires inference of the generator posterior. To ensure accurate and efficient inference, we adopt the MCMC posterior sampling and introduce complementary inference model to initialize such latent MCMC sampling. We show that three separate models can seamlessly integrated into our joint framework through two (dual-) MCMC teaching, enabling effective and efficient EBM learning.
Introduction
Deep generative models have made significant progress in learning complex data distributions [35, 18, 37, 34, 19, 22, 5] and have found successful applications in wide range of real-world scenarios [26, 6, 11, 15]. Among these, the energy-based model (EBM) [5, 6, 29, 8, 39, 2, 3] has gained particular interest as flexible and expressive generative model with an energy function parameterized by a neural network. Learning the EBM can be accomplished via the maximum likelihood estimation (MLE), which involves the Markov Chain Monte Carlo (MCMC) sampling in dige-dimmesional data space. However, such MCMC sampling has shown to be challenging [30, 7, 33, 13], as it may take long time mix between different local modes with non-informative noise initialization [43, 15].
To address this challenge, recent advances have explored employing complementary models to substitute for MCMC sampling [15, 16, 12, 17, 24]. One notable example the generator model. The generator model incorporates top-down generation network that is capable of mapping low- dimensional latent space to high-ding -ige--iinnsional data space and admits efficient sample generation. The generator is learned to match the EBM so that MCMC sampling can be replaced by generator ancestral
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

sampling. However, such direct generator sampling has shown to be less accurate and suboptimal [43]. To alleviate this issue, [42, 43] introduced cooperative learning, where samples generated by the generator model serve as initial points, and then followed by finite-step MCMC revision process. While this gradient- basee MCMC revision process can be more accurate, the generator model learned relies solely on the EBM and has no access to the observed empirical observations. As result, this learning scheme may render biased generator learning, which in turn caps the potential of learning a strong EBM. An effective joint learning scheme for the EBM and its complementary generator model needed, yet still in its infancy.
In this paper, we present a novel learning scheme that can seamlessly integrate the EBM and comple- mentary models into a joint probabilistic framework. Specifically, both the EBM and complementary generator model are learned to match the empirical data distribution, while the generator model, at the same time, is also learned to match the EBM. Learning the generator model with empirical training examples can be achieved with MIE,, which typically requires access to the generator posterior as an inference process. To ensure an effective and efficient inference, we employ the MCMC posterior sampling with the complementary inference model learned as an initializer. Together with MCMC sampling of EBM being initialized by the generator model, such two MCMC samplings can be further used as two MCMC revision processes that teach the generator and inference model to absorb MCMC-revised samples, thus we term our framework dual-MCMC teaching. We show that our joint framework is capable of teaching the complementary models and thus learning strong EBM.
Our contributions can be summarized as follows:
We introduce novel method that integrates the EBM and its complementary models into joint learning scheme.
We propose the use of dual-MCME teaching for generator and inference models facilitate efficient yet accurate sampling and inference, which in turn leads to effective EBM learning.
We conduct extensive experiments to demonstrate the superior performance our EBM.
2 Preliminary
2.1 Energy-based Model
Let x E RP be the high-dimensional observed examples. The energy-based model (EBM) [41, 6, 5, 29] represents data uncertainty with an undirected probability density defined as
(x)
exp [fa (x)],
(I)
Z(xx)
where (x) is the energy function parameterized with parameters a, and Z(a) (= Sx (x)xxx is the partition function or normalizing constant.
Maximum likelihood estimation. The maximum likelihood estimation (MLE) is known for being an asymptotically optimal estimator and can be used for training the EBM. In particular, with observed examples, (x((), i = 2,,, ..... ), the MLE learning of EBM maximizes the likeliiod n
L, (a) 1 Σn log Ta (x()). If the sample size is large enough, the maximum likelihood estimator minimizes the (L (x) To (x)) which is the Kullback-Leibler (KL) divergence between the empirical data distribution pa (x) and the EBM distribution n. (x). The gradient 2 computed as
Ba La (a) is
max L (a) = min (L(pa(x) (T (x)), where
2 _ fa (x)) - bx) 2
da La (a) = (p))) (a)=
(x))
da (x)]
Da
Given such gradient, the EBM can be learned via stochastic gradient ascent.
Sampling from EBM. The Eqn.2 requires sampling from the EBM n, (x), which can be achieved via Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics [28]. Specifically, sample from the EBM, the Langevin dynamics iteratively updates
θ X X++1 =X, + Ox,,
log Ta (x+) + 2sU,
(3)
2

---

where 7 indexes the time step, s is the step size and U, ~ (0,
oo, the distribution of x, will converge the target distribution To (x+) regardless of the initial distribution of X [28]. The existing practice [29, 30, 5] adopts non-informative distribution for Xp, such unit Gaussian uniform, to initialize the Langevin transition, but it can be extremely inefficient and ineffective as they usually take a long time to converge between different modes and are also non-stable in practice [43]. The ability generate efficient and effective samples from the model distribution becomes the key step toward training successful EBMs. In this paper, we study the complementary model, i.e., generator model, as an informative initializer for effective yet efficient MCMC exploration toward better EBM training.
As 8 O, and
2.2 Generator Model
Various works [15, 12, 17, 24] have explored the use of the generator as an amortized sampler to replace the costly coise-initialized MCMC sampling for EBM training. Such a learning approach relies on samples directly drawn from complementary models, which can be less accurate than iterative MCMC sampling as it lacks fine-grained exploration of the energy landscape. [42, 43] propose the cooperative scheme in which MCMC sampling of EBM is initialized by the generated samples from the generator model. However, the generator model has no access to the observed training examples, and such biased generator learning makes the EBM sampling ineffective and renders limited model training.
In this paper, the generator model is learned by MLE match both the empirical data distribution and the EBM distribution. Such training ensures stronger generator model which will further facilitate more effective EBM sampling and learning. We present the background of the generator model and its MLE learning algorithm below, which shall serve as the foundation of our proposed method.
Generator model. Let Rd (d D) be the low-dimensional latent variables. The generator model [14, 10, 21] seeks to explain the observation signal x by a latent vector z and can be specified
po 2) = =p(z)pp((f p(z)po
where p(z) is known prior distribution such unit Gaussian, e.g., p(z) NN00 Ia), and px(x ((go IIp) is the generation model that specified by neural network go(-) that maps from latent space data space.
Maximum likelihood estimation. The MLE learning of the generator model computes log-likelihood over the observed examples as (0) 1
n E log Po (x("), where Po (x) fe (x, z) the marginal distribution. If the sample size is large, it is equivalent to minimizing the KI. divergence KL (x) (x)). The gradient of the likelihood (f) can be obtained via:
max I. (0) min (T.
where
0 (0) = (x) log px
(5)
00
00
With such gradient, the generator model can be learned via stochastic gradient ascent.
Sampling from generator posterior. The Eqn.5 requires the sampling from the generator posterior px(2) One can use MCMC sampling such as Langevin dynamics [28] that iterates
à 2r+1 = Zo + 8 Oz,
log po (x) + 22
(6)
where U,- ~ NO0 Id). Such a Langevin process is an explaining -wwy inference where the latent factors compete with each other explain each training example. As s distribution of will converge to the posterior Po(z(x) regardless of the initial distribution of zo [28]. However, noise-initialized Langevin [14, 31] can be ineffective in traversing the latent space and hard to mix. In this paper, we introduce complementary model, i.e., inference model, as an informative initializer for effective yet efficient latent space MCMC exploration for better generator and EBM training.
and - 30, the
3

---

2.3 Inference model
The inference model (z(x))) is adopted in VAEs [21, 32] as an amortized sampler to bypass the costly poise-initialignn latent space MCMC sampling. In VAEs, (z(z]x) is Gaussian parameterized, ie.. (Nuss (x()),, where (x) is the mean d-dimensional mean vector and Vx(x) is the d- dimensional diagonal covariance matrix. Such a Gaussian parameterized inference model is a tractable approximation to the true generator posterior po (z) x), but can be limited to approximate the multi- modal posterior. We adopt the same Gaussian parametrization of go (z) in this paper, but unlike the VAEs, our inference model serves as an initializer network that jump-starts the latent MCMC sampling from an informative initialization. The marginal distribution obtained after Langevin can be more general and multi-modal than the Gaussian distribution.
3 Methodology
To effectively learn the EBM, we propose joint learning framework that interweaves maximum likelihood learning algorithms for both the EBM and its complementary models. For the MLE learning of the EBM, MCMC sampling can be initialized through the complementary generator model, while for the MLE learning of the generator model, the latent MCMC sampling can be initialized by the complementary inference model. Three models are seamlessly integrated into our joint framework and are learned through dual-MCMC teaching.
3.1 DCal-MCMC Sampling
The EBM Ta (x), generator (x) and the inference model go(z/x) defined in Sec.2 naturally specity the three densities on joint space (x, z), i.c.
Po(x, 2) = po (x (z)))()))
(x, To (x) (z)x) Qx(x, 2) = Pa pa(x)gg(2(x))
The generator density (x z) specifies the joint density through ancestral generator sampling from prior latent vectors. Both the joint EBM density Il (x, (x,z) and data density Qx z) include inference model (z(x) to bridge the marginal distribution to joint (x, z) space. However, go (z]x) is modeled and learned from two different perspectives, one empirical observed data distribution (x) for real data inference, and one on EBM density (x) for generated sample inference.
The joint learning schemes [15, 12, 17, 24] based on these joint distributions can be limited, because 1) the generator samples from Po (x, z) is conditionally Gaussian distributed (Sec.2.2) which can be ineffective to capture the high-dimensional multi-modal empirical data distribution, and 2) the inference model qo (z) x) on observed training examples assumed to be conditionally Gaussian distributed (Sec.2.3) that is incapable of explaining-away inference [14].
To address the above limitations, we introduce two joint distributions that incorporate MCMC sampling as revision processes,
(x,a(x,2) 2) =
Q0.0(x, 2) = pa (x)TT)
where (.) denotes the Markov transition kernel of finite step Langevin dynamics that samples z from po (2(x) (see (.n.6). and (.x denotes the transition kernel that samples x from n. (x) shown in Eqn.3. Therefore, (x) Ixx' J₂ T.x (xx) )Pp 2)dzdx') indicates the marginal distribution of x obtained by running MCMC transition T (-) that is initialized from po (x). Similarly, To (z|x) represents the marginal distribution of z obtained by running To (-) that is initialized from q0(z given observation x (i.c., Tg go (z|x) = fa. (z"(z) (2) (z') (z)'))
The Po.a z), as a revised generator densiry, is more expressive on x-space than Po z) as the generated samples from pa (x) are refined via the EBM-guided MCMC sampling. Q0,e z). as revised data density, can be more expressive on z-space than Q& (x, z), as the latent samples from (z(x)) are revised via the generator-guine guided explaining-away MCMC inference. These MCMC- revised joint densities will be used for better EBM training, while at the same time, they will guide and teach the generator and inference model to better initialize and facilitate MCMC samplings.
We jointly train three models within a probabilistic framework based on KL divergence between joint densities. We present below our learning algorithm in an alternative and iterative manner where the new model parameters are updated based on the current model parameters. We present the learning algorithm in Appendix.

---

3.2 Learning Energy-based Model
Learning the EBM based on the minimization of KL divergences as
min (a) min KT 0. (x, T) 2)) - KT (T. (x, 2) 2))lTT,
To fa(()) - ET, PM, (x)] a Oa fa()x)
3
where Dr (a) = [x)x))
(7)
da
da
where Ot, Oz, Ur. denote fixed copies of EBM, generator, and inference model the 1-th step in an iterative algorithm. The joint densities Qu. and Po..o. are based this current iteration.
Comparing Eqn.7 to Eqn.2, we compute sampling from EBM through Langevin transition with current pe, (x) as initializer, i.c., Pe, (x). Such generator initialized MCMC is more effective and efficient compared to the noise-initialized transition, T% (6x) where 6x ~ (O,, Ip), that is used in recent literature [29, 5, 6].
MLE perturbation. The above joint space KL divergences are equivalent to the marginal version, min (a) min (x) - KL
TT,,, Pe, (x) (x))
(8)
Additionally, To, (x) + na, (x) 8 - 0 and r - oo (see Eqn.3), thus Eqn.8 amounts to the approximation of MLE objective function with KL. perturbation term, i.e.,
(I(aa(x)]|=aaxx) KL (T.
Such surrogate form is tractable than the MLE objective function, since the log 7(a) term is canceled out. The (x) seeks to approach the data distribution (x() while escapes from its current version To, (x), thus can be treated as its own critic. The learning of EBM can then be interpreted as a self-adversarial learning [15, 40].
Connection to variational learning. It is also tempting learn the EBM without MCMC sam- 0
1 So (x)) (i.e., min, KI[Q,, (x, 2) To. (x, 2)) pling via gradient (pa(x) da [x)] - Eps, (x) (x)laa
KL(Ps, (x, z)|lI (x, z)) ). which underlies the variational joint learning [15, 4, 12, 24]. Compared
to 7q their generator serves as a direct sampler for EBM, while we perform the EBM self-guided MCMC revision for more accurate samples.
3.3 Learning Generator Model via Dual-MCMC Teaching
As complementary model for learning the EBM. the generator model becomes a key ingredient toward success. The generator model is learned through the minimization of KL divergences as
min (p) = min KL(Q 0. (x, z) Px z)) + KL(P (x, z) Px
0
θ (7) Epa(x))T 90. (x/x)
8
where
log po (x, 2)] ETy (xl2)p(2)
log po (x( (x, 2)] (10)
00
00
where both the Q0..0, and Po.a. are based on the current iteration. The revised data density Q0..0 teaches the generator better match with empirical data observations through the first KL term, and the revised generator density Po.. teaches the generator to better match with generated samples through the second KI. term. As we describe below, such a joint minimization scheme provides a tractable approximation of the generator learning with marginal distribution, i.e.,
min (x)p
KL( (x) (x))))
where generator model (x( learns match the (xx) on empirical data observations and catch up with the current EBM density Ta. (x) through guidance of its generated samples.
MLE perturbation on Pa- Our generator model matches empirical data distribution (a(x) through KL( (x, z) Po 2)) and is equivalent the marginal version that follows,
KL (x. z) xx 2)) (x(pa(x) (x)) + Ex)(x) KL 9o. (z) (p)o))))
(11)
Given s + and + Dc. TØ. (z|x) + ps. (z(x) (see Eqn.6). the first KL term (in Eqn. thus approximates the true MLE objective function with additional KL perturbation term, i.c.,
(L(pa(x) Ippa (x)) + (x)( [pL, (z/x)
(12)
5

---

Such surrogate form in joint density bounr-bound (i.e., majorizes) the true MLE objective (I)pa(x)]lPo(S) and can be more tractable as it involves the complete-data model with latent vector z has been inferred in the current learning step. Minimizing the surrogate form in the iterative algorithm makes the generator po(x) to be closer to the empirical pa(x) due to its majorization property [15].
MLE perturbation on To., Our generator model is learned catch up with the EBM model Ta (x) through the second term (L(Pe,, (x, 2) (x z)). ) is equivalent the marginal version as
(Ps.... (x, z) Pa (x. z)) = KL Pe, (x) (x())) + BT,, po, (x) (po, (x] (zx((()) (13)
With + 0 and 7 co. Ta Ta, (x) > aaz (x) (see Eqn.3), our second KI. term approximates (in Eqn. 10) the MLE objective on Nay for generator,
KT.(Ta. (x)) + Br.,, (x) [pT, (2/x)]]]0(22)
(14)
Such surrogate in joint density again apper-bounds (i.e., majorizes) the true MLE objective on generated samples, i.e., KL(T (x) (x)), and thus the generator (x(x) updates to be closer to the EBM Fa (x) at the current iteration.
Connection to variational learning. Without MCMC inference, the generator model can be learned with inference model match the empirical data distribution, i.c., ming (L(Q (x, z) P z)). which underlies VAEs [21, 32, 26, 37]. Compared to Equ. VAEs seek to minimize (L(pa (x(paxx)lpppxx) (x)) + (x) [KL(qs, (z) (<]((p(z(x)) where qo (z(xx) x) is assumed be Gaussian dis- tributed which has limited capacity for generator learning.
Connection to cooperative learning. Cooperative learning schemes [42, 43] share similar EBM train- ing procedures but can be fundamentally different in generator learning. The generators are learned through ming KT (x, 2) || Pa (x, z)) [43] or ming KT. generators have access to the empirical observations which lead to biased and sub-optimal generator models.
2)) [42], however,
3.4 Learning Inference Model via Dual-MCMC Teaching
The inference model qs (z)x) serves as a key component for generator learning which will in turn facilitate the EBM training. In this paper, the inference model is learned by minimizing the KL divergences (d) as
min (0) min KL z) 2))l z)) + KL a. (x, 2) o z))
0 Dg( = Epalx) 90, xx) do d6
where
log go (z(x)] + E po, (xxx)
log d0 (z/x)] (15)
do
The revised data density Q00.0, teaches the inference model on empirical data observations for better real data inference through the first KL term, and the revised generator density Pa.... teaches the inference model for better generated sample inference through the second KL term.
Real data inference. Optimizing the first term ( .. (x, 2) Q 2)) in Eqn.15 is equivalent to min& (x)( [KT, go. (z) x)) (z|x))] Given the long-run optimality condition of the MCMC transition To,4o, go. (z) (2|x) + Po, (z) x). our first term (T. (x, |[ (x, 2)) tends to learn go (z|x) by minimizing the Ep [KL ( (z) x) || (z x- The inference model is learned to match the true generator posterior Po (z) x) on real observations in the current learning step. Specifically, latent samples are initialized from current qs. (z/x), and the generator guided MCMC revision TØ 9o. go. (z/x) is then performed to obtain the revised latent samples. The inference model is updated to amortize the MCMC and absorb such sample revision. The MCMC revision not only drives the evolution of the latent samples, but also drives the evolution of the inference model.
Generated sample inference. Optimizing the second term (L(Po.... z) [o., z)) in Eq is equivalent to ming ETg,,, (x) [KL(po, (z/x)]lu((z)x)) which tends to minimizing En., (x) [pL (z]x)]l9.(z(x)) given long-run optimality condition (i.e., T Po, (x) - Na, (x)).
The inference model is learned to match the true generator posterior po po(x)) on generated samples from EBM in the current learning step. Noted that both the generated sample and its latent factor can be readily available where the latent factor z is drawn from prior distribution p(z), which assumed to be unit Gaussian, and the generated sample is obtained directly from generator.

---

4 Related Work
Energy-based model. The EBM is flexible and theoretically appealing with various approaches to learning, such as the noise-contrastive estimation (NCE) [1, 38] and the diffusion approach [8]. Most existing works learn the EBM via MLE [29, 30, 5, 6, 39], which typically involves MCMC sampling, while some advance [15, 16, 12, 17, 24] propose to amortize MCMC sampling with the generator model and learn EBM in a close-formed, variational learning scheme. Instead, [42, 43] recruit ancestral Langevin dynamics with the generator model being the initializer model. In this paper, we propose a joint framework where the generator model matches both the EBM and empirical data distribution through dual-MCMC teaching to better benefit the EBM sampling and learning.
Generator model. In recent years, the success of generator models has given rise various ways of learning methods. Generative adversarial network (GAN) [10, 19, 27, 20] jointly trains the generator model with discriminator, while VAE [21, 32, 36, 9] is trained with inference model (or encoder) approximating the generator posterior. Without the inference model, [14, 31] instead utilize MCMC sampling sample from the generator posterior. Our work differs from theirs by employing the MCMC inference based on informative initialization from the inference model, and we aim to learn the generator model to facilitate effective learning of the EBM.
5 Experiment
In this section, we address the following questions: (1) Can our method learn an EBM with high- quality synthesis? (2) Can both the complementary generator and inference model successfully match their MCMC-revised samples? and (3) What is the influence of the inference model and generator model? We refer to implementation details and additional experiments in Appendix.
5.1 Image Modelling
We first evaluate the EBM in image data modelling. Both the generator model and EBM are learned to match empirical data distribution, and if the generator model is well-trained, it can serve as an informative initializer model, making the EBM sampling easier. As a result, the EBM should be capable of generating realistic image synthesis. For evaluation, we generate images from the EBM by obtaining Xg from the generator and running Langevin dynamics with Xg being the initial points.
Table 1: FID and IS on CIFAR-10 10 and CelebA-64.
Table 2: FID on CelebA-HQ-256 and LSUN- Church-64.
CIFAR-10
CelebA-64 FID (J)
Methods
IS (t)
FID (L)
Methods
CelebA-HO-256
LSUN-Church-64
Ours
8.55
9,26
5.15
Ours
15.89
4.56
Cooperative EBM [42]
6.55
33.61
16.65
Dillusion EBM [8]
7.02
Amortized EBM [43]
6.65
VAEBM [39]
20.38
13.51
Divergence Triangle [15]
7.23
30
18.21
NCP-V
No MCMC EBM [12]
27.5
GLOW[22]
68.93
59.35
PGGAN [19]
Short-run EBM [29]
6.21
23.02
IGEBM [5]
6.78
38.2
ImprovedCD EBM [6]
7.85
25.1
Diffusion EBM [8]
8.30
9.58
5.98
VAEBM [39]
8.43
12.19
5.31
VAP-V
24.08
5.25
SNGAN [27]
8.22
21.7
6.1
StyleGANv2 w/o ADA[20]
8.99
9.9
2.32
NCSN[34]
8.87
25.32
25,30
DDPM[18]
9.46
3.17
3.93
Figure 1: Image synthesis on CIFAR-10.
We benchmark our method on standard datasets such as CIFAR-10 [23] and CelebA-64 [25], as well as challenging high-resolution CelebA-HQ-256 [19] and large-scal scale SSUN-Church664 [44]. We consider the baseline models, including Divergence Triangle [15], No MCMC EBM [12], Cooperative EBM [42], and Amortized EBM [43], as well as modern advanced generative models, including other EBMs [29, 6, 8, 39, 1]. GANs [27, 20] and score-based models [34, 18]. We recruit Fréchet Inception Distance (FID) and Inception Score (IS) metrics to evaluate the quality of image synthesis. Results are reported in Tab. and Tab.2 where our EBM shows the capability of generating realistic image synthesis and renders competitive performance even compared to GANs and score-based models.
7

---

5.2 MCMC Revision
The complementary generator and inference model are learned to match their MCMC-revised samples and thus model can successfully catch up with the MCMC revision. We train our model on CelebA-64 using Langevin steps xx 30 for the MCMC revision on and k₂ 10 for the MCMC revision on
informative initializers. We demonstrate that both the generator and inference
Generator model. the generator model captures different modes of the EBM, the MCMC revision on should only need to search around the local mode and correct pixel-level details. To examine the generator model, we visualize the Langevin transition by drawing x; for every three steps from generated samples Xg to MCMC-revised samples X₃. As shown in Fig.2, only minor changes be observed during the transition, suggesting that the generator has matched the EBM-guided MCMC revision. By measuring FID of Xo and Xg, it still improves from 5.94 5.15, which indicates pixel-level refinements.
Inference model. We then show the Langevin transition mapped to data space via the generation network. We draw z, for each step and show corresponding images in Fig.2, where the inference model also catches up with the generator-guides explaining-away MCMC inference, leading to faithful reconstruction as a result.
For visualization, latent codes
019
Figure 2: Left top: MCMC revision on The leftmost images are sampled from the generator model, and the rightmost images are at the final step of the EBM-guided MCMC sampling. Left bottom: Energy profile over steps. Right top: MCMC revision reconstructed by latent codes inferred from the inference model, and the rightmost images are reconstructed by latent codes at the final step of the generator-guie guided MCMC inference. Right Mean Squared Error (MSE) over steps.
The leftmost images are
bottom:
5.3 Analysis of Inference Model
The inference model serves as an initializer model for generator learning which in turn facilitates the EBM sampling and learning. To demonstrate the benefit of the inference model, we adopt noise-initialized Langevin dynamics for generator posterior sampling and compare with the Langevin dynamics initialized by the inference model.
Table 3:
Comparison of MSE. Inf+L=10 de- notes using Langevin dynamics initialized by inference model for k₂ 10 steps.
Methods
CIFAR-10
CelebA-64
VAB[21]
0.0341
0.0438
WAE[3]]
0,0291
0.0237
RAE[9]
0.0231
0.0246
ABP[14]
0.0183
0.0277
SR-ABP[31]
0.0262
0.0330
Cooperative EBM[42]
0,0271
0.0387
Divergence [ri5
0,0237
0.0281
Ours (Inf)
0.0214
0.0227
GPAR13
GelbA 64
Ours (Inf+L=10)
0.0072
0.0164
Figure 3: MSE(). SSIM(1) and PSNR (t).
8

---

Specifically, we conduct ioise-initializzed Langevin dynamics with increasing steps from l2 = 10 to k2 = 30, and compare with the Langevin dynamics using only h₂ = 10 steps but is initialized by the inference model. We recruit MSF, Peak Signal-to-Non Noise Ratio (PSNR), and Structural SIMilarity (SSIM) to measure the inference accuracy of reconstruction and present the results in Fig.3. As the Langevin steps increase, the inference becomes more accurate (lower MSE, higher PSNR, and SSIM), however, it is still less accurate than the proposed method (L=30 vs. Inf+L=10), This result highlights the inference model in our framework. We then compare with other models that also characterize an inferential mechanism, such as VAE [21], Wasserstein auto-encoders (WAE) [36], RAE [9], Alternating Back-prop propppropggttion (ABP) [14], and Short-run ABP (SR-ABP) [31]. As shown in Tab.3, our model can render superior performance with faithful reconstruction.
5.4 Analysis of Generator Model
With the generator model being the initializer for EBM sampling, exploring the energy landscape should become easier by first traversing the low- dimensional latent space. We intend to exam-
ine our generator model can deliver smooth interpolation on the latent space, thus making a smooth transition in the data space. We em- play linear interpolation among latent space, i.e., 2 (1- a)- z1 + a Zg, and consider two see- marios, such the image synthesis and image reconstruction. As shown in Fig.4, our genera- tor model is capable of smooth interpolation for both scenarios, which suggests its effectiveness in exploring the energy landscape.
Figure 4: Linear interpolation latent space. The top and bottom three rows indicate image genera- tion and reconstruction, respectively.
6 Ablation Studies
MCMC steps of Ti. We analyze the impact of the inference accuracy in our framework by increasing the Langevin steps of TV (z)x). With an inference model initializing the MCMC posterior sampling, further increasing the MCMC steps should deliver more accurate inference and thus benefit the generator and EBM for better performance. Thus, we compute the FID, MSE, and wall-clock training time (seconds/ / per iteration) in Tab.4. It can be seen that increasing MCMC steps from 10 to 30 indeed slightly improves the generation quality and inference accuracy but requires more training time. We thus report the result of Inf+/.=/0 /0 in Tab. I and Tab.3.
MCMC steps of T Then, we discuss the impact of the Langevin steps of T. Increasing the MCMC steps of Tx should explore the energy landscape more effectively and render better performance in the generation. In Tab.5, starting MCMC steps from 10 to 30, our model exhibits largely improved performance in generation quality but only minor improvement even when we use = 50 steps. Thus, report L = 30 steps in 1.
Table Increasing MCMC steps of TØ
Table 5: Increasing MCMC steps of T.
L=10
L=30
Inf+L=10
Inf+L=30
L=10
L=20
L=30
L=50
FII)
17.32
14.51
9.26
9.18
FID Time (s)
14.78
11.51
9.26
9.07
MSE
0.0214
0.0164
0.0072
0.0068
Time (5)
1.576
2.034
1.594
2.112
0.861
1.241
1.594
2.454
7 Conclusion
We present a joint learning scheme that can effectively learn the EBM by interweaving the maximum likelihood learning of the EBM, generator, and inference model through dual-MCMC teaching. The generator and inference model are learned to initialize MCMC sampling of EBM and generator posterior, respectively, while these EBM-guides MCMC sampling and generator-guides MCMC inference, in turn, serve as two MCMC revision processes that are capable of teaching the generator and inference model. This work may share the limitation with other ICCMC-based methods in terms of the computational cost, but we expect to impact the active research of learning the EBMs.

---

References
[1] Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480-499 493, 2021.
[2] Jiali Cui, Ying Nian Wu, and Tian Han. Learning joint latent space ebm prior model for multi- layte generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3603-3612, June 2023.
[3] Jiali Cui, Ying Nian Wu, and Tian Han. Learning hierarchical features with joint latent space energy-based prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2218 2227, October 2023. 1
[4] Zihang Dai, Amjad Almahairi, Philip Bachman, Eduard H. Hovy, and Aaron C. Courville. Calibrating energy-hased generative adversarial networks. In 5th International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 5
2017. URL https: http///pppeeeeeeee/eeee/ee net. petee//orumiiddSyyee22200
[5] Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models, arXiv preprint Xxxv:/903 08689, 2019. 3,5,7
[6] Yilun Du, Shuang Li, Joshua Tenenhaum, and Igor Mordatch, Improved contrastive divergence training of energy based models. arXiv preprint 201.0:00 2020. 1, 5,7
[7] Marylou Gabrié, Grant M Rotskoff, and Eric Vanden-Eijndeen Adaptive monte carlo augmented with normalizing flows. Proceedings of the National Academy Sciences, (19(10):2210420013 19, 2022. I
[8] R Gao, Y Song, B Poole, YN Wu, and DP Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations (ICLR 202/), 2021. 1,7
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Berhhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint Xrx:: 12436, 2019. 7,8,9
[10] Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, Advances in neural information processing systems, 27, 2014. 3,7
[11] Will Grathwohl, Kuan-Chieh Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly energy based model and you should treat it like one. arXiv preprint 1rX1v:111..0366,, 2019. 1
[12] Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No (meme) for me: Amortized sampling for fast and stable training of energy-basse based models. In International Conference on Learning Representations, 2021. URL https:// 3,3,4,5, 4,7
net/forum?id:ixppnie
[13] Louis Grenioux, féii Moulines, and Marylou Gabrié. Balanced training of energy-based models with adaptive flow sampling. arXiv preprint arXiv:2306.0068 2023.
[14] Tian Han, Yang Lu, Song-Chus Zhu, and Ying Nian Wu. Alterntting pock-aropagation for generator network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017. 3, 4, 7,
[15] Tian Han, Erik Nijkamp. Xiaolin Fang. Mitch Hill. Song- Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training generator model, energy- hased model, and inferential model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8670-8679, 2019. 1,3, 4, 5,
6, 7, 8
[16] Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy- based model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 1.7
[17] Mitch Hill, Erik Nijkamp, Jonathan Craig Mitchell, Bo Pang, and Song-Chun Zhu. Learning probabilistic models from generator latent spaces with hat EBM. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems. 2022. URL https: //openreview.coo net
Zy. 1,3, 4,7
[18] Jomathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-668 6851, 2020. 7,7
10

---

[19] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710. 10199, 2017. 1,7
[20] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, Advances in Neural Information Processing Systems, 33:12104-12114, 2020. 7
[21] Diederik P Kingma and Max Welling, Auto-encodio variational bayes. arXiv preprint arXiv:1312.6114 2013. 3,4, 6, 8.9
[22] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018. 7
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7
[24] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-hased models. arXiv preprint MoXivv/901.085000 2019. 1, 5,7
[25] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. CoRR, abs/1411.7766, 2014. URL http:/ /arpiv/ org/ abs/1411. 1411.7766..
[26] Lars Maaloe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019. 1, 6
[27] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 1802.0597, 2018. 7
[28] Radford M Neal et al. Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):
2, 2011. 1.. 2,3
[29] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- non-persistent short-run meme toward energy-based model. Advances in Neural Information Processing Systems, 32, 2019.
2,
[30] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chur Zhu, and Ying Nian Wu. On the anatomy of meme-basee maximum likelihood lcarning of energy-hased models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5272-5280, 2020. 3,7,
[31] Erik Nijkamp, Bo Pang. Tian Han, Linqi Zhou, Song- Chun Zhu, and Ying Nian Wu. Learning multi- layer latent variable model via variational optimization of short run meme for approximate inference. In European Conference on Computer Vision, pages 368-378. Springer, 2020. 3, 3,
[32] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference machine learning, pages 1278-1286. PMLR, 2014. 4, 6,
[33] Sergey Samsonov, Evgeny Lagutin, Marylou Gabrié, Alain Durmus, Alexey Naumov, and Eric Moulines. Local-global meme kernels: the best of both worlds. Advances in Neural Information Processing Systems, 35:5178- 1111778-919,,, 2022.
[34] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019. 1.7
[35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint 1331v 2020.
[36] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- ento-enc arXiv preprint arXiv:1711.0155.. 2017. 7, 8,9
[37] Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667-19679, 2020. 1.6
[38] Zhisheng Xiao and Tian Han. Adaptive multi-stage density ratio estimation for learning latent space energy-based model. In NeurlPS, 2022, URL http:://apperrs nips. wwwp:cc/papps......eeepppppppppe cc/ppaperfilleseeeeeeeeeeeeeeeee ftless/ppppe//02//
htt.r//eeeeeeeeee00ooooaaaeaaaaa Conferencee html.
[39] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy- based models. In International Conference on Learning Representations, 2020.
7

---

[40] Zhisheng Xiao, Qing Yan, and Yali Amit. Ebms trained with maximum likelihood are generator models trained with self-adverserial loss. arXiv preprint arXiv:2102.1137, 2021.
[41] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pages 2635-2644. PMLR, 2016.
[42] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27-45, 2018. 2, 2,3, 7.8
[43] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy baargy-based model with variational auto-encoder amortized sampler. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10441-10451, 2021. 2, 3, 6,
[44] Fisher Yu. Yinda Zhang. Shuran Song, Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans the loop. arXiv preprint urXiv:1506.03360 2015. 7
12


Reviews:
[{"review_id": "mYl63f3boS", "reviewer": "Reviewer_uj7J", "summary": "This work investigates learning EBMs for image data using auxiliary generator and inference models. The model generates samples by drawing a latent normal vector, passing it through the generator, and refining the generator sample using MCMC with the EBM. An inference network, which predicts latent vectors from images, is used to assist with training the EBM and generator. The key innovation is to update both generator samples using image space MCMC and inferred latent vectors for data using latent space MCMC. This allows the generator to be efficiently trained using both revised MCMC samples, as in Cooperative Learning, and revised latent codes for data, which is unique to this work. Training the generator to match both EBM and data samples is crucial for improving performance beyond the results achieved by related methods. Training algorithms for the EBM, generator, and inference network are presented using several KL terms between different joint distributions of images and latent codes. Experiments show strong EBM synthesis results for CIFAR-10, CelebA, CelebA HQ, and LSUN-Church-64.", "questions": "Could the authors provide a detailed comparison between their proposed method and the Divergence Triangle?", "limitations": "A broader impact section was not included. Computational limitations are very briefly discussed, but more thorough details about runtime and compute costs compared to related methods would strengthen the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The work identifies and addresses a key limitation of the Cooperative Learning framework, which is the fact that the generator is trained to reconstruct only refined EBM samples. This can limit the potential of the generator since EBM samples are less realistic than true data. While other works such as Divergence Triangle train the generator to reconstruct data images using inferred codes for observed data, these works lack an MCMC refinement stage which is crucial for improving sample quality. This work bridges the gap between Cooperative Learning style methods and Divergence Triangle style methods to incorporate both MCMC sampling and generator data reconstruction into a single learning framework.\n* Experimental results show significant benefits from the proposed method compared to a variety of other EBM learning methods.", "weaknesses": "* The loss functions appear to be significantly inspired by the Divergence Triangle, but the paper lacks a thorough comparison between the proposed method and Divergence Triangle. Such a discussion would help elucidate the differences between the proposed method and prior work. To my understanding, the primary differences are 1) replace all LHS of KL terms with distributions defined by short-run MCMC distributions with frozen model parameters, 2) add KL($\\tilde{P} || P$) and KL($\\tilde{Q} || Q$) terms to the generator and inference network updates respectively.\n* The motivation of each loss term is not entirely clear, which makes the proposed loss gradients appear somewhat ad-hoc. An intuitive explanation of the role of each KL term would be very helpful for the reader. A clearer discussion of connections to the Divergence Triangle could also help with understanding the KL terms.\n* The proposed method is somewhat complex, and the loss functions are defined sequentially given the current fixed model weights. Such an approach is common to Cooperative Learning and related methods, while amortized methods like Divergence Triangle have a loss function which can be written in closed form without reference to fixed model weights. While this is not a major issue in my view, the reliance on loss defined by a sequence of fixed model weights as opposed to a single loss function should be discussed as a limitation of the proposed method.\n\nMinor Problems\n* The related work [12] uses MCMC sampling initialized from the generator to update the EBM similar to [36, 37], instead direct generator samples like [11, 9, 19].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback. We appreciate the correction of the references and shall fix them in our final manuscript. \n\nWe provide the clarification in **General Response** for\n- Connection with Divergence Triangle (**G.2**)\n- Motivation of our method (**G.1**)\n- Computational cost (**G.3**) (see also the attached **PDF**)\n\n----\n\n**Q1. Closed-form Objective.** \n\nYou are correct. Our MCMC teaching-based framework, indeed, cannot be learned within a single objective function. The MCMC revision shall be conducted based on the fixed model weights. We shall include this point and a thorough discussion of the limitation in our revision."}, {"review_id": "QlMr1wfjZ6", "reviewer": "Reviewer_RJbi", "summary": "This work presents a new approach for training energy-based models (EBMs). These are commonly trained by approximating the maximum-likelihood estimate of the model parameters. This is done by minimising a certain Kullback--Leibler divergence via stochastic-gradient descent. To estimate the necessarily gradient, one must approximate an expectation w.r.t. the law of the data under the energy-based model. This is typically done via an MCMC algorithm.\n\nThe present work attempts to speed up this procedure by training an additional surrogate model (the \"generator\" model) from which the MCMC chain can be initialised (so that fewer MCMC iterations are needed). This generator model and the EBM are trained as part of a larger scheme which additionally trains an inference model (i.e. the posterior distribution of the latent variables given the observed data).\n\n-----------------------\nEDIT (2023-09-04): I have now read all the reviews and rebuttals. All my questions have been addressed. I continue to be happy for this paper to be published.", "questions": "In Table 1 (and also in the numerical results from Sections 5.2 and 5.3), is the computational cost of all algorithms comparable?", "limitations": "NA", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method appears to be sound and novel.\n2. The paper is mostly well written.", "weaknesses": "Main comments:\n\nMy expertise with training EBMs in this fashion is too limited to offer deeper insights. However:\n\n1. Some of the objectives used for training the three models appear a little ad-hoc and their motivation/justification sometimes leans on convergence of the MCMC chains (which seems unlikely to occur in practice given the small number of MCMC iterations used). At the very least, some added justification/derivation of these objectives would help the reader gain more intuition for the method.\n2. I'd ask the authors to confirm that the numerical comparisons are fair in the sense that all algorithms have roughly the same computational cost (see question below).\n3. I think it would help to have some (brief/high-level) pseudo code in the main paper.\n\n\nMinor comments/typos:\n\nL114: grammar in \"such Langevin process\"\nL180: are the KL divergences equivalent to the marginal version or is the /minimisation/ of these KL divergences equivalent to minimising the marginal divergences?\nL152--160: I understand what the authors mean with the \"transition-kernel\" notation. But I think the notation/explanation in this paragraph needs to be improved. For example, in Line 156, the stated expression is referred to as \"marginal distribution on $x$\" but it clearly represents distribution on joint the space of $(x, z)$. I would state the formal definitions (i.e. those found in Lines 158 and 160) right from the start to enhance clarity. It might also be worth reminding the reader that $p_\\theta(x)$ is the marginal of $p(z)p_\\theta(x|z)$.\nL256--257: the MCMC sampling -> MCMC sampling\n\n- there are sometimes missing spaces between \"Eq\" and the equation number and between \"Sec.\" and the section number.\n- punctuation is needed even if a sentence ends in a (displayed) equation\n- the font size in most of the figures is too small (especially in Figure 3) and axis labels are sometimes missing\n- it may just be my lack of familiarity with the literature in this area but is it clear to the reader what \"explaining away inference\" means in this specific context? If not, maybe it'd be worth adding a brief explanation.\n- table captions normally go /above/ tables (unless the NeurIPS style guide says otherwise)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your supportive comments and all the corrections, such as typos, formats, and captions, and we shall correct them in our final version. Please also find a brief description and model comparison in the **General Response** and the attached **PDF**.\n\n----\n\n**Q1. Convergence of the MCMC Chain.** \n\nThe convergence analysis of MCMC chains is mainly used for asymptotically theoretical understanding and is meant to establish the connection with the standard maximum likelihood estimation (MLE) and other relevant models. In practice, it can be costly and even infeasible to run a mixed and convergent MCMC sampler in each iteration. As such, a short-run sampling strategy (i.e., using only a small number of sampling steps) is commonly used and can provide meaningful learning signals for the model traversing as observed in [1]. We add the experiment for the long-run Langevin traversing (with $3,000$ steps, please see Figure 2 in the attached **PDF**), where only minor changes can be observed. It suggests that the generator, though trained with short-run MCMC revision, could essentially learn to match the long-run (i.e., near convergent) MCMC samples.\n\n**Q2. Computation Cost of All Algorithms.** \n\nWe follow the standard evaluation protocol in generative modeling (e.g., [2, 3] and others) and report the best-performing results for the baseline models. MCMC-based methods typically involve an inner loop for sampling, which introduces computational overhead compared to variational-based or adversarial methods (see the comparison of computational cost in **General Response** and the attached **PDF**). However, we note that our proposed joint learning scheme allows complementary models (e.g., inference model) to act as informative starting points and jump-start MCMC procedures (e.g., MCMC sampling on generator posterior), making them more efficient and effective than other noise-initialized MCMC methods. For example, as demonstrated in *Analysis of Inference Model* (Section 5.3), the inference accuracy of our 10-step MCMC revision with inference model initializer can be even better than 30-step, noise-initialized MCMC sampling. \n\n**Q3. Pseudo Code, KL Divergences Equivalence, Notation and Expression.** \n\nThanks for your valuable suggestions. The detailed derivation for the KL divergence equivalence can be found in *Methodology* (Section 2.2 in Appendix). We shall follow your advice to change the notations and expressions (e.g., marginal distribution in L156) in the revision, and we shall open-source our implementation.\n\n**Q4. Explaining-away Inference.** \n\nSorry for the confusion. The \"explaining-away inference\" refers to the situation where the knowledge of one latent variable reduces the influence or importance of another latent variable in explaining observed data. We employ this term here to emphasize that during the Langevin process, latent factors compete with each other to explain the given training example. We shall make it more clear in the revision.\n\n---\n\n[1] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems, 32, 2019.\n\n[2] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2020.\n\n[3] Gao, Ruiqi, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma. Learning Energy-Based Models by Diffusion Recovery Likelihood. In International Conference on Learning Representations, 2021."}, {"review_id": "g7mPZ6n39W", "reviewer": "Reviewer_iX9Y", "summary": "This paper investigates the problem of Maximum Likelihood (ML) estimation for Energy-Based Models (EBM). Building on previous research, the authors propose a novel approach that involves learning a surrogate generative model to initialize the costly Langevin steps. The authors chose a latent-based generative model thus requiring additional Markov Chain Monte Carlo (MCMC) sampling for estimation. This new MCMC procedure is amortized using a simpler third generative model, referred to as the inference model. This work provides a procedure to learn the three models jointly. Through extensive experimental evaluations, the authors demonstrate the effectiveness of their proposed framework. They showcase the benefits of incorporating the additional models and highlight the superiority of their combined algorithm over other existing methods. The results substantiate the value of joint learning, showcasing improved accuracy.", "questions": "- One of the main justifications of your learning procedure is to learn the generator using training data. Can you explain why not doing this (like in cooperative learning [1,2]) would introduce a bias ? This claim was made multiple times in the main paper.\n- As a justification of your revisited densities (or in L129), you say that Langevin steps could transform something unimodal into something multimodal which is not the case (see for instance [3]) as shown in Fig 2 of Sec 5.2. Is there another justification for those Langevin steps ?\n- The ablation studies (Sec 5.2 and 5.3) highlight the usefulness of the inference model with surprisingly low computational overhead. What about the memory overhead ?\n\n[1] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):27–45, 2018.\n\n[2] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10441–10451, 2021\n\n[3] Véronique Gayrard, Anton Bovier, Markus Klein, Metastability in reversible diffusion processes II: precise asymptotics for small eigenvalues. J. Eur. Math. Soc. 7 (2005), no. 1, pp. 69–99", "limitations": "Not applicable.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Experiments are well detailled.\n- Experiments on Out-of-Distribution Detection (OOD) in App 1.2 allow to evaluate the EBM beyond generating samples.", "weaknesses": "- The motivation to learn the generator with observed samples is unclear.\n- The improved accuracy of this method is not balanced with the computational and memory costs generated.\n- The quality of the density estimation task is barely considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed reviews, and we appreciate the time you spent on reviewing our paper. In **General Response**, we describe the motivation behind our method and discuss the computational and memory costs (see also the attached **PDF**). \n\n---\n\n**Q1. Density Estimation.** \n\nThanks for the note on the density estimation task. Evaluating the density of learned EBM requires the computation of intractable normalization constant (or partition function), which can be computationally expensive and non-exact as observed in [1]. Therefore, we consider standard metrics, such as FID and IS, for the sample quality of our EBM. We also consider out-of-distribution detection for EBM evaluation (please see *Out-of-Distribution Detection*, Section 1.2, Table 3 in Appendix). \n\n**Q2. Biased-learned Generator.** \n\nOur generator model is learned to match the EBM and the true data distribution. In the absence of observed training data, the generator can only learn from the EBM samples, and any approximation error of the EBM may carry over to the generator learning. Consequently, this leads to a biased-learned generator, which, in turn, affects the EBM learning by providing sub-optimal initial points. This is particularly true in the early stage of learning. To demonstrate, we adapt our model and learn the generator without using true data. As shown in Figure 1 in the attached **PDF** such a biased-learned generator might generate noisy and less realistic samples, resulting in sub-optimal EBM training and, consequently, an overall weaker generation performance (see, e.g., cooperative-style baselines in *Image Modelling*, Section 5.1, Table 1).\n\n**Q3. Justification of Langevin Steps.** \n\nThe use of Langevin step is intended to refine the direct samples obtained from generator and inference model, guiding them towards target distributions. For example, in L129, the starting (reference) distribution is unimodal Gaussian $q_\\phi(z|x)$, using K steps of Langevin would drive such distribution towards the exact generator posterior (i.e., $p_\\theta(z|x)$, which is the target distribution of latent MCMC, and can be multi-modal). This is based on the monotonicity property of KL [2], i.e., $KL(T_{\\theta_t}^{z}q_{\\phi_t}(z|x)||p_{\\theta_t}(z|x))\\le KL(q_{\\phi_t}(z|x)||p_{\\theta_t}(z|x))$. \n\nThe Figure 2 in Section 5.2 is shown to demonstrate the learned generator and inference model. If they are well-learned and successfully absorb the corresponding MCMC revisions, then they should generate ancestral samples that are on (or lie close to) the major modes of target distributions (or stationary distributions of MCMC transitions). For example, for latent MCMC revision (Figure 2, right panel), the samples exhibit only mild pixel changes along the trajectory, indicating the inference model $q_\\phi(z|x)$ is well-learned and close to the stationary distribution, i.e., $KL(T_{\\hat{\\theta}}^{z}q_{\\hat{\\phi}}(z|x)||q_{\\hat{\\phi}}(z|x))\\rightarrow 0$ ($\\hat{\\theta}$ and $\\hat{\\phi}$ denoted for  learned generator and inference models). This phenomenon (albeit within the realm of image space) is also demonstrated in Section 6.1, Figure 7.b in [3]. \n\n\n\nWe humbly request that you could reconsider the decision given our response. Please feel free to let us know if you have more questions about the paper. We will try our best to address your concerns. Thank you!\n\n---\n\n[1] Du, Yilun, and Igor Mordatch. Implicit generation and generalization in energy-based models. Advances in Neural Information Processing Systems, 32, 2019.\n\n[2] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley \\& Sons, 1991.\n\n\n\n[3] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy-based model with variational auto-encoder as amortized sampler. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021"}, {"review_id": "nrxd6VD2zB", "reviewer": "Reviewer_XcPg", "summary": "This paper proposes a novel method for training and inference of energy-based model (EBM). The author claim that although the use of generator as an initializer model may improve MCMC sampling, training unbiased generator is an open problem . Thus they propose a joint learning framework in which generator is learned by MLE via posterior sampling. While EBMs have a number of appealing properties they are difficult to inference. Therefore learning an unbiased generator for faster MCMC sampling from the EBM distribution is a hot topic.", "questions": "1. It was noticed [1] that for EBM distribution with density $\\pi(x)$ and generator $p_{\\theta} = p_{\\theta}(x|z)p(z)$ sampling latent variable from $p(z)\\pi(g_{\\theta}(z))$ and pushing it by $g_{\\theta}$ to the ambient space is equivalent to sampling from $\\pi(x)$. Why do you propose to perform EBM inference in the ambient space instead of latent space? As traversing high-dimensional space with MCMC sampling is a hard problem. Please elaborate on why don’t you use an advantage of sampling from EBM in the latent space of the generator model.\n\n2. The paper massively adopts the method proposed in [2], nevertheless considering results on CIFAR-10 dataset FID drops from $30.10$ for [2] to $9.26$ for the current paper. What is a major reason for such a dramatic improvement? In the Appendix you provide a quality for a \"simple network\" which seems to share architecture with paper [2]. It would be helpful for understanding if you provide a comparison of model complexity and/or time, GPU consumption in addition to image quality comparison.\n\n[1] Che, Tong, et al. \"Your gan is secretly an energy-based model and you should use discriminator driven latent sampling.\" Advances in Neural Information Processing Systems 33 (2020): 12275-12287.\n\n[2] Han, Tian, et al. \"Divergence triangle for joint training of generator model, energy-based model, and inferential model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.", "limitations": "The limitations of the method are obscured. The authors only mention that “this work may share the limitation with other MCMC-based methods in terms of the computational cost” which is a bit vague. \n\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is clearly written and well presented. The potential is impact of the proposed dual-MCMC teaching is considerable. The problem arised in the paper is actual and mostly uncovered by the previous research. The paper provides a number of novel ideas which could encorage the future research in this direction. The experiments are very clear and solid including fair comparison with concurent works. Remarkably, the numerical results for the proposed method are spectacular.\n", "weaknesses": "1. The section 2 is clear-written and self-contained but it is large and has little connection to the methodology proposed in the paper. As the authors reforlulate each component described in the section 2 it might be helpful to better explain the benefits of the choice for proposed method. \n\n2. The objective proposed for learning EBM includes maximization of Kulback-Leibler divergence term (equation 7) which is stated to perform as a self-critic for EBM. The grounding beneath this “​​self-adversarial learning” is not very confident. Either literature referenced by the authors lack the verification or some formal reasoning behind this objective. Compared to the objectives for generator and inference model this one doesn’t pose an upper bound for MLE objective.\n\n3. In the Figure 2 in the main text you provide an energy profile after performing $30$ steps (as during training) of MCMC revision on $x$. FID improves from $5.94$ on $x_0$ to $5.15$ on $x_{30}$, still the energy profile is not convergent. It would be more convincing if you provide a result for long-run MCMC (until energy profile convergence) revision for learned EBM to verify your EBM. \n\n4. Please include the training details, including optimization hyperparameters, step sizes for Langevin dynamics, etc.\n\nTypos:\n1. In the line 156 there is a typo for the marginal distribution of x, but inside the brackets you use a proper notation. \n2. In the line 214 “second” is used instead of “first”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging feedback and the correction of typos. We shall follow the suggestion and fix the typos in the revision. In the **General Response**, we provide clarifications for concerns regarding the motivation of our method, connection to Divergence Triangle, and computational cost  (see also the attached **PDF**).  \n\n---\n\n**Q1. Self-adversarial Interpretation of EBM Learning.** \n\nWe learn our EBM by minimizing $\\min_\\alpha KL(p_{\\rm data}(x)\\| \\pi_\\alpha(x)) - KL(T^x_{\\alpha_t} p_{\\theta_t}(x)\\|\\pi_\\alpha(x))$ (Eqn. 8), which actually lower bounds the traditional MLE objective. Such \"$KL -KL$\" objective has the self-adversarial interpretation, as the energy value for observed examples is learned to decrease, whereas it increases for model samples. Therefore, the model is learned to criticize its own samples at current iteration. On the other hand, the model sample along the MCMC iteration process (Eqn. 3) is adjusted through lowering its sample energy, which can be viewed to fool the EBM itself at the current iteration.\n\n**Q2. Long-run Langevin Trajectory.** \n\nWe conduct the long-run experiment and show the result in the attached **PDF**, where we run MCMC revision on $x$ for 3000 steps (much longer than 30 training steps). We observe only minor changes during the process, which indicates that the generator model could successfully match the long-run MCMC samples from EBM and approach the target distribution $\\pi_\\alpha(x)$.\n\n**Q3. Training Hyper-parameters.** \n\nWe shall include training details in the Appendix and open-source our implementation along with checkpoints.\n\n**Q4. Traversing Latent Space.** \n\nThank you for providing such an insightful suggestion. As stated in [1], the data distribution $p_{\\rm data}(x)$ and generator distribution $p_\\theta(x)$ need to have the same support to avoid the mode-dropping phenomenon. In other words, in the case when the generator model fails to cover some of the modes in data distribution, the latent sampling from such an induced model will also miss these modes. Addressing such an issue would require an additional treatment (Corollary 1 in [1]). Our explicit EBM, on the other hand, allows a further MCMC revision, so even when the generator model only partially covers the data distribution, such MCMC revision would still effectively guide the samples toward the data manifold. \n\n\n\nComparison of sample quality, wall-clock training time (seconds / per-iteration).\n\n|                | Traversing on $\\mathbf{z}$ | Traversing on $\\mathbf{x}$ |\n| :------------: | :------------------------: | :------------------------: |\n|      FID       |           40.35            |            9.26            |\n|    Time (s)    |           1.742            |           1.594            |\n| Langevin Steps |             30             |             30             |\n\n\n\nEmpirically, we conducted an initial experiment where we adapted our method by traversing the induced latent space for EBM sampling, rather than directly performing MCMC on the data space. The results can be found in the table below, where we observe a weaker performance of the adapted model. Additionally, such a sampling method would increase the training time as it requires back-propagation of the gradient through the generation network. For comparison, we use the same network structure and other hyper-parameters (e.g., Langevin steps, etc.),  while we shall explore its potential in greater depth in our future study. Thanks again for such an insightful comment.\n\n**Q5. Discussion of Limitation.** \n\nThe details of our computational cost can be found in **General Response**. We shall add a thorough discussion of the limitation, including our computational cost and the point (provided by **Reviewer** **uj7J**, Q1), in the revision.\n\n---\n\n[1] Che, Tong, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. Advances in Neural Information Processing Systems, 33, 2020."}]

Author rebuttal:
We thank the valuable comments from all the reviewers. The reviewers note that (1) our idea is *novel* (**Reviewer** **uj7J**, **RJbi**, **XcPg**), (2) our experiments are *clear* and *solid* (**Reviewer** **uj7J**, **iX9Y**, **XcPg**), (3) our paper is *well-written* (**Reviewer RJbi**, **XcPg**). We are also encouraged to hear that you found our paper addressed the *key limitation* and *actual problem* for future research (**Reviewer uj7J,** **XcPg**). 

We provide the general response for the common questions below and respond to each reviewer in their individual comments.  

We refer to the attached **PDF** for additional ablation results.

----

### G.1 Motivation (Reviewer **uj7J, RJbi, iX9Y, XcPg**).

**Dual-MCMC Teaching.** This paper studies the learning of the energy-based model (EBM). The EBM can be typically learned through the MLE, which involves MCMC sampling and is known to be challenging in practice. To tackle this challenge, we introduce a probabilistic learning framework, where a generator and inference model are jointly trained with the EBM.

Firstly, our joint learning scheme is designed to facilitate the intractable EBM and generator learning, and both generator and inference models are learned to jump-start MCMC samplings for EBM and generator posterior distribution, respectively. Such a joint learning scheme is related to work **Divergence Triangle** [1] (as noted by **Reviewer** **uj7J**, **XcPg**). However, DT relies on ancestral samples from both the generator and inference model, which may be sub-optimal. Additionally, the learning can be fundamentally different (please refer to **G.2** for a detailed breakdown and comparison). Secondly, the KL terms used in generator and inference model learning are designed to integrate two MCMC revised samples (one in the latent space and another in the image space) to steer the generator and inference learning (as highlighted by **Reviewer uj7J**). Consequently, this approach would enhance EBM learning by providing improved initial samples for MCMC samplings. We shall add a detailed discussion of the motivation in our final version.

### G.2 Connection to Divergence Triangle (DT) [1] (Reviewer **uj7J, XcPg**).

We denote $Q$ for $Q_{\phi}(x, z)$ and other joint densities for notation simplicity. Recall that $Q=q_\phi(z|x)p_{\rm data}(x)$,$P=p_\theta(x|z)p(z)$, $\Pi=\pi_\alpha(x)q_\phi(z|x)$, and $\tilde{Q}=p_{\rm data}(x)T^z_{\theta_t} q_{\phi_t}(z|x)$, $\tilde{P}=T^x_{\alpha_t} p_{\theta_t}(x|z)p(z)$, where $T^z_{\theta_t}(\cdot)$ denotes the Markov transition kernel of finite step Langevin dynamics that samples $z$ from $p_{\theta_t}(z|x)$, and $T^x_{\alpha_t}(\cdot)$ denotes the transition kernel that samples $x$ from $\pi_{\alpha_t}(x)$. 

- For learning the EBM ($\alpha$), we use $\min_\alpha KL(\tilde{Q}\| \Pi) - KL(\tilde{P}\|\Pi)$  (Eqn. 7), and DT considers $\min_\alpha KL(Q\| \Pi) - KL(P\|\Pi)$. Specifically, we utilize the MCMC revised samples (i.e., joint densities of $\tilde{Q}$ and $\tilde{P}$) for learning the EBM, while DT only considers ancestral samples (i.e., joint densities of $Q$ and $P$). Such MCMC-revised samples should render more effectively learned EBM, as model samples revised by the EBM itself can be more accurate than samples directly drawn from the generator, i.e., $KL(T^x_{\alpha_t}p_{\theta_t}(x)\|\pi_{\alpha_t}(x))\le KL(p_{\theta_t}(x)\|\pi_{\alpha_t}(x))$.
- For learning the generator model ($\theta$), the KL terms for our work are $\min_\theta KL(\tilde{Q}\| P) + KL(\tilde{P}\|P)$ (Eqn. 10), and DT's are $\min_\theta KL(Q\|P) + KL(P\| \Pi)$. In comparison, for training observation, our generator is learned with revised latent samples ($KL(\tilde{Q}\| P)$ v.s. $KL(Q\|P)$). Such samples can be more accurate towards the generator posterior, i.e., $KL(T^z_{\theta_t}q_{\phi_t}(z|x)\|p_{\theta_t}(z|x))\le KL(q_{\phi_t}(z|x)\|p_{\theta_t}(z|x))$. For generated model samples, our model, i.e., $KL(\tilde{P}\|P)$, learns to match the revised MCMC samples from $\pi_\alpha(x)$, while DT intends to chase the major modes of $\pi_\alpha(x)$ through variational approximation (i.e., $KL(P\| \Pi)$).
- For learning the inference model ($\phi$), we utilize $\min_\phi KL(\tilde{Q}\| Q) + KL(\tilde{P}\|\Pi)$ (Eqn. 15), and DT uses $\min_\phi KL(Q\|P) + KL(P\|\Pi)$. Two learning schemes can be very different. On the one hand, for training observation, our model, i.e., $KL(\tilde{Q}\| Q)$, is trained by amortizing the latent MCMC, while the DT follows variation inference as in VAEs (i.e., $KL(Q\|P)$). On the other hand, on generated model samples, our inference is learned to match the revised generator samples from $\pi_\alpha(x)$ (i.e., $KL(\tilde{P}\|\Pi)$), while the DT directly considers ancestral generator samples, i.e., $KL(P\|\Pi)$, which can be sub-optimal. 

### G.3 Computational and Memory Cost (Reviewer **uj7J, RJbi, iX9Y, XcPg**).

Our learning algorithm belongs to MCMC-based methods and can indeed incur computational overhead due to its iterative nature compared to variational-based or adversarial methods. We provide further analysis by computing the wall-clock training time and parameter complexity for our related work Divergence Triangle [1] (variational and adversarial-based joint training without MCMC) and our model (see Table 1 in the attached **PDF**), where the proposed work requires more training time but can also render significantly better performance. In terms of memory cost, it's important to note that we didn't observe further improvement by just increasing parameter complexity (see *Parameter Efficiency* in Section 1.1 of the Appendix). This emphasizes the effectiveness provided by our learning algorithm.

---

[1] Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training of generator model, energy-based model, and inferential model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019

Meta-review: The paper presents a significant advance in the field of energy-based models (EBMs). The authors identify and tackle existing limitations in Cooperative Learning frameworks, particularly focusing on the generator's constraint in only being able to reconstruct refined EBM samples. 

The proposed method effectively bridges existing methods, integrating elements from both Cooperative Learning and Divergence Triangle techniques to create a comprehensive framework that features MCMC sampling and generator data reconstruction.

The results from the experiments carried out provide strong support for the proposed methodology, indicating its advantages over existing EBM learning methods. Furthermore, the authors were engaging and responsive during the rebuttal period, providing reproducible code and clear theoretical explanations, which were well received by the reviewers.
Decision: Accept (poster)