Improving Adversarial Energy-Based Model via Diffusion Process

Cong Geng 1 Tian Han 2 Pengtao Jiang 1 Hao Zhang 1 Jinwei Chen 1 Søren Hauberg 3 Bo Li 1

Abstract

expression for the normalization constant. MCMC-based
EBMs (Du & Mordatch, 2019; Nijkamp et al., 2019) evaluate the gradient of the objective through Markov chain
Monte Carlo (MCMC) sampling on the defined energy function, which can be computationally expensive for both training and sampling. Adversarial EBMs (Grathwohl et al.,2021; Geng et al., 2021) introduce a generator to form aminimax game between alternative optimization of this generator and energy function, allowing for MCMC-free EBM
training and fast sampling.

Generative models have shown strong generation
ability while efficient likelihood estimation is less
explored. Energy-based models (EBMs) define
a flexible energy function to parameterize unnormalized densities efficiently but are notorious forbeing difficult to train. Adversarial EBMs introduce a generator to form a minimax training gameto avoid expensive MCMC sampling used in traditional EBMs, but a noticeable gap between adversarial EBMs and other strong generative models
still exists. Inspired by diffusion-based models,
we embedded EBMs into each denoising step tosplit a long-generated process into several smaller
steps. Besides, we employ a symmetric Jeffrey divergence and introduce a variational posterior distribution for the generator’s training to address themain challenges that exist in adversarial EBMs.
Our experiments show significant improvementin generation compared to existing adversarial
EBMs, while also providing a useful energy function for efficient density estimation.

Although adversarial EBMs have great potential in distribution modeling, they still have some limitations that canbe mainly attributed to three reasons. First, as is pointed
out in Mescheder et al. (2018) and Geng et al. (2021), minimax training can be unstable if two alternative optimization
steps are not well balanced. This instability poses a significant challenge in fitting the marginal energy distribution
through adversarial training, particularly when dealing withlarge-scale, complex, and multi-modal data distributions.
Secondly, most adversarial EBMs adopt KL divergence tooptimize the generator. Since KL divergence is an asymmetric measure of divergence, relying solely on KL divergence
may not be sufficient (Arjovsky et al., 2017). Thirdly, optimizing the generator requires computing an intractableentropy term, leading to a trade-off between generation anddensity estimation (Grathwohl et al., 2021).

1. Introduction
Energy-based models (EBMs) are known as one type ofgenerative models that draw inspiration from physics andhave been widely studied in machine learning (Hopfield,
1982; Hinton & Sejnowski, 1983; Smolensky, 1986). EBMs
define an unnormalized probability distribution over data
space from a Gibbs density, which can be useful for several visual tasks, such as image classification (Grathwohl
et al., 2020), out-of-distribution (OOD) detection (Liu et al.,2020) or semi-supervised learning (Gao et al., 2020). However, training an EBM using maximum likelihood estimation can be challenging due to the lack of a closed-form

To address these issues, we get inspiration from diffusionbased models (Ho et al., 2020) which specify a diffusion
process to transform a data distribution by adding Gaussian noise with multiple steps until obtaining an approximate Gaussian distribution. A denoising diffusion process
is learned by minimizing the KL divergence between true
conditional denoising distribution q (xt−1 | xt ) and a parameterized conditional distribution pθ (xt−1 | xt ) at eachnoise step. Diffusion Recovery Likelihood (DRL) (Gaoet al., 2021) and Denoising diffusion GAN (DDGAN) (Xiaoet al., 2022) combined EBM or GAN with diffusion processrespectively. They trained a sequence of EBMs or GANs bymatching true and modeled denoising distributions at eachtime step. The efficacy of these methods demonstrates thatmatching two denoising distributions is more tractable, as
sampling from these conditional distributions is much easierthan sampling from their marginal distributions. However,
DDGAN lacks a useful density estimate for its discriminator

1

vivo Mobile Communication Co., Ltd, China 2 Departmentof Computer Science, Stevens Institute of Technology, USA
3
Technical University of Denmark, Copenhagen, Denmark. Correspondence to: Søren Hauberg <sohau@dtu.dk>, Bo Li<ibra@vivo.com>.
Proceedings of the 41 st International Conference on Machine
Learning, Vienna, Austria. PMLR 235, 2024. Copyright 2024 by
the author(s).

1Improving Adversarial Energy-Based Model via Diffusion Process

and DRL has only one energy function, making sampling inefficient. These limitations are precisely where the strengthsof EBM lie. Based on these arguments, we also incorporate
adversarial EBMs into diffusion processes at each noise
step. Thus for each EBM, the target distribution is a conditional distribution that is less multi-modal and easier to learn.
Besides, with our defined generated denoising distribution
through a generator, a symmetric Jeffrey divergence (Jeffreys, 1946) can be easily employed to remedy inadequate
fitting, and a variational posterior distribution is introduced
to compute the entropy term. Therefore, three main challenges in training adversarial EBMs can be overcome. To
our knowledge, it is the first time that adversarial EBM has
been well integrated into the framework of diffusion models.
In summary, the contributions of our paper are as follows:

optimization problem respectively. Adversarial EBMs inherit the advantages of GANs and avoid MCMC sampling,
but they also bring the risk of unstable training.
The original diffusion-based model (Ho et al., 2020; SohlDickstein et al., 2015) learns a finite-time reversed diffusionfrom a forward process and gets strong performance on image generation, but the sampling process is slow. Several
works (Song et al., 2021a; Lu et al., 2022b; Bao et al., 2022)
improve training-free samplers and significantly improve
the sampling speed going from 1000 steps sampling to only
a few. DDGAN (Xiao et al., 2022) proposed to model eachdenoising step using a multimodal conditional GAN, tackling the generative learning trilemma of fast sampling while
maintaining strong mode coverage and sample quality. However, its discriminator fails to provide a density estimate,which is crucial for many tasks (Du et al., 2023; Zhang et al.,2022; Du et al., 2022).

• We propose an MCMC-free training framework for
EBMs to incorporate a sequence of adversarial EBMs
into a denoising diffusion process. This framework
avoids MCMC in both training and sampling.

Although some diffusion-based methods (Kingma et al.,2021; Song et al., 2021b; Lu et al., 2022a) use score functions or variational lower bounds to estimate the density ofthe data distribution, these methods tend to be complicatedand inefficient, which is precisely where energy-based models excel. DRL (Gao et al., 2021) and CDRL (Zhu et al.,2024) tried to combine EBM with a diffusion-based modeland got state-of-the-art generation performance on several
image datasets. Still, they relied on MCMC sampling which
remains inefficient.

• We learn EBMs by optimizing conditional denoising
distributions instead of marginal distributions to alleviate the training burden. We train the generator byminimizing a symmetric Jeffrey divergence to help distribution matching and introduce a variational posterior
distribution to compute the entropy term.
• We demonstrate that our model achieves significant
improvements on sample quality compared to existing
adversarial EBMs. We also verify the ability of ourenergy function in density estimation.

3. Denoising Diffusion Adversarial EBM
Preliminaries Let x ∼ q(x) be a training example froman underlying data distribution. An EBM is specified interms of an energy function Eθ : X → R, that is parameterized by θ, defines a probability distribution over X from the
Gibbs distribution:1
exp (Eθ (x)) ,(1)pθ (x) =Zθ

2. Related Work
There is a long history of EBMs in machine learning datingback to Hopfield networks (Hopfield, 1982) and Boltzmann
machines (Hinton & Sejnowski, 1983). Recently, deep
EBMs have grown in popularity, especially for image
generation, and can broadly be classified into two categories.
MCMC-based methods simulate Markov chains during
training (Du & Mordatch, 2019; Pang et al., 2020; Arbel
et al., 2021) or sampling (Song et al., 2020; Song & Ermon,
2019). This can be expensive, slow, and hard to control.
Cooperative learning methods (Xie et al., 2020; 2021; 2022;
Cui & Han, 2023) jointly train a generator and an energyfunction for MCMC teaching, using the generator as a fast
initializer for Langevin sampling to alleviate the MCMC
burden. They, however, remain inefficient.

where Zθ is the normalization constant (or partition function). In principle, any density can be described in this way
for a suitable choice of Eθ . We learn the EBM throughmaximum likelihood estimation (MLE), i.e. we seek θ∗ thatmaximizes the data log-likelihood:

The other category is adversarial EBMs (Zhai et al., 2016;
Geng et al., 2021; Kumar et al., 2019; Zhao et al., 2017),
which introduce a ’generator’ to form a minimax game between the energy function and the introduced generator. Han
et al. (2019; 2020) and Kan et al. (2022) explained adversarial EBM from the view of triangle divergence and bi-level

The fundamental challenge for training EBMs is the lack of
a closed-form expression for the normalization constant. A
common way to solve this is to approximate the gradient ofL(θ) directly and apply gradient-based optimization:




∂L
∂
∂
= Ex∼q(x)Eθ (x) − Ex∼pθ (x)Eθ (x) . (3)
∂θ
∂θ
∂θ

L(θ∗ ) := max Ex∼q(x) [log pθ (x)]θ= max{Ex∼q(x) [Eθ (x)] − log Zθ }.

(2)

θ

2

Improving Adversarial Energy-Based Model via Diffusion Process

Z

Estimating the above equation needs MCMC sampling fromthe energy distribution pθ (x), which is both unstable andexpensive. Adversarial EBMs (Grathwohl et al., 2021; Geng
et al., 2021; Han et al., 2019; Kan et al., 2022) avoid MCMC
by introducing a variational distribution pϕ to approximatepθ . Therefore, a minimax game can be formed to alternately
optimize two adversarial steps:

where Eθ (xt , t) : RD × R → R is the energy function
parameterized by θ. Thus L can be simplified and expressedby energy function:XL=
Eq(xt−1 ,xt ) Eθ (xt−1 , t − 1)
t≥1
(7)

+ log q(xt |xt−1 ) − log Z̃θ,t (xt ) .

1. min D(pϕ , pθ ) with a certain proper divergence D.
pϕ
2. max Ex∼q(x) [Eθ (x)] − Ex∼pϕ (x) [Eθ (x)] .

It is easy to see that when Eq. (7) is optimized, pθ (xt ) =exp (Eθ (xt ,t))= q(xt ) (see Appendix).
Zθ,t

Eθ

This two-step training strategy can be interpreted as a bilevel optimization problem (Liu et al., 2022), where theminimization step is a lower-level (LL) subproblem and the
maximization step an upper-level (UL) subproblem. Geng
et al. (2021) demonstrated that if pϕ fails to be optimized,
it runs the risk of maximizing an upper bound, causing
a noticeable performance gap compared with mainstream
generative models in complex and sparse datasets.

Since Z̃θ,t is intractable, if we follow the common choice
to compute the gradient of the objective:


∂L X
∂
Eq(xt−1 ,xt )=Eθ (xt−1 , t − 1)
∂θ
∂θ
t≥1
(8)


∂
− Eq(xt )pθ (xt−1 |xt )Eθ (xt−1 , t − 1) ,∂θ
the second term has to be estimated by MCMC-samplingfrom the distribution q(xt )pθ (xt−1 |xt ) (Gao et al., 2021),
which can be computational demanding and unstable during
training.

3.1. Adversarial EBM with Denoising Diffusion Process
To make adversarial EBMs scalable to complex distributions, we borrow the diffusion-based framework to split
the generation process into multiple steps. For each step,
we only need to learn a conditional distribution rather thana complex marginal distribution. Similar to DDPM (Hoet al., 2020), we also use a Markov chain for the diffusion
process. Specifically, for the forward process, we gradually add noise to the real data x0 ∼ q(x0 ) in T steps withpre-defined variance schedule βt :Yq (x1:T | x0 ) =q (xt | xt−1 ) ,(4)

Therefore, we adopt an adversarial EBM by introducing avariational conditional distribution pϕ (xt−1 | xt ) parameterized by ϕ to form a two-step minimax game:
DDAEBM minimax gameP
1. min t≥1 Eq(xt ) [D (pϕ (xt−1 |xt ), pθ (xt−1 |xt ))]pϕ

2. maxEθ

t≥1

q (xt | xt−1 ) = N xt ;pXEq(xt−1 ,xt ) Eθ (xt−1 , t − 1)−t≥1

Eq(xt )pϕ (xt−1 |xt ) Eθ (xt−1 , t − 1)



1 − βt xt−1 , βt I .

This adversarial training strategy alternately optimizes
pϕ and Eθ , maintaining one as fixed while optimizing
the other. This gives a tractable and MCMC-free approach to EBM training. We simply refer to our proposed
method as Denoising Diffusion Adversarial Energy-Based
Model (DDAEBM).

Then we can get the denoising distribution q (xt−1 | xt ) =q(xt−1 )q(xt |xt−1 )as our target distribution to learn. In
q(xt )
diffusion-based models, the regular training objective is
to minimize the KL divergence between the true denoising
distribution q (xt−1 | xt ) and a modeled one pθ (xt−1 | xt )for each t, i.e.
XL =−Eq(xt ) [DKL (q (xt−1 | xt ) ∥pθ (xt−1 | xt ))]+C.

3.2. Minimization w.r.t. pϕt≥1First, we need to have a specific form of our introduced variational conditional distribution pϕ (xt−1 |xt ).
DDIM (Song et al., 2021a) and DDPM (Ho et al., 2020)
define pϕ (xt−1 |xt ) as a Gaussian distribution to match theGaussian posterior distribution q (xt−1 | xt , x0 ). However,as demonstrated in DRL, this normal distribution approximation is only accurate when βt is small, it may not be reasonable when there are few denoising steps, as the denoising(5)
Inspired by DRL (Gao et al., 2021), we design a sequenceof conditional EBM using an energy function conditioned
on t to define our modeled denoising distribution,pθ (xt−1 | xt ) =exp(Eθ (xt−1 , t − 1))q(xt |xt−1 )dxt−1Z̃θ,t (xt ) =exp(Eθ (xt−1 , t − 1))q(xt |xt−1 )Z̃θ,t (xt )(6)3

Improving Adversarial Energy-Based Model via Diffusion Process

distributions can be complex and multimodal. Therefore,
we define pϕ (xt−1 |xt ) with a reparameterization trick,Z
pϕ (xt−1 | xt ) := pϕ (x0 | xt ) q (xt−1 | xt , x0 ) dx0
Z= p(z)q (xt−1 | xt , x0 = Gϕ (xt , z, t)) dz,(9)have been proposed for efficient entropy approximation, butthese are all either time or memory-demanding. We need tocompute a conditional entropy:
H[pϕ (xt−1 |xt )] = H[pϕ (xt−1 , z|xt )] − H[pϕ (z|xt−1 , xt )]
(14)
where

where pϕ (x0 | xt ) is an implicit distribution imposed by aneural network called the generator Gϕ (xt , z, t) : RD ×
Rd × R → RD . Here, p(z) is a d-dimensional latent variable following a standard Gaussian distribution N (z; 0, I),
and q (xt−1 | xt , x0 ) is the posterior distribution defined inforward diffusion process. It is worth noting that this definition is also explored by Xiao et al. (2022), which has beenverified to be effective in fitting complex and multi-modal
conditional distributions, although they don’t leverage an
EBM structure to characterize the data distribution.

pϕ (xt−1 , z|xt ) = p(z)q (xt−1 | xt , x0 = Gϕ (xt , z, t)) .
(15)
H[pϕ (z|xt−1 , xt )] is intractable while H[pϕ (xt−1 , z|xt )]is a constant that can be ignored (see Appendix). Minimizing Eq. (13) can be problematic since we do not have accessto H[pϕ (z|xt−1 , xt )], and we instead minimize a variational
upper bound by introducing an approximate Gaussian posterior qψ (z|xt−1 , xt ). Here the mean and variance are represented through an encoder parameterized by ψ with xt−1 ,xt and t as the input. We can easily obtain:

The KL divergence is, by far, the most common choice of divergence D for adversarial EBM training (Geng et al., 2021;
Grathwohl et al., 2021; Zhai et al., 2016). However, giventhat the KL divergence is an asymmetrical metric, it compels pϕ (xt−1 |xt ) to chase the major modes of pθ (xt−1 |xt ).
Relying solely on the KL divergence as our objective can,therefore, be insufficient for the generator to effectively
capture the energy distribution (Geng et al., 2021). Following Kan et al. (2022), we choose a symmetric Jeffrey
divergence, to integrate both KL divergence and reverse KL
divergence into our objective:

H[pϕ (z|xt−1 , xt )] ≤ −Eq(xt )pϕ (xt−1 ,z|xt ) log qψ (z|xt−1 , xt ).
(16)
Expressing log pθ (xt−1 |xt ) with Eq. (6) and applying
Eq. (16) to Eq. (13), the first term Eq. (11) can be minimized by its simplified variational upper bound:
Upper bound of L1
L1 ≤

X

h
Eq(xt )pϕ (xt−1 ,z|xt ) −Eθ (xt−1 , t−1)

(17)

t≥1

i
− log q(xt |xt−1 ) −log qψ (z|xt−1 , xt )Ideal objective
min
pϕ

X

Bounding the second term of the ideal objective. We
now proceed with the minimization of our ideal objective,
treating the second term Eq. (12). This term can also
be bounded from above by utilizing an Evidence Lower
Bound (ELBO) associated with our introduced Gaussian
posterior qψ (z|xt−1 , xt ), since
Eq(xt ) DKL (pϕ (xt−1 |xt )∥pθ (xt−1 |xt ))t≥1

+ DKL (pθ (xt−1 |xt )∥pϕ (xt−1 |xt )) .

(10)

We divide our ideal objective into two terms for separate
handling:
X
L1 =Eq(xt ) DKL (pϕ (xt−1 |xt )∥pθ (xt−1 |xt )), (11)log pϕ (xt−1 |xt ) ≥ −DKL (qψ (z|xt−1 , xt )∥p(z))+ Eqψ (z|xt−1 ,xt ) log q(xt−1 |xt , Gϕ (xt , z, t))

(18)

t≥1

L2 =

X

We can get the upper bound of Eq. (12) by applying Eq. (18):Eq(xt ) DKL (pθ (xt−1 |xt )∥pϕ (xt−1 |xt )). (12)Upper bound of L2t≥1

Bounding the first term of the ideal objective. To minimize our ideal objective, we first handle Eq. (11). It can be
simplified by omitting some unrelated terms and obtaining:X−H[pϕ (xt−1 |xt )] − Eq(xt )pϕ (xt−1 |xt ) log pθ (xt−1 |xt )

L2 ≤

X
t≥1

Eq(xt )pθ (xt−1 |xt ) DKL (qψ (z|xt−1 , xt )∥p(z))(19)−Eq(xt )pθ (xt−1 |xt )qψ (z|xt−1 ,xt ) log q(xt−1 |xt , Gϕ (xt , z, t))

t≥1

(13)
Computing the entropy of the generated distribution is always a challenging task in EBMs. Several approaches (Kumar et al., 2019; Grathwohl et al., 2021; Geng et al., 2021)

A Monte Carlo approximation of Eq. (19) requires samplingfrom q(xt )pθ (xt−1 |xt ). As pθ (xt−1 |xt ) is designed to fitq(xt−1 |xt ), we use samples of q(xt−1 , xt ) with an importance ratio to replace those of q(xt )pθ (xt−1 |xt ), as done in4Improving Adversarial Energy-Based Model via Diffusion Process

BiDVL (Kan et al., 2022):pθ (xt−1 |xt ) = λ(t − 1)q(xt−1 |xt )(20)

The importance ratio is designed inspired by ACT (Konget al., 2023):
λ(t − 1) = w (t′ (t))

log 1 (
2

wmid
w

)

,

(21)

where w and wmid are the hyperparameters selected basedon the datasets. t′ (·) is a function denoting the time of xt inthe Variance Preserving (VP) SDE (Song et al., 2021c), as
our diffusion process can be viewed as the discretization of
the continuous-time VP SDE (see Appendix). The ultimate
overall objective for implementation of the minimization
step is the sum of Eq. (17) and Eq. (19) w.r.t pϕ and qψ .

MEG

VERA

FCE

BiDVL DDAEBM

Figure 1. Density estimation and generation on 25-Gaussians andpinwheel datasets. For each dataset, the first row shows the estimated densities and the second row shows generated samples.

3.3. Maximization step w.r.t. Eθ

of the network. For the energy function, we adopt traditional NCSN++ or DDPM++ architectures from Score SDE
except that we remove the last scale-by-sigma operation andreplace it with a negative ℓ2 norm between input xt and theoutput of the Unet (Du et al., 2023), i.e.

After minimizing step w.r.t pϕ and qψ , we approximately
assume pϕ (xt−1 |xt ) = pθ (xt−1 |xt ). Next, we should optimize the energy function by maximizing:XEq(xt−1 ,xt ) Eθ (xt−1 , t − 1)
t≥1
(22)

− Eq(xt )pϕ (xt−1 |xt ) Eθ (xt−1 , t − 1)

1Eθ (xt , t) = − ∥xt − fθ (xt , t) ∥2 .
2

Similar to most adversarial EBMs, we also find it helpful tostabilize training by adding a ℓ2 -regularizer for the gradientof our energy:hiγ2Eq(xt−1 ) ∇xt−1 [Eθ (xt−1 , t − 1) + log q(xt |xt−1 )]2(23)where γ is the regularization coefficient. This regularization
is a gradient penalty that alleviates training instability due
to insufficient training at the minimization step (Grathwohl
et al., 2021; Kumar et al., 2019).

(24)

4.1. Performance on 2D synthetic data
Fig. 1 demonstrates the density estimation and generation
results of our DDAEBM on the 25-Gaussians and pinwheeldatasets, which are two challenging toy datasets. We compare our model with several mainstream adversarial EBMs.
These methods are all MCMC-free through training a generator and an energy function alternatively in two steps.
We can observe that only our DDAEBM can get satisfying
performance both for generation and density estimation onthese two datasets. For pinwheel, most methods fail on density estimation except FCE and our DDAEBM. FCE (Gaoet al., 2020) trains an energy function using constrastive
learning which has a strong ability on density estimation,
but it uses a GLOW network as its generator which limits
its ability on generation. For 25-Gaussians, although thegeneration of our DDAEBM is more dispersed than thatof BiDVL around each mode, samples are mostly centeredaround each mode instead of being between two modes.

4. Experiments
We evaluate our DDAEBM in different scenarios acrossdifferent data scales from 2-dimension toy datasets to largescale image datasets. We test our energy function mainly on
toy datasets and MNIST datasets which are easy to visualize
and intuitive to measure. For large-scale datasets, we focus
on image generation. We further perform some additional
studies such as out-of-distribution (OOD) detection and ablation studies to verify our model’s superiority. We briefly
introduce the network architecture design, while additional
implementation details are presented in the Appendix. For
the generator trained on large image datasets, we adopt thesame modified NCSN++ architecture as DDGAN (Xiaoet al., 2022) or DDPM++ with a slightly different structure
as described in Score SDE (Song et al., 2021c), where diffused samples xt , time t and latent variable z are the input

4.2. Fitting flow models with energy function
Since our energy function is an unnormalized likelihood estimator with a difficult-to-estimate normalization constant,
we consider an example where we can evaluate the exact
log-likelihood associated with our energy function. Following Grathwohl et al. (2021), we train the NICE model (Dinhet al., 2015) as an energy function on the MNIST dataset.
NICE is a normalizing flow model that allows for exact
5

Improving Adversarial Energy-Based Model via Diffusion Process
Table 1. Log-likelihoods for NICE models as the energy function
MethodMLE

MLE(t)

SSM(Song et al., 2020)

Test LL

-791

-879

-2039

DSM
(Vincent, 2011)

CoopNet(Xie et al., 2020)

WGAN-0GP(Mescheder et al., 2018)

MEG(Kumar et al., 2019)

VERA(Grathwohl et al., 2021)

DDAEBM(ours)

-4363

-1465

-1214

-1023

-1021

-902

likelihood computation and sampling. We get -879 on the
MLE test using our network, which is close to the -791 of
the traditional NICE model provided by Song et al. (2020).
Data is preprocessed following Grathwohl et al. (2021).
We observe that using the NICE network leads to worse
WGAN-0GP

MEG

VERA

CelebA (Liu et al., 2015), and 128×128 LSUN church (Yuet al., 2015) datasets. Each dataset represents a significant
increase in complexity. All datasets are scaled to [−1, 1]
for pre-processing. For quantitative results, we adopt commonly used Fréchet inception distance (FID) and Inception
Score (IS) to evaluate sample fidelity and the number offunction evaluations (NFE) to evaluate sampling time. Table 2 shows quantitative results on CIFAR-10. We observe
our model achieves FID 4.82 and IS 8.86, outperforming
existing adversarial EBMs by a significant margin and performing comparably to strong baselines from other generative models such as GANs and diffusion-based models,
which struggle with efficient density estimation. Note that

DDAEBM

Exact Samples

Generator Samples

Table 2. Results for generation on CIFAR-10 dataset

Figure 2. Exact samples from the NICE model and generated
samples from the generator for WGAN-based methods and our
DDAEBM.

Model

FID↓

IS↑

NFE↓

IGEBM (Du & Mordatch, 2019)
JEM (Grathwohl et al., 2020)
EBM-SR (Nijkamp et al., 2019)
EBM-CD (Du et al., 2021)
Hat EBM (Hill et al., 2022)
DAMC (Yu et al., 2023)
CoopNets (Xie et al., 2020)
VAEBM (Xiao et al., 2021)
CLEL-large (Lee et al., 2023)
DRL (Gao et al., 2021)

38.2
38.4
44.50
25.1
19.30
57.72
33.61
12.2
8.61
9.58

6.78
8.76
6.21
7.85
6.55
8.43
8.30

60
20
100
500
50
100
50
16
1200
180

EBM Triangle (Han et al., 2019)
MEG (Kumar et al., 2019)
VERA (Grathwohl et al., 2021)
EBM-BB (Geng et al., 2021)
FCE (Gao et al., 2020)
BiDVL (Kan et al., 2022)
DDAEBM(ours)

28.96
35.02
27.5
28.63
37.30
20.75
4.82

7.30
6.49
7.45
8.86

1
1
1
1
1
1
4

21.7
14.73
2.92
10.87
4.67
2.20
3.75
48.9

8.22
9.22
9.83
8.40
8.78
9.89
9.63
3.92

1
1
1
1000
50
2000
4
1

Energy-based models

performance of our DDAEBM than other WGAN-based
methods such as WGAN-0GP, MEG, and VERA. The reason is that for WGAN-based methods, the gradients of theenergy function’s output are significantly smaller for fakesamples compared to real samples, making the training ofenergy function similar to MLE. Therefore we add a weightof 0.1 for the second term in Eq. (22) to reduce the effect offake samples. We use the same trick on WGAN-based methods and find it can also improve their likelihood fitting but
help little on generation even using a larger generator. We
use an MLP network for our generator. Other experimentsettings are the same as VERA (Grathwohl et al., 2021).
Full experimental details can be found in the Appendix.

MCMC

No MCMC

Other Generative Models
SNGAN (Miyato et al., 2018)
BigGAN (Brock et al., 2019)
StyleGAN2 w/ ADA (Karras et al., 2020a)
NCSN-v2 (Song & Ermon, 2020)
DDIM (Song et al., 2021a)
Score SDE (Song et al., 2021c)
DDGAN (Xiao et al., 2022)
GLOW (Kingma & Dhariwal, 2018)

From Table 1 we observe that our method obtains the maximum log-likelihood on test data except MLE which maximizes the log-likelihood of training data as its objective.
Fig. 2 shows the exact samples from NICE models andgenerated samples from the generator. Although all WGANbased methods yield a good fit of log-likelihood, we canobserve that WGAN-0GP and MEG fail to generate diverseand good-quality samples, VERA performs better but still
has some mode collapse, our DDAEBM generates highquality samples that match exact samples and reasonably
capture the data distribution.

our DDAEBM is still less performant than DDGAN eventhough they use the same generator structure and parameterization of the generated denoising model. We further add
an additional denoising step (Song & Ermon, 2020) using
our energy function to refine the generated samples, we findthe FID score can be improved to 3.73, which is on parwith DDGAN. This denoising step can be approximately
viewed as a one-step MCMC using our energy function.
With this minor modification, we successfully bridge thegap between DDGAN and DDAEBM in terms of generation
performance. Besides, the energy function in DDAEBM is

4.3. Image generation
To confirm the ability of our model to scale to large-scale
datasets effectively, we conduct experiments of generation
task on 32×32 CIFAR-10 (Krizhevsky et al., 2009), 64×64
6

Improving Adversarial Energy-Based Model via Diffusion Process

Figure 3. Randomly generated images with DDAEBM on 32 × 32 CIFAR-10, 64 × 64 CelebA and 128 × 128 LSUN church datasets.

tasked with providing a density estimate, not merely functioning as a discriminator. Therefore, the energy function in
DDAEBM carries a heavier workload than the discriminator in DDGAN. In contrast, the discriminator in DDGAN
merely distinguishes between real and fake sample pairs,
primarily concentrating on the quality of the samples rather
than on optimizing data likelihood. This explains why the
network of our energy function is stronger than the discriminator of DDGAN.

of-distribution (OOD) regions and high values on data regions, which is suitable for detecting OOD samples. To
test this, we first train our DDAEBM on CIFAR-10 andcalculate the energy Eθ (x0 , 0) on in-distribution images
from the CIFAR-10 test set and out-of-distribution images
from datasets including SVHN (Netzer et al., 2011), Texture (Cimpoi et al., 2014), CIFAR-100 (Krizhevsky et al.,2009) and CelebA. Following Xiao et al. (2021), we use
the area under the ROC curve (AUROC) as a quantitative
metric on the energy scores, where high AUROC indicatesthat the model correctly assigns low density to OOD samples. Results are shown in Table 4, from where we can see
our model achieves comparable performance with most of
the baselines chosen from recent EBMs, VAEs, and Glow.
Our model performs the best on SVHN and Texture datasets
while on CIFAR-100 and CelebA it performs at a moderatelevel. Note that VAEBM performs slightly better than ourson most datasets, but it requires MCMC sampling to refine
its generation, which can be inefficient. As is pointed out
by Zhang et al. (2021), no method can guarantee better thanrandom chance performance without assumptions on which
out-distributions are relevant. Hence we see no clear winneron this task across all the datasets and we should take theresults with a grain of salt.

For CelebA, we only report FID scores in Table 3 since
the IS score is not widely reported. Our model gets thebest performance among adversarial EBMs and outperforms score matching-based model NCSNv2 and VAEbased model NVAE. For LSUN church, Fig. 3 depicts high-Table 3. FID scores on CelebA 64× 64 dataset

Model

FID↓

SNGAN (Miyato et al., 2018)
COCO-GAN (Lin et al., 2019)

50.4
4.0

NVAE (Vahdat & Kautz, 2020)

14.74

VAEBM (Xiao et al., 2021)
NCSNv2 (Song & Ermon, 2020)
DDPM (Ho et al., 2020)
DRL (Gao et al., 2021)

5.31
26.86
3.50
5.98

joint EBM Triangle (Han et al., 2020)
BiDVL (Kan et al., 2022)
DDAEBM (ours)

24.7
17.24
10.29

Fig 4 further shows histograms of the energy output at t = 0
between the CelebA test dataset and several OOD distributions. For fake data, we diffuse real data x0 for one step anduse pϕ (x0 |x1 ) to generate fake data. For noisy data, we add
noise to the real data with standard deviations of 0.01, 0.1,
and 0.5. From Fig. 4 we observe our energy function assigns
high values to real data and low values to fake and noisy
data. Additionally, the energy values gradually decrease as
the added noise increases. Although fake data and noisy
data with 0.01 standard deviation have imperceptible differences for visualization, their energy values still exhibit clear
distinctions when compared to real data. This demonstrates
our energy function’s capability for OOD detection.

fidelity synthesis sampled from the generator. We calculate
FID on 50,000 samples using a PyTorch implementationfrom DDGAN and our model achieves 13.80 for this metric.
4.4. Out of distribution detection
The energy function of EBMs can be viewed as an unnormalized density function that assigns low values on out7

Improving Adversarial Energy-Based Model via Diffusion Process

Figure 4. Histogram of unnormalized log-likelihood for comparison of real data and fake data or noisy data with the standard deviations
being 0.01, 0.1, and 0.5. We provide the histogram comparison on CelebA test images.

Table 4. AUROC for out-of-distribution detection on SVHN, Texture, CIFAR-100 and CelebA test datasets, with CIFAR-10 as thein-distribution dataset.
Model

SVHN

TextureCIFAR-100

CelebA

IGEBM (Du & Mordatch, 2019)
Glow (Kingma & Dhariwal, 2018)
NVAE (Vahdat & Kautz, 2020)
SVAE (Chen et al., 2018)
joint EBM Triangle (Han et al., 2019)
VERA (Grathwohl et al., 2021)
BiDVL (Kan et al., 2022)
VAEBM (Xiao et al., 2021)
DDAEBM(ours)

0.63
0.05
0.42
0.42
0.68
0.83
0.76
0.83
0.83

0.48
0.5
0.56
0.62

0.5
0.55
0.56
0.73
0.62
0.60

0.7
0.57
0.68
0.52
0.56
0.33
0.77
0.77
0.70

Figure 5. The FID score vs. the number of training epochs for adifferent number of time steps.

4.5. Ablation studies
score versus training epochs for different time steps in Fig 5.
Note that T = 1 corresponds to training a standard adversarial EBM, which leads to inadequate performance andunstable training. T = 2 can get decent results very early
but becomes unstable in later stages. T = 4 gives excellent
generation results and stable training which is consistent
with DDGAN. When T becomes larger, the training can bestable but there is an obvious degradation in performance.
We hypothesize that although training for each step can beeasier with a larger T , a higher capacity of the energy function is also required to accommodate more time steps. Thus,
the choice of T is important. We consistently chose T = 4.

Importance of proposed modifications. First, we investigate the effects of some proposed modifications in ourmodel including the latent variable in our defined denoising
distribution, introduced posterior qψ (z|xt−1 , xt ) and Jeffrey divergence for the generator’s training. Table 5 reports
FID and IS scores on CIFAR-10 dataset and AUROC for the
OOD task with CIFAR-10/SVHN as the in-distribution/outof-distribution datasets. We replace sampling from p(z) in
Eq. (9) with all zero-vectors to remove the effect of z andobserve the FID and IS scores are significantly worse. We
also train our model with the most commonly used KL divergence and get similar performance on generation, but
OOD performance significantly drops. We further removelog qψ (z|xt−1 , xt )-related term in our objective, which isequivalent to training a sequence of WGAN-0GPs embedded in a diffusion process. The generation and OOD performance are similar to the model trained with KL divergence.
This experiment implies that asymmetric KL divergenceand lack of log qψ (z|xt−1 , xt ) lead to inaccurate learningof the energy function which, in turn, affects e.g. OOD
performance. Symmetric Jeffrey divergence and entropy
term qψ (z|xt−1 , xt ) ensure a better fit between the generated distribution and the energy function. This improves
the energy training, resulting in improved OOD detection.
More results can be found in the Appendix.

Table 5. Ablation studies for some proposed modifications
OOD
Model Variants
FID↓ IS↑(CIFAR-10/SVHN)↑
No latent variable
KL divergence
KL w/o qψ (z|xt−1 , xt )DDAEBM

10.09
4.99
4.90
4.82

8.25
9.11
8.90
8.86

0.23
0.37
0.38
0.83

5. Conclusion
We propose to integrate adversarial EBMs into the denoising diffusion process to learn the complex multimodal data
distribution in several steps. This allows us to split a long
generated process into several small steps, making each

Number of time steps. We also examine the influenceof varying the number of time steps T by plotting the FID
8Improving Adversarial Energy-Based Model via Diffusion Process

EBM easier to train. We define the generated denoising
distribution by introducing a latent variable z, which greatly
accelerates sampling. With this definition, we employ asymmetric Jeffrey divergence to calibrate the training ofour generator and introduce a variational posterior distribution qψ (z|xt−1 , xt ) to compute the entropy term, these
operations address the long-standing training challenges
existing in adversarial EBMs. Our model reduces the gap
between adversarial EBMs and current mainstream generative models in terms of generation and provides a usefulenergy function with notable potential for a wide range ofapplications and downstream tasks.

Conference on Artificial Intelligence and Statistics, pp.
661–669. PMLR, 2018.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., andVedaldi, A. Describing textures in the wild. In IEEE
Conference on Computer Vision and Pattern Recognition,
pp. 3606–3613, 2014.
Cui, J. and Han, T. Learning energy-based model via dualmcmc teaching. Advances in Neural Information Processing Systems, 2023.
Dieng, A. B., Ruiz, F. J., Blei, D. M., and Titsias, M. K. Prescribed generative adversarial networks. arXiv preprint
arXiv:1910.04302, 2019.

Acknowledgements

Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear
independent components estimation. In International
Conference Learning Representations Workshops, 2015.

This work was supported by a research grant (42062) fromVILLUM FONDEN. This project received funding fromthe European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement 757360). The work was partly
funded by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science(NNF20OC0062606).

Du, Y. and Mordatch, I. Implicit generation and generalization in energy-based models. Advances in Neural
Information Processing Systems, 2019.
Du, Y., Li, S., Tenenbaum, J., and Mordatch, I. Improved
contrastive divergence training of energy based models.
International Conference on Machine Learning, 2021.

Impact Statement

Du, Y., Li, S., Tenenbaum, J., and Mordatch, I. Learning iterative reasoning through energy minimization. In
International Conference on Machine Learning, pp. 5570–
5582. PMLR, 2022.

This paper presents work whose goal is to advance the fieldof Machine Learning. There are many potential societal
consequences of our work, as all high-capacity generativemodels carry the risk of being used for misinformation, andour model is no exception, none of which we feel must be
specifically highlighted here.

Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., andGrathwohl, W. S. Reduce, reuse, recycle: Compositional
generation with energy-based diffusion models and mcmc.
In International Conference on Machine Learning, pp.
8489–8510. PMLR, 2023.

References
Arbel, M., Zhou, L., and Gretton, A. Generalized energy
based models. International Conference on Learning
Representations, 2021.

Gao, R., Nijkamp, E., Kingma, D. P., Xu, Z., Dai, A. M.,
and Wu, Y. N. Flow contrastive estimation of energybased models. In IEEE Conference on Computer Vision
and Pattern Recognition, pp. 7518–7528, 2020.

Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In International Conference
on Machine Learning, pp. 214–223. PMLR, 2017.

Gao, R., Song, Y., Poole, B., Wu, Y. N., and Kingma, D. P.
Learning energy-based models by diffusion recovery likelihood. International Conference on Learning Representations, 2021.

Bao, F., Li, C., Zhu, J., and Zhang, B. Analytic-dpm: ananalytic estimate of the optimal reverse variance in diffusion probabilistic models. International Conference on
Learning Representations, 2022.
Brock, A., Donahue, J., and Simonyan, K. Large scale gan
training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019.

Geng, C., Wang, J., Gao, Z., Frellsen, J., and Hauberg, S.
Bounds all around: training energy-based models withbidirectional bounds. Advances in Neural Information
Processing Systems, 34:19808–19821, 2021.

Chen, L., Dai, S., Pu, Y., Zhou, E., Li, C., Su, Q., Chen,C., and Carin, L. Symmetric variational autoencoder
and connections to adversarial learning. In International

Grathwohl, W., Wang, K.-C., Jacobsen, J.-H., Duvenaud, D.,
Norouzi, M., and Swersky, K. Your classifier is secretly
an energy based model and you should treat it like one.
9

Improving Adversarial Energy-Based Model via Diffusion Process

Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J.,
and Aila, T. Analyzing and improving the image qualityof stylegan. In IEEE Conference on Computer Vision and
Pattern Recognition, pp. 8110–8119, 2020b.

International Conference on Learning Representations,
2020.
Grathwohl, W. S., Kelly, J. J., Hashemi, M., Norouzi, M.,
Swersky, K., and Duvenaud, D. No MCMC for me:
Amortized sampling for fast and stable training of energybased models. In International Conference on Learning
Representations, 2021.

Kingma, D., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. Advances in Neural Information
Processing Systems, 34:21696–21707, 2021.

Han, T., Nijkamp, E., Fang, X., Hill, M., Zhu, S.-C., andWu, Y. N. Divergence triangle for joint training of generator model, energy-based model, and inferential model.
In IEEE Conference on Computer Vision and Pattern
Recognition, pp. 8670–8679, 2019.

Kingma, D. P. and Dhariwal, P. Glow: Generative flow
with invertible 1x1 convolutions. Advances in Neural
Information Processing Systems, 31, 2018.
Kong, F., Duan, J., Sun, L., Cheng, H., Xu, R., Shen, H.,
Zhu, X., Shi, X., and Xu, K. Act: Adversarial consistencymodels. arXiv preprint arXiv:2311.14097, 2023.

Han, T., Nijkamp, E., Zhou, L., Pang, B., Zhu, S.-C., andWu, Y. N. Joint training of variational auto-encoder andlatent energy-based model. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7978–7987,
2020.

Krizhevsky, A., Hinton, G., et al. Learning multiple layersof features from tiny images. 2009.
Kumar, R., Ozair, S., Goyal, A., Courville, A., and Bengio,
Y. Maximum entropy generators for energy-based models.
arXiv preprint arXiv:1901.08508, 2019.

Hill, M., Nijkamp, E., Mitchell, J., Pang, B., and Zhu, S.C. Learning probabilistic models from generator latent
spaces with hat ebm. Advances in Neural Information
Processing Systems, 35:928–940, 2022.

Langley, P. Crafting papers on machine learning. In Langley,
P. (ed.), International Conference on Machine Learning,pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann.

Hinton, G. E. and Sejnowski, T. J. Optimal perceptual
inference. In IEEE Conference on Computer Vision and
Pattern Recognition, volume 448, 1983.

Lee, H., Jeong, J., Park, S., and Shin, J. Guiding energybased models via contrastive latent variables. International Conference on Learning Representations, 2023.

Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information
Processing Systems, volume 33, pp. 6840–6851, 2020.

Lin, C. H., Chang, C.-C., Chen, Y.-S., Juan, D.-C., Wei,
W., and Chen, H.-T. Coco-gan: Generation by parts via
conditional coordinating. In International Conference on
Computer Vision, pp. 4512–4521, 2019.

Hopfield, J. J. Neural networks and physical systems withemergent collective computational abilities. Proceedingsof the National Academy of Sciences, 79(8):2554–2558,
1982. ISSN 0027-8424.

Lin, Z., Khetan, A., Fanti, G., and Oh, S. Pacgan: The
power of two samples in generative adversarial networks.
IEEE Journal on Selected Areas in Information Theory,
pp. 324–335, May 2020.

Jeffreys, H. An invariant form for the prior probability inestimation problems. Proceedings of the Royal Society of
London. Series A. Mathematical and Physical Sciences,186(1007):453–461, 1946.

Liu, R., Gao, J., Zhang, J., Meng, D., and Lin, Z. Investigating bi-level optimization for learning and vision from a
unified perspective: A survey and beyond. IEEE Transaction on Pattern Analysis and Machine Intelligence, pp.
10045–10067, 2022.

Jolicoeur-Martineau, A., Piché-Taillefer, R., Combes, R.
T. d., and Mitliagkas, I. Adversarial score matching andimproved sampling for image generation. International
Conference on Learning Representations, 2021.
Kan, G., Lü, J., Wang, T., Zhang, B., Zhu, A., Huang,L., Guo, G., and Snoussi, H. Bi-level doubly variational
learning for energy-based latent variable models. In IEEE
Conference on Computer Vision and Pattern Recognition,
pp. 18460–18469, 2022.

Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based outof-distribution detection. Advances in Neural Information
Processing Systems, 33:21464–21475, 2020.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning
face attributes in the wild. In International Conference
on Computer Vision, pp. 3730–3738, 2015.

Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J.,
and Aila, T. Training generative adversarial networks
with limited data. Advances in Neural Information Processing Systems, 33:12104–12114, 2020a.

Lu, C., Zheng, K., Bao, F., Chen, J., Li, C., and Zhu, J.
Maximum likelihood training for score-based diffusion
10Improving Adversarial Energy-Based Model via Diffusion Process

odes by high order denoising score matching. In International Conference on Machine Learning, pp. 14429–
14460. PMLR, 2022a.

Song, Y. and Ermon, S. Improved techniques for training score-based generative models. Advances in Neural
Information Processing Systems, 33:12438–12448, 2020.

Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J.
Dpm-solver: A fast ode solver for diffusion probabilistic
model sampling in around 10 steps. Advances in Neural
Information Processing Systems, 35:5775–5787, 2022b.

Song, Y., Garg, S., Shi, J., and Ermon, S. Sliced scorematching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp.
574–584. PMLR, 2020.

Mescheder, L., Geiger, A., and Nowozin, S. Which training
methods for gans do actually converge? In International Conference on Machine Learning, pp. 3481–3490.
PMLR, 2018.

Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum
likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415–1428, 2021b.

Misra, N., Singh, H., and Demchuk, E. Estimation of theentropy of a multivariate normal distribution. Journal of
multivariate analysis, 92(2):324–342, 2005.

Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modelingthrough stochastic differential equations. International
Conference on Learning Representations, 2021c.

Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks.
International Conference on Learning Representations,2018.

Srivastava, A., Valkov, L., Russell, C., Gutmann, M., andSutton, C. Veegan: Reducing mode collapse in gansusing implicit variational learning. Advances in Neural
Information Processing Systems, Dec 2017.

Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B.,
and Ng, A. Y. Reading digits in natural images with
unsupervised feature learning. 2011.

Vahdat, A. and Kautz, J. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information
Processing Systems, 33:19667–19679, 2020.

Nijkamp, E., Hill, M., Zhu, S.-C., and Wu, Y. N. Learning
non-convergent non-persistent short-run mcmc toward
energy-based model. Advances in Neural Information
Processing Systems, 32, 2019.

Vincent, P. A connection between score matching anddenoising autoencoders. Neural Computation, pp.
1661–1674, Jul 2011.

Orlitsky, A. Information theory. In Meyers, R. A. (ed.),Encyclopedia of Physical Science and Technology (Third
Edition), pp. 751–769. Academic Press, New York, third
edition edition, 2003.

Xiao, Z., Kreis, K., Kautz, J., and Vahdat, A. Vaebm: A
symbiosis between variational autoencoders and energybased models. International Conference on Learning
Representations, 2021.

Pang, B., Han, T., Nijkamp, E., Zhu, S.-C., and Wu, Y. N.
Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33:
21994–22008, 2020.

Xiao, Z., Kreis, K., and Vahdat, A. Tackling the generative learning trilemma with denoising diffusion gans.
International Conference on Learning Representations,2022.

Smolensky, P. Information Processing in Dynamical Systems: Foundations of Harmony Theory, pp. 194–281.
MIT Press, Cambridge, MA, USA, 1986.

Xie, J., Lu, Y., Gao, R., Zhu, S.-C., and Wu, Y. N. Cooperative training of descriptor and generator networks.
IEEE Transaction on Pattern Analysis and Machine Intelligence, pp. 27–45, Jan 2020.

Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and
Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on
Machine Learning, pp. 2256–2265. PMLR, 2015.

Xie, J., Zheng, Z., and Li, P. Learning energy-based modelwith variational auto-encoder as amortized sampler. In
Association for the Advancement of Artificial Intelligence,
volume 35, pp. 10441–10451, 2021.

Song, J., Meng, C., and Ermon, S. Denoising diffusion
implicit models. International Conference on Learning
Representations, 2021a.

Xie, J., Zhu, Y., Li, J., and Li, P. A tale of two flows: Cooperative learning of langevin flow and normalizing flow
toward energy-based model. International Conference on
Learning Representations, 2022.

Song, Y. and Ermon, S. Generative modeling by estimating
gradients of the data distribution. Advances in Neural
Information Processing Systems, 32, 2019.
11

Improving Adversarial Energy-Based Model via Diffusion Process

Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., andXiao, J. Lsun: Construction of a large-scale image dataset
using deep learning with humans in the loop. arXiv
preprint arXiv:1506.03365, 2015.
Yu, P., Zhu, Y., Xie, S., Ma, X., Gao, R., Zhu, S.-C., and Wu,
Y. N. Learning energy-based prior model with diffusionamortized mcmc. Advances in Neural Information Processing Systems, 2023.
Zhai, S., Cheng, Y., Feris, R., and Zhang, Z. Generative
adversarial networks as variational training of energy
based models. arXiv preprint arXiv:1611.01799, 2016.
Zhang, J., Xie, J., Zheng, Z., and Barnes, N. Energy-based
generative cooperative saliency prediction. In Association
for the Advancement of Artificial Intelligence, volume 36,
pp. 3280–3290, 2022.
Zhang, L., Goldstein, M., and Ranganath, R. Understanding
failures in out-of-distribution detection with deep generative models. In International Conference on Machine
Learning, pp. 12427–12436. PMLR, 2021.
Zhao, J., Mathieu, M., and LeCun, Y. Energy-based generative adversarial network. International Conference on
Learning Representations, 2017.
Zhu, Y., Xie, J., Wu, Y., and Gao, R. Learning energybased models by cooperative diffusion recovery likelihood. International Conference on Learning Representations, 2024.

12

Improving Adversarial Energy-Based Model via Diffusion Process

A. Extended derivations
A.1. Derivations of Eq. (7)q (xt−1 | xt )dxt−1pθ (xt−1 | xt )= Eq(xt−1 |xt ) log q(xt−1 |xt ) − Eq(xt−1 |xt ) log pθ (xt−1 |xt )Z

DKL (q (xt−1 | xt ) ∥pθ (xt−1 | xt )) =q (xt−1 | xt ) log

Thus the objective Eq. (5) can be written as:XEq(xt−1 ,xt ) log q(xt−1 |xt ) − Eq(xt−1 ,xt ) log pθ (xt−1 |xt ) + C
L=−

(25)

(26)

t≥1

Since the first and the third term are independent of θ, they can be disregarded during optimization. Thus the objective isonly the second term. Plugging Eq. (6) into the second term, we can obtain:XL=
Eq(xt−1 ,xt ) Eθ (xt−1 , t − 1) + log q(xt |xt−1 ) − log Z̃θ,t (xt )
(27)
t≥1

A.2. Derivations of the optimum in Eq. (7)When Eq. (7) gets optimum, due to the equivalence of Eq. (5) and Eq. (7), we can easily obtain
DKL (q (xt−1 | xt ) ∥pθ (xt−1 | xt )) = 0 for each time step, i.e., q (xt−1 | xt ) = pθ (xt−1 | xt ), which means:Eθ (xt−1 , t − 1) + log q(xt |xt−1 ) − log Z̃θ,t (xt ) =log q(xt−1 ) + log q(xt |xt−1 ) − log q(xt )(28)Then we can obtainEθ (xt−1 , t − 1) − log q(xt−1 ) = log Z̃θ,t (xt ) − log q(xt )(29)

The left side of the Eq. (29) is just related to xt−1 , while the right side is just related to xt , as xt−1 can respond to all xt inthe whole space of X , then we can get Eθ (xt−1 , t − 1) − log q(xt−1 ) = C, therefore, for ∀t, we haveZ
exp (Eθ (xt , t))= q(xt ), Zθ,t = exp (Eθ (xt , t))dxt
(30)
Zθ,t
We already have pθ (xT ) = q(xT ) at t = T , when Eq. (5) gets optimum,pθ (x0:T ) = q (xT )TYpθ (xt−1 | xt )t=1

= q (xT )

T
Y

(31)

q (xt−1 | xt ) = q(x0:T )t=1
θ (xt ,t)).
Therefore, we have pθ (xt ) = q(xt ) = exp (EZ θ,tA.3. Derivations of Eq. (8) ∂ Eθ (xt−1 , t − 1) − log Z̃θ,t (xt ) ∂L X=
Eq(xt−1 ,xt )
∂θ
∂θ

(32)

Z
∂ log Z̃θ,t (xt )1 ∂ Z̃θ,t1
exp(Eθ (xt−1 , t − 1))q(xt |xt−1 )
=
=
dxt−1
∂θ
∂θ
Z̃θ,t ∂θZ̃θ,tZ
exp(Eθ (xt−1 , t − 1))q(xt |xt−1 ) ∂Eθ (xt−1 , t − 1)
=
dxt−1
∂θ
Z̃θ,t
Z
∂Eθ (xt−1 , t − 1)
= pθ (xt−1 | xt )
dxt−1
∂θ

(33)

t≥1

where

13

Improving Adversarial Energy-Based Model via Diffusion Process

Therefore, we can obtain:∂Eθ (xt−1 , t − 1)∂Eθ (xt−1 , t − 1)
∂L X=
Eq(xt−1 ,xt )− Eq(xt )pθ (xt−1 |xt )
∂θ
∂θ
∂θ

(34)

t≥1

A.4. Derivations of Eq. (13)

DKL (pϕ (xt−1 |xt )∥pθ (xt−1 |xt )) = Epϕ (xt−1 |xt ) log pϕ (xt−1 |xt ) − Epϕ (xt−1 |xt ) log pθ (xt−1 |xt )

(35)

Thus
X

Eq(xt ) DKL (pϕ (xt−1 |xt )∥pθ (xt−1 |xt ))

t≥1

=

X

Eq(xt )pϕ (xt−1 |xt ) log pϕ (xt−1 |xt ) − Eq(xt )pϕ (xt−1 |xt ) log pθ (xt−1 |xt )

(36)

t≥1

=

X

−H[pϕ (xt−1 |xt )] − Eq(xt )pϕ (xt−1 |xt ) log pθ (xt−1 |xt )t≥1

A.5. Derivations of the entropy
Since
pϕ (xt−1 , z|xt ) = p(z)q (xt−1 | xt , x0 = Gϕ (xt , z, t)) ,(37)

according to the property of entropy (Orlitsky, 2003)
H[x, z] = H[z] + H[x|z],(38)

H[pϕ (xt−1 , z|xt )] = H[z] + H[q(xt−1 |xt , Gϕ (xt , z, t))](39)

we can get

Since p(z) and q(xt−1 |xt , Gϕ (xt , z, t)) are both Gaussian distributions, according to (Misra et al., 2005), a m-dimensional
Gaussian distribution p(x) with mean µ and a d × d positive definite covariance matrix Σ, its entropy is
H [p(x)] =ln |Σ|
m[1 + ln(2π)] +
.
2
2

(40)

Therefore, the entropy of p(z) and q(xt−1 |xt , Gϕ (xt , z, t)) can be computed directly:
H [p(z)] =d(1 + log(2π))
2H[q(xt−1 |xt , Gϕ (xt , z, t))] =DD(1 + log(2π)) + log β˜t
2
2

(41)

(42)

A.6. Derivations of Eq. (16)H[pϕ (z|xt−1 , xt )] = −Eq(xt )pϕ (xt−1 ,z|xt ) log pϕ (z|xt−1 , xt )= −Eq(xt )pϕ (xt−1 ,z|xt ) log

pϕ (z|xt−1 , xt )− Eq(xt )pϕ (xt−1 ,z|xt ) log qψ (z|xt−1 , xt )qψ (z|xt−1 , xt )= −Eq(xt )pϕ (xt−1 |xt ) DKL (pϕ (z|xt−1 , xt )∥qψ (z|xt−1 , xt )) − Eq(xt )pϕ (xt−1 ,z|xt ) log qψ (z|xt−1 , xt )≤ −Eq(xt )pϕ (xt−1 ,z|xt ) log qψ (z|xt−1 , xt )
14

(43)

Improving Adversarial Energy-Based Model via Diffusion Process

A.7. Derivations of Eq. (18)Z
log pϕ (xt−1 |xt ) = logp(z)q (xt−1 | xt , x0 = Gϕ (xt , z, t)) dzZp(z)= log qψ (z|xt−1 , xt )q (xt−1 | xt , x0 = Gϕ (xt , z, t)) dzqψ (z|xt−1 , xt )


Z
p(z)
≥ qψ (z|xt−1 , xt ) log+ log q (xt−1 | xt , x0 = Gϕ (xt , z, t)) dzqψ (z|xt−1 , xt )(44)

= −DKL (qψ (z|xt−1 , xt )∥p(z)) + Eqψ (z|xt−1 ,xt ) log q(xt−1 |xt , Gϕ (xt , z, t))
A.8. Derivations of Eq. (19)X

Eq(xt ) DKL (pθ (xt−1 |xt )∥pϕ (xt−1 |xt ))

t≥1

=

X

(45)
Eq(xt )pθ (xt−1 |xt ) log pθ (xt−1 |xt ) − Eq(xt )pθ (xt−1 |xt ) log pϕ (xt−1 |xt )t≥1

The first term can be disregarded since it’s independent of ϕ, thus the variational upper bound of Eq. (12) can be written as
follows by applying Eq. (18) to the above equation:
XX−
Eq(xt )pθ (xt−1 |xt ) log pϕ (xt−1 |xt ) ≤Eq(xt )pθ (xt−1 |xt ) DKL (qψ (z|xt−1 , xt )∥p(z))
t≥1
t≥1
(46)
− Eq(xt )pθ (xt−1 |xt )qψ (z|xt−1 ,xt ) log q(xt−1 |xt , Gϕ (xt , z, t))

B. Experimental details
B.1. Diffusion process
We use the discretization of the continuous-time VP SDE (Song et al., 2021c) as our diffusion process, which is the same as
DDGAN. For all datasets, we set the number of time steps T to be 4. The variance function of VP SDE is given by:′′2

σ 2 (t′ (t)) = 1 − e−βmin t (t)−0.5(βmax −βmin )t (t) ,(47)

where t′ (t) ∈ [0, 1], t′ (t) is a function of time step t denoting the time of xt in VP SDE. t′ (t) can be any flexible timeschedule of a forward SDE. The constants βmax and βmin are chosen differently for different datasets. βt is designed asfollows:′′′2′2
1 − σ 2 (t′ (t))= 1 − e−βmin (t (t)−t (t−1))−0.5(βmax −βmin )(t (t)−t (t−1)) ,(48)
βt = 1 −2′
1 − σ (t (t − 1))
where t ∈ {0, 1, 2, . . . T }. Except for LSUN church dataset, we use equidistant steps in time for t′ (t) on other datasets,
i.e. t′ (t) = Tt . For LSUN
( church, we borrow the time schedule from DRL, which focuses on the first stage of thet⌊ T500−1 ⌋999

t ≤ T , we assume T ≤ 999 in most cases. We choose this time schedule because
1t=T
signal-to-noise ratio (SNR) (Kingma et al., 2021) is strictly monotonically decreasing in time, in high-dimensional space,data becomes quite sparse, thus training more on the first stage may be more important for generation.
′diffusion process: t (t) =B.2. Network structure
B.2.1. T OY DATA
For toy datasets, our encoder, generator, and energy function all have two embedding networks to encode the toy data,latent variable, and time t into two features. Then a decoder is used to decode the concatenation of these features into ourdesired output. Network structures are illustrated in Table 6. We use the Adam optimizer with a learning rate of 10−4 for
all the networks. The batch size is 200, and we train the model for 180k iterations. For FCE, we also choose Glow as the15Improving Adversarial Energy-Based Model via Diffusion Process
Table 6. Network structures for toy datasets. BN denotes batch normalization. SE denotes sinusoidal embedding. xout , tout denote theoutputs of two embedding networks with xt -related input and sinusoidal embedding of t as the inputs.

Energyxt
SE(t)FC 16 PReLU FC 16 PReLU
FC 32
FC 32
concat [xout , tout ]FC 300 PReLU
FC 300 PReLU
FC 1Generator
concat [xt ,z]SE(t)FC 16 PReLU BN FC 16 PReLU
FC 32
FC 32
concat [xout , tout ]FC 300 PReLU BN
FC 300 PReLU BN
FC 2

Encoderconcat [xt−1 , xt ]SE(t)FC 16 PReLU BN FC 16 PReLU
FC 32
FC 32
concat [xout , tout ]FC 300 PReLU BN
FC 300 PReLU BN
FC 16
mean: FC 2logvar: FC 2

Table 7. Network structures for MNIST dataset. permutate(·) denotes the permutation operation in Glow.

Energypermutate(xt ) SE(t)4 NICE layers
NICE scale layer
NICE layer
FC 1000 PReLU+ FC 1000 PReLU (SE(t))
FC 392×4

Generator
concat [xt ,z]SE(t)FC 1600 PReLU FC 50 PReLU
FC 1600
FC 100concat [xout , tout ]FC 3200 PReLU BN
FC 1600 PReLU BN
FC 784

Encoderconcat [xt−1 , xt ]SE(t)FC 3200 PReLU FC 50 PReLU
FC 1600
FC 100concat [xout , tout ]FC 1600 PReLU BN
FC 1600 PReLU BN
mean: FC 50
logvar: FC 50

generator, where we use 5 affine coupling layers amounting to 15 fully connected layers with 300 hidden units. For otheradversarial EBMs, both the generator and energy function have 3 fully connected layers each with 300 hidden units and
PReLU activations. Batch normalization is used in generators. The latent dimension is set to 2.
B.2.2. MNISTFor our NICE t network, similar to NICE model in VERA (Grathwohl et al., 2021), we also have 4 coupling layers andeach coupling layer has 5 hidden layers with 1000 units and PReLU nonlinearity. We integrate parameter t into NICE byadding a linear layer with the time embeddings as the inputs and PReLU nonlinearity to the output of each hidden layer. For
other WGAN-based methods, we use the same NICE model as in VERA. We deepen their generators, which allows fora fairer comparison of our method. Their generators all have a latent dimension of 100 and 5 hidden layers with 1600 or
3200 units each. We adopt PReLU nonlinearity and batch normalization in their generators as is common with generator
networks. Our generator and encoder employ the same structure as those for toy datasets except for more hidden units. The
latent dimension is set to 50. Network structures are shown in Table 7. All models were trained for 400 epochs with theAdam optimizer. We use the learning rate 10−4 for all our networks. For other WGAN-based methods, we use learning rate
3 × 10−6 for energy function and 3 × 10−4 for generator. We use a batch size of 128 for all models.
B.2.3. L ARGE - SCALE DATASETSFor large-scale datasets, we use sinusoidal positional embeddings for conditioning on integer time steps. For CIFAR-10
and CelebA datasets, our generator follows the modified Unet structure in DDGAN (Xiao et al., 2022), which provides
z-conditioning to the NCSN++ architecture (Song et al., 2021c). Our energy function mostly follows NCSN++ in Score
SDE, which also takes xt and t as its inputs. The only difference is we remove the last scale-by-sigma operation in NCSN++
and replace it with a negative ℓ2 norm between input xt and the output of the NCSN++ network as in Eq. (24). Our
encoder is designed as a CNN network that incorporates time embeddings and concatenates xt−1 and xt . The Encoder
network is shown in Table 8. For LSUN dataset, we simply removed FIR upsampling/downsampling and progressive
growing architecture from the z-conditioned NCSN++ and named it DDPM++ as our generator, following the naming
convention of Score SDE. For energy function, we adopt DDPM++ in Score SDE except for the same modifications usedin NCSN++. We find that DDPM++ helps generation on LSUN dataset. Our encoder borrows the network structure of16Improving Adversarial Energy-Based Model via Diffusion Process

the discriminator in DDGAN, but replaces the final dense layer with two dense layers that separately output the mean andvariance of qψ (z|xt−1 , xt ). See DDGAN for more details.
Table 8. Encoder structures for CIFAR-10 and CelebA datasets.

Encoder of CelebA
Encoder of CIFAR-10SE(t)FC 256 LeakyReLU (0.2)
concat [xt−1 , xt ]FC 256 LeakyReLU (0.2)
3×3 Conv2D 64, LeakyReLU (0.2)+ Dense 64 (tout )4×4 Conv2D 128, LeakyReLU (0.2)+Dense 128(tout )4×4 Conv2D 256, LeakyReLU (0.2)+Dense 256 (tout )4*4 Conv2D 512, LeakyReLU (0.2)+Dense 512 (tout )mean: 4×4 Conv2D 100 logvar: 4×4 Conv2D 100

SE(t)FC 256 LeakyReLU (0.2)
FC 256 LeakyReLU (0.2)
3×3 Conv2D 64, LeakyReLU (0.2)+ Dense 64 (tout )4×4 Conv2D 128, LeakyReLU (0.2)+Dense 128(tout )4×4 Conv2D 256, LeakyReLU (0.2)+Dense 256 (tout )4*4 Conv2D 512, LeakyReLU (0.2)+Dense 512 (tout )4×4 Conv2D 1024, LeakyReLU (0.2)+Dense 1024 (tout )mean: 4×4 Conv2D 100 logvar: 4×4 Conv2D 100
concat [xt−1 , xt ]B.3. Hyperparameter settings
We specify the hyperparameters used for our generators and training optimization on each dataset in Table 9 and Table 10.
Table 9. Hyper-parameters for our generator network# of ResNet blocks per scale
Initial # of channels
Channel multiplier for each scale
Scale of attention block
Latent Dimension# of latent mapping layers
Latent embedding dimension

CIFAR-102
128(1,2,2,2)
16
100
4
256

CelebA
2
64(1,1,2,2,4)
16
100
4
256

LSUN church
264(1,2,2,4,4)
16
100
4
256

B.4. Evaluation
When evaluating FID and IS scores, we use 50k generated samples for CIFAR-10, CelebA, and LSUN church datasets.
We use Pytorch 1.10.0 and CUDA 11.3 for training. Our training converges approximately 3 times faster than DDGAN,
resulting in a comparable overall training time despite our model’s slower per-epoch training speed compared to DDGAN.

C. Additional results
C.1. More comparisons on Toy dataset
Fig 6 further demonstrates our model’s strong capability on density estimation by comparing with score flow (Song et al.,2021b), DRL and DDGAN. Score flow and DRL specialize in density estimation, especially for large-scale datasets, butthey can’t perform well on toy datasets. We choose sub-VP SDE for the diffusion process and replace the score networkwith the gradient of an energy function, the network of which is the same as ours except for changing positional embeddings
to random Fourier feature embeddings for conditioning on continuous time steps. DRL requires multiplying its energyfunction by a scaling factor of 0.01, which introduces a temperature parameter between the trained energy function and thetarget one. DDGAN combines the diffusion process with GAN instead of EBM, as we expected, DDGAN is not suitable asa density estimator because GAN-based models are not supposed to provide a density estimate.
17

Improving Adversarial Energy-Based Model via Diffusion Process

Table 10. Hyper-parameters for our training optimization

Initial learning rate
βmin , βmax in Eq. (47)w, wmid in Eq. (21)Adam optimizer β1 , β2
EMA
Batch size
# of training epochs# of GPUs

real data

Score flow

MNIST
1e-4
(0.1, 10)(1, 1)(0.0, 0.9)
None
128
400
1

CIFAR-101e-4(0.1, 20)
(1, 1)(0.0, 0.9)
0.9999
64
1200
4

DRL
w/o scaleDRL
w/ scale 0.01

CelebA5e-5(0.1, 20)
(1, 1)(0.0, 0.9)
0.999
32
400
4

LSUN church
5e-5(0.1, 20)
(0.6, 0.2)
(0.0, 0.9)
0.999
12
400
4

DDGAN

DDAEBM

Figure 6. Density estimation on pinwheel dataset for different methods.

Swiss roll

DDGAN

DDAEBM

Swiss roll

DDAEBM (KL) DDAEBM (Jeffrey)

Figure 7. More generation results on Swiss roll and 25-Gaussians datasets

18

Improving Adversarial Energy-Based Model via Diffusion Process

We also show more generation results in Fig. 7 to compare our model with DDGAN and DDAEBM trained with KL
divergence. Although DDGAN can achieve impressive sample quality on the 25-Gaussians dataset, as demonstrated in
its initial paper, it gets limited quality on the Swiss roll dataset, which means DDGAN is not stable and robust enoughacross various datasets. Our DDAEBM can get much better generation performance on Swiss roll dataset, verifying ourmodel’s advantage in terms of robustness. We can also observe that without the reversed KL divergence, DDAEBM fails
to converge to each mode, leading to poor sample quality. Our DDAEBM trained with symmetric Jeffrey divergence can
improve generation a lot, resulting in satisfactory sample quality on each mode. This experiment demonstrates besides thetraining of energy function as shown in Table 5, training with Jeffrey divergence also has superiority to the generator’s
training by adding an extra reversed KL divergence.
Table 11. Mode coverage on StackedMNISTModel
VEEGAN (Srivastava et al., 2017)
PresGAN (Dieng et al., 2019)
StyleGAN2 (Karras et al., 2020b)
Adv.DSM (Jolicoeur-Martineau et al., 2021)
VAEBM (Xiao et al., 2021)
DDGAN (Xiao et al., 2022)
MEG (Kumar et al., 2019)
EBM-BB (Geng et al., 2021)
DDAEBM(ours)

Modes
762
1000
940
1000
1000
1000
1000
1000
1000

KL
2.173
0.115
0.424
1.49
0.087
0.071
0.042
0.045
0.033

C.2. Mode counting
Generative Adversarial Networks (GANs) are notorious for mode collapse which is a phenomenon in which the generator
function maps all samples to a small subset of the observation space. It’s well known that EBM can alleviate mode collapse
because of its entropy term. Therefore, we also evaluate the mode coverage of our model on the StackedMNIST dataset.
StackedMNIST is a synthetic dataset that contains images generated by randomly choosing three MNIST images andstacking them along the RGB channels. Hence the true total number of modes is 1,000, and they are counted using a
pretrained MNIST classifier. Similar to DDGAN, we also follow the setting of (Lin et al., 2020) and report the number ofcovered modes and the KL divergence from the categorical distribution over 1000 categories of generated samples to true
data in Table 11. It shows that our model covers all modes and achieves the lowest KL compared to GAN-based models andEBMs. This demonstrates our model is effective in mitigating mode collapse.
C.3. Additional results on LSUN church
We provide more results on 128 × 128 LSUN church dataset. Since there are few baselines on this resolution, we justcompare our model with DRL (Gao et al., 2021), which is superior to the majority of EBMs on various image datasets. Our
DDAEBM can get better visual quality compared to DRL as illustrated in Fig. 8 and Fig. 9. We calculate FID on 50,000
samples using a PyTorch implementation from DDGAN. Our DDAEBM has 13.80 FID. Original paper of DRL reports 9.76
FID with TensorFlow for calculation. But we used officially provided checkpoints and evaluated FID using our PyTorch
implementation, we only got 26.69 FID which is much worse than ours.

19Improving Adversarial Energy-Based Model via Diffusion Process

Figure 8. Generated samples using DDAEBM on LSUN church dataset. FID=13.80

20Improving Adversarial Energy-Based Model via Diffusion Process

Figure 9. Generated samples using DRL on LSUN church dataset. FID=26.69

21