Revisiting Contrastive Divergence for Density Estimation and Sample Generation
Authors: Azwar Abdulsalam
Joseph G. Makin
Venue: Accepted by TMLR ()

Abstract: Energy-based models (EBMs) have recently attracted renewed attention as models for complex distributions of data, like natural images. Improved image generation under the maximum-likelihood (MLE) objective has been achieved by combining very complex energy functions, in the form of deep neural networks, with Langevin dynamics for sampling from the model. However, Nijkamp and colleagues have recently shown that such EBMs become good generators without becoming good density estimators: an impractical number of Langevin steps is typically required to exit the burn-in of the Markov chain, so the training merely sculpts the energy landscape near the distribution used to initialize the chain.  Careful hyperparameter choices and the use of persistent chains can significantly shorten the required number of Langevin steps, but at the price that new samples can be generated only in the vicinity of the persistent chain and not from noise. Here we introduce a simple method to achieve both convergence of the Markov chain in a practical number of Langevin steps (L = 500) and the ability to generate diverse, high-quality samples from noise. Under the hypothesis that Hinton’s classic contrastive-divergence (CD) training does yield good density estimators, but simply lacks a mechanism for connecting the noise manifold to the learned data manifold, we combine CD with an MLE-like loss. We demonstrate that a simple ConvNet can be trained with this method to be good at generation as well as density estimation for CIFAR-10, Oxford Flowers, and a synthetic dataset in which the learned density can be verified visually.

Published in Transactions on Machine Learning Research (10/2025)
Revisiting Contrastive Divergence for Density Estimation and Sample Generation
Azwar Abdulsalam
orwar couor.abdussalam/ 1729gmmalll com
Elmore School of Electrical and Computer Engineering Purdue Universily
Joseph G. Makin
jgmakin Depurdue.comm edu
Elmore School of Electrical and Computer Engineering Purdue University
Reviewed on OpenReview: /t net/ forum? id= 15X4SZeqtg
Abstract
shergy-baseed models (EBMs) have recently attracted renewed attention as models for com- plex distributions of data, like natural images. Improved image generation under the maximum-ikclhhodd (MLE) objective has been achieved by combining very complex en- ergy functions, the form of deep neural nerksrks, with Langevin dynamics for sampling from the model. However, Nijkamp and colleagues have recently shown that such EBMs become good generators without becoming good density estimators: an impractical number of Langevin steps is typically required to exit the burn-in of the Markov chain, the train- ing merely sculpts the energy landscape near the distribution used to initialize the chain. Careful hyperparameter choices and the use of persistent chains can significantly shorten the required number Langevin steps, but at the price that new samples can be generated only in the vicinity of the persistent chain and not from noise. Here we introduce a simple method to achieve both convergence the Markov chain practical number of Langevin steps (L= = 500) and the ability to generate diverse, high-quality samples from noise. Under the hypothesis that Hinton's classic contrastive -ivergeeeee (CD) training does yield good density estimators, but simply lacks mechanism for connecting the noise manifold to the learned data manifold, we combine CD with an MLE-like loss. We demonstrate that sim- ple ConvNet can be trained with this method to be good al generation as well as density estimation for CIFAR-10, Oxford Flowers, and synthetic dataset in which the learned density can be verified visually.
1 Introduction
One long-standing approach to fitting probability distributions to observed data is to model the distribution only to the normalizer. Relaxing the constraint that the model integrate to unity allows for the use of any arbitrary function from the space of data to the real line, including deep neural networks. Exponentiating this "energy" function (or, by convention, its additive inverse) yields an unnormalized probability distribution:
fa:; 0) x -xp{-F((x,0))
(1)
(Circumflexes indicate models throughout.) The flexibility comes at a price, however. The classical approach to fitting distributions to data to minimize the relative entropy of these distributions, but for unnormalized models this requires taking expectations under the model distribution. In particular, the gradient of the
relative entropy is (Hinton 1999)
(scc Section A.5
TE E
(x,0) .0 00
DKL (xL Ex
X 0)
(2)
do
00

---

Published in Transactions on Machine Learning Research (10/2025)
The more complex the energy function, the less likely the integral implicit in the second term is to be analytically tractable. (The first expectation, under the data, X, will be approximated by sample average.) Low-dimensional observation spaces can be discretized and the integrals replaced with sums; but datasets of interest today, like images, occupy far too many dimensions for this approach to be feasible, since the required number of discrete bins grows exponentially in dimensionality.
The dominant approach to approximating Eq. 2 is therefore replace the intractable expectation with an empirical average under samples drawn from the model distribution, typically with some form of Markov- chain Monte Carlo (MCMC). In MCMC, an initial, randomly chosen sample is updated according to some transition operator that guides it into the "thick" part of the distribution of interest, in this case the model represented by Eq- [ More precisely, the samples generated by this procedure are guaranteed, after some number of iterations the "burn-in" to be distributed according to Eq. [ The most powerful procedures, like Hamiltonian Monte Carlo and Langevin dynamics (I.D) (Neal 2011), p exploit the gradient of the energy, 0E/0z, to accelcrate convergence. For example, the transition operator in LD encourages descent of the energy but also encourages exploration by injecting Gaussian noise:
aE Ox
R = X1 -x (x,0)++ + 22
(3)
where 2, ~ ~NO I). Contimoous-time Langevin dynamics is guaranteed to have Eq. as its stationary distribution, but Eq. 3 its discretization, is not (except in the limit as the step size approaches 0). A Mctropolis adjustment Mctropolis et al.) 1953) is therefore required in theory, although frequently neglected in practice. The discrete Langevin dynamics of Eq. 3 can also be used to generate samples from trained models at test time, as well as to evaluate the loss gradient, Eq. 2 during training.
Recently, energy-based models (EBMs) trained by descending the gradicnt of Eq. 2 have performed well at conditional and unconditional image generation, generating molecular graphs, image completion, composi-
tional generation, and out-of- generalization (Nijkamp et al. 2019 2020 Du & Mordatch 2019
Du et al.
2021 2023 Liu et al. 2021 Gao et al. 2021]. Recent work has demonstrated that EBMs trained
and tested in this or similar ways can generate realistic images in a variety of contexts (Zhao et al. 2021
2020 e.g. by using a ConvNet as the energy function. And yet, contrary to the orthodox theory just reviewed here (and underlying the papers just cited), it seems that none of these EBMs has learned the data distribution (Nijkamp al. 2019 2020). More precisely, the distribution generated by the sampling process in Eq. g does resemble the data distribution, even though Eq. does not. In an important pair of papers, Nijkamp and colleagues showed that for the parameter choices typically employed in EBM training, in particular for any number of Langevin steps L less than about 10,000 (L = 100 is typical), the Markov chain does not make it out of the burn-in stage. Consequently, the training sculpts this energy landscape instead of that of the model.
Xiao et al.) 2021a Xie et al. 2022 Pang et al.
Thus the training procedure with L RV 100, which Nijkamp et al. 2019 2020 call "non-convergent, short-run MCMC," " yields a useful generator. But the resulting EBM cannot be used to assign probabilities to samples, either relative or absolute, nor consequently to identify out-of-distribution data. Nijkamp and colleagues argue that the Langevin dynamics can be adjusted to yield samples from the model distribution, i.c. to get past the burn-in, with careful choice of the step size, e; and with "persistent initialization" of the Markov chain, Le. initializing X, at training iteration i with &, from training iteration i 1. (X, at training iteration = 1 is drawn from a noise distribution.) But this makes generation of new data dependent upon a bank of persistent samples. In addition to imposing a storage cost (or a computational cost, if the bank is to be regenerated from scratch), any finite sample bank also limits the diversity of new samples.
In sum, EBMs putatively trained by descent of the gradient. in Eq. 2 or, equivalently, to maximize the likelihood of the model parameters under the observed data either fail to assign the correct probabilities to data, cannot generate new samples from noise, or require impractically long training (I. >00,00).
Here we propose and verily a simple procedure that satisfies all three desiderata. Our point of departure is the observation that density estimation and generation make demands on the model that are to some extent orthogonal. Assigning the right probabilities to data requires getting the energy function correct on or near the data (e.g., image) manifold, whereas generating new samples from noise requires that the energy steadily decrease from the "noise manifold" (e-g., for high-dimensional Gaussian noise, a hypersphere)
2

---

Published in Transactions on Machine Learning Research (10/2025)
toward the data manifold. These two manifolds will not in general intersect, so these are distinct demands. Nevertheless, there exist, losses that plausibly enforce each of them. Short-run MCMC evidently sculpts a path from noise manifold to data manifold. And we hypothesize, and then show, that Hintonjs method of contrastive divergence (CD) is, despite recent work on its putative shortcomings (discussed below), in fact an excellent densily estimator. This has been overlooked because CD does not yield good generators; but on our account, this to be expected, since it does not shape the energy landscape far from the data manifold.
In this study, we show that combining these two training methods in the right proportions yields high-quality sample generation and -itto---iittibttiinn detection on datasets ranging in difficulty from simple synthetic data up to CIFAR-10 and Oxford Flowers, outperforming recently proposed alternatives for training EBMs. Generation requires only a few hundred (L R 500) steps of Langevin dynamics. But unlike in models trained with short-run MCMC alone, the Langevin sampler continues to produce high-quality images from the data distribution for as long it run (out to at least. 100,000 samples).
In what follows, we elaborate on the underlying theory (Section 2, describe our methods in detail (Section
8 present these results (Section 4 and discuss their implications (Section 5
2 Remedies for MLE
The problem of "non-convergent MCMC" (Nijkamp et al. 2020) has its roots in the slow rate of convergence of Langevin dynamics for non-conver functions. Under standard theory, the initial state of an ergodic Markov chain will be "forgotten" " after some number of transitions the burn-in period. However, the convergence rate is provably exponential in (a measure of) the non-convexity of the energy (Cheng al. 2020). Energies defined by neural nelworks are (unless otherwise constrained) non-convex, which is consistent with the exceedingly slow convergence rates observed in practice (Nijkamp et al. 2019 2020
Persistent Markov chains? A potential remedy is to allow the Markov chain, originally initialized random sample, to "persist" across weight updates. The idca is that the first weight update, although occurring before the Markov chain has converged to its stationary distribution, will not alter that stationary distribution so much as to render the samples up to that point useless. Intuitively, if the weight updates change the stationary distribution slowly enough, the Markov chain will eventually "catch up" and enter the typical set of the model distribution, greatly decreasing the required burn-in period. In practice, multiple Markov chains are maintained simultaneously in a "bank" of samples, in order to provide an unbiased estimate of the expected energy gradient,
TE 00
/OE | 00
Ex
{ 0)
(x,0)
(4)
The angle brackets indicate a Monte Carlo estimate (sample average) under the samples from the persistent Markov chains, (
However, as we have noted, this holds generation hostage to the sample bank, whereas one would prefer to generate new images at will from noise. To alleviate this, Du & Mordatch| (2019): Du et al. (2021) propose to reserve some fraction of the sample bank, typically about 5%, for noise samples. This does improve the quality of samples generated from noise, but with limited mode coverage. To increase sample diversity, more tricks are needed (Du et al. 2021).
Data-initialized LD for density estimation. Furthermore, in addition to the question of bias in Eq. under MCMC, there is the question of variance. Hinton (199) has argued that even after convergence, Monte Carlo estimates of the expected energy gradient will suffer from high variance because samples are being drawn from all over the model's (initially random) energy landscape. To remedy this, he proposed another initialization scheme (Hinton 1999), namely, to start the Markov chains at observed data. This certainly sensible late in training, when the model resembles the data, and such an initialization will shorten the burn- in period. But the procodure can also be justified as the approximate gradient of different loss function, a difference contrast of KT. divergences, which has the same minimum the original KT divergence (cf. Eq.
3

---

Published in Transactions on Machine Learning Research (10/2025)
2 [Hinton 2002
Accordingly, Hinton called data-initializing the Markov chain the method of "contrastive divergence" (CD). The pairing of each model sample with data sample reduces variance (see Section A.3) However, contrastive divergence can also be used in conjunction with persistent chains; indeed, this is (to our knowledge) where persistent chains originated (Tieleman 2008).
The approximation in CD comes from neglecting a term that is supposedly small in practice (Hinton 2002 and the recent work of Du et al. (2021) has argued that this term can and should be modeled. But there are several reasons to believe this is not the case. First, the training scheme used by Du et al. (2021) is not in fact (persistent) contrastive divergence, but rather the moise-initiallzed persistent Markov chains lately described, so it is not clear what can be concluded about CD from these experiments.
Second, recent work has shown that, in certain set Lings, parameter estimation with contrastive divergence is consistent, asymptotically converges at close to the optimal rate, and approaches the optimal precision (the Cramér-Rao bound) (Jiang al., 2018: Glaser al.) 2024). The results have so far been shown only for EBMs in which the parameters enter the energy linearly, but this nevertheless suggests there is nothing intrinsically problematic about CD density estimator.
Third and perhaps most compellingly, CD can be justified from other perspectives that do not invoke any approximation at all. For example, Omer & Michaeli (2021) show that CD can be seen as an instance of (conditional) noise-contrastive estimation (Gutmann & Hyvärinen, 2012). In this scheme, the Markov chain initialized at data is interpreted as a generator for a "negative-sample" distribution. With this generator fixed, the EBM trained so as to make the initializing data (positive samples) and the samples from L steps down the Markov chain (the negative samples) maximally distinguishable, which requires the EBM to become a better model for the data. Since the generator is in fact derived from the model, this in turn makes the discrimination problem more challenging, leading to more efficient training. Crucially, this interpretation retains the requirement that model and data samples be paired, which is critical for keeping the variance of the gradient small.
Woise-initialized LD for generation. Our hypothesis, then, is that CD is in fact a powerful density estimator, even for complex EBMs, but that this has been overlooked in the literature in part because CD-trained models do not generate high-quality images. That because, conceptually, density es- timation and (image) generation are to some extent orthogonal demands. Density estimation requires assigning the right prob- abilities to data, but what probabilities should be assigned to the noise vectors used to initialize sampling? Indeed, it is not clear that it makes sense to assign probabilities to data that are so far from the manifold to which the training data are confined; and if the data distribution has no support in this region, the gradient is zero. But if want to generate sam- ples, must shape the energy landscape between the "noise m and the data manifold.
Musise
Enotse-tnit
Cestn-tnit
Mania
Casto-tnit
Cominniii
On the other hand, non-convergent short-run MCMC models have shown how to shape the path from noise manifold to data manifold. This training procedure, although originally derived as an attempt
Figure 1: Illustration of the hypothesized roles of the two loss functions in shaping the energy landscape. The loss based data-initialized Markov chains (i.e., CD) ensures that the data manifold has lower energy than surrounding points. The loss based on noise-initialized Markov chains ensures that the energy decrease from the noise manifold loward the data man- ifold.
maximum-likellhod estimation, i.e. following the gradient in Eq. 2 is perhaps best understood as training Wasserstein GAN in which the generator and discriminator are different aspects of the same model (Xiao et al. 2021b). In par- ticular, the (negalive) energy acts as a discriminator wet data and samples produced by a generator, which in this case is LD operating on this same energy (Eq. 3). and initialized at noise. This encourages the EBM to assign lower energies to data samples than to noise (the discriminator); but also for the
4

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 1: Hybrid training of EBM
def OneStepUpdate( x(1) (N)
18
// batch of training data // current parameters of the EBM
19 def LangDyn (ag
(N)
0, e, T):
// in practice vectorize
L,
// number Langevin steps
20
for 1...,. do
// size of Langevin step
② - (())
21
T,
1/ temperature of Langevin dynamics
22
for 1 ,.... do
// size of descent step // relative weight of loss
23
~~ N0, 1)
X
+ à - 2 (a: 0) 2
):
10
Idata
AnghnergyGradient((r (1)
(N)
25
á (1) à
11
/ LangDyn x (1) x
(M) L,
26
return 2 (NN
12
Idita-inis
AvgEnergyGradient (11) (N)
27
def AvgEnergyGradieng (x (1)
(N) ())
13
#0
0
(N) NO0,1 I)
28
return En.. 30 (x"
3E
1
29
14
(0)
$1
(N)
LyngDyn (1)
!..., 0 L,, T
30
ArgEnergyGradient (0
..... (
def ParamUpdate (gdata: Idate inst, Inoise init, O 17): return
Guoise-init
32
16
ParamUpdate Idata-init, Inoise-init
(l(daata Gnoise init
A(gilaaa gdlat-innit)
17
return
LD to transform noise samples into data samples (the generator). Together these demands oblige the energy to decrease from the "noise manifold" toward the data manifold.
It is not a coincidence that (self- Jadversaaaa losses provide convenient interpretations for noise-initialized sampling as well as inttainitialized sampling. This is what licenses writing the so-called (Nijkamp et al. 2020 "compttatinnal loss" for each a difference in expected energies,
[( = [E(X,0) ((x) - (F((X,0) (8))
where ggen () is the distribution of data produced by the generator: Since for adversarial losses, the generator is fixed during improvement of the discriminator, derivatives with respect to the parameters 0 can pass inside the second sample average, even though the generator itself temp--armprrrlly fixed-also depends on these parameters.
A hybrid loss function. If density estimation and generation are indeed distinct demands, then achieve both in one model will require separately enforcing both. Accordingly, we propose a hybrid loss combining the two schemes just described (see Fig. 1 data-initialized version of Eq. 5 which enforces self-adversarial noise-contrastive loss for modeling the data distribution; and noise-initialized version of Eq. 5] which enforces self-adversarial WGAN loss for generating from noise:
(()) = ( ([((X)))
x()) - (((8,0) Inoise-inis {3)
+ + ( p(x) -
- (E(X, Iaata-imax
(6)
The relative weighting of the losses, X, is a hyperparameter and must be tunod. We emphasize the simplicity of the loss gradient: although the more exact interprelation is in terms of self-adversarial losses, it amounts combining the pseudo-MLE gradient (i.e., short-run MCMC) with the contrastive-diverguuee gradient. The training algorithm is summarized in Algorithm
3 Methods
Synthetic dataset. We illustrate the main ideas on a synthetic dataset designed replicate settings where the data lie on low-dimensional manifold embedded in higher-dimennsonal ambient. space. Specifically,
5

---

Published in Transactions on Machine Learning Research (10/2025)
the dataset consists of three concentric rings of radius 1, 2 and 3 with equal probability in a 2D ("X-Y") plane. To each point on the ring, we add isotropic Gaussian noise with standard deviation 0.1 in the X-Y-Z plane (i.e., in the original X-Y plane as well as one orthogonal dimension); and 0.01 along each of seven additional orthogonal dimensions, resulting in a dataset with an intrinsic 2D structure embedded in a 10D ambient space (see Section A.1 for full details).
We investigate five different training schemes, each based on different initializations of the Langevin dynamics, including the hybrid approach just proposed; these are detailed in the next section. For all experiments, we use four-layer ConvNet architecture. For persistent and persistent-trffessh initializations, we apply Langevin dynamics with a step size of c= 0.05 and a temperature (sce Section A.4) of T = 0.005. For data and hybrid initializations, we employ an adaptive step size: at each training iteration, the step size is set as = (.0005/||E(())) allowing for larger exploration steps early in training and gradual annealing as the training progresses. All models were trained for 10,000 iterations using single V100 GPU. We experimented with using adaptive step size for the persistent and noiso-init; however, we found best results with constant step size. To generale KDE plots, we initialize the MCMC chains from standard normal noise, using the same Langevin parameters employed during training.
Natural datasets. We also apply these five training schemes CIFAR-10 and the Oxford Flowers dataset. We employ an architecture similar to the one detailed by Nijkamp et al. (2019), with their ny parameter set to 128; this translates to 1024 output channels in the final convolutional layer. We found that incorporating attention layers significantly enhances the learning of convergent models in these more complex datasets, compared to merely stacking convolutional layers. A similar observation was reported by Nijkamp et al. (2019). We train these models with the same five schemes as for the synthetic dataset, in this case for 50,000 iterations and again each on a single V100. We apply the same MCMC parameters across all schemes, setting the step size c and temperature T 5c-5 (as for the synthetic datasot), and taking 500 steps of Langevin dynamics.
At test time, "short-run" " samples are generated by initializing Langevin dynamics at coaudard-normal noise samples and taking the same number of steps as during training, L = 500. The exception is the hybrid model, for which L = 1000 and
yields slightly better looking samples (it. does not improve any other models). We also continue these chains for 100,000 steps to investigate their long-run behavior. Data-initialized chains are run for L = 10, 000 steps, starting from samples from the test partition of the relevant data sets. All models are initialized at the same test-data samples to facilitate comparisons between them.
Common training and testing details. For the hybrid loss we set À = 25 for both datasets (see Eq. We tried a range of values for X and found 25 to give the best image generation.
6
In persistent training, we initialize a sample bank of size 60,000 with random noise. During each training iteration, we randomly select batch from this sample bank, run Markov chain starting from that batch, and replace the selected batch with the resulting samples. Following Du et al. (2021), in the "refresh" variant, we use this procedure only 95% of the time. In the remaining 5% of iterations, instead of starting from a batch in the sample bank, we "refresh" " the bank by initializing MC chains from pure noise and then replacing a randomly selected batch in the bank with the newly generated samples.
Out-of-distribution detection. We follow the standard procedure (Hendrycks & Gimpel 2017 to de- termine how well the model separates the probability densities of two datasels, in this case CIFAR-10 (on which EBM was trained) and SVHN. In particular, we first calculate the energies for all images in the test sets of CIFAR-10 and SVHN. We then calculate the fractions of true and false positives for every choice of an energy threshold putatively separating the two datascts. Finally, we report the area under the graph of true positives vs. true negatives, i.e. the area under the receiver-operator characteristic (acestic(CAAR (AUROC).
6

---

Published in Transactions on Machine Learning Research (10/2025)
columns: initia lixation during training
data
noise
persistent Somaie XI sine
persistent +refresh
hybrid Sarples aisass
Somplee
Sandie
Samales sione
short- run samples
data density
Poabd cxesity #1
Pucelesslyy 91
Poceldeniyy on
Veleldensly on
Made denily By on
model clensity
Figure 2: Data distribution (left) and learned densities (right) under different EBM training schemes. Top row shows
kenselddesstty estimate the distribution learned by the sampler (Bq. 3); bottom row shows true model densities
(Eq. D
4 Results
4.1 Synthetic (rings) dataset
Natural data, like images, believed to be confined manifold of much lower dimension than the ambient (pixel) space (Bengio et al. 2013). To explore such a setting without losing the ability to visualize that manifold, we begin with a 10-dimensional distribution with most of its mass concentrated in concentric rings in a two-dimensional subspace (see the leftmost. panel in Fig. 2 the description in Section B and a more detailed description in Section A.I The extra eight dimensions make the task finding the data manifold from noise more challenging. It not possible to visualize the entire energy landscape learned by the model, but we can view the space of most interest, the "X-Y" plane in which the probability mass is concentrated. We also examine kernel density estimates in this 2D space of samples generated by short-run MCMC. As Nijkamp al. 2019 2020 have pointed out, these distributions need not match.
We trained identical EBMs (see Section 3 using five distinct schemes (columns at right in Fig. 2 (1) initialization from data (i.e., contrastive divergence); (2) initialization from noise; (3) persistent initialization (à la Nijkamp and collcagues, i.c. with chains originally initialized at noise rather than data); (4) persistent initialization with 5% of chains randomly restarted at noise; and (5) our hybrid loss, Eq.
Training models with data-initialized Markov chains (Fig, 2 first column) accurately shapes the energy landscape and very near the data manifold, as seen in the learned model density (bottom row). On the other hand, the Markov chains run during training tend to remain close to the data manifold and fail to explore distant regions of the space. Since these regions of the energy landscape are not visited during training, they are not (appropriately) modified by gradient descent either. Consequently, at test time, Markov chains initialized at standard noise struggle to reach the data manifold (top row).
In contrast, models trained with noisc-initialized Markov chains (socond column) learn generate from the data distribution (lop row), but do not actually learn the correct energy (bottom row), noted by Nijkamp and colleagues. Allowing chains to persist across weight updates (third column) improves density estimation near the data (bottom row), but at the price of poorer generation from noise (top row). The latter problem, in turn, can be partially ameliorated by randomly restarting 5% of the persistent chains from noise at every step of training- but at the price of unbalancing coverage of the data (bottom row).
Finally, EBMs trained with our hybrid loss (Eq. 6 combine the merits the models trained with noise- and data-initialized MCMC: generating new samples from noise in virtue of the former, and assigning correct probabilities the data manifold in virtue of the latter. The important empirical finding is that, although

---

Published in Transactions on Machine Learning Research (10/2025)
columns: initialization during training
data
noise
persistent
persistent +refresh
hy hybrid
noise (short-rum)
rows: test-time initialization
nobse ==00,0000
data
Figure Data generation for networks trained CIFAR-10. Each column shows the results from model trained with particular initialization scheme. Each row shows samples generated at test time with Langevin dynamics (LD). First row: LD initialized noise and run for hundreds of steps ("short-run"). Second row: LD initialized noise and run for 100,000 steps ("long run"). Third row: LD initialized at data and run for 10,000 of step. For ease of comparison, all models were initialized at the same data samples.
the two component losses impose at least partially distinct constraints, they do not interfere with each other; and together they cover the relevant portions of the energy landscape.
4.2 Natural datasets
We now ask if these findings extend to complicated data distributions, wit, CIFAR-10 and the Oxford Flowers datasets. As with experiments the rings dataset, we trained otherwise identical models using the same five distinct approaches. It is obviously not possible to view the probability densities of these distributions, so we instead examine individual samples from the distributions of interest (Figs. 3 and [ and the energies assigned by models to selected samples (Fig. 5) and then quantify performance terms of FID score (Table [] and out-of-distribution detection (Table 2
4.2.1 Qualitative results
We begin by reproducing the results of Nijkamp et al. (2019 2020 namely, that EBMs trained with noise- initialized Markov chains do not produce convergent Langevin dynamics at test time. In such models, energy continues to decline long as the Markov chain is run, at least out to L = 10,000 steps (Fig. gb. Under the other four training schemes, Markov chains do converge (Fig. But do they converge to the data distribution?
We generate samples from the models three different ways: short-run MCMC from noise, verify whether the model can be used to generate images; long-run MCMC from noise, determine what (if anything) this sampler converges; and MCMC from data, to determine the quality of the models the data.
8

---

Published in Transactions on Machine Learning Research (10/2025)
columns: initialization during training
data
noise
persistent
sersisseen++refieen
hy brbriid
noise (short-rum)
rows: test-time initialization
nobee (L ===00000000 100.0000
data
Figure 4: Data generation for networks trained on the Oxford Flowers. Each column shows the results from model trained with a particular initialization scheme. Each row shows samples generated at test time with Langevin dynamics (LD). First row: LD initialized at noise and run for hundreds of steps ("short-run") Second row: LD initialized at noise and run for 100,000 sleps ("long run"). Third row: LD initialized at data and run for 10,000 of step. For case of comparison, models were initialized at the same data samples.
As with the rings data set, models trained on CIFAR-10 and Oxford Flowers purely with contrastive diver- gence (i.e., data-initialized Markov chains, first column of Figs. and 4 learn excellent models of the data: Chains initialized at test (unseen) images yield high-quality samples even after L = 10,000 LD steps. But the model cannot generate images from noise, because during training it never explored regions far from the data manifold. This is true in the long run (middle row) as well as the short run (top row); that is, Langevin dynamics does not ever converge on the data distribution. Indeed, this likely the reason that rastivve divergence as introduced by Hinton has not been used to train EBMs despite apparently learning very good models of the data. Even when the model has learned to assign correct energies on and near the data manifold, it need not have shaped the energy landscape in vast regions of the ambient space. For example, there is no reason why vector of standard normal noise should have higher or lower energy than nearby vector that is closer to the data manifold so the energy gradient in Eq. 3 is not helpful. And since the true data manifold has a much lower intrinsic dimension than the ambient space, most steps of LD are orthogonal to the manifold (so the noise term Eq. 3 does not. help either).
As has been observed in the literature, noise initialization (second column in Figs. 3ndd 4 yields a generator of high-quality images (top row), but the model has not learned the correct densities, since the sampler moves toward saturated images both under data-initialization (bottom row) and in the long run under noise initialization (middle row). Again we see that persistent initialization (third column) improves the model density near data (bottom row), but at the price of being unable to generate new samples from noise (top and middle rows). This also shows that the energy convergence observed in Fig. g even coupled with a good model near the data, cannot by itsclf be taken to imply convergence the data distribution. Indood, the energies of the (poor) images generated by long-run, noise-initialize chains are similar to the energies of images generated by the data-initialized chains (Fig. "persistent.)) "), so the resulting model is evidently
9

---

Published in Transactions on Machine Learning Research (10/2025)
data
noise
persistent
persistent--
hybrid
10
long-run
10
10
10
density
data
0.5
5
0
-0.5
0.2-0.1 O 0.1
-0.3 1..3-0.2-10 0.1
-0.2 -0.1
Energy
Energy
Energy
Energy
Energy
Figure Distributions of energies assigned by models after training on CIFAR-10. Blue: long-run samples; orange: test (held-out) CIFAR-10 samples. Plot titles specify the MCMC initialization scheme used during training.
not a good density estimator. (Fig. 5 shows the distribution of energies for CIFAR-10; for Oxford flowers,
see Fig. 10 in the appendix.)
During training, occasionally initializing the Markov chains from noise rather than from the sample bank (fourth column of Figs. 3 Jand see Section 3), as proposed by Du & Mordatch (2019), does markedly improve sampling from noise (first and second rows). However, the resulting samples are either unacceptably poor (Fig. or concentrated in one or two modes of the distribution (Fig. It is true that this method was not intended to be used with coise-initializatioon models. But furthermore, like its counterpart without sample-bank refreshing, this model assigns very similar energies to the (poor) samples generated from long-run, ioise-initialized chains as to (good) samples from data-initialized chains (Fig. 5t. So the model still not good density estimator.
we have already argued that this is a limitation of such
As with the synthetic dataset, if do desire to generate samples from noise, need to shape the energy in this region of pixel space well in the present case, essentially n-sphere with radius /Naims, since we use standard normal noise and the space is nigh-dimensional. In particular, the energy must decrease roughly monotonically from this noise manifold toward the data manifold. Combining the data-initialized and noise-initialized losses during training (Eq. 6 again yickds a model (final column of Figs. 8 and 4 that can generate high-quality samples from noise the short run (top row) and converges to such samples the long run (middle row), without compromising the energy landscape near the data (bottom row). The energies assigned to long-run samples are very close to, but slightly higher than, the energies assigned to test samples (Fig. 5 final panel). Although cannot rule out the existence of other points with even lower energy, this strongly suggests that models trained with the hybrid loss assign the highest probability to images resembling the training dataset. Thus, models trained with the hybrid loss make good generators as well as density estimators.
4.2.2 Quantitative results
To quantify the image generation from the previous section compute FID scores (Heusel et al. 2017 for CIFAR-10 for the five training schemes (Table
Table 1: FID scores for CIFAR-10.
D generating samples from each with short-run LD initialized at noise. Our hybrid-trained EBMs
test-time init
Ltest
training scheme
FID
data
292
(A = 25) achieve the lowest scores, lower even than models trained with noise initialization. Thus, so far from compromising image generation from noise, the addition of the inta-initializee loss term improves it. Of course, performance is not monotonically in- creasing in X, since in the limit, the hybrid schome becomes contraslive-divergenn
noise
42.3
noise
short
persistent
243
persistent-trefrsh
hybrid, X == 10
38.7
hybrid, 25
36.8
hybrid, À 50
44.5
training (i.e., pure data initialization, the first row of Table D By about À= 50, hybrid training becomes worse than pure noise initialization, presunably because the path from noise data is no longer well-learned.
noise
long
hybrid, À 25
35.3
persistent
40.1
sample bank
short
sersistent++refrees
35.1
10

---

Published in Transactions on Machine Learning Research (10/2025)
Note that if we continue to run the Langevin dynamics in the hybrid-trained EBMs for very long time (Trest = 100, 000), the FID score does not much change (middle row of Table | FID = 35.3). This is consistent with the observation that the short- and long-run samples from this model look qualitatively similar to each other (Fig. 3 last column, top and middle rows), and corroborates the conclusion that short- run LD convergences to the stationary distribution in EBMs trained with the hybrid loss, whereas does not under the noise-initialized loss and its variants.
For comparison, also include the performance of EBMs trained with persistent Markov chains and then tested by initializing LD at the accumulated bank of samples rather than noise (bottom row of Table I Even when initialized at a bank of high-quality samples, these models achieve at best comparable FID scores to the hybrid model initialized at noise.
The complementary question, how well the model has learned the underlying density, can be quantified indiroctly by asking how well the distribution of model energies for samples from CIFAR-10 can be distinguished from the same model's energies for samples from another dataset, in this case SVHN (Netzer et al. 2011). Following the literature (Hendrycks & Gimpel 2017), we report the area under the receiver-operator characteristic (AUROC), for the training schemes we have been considering so far, in Table 2 We include models trained with our hybrid loss for various values of the relative weighting X, including A= = 25, used throughout up until this point. Even at À 10, models trained with the hybrid loss outperform models trained with noise- and persistently initialized Markov chains. In fact, performance increases function of X, achieving maximal values for pure CD (all weight the second component of the loss in Eq. (). This suggests that including noise- initialized Markov chains in the loss function does degrade density estimation by some small amount-the price to be paid for genorating new samples from noise.
Out-of-distribution performance for models trained on CIFAR-10 and tested CIFAR-10 and SVHN. AUROC Area under ROC curve.
Table 2:
MCMC init
AUROC
data noise
0.95
0.57
persistent
0.68
persistent-trefieet
0.66
hybrid, À =
0.75
hybrid, x = 25
0.87
hybrid, À = -50
0.91
That OOD detection improves monotonically with the weight on the data-initialized loss (A), but FID does not, is intuitive and related to the fact that in general we require À >> 1, i.c. that the data-initialized loss be weighted much more highly than the noise-inilialized loss. The EBM needs to learn more detailed model of the data than of the noise distribution, or of the path from noise to data, since it does not matter much how the sampler arrives at the data manifold, as long as it does; whereas generating high-quality images or assigning the right probabilities to data requires many more bits. However, this is not the only reason that required for good performance: also decreases the variance of the gradient, which we show next.
X >>>>>
4.2.3 Evolution of the computational losses
Recall that the "computational losses" defined in Eqs. 5 and treat the generator I(X) as fixed (i.c., "stop-gradient" operation is applied to them). As result, these equations do not correspond to genuine loss functions. In particular, we do not expect them to decline roughly monotonically throughout training, as genuine loss functions do, because stepping downhill under the pretense of a fixed generator will in fact change the generator and may cause the total loss to increase. Instead, the functions in Eqs. 5 and D correspond loss grudients (with respect to a fictive coldness parameter; see Section A.3), and so we expect them to decline toward zero and then oscillate. Nevertheless, large fluctuations these updates indicate instability and are undesirable.
We invoked at the outset (Section 2 Hinton's 1999 argument that iata-initializing the sampler reduces variance in the estimate of the gradient. This explains why our method is successful while the superficially similar training based on persistent sample banks with random restarts at noise (Du & Mordatch 2019 Du et al.) 2021) is not: The opposing average energy gradients in Eq. 2 can be "paired" in CD both are "initialized" Th at the same data, with the second gradient evaluated farther down the Markov chains. In contrast, samples from a persistent bank will eventually be near the data manifold, but not necessarily near precisely those data showing up in the first. (data-averaged) energy gradient. That is, the variance of the
11

---

Published in Transactions on Machine Learning Research (10/2025)
std. dev. of comp. loss
hybrid, data-init hybrid, nolse-init
20
SNR of comp. loss
data-init (CD)
persistent refresh
0.5
200
400
600
800
1,000
0
200
400
600
800
1,000
training iteration
training iteration
A
B
Figure 6: The evolution of the compulational losses and their components (see legend) across training on CIFAR. (A) Standard deviations. (B) SNR (mean standard deviation). Curves have been smoothed for clarity.
computational loss is
Var[C] Var
( (0)
Σ E & (m) 0)
N
N
n=1
n=1
Var E(x (n) 00] Var ((8(m) (m) 00] (1))8))))))))))))) 2Cov E( .0))
N2
with N the batch size. When ( and (x) are independent (for all n); as in persistent or noise initializa- tions, the covariance term vanishes. Under data initialization, each sample (n) is generated from a Markov chain that starts (()) If the Markov chain is not run past the burn-in stage, then the samples and their corresponding energies (under the model) will be positively correlated, thus reducing the overall variance.
We can observe this empirically by plotting the standard deviation (across batch) of the computational loss at each training iteration (Fig. 6A As expected, the estimates of the computational loss have the highest variance under persistent initialization, and the lowest under data initialization (CD). Under our hybrid scheme, both components' deviations are much closer to that of CD. The noise-initialized loss is larger, as expected, but recall that during training, is effectively scaled down by a factor of A 25 relative to the data-initialized loss.
It might be argued that the relative, rather than absolute, variability of the computational loss is the critical quantity. But the same conclusion holds: persistent initialization also yields the loss estimates with the lowest signal-to-noise (mean-to-standardddeewe estimates of the loss are found in the noise-initialized component of the hybrid loss, but this is arguably misleading since the energy need not be estimated to high precision far from the data manifold.)
ratio (Fig. 6B especially the inset). (The highest SNR
4.2.4 Sample diversity
Another failure mode of generative models is the inability to generate novel samples. For example, EBMs trained with persistent Markov chains generate higher-quality examples at test-time when initialized at samples from a bank (acoumulated during training) rather than at noise samples (recall Table D But the price to be paid is that, least without more "tricks," generated sample will frequently be very similar to the sample from the bank which the LD was initialized.
We show this in Fig. 7 Fig. 7A shows sample bank accumulated during training of an EBM on the Oxford Flowers, and Fig. 7B shows the samples generated from the trained EBM after L = 500 steps of LD, starting form the sample bank. The resemblance between Fig. [A is very strong; we might even say that the "generation" process is mostly just copying samples out of the bank.
12

---

Published in Transactions on Machine Learning Research (10/2025)
In contrast, EBMs trained under the hybrid loss generate novel samples. Fig- 7C shows samples from such an EBM in the first col- umn. The next three columns show their (pix- elwise) nearest neighbors the training set. Evidently, the generated images do not closely resemble the training data, which shows that the EBM has learned the structure of (CIFAR- 10) images rather than merely memorizing ex- amples.
A
B
C
Figure 7: Sample diversity. (A) A bank of samples accumplated during training of an EBM with persistent Markov chains (with periodic refreshing). (B) Samples generated from this EBM after training, using short-run Langevin dynamics initialized at the samples in (A). (C) The first column shows images generated by an EBM trained under the hybrid loss; the remaining columns show their nearest neighbors in the training set.
4.2.5 Metropolis
adjustment and the stationary distribution
Theoretically, to ensure that Langevin sam- pling has the correct stationary distribution despite errors introduced by the discretization (see Section [] some samples must be rejected [Metropolis et al. 1953 In particular, sam- plcs can be "accepted" according to a formula that guarantees that the Markov chain satisfies detailed balance, and consequently has the correct stationary distribution. However, when using LD sample from EBMs, the acceptance rate typically drops very small values, and the sampling procedure slows to crawl (rejected samples do not count toward Langevin steps). So this "Metropolis adjustment" is frequently omitted, as we have until this point.
Nevertheless, to demonstrate more rigor- ously that the learned models do have the correct stationary distribution, we now apply the Metropolis adjustment. To keep the chain from "scizing," " we replace the fixed step size with dynamic step that decreases in proportion as the accep- tance ratio falls below the optimal ratio of 0.67 (Neal 2011). Here we use this procedure to generate samples, starting from noise, from an EBM trained with our hybrid loss on CIFAR. Fig. [A shows that this technique does indeed maintain acceptance ratio close to the optimum. Notice that continue to sample two orders of magnitude beyond the number of LD stops taken during training. Fig. SB shows samples from such chains after 50,000 L.D steps. Critically, the resulting images are once again very high quality, and resemble the corresponding images generated without the Metropolis adjustment (Fig. 3 final column), both short-run (top row) and long-run (middle row). This strongly suggests that the learned model has the correct stationary distribution i.e., that it is good densily estimator, well as being a good generator.
swatie
A
B
Figure 8: Image generation with the Metropolis adjustment. (A) The acceptance ratio as a function of training iterations. (B) Samples after 50,000 Langevin steps, Metropolis adjusted, from an EBM trained with the hybrid loss.
5
Discussion
The main goal of this paper has been to revisit contrastive divergence as a training method and to understand why, despite its early successes with models such as restricted Boltzmann machines, has become less popular for training modern energy-based models using neural networks. A erucial reason for this decline is
13

---

Published in Transactions on Machine Learning Research (10/2025)
the current standard for evaluating EBMs, which primarily emphasizes the quality of images generated from known noise distributions. Since CD initializes training near the data manifold, it is inherently disadvantaged in generating paths from noise distributions, making it unreasonable to expect CD-trained models to excel at noise-to-data image generation.
In contrast, training schemes based on noise-initialiee short-run MCMC, although able to produce visually appealing samples, have known shortcomings. Specifically, the images generated through such approaches do not represent samples from the stationary distribution of the trained model. Nijkamp et al. (2020) have previously discussed this issue and proposed remedies involving maintenance of a persistent sample bank (originally initialized from noise). However, our experimental findings on synthetic datasots- of low-dimensional data embedded in high-dimensional spaces demonstrate that persistent noise-initialized training often fails to recover the true underlying data distribution completely, whereas CD reliably suc- ceeds. Furthermore, although persistent methods frequently showcase samples from stored filter banks, these approaches face significant difficulties in directly generating new images starting purcly from noisc iost as have shown.
To address this challenge comprehensively, we introduced simple yet highly effective hybrid training ap- proach that combines the advantages of CD and noise-based methods. Specifically, we augment the tra- ditional CD loss with a scaled ML.E. loss. This combined strategy ensures robust density estimation while simultaneously enabling high-quality image generation from noise. The procedure is simple and intuitive and does not require persistent chains, data augmentation, resetting Markov chains, or other expedients. Besides the two Langevin hyperparameters (number and size of steps), only the relative importance of the two penalties must. be set beforehand, and we found single value (A 25) to work well for all experiments. Increasing the weight on the data-initialized chain beyond this point (i.e., making training more CD-like) improves -itt----istritttion detection (Table 2, but makes image generation worse (Table I.
To distinguish cleanly the effects of the various training schemes, including ours, we have restricted their applications to EBMs based on a simple neural network and trained on correspondingly simple data sets. We expect scaling it to more complicated or higher-resolution images will require more work. Nevertheless, we believe that this hybrid training scheme offers practical pathway to improve both density estimation and imago-generation capabilities in EBMs.
References
Y. Bengio, A. Courville, and P. Vincent. Representation Learning: A Review and New Perspectives. IEFF Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798 1828, August 2013. ISSN 0162- 8828, 2160-9292. doi: O0.109/TPAMM.2011
Xiang Cheng, Niladri S. Chatterji, Yasin Abbasi- Yadkori, Peter L. Bartlett, and Michael I. Jordan. Conver- gence Rates for Langevin Monte Carlo in the Nonconvex Setting. Technical report, 2020.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy-based models. In Advances in Newral Information Processing Systems, volume 32, 2019.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved Contrastive Divergence Training of Based Models. International Conference on Machine Learning, 139:2837 2848, 2021. ISSN 26403498.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC. In International Conference Machine Learning, pp.
1 1-22, 2023.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik Kingma. Learning Energy-Basod Models by Diffusion Recovery Likelihood. In International Conference on Learning Representations, number 2015, 2021. URL http::ttp/// /ttp::///axiv... wttg/abs/2022.01122
14

---

Published in Transactions on Machine Learning Research (10/2025)
Pierre Glaser, Kevin Han Huang, and Arthur Gretton. Near- Dear-Optimality of Contrastive Divergence Algo- rithms. In Advances in Newral Information Processing Systems 37 (NeurIPS 2024), December 2024.
Michael U. Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13:307-361, 2012. ISSN 15324435.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations, pp. 1-12, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale updale rule converge to local Nash equilibrium. Advances in Neural Informa- tion Processing Systems, 2017-Decem (Nips):6627-6633 2017. ISSN 10495258. doi: 10.18034/ 20844//jja..881 v8i1.9.
Geoffrey E. Hinton. Products of experts. In 9th International Conference on Artificial Neural Networks: ICANN '99, volume 2, pp. 6. London; Institution of Electrical Engineers; 1999, 1999. ISBN 85296 721 7. doi: 10. 10.1049/cp:19990) 19991075.
Geoffrey E. Hinton. Training Products of Experts by Minimizing Contrastive Divergence. Newral Computa- tion, 14:1771 1800, 2002.
Bai Jiang, Tung- Yu Wu, Yifan Jin, and Wing H. Wong. Convergence of Contrastive Divergence Algorithm in Exponential Family, February 2018.
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBBM Molecular Graph Generation with Energy-Based Models. In International Conference on Learning Representations, ICLR Workshop Track Proceedings, number 1, pp. 1-16, 2021. URL httt:///arxvv http://..orrgbb///22.0000 2atbbb/210..0044
Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Roseubluth, Augusta H. Teller, and Edward Teller. Decomposition approaches for the efficient solution of short-term schoduling problems. The Journal of Chemical Physics, (6)::1088 1093, 1953. ISSN 00981354. doi:
Radford M. Neal. MCMC using Hamiltonian Dynamics. In Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng (eds.), Handbook of Markoe Chain Monte Carlo, pp. 113-162. CRC Press, 2011. ISBN 978-1-4200-7942-53 doi: (0.1201/b10905
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading Dig- its in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, pp. 9, 2011.
Erik Nijkamp, Mitch Hill, Song Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run MCMC toward energy-based model. In Advonces in Newral Information Processing Systems, volume 32, 2019.
Erik Nijkamp, Mitch Hill, Tian Han, Song Chun Zhu, and Ying Nian Wu. On the anatomy of MCMC-basod maximum likelihood learning of energy- models. In AAAI 2020 34th AAAI Conference on Artificial Intelligence, pp. 5272 -2800, 2020. ISBN 9781577358350, doi: 10. 111.6/9//aaiv340000 v330..3440
Yair Omer and Tomer Michaeli. Contrastive Divergence Learning is a Time Reversal Adversarial Game. International Conference on Leurning Representations, pp. 16, 2021.
Bo Pang, Tian Han, Erik Nijkamp, Song Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In Advances Neural Information Processing Systems, pp. 1-15, 2020.
Tijmen Ticleman. Training Restricted Boltzmann Machines Using Approximations the Likelihood Gra- dient. Proceedings of the 25th International Conference on Machine Learning, pp. 1064-1071, 2008. doi: 10.1145/1 44/13990566 56.1390290.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models. In International Conference on Learning Represenlations, pp. 1 1-33, 2021a. URL httt:://arxiv.org/aaee http://bs/221..0055550
Zhisheng Xiao, Qing Yan, and Yali Amit. FBMs Trained with Maximum Likelihood Generator Models Trained with Self-adverscrial Loss, April 2021b.
Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. a Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Base Basod Model. In International Conference on Learning, pp. 1 1-23, 2022.
Yang Zhao, Jianwen Xie, and Ping Li. Learning Energy-Based Generalive Models Via Coarse- Coarse-To-Fine Expanding and Sampling. In International Conference on Learning Representations, pp. 1 1-29, 2021.
16

---

Published in Transactions on Machine Learning Research (10/2025)
A
Appendix
A.1 The concentric-rings datasets
The generative process for the concentric rings is depicted in the graphical model at right (we use 7 2x). The covariance matrix D diagonal with ii 0.1² for the first three diagonal entries and small for the remaining seven. The three radius lengths are r E (1.0, 2.0, 3.0). We visualize the distribution of (2), that is, the "thick" parts of this distribution, by explicitly marginalizing out Z and & (and ignoring the other dimensions of X). Anticipating that the distribution should be invariant to the angle (2), we define the length == V x Then we obtain
p(2) 1/3
p(0) = ((0
0.01
Z
X
C (6)
p
= /
r;
sin (5)
D
p(21, x22)
22 2)p(d)p(2) dø Σ
cOS(O)
Σ
N
oig I dø
sin(
f 2r, (2) cos o + 22 sin @) +
Σ
exp
do
20
Tgl
r
10
exp
20 2ig
big
rzl
f2
2r +
To
exp
big
20 big
r=l
>
To
N Nlrroz,
bbig).
big
with To rl BaR
a scaled version of the modified Bessel function of the first kind (to wit, i0e in scipy). Since bbig 0.1 is small compared to the inter-ring differences (1.0), r.l ze r². Thus, the distribution is approximately one-dimensional Gaussian mixture model along any radius, with the Bessel functions providing the mixing weights.
A.2
Energy evolution during Langevin dynamics
Fig.
9
shows the evolution of the energy across LD in models fully trained on CIFAR-10.
data
noise
persistent
persistent-
hybrid
20
20
20
15
-2,000
15
energy
4,000
10
10
5
6,000
0.5
0
0.5
0.5
0.5
0.5
step
-104
step
-104
step
.104
step
.104
step
.104
Figure 9: Energy vs. Langevin steps in models trained with various schemes, identified in the column labels, CIFAR-10.
17

---

Published in Transactions on Machine Learning Research (10/2025)
A.3
The computational loss
Nijkamp el al. (2020) identily the difference in average energies,
d == (((X 0) - (E((X
(7)
as a critical quantity during EBM training. (We leave open for now under precisely what distribution the second expectation is taken, but assume that it depends on the parameters 0.) They make, in particular, the empirical observation that d and the fluctuations in the average magnitude of the force 0E/Ox correlate highly and positively at the same training batch, but negatively lags of two three batches. Thus stable training requires d to balance roughly between positive and negative values across gradient updates.
The computational loss is a gradient. It is tempting to interpret Eq. 7 as the loss function itself because its gradient superficially resembles 2 But the averaging brackets in the second term of Eq. 7 depend on the parameters, so the gradient cannot pass inside. Nijkamp and colleagues point out this similarity to the loss but rightly resist the temptation to assimilate the two. But what then is the relationship.
Let re-express the energy function (without loss of generality) the product of fixed-scale energy É E [-11, 1J, and "coldness" parameter 3 > 0: E 3E. This parameter could be explicit, or it could be merely a fictitious stand-in for the net effect of all the parameters on the energy magnitude. In either case, the loss gradient, Eq. 2 with respect to this parameter is
d DKL p( 0)} = Ex
T 05
TE
X 0)
- Ex
(X
0)
dB
08
1
d 8
[x 0)]
= E X 8)
8
Thus, the difference in expected energies is proportional to the gradient of the loss with respect to the energy magnitude.
This explains the relationships observed in NNikkamp ct al. 2020 If Eq. 7 were a proxy for the loss, gradient
descent (with sufficiently small gradient steps) would always drive it downwards. Instead, d a loss gradiend, so during successful training runs oscillates around zero. Negative values of imply that the coldness B needs to be adjusted upward by gradient descent. Equivalently, negative implies that the energy E BÉ, and its gradient with respect to the data 0E/Oz, the force, are too small. Positive values of d imply, by the same logic, that the force is too large. This explains the correlation between the force magnitude and d. Likewise, gradient descent will increase negative d and decrease positive d, equivalently increase forces that are too small and decrease forces that are too large. This explains the negative cross-correlation at short lags.
The difference expected energies therefore corresponds to the change made the overall magnitude of the energy that point in training. Large fluctuations in the magnitude updates indicate instability and are undesirable.
A.4 The temperature parameter
In the limit as the step size e approaches zero, the discretized Langevin dynamics
OE
X X,
X 0) + V2e2i
(8)
Zx
has as its stationary distribution Eq. which repeat here for convenience:
f(x; 0)
0))
In practice when running Langevin dynamics, is useful to be able scale the gradient and noise terms independently: the latter must be small for stability, but the former must not be too small lest. numerical
18

---

Published in Transactions on Machine Learning Research (10/2025)
data
noise
persistent
persistent-trefessh
hybrid
15
0.4
long-run
15
0.6
10
density
data
10
10
0.4
0.2
0.2
-0.5
15
10
-5
-0.1
0.1
0.2
2
0
-..40.22 0
0.2
energy
energy
energy
energy
energy
Figure 10: Distributions of energies assigned by models after training on the Oxford Flowers. Blue: long-run samples; orange: test. (held-out) the Oxford Flowers samples. Plot titles specify the MCMC inirialization scheme used during training.
precision of the gradient be lost. This particularly true early in training, when the energy landscape produced by the neural network is very flat.
Including the parameter T in Eq. allows for independent scaling of the gradient and noise. Of course, this does change the stalionary distribution; in particular, the Langevin update can be re-written
OE dx
4+1 X
( 0) + 2
c
1= T
& 4+
& T dr
8 0) + /2
(9)
and the stationary distribution bocomes
f(:; 0) x
0)/T},
as can be seen by comparing Eqs. 8 and with Eq.
9 (and grouping 1/T with the energy).
In the literature, T frequently omitted from the stationary distribution order to reduce clutter, as we have in Eq. [ and in the loss and its gradients, Eqs.
and 6 Indeed, because normalized probabilities are rarely computable (or computed), T is in some sense irrelevant to the reported results-it it merely sets the "units" of the energy. However, in Fig. 2 we display the model densities themselves, and therefore have taken T into account.
A.5 The relative-entropy gradient for EBMs
Although it is well known, for completeness we include the derivation of the gradient of the relative entropy for energy-based models, here keeping the temperature parameter explicit:
p ((X)
DKL (x do
Ex
log
d0
p p(::0) O)
d
1 expl EE () dē
Ex
log
d0
exp ((x, 0)/T}
(10)
d
S1 exp( FF 0) d.
1 OE (X,0) 0)
+ Ex
La e - -E(@, [T] dı
T 00
1
0E
AE
Ex
X 0)
E
0)
T
00
00
Ignoring the temperature parameter (or setting it to unity) produces Eq.
2 of the main text.
19


Reviews:
[{"review_id": "fOfTwWCfV2", "reviewer": "Reviewer_grgR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: I am not familiar with image generation, but I learned much from this work. I think the paper is well-structured, mostly self-contained, and provides useful, sufficient information to the audience. While the idea behind their hybrid loss and training scheme sounds natural (combining two to achieve more), the results are solid on both synthetic and natural datasets. \n\nWeaknesses: I do not see this paper has significant weaknesses. But I do have some questions, mostly out of curiosity for this work. \n\n1. How is the complexity of the training scheme proposed by the author(s), compared to the other training schemes (noise, data, persistent, etc.). It would be helpful to either discuss it or illustrate it in the paper. \n\n2. Why the $\\lambda = 25$ gives the best performance? How does the best value of $\\lambda$ depend on the training/testing set? It would be good to discuss it, even if it is just for the experiments done in this paper. \n\n3. I wonder if there is any theoretical results regarding the better behavior of the training scheme proposed by the author(s).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contributions of this paper are: 1. The author(s) compared several different training schemes of image generation, discussing their strengths and weaknesses and performed experiments to validate them; 2. The author(s) proposes a method which combines both data- and noise-initialized schemes; 3. The author(s) show by experiments that this new method performs well on both synthetic and natural datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some details in this paper are not made clear enough. See below. \n\n1. The variable $T$ in eq (3) is not defined. \n2. There are many things like “$L \\sim$ 100/500/etc.” The author(s) should make clear how the “$\\sim$” is defined. Since we are dealing with finite numbers here, for example $L \\sim 100$ should not mean \"$100 c_1 \\le L \\le 100 c_2$ for some constants $c_1$, $c_2$. \n3. In the experiment for synthetic dataset, the rings are in 2D plane, the meaning of “xyz-plane”, especially the z-direction, in the first paragraph of “Section 3, Synthetic dataset” should be made clear. \n4. In Figures 2,3,4, I suggest changing the title “Initialization during training” to something like “Results of different training schemes”. \n5. There are many “$\\hat{X}_{subscript}$” in the paper. It would be helpful to mention their meanings. For example, do the $\\hat{X}_l$ in eq (3) the same as the samples $\\hat{X}_n$ in Section 4.2.4?", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jMVu7WJ23v", "reviewer": "Reviewer_rfCw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Authors tackle a hard and interesting problem\n- I really like the emphasis of trying to bring an algorithm that converges with a practical number of sampling steps\n\nWeaknesses:\n- Clarity\n- Empirical validation of the proposed loss\n\nI detailed the 2 weaknesses below.\n\nDisclaimer: I am not an expert in energy-based models\n\n**Clarity**: I struggled to understand section 1 after equation 3.\n- What does it mean to be a 'good generators without becoming good density estimators'?. In particular, experiments report the FID, which evaluates the generation quality.\n- Can you clarify the following statement?  'And yet, contrary to the orthodox theory just reviewed here (and underlying the papers just cited), it seems that none of these EBMs has learned the data distribution (Nijkamp et al., 2019; 2020). More precisely, the distribution generated by the sampling process in Eq. 3 does resemble the data distribution, even though Eq. 1 does not'\n    - What is exactly the setting here?\n        - Do you have a 'reward' function you try to sample from?\n        - You have a finite dataset and want to estimate the underlying probability distribution? With the specific shape of an energy function?\n- ' In an important pair of papers, Nijkamp and colleagues showed that for the parameter choices typically employed in EBM training, in particular for any number of Langevin steps L less than ∼10,000 (L = 100 is typical), the Markov chain does not make it out of the burn-in stage. Consequently, the training\nsculpts this energy landscape instead of that of the model'\nCan you give more details on this? Especially, can you explain more the last sentence? I do not understand the causality link between the 2 sentences.\n- ' “persistent initialization” of the Markov chain, i.e. initializing X̂ 1 at training iteration i with X̂ L from training iteration i − 1. (X̂ 1 at training\niteration i = 1 is drawn from a noise distribution.)  data dependent upon' I did not understand what is the 'persistence initialization'\n- 'training procedure with L ∼ 100' I am confused here. From what I understand, L is the number of steps in the Langevin steps at inference. Why do you keep referring to L for the training procedure? Do authors do simulation-based training? Maybe if would help to write done the algorithm used for training and sampling.\n- 'Short-run MCMC evidently sculpts a path from noise manifold to data manifold' Why is this 'evident'?\n- ´The problem of “non-convergent MCMC” (Nijkamp et al., 2020) has its roots in the slow rate of convergence of Langevin dynamics for non-convex functions.´\nIs this also something you observed in practice? That can be seen on the figures in the experiments.\n- I am not sure if I understood the take-away of Figure 1.\n- Overall the paper heavily relies on Nijkamp 2020, maybe it could be worth to provide extensive reminder at the beginning of the manuscript.\n\nNijkamp, E., Hill, M., Han, T., Zhu, S.C. and Wu, Y.N., 2020, April. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 5272-5280).\n\n**Experiments**:\n- 'In contrast, models trained with noise-initialized Markov chains (second column) learn to generate from the data distribution (top row), but do not actually learn the correct energy (bottom row), as noted by Nijkamp and colleagues.'\n- Can you give more details on the plots in Figure 5: what d they represent, and what is the takeaway from this Figure?\n- Qualitative experiments are amazing to grab intuition (Figure 3 and 4 are very helpful). The only quantitative experiment I saw Figure 5 and Table 1 (+ the ablation study in Table 2.), that provides results for CIFAR-10.\nWould it be possible to provide a wider range of experiments to support the claim the proposed loss is better/interesting.\n\nTypos:\n- page 11, durng >> during", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors tackles the problem of sampling from energy-based models.\nAuthors propose to combine contrastive-divergence and maximum likelihood.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify the introduction: the problem setting and the challenges\n- Provide the algorithm for training and inference\n- Provide additional experiments (not necessarily SOTA) to back up the proposed loss", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BG8NbTnnwB", "reviewer": "Reviewer_vVbU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. An excellent championing of contrastive divergence back to EBM. The authors defend the use of the hybrid approach, and explain why CD has been overlooked by the EBM literature. The hybrid approach is quite effective since the lambda=25 is in general transferrable, e.g. turning is not too hard.\n\n2. Addition of a regularization term based on contrastive divergence is quite effective. It allows for density estimation (coming from the regularization term and good generation of samples from noise). I do appreciate the deep dives, especially the SNR computation.\n\n3. The presentation of this work is clear and effective. I believe the audience will appreciate the insights.\n\nWeaknesses:\n\n1. I have a small question about the OOD detection, please see it below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address a challenge with Energy Based Models that it is hard to have 1) good density estimation; 2) generating new samples from noise, and 3) efficient training (small number of Langevin steps) at the same time.\n\nThe first approach to remedy is by using persistent Markov chains where the gradient of the energy is estimated by an average of samples from persistent Markov chains.\n\nThe issue with the first approach is not allowing to sample freely from noise. To address that issue the authors revisit contrastive divergence methods.\n\nThe result is a loss function which has two components, which can be interpreted as the (short-run MCMC) + lambda * (contrastive-divergence gradient). The first term has initialized the Markov Chains at noise, and the second term has initialized the Markov chains at the data manifold.\n\nThe authors demonstrate that this hybrid loss works the best in both estimating the density and sampling from noise freely in a synthetic dataset of concentric circles.\n\nAn interesting empirical finding of this work is that the two loss terms do not conflict with each other, which is inline with the visualized hypothesis of the authors in Figure 1.\n\nThe authors demonstrate the usefulness of the hybrid approach on the Oxford flowers and CIFAR-10 datasets, as well as testing out of distribution energies.\n\nThe authors dig deeper to demonstrate that he hybrid approach yields the best SNR with training iterations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* How exactly do you compute the AUROC in Table 2? E.g. how do you measure exactly the distinguishing between the the energies from the in-distribution and the out-of-distribution datasets? Do you train a separate classifier? Can you clarify this in the main text?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)