Energy Discrepancies: A Score-Independent Loss for Energy-Based Models
Authors: Tobias Schröder
Zijing Ou
Jen Ning Lim
Yingzhen Li
Sebastian Josef Vollmer
Andrew Duncan
Venue: NeurIPS 2023 poster (2023)
Keywords: Energy-based models
statistical discrepancy
latent-variable model
density estimation
TL;DR: This paper proposes a new loss-function for energy-based modelling which can be evaluated without computing scores. We describe strong theoretical properties of the loss and demonstrate its efficacy for learning positive data distributions.

Abstract: Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that energy discrepancy approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum energy discrepancy estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.

Energy Discrepancies: A Score-Independent Loss for Energy-Based Models
Tobias Schröder! Zijing Ou Yingzhen Li"
Jen Ning Lim?, Andrew B. Duncan
Sebastian J. Vollmer'
Imperial College London
University of Warwick "The Alan Turing Institute
DDFI and RPTU Kaiserslautern
{t. schroeder21, 2. ou22, yingzhen. 1i, a. duncan) iimeerial .u.uuk jen-ning www.warwickk.a... uk sebastian vollmer@dfki.
Abstract
Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely the computation of scores expensive Markov chain Monte Carlo. We show that energy discrepancy approaches the explicit score matching and negative log- eegllikelhhood loss under different limits, effectively interpolating between both. Consequently, minimum energy discrepancy estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns dindemniiional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high- nigh-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy- based model as a prior of a variational decoder model.
Introduction
Energy-Bas Based Models (EBMs) are class of parametric unnormalised probabilistic models of the general form Pebm exp(-U') originally inspired by statistical physics. EBMs can be flexibly modelled through wide range of neural network functions which, in principle, permit the modelling of any positive probability density. Through sampling and inference on the learned energy function, the FBM can then be used as generative model or in numerous other downstream tasks such as improving robustness in classification or anomaly detection (Grathwohl et al., 2020), simulation-basee inference (Glaser et al., 2022), or learning neural set functions (Ou et al., 2022).
Despite their flexibility, EBMs are limited in machine learning applications by the difficulty of their training. The normalisation constant of EBMs, also known the partition function, is typically intractable making standard techniques such as Maximum Likelihood Estimation (MLE) infeasible. For this reason. EBMs are commonly trained with an approximate maximum likelihood method called Contrastive Divergence (CD) (Hinton, 2002) which approximates the gradient of the log -ggliiili using short runs of Markov chain Monte Carlo (MCMC) method. However, contrastive divergence with short run MCMC leads to malformed estimators of the energy function (Nijkamp et al., 2020a), even for relatively simple restricted Boltzmann machinns Chareira--Prpinnee & Hinton, 2005). This can, in part, be attributed to the fact that contrastive divergence is not the gradient of any fixed
"Correspondence Tobias Schröder, www.ccoreder221emmriiaalll ac.uk
I CCode https://githuu com/ wtt.e/////////eeeeeeeeee citt///nnnrgyydccepaancc
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

objective function (Sutskever & Tieleman, 2010), which severely limits the theoretical understanding of CD and motivated various adjustments of the algorithm (Du et al., 2021; Yair Michaeli, 2021).
Score-based methods such as Score Matching (SM) (Hyvärinen & Dayan, 2005; Vincent, 2011; Song et al., 2020) and Kernel Stein Discrepancy (KSD) estimation (Liu et al., 2016; Chwialkowski et al., 2016; Gorham & Mackey, 2017; Barp et al., 2019) are a family of competing approaches which offer tractable loss functions and are, by construction, independent of the normalising constant of the distribution. However, such methods suffer from nearsightedness as they fail to resolve global features the distribution without vast amounts of data. In particular, both SM and KSD estimators are unable to capture the mixture weights of two well-separated Gaussians (Song & Ermon, 2019; Zhang et al., 2022; Liu et al.. 2023).
We propose a new loss functional for energy-based models called Energy Discrepancy (ED) which compares the data distribution and the energy-basee model via two contrasting energy contributions. By definition, energy discrepancy only depends on the energy function and is independent of the scores or MCMC samples from the energy-based model. In our theoretical section, we show that this leads to loss functional that can be defined on general measure spaces without Euclidean structure and demonstrate its close connection to score matching and maximum likelihood estimation in the Euclidean case. In our practical section, we focus on a simple implementation of energy discrepancy on Euclidean space which requires less evaluations of the energy-basee -ased model than the parameter update of contrastive divergence or score- wore-matching. We demonstrate that the Euclidean energy-discrepane alleviates the problem of nearsightedness of score matching and approximates maximum-likelime estimation with better theoretical guarantees than contrastive divergence.
On high-dimensional image data, energy-based models face the additional challenge that under the manifold hypothesis (Bengio et al., 2013), the data distribution is not positive probability density and does, strictly speaking, not permit a representation of the form of an energy-based model. Energy discrepancy is particularly sensitive to singular data supports and requires the transformation of the data distribution to a positive density. We approach this problem by training latent energy-based priors (Pang et al., 2020) which employ a liwer-dimensional latent representation in which the data distribution is positive.
Our contributions are the following: 1) We present energy discrepancy, a new estimation loss for the training of energy-based models that can be computed without MCMC or spatial gradients of the energy function. 2) We show that, as a loss function, ED interpolates between the losses of score matching and maximum-likellhon
estimation and overcomes the nearsightedness of score- based
methods. 3) We introduce a novel variance reduction trick called w-stabilisation that drastically reduces the computational cost of approximating energy discrepancy stably.
2 Training of Energy-Based Models
In the following, let (x) be an unknown data distribution which we are trying to estimate from independently distributed data } ~ Pdata- Energy- models are parametric distributions of the form
(x(x) x exp(-Eo(x))
for which we want to find the scalar energy function Fg such that pa R Pdata- Typically, energy-based models are trained with contrastive divergence which estimates the gradient of the log- likeliidlhoo
Va lo (x) Eps (y) [o)) (x)
(1)
using Markov chain Monte Carlo methods to approximate the expectation for po. For computational efficiency, the Markov chain is only run for a small number of steps. As a result, contrastive divergence does not learn the maximum-likellhood estimator and can produce malformed estimates of the energy function (Nijkamp et al., 2020a).
Alternatively, the discrepancy between data distribution and energy-based model can be measured by comparing their score functions Vx log Pdata and Vx log pe which, by definition, are independent of the normalising constant. The comparison of the scores is achieved with the Fisher divergence. After applying an antegration-yy-pute and discarding constants with respect to 0, this leads to the score-matching loss (Hyvärinen & Dayan, 2005)
[- Foxx) 1 Eo)
SM(pdata, Fω) = Bxsau(x))
(2)
2

---

As this only requires the computation of expectations with respect to Plata, the requirement that the data distribution attains a density can be relaxed, yielding a loss function for po which can be readily approximated. Score-based methods are nearsighted as the score function only contributes local information to the loss. mixture of well-separated distributions, the score matching loss decomposes into a sum of objective functions that only see the local mode and are not capable of resolving the weights of the mixture components (Song & Ermon, 2019; Zhang et al., 2022).
3 Energy Discrepancies
To illustrate the idea behind the proposed objective func- tion, we start by motivating energy discrepancy from the perspective of explicit score matching. In the following, we will denote the EBM as Pebm X exp(-U), where U is
SM
MLE
3.18
MLE/ EDI
ED..
ED
the energy function that is learned. The nearsightedness of
3.00
ED.
score matching arises due to the presence of large regions of low probability which are separating the modes of the data distribution. To increase the probability mass in these regions, we follow (Zhang et al., 2020) and perturb Pdata and Pebm through convolution with a Gaussian kernel 7s(y x) x exp( -lly x1|2/2s), i.e.
2.00
0.00
Misure weight
Figure 1: Loss of energy discrepancy, score matching, and maximum likeli- hood estimation the task of estimating the weight in a mixture of two Gaussians. For details, see Appendix D.I.
Ps(y) = / 7s x)pdata(x)dtt
/ (y
The resulting perturbed divergence SMiPaata: U) == SM(P,, Us) retains its unique optimum at exp(-U")) x Pdata (Zhang et al., 2020) but alleviates the nearsightedness as the data distribution is more spread out. The perturbation with 7/s simultaneously makes the two distributions more similar which comes at potential loss of discriminatory power. We mitigate this by integrating the score matching objectives over s over an interval of noise-scales [0, 4. It turns out that this integral can be evaluated as the difference of two contrasting energy conrryycottibbtioon:::
L SMips, U.)ds Expsata (x) [Ux)) (x) (x: (x:))
The proof is given in Appendix A.3. The contrasting expression on the right-hand is now independent of the score and normalisation of the EBM. We argue that such constructed objective functions useful losses for energy-basse -aaed modelling.
3.1 A contrastive approach to learning the energy
We lift the idea of learning the energy-based distribution through the contrast of two energy- contributions to a general estimation loss called Energy Discrepancy. We will show that energy discrepancy can be defined independently of an underlying perturbative process and is well-posed even on non-Euclidean measure-spe measure-spaces:
Definition (Energy Discrepancy). Let pdata be a positive density on a measure space (X, and and let q(-1x) be conditional probability density. Define the contrastive potential induced by q as
Ugly): == -log expl)))
(4)
We define the energy discrepancy between Pdata and U induced by q as EDy U) == (x) - Epanatxx (x) E. (u(y)))
(5)
In this paper, we shall largely focus on the case where the data is Euclidean, i.e. X' = Rd, and the base-distribution dx is the standard Lebesgue measure. However, this framework also admits X' being discrete spaces like spaces of graphs, or continuous spaces with non-trivial base measures such as manifolds. Specifically, the validity of our approach is characterised by the following non-parametric estimation result:
'The integrals should be interpreted as Lebesgue integrals, i.e., if X' is discrete, the integral will be sum.
3

---

Theorem 1. Let Pdata be a positive probability density on (X, dx), and let q(- be a conditional probability density. Under mild assumptions on q and the set of admissible energy functions U, energy discrepancy g is functionally convex in U and has, up to additive constants, uuique global minimiser U arg min FD2 U) with exp(-U") x Pdata-
The assumption on the conditional density describes that y ~ g(-1x) involves some loss of information. The assumption that Pdata has full support turns out to be critical when scaling energy- discrepancy to high- nigh-dimensional data. The proof of Theorem is given in Appendix A.I.
3.2 Choices for the conditional distribution 9
Def. 1 offers wide range of possible choices for the perturbation distribution q. In Appendix D.3 we discuss a possible choice in the discrete space {0. 1)a. For the remainder of this paper, however, we will focus on Fuclidean data x E Rd and hope that the generality of our result inspires future work. Our requirements on are that simulating from the conditional distribution y ~ q(y/x) and computing the convolution that defines Ug are numerically tractable. On continuous spaces, a natural candidate for q is the transition density of a diffusion process which arises as solution stochastic differential equations of the form dx; = a a(x; )t + dw, with drift a and standard Brownian motion We (see Øksendal, 2003). The conditional density ge (-Ix) represents the probability density of the perturbed particle X2 that was initialised at Xy = x. The resulting transition density then satisfies both of our requirements by employing the Feynman -ay formula as we line out in Appendix B.2. Although this approach makes the choice of q flexible, the following interpolation result stresses that not much is lost when choosing Gaussian transition density 7 x exp( -|x y|2//20)
Theorem 2. Let qu be the transition density of the diffusion process dx, = a(x;) )dt + dw., let Pehm & exp(-U) be the energy-based distribution and Pr. the data-distribution convolved with qr-
1. The energy discrepancy is given by multi-noise scale score matching loss
[-SV,, F. (p..o..) [- (x.) 2 + VV (x.a)) ds + const.
gD (Pdata: U)
2. Ifa O, i.e. qe is the Gaussian transition density "It, the energy discrepancy converges to the loss of maximum likelihood estimation a linear rate in time
ED (Panta: U) + Epsaa(x) [log Pebm c(t))
2t W2 (Pdatas Pebm)
where c(t) is independent U and W2 denotes the Wasserstein distance?.
Theorem 2 has two main messages: All diffusion- based energy discrepancies behave like multi-noin scale score matching loss, which is independent of the drift a. In fact, we show in Appendix A.2 that for a linear drift ax2, the induced energy discrepancy always equivalent to the energy discrepancy based on a Gaussian perturbation. Furthermore, estimation with Gaussian-based energy discrepancy approximates maximum likelihood estimation for large t. thus enjoying its attractive asymptotic properties provided ED,, (Pdata. U)) can be approximated with low variance. We demonstrate the result in a mixture model in Figure/ I and Appendix D.I and give a proof of above theorem in Appendices A.3 and A.4.
Connection to Contrastive Divergence. Due to the generality of our result we can also make a direct connection between energy discrepancy and contrastive divergence. To this end, suppose that for fixed, yg satisfies the detailed balance relation g /(y|x) exp(-Eo(x)) = q(x(y) exp( Ey))). In this case, energy discrepancy becomes the loss function
(6) which, after taking gradients in 0 yields the contrastive divergence update". See Appendix A.5 for details. The non parammeetii estimation result from Theorem I holds true for the contrastive objective in (6). This means that each step of contrastive divergence optimises an objective function with minimum at Pdata R po- However, the objective function is adjusted in each step of the algorithm.
(Páata: Ea) (x) [Ex - E (x) B(y)))) [Eo
2 For a reference on Wasserstein distances, see Peyré & Cuturi (2019)
'The implicit dependence of q on 0 is ignored when taking the gradient. Notice that CD is not the gradient of
any fixed objective function (Sutskever & Tieleman, 2010).

---

Training Energy-Based Models with Energy Diserepancy
In sight of Theorem 2, we will discuss how to approximate ED from samples for Euclidean data {x*} C I2 and a Gaussian conditional distribution 4 - x) x exp(-lly x2l²/2t.. First, the outer expectations on the right-hand of (5) can be computed as plug-in estimators by simulating the Gaussian perturbation x x' + vfg for & No I) and averaging (U(x)} and (U=, (xx)). The critical step is then finding a low-variance estimate of the contrastive potential Uy, itself.
Due the symmetry of the Gaussian transition density, we can interpret "t as the law of a Gaussian random variable with mean x² and variance i.e. the conditioned random variable x + /xf for g' (O, I) has the density (x] x). Hence, we can write Un, as the expectation
trut
estmmated energy
2.5
Uy, (x)) log (e) (e)exp(-UUX + ((())
3.0
for i 1........ N. The conditioning expresses that keep x; fixed when taking the expectation with respect to E'. It is then possible to calculate the expectation by sampling Ed ~ (0, I) and calculating the mean as for the outer expectations. However, we find that this approximation is not sufficient as is biased due to the logarithm and prone numerical instabilities because of missing bounds on the value of U (x)). To stabilise training, we augment the Monte Carlo estimate of U,, (x() by an additional term w/M cxp(-U (xx)) which we call s-stabilisation. This results in the following approximation of the contrastive potential:
Figure 2: Estimated energy functions for Gaussian data with various choices of w. Increasing leads to flatter energy- landscapes, and training becomes unsta- ble if 0. For details, see Appen- dices B. and D.2.
M
Uge (x+) (xx))===oos log
exp(-U(xx)) M
M Σ
1
(xp(-U(x(+
goh N(0.I)
The w-stabilisation dampens the contributions of contrastive samples x + Vigii. whose energy is higher than the energy of the data point x'. This provides deterministic upper bound for the approximate contrastive potential (4) and reduces the variance of the estimation. We find that the w-stabilisation drastically reduces the number of samples M needed for the stable training of deep energy-based models. We illustrate the effect of the stabilisation in Figures and 24 and discuss reasoning in more details in Appendix B.1. The full loss is formed for U Eg with e¹ ~ (0, I). Edh. ~ NO0, I) and tunable hyperparameters I, M, w as
M
log
W M
Ex((()(x') Ex(x' Vie + vig M
Σ 1=1
N
1=1
The loss is evaluated using the numerically stabilised logsumexp function. The justification of the approximation is given by the following theorem:
Theorem 3. Assume that x ト exist N and (NN such that |C,, (0) ED,, (Pdata Ea) < almost surely.
is uniformly bounded. Then, for every E > 0 there
We give the proof in Appendix B.I. This result forms the basis for proofs of asymptotic consistency of our estimators which we leave for future work. It is noteworthy that analogous implementation of energy discrepancy for other perturbation kernels and other spaces possible. For example, we construct similar loss for discrete spaces using a Bernoulli perturhation in Appendix D.3. Similarly to the Gaussian case, the w--stabilisation provides a useful variance reduction for the training with Bernoulli- energy discrepancy.
4.1 Training Energy-Based Models under the Manifold Hypothesis
LeCun (2022) suggests that maximun-likelihood-ee training of energy-based models can lead to the formation of undesirable canyon shaped energy mergy-fuucction. Indeed, energy discrepancy yields an
5

---

Daya
Figure 3: Comparison of energy discrepancy, score matching and contrastive divergence on density estimation. The 1st and 2nd rows are the estimated density and synthesised samples, respectively.
energy with low values on the data-support and rapidly diverging values outside of it when being used on image data, directly. Such learned energies fail to represent the distribution between data- sota-points and are not suitable for inference or image generation. We attribute this phenomenon to the manifold hypothesis, which states that the data concentrates in the vicinity of liw-dimensional manifold (Bengio et al., 2013). In this case, the data distribution is not positive density and can not be written energy-based model log Pdata is not well-defined Additionally, Gaussian perturbations of a data point == x & are orthogonal to the data manifold with high-probability and the negative samples in the contrastive term are not informative.
To resolve this problem, we will work with a live-dimensional latent representation of the data distribution in which positivity can be ensured. We follow Pang et al. (2020) and de- fine a variational decoder network ps (x(z) and an energy-based tilting of a Gaussian prior pz(z) exp(-EEx (z)))E models (LEBMs) Pe. & (x|z)))p dz. To obtain the latent representation sample from the posterior (z.a(z)x) (x(z) (o(--E(z)))r the training of model, the contrastive divergence algorithm is replaced with energy discrepancy (for details see appendix C). LEBMs provide interesting benchmark compare energy discrepancy with contrastive divergence in high dimensions. However, other ways to tackle the manifold problem elements of ongoing research.
latent space, resulting in the so-called latent energy- baased prior
data we using Langevin sampler. For
5 Empirical Studies
To support our theoretical discussion, we evaluate the performance of energy discrepancy on low- dimensional datasets as well as for the training latent EBMs on high-dimensional image data sets. In our discussion, we emphasise the comparison with contrastive divergence and score matching as the dominant methodologies in EBM training.
5.1 Density Estimation
We first demonstrate the effectiveness of energy discrepancy on several two-dimensional distributions. Figure 3 displays the estimated unnormalised densities as well as samples that were synthesised with Langevin dynamics for energy discrep- ancy (ED), score matching (SM) and contrastive divergence (CD). More experimental results and details given Ap- pendix D.4. Our results confirm the aforementioned nearsight- edness of score matching which does not learn the uniform weight distribution of the Gaussian mixture. For CD, it can be seen that CD consistently produces flattened energy landscapes which can be attributed to the short-run MCMC (Nijkamp et al., 2020a, see) having converged. Consequently, the synthe- sised samples of energies learned with CD can lie outside of the data-support support. In contrast, ED is able to model modti-modal distributions faithfully and learns sharp edges in the data support as in the chessboard data set. We quantify our results in Figure 4 which shows the mean squared error of the estimated log-density of 25-Gaussians over the number training iterations. The partition function is estimated using importance sampling log Z logsumexp( E(x,) -log Paata Pxata(x;)) (x;)) -logg N, where x; is sampled from the data distribution (x) and N 5, 000. shows that outperforms SM and CD with faster convergence, lower mean square error, and better stability.
of Aonnnnn
20c
42k
Iterattoons
Figure 4: Density estimation aceu-
racy in the 25-Gaussians dataset.

---

81
1
"
66
88
85
3
150
5
8
3
6
R5%
2
40
33
151
12
2
2
41
0
91
000
1
6
4
09
180
63
8
19
33
24
$
23
366
8
3
8
55
(a) SVHN (32 x 32)
(b) CIFAR-I0 (32 32)
(c) CelebA (64 x 64)
Figure 5: Generated examples by the ED-LEBM trained on SVHN, CIFAR- 10, and CelebA.
Table I: Comparison of MSE(L) and FID(L) on the SVHN, CIFAR-10, and CelebA datasets.
SVHN
CIFAR-I0
CelebA
MSE
FID
MSE
FID
MSE
FID
VAE (Kingma & Welling, 2014)
0.019
46.78
0.057
106.37
0.021
65.75
2s- &--VE (Dai & Wipf, 2018)
0.019
42.81
0.056
72.90
0.021
44.40
RAE (Ghosh al., 2019)
0.014
40.02
0.027
74.16
0.018
40.95
SRI (Nijkamp al., 2020b)
0.018
44.86
0.020
61.03
SRI (I.=5) (Nijkamp et al., 2020b)
0.011
35.32
0.015
47.95
CD-LEBM (Pang et al., 2020)
0.008
29.44
0.020
70.15
0.013
37.87
SM-LEBM
0.010
34.44
0.026
77.82
0.014
41.21
ED-LEBM (ours)
0.006
28.10
0.023
73.58
0.009
36.73
5.2 Image Modelling
In this experiment, methods evaluated by training latent EBM three image datasets: SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), and CelebA (Liu et al., 2015). The effectiveness of energy discrepancy is diagnosed through image generation, image reconstruction from their latent representation, and the faithfulness of the learned latent representation. The model architectures, training details, and the choices of pyper-parameters can be found in Appendix C.3.
Image Generation and Reconstruction. We benchmark latent EBM priors trained with energy discrepancy (ED-LEBM) and score matching (SM-LEBM) against various baselines for latent variable models which are included in Table Note that the original work on latent EBMs (Pang et al., 2020) uses contrastive divergence (CD-LEBM) (see appendix for details).
For a well-learned model, the latent EBM should produce realistic samples and faithful reconstructions. The reconstruction eтror is measured via the mean square error while the image generation measured with the FID (Heusel et al., 2017) which are both reported in Table 1. We observe that ED improve the contrastive divergence benchmark on SVHN and CelebA while the results on CIFAR-10 could not be improved. However, we emphasise that ED only requires M (here, M 16) evaluations of the energy function per data point which is significantly less than CD and SM that both require the calculation of a high-dimensional spatial gradient. Besides the quantitative metrics, we present qualitative results of the generated samples in Figure 5. It can be seen that our model generates diverse high-quality images. The qualitative results of the reconstruction are shown Figure 6, for which we use the test set of CelebA 64 x 64. The right column shows the original image x to be reconstructed. The left column shows the reconstruction based ~ Pz(z), and
X
XK
X
on latent variable initialised from the base distribution the middle column displays the image reconstructed from Z₃ ~ (z)x( which is sampled via Langevin dynamics. One can see that our model can
Figure Image re- construction results on CelebA 64 x 64.
successfully reconstruct the test images, verifying the validity of the latent
prior learned with energy discrepancy. In addition. we showcase the scalability of our approach by
7

---

Figure 7: Image interpolation results on CelebA 64 x 64. The first and last columns show the observed
images, while the middle columns display the interpolation results using the inferred latent vectors.
(a) Smiling
(b) Male
(c) Eyeglasses
(d) Blond Hair
Figure 8: Attribute manipulation by interpolating the latent variable along an attribute vector. The
middle image corresponds to the original image, and each subfigure represents a different attribute.
Table Comparison of AUPRC(") for unsupervised anomaly detection on MNIST dataset.
Heldout Digit
4
5
7
9
VAE (Kingma & Welling, 2014)
0.063
0.337
0.325
0.148
0.104
ABP (Han et al., 2017)
0.095 + 0.03
0.138 + 0.04
0.147
= 0.03
0.138
= 0.02
0.102 + 0.03
MEG (Kumar al., 2019)
0.281 1 0.04
0.401
0.402
1:002:000 0.06
0.290
0.04
0.342 1 0.03
BiGAN-o (Zenati et al., 2018)
0.287 0.02
1: 0.443
0.514
0.347
1.02 0.02 0.307
0.03
CD-LEBM (Pang et al., 2020)
0.336 + 0.01 0.630 1 0.02 0.619
0.463
0.01
0.01
0,413
0.01
SM-LEBM
0.285 + 0.01 0.663 = 0.01
0.610
0.01
0.471 = 0.01
0.422 + 0.01
ED-LEBM (ours)
0.342 + 0.01 0.740 + 0.01 0.708
0.01 0.501 = 0.02 0.444 + 0.01
applying it successfully to high-resolution images (CelebA 128 128). More results can be found in Appendix D.5.
Image Interpolation and Manipulation. We use a latent-variable to model the effective low- dimensional structure in the data set. To probe how well the latent space describes the geometry and meaning of the Sata-manifold we analyse the structure of the latent space through interpolation and attribute manipulation. For the interpolation between two samples we linearly interpolate between their latent representations which were sampled from the posterior distribution. The results in Figure 7 demonstrate that the latent space has learned the data-manifold well and almost all intermediate samples appear as realistic faces. Further results are given in Figures 19 and 20. In addition, we can utilize the labels in the CelebA dataset to modify the attributes of an image through the manipulation technique proposed in (Kingma & Dhariwal, 2018). Specifically, each image in the dataset is associated with a binary label that indicates the presence or absence of attributes such as smiling, male, eyeglasses. For each manipulated attribute, we compute the average latent vectors ppos of images with and Zneg of images without the attribute in the training set. Then, we use the difference Zpos Zneg the direction for manipulating the attribute of image. The results presented in Figures and 22 confirm the mcaningfulness of the latent space learned by energy discrepancy.
5.3 Anomaly Detection
In a well-learned energy-based model the likelihood should be higher for in-distribution examples and lower for -idtt--diittibttion examples. Based on this principle, we conduct anomaly detection experiments to compare our method with other baselines. Specifically, given test sample x, we first sample z from the posterior Po (z)x) by Langevin dynamics, and then compute the unnormalised log- log-density Po (x, z) as the decision function. Following the protocol in (Zenati et al., 2018), we designate each digit class in the MNIST dataset as an anomaly and leave the rest as training data. The area under the precision-recall curve (AUPRC) is used to evaluate different methods. As shown in Table 2, our model consistently outperforms the baseline methods, demonstrating the advantages of training latent EBMs with energy discrepancy.
8

---

6 Related Work
Training Energy-based models. While energy-based models have been around for some time (Hinton, 2002), the training of energy-based models remains challenging. For summary on existing methods for the training of energy-based models see Song & Kingma (2021) and LeCun et al. (2006). Contrastive divergence is still the most used option for training energy-bass models. Recent extensions have improved the scalability of the basic algorithm, making it possible to train EBMs on high-dimensional data (Ngiam et al., 2011; 1; Dai et al., 2015; Xie et al., 2016; Nijkamp al., 2019; Du & Mordatch, 2019). Despite these advances, it has been noted that contrastive divergence is not the gradient of any fixed loss-function (Sutskever & Tieleman, 2010) and can yield energy functions that do not adequately reflect the data distribution (Nijkamp et al., 2020a). This has motivated improvements to the standard methodology (Du et al., 2021) by approximating overlooked entropy terms or by improving the convergence of MCMC sampling with an underlying diffusion model (Gao al., 2021).
Score-based methods (Hyvärinen & Dayan, 2005; Vincent, 2011; Song et al., 2020) are typically implemented by directly learning the score function instead of the energy function. Song & Ermon (2019); Song et al. (2021b) 1b) point out the importance to introduce multiple noise levels at which the score is learned. Li et al. (2019) adopt the strategy of learning an energy-baset model using multi-scale denoising score matching.
We learn a latent EBM prior (Pang et al., 2020) on high-dimensional data to address situations where the data lives on lower dimensional submanifold. This methodology has been improved with multi-stage approach (Xiao & Han, 2022) with score-independent noise contrastive estimation (Gutmann & Hyvärinen, 2010).
Related loss functions. Energy discrepancies can be derived from Kullback cobber-conttraction divergences (Lyu, 2011) which we also discuss briefly in Appendix A.6. Diffusion recovery likelihood (Gao et al.. 2021) technically optimises the same loss as us but the EBM is trained with contrastive divergence. Luo et al. (2023) make similar observations to us while estimating the KL-contraction by leveraging the time evolution of the energy function along diffusion process. Energy discrepancy also shares similarities with other contrastive loss functions in machine learning. The o-aatbilised energy -eergyddicrrppncyy loss with w, M = 1 is equivalent to conditional noise contrastive estimation (Ceylan & Gutmann, 2018) with Gaussian noise. For M > 1, the stabilised ED loss shares great structural similarity to certain contrastive learning losses such as InfoNCE (van den Oord et al., 2018). We hope that such observations can lead to interpretations of the w-stabilisation.
Theoretical connections between likelihood and score-based functionals. The connection between likelihood methods and score-based methods generalises the de Bruijn identity (Stam, 1959; Lyu, 2009) which explains that score matching appears frequently as the limit of EBM training methods. Similar connections have been mentioned and exploited by Song et al. (2021a) to choose a weighting scheme for a combination of score matching losses.
Generative modelling for manifold-valued data. The manifold hypothesis was, for example, described in Bengio et al. (2013). Prior work to us shows how score-based diffusion models detect this manifold (Pidstrigach, 2022) and give reasons why CD is more robust to this issue than other contrastive methods (Yair& & Michaeli, 2021). Arbel et al. (2021) learn an energybbased model as a tilt of GAN-based prior that models the data-oonifold. We believe that a combination of above results with energy discrepancy could enable training EBMs with energy discrepancy on data space.
7 Discussion and Outlook
We demonstrate that energy discrepancy provides a new tool for the fast training of energy-based models without MCMC or scores. We show for Euclidean data that ED interpolates between score matching and maximum likelihood estimation, thus alleviating problems of nearsightedness of score matching without additional annealing strategies as in score-based diffusion models. Based on our theoretical analysis, we show that training EBMs using energy discrepancy yields more accurate estimates of the energy function for two-dimensional data than explicit score matching and contrastive divergence. In this task, it is robust to the hyperparameters used, making energy discrepancy useful tool for energy-based modelling with little tuning required. We further establish that energy discrepancy achieves comparable results to contrastive divergence at lower computational cost

---

when learning lowe--dimensionll energy-based prior for nigh-dimensional data. This shows that energy discrepancy is scalable high dimensions.
Limitations: Energy-based models make the assumption that the data-distributtin has a positive density which is violated for most high-dimensional data sets due to the manifold hypothesis. Compared to contrastive divergence or diffusion- based strategies, energy discrepancy is especially sensitive such singularities the data set. Currently, this limits energy discrepancy to settings in which a lower-dimensional representation with non-singular support can be learned or constructed efficiently and accurately.
This work can be extended in various ways. Firstly, we want to explore other choices for the conditional distribution q. In particular, the effect of different types of noise such Laplace noise or anisotropic Gaussian noise are open questions. Furthermore, energy discrepancy is a well- suited objective function for discrete data for appropriately chosen perturbations. We present preliminary study of energy discrepancy on discrete spaces in the workshop paper Schröder et al. (2023). Secondly, we believe that improvements to our methodology can be made by learning the energy function of image-data on pixel space directly, as in this domain specific architectures such as U-nets (Ronneberger et al., 2015; Song & Ermon, 2019) can be used in the modelling. However, this requires further work in learning the data-manifold during training so that early saturation can be prevented. Finally, the partial differential equations arising Euclidean energy discrepancies promise exciting insights into the connections between energy-based learning, stochastic differential equations, and optimal control which we want to investigate further.
Outlook:
Acknowledgements
TS and ABD are grateful to G.A. Pavliotis for insightful discussions throughout the project. TS was supported by the EPSRC-DTP scholarship partially funded by the Department of Mathematies, Imperial College London. ZO was supported by the Lee Family Scholarship. JNL was supported by the Feuer International Scholarship in Artificial Intelligence. ABD was supported by Wave I of The UKRI Strategic Priorities Fund under the EPSRC Grant EP/T001569/ and EPSRC Grant EP/W006022/1, particularly the "Ecosystems of Digital Twins" theme within those grants and The Alan Turing Institute. We thank the anonymous reviewer for their comments.
References
Arbel, Michael, Zhou, Liang, and Gretton, Arthur. Generalized energy based models. In International Conference on Learning Representations, 2021.
Barp, Alessandro, Briol, Francois-Xavier, Duncan, Andrew, Girolami, Mark, and Mackey, Lester. Minimum stein discrepancy estimators. Advances in Neural Information Processing Systems, 32, 2019.
Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1788)11798 2013.
Perpita- Miguel Á. and Hinton, Geoffrey. On contrastive divergence learning. In Proceed- ings of the Tenth International Workshop on Artificial Intelligence and Statistics, volume RS of Proceedings of Machine Learning Research, pp. 33-40. PMLR, 06-08 Jan 2005.
Ceylan, Ciwan and Gutmann, Michael U. Conditional noise-contrastive estimation of unnormalised models. In Dy, Jennifer and Krause, Andreas (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 726-734. PMLR, 10-15 Jul 2018.
Chwialkowski, Kacper, Strathmann, Heiko. and Gretton, Arthur, A kernel test of goodness of fit. In International conference on machine learning, pp. 2606-2615, PMLR, 2016.
Dai, Bin and Wipf, David. Diagnosing and enhancing vae models. In International Conference on Learning Representations, 2018.
10

---

Dai, Hanjun, Singh, Rishabh, Dai, Bo, Sutton, Charles, and Schuurmans, Dale. Learning discrete energy based models via auxiliary-varrii variable local exploration. Advances Neural Information Processing Systems, 13:10443-10455 2020.
Dai, Jifeng, Lu, Yang, and Wu, Ying Nian. Generative modeling of convolutional neural networks. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Du, Yilun and Mordatch, Igor. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
Du, Yilun, Li, Shuang, Tenenbaum, Joshua, and Mordatch, Igor. Improved contrastive divergence training of energy-based models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2837-2848. PMLR, 18-24 Jul 2021.
Friedman, Avner. Stochastic differential equations and applications. Courier Corporation, 2012.
Gao, Ruiqi, Song. Yang, Poole, Ben, Wu, Ying Nian, and Kingma, Diederik Learning energy- based models by diffusion recovery likelihood. In 9th International Conference Learning Representations, ICLR, 2021.
Gelman, Andrew and Rubin, Donald B. Inference from iterative simulation using multiple sequences. Statistical science, pp. 457-472, 1992.
Ghosh, Partha, Sajjadi, Mehdi SM, Vergari, Antonio, Black, Michael, and Scholkopf, Bernhard. From variational to deterministic autoencoders. International Conference on Learning Representations, 2019.
Glaser, Pierre, Arbel, Michael, Doucet, Arnaud, and Gretton, Arthur. Maximum likelihood learning of energy-based models for simulation-bass -base inference. arXiv preprint arXiv:2210. 14756, 2022.
Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256 JMLR Workshop and Conference Proceedings, 2010.
Gorham, Jackson and Mackey, Lester. Measuring sample quality with kernels. In International Conference on Machine Learning, pp. 1292-1301. PMLR, 2017.
Grathwohl, Will, Wang, Kuan-Chich, Jacobsen, Jörn-Henrik, Duvenaud, David, Norouzi, Mohammad, and Swersky, Kevin. Your classifier is secretly an energy based model and you should treat it like one. In 8th International Conference on Learning Representations, ICLR, 2020.
Grathwohl, Will, Swersky, Kevin, Hashemi, Milad, Duvenaud, David, and Maddison, Chris. Oops i took a gradient: Scalable sampling for discrete distributions. International Conference on Machine Learning, pp. 3831-3841 PMLR, 2021.
Gutmann, Michael and Hyvärinen, Aapo. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Han, Tian, Lu, Yang. Zhu, Song-Chun, and Wu, Ying Nian. Alternating back- nack-propagation for generator network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Heusel, Martin, Ramsauer, Hubert, Unterthiner, Thomas, Nessler, Bernhard, and Hochreiter, Sepp. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Hinton, Geoffrey E. Training products of experts by minimizing contrastive divergence. Neural aug 2002.
Comput.,
Hyvärinen, Aapo and Dayan, Peter. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.

---

Kingma, Diederik P and Ba, Jimmy, Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015.
Kingma, Diederik P and Welling, Max. Auto-encoding variational bayes. 2nd International Confer- ence on Learning Representations, ICLR, 2014.
Kingma, Durk P and Dhariwal, Prafulla. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Krizhevsky, Alex, Hinton, Geoffrey, et al. Learning multiple layers of features from tiny images. 2009.
Kumar, Rithesh, Ozair, Sherjil, Goyal, Anirudh, Courville, Aaron, and Bengio, Yoshua. Maximum 2019.
entropy generators for energy -bagyy models. arXiv preprint
LeCun, Yann. From machine learning autonomous intelligence lecture 2, 2022. URL https: //leshouches2022 www.gg.t.........ii...
02.eas--2.... 02.
LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, and Huang, Fujie. tutorial energy- based learning. Predicting structured data, 1(0), 2006.
Li, Zengyi, Chen, Yubei, and Sommer, Friedrich T. Learning energy- models in high- dimensional spaces with multiscale denoising- scooee matching. Entropy, 2019.
Liu, Qiang, Lee, Jason, and Jordan, Michael. A kernelized stein discrepancy for goodness-of-N tests. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 276-284. PMLR, 2016.
Liu, Xing, Dunean, Andrew B, and Gandy, Axel. Using perturbation improve goodness-of-fh tests based on kernelized stein discrepancy. International Conference on Machine Learning ICML, 40, 2023.
Liu, Ziwei, Luo, Ping. Wang. Xiaogang, and Tang, Xiaoou. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Luo, Weijian, Jiang, Hao, Hu, Tianyang, Sun, Jiacheng, Li, Zhenguo, and Zhang, Zhihua. Training energy- baaeg models with diffusion contrastive divergences, 2023.
Lyu. Siwei. Interpretation and generalization score matching. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence. UAI '09, pp. 359-366, Arlington, Virginia, USA, 2009, AUAI Press. ISBN 9780974903958.
Lyu, Siwei. Unifying non-maximum likelihood learning objectives with minimum kl contraction. In Shawe-Taylon J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
Majerek, Dariusz, Nowak. Wioletta, and Ziba, Wies. Conditional strong law of large number. International Journal of Pure and Applied Mathematics, 20, 01 2005.
Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng. Andrew Y. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Ngiam, Jiquan, Chen, Zhenghao, Koh, Pang Wei, and Ng, A. Learning deep energy models. In International Conference Machine Learning, 2011.
Nijkamp. Erik, Hill, Mitch, Zhu, Song-Chup Chum, and Wu, Ying Nian. Learning non con-convergent non- persistent short-run meme toward energy-based model. In Advances in Neural Information Processing Systems, volume 32, 2019.
Nijkamp, Erik, Hill, Mitch, Han, Tian, Zhu, Song-Chun, and Wu, Ying Nian. On the anatomy of meme-based maximum likelihood learning energy-based models. In Proceedings of the AAAI Conference Artificial Intelligence, 2020a.
12

---

Nijkamp, Erik, Pang, Bo, Han, Tian, Zhou, Linqi. Zhu, Song-Chun, and Wu, Ying Nian. Learning multi-layer latent variable model via variational optimization of short run meme for approximate inference. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 361-378. Springer, 2020b.
Ou, Zijing, Xu, Tingyang. Su, Qinliang, Li, Yingzhen, Zhao, Peilin, and Bian, Yatao. Learning neural set functions under the optimal subset oracle. Advances in Neural Information Processing Systems, 35:35021-35034 2022.
Pang, Bo, Han, Tian, Nijkamp, Erik, Zhu, Song-Chun, and Wu, Ying Nian. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33:21994-2200 2020.
Peyré, Gabriel and Cuturi, Marco. Computational optimal transport. Foundations and Trends in Machine Learning, 11 (5-6):355-602, 2019.
Pidstrigach, Jakiw. Score-based generative models detect manifolds. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp- 35852-35865. Curran Associates, Inc., 2022.
Raginsky, Maxim and Sason, Igal. Concentration of measure inequalities in information theory, communications, and coding. Foundations and Trends@ in Communications and Information Theory, 10(1-2):1-266, 2013. ISSN 1567-2190. doi: 10. 561/0000000044.
Ronneberger, Olaf, Fischer, Philipp. and Brox, Thomas. U-net: Convolutional networks for biomedi- cal image segmentation. Medical Image Computing and Computer-Assisted Intervention -MICCA,, 9351:234-28 241, 2015.
Schröder, Tobias, Ou, Zijing. Li, Yingzhen, and Duncan, Andrew B. Training discrete ebms with energy discrepancy. Sampling and Optimization in Discrete Spaces workshop, ICML, 2023.
Song, Yang and Ermon, Stefano. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Song, Yang and Kingma, Diederik P. How to Train Your Anergy-Basees Models, arXiv e-prints, 2021.
Song. Yang, Garg, Sahaj. Shi, Jiaxin, and Ermon, Stefano. Sliced score matching: scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-584. PMLR. 2020.
Song, Yang, Durkan, Conor, Murray, lain, and Ermon, Stefano. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428,
2021a.
Song, Yang, Sohl-Dickstein, Jascha, Kingma, Diederik P., Kumar, Abhishek, Ermon, Stefano, and Poole, Ben. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR, 2021b.
Stam, A.J. Some inequalities satisfied by the quantities of information of fisher and shannon. Information and Control, 2(2):101-112, 1959. ISSN 0019-9958.
Sutskever, llya and Tieleman, Tijmen. On the convergence properties of contrastive divergence. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 789-795. JMLR Workshop and Conference Proceedings, 2010.
van den Oord, Aäron, Li, Yazhe, and Vinyals, Oriol. Representation learning with contrastive predictive coding. arXiv preprint 1807.03748. 2018.
Vincent, Pascal. A connection between score matching and denoising autoencoders. Neural computa- tion, 23(7):1661-1674 2011.
Xiao, Zhisheng and Han, Tian. Adaptive multi-stage density ratio estimation for learning latent space energy-based model. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances Neural Information Processing Systems, volume 35, pp. 21550-21601. Curran Associates, Inc., 2022.
13

---

Xie, Jianwen, Lu, Yang. Zhu, Song-Chun, and Wu, Yingnian, A theory of generative convnet. In International Conference on Machine Learning, pp. 3635-2644 PMLR, 2016.
Yair, Omer and Michaeli, Tomer. Contrastive divergence learning a time reversal adversarial game. In International Conference on Learning Representations, 2021.
Zenati, Houssam, Foo, Chuan Sheng, Lecouat, Bruno, Manek, Gaurav, and Chandrasekhar, Vijay Ra- maseshan. Efficient gan-based anomaly detection. arXiv preprint arXiv: 2018.
Zhang. Mingtian, Hayes, Peter, Bird, Thomas, Habib, Raza, and Barber, David. Spread divergence. In III, Hal Daumé and Singh, Aarti (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11106-11116. PMLR, 2020.
Zhang, Mingtian, Key, Osear, Hayes, Peter, Barber, David, Paige, Brooks, and Briol, François-Xavier Towards healing the blindness of score matching. NeurlPS 2022 Workshop on Score-Based Methods, 2022.
Øksendal, Bernt. Stochastic Differential Equations An Introduction with Applications. Universitext. Springer, Berlin, Heidelberg, sixth edition. edition, 2003. ISBN 77886421433448
14

---

Appendix for "Energy Discrepancies: A Score-Independent Loss for Energy-Based Models"
Contents
Proofs and Derivations
15
A.1 Proof of the Non-Parametric Estimation Theorem 1.
15
A.2 Equivalence of Energy Discrepancy for Brownian Motion and Ornstein-
Ornstein-Ullenber Uhlenbeck Processes
17
A.3 Interpolation between Score-Matching and Maximum-Likelilann Estimation
18
A.4 Representing Energy Discrepancy as Multi-Scale SM for General Diffusion Processes
21
A.5 Connections of Energy Discrepancy with Contrastive Divergence
23
A.6 Derivation of Energy Discrepancy from KL Contractions
24
Aspects of Training Energy-Based Models with Energy Discrepancy
24
B.1 Conceptual Understanding of the w-Sttabilisation
24
B.2 Approximation of Energy Discrepancy based on general Ito Diffusions
27
Latent Space Energy-Based Prior Models
28
C.I Brief Review LEBMs
................................
28
C.2 Langevin Sampling, Reconstruction, and Generation
................................
30
C.3 Experimental Details of LEBMs
30
Additional Experimental Results
31
D.I Experimental Sctup for Figure (Healing the nearsightedness of score-matching)
31
D.2 Experimental Sctup for Figure (Understanding the w-stablisation)
32
D.3 Energy Discrepancy on the Discrete Space (0, 13
32
D.4 Additional Density Estimation Results
33
D.5 Additional Image Modelling Results
34
D.6 Qualitative Results the Effect of f, M, and
34
A Proofs and Derivations
In this section we are going to prove Theorem and Theorem Furthermore, we are going to generalise Theorem 2 to arbitrary diffusion processes and discuss the connections of energy discrepancy to other training losses for energy-based models.
A.1 Proof of the Non-Parametrie Estimation Theorem
In this subsection we give a formal proof for the uniqueness of minima of FD, (paata, U) as a functional in the energy function U. We first reiterate the theorem stated in the paper:
Theorem 1. Let Paata be a positive probability density on (X, dx). and let g( x) be a conditional probability density. Under mild assumptions on and the set of admissible energy functions U. energy discrepancy E.D. is functionally convex in and has, up to additive constants, unique global minimiser U+ arg min F.D.(Pdata, U) with exp( -U*) x Pdata-
For this theorem we need to specify additional assumptions on the conditional distribution g and on the optimisation domain to guarantee uniqueness. Firstly, we require the energy-based distribution to be normalisable which implies that exp(-U) T2* dx). For the existence and uniqueness of min- imisers we have constrain the space of energy functions since BDy(Paats U) ED, (pasta: U +c)
15

---

for any constant e E R. Hence, we restrict the optimisation domain to functions U that satisfy minxex (xx To specify the condition on q we define an equivalence relation on X:
Definition 2 (g-equivalence), For pair of samples X E X we say that and x2 are neighbours if there exists a y such that X1, X2 E supp(g(y).)) We denote the set of of q-neighbours of as (x). We then define the q-equivalence relation as:
X1 r X2 there exist (Z1.n) with 2 X1, Z = X2. and Zr+1 (Ng(z,)
is easy to see that g-equivalence defines an equivalence relation, i.e. is symmetric, reflexive, and transitive. We summarise the assumptions for the non non-parametric approximation theorem follows: Assumption 1. We define the optimisation domain
G: X such that exp(-U) E (x), U E L' (pdnta), and min ((x) 0f We then make the following assumptions on q and G:
xeX
1. All elements of X are g-equivalent, i.e. for every x E X it holds that [x]g = X.
2. There exists a U E such that cxp( X Pdata-
Under Assumption 1, ED (Pdata U') has unique global minimiser U log Pdata cn G. We prove this by computing the first and second variation ED2- Note that & may not be a vector space in general since in some cases G. Furthermore, this technical issue in this discussion. We start from the following lemmata and complete the proof of Theorem in Corollary 1.
did not specify norm on G. We will omit
Lemma 1. Let h E G be arbitrary. The first variation of ED, is given by
de F.D, (Paata; U + +hh)
Ex (x) [h(x)] Pa (x) Euts x)
c=0
where pu (zly)
<(y'z) exp( ((x)) J ('olyla exp( (x
Proof. We define the short-hand notation U. U ch. The energy discrepancy at Ug reads
ED, U.) Ex (xx) [x)] + (x Byty [oo / ((yl2) (xll)))ex(-UUx)))ee)))
For the first functional derivative, we only need calculate
de log / (yylz) exp(
/ (xxple))())exp((UU(e() J exp()U, dz = -E,., (aly) [4(z). (8)
Plugging this expression into F.D,, (Paata: U.) and setting = 0 yields the first variation of F.D
Lemma 2. The second variation of ED2 is given by
d²
F.D (Pdatan U + ch) E (x) By(yix) Varpo(aay)) [M()).
de
Proof. For the second order term, we have based on equation 8 and the quotient rule for derivatives:
d
log
4(vlz) (x(())))x(UU
dea
[q(1l2) exp(U. 12 (z) dz q(ylz') (2
([q(vlz) exp( (xp(-U,(xx))dll
( (x)(U. q(yl22 (z))))) exp exp((U
(ʃ q exp( (x)))
E Epu, (ely) ( (x)) Epo, (2ly) [1x2)]2 = Var pu. (zly) [h(2)]
We obtain the desired result by interchanging the outer expectations with the derivatives in
16

---

Corollary 1. Let c minxex(- log Pdata (x)) For U
log((data) G it holds under
Assumption I that
d
F.Dg (Panta: U eli)
=0
dc d² (D,a(Pataa U + thi
c=0
>
for all
E g.
de
==0
Furthermore, U is the unique global minimiser of ED (Dy(Paaaas' in g.
Proof. By definition, the variance is non-negative, i.e. for every h g:
d² ED, (pdata: U +hh) (xly) (x)))) 20.
de
Consequently, the energy discrepancy is convex and extremal point of (Dyapaatas (Pdata: .) is global minimiser. We are left to show that the minimiser is obtained at U and unique. First of all, we have:
q(ylzz) exp((U h(z)dz /
q(ylz) (x)(-U"(z)
(aalz) (z)
(xx)
gly ((lylz))Pasaal" Paasa z2)dz' h(z)dz.
By applying the outer expectations we obtain
E (xly) [h(z)] / Pu (x)x / ((ylz)Pdatal h(z) dy dz
[ (qy(z' Pdata z')dz'
/ ((tlx)) (z) dy dz pa (z) [h(z]],
where we used that the marginal distributions Jf (PPaaaaa))(())) (x)((x)x)dd))))))) cancel out and the conditional probability density integrates to one. This implies
d ED, EDy U + ch)
Exxmax(x) [(zz)] - (x) [h(z)] 0.
de
l==0
for all h E G.
We now show that the second variation is also positive definite, i.e.
d² ED (Pdntay U + +ch)
Epssa (x) Buty x) Varpass [1(2)] > 0.
de
c==
where Pdata (zly) T Tolsiz "Paatalz ydx"" Assume that there exists test function h G such that (xnnaa paseaa [n(z)] = O. Since this is the expectation of non-negative random variable, it follows for all y E V that Varpanss(tly) [z)) = O. Consequently. lu(z) has to constant Pdata P- --losst surely and there exists a measurable function g y - such that h(z) g for all z supp(p(-ly)). Next, assume that z and are g-neighbours. By definition, there exists y such that 21, 22 E supp(4(y)))) Since the data density positive, the support of q(yl.) equals the support of Paata (-ly). Consequently, it holds that h((2)) = g(y) h(zz). It now follows from the definition of q-equivalence that h has to be constant almost surely on all geequivalence classes [x],, since any two elements in an equivalence class can be connected by chain of g-neighbours. By Assumption 1, [x], = X, and thus constant on X. However, there are no non-trivial constant test function in G. Hence, the second variation has to be positive definite for all l E G. Consequently, U** the unique global minimiser of ED, which completes the statement Theorem 1.
glylz)pnaa((e (x))
A.2 Equivalence of Energy Diserepancy for Brownian Motion and Processes
In this subsection we show that energy discrepancy based on Orsteen-UUhlenbeck process is equivalent to the energy discrepancy based on a time-changed Brownian motion.
Proposition 1. Let qi be the transition density for the Ornstein- Uhhehnne-hlebbbekeeee process dx, ax,dt /Bdw, with standard Brownian motion W,, and let (e(y))) x exp( Ily x||2/20 be the Gaussian transition density of Brownian motion. Then,
EDg (pasta: U) ED 702 (Ddatar U) at
where ao (t) v (c2at 1) and andao(t) (t = Bt.
17

---

Proof. At time t, the Ornstein- OrnseennUUlenbeee Uhlenbeck process has distribution
X edd X0 oo(E)E, 8 {~N(00,1)
with (1) (e2at -1) 1) and (n VBl. The Ornstein- -hhaeennU process is variance
exploding for 2 0 and variance preserving for a < 0. Based on (9). the transition density of X, is given 2a
V2TO (- 202(()
ly xx12
da (x(x)x)
exp
Hence, we obtain via the change of variables = (y (x)= ~ NO0 I) for the contrastive potential
/ exp(()))xx
Ucly) - log
/ (n(5)) exp (-U (y--o(tt)E)) de' at.
log
- log
We now evaluate the contrastive potential at the forward process = which yields
Up(xi) log 7/1 (E() exp (-U (e (E" + oa(t) (t()))))) de' - at
-logg / 71((") exp (-U (xo + --)((1) - --a(1) de' al -
-lo log / 702a(1) (Woz_(1) - -x exp(_Uxx)xx=
where we used that rt aa (t) = (t) in the second equality and the change of variables x Wo2., (1) - E' in the third equality. Hence. the energy discrepancy for the Ornstein- Omstein-Uhlenbecck Uhlenbeck process is equivalent to the energy discrepancy for Brownian motion with time parameter
B 2a
(t) =
(1 2ar)
Notice that for the variance-exploding process with a > 0 the contrasting particles have a finite co. Hence, the maximum maximum-likelihodd limit in Theorem 2 is only achieved for the variance preserving process with < 0 and for the critical case of Brownian motion
horizon since (t) 1-000
with a = O.
A.3
Interpolation between Score-Matching and
Estimation
We first prove the result as stated in the Gaussian case. We then show how the result can be generalised arbitrary diffusions by using Ito calculus.
Gaussian case Denote the Gaussian density
|ly
x
7 x) = /2mt
exp
21
and define the convolved distributions Pt Pdata and == 7 (xp(-U) Proposition 2. The energy discrepancy is the multi noise-scale score-matching loss
(paata: U) = (p.)
-AUs(xa) + (x,())))1 ds
18

---

Proof. It is known that V/ is the solution of the heat equation:
d -x x] - -
Consequently, both, Pt and exp(-Ui satisfy the heat-equation because the integral commutes with the differential operators. Based on the heat-equation differential equation for the contrastive potential U₁:
derive the following non-linear partial
d,e d.e Ucl)
Oomly - x)e Ux
/ Ayrily -x)- -xx
Aye-Ui -U(())
1 Vy (VVUU(()))--000)
2
VyUi(y))12 - U(ly)
Since 0,e-u U. -e---a we get after cancellation of the exponentials:
o,U,(y) = 2 A,, _ [yU((>))
The integral notation of the contrastive term in energy discrepancy takes the form
E Epaata (x) E -x) (U) = U(yy))(y)d..
We now take derivative of the energy discrepancy and find
orED,, (Pdata: U) - Ucly)po(y)dy
(eUoly))pely)ee
U,ly)drrlyddy
/ Aypey)dy
(QUu,()))
/ (a.U.(y)) prly)dy
/ (A)U)(y)) pely)dy
where we used integration by parts twice in the final equation shift the differential operator from Pe to Ur. Now, plugging in the differential equation for U, we find
O,ED,, (paata; U) (-
+ 1 1 - - (,,))()))
(r(y)dy
E (x) Tey -,U(( + + [V
Finally, we obtain energy discrepancy by integrating above expression:
ED7, (Paata: U) = //¹ O,ED,, (Pdatas U'ds
L/¹ (x)
1V, ds
This gives the desired integral representation in Proposition 2.
Proposition 3. Let "it be the Gaussian transition density and Pebm cxp(-U) the energy-based
distribution. The energy discrepancy converges to a cross entropy loss at linear rate in time
1 W2 (Paata- Pobm)
ED7, (Pentas U) + Epsaia()) [log Petm (x) - -c()) <
2t
19

---

where c(t) is renormalising constant independent of U.
For the proof employ the following lemma of Yihong Wu which was given in Raginsky & Sason (2013).
Lemma 3. Let T/t be the Gaussian transition density of a standard Brownian motion. Let v.v be probability distributions and denote p4 == */t and U4 == *n. The following information-treaspor inequality holds:
KL || m4) - W2(,,v)
2t
Proof. Let ıT be a probability density (x, x') with marginal distributions (x(x) and vxx') (also called coupling in optimal transport). We have by the decomposition of Ballback-Leibble divergences
f KKL( -x) x) 70(- x (x),xx x' x')xxxx - KL(He || 1/)
- -fKL KL Htly x)x x')
|| yol x) x ) 2 0
(x)
vy(y)
Hence, we find by rearranging the inequality
KL(m |l m4) - KL(M( x) ll 7e(- x') (x),, x' dxxx''
The right hand side is the Kullback-Leibie divergence between Gaussians, so
KL(X((-- x) rel- x')) x
Since the coupling was arbitrary, we can minimise over all couplings π of μ and v which results in the daaserrstein-distanc
/ KL((((- x) (x((-xx)art x'xxxx)dok x'dxxx'' w2(H,))
KL( p4) min
rell(w)))
where [1 v) denotes the set of all joint distributions with marginals and v.
The proof of Proposition 3 then follows:
Proof. Let μ Pdata and v = Pebm denote the convolved distributions as Pr,data and Pr.ebm- Notice that for any x, y E Rd
Pt. (y) (xx (v)
log
Pobm (x)
since both models have the same normalising constant. We then have for arbitrary x
/ (logPt.. log Pt, (y)) Pr.datalyddy
KT(pe,data || Pr.cbm)
/ (IogPr,dustu())) Pr,daata()) loog Pt,cbm (y)
log Pebmm(x) Pr,data Jyy
Pobm((X
/ (log U.(y) - U(x) Pe,data (y)dy log Pebiu (x)
c(t)) + f U Uely)pt,datay - Uxx) - log Pebm
with U independent entropy term c(t) : Epy [log Pr,data(y)) Since x was chosen arbitrarily, we can integrate with respect to Pdata(x) and find
0 < (T, Pr.ctom)
c(t) / [U:(Y)P (y)dy /
/ log Pebm (x)aaa (x)
(UCe---UXxxpA U(x)pddncxxx))
1
= =((t) -ED,, (pdata: U) - Epeara (x) (log Dobm (x) <
w3 (paata Pobm)
21
20

---

A.4 Representing Energy Diserepancy as Multi-Scale SM for General Diffusion Processes
We now prove the connection between energy discrepancy and multi-noise scale score matching in general context. For all following results we will assume that is some stochastic diffusion process which satisfies the SDE dx; = a(x,) ) + xt )dw, and assume that X ~ Pdata- Let şt denote the associated transition probability density. To make the exposition cleaner we write U, == Uqe
The main idea will be the following observation:
Proposition 4. The diffasion-based energy discrepancy is given by the expectation of the Ito integral
E. (Pdata, U) = -B [ (U.)).)
Proof. The stochastic integral with respect the differential dU, (x,) is defined to satisfy the following generalisation of the fundamental theorem of calculus:
Ux(x.) Uo(x)) 1 (x,(
We obtain the desired result by taking expectations on both sides in (Xy)usaSt.
Notice that the law the random variable x, is fixed by the initial distribution of the diffusion Xg ~ Pdata- These distributions are implied when taking the expectation. We will explore this connection further. For this we make some basic assumptions which allow us to connect stochastic differential equations with partial differential equations.
Assumption 2. Consider the stochastic differential equation dx, a dt b(x,) for drift Rdd - Rd and b IRd - RR Further, define the diffusion matrix Exx) = b(x) e Rdxdd We make the following assumptions:
1. There exists Js such that for all &, TRd (E. E(x)E) /
2. Σ and bounded and uniformly Lipschitz-comtimoous on every compact subset of R
3. Σ is uniformly Hölder-continuous in X
Theorem 4 (Fokker-Planck equation). Under Assumption 2, Xt has a transition density function given by
IP(x, A|xo= = x)= gr (x(xx)....
Furthermore, qt satisfies the Fokker-Planck partial differential equation
d
0.g((y(x) Σ 0y.
-a(x)a + Σ By, (2 (2)(())
(10)
1=1
(y(x)) = =0ly x)
For a reference, see (Friedman, 2012, Theorem 5.4)
The Fokker-Planck equation yields the following important differential equation for the contrastive potential U,:
Proposition 5. Consider the stochastic differential equation dx, = a(xx)dt + ddw, for drift Rd Idd and Rdd + IR and diffusion matrix Σ(x) b(x)b(x))* Raxd that satisfies assumptions 2. Lel qr be the associated transition density and define the contrastive potential U,(y) :=
-log = qg(y)xx ())x)) exp(-U(x))dxx Farthermore, we define the scalar field
d Σ Oy. By Eij
N)) = Σ 1=1
Oy (g. 2 j=1
21

---

and the linear operator
(a, 2.
Σ
04
0
+
1
Σ
20y, Eij dy:
0
Enj
8
oy:
dy. ,oy
Then, the contrastive potential satisfies the non-linear partial differential equation
0,U,(y) C(a, 1 (y)VU((y>)1 c(a, (y)
Proof. We commute the linear operator of the Fokker-Planck equation that e- satisfies the Fokker-Planck equation in Theorem 4, too, i.e.
1 + dje Uily) = Σ Oy, -a:(y)e - Ucly)
+ Eoy, Oys (Zis(y))-
i=1
1=1
Unly) U(x)
We now expand the term corresponding the drift term:
Σ ay. (-a,
(-0x,a(x)++(())),1 -U(y))
Similarly, we treat the diffusion term:
d Σ
Oy,0y, (Eis(y)e-
Oy 2110))))
2 4=
2 i,j=1
20y, Ei 10y U.ly) Oy.
Finally, the time derivative simply becomes 0,e U,ly) -VU Uily), We can now collect all terms independent of U and identify
c(a, E)(y)
Σ
1 ag,
Σ
By. Oy Eny(x)
2
as well the linear operator term
d
C(a, E) =
Σ
04 0y:
+
1
Σ
20y,
Eij Ny:
+ Eij By 0y,dy,
2
1=1
i==1
Finally, we have
Σ
ba (0)
i-==1
U -1b"
This gives the partial differential equation
-a,u(yy)--uice Uc(y) ( (-L(a, E)U,)) + 1 (y) c(a,,)(x) E)rr
-U(y))
Cancelling all exponentials from both sides of the equation yields the desired result.
Theorem The energy discrepancy takes the form of a generalised multi-noise scale score matching
loss:
ED,, qe (Pdntar U) /'' Σ Ox (E.)(x,) (xx)) + 1 1 (x.))U,(xx)) + const.
11=1
22

---

Proof. For this proof we return to the stochastic process U₂ (x,) from Proposition 4. By Ito's formula,
(x,) satisfies the stochastic differential equation
Σ (x, Ux(x,)) + 1 Σ
dU,(x,)) (x,, +
Eyy(y)xx, 0x, U(x)) ds
1,j=1
8x. U, (x,
Under the additional integrability condition that E 16 (x.)VU, (x.())) < co, the stochastic integral with respect to Brownian motion dw, has expectation zero. Furthermore, we can replace (x,) with our previously obtained non-linear partial differential equation
(xU,(x,) C(a, (x.) + 116" (x,))U((x))) c(a, Ex.)
Due to opposing signs, the drift cancels, i.c.
Oxx + 1 2,j=1
C(a, E),, + Σ
Σ
(xx))
1=1
d
+ Eij(y) 0²
Oy, Eij(y) Qy:
Ux(x.)
,x,0x;
1,==1
Öz, Oz Uxx))
1,j=1
Consequently, we obtain the final energy discrepancy expression using Proposition
EDg (Pdata: U) = -E
(x,(
E
Σ i,j=1
Oxy (xx) (xx)) _ 11)))))))))))))))))))))))) (x.)(x)UV) + const.
with U-independent constant Jg c(a, (x.x))ss. This completes the proof.
As a corollary we obtain the proof of the first statement in Theorem Assume that qe is de- fined through the stochastic differential equation dx, = a(x, )dt + dw, In this case, E = I and 1 0x, (x, Uxx)) (x,).. Consequently, matching representation of ED,, in Theorem 2. In the special case that Σ = 66 is independent obtain integrated sliced score-matching loss (Song al., 2020).
obtain from Theorem the score
A.5 Connections of Energy Discrepancy with Contrastive Divergence
The contrastive divergence update can be derived from an energy discrepancy when, for Eg fixed, q satisfies the detailed balance relation
q(y(x) exp(-Ex q(x exp(-Ee(y)).
To see this, we calculate the contrastive potential induced by We have
llog| q(y exp((Ex(x))ddx - log / q(xly) exp(-Eooy))eee Eo
Ealy).
Consequently, the energy discrepancy induced by q is given by
F.D, (Paata: Fig) pxaatx (x) (x)) Eppssee (x) Byyy [y)(y))
Updating 0 based on sample approximation of this loss leads to the contrastive divergence update
1 A0 x N
1 Σ VoEo(y") g'~g(-Ix") N
Σ VoEo(x))
Three things are important to notice:
23

---

1. Implicitly, the distribution g depends on Eg and needs to adjusted in each step of the algorithm
2. For fixed g, ED, (Paata: Ea) satisfies Theorem 1. This means that each step of contrastive divergence optimises loss with minimiser Ej = log Paata + c. However, y needs to be adjusted in each step as otherwise the contrastive potential is not given by the energy function Eg itself.
3. This result highlights the importance to use Metropolis- Hastings adjusted Langevin-sampler to implement CD to ensure that the implied distribution satisfies the detailed balance relation. This matches the observations found by Yair & Michaeli (2021).
A.6 Derivation of Energy Discrepancy from KL Contractions
Kullback-Leibler contraction the divergence function KL(Paata (llbm)-KL(Q2)) (KPaaaa ll QPebm) (Lyu, 201 for the convolution operator Qp(y) = f g(y (x)x))(((xx)) The linearity of the convolution operator retains the normalisation of the measure, i.e. for the energy-based distribution pobm we have
Peebm = 2 / g(ylx) exp(-U((xx with Zu Ru=1 cxp((U(xx)dxx.
Zu
The KL divergences then become with Ug == - log Q exx(((((x))
(L |I Pebm) = Eaa.t [log Pd (x)] + (x) (x + log Zu KL || Qpebm) = Eqpases llog QP)ataay) + EEpasca (y) [y)(y) + log Zu Since the normalisation cancels when subtracting the two terms we find
(P || Pebm) (T.(QPaata || QPebm) (PData U) + c where c is a constant that contains the U U-independent entropies of pdata and Qpdata-
B Aspects of Training Snergy-Bassed Models with Energy Discrepancy
In this section, we discuss the w-staailliation in depth. Furthermore, we give proof for Theorem and show energy discrepancy can be upproximated for other diffusion processes via the Feynman- formula.
B.1 Conceptual Understanding of the w-Stabilisation
The w-stabilisation is a useful stabilisation for all types of perturbation q. For example, we use the same stabilisation in the discrete setting (see Appendix D.3). To provide a better intuition for the stabilisation, we will investigate more closely for the Gaussian perturbation.
The critical step for using energy discrepancy in practice a stable approximation of the contrastive write the contrastive potential as yy) - log E[exp(-UU( + V({')) with g' ~ N0. I) and y E Rd. A naive approximation of the expectation with a Monte-Carlo estimator, however, is biased because of Jensen's 's inequality, i.c. for e', " ~ (00, I) we have
potential. For the Gaussian-basse energy discrepancy,
M
(v) = - log E (exp((U ViE'M < -E log
exp(-U'(y + Vte"))
M
j=1
Our first observation is that the appearing bias be quantified to leading order. For this, define v == (exp(-U(y + VFE()) and dy(y) = 1 Σ/e exp(-U(y + VFE")). We use the Taylor-expansion of log(1 (1 + u) s u - 1 /2 + h.o.t. which gives
1 1 or(y) - vx())) ]
E Uog log (y) = log
(1))
vx(y)
1
(or(y) Ve 2x))
h.o.t.
2
4x(y)²
1
Var (exp(-U(xx + ViE")) + hott.
2y)
24

---

The linear term in the Taylor-expansion does not contribute because (y): = (y)).. In the final equation we used that (y) = Var(exp(-Ui(y /IM because all şd are independent. The Taylor expansion shows that the dominating contribution to the bias is the variance of the approximated convolution integral.
Our second observation is that this occurring bias can become infinite for malformed energy functions. For this reason, the optimiser may start to increase the bias instead of minimising our target loss. To illustrate how a high-variance estimator of the contrastive potential be divergent, consider the energy function
( 0 for x S 0 U(x) = bx
for x > 0
The energy function does not strictly adhere to our conditions that the energy based model should be normalisable. Our argument still holds when exp(-U) is changed be normalisable. In theory, the contrastive potential at 0 is upper bounded because
(0) < lim log IE eexp(=U(([eeee
= log ( S
log(1/2)
because exp( (rpp((U(x)) (x)) converges to an indicator function on (x S 0} as b - so. The Monte Carlo estimator of the contrastive potential, on the other hand, has upper bound
0,(0) log M
1
Σ
exp(-U(ViE") S min(U((Vee)) ),. j=1
" U(Vig(M) log(M)
which can be seen by applying standard inequalities for the logsumexp function Hence, as long as there exists a j such that E < O, the estimated contrastive potential does not diverge. If, however, E for every j = .,.., M, then
Ú,(0) 2 min ((ViE")) ..... (V) (V.E"))).....))))) by minlg") ( bu20 0o.
Consequently, the approximate contrastive potential may attain diverging values discontinuities in the energy function. Indeed, this phenomenon is observed for w = 0 in Figure 2. Here, the learned energy becomes discontinuous the edge of the support and the energy discrepancy loss diverges during training. In low dimensions, this problem can be alleviated by using variance reduction techniques such as antithetic variables or by using large enough values of M during the training. The stabilising effect of M is observed in our ablation studies in Figure 24. In high- -ige-dimensional settings, however, such variance reduction techniques are infeasible.
The idea of the w--stabilisation is that the value of the energy at non-perturbed data points U(xo) is guaranteed to stay controlled since is minimised in the optimisation of ED. Hence, the diverging contrasting potential can be controlled by including (xx)) in the summation in the logsumexp operation which acts as a soft-min over all contrasting energy contributions. Indeed, this augmentation provides a deterministic upper bound to the approximated contrastive potential:
M
(x,) = - log
w M
exp(-U
Σ 1=1
exp(-U((x + )Fg"))
M
min [xx; +
U(x + Vig -lg)-oog(ee log(w)) log(M)
Additionally, the w-stabilisation introduces negative bias to the approximated contrastive potential. Hence, if tuned correctly, it counteracts the bias introduced by the Jensen-gan of the logarithm.
To gain additional intuition on the effect of w, notice that by the same bounds as before,
U (xx) - (x.) S min (U(x)) - Ux+ + )...))...... U(xo) U(x + Vig log(w))
for every data point This tells us that, roughly speaking, perturbed data point with U (xx) U(x, vTg') > log(w) should have small contribution the loss and the optimisation converges the data distribution learned or when the bound is violated at all perturbed data points. Thus, log(ur (w) describes weak notion of margin between positive and negative energy contributions. Consequently, large values for w E (0, 1) tend to lead to flatter learned energies, while smaller values lead to steeper learned energies. This intuition is confirmed by Figures 2 and 24.
It holds that min(u1, u2...., uM)) log(M) v - -LSE( -2, U2....,
........uuM -uM) min(us, (1,, U2, ..., uM)
25

---

Asymptotic consistency of sample approximation of ED We give proof for Theorem 3 which states that our approximation of energy discrepancy is justified. To make the exposition easier to understand, we first show how the energy discrepancy is transformed into conditional expectation. Recall the probabilistic representation of the contrastive potential Section 4. Using Ex = log(exp (xxp( Eg (x))) we obtain the following rewritten form of energy discrepancy:
EDy, (pdata, Ea) = E Expaaa (x) [x(x)) + pa (x) Byy. (y-(()))) x)
log (e))))) (exp(_FFor Eoly + Vig'ly My
Epenes (x) [x)] + Ex (x) Bry log E (5))
exp( Hep(-Lop 1x + Vig + Vtt') x { exp( - Ex (x + Vie + v |xx, s|]
Bpacss (x) (E) (E)
log(exp(EE Eo
+ log E (E")
Bpaas (x) Bn
log
exp E (x) - Eo(x + Vtg + vtE') x ( 5
The conditioning means that the expectation is not taken with respect to y or x and in the inner expectation. The conditioning is important to understand how the law of large numbers is to be applied. We now come to the proof that our approximation is consistent with the definition of energy discrepancy:
Theorem 3. Assume that x + exp(-Eg(x) is uniformly bounded. Then, for every E >> 0 there exist N and M(N such that (N ED,,, (Pdata: Fea) & almost surely.
Proof. First, consider independent random variables x ~ Pdata. & ~ -N(00,)) I), and gos ind NNO, I). Using the triangle inequality, can upper bound the difference ED y, (Pdatas Eg) Lr, Mo by upper bounding the following two terms, individually:
ED to (pdata: Foo
1
Σ
log E. (xx((Fo(x')= FFoxx + vig + Vigh) xx
N
1=1
N
log E (xpp(E((x)) Ex(x' + Vig + vieli) xxx - Ma()))(
+
N
The first term can be bounded by sequence En - 0 due to the normal strong law of large numbers. The second term can be estimated by applying the following conditional version of the strong law of large numbers (Majerek al., 2005, Theorem 4.2):
exp [Fo(x) - (x + Vig + vigh)) a.e. E. (exp(Fa(x) - Fx + vig + vig') E
M 1=1
Next, we have that the deterministic sequence w w/M - O. Thus, adding the regularistion w/M does not change the limit in M. Furthermore, since the logarithm is continuous, the limit also holds after applying the logarithm. Finally, the estimate translates to the sum by another application of the triangle inequality. We define
Aco(x, E,E') == exp(Ee(x) - Eg(x + Vig + ViE))
For each i = 1, 2,.,. N there exists a sequence EL,M a.R., such that
N
(,,5)xxxx/ xx,== LL,, log E A .',,
N
i=1
N
1 M
log E Aco(xi &', (x,5) xx llog M j=1
Aco(x s'oxx'''',5" g'
N
N
M, S max
N
Hence, for each 6 0 there exists an N e and an M(N) E N such that [DD,, (pdatay Fe) (,, E almost surely.
26

---

B.2 Approximation of Energy Diserepancy based on general Ito Diffusions
Energy discrepancies are useful objectives for energy-based modelling when the contrastive potential can be approximated easily and stabily. In most cases this requires us to write the contrastive potential as an expectation which can be computed using Monte Carlo methods. We show how such probabilistic representation can be achieved for much larger class of stochastic processes via application of the Feynman-Kac formula. We first highlight the difficulty. Consider the integral f'y )g (y px)aaata (aiaaxxxxxxe (x)dxdy, Since the expectation is taken in the integral can be represented as an expectation of the forward process associated with gr, i.e.
N Σ f(x)
I ((x))g(s/x//uuaa))dddd = Eppsia (x)) (x)))f((()()))) [f(x))) ~ N
where x are simulated processes initialised at x-j x² Pdata Next, consider the integral
yt, == / 9
This integral is more difficult approximate because the function g is evaluated at the starting point of the diffusion x, but weighted by it's transition probability density. To compute such integrals without sampling from g we use the Feynman-Kac formula, see e.g. Øksendal (2003):
Theorem 6 (Feynman-Kace)
Let g (R2) and e E C(RR). Assume that E (1.2 Rd) is bounded on K x Idd with K compact and satisfies
{ [ru(l,y)) y) Au(l,y) +cl))(((,))
for all 1. >> 0, E IRd for all y E R2d
(12)
v(0,y)= ggy))
Then, v has the probabilistic representation
vt,y = By exp ((scs)) )
where (y:)ezo is diffusion process with infinitesimal generator A.
We will establish that vt, y) satisfies partial differential equation of the above form which yields probabilistic representation the contrastive potential. We know that vtt,y) satisfies the Fokker- Planck equation (10). By applying the product rule to each term in the Fokker-Planck equation we find
a(v)
d
d
d Σ b
Qru(1,y)=
=
ax(y)= +
Σ 3=1
By; Enj(y)
Oy. + 2
Egg Ey),,
v(l,y)
(=1
d
+
1
By, Ey,
Σ 0xy) v(t.y)
2
cly)
(0,y)) =g()).
By comparing with Theorem 6, we identify the infinitesimal generator
0²
A
Eij (y)
By. By;
==1
Hence we associate the forward diffusion process dx; = a(x))dl + b(x, )dw, with it's backwards process with infinitesimal generator A
dy: = aly,)dt b(y:)dw'
27

---

with E(y) = b(y)b"(y)) (y). This yields the probabilistic representation of v) backward process y::
in terms of the
v(l,y) ( = By eepp (foovv)dls s(xx))
Hence, we also obtain probabilistic representation for the contrastive potential by choosing g(x) == exp(-U(x)). This finally gives
Utly) - -lg Boy exp (
Unlike the contrasting term in contrastive divergence, this expression can indeed be calculated by simulating stochastic processes that are entirely independent U. For this we simulate from the forward process starting at x which yields X,, where the tilde denotes that this simulation may not be exact. We then simulate M copies of the reverse process and keep values at intermediate steps, i.e. (5% = St........ -ix=() for =1,... 1,.... .M. Finally we evaluate the contrastive potential as
M
(ta - ta-..) U(F)) (C
K
Ux(x,) = - llog M
Σ exp 5=1
The simulation method for the stochastic process and for the integration Io c(y.) may be altered in this approximation. At this stage, it is unclear what practical implications the weighting term So e(y,)ds has. Notice that the process y, initialised at the final simulated position of the forward process X, Furthermore, the bias correction with the ws-ttabilisation or an alternative method should still be relevant for stable training of energy-based models.
C Latent Space Energy-Based Prior Models
In this section, we first briefly review the latent space energy- masey prior models (LEBMs) and its variants: CD-LEBM, SM-LEBM, and ED-LEBM. We then proceed the experimental details.
C.1 A Brief Review of LEBMMs
Latent space energy-based prior models (Pang et al., 2020) seek to model latent variable models (x) (x(z))oo with an EBM prior pz(z) exp( Eo(x))po(x) where pz is a base Zo distribution which we choose as standard Gaussian (Pang et al., 2020). LEBMs often perform better than latent variable models with fixed Gaussian prior like VAEs since the EBM prior more informative and expressive (Pang et al., 2020, Appendix C). However, training LEBMs is more expensive compared to latent variable models with fixed Gaussian prior because of the cost of training for energy-based models. This motivates to explore various training strategies for EBMs such as contrastive divergence, score matching, and the proposed energy discrepancy, where we find that energy discrepancy is the most efficient in terms of computational complexity.
The parameter update for the LEBM can be derived from maximum likellhood estimation of (x)) Using the identity (x))(x))) Vg. Ps.e ez.x(z|x)) =0, the gradient of the log- likeliiol of a data point x is given by
=E V Po. P (x) = Ep (x/xx) V 0.e Po, (z, x)] = E (x|xx) V log ps + Ve log pe (z)].
The posterior Po.s (z x) prescribes the latent representation of the data point x. Consequently, in each parameter update, samples are generated from the posterior distribution (z(x()()) via running Langevin dynamics and are treated as data on latent space. The generator is then updated via
V Po. (x)xx V log (x|2)).
Similarly, the maximum likelihood update for the EBM parameters 0 is given by Vo log Px.e(x) (x)x) Va logpo(22 As with any EBM, this gradient can not used, directly, since this would require a tractable normalisation constant Zg. To make this update tractable, we replace the gradient of the log- likelihoeooo with contrastive divergence, score matching, and energy discrepancy as outlined below.
28

---

Algorithm CD-LEBM
Algorithm 2 SM-LEBM
Algorithm ED-LEBB
1: sample from posterior and prior
sample from posterior
1: sample from posterior
2+ ~ p(z]x); 2- pa (z)
2 ~ p(zzx)
z ~ p(z]x)
2: evaluate the energy difference
2: evaluate the score difference
2: evaluate the energy difference
dg t [z(z+) Ee (z-)
Volog -Vgl))-Vlgggiii
dg / B/a-///o///EVVVT
3: Update parameter 0 using (13)
3: Update parameter 0 using (14)
3: Update parameter using (15)
ye Veda
0 < 0
o/o Va de
0 + 8-17 V log (w/M + +
Figure 9: The training procedure for the EBM prior. We one training sample only illustrate.
Algorithm 4 Learning latent space energy-based prior models
1:
repeat
2: Sample training data points {x' Ji= ~ Px)ta(x)
For each x", sample corresponding latent variable z' (z. via
Zi++1 z + 2 Va log (z.x(z)x') + Vews. we ~ NO00 1), zú ~ Po(z)
4: Update parameter Φ by maximizing log-likelihood
á <- Φ + To V Ei log p& (z') (((x)
5: Update parameter a by running Algorithms 1, 2, or
T/e V VeL. sM, a (0)
6: until convergence parameters (0,
CD-LEBM (Pang et al., 2020). The contrastive divergence update is obtained as per usual by expressing the gradient of the log likelihood in terms of the energy function
V log (x) Ep,, (xlx)) (alx)]V log po pz)z] = (x) Vol VoL
Therefore, the EBM prior be learned by minimising
N
(t)
1
Σ
Eo(z)) ), z (s.o(zlx'), z'_ ~ (z).
(13)
N
Note that optimizing CD-LEBM is computationally expensive, as training the EBM prior requires simulating Langevin dynamics to sample z frompe.0 (x)0(2)x) to generate positive samples and (z to generate negative samples.
SM-LEBM. The second solution is to minimize the Fisher divergence between the posterior and prior, which has the following form
Ep. V lo) Vz log Ps.0 (z)x)
This equivalent score matching (Hyvärinen & Dayan, 2005) when (z)x( is treated as parameter independent data distribution. We refer to this approach as score-matching LEBM, in which the EBM prior is learned by minimising
N Σ
(SM(O)
V lo (o()) V. log x1 x² po.0(z)x)).
(14)
where the parameters of Ps.0 (2/x) are suppressed in the update. Note that score matching generally requires computing the Hessian of the log density as in but in score-matching LEBM, we have Va log p(z))) = Vz log p(x|z) + Vz log p(z).
ED-LEBM. Finally, the EBM prior can be learned by minimising the energy discrepancy between the posterior and the EBM prior with Ez(z) == (z) log Pa (z)).. which can be estimated as follows
M
(ED = N log
W
Σ
Exp((E((z)) - Eo + Vtt + (lghh)))
(15)
M
M
with z' PPs ( (z)x'). Note that energy discrepancy does not require simulating MCMC sampling on the EBM prior and calculating the score of the log density, which is computationally friendly for
large-scale training. It is critical to include the base distribution po(z) in the energy function Eg. We summarize the training process of the EBM prior using CD-, SM- and ED-LEBM in Algorithms 1, 2, and 3, with the training procedure of LEBM given in Algorithm 4.
29

---

Table 3: Model architectures of LEBMs on various datasets.
(a) Generator for SVHN x 32, ngf 64
(b) Generator for CIFAR-I0 32, ngf 128
Layers
In-Out Size
Stride
Layers
In-Out Size
Stride
Input: x
1xlx100
Input: x
1xlx128
4x4 convT(nng x 8), IRRLU
4x4x(ngf 8)
8x8 convT(nng's x 8), LReLU
8x8x(ngf 8)
4x4 convT((ngf 4), I.RelU
8x8x(ngf 4)
2
4x4 convT(nngf 4), IRRLJU
16x 16x16x(ngf x 4)
2
4x4 convT(nng 2), I.ReLU
16x16x(ngf x
4x4 convT(nng' 2), L.ReIU 32x32x(ngf x
4x4 T(3), Tanh
32x32x3
3x3 convT(3), Tanh
32x32x3
(c) Generator for CelebA 64
64, ngf 128
(d) Generator MNIST 28 28, ngf 16
Layers
In-Out Size
Stride
Layers
In-Out Size
Stride
Input: x
Ixlx100
Input: x
16 4x4x(ngf 8)
4x4 convT(ngi's x 8). LReLU
4x4x(ng[ 8)
4x4 convT(ngg's x 8), LReLU
1
4x4 convT(ngf'X 4), LReLU
8x8x(ng[ 4)
3x3 convT(ngg's x 4), LReLU
7x7x(ngf 4)
4x4 convT(ngf 2), LReLU
16x16x(ngf x
4x4 convT(ngf 2), LReLU
14x 14x14x(ngf x
4x4 convT(ngf'x x 1), LReLU
32x32x(ngf x 1)
4x4 convT(1), Tanh
28x28x1
2
4x4 conv T(3), Tanh
64x64x3
2
(d) EBM prior
Layers
In-Out Size
Input:
16/100/128
Linear, LReLU
200
Linear, LReLU 200
Linear
C.2 Langevin Sampling, Reconstruction, and Generation
To sample from the EBM prior po (z) and posterior Po.o (z|xx we employ a standard unadjusted Langevin sampling routine, i.e. we repeat for k = 0, 1,...... K
z 2 = 2 + v logp(z) + Vewk, we ~ I))
where Zy ~ (z() and the distribution p(z) is replaced by the prior or posterior densities, respectively.
The generator is modelled as the Gaussian Po z) = -N(p((2), I²I). In reconstruction of x, we sample from the posterior zx ~ P4.0 (2).(2(() and compute the reconstruction as = (xx22)) In data generation, we sample from the EBM prior Zgen ~ pz() and compute the generated synthetic data point as X Xgen (2(en))).
C.3 Experimental Details of LEBMs
Datasets. We use the following datasets in image modelling: SVHN (Netzer et al., 2011), CIFAR- 10 (Krizhevsky al., 2009), and CelebA (Liu al., 2015). SVHN of resolution 32 32, and containts 73, 257 training images and 26, 032 test images. CIFAR-10 10 consists of 50, 000 training images and 10, 000 test images with a resolution of 32 x 32. For CelebA, which contains 162, 770 training images and 19, 962 test images, we follow the pre-processing step in (Pang et al., 2020), taking 40, 000 examples of CelebA as training data and resizing it to x 64. In anomaly detection, we follow the setting in (Zenati et al., 2018) and the dataset can be found in their published code".
Model Architectures. We adopt the same network architecture used in CD-LEBM (Pang et al., 2020), with the details depicted in Table 3, where conv' conyT(() indicates a transposed convolutional operation with n output channels. We use Leaky ReLU as activation functions and the slope is set to be 0.2 and 0.1 in the generator and EBM prior, respectively.
Details of Training and Inference. Here, we provide detailed description of the hyperparameters setup for ED-LEBM. Following (Pang et al., 2020), we utilise Xavier normal (Glorot & Bengio, 2010)
Detection
30

---

to initialise the parameters. For the posterior sampling during training, we use the Langevin sampler with step size of 0.1 and run it for 20 steps for SVHN and CelebA, and 40 steps on CIFAR-10. We set = 0.25,, M = 16, wo = 1 throughout the experiments. The proposed models are trained for 200 epochs using the Adam optimizer (Kingma Ba, 2015) with fixed learning 0.0001 for the generator and 0.00005 for the EBM prior. We choose the largest batch size from {128, 256, 512) such that it can be trained on a single NVIDIA-GeForce- -ET-Foo---R--2200----- 2080-Ti GPU. In test time, we observed that slightly increasing the number of Langevin sampler steps can improve reconstruction performance. Therefore, we choose 100 steps with a step size of 0.1 for posterior sampling. Based on the insights gained from the MCMC diagnostic presented in Figure 21, choose 500 steps with step size of 0.2 to ensure convergence of the Langevin dynamics when sampling from the EBM prior.
Evaluation Metries. In image modelling, we use FID and MSE to quantitatively evaluate the quality of the generated samples and reconstructed images. On all datasets the FID is computed based on 50, 000 samples and the MSE is computed on the test set. Following (Zenati et al., 2018 20;8; Pang et al., 2020), we report the performance using AUPRC in anomaly detection and results are averaged over last 10 epochs to account for variance.
D Additional Experimental Results
In this section we present additional experimental results to highlight the effect of various parameters of our approach.
D.1 Experimental Setup for Figure 1 (Healing the nearsightedness of score-matching)
major problem of score-based methods is their nearsight- edness, which refers to their inability to capture global properties of distribution with disjoint supports such as the mixture weights of two well-separated modes (Zhang et al., 2022). In sight of Theorem 2. energy discrepancy should alleviate this problem as implicitly compares the scores of both distributions at multiple noise-scales. Fol- lowing Zhang et al. (2022), we investigate this by comput- function of the mixture weight
10
MILE
10
10-1
ing energy discrepancy p for the mixture of two Gaussians 91 == ( 1) and G2 == M5, 1), i.e.
10a
26
Figure 10: Study of the influence of t and M on estimating mixing weights.
Pr(x)= Pg1 (x) + (1 (x).
where the true data has the mixture weight = 0.2. We compare energy discrepancy (I) (p) ~ ED(Pp=0.2, log with the objective of maximum likelihood estimation MLE(p) = E 2())) |- log (z)] and the score matching objective which here is given by the Fisher divergence SM(p)= ;= _ E p.=0 [|| logPp=o.2(z)) V (x)) [[]]. The losses as functions of p are shown in Figure We find that energy discrepancy convex as a function of the mixture weight and approximates the negative log- liglliieldhodd as f increases. Consequently, energy discrepancy can capture the mixture weight well for sufficiently large values of SM, on the other hand, is a constant function and is blind to the value of the mixture weight.
To further investigate the impact of and M the efficiency of energy discrepancy, minimise the energy discrepancy loss L2,M=33,wm (p) as function of the scalar parameter p for various choices of M and l. We compute the mean-square error of 50 independent estimated mixture weights for choice of and M. As shown in Figure 10, the estimation performance approaches that of the maximum likelihood estimator as f increases, which verifies the statement in Theorem 2. Moreover, if the number of samples M used to estimate the contrastive potential is increased, the estimation performance can be further increased towards the mean- square error of the maximum- likelloood estimator.
31

---

Figure 11: Potential outcomes for the estimated energy and loss history when ED does not converge with w 0.
D.2
Experimental Setup for Figure 2 (Understanding the w-stablisation)
To probe our interpretation of the w-stablisation, we train a neural-network learn the energy function using 4, 096 data points of a one-dimensional standard Gaussian Pdata (2) xc exp(-x² (000222/2) /2). The neural network uses an input layer, hidden linear layer of width two IR² - RR2 and scalar output layer IR2 - IR with Sigmoid Linear Unit activation between the layers. This neural network has sufficient capacity to model the Gaussian data as well as degenerate energy functions that illustrate potential pitfalls of energy discrepancy for O. The energy discrepancy is set up with hyperparameters M = / = 1, and w E {0, 0.05, 0.25, 2. and is trained for 50 epochs with Adam. Our results are shown in Figure which confirms the relevance of the w-stablisation to obtain stable optimisation of energy discrepancy. We remark here that the degenerate case w = not strictly reproducible. Different types of lacking smoothness of the energy-function at the edge of the support lead to diverging loss values. We chose result that illustrates the best the theoretical exposition of the w-stablisation in Appendix B. and refer to Figure to reflect other malformed estimated energies as well as an example of a diverging loss history.
D.3 Energy Discrepancy on the Discrete Space {0, 1)dd
Energy discrepancies are, in principle, well-defined on discrete spaces. To illustrate this point, we describe the energy- desery-diyerypancy loss for {0, 1) valued data such images with binary pixel values, in which case the discrete energy-discrepancy is straight forward to implement. We will replace the Gaussian transition density with Bernoulli distribution. For (0, 1), let Bernoulliee).. Then the transition y = x + & mo)(2) is symmetric and induces symmetric transition density 4(y x). Because of the symmetry, the energy discrepancy can be implemented in the same way in the continuous case, i.c.
((0)
exp (xx) (x @ g' @ gths))
N
with &', g.. ~ Bernoulli and where @ is the addition modulo 2. To test the effectiveness of energy discrepancy with Bernoulli perturbation, we test energy discrepancy on various discrete settings. In Figure 12, estimate the symmetric connectivity matrix of Ising models following the exposition Grathwohl et al. (. We then learn the densities of two-dimensional synthetic data sets that mapped to the space {0, 1 32 via grey codes in Figure 13, following Dai et al. (2020). Finally, we estimate the densities of image data with binary pixel values for the static and dynamic MNIST dataset, the omniglot dataset, and the caltech silhouettes dataset. As shown in Figure 14, energy discrepancy performs competitively to contrastive divergence. We propose other potential discrete perturbations Schröder et al. (2023).
Figure 12: Results of learning Ising models. Left: ground truth; Right: learned pattern.
Figure Visualisation of the energy function learned on 2D plan. Here we train discrete EBM on 32-dim grey codes which are one-to-one mapping from Ig² to {0, 1332.
32

---

2
9 1O0
2
Figure 14: Generated samples on discrete image modelling. Here we train discrete EBM using the Bernoulli transition. Left to right: Static MNIST, Dynamic MNIST, Omniglot, Caltech Silhouettes.
24
FD
FD
32
Figure 15: Additional results on density estimation.
D.4 Additional Density Estimation Results
Here, we provide additional details and results on the density estimation experiments.
Details of Training and Inference. Our choice for the energy-net for density estimation a 4-layer feed-forward neural network with 128 hidden units and softplus activation function. In the context of energy discrepancy, we select 1, MM 4, and divergence approach, we utilise CD-1, in which the gradient of the log liglliielihood in Equation (1) estimated by employing Langevin sampler with a single step and step size of 0.1. For score matching, we train EBMs using the explicit score matching in (2), where the Laplacian of the score is explicitly computed. We train the model using the Adam optimizer with a learning rate of 0.001 and iterations of 50, 000. After training, synthesised samples are drawn by simulating Langevin dynamics with 100 steps and a step size of 0.1.
as hyperparameters. For the contrastive
The additional results depicted in Figure 15 demonstrate the strong performance energy discrepancy on various toy datasets, consistently yielding accurate energy landscapes. In contrast, contrastive divergence consistently produces flattened energy land- scapes. Despite the success of score matching in these toy examples, score matching struggles to effectively learn distributions with disjoint support which can be seen in the results in Figure 3.
Additional Experimental Results.
Comparison with Denoising Score Matching We further compare ergy discrepancy with denoising score matching (DSM) (Vincent, 2011).
ro: Ta
DC: 7.0.03
retiopend detic
Specifically, we set w = M. = 4 and experiment with various As
shown in Figure 16, DSM fails to
Conutt
work when the noise scale too large or too small. This is because DSM is a biased estimator which is optimised for pa- (y) = s x)Paatall )xx In contrast, energy discrepancy is
Figure 16: Comparing energy discrepancy (ED) with denois- ing score matching (DSM) with different noise scales.
more robust to the choice of t since energy discrepancy considers all noise scales up to simulta-
33

---

Figure 17: Generated images on CelebA 128 128.
neously and has an unique optimum pa- (x) Paata (x). However, in the case that / is large and M small, estimation with energy discrepancy deteriorates due high variance of the estimated loss function. This provides explanation for the superior performance of energy discrepancy at vi compared to vt = 10. Further ablation studies are presented in Figure 23.
D,5 Additional Image Modelling Results
Additional Image Generation and Reconstruction Results. Figures 17 and 18 show additional examples of image generation on CelebA 128 x 128 and image reconstruction on CelebA 64 x 64. The images computed through the sampling process outlined in Appendix C.2.
Additional Image Interpolation and Manipulation Results. Figures 19, 20 and 22 show additional results of image interpolation and manipulation on CelebA 64 x 64. Note that there are two types of interpolations: posterior interpolation and prior interpolation. For posterior interpolation, we consider two real images X1 and x2 from the dataset and perform linear interpolation among their correspouding latent variables 1 ~ Pá.e (z(x1) and Z2 ~ Pa.e (z) (2)x)). For prior interpolation, we apply linear interpolation between Z1 pz(z) and Z2 p& (z)..
Long-run MCMC Diagnostics. Figure 21 depicts several convergence diagnostics for long-run MCMC on the EBM prior, where we simulate Langevin dynamics with large number of steps (2, 000). Firstly, the energy profiles converge at approximately 250 steps, as demonstrated in Fig- ure 21a, and the quality of the synthesized samples improves as the number of steps increases. Secondly, compute the Relman-Rubin statistic i (Gelman & Rubin, 1992) using 64 chains. The histograms of R over 5, 000 x 64 chains are shown in Figure 21b, with mean of 1.08 < 1.20, indicat- ing that the Langevin dynamics have approximately converged. Thirdly, we present auto auro-correlation results in Figure 21c using 000 chains, where the mean is depicted deviation as bands. The auto-correlation decreases to zero within 200 steps, which is consistent with the Gelman-Rubin statistic that assesses convergence across multiple chains.
line and the standard
D.6 Qualitative Results on the Effeec of t, M,, and w
The hyperparameters I, M, play important roles in energy discrepancy. Here, we provide some qualitative results to understand their effects. According to Theorem 2, controls the nearsight- edness of energy discrepancy. For small energy discrepancy behaves like score matching EED,, (pdata U) + f SM(p., U.)ds x SM(panta, U) and expected to be unable to resolve local mixture weights. This assertion can be confirmed by qualitative results depicted in Figure 23, which show that when / = 0.0025, energy discrepancy fails to identify the weights of components in the 25- Gaussians and pinwheel datasets. For large t, energy discrepancy inherits favourable properties
34

---

of the maximum likelihood estimator. While large values of t consequently mitigate problems of nearsightedness, is worth noting that energy discrepancy may encounter issues with high variance when become excessively large. In such situations, it is necessary to consider increasing the value of M reduce the variance.
We also investigate the effect of w in Figure 24. As pointed out by the analysis in Appendix B.I, serves stabilises training of energy based models with energy discrepancy. Based on our experimental observations, when w = 0 and M is small (e.g, M < 128 in the 25-Gaussians dataset and .M Σ the pinwheel dataset), energy discrepancy exhibits rapid divergence within 100 optimisation steps and fails to converge in the end. If, however, w is increased, e.g. to 1, energy discrepancy shows stable convergence even with M 1. This property highly appealing as significantly reduces the computational complexity. Additionally, we find in Figure 2 that larger w tends to result in flatter estimated energy landscapes which aligns with our intuition gained in Appendix B.1.
35

---

3
10
12
0
10
10
7
12
7
16
5
4
7
16
8
5
4
5
O
0
50
)
5
O
000
6
3
6
17
6
3
6
17
86
25
440
10
36
25
3
140
0
10
3
4
4
6
3
4
4
66
3
28
14
58
2
3
2
1
8
14
22
3
22
0
5
17
0
5
17
Figure 18: Qualitative results of reconstruction on test images. Left: real image from the dataset. Right: reconstructed images by sampling from the posterior.
36

---

Figure 19: Linear interpolation results in posterior latent space between real images.
Figure 20: Linear interpolation results in prior latent space between generated images.
-100
Erwrgy Protlee
-200
Conmeation
25
-500
1200 1508 1750
2000 1005 1.03 1:75
1000
1.125 1.30
1:575
1200
420
800
Rerations
Lag
(a) Energy Profile
(b) Gelman-Ruun Rubin
(c) Auto-correllaion
Figure 21: Diagnostics for the mixing of MCMC chains with 2, 000 steps on CelebA 64 x 64. Top: Trajectory in the data space. Bottom: (a) Energy profile over time; (b) Histograms of Gelman-Rubin statistic multiple chains; (c) Auto-correlation of a single chain over time lags.
37

---

(a) Smiling
(b) Male
(c) Eyeglasses
(d) Blond Hair
Figure 22: Attribute manipulation results on CelebA 64 x 64. Each row is made by interpolating the latent variable along an attribute vector, with the middle image being the original image.
38

---

M=11
M=
M-44
M
16
M=:32
M-64
M M=128
M=256
M=512
vt = 0.05
1100
=t= 50.0
M=4
M-16
M=64
M=256
M=517
=
1.00
S
50.
Figure 23: Density estimation on 25- Gausssians and pinwheel with different t, M and
M=7
M=4
M=B
M= 16
M=:37
M=
M= 178
M=756
M=517
w=0.00
w=0.05
00
50.0
M=1
M=2
M=4
M=8
M=16
M=32
M=64
M=128
M=256
M=512
w=0.00
0.00
W=0.05
56
1,00
5050.0
Figure 24: Density estimation 25- Gausssians and pinwheel with different w, M and
39


Reviews:
[{"review_id": "QZx86AArmM", "reviewer": "Reviewer_BMfu", "summary": "In this paper, the authors consider the problem of training energy-based models. Due to the limitations of two current approaches known as approximate maximum likelihood methods (which might lead to malformed estimators of the energy function) and score-based methods (which fail to resolve global features in the distribution without big data), the authors propose a novel loss function for this task named Energy Discrepancy (ED) to overcome those issues with rigorous theoretical guarantee. Moreover, they also conduct several experiments to demonstrate the efficacy of the ED loss function in learning low-dimensional data distributions compared to two previous methods. However, the authors point out that their approach does not work for high-dimensional data due to some manifold hypothesis. ", "questions": "1. In equation (2), why is the score-matching loss defined between $p_{\\text{data}}$ and $E_{\\theta}$ rather than between $p_{\\text{data}}$ and $p_{\\theta}$?\n\n2. In the beginning of section 3, can we use other kernels rather than Gaussian kernels to perturb $p_{\\text{data}}$ and $p_{\\theta}$? \n\n3. In equation (5), is the energy discrepancy $ED_q$ a proper metric?\n\n4. In Theorem 2, the authors should either briefly introduce the definition of Wasserstein distance or cite relevant papers to that distance so that readers from different communities can understand.\n", "limitations": "The limitations are discussed in Section 7 of the paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: Energy Discrepancy (ED) is a novel loss for training enerygy-based models.\n\n2. Quality: Theoretical results are solid and associated with rigorous proof, but I do not spend time double-checking all of them. Additionally, a lot of experiments are carried out to empirically justify the effectiveness of the proposed loss function ED. This significantly strengthens the contributions of the paper. \n\n3. Clarity: The paper is well written and organized, which makes it easy to follow.\n\n4. Significance: Although the ED loss function work for only low-dimensional data distributions, this work lays the basis for alleviating the issues of maximum likelihood methods and score-based methods for training energy-based models.", "weaknesses": "1. Clarity:  There are some places that the authors introduce results without necessary intuitions. For instance, in line 96, they directly suggest using Gaussian kernels without explanation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments. Here is our response to your questions:\n\n> There are some places that the authors introduce results without necessary intuitions. For instance, in line 96, they directly suggest using Gaussian kernels without explanation.\n> \n\n**ANSWER:** Thank you for making us aware of this. The intuition for Gaussian kernels is two-fold: Firstly, Gaussian kernels are the simplest choice as they are easy to sample from and allow easier calculations for the analysis of the method. They are often used to apply noise in score-based methods [1], contrastive methods [2], and as spread noise [3]. Consequently, this is the first perturbation we tried. Secondly, we observed empirically that Gaussian perturbations work well in practice. They allow us to make the connection to score matching which we describe in our motivation in equation (3) as well as maximum likelihood estimation which characterises Gaussian perturbations as optimal among a large class of Markov transition kernels. We will include an additional sentence to motivate our choice of perturbation in line 96 and leave other choices for follow-up work. \n\n> In equation (2), why is the score-matching loss defined between $p_{\\mathrm{data}}$ and $E_\\theta$ rather than between $p_\\mathrm{data}$ and $p_\\theta$?\n> \n\n**ANSWER:** Thank you for pointing out that this notation is unclear. This notation was chosen for the following reasons:\n\n1. Unlike the Fisher divergence from which score matching is derived, the score-matching objective in equation (2) is not a statistical divergence between $p_\\mathrm{data}$ and  $p_\\mathrm{ebm}$, but an estimation criterion for the energy function. For this reason, we treat score-matching as a functional of the learned energy function $U$ or $E_\\theta$, respectively and make this explicit in the notation.\n2. We also reuse the notation in equation (3) where we find this notation more concise than the alternatives.\n3. Finally, this notation is consistent with our notation for Energy Discrepancy, which we treat as a functional of the energy function in the same way.\n\nWe are going to edit the sentence that precedes the equation to make the chosen notation more logical.\n\n> In the beginning of section 3, can we use other kernels rather than Gaussian kernels to perturb $p_\\mathrm{data}$ and $p_\\theta$?\n> \n\n**ANSWER:** The proposed Energy Discrepancy can indeed be defined for other kernel functions. The connection to score matching made at the beginning of section 3, however, holds only for Markov transition kernels associated with SDEs of the form $\\mathrm d\\mathbf x_t = \\mathbf a(\\mathbf x_t)\\mathrm dt + \\mathrm d\\mathbf w_t$.\n\nIn practice, Gaussian kernels are favourable due to easy sampling and many analytical tools available, so we conducted our analysis and experiments for this choice of kernel, first. Additionally, Theorem 2 shows that Gaussian kernels are optimal among possible Markov transition kernels in the sense that the maximum likelihood objective can be approximated with a Gaussian kernel of sufficiently large variance.\n\nOne example of a non-Gaussian kernel for Energy Discrepancy is the Bernoulli perturbation that can be applied in discrete spaces (see section B.3 in the appendix). The additional experimental results in the attached PDF (Figure 1-4) demonstrate that this perturbation is effective in training EBMs in various discrete settings. We will explore other kernels in future work.\n\n> In equation (5), is the energy discrepancy a proper metric?\n> \n\n**ANSWER:** Energy Discrepancy is not a metric but is designed as a criterion for density estimation, similar to maximum likelihood estimation. This means that for Energy Discrepancy, $p_\\mathrm{data}$ is always fixed while $p_\\theta$ is learned and the role of the two arguments can not be interchanged. In particular, ED is not symmetric. When Energy Discrepancy is minimised in $p_\\theta$, the minimum is attained at the ground truth $p_\\theta = p_\\mathrm{data}$. \n\nWhile not a metric, Energy Discrepancy can be related to a statistical divergence called KL contraction, which is a weaker notion of distance. (see section A.6 in the appendix).\n\n> In Theorem 2, the authors should either briefly introduce the definition of Wasserstein distance or cite relevant papers to that distance so that readers from different communities can understand.\n> \n\n**ANSWER:** Thank you for pointing this out, we will refer to [4] in our revision.\n\n[1] Song, Yang and Ermon, Stefano: Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS 2019\n\n[2] Michael Gutmann and Aapo Hyvärinen: Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, AISTATS 2010\n\n[3] Mingtian Zhang, Peter Hayes, Tom Bird, Raza Habib, David Barber: Spread Divergence, ICML 2020\n\n[4] Peyre, Gabrial and Cuturi, Marco: Computational Optimal Transport, Foundations and Trends in Machine Learning 2019"}, {"review_id": "KuTwyQzGM2", "reviewer": "Reviewer_BJCK", "summary": "The authors propose a new loss function for training energy-based models, which they dub \"energy discrepancy.\"  The aim is to provide a viable alternative to contrastive divergence and score matching based methods that suffer from near-sightedness -- these approaches lack global information and can have difficulties fitting well-separated Gaussians.  The energy discrepancy method seeks to overcome this difficulty by perturbing the distributions to increase the mass of the low probability regions separating the peaks of the distribution.  This is done at different noise levels and then integrated.  The proposed loss is theoretically justified and then validated experimentally in both synthetic and real-world settings.", "questions": "-  There is a bit of magic in the w-stabilisation procedure.  I read the explanation in Appendix B, but I feel that more explanation is needed in the main text.  In particular, does this approach generalize beyond the Gaussian case?  This makes me think that the proposed approach is really quite specific (at least if you want to be able to do it in practice).    Does this need to be included as a limitation?\n\n-  Is it possible to include some discrete setting experiments as well?  \n\nMinor typos:\n\n- \"normalisation of EBMs, also known as partition function\" sounds funny.  Maybe \"normalisation constant\"?\n- The sentence in lines 30-33 doesn't really make sense.\n- \"i.e.\" and \"e.g.\" must always be followed by a comma.\n- \"line out in\" -> \"outline in\"?\n- Lots of \"which\" usage that I think requires a comma.\n\n", "limitations": "Yes, but see above comments.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach is well-motivated for the most part and the presentation is mostly clear.  The idea is novel to me and experimentally improves over existing approaches.  ", "weaknesses": "I'm not certain that the approach is potentially as practical as it appears.  My main concern is that the w-stabilisation procedure seems to be doing a lot of heavy lifting and that is tailored to the Gaussian case.  Given other discussion in this work, I don't think this is a huge problem.  However, the case for acceptance would be bolstered if a broader set of applications were considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your valuable comments. Your suggestions are very helpful in further improving the work, and we are refining the manuscript accordingly.\n\n> My main concern is that the w-stabilisation procedure seems to be doing a lot of heavy lifting and that is tailored to the Gaussian case. Given other discussion in this work, I don't think this is a huge problem. However, the case for acceptance would be bolstered if a broader set of applications were considered.\n> \n\n**ANSWER:** Thank you for raising this concern. The $w$-stabilisation is critical to scale up our experiments to high dimensions and significantly reduce the computational cost. The stabilisation is, however, not restricted to the Gaussian case. To support our case, we have added experiments using a Bernoulli perturbation in various discrete settings. We observe a similar effect of the $w$-stabilisation in discrete spaces (see Figure 4 in the attached pdf).\n\nAdditionally, it is possible to avoid stabilisation in certain use cases. For example, it is feasible to estimate densities using a large value for $M$ such as $M=512$ in low-dimensional data (see e.g. figure 21 in the appendix). The stabilisation is also not needed if the energy function is sufficiently regular. This typically applies for models that are not deep architectures and that have interpretable parameters, such as products of Gaussian experts with finite variance.\n\n> There is a bit of magic in the w-stabilisation procedure. I read the explanation in Appendix B, but I feel that more explanation is needed in the main text. In particular, does this approach generalize beyond the Gaussian case? This makes me think that the proposed approach is really quite specific (at least if you want to be able to do it in practice). Does this need to be included as a limitation?\n> \n\n**ANSWER:** Thanks for the comment. We will extend the main text to make the w-stabilisation more intuitive. The w-stabilisation can be generalised beyond the Gaussian case and we choose the same stabilisation successfully in the discrete setting which uses a Bernoulli perturbation. We included a study on the effect of w for the Bernoulli perturbation in Figure 4 on the attached PDF. The main idea behind the stabilisation is the following:\n\nIn theory, the difference between the positive energy term and the contrastive energy term is bounded, and thus Energy Discrepancy has an existing minimum at $\\exp(-U^\\ast)\\propto p_\\mathrm{data}$. In practice, Energy Discrepancy requires approximating the contrastive energy term with samples, which we call “contrastive samples”.  At the edge of the data support, it can happen at random that all contrastive samples have high energy. The resulting parameter gradient encourages energy landscapes for which the energies of contrastive samples go to infinity. We visualise this in Figure 2 in our paper. The w-stabilisation effectively adds the unperturbed data point to the set of contrastive samples. If all contrastive samples have unusually high energies, the log-sum-exp operation is dominated by the energy of the unperturbed data point which does not blow up because it is minimised in the non-contrastive term at the same time. This heuristic argument applies to all perturbations.\n\nWe agree that further theoretical work on the effect of the w-stabilisation and its optimal value would be of great interest and it is conceivable that different stabilisation procedures could improve the effectiveness of our method. However, at this stage of the work, the w-stabilisation has several big advantages:\n\n- It is simple, adds no computational complexity to the optimisation, and there is numerical evidence for its effectiveness\n- The $w$-parameter requires little tuning, and experimental results are of similar quality for all values that are of the order of 1.\n- It is theoretically justified by Theorem 3, which we have generalised to other perturbations in the meantime\n\nWe hope we could address your concerns and are happy to discuss further.\n\n> Is it possible to include some discrete setting experiments as well?\n> \n\n**ANSWER:** We have included new experiments on discrete data using the Bernoulli perturbation in the attached PDF, following the setting in [1]. The experiments are conducted on predicting the connectivity matrix of an Ising model (Figure 1), learning the density of 2D data mapped via grey codes to the discrete space $\\\\{0, 1\\\\}^{32}$ (Figure 2), as well as binary MNIST, Polyglot, and Silhouettes data sets (Figure 3). It is noteworthy that there were no latent variable models used in the discrete image modelling while achieving competitive results. We leave it as future work to find suitable perturbations for other structured data like graphs and text.\n\n[1] Dinghuai Zhang et al.: Generative Flow Networks for Discrete Probabilistic Modeling, ICML 2022"}, {"review_id": "BRgkuTpFL6", "reviewer": "Reviewer_NEwe", "summary": "This paper proposes a new loss function for training energy-based models, called energy discrepancy (ED). ED does not rely on score functions and MCMC samples. Instead, it is defined as the difference between the energy function of data and some conditional samples. They prove that optimizing this objective function yields the appropriate energy function. Then, they build connections between ED, score matching and contrastive divergence. Finally, this paper focuses on training the latent energy-based prior model, which is a VAE with EBM prior.\n\nFor experiments, the authors first showcase the density estimation on several 2D pdfs. ED outperforms SM and CD on pdf estimation and sampling. Then, they train latent EBMs on SVHN, CIFAR-10 and CelebA. ED also outperforms SM and CD on image reconstruction/generating and out-of-distribution detection.\n", "questions": "1. In the Equation between Line 187 and 188, is there $E_\\theta(x^i)$ missing behind $w / M$?", "limitations": "None", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Training EBMs is an important problem in generative modeling. This paper proposes a new training criteria, which does not rely on possibly ill-conditioned score functions and time-consuming MCMC samples.\n\n2. The authors also build connections between the proposed method and score-matching estimates or MLEs.\n", "weaknesses": "1. The authors do not train EBMs directly on the data space. Instead, they train a VAE with a EBM prior. Since CD or SM can lead to competitive EBMs, I am wondering if there is any empirical result on training EBMs directly on the data space using the proposed loss?\n\n2. Since the proposed loss function is similar to the conditional NCE, I think one promising baseline could be training the same model using CNCE loss.\n\n3. The authors said ED provides fast training (in Line 338). Is there any comparison about the running time?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback! Here are our responses to the mentioned weaknesses and questions:\n\n**Weaknesses:**\n\n> The authors do not train EBMs directly on the data space. Instead, they train a VAE with a EBM prior. Since CD or SM can lead to competitive EBMs, I am wondering if there is any empirical result on training EBMs directly on the data space using the proposed loss?\n> \n\n**ANSWER:** In our density estimation experiments, the energy-based models are trained on data space, and Energy Discrepancy outperforms CD or SM in this domain. Furthermore, we have new empirical results for discrete EBMs trained directly on pixel space for various binary image data sets (See Figure 3 in the attached PDF).\n\nThe VAE approach is used in our image modelling experiments, only. The so-called manifold hypothesis suggests that most image data lives on a lower-dimensional subset of the higher dimensional pixel space. As a consequence of this, the energy function outside of the manifold is undefined, i.e. for a noisy data point, we have $p_\\mathrm{data}(\\tilde{\\mathbf x}^i)=0$ which implies for the learned energy that $E_\\theta(\\tilde{\\mathbf x}^i) \\approx -\\log p_\\mathrm{data}(\\tilde{\\mathbf x}^i) \\to \\infty$. For this reason, we use a latent variable model to construct representations of the data that are supported in the whole latent space. The probability of a noisy latent representation is still positive $p_z(\\tilde{\\mathbf{z}}^i)>0$ and we can learn a well-defined energy.\n\nContrastive divergence is not as sensitive to this problem because it only pulls up low-energy states, and the value of the energy function stays bounded. However, this comes at the cost of biased energy landscapes as shown in Figure 3, which pose difficulties in likelihood-based tasks like anomaly detection as shown in Table 2. Furthermore, sampling from the learned energies remains challenging.\n\nScore-based generative models normally require the use of multiple noise scales [1, 2] to alleviate issues with its nearsightedness and scores being undefined outside the data manifold. Training EBMs with score matching directly on data space without annealing faces challenges similar to the ones outlined for Energy Discrepancy with the additional difficulty of SMs nearsightedness.\n\n> Since the proposed loss function is similar to the conditional NCE, I think one promising baseline could be training the same model using CNCE loss.\n> \n\n**ANSWER:** This is a great suggestion. In fact, in the special case of Gaussian perturbations with one contrastive sample ($M= 1$ in ED, $\\kappa = 1$ in CNCE [3]), and $w= 1$ , the two approaches coincide.\n\nTo further explore CNCE as a baseline, we conducted experiments on the 25-gaussian dataset for different choices of $M$ ($\\kappa$ in CNCE) and different choices for $t$ ($0.5\\epsilon^2$ in CNCE). Our results (see Figure 6 in the attached pdf) show that the performance of CNCE is comparable to Energy Discrepancy. On image data, we expect that CNCE performs similarly to Energy Discrepancy and that many of the techniques we use for ED like the latent EBM would also improve the experimental results for CNCE. We believe that both, CNCE as well as our proposed loss function are promising approaches for fast and accurate training of EBMs.\n\nConceptually, Energy Discrepancy has the appeal of being closely connected to the original maximum likelihood estimation problem, which has the lowest asymptotic variance amongst all unbiased estimators. For this reason, Energy Discrepancy may be more data efficient if sufficient compute is available. Additionally, CNCE requires tractable likelihoods of the perturbation, while Energy Discrepancy only requires being able to simulate the perturbation. Thus, ED is applicable to different non-Gaussian perturbations in which the likelihood is intractable.\n\n> The authors said ED provides fast training (in Line 338). Is there any comparison about the running time?\n> \n\n**ANSWER:** The statement is based on complexity analysis. Energy Discrepancy requires $\\mathcal O(M)$ evaluations of the energy net, while SM and CD methods require at least $\\mathcal O(d)$ evaluations of the energy net to compute the gradient, where $d$ denotes the dimension.\n\nFor the density estimation experiment, one step of the optimiser for ED, CD, and SM takes 0.006s, 0.018s, and 0.021s, respectively. Additionally, ED converges more quickly as shown in Figure 4 in our paper. The resulting run time can be compared in Figure 5 in the attached pdf. In the image modelling experiments, ED reduces the computational cost significantly as $M=16$ is used throughout all image modelling experiments irrespective of the dimension.\n\n**Questions:**\n\n> In the Equation between Line 187 and 188, is there $E_\\theta(x^i)$ missing behind $w/M$?\n> \n\n**ANSWER:** In this equation, the energy term behind the $w/M$ cancels with the non-contrastive energy term, i.e. for each $\\mathbf x^i$ we write the energy as $E_\\theta(\\mathbf x^i) = \\log \\exp(E_\\theta(\\mathbf x^i))$ and use $\\log(a)+\\log(b) = \\log(ab)$ to take the logarithm outside of all other operations. This gives for each sample in the batch:\n\n$E_\\theta(\\mathbf{x}^i) + \\log\\left(\\frac{w}{M}\\exp(-E_\\theta(\\mathbf{x}^i)) + \\frac{1}{M}\\sum_{j=1}^M \\exp(-E_\\theta(\\tilde{\\mathbf{x}}^{i, j}) \\right) = \\log\\left(\\frac{w}{M} + \\frac{1}{M}\\sum_{j=1}^M \\exp(E_\\theta(\\mathbf x^i)-E_\\theta(\\tilde{\\mathbf{x}}^{i, j})\\right)$\n\nWe will add the identity to the appendix and hope this makes things clearer.\n\n[1] Song, Yang and Ermon, Stefano: Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS 2019\n\n[2] Zengyi Li, Yubei Chen, Friedrich T. Sommer: Learning Energy-Based Models in High-Dimensional Spaces with Multi-scale Denoising Score Matching, 2019\n\n[3] Ceylan, Ciwan, and Michael U. Gutmann. \"Conditional noise-contrastive estimation of unnormalised models.\" *ICML*, 2018."}, {"review_id": "VWI7JJliBt", "reviewer": "Reviewer_M9yD", "summary": "This paper has proposed a new loss funciton, i.e., Energy Discrepancy to train energy-based models without computaiton of MCMC. The proposed loss function could be directly derived from the energy function without relying on MCMC samples.", "questions": "1. Model in [1] utlizes MCMC-based maximum likelihood algorithm to train a normalizing flow model for image generation. Could the proposed energy discrepancy used in this model?\n2. According to Table 1, the performance of CD-LEBM and the proposed ED-LEBM are quite closed. The gap is much smaller than the experiments in density estimation.\n3. Are there any comparison with GAN-based models?\n4. For comparison with score-based methods, [2] shows a much better performance in FID on Cifar10. Could the author give some discussion about these results as the author has claimed that the proposed energy discrepancy has advantages of score matching?\n\nReference:\n\n[1] Xie, Jianwen, et al. \"A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference.\" arXiv preprint arXiv:2301.09300 (2023).\n\n[2] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" arXiv preprint arXiv:2011.13456 (2020).", "limitations": "Please refer to the question section.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The proposed energy discrepancy could be directly computed from the energy function and alleviate the problem of nearsightedness of score matching and approximates maximum-likelihood estimation. The experiments show that the energy discrepancy could achieve better performance than score matching and contrastive divergence in image generation.", "weaknesses": "I think this is very good paper that proposes a novel optimization method in enery-based models. I only have several questions about the experiment part. Pleare refer to the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments and helpful suggestions! Our answers are listed below.\n\n> Model in [1] utlizes MCMC-based maximum likelihood algorithm to train a normalizing flow model for image generation. Could the proposed energy discrepancy used in this model?\n> \n\n**ANSWER:** Thank you for the interesting reference. The Model in [1] works analogously to the latent EBM [3] used for image data in our work, with the EBM prior being replaced with a normalising flow prior. Since the normalising flow has a tractable likelihood, the maximum likelihood training is possible without training approaches for EBMs like CD, SM, and Energy Discrepancy. However, [1] requires MCMC sampling from the posterior $p(\\mathbf z \\vert \\mathbf x)$ to generate latent representations of data, and this step is necessary in our work as well.\n\nIt is an exciting avenue for future research to use a normalising flow as the base distribution of an EBM [4], which would enable training of EBMs with Energy Discrepancy without the need of a latent variable model or MCMC.\n\n> According to Table 1, the performance of CD-LEBM and the proposed ED-LEBM are quite closed. The gap is much smaller than the experiments in density estimation.\n> \n\n**ANSWER:** This is true and we can currently only make assumptions why this is the case. Most likely, the reason lies in the latent variable model used in both methods: \n\nIn the density estimation experiments, we observed that CD learns energy landscapes that are biased towards being too smooth. In the image modelling experiments, the latent representations obtained by sampling from $p(\\mathbf z\\vert \\mathbf x)$ may be fairly noisy which corresponds to smooth energy landscapes that can be learned equally well by CD and ED. Flat energy landscapes are also suggested by the interpolation results which show that recognisable images can be produced from midpoints between two latent representations.\n\nAdditionally, it is noteworthy that the results are compared in different metrics. In our density estimation results we are comparing the MSE between the estimated density and ground truth density. For the image modeling results, we are comparing the FID of images, which not only measures the quality of the learned EBM prior but also that of the decoder. We do not have access to the true prior to assess the accuracy of the learned EBM and to what extent the decoder compensates for poorly learned energies.\n\nWe would like to point out that Energy Discrepancy requires significantly fewer computations per iteration to produce comparable results. Furthermore, we hope that we can improve our experimental results in the future by finding alternatives to the latent EBM model.\n\n> Are there any comparison with GAN-based models?\n> \n\n**ANSWER:** Neither VAEs nor EBMs can currently outperform GAN-based models if it comes to image generation. The appeal of EBMs lies in the fact that they don’t just generate data but also encode data into a probability distribution. An exciting research direction, however, are Generalised EBMs [5] which combine EBMs with GANs and outperform both methods. Exploring new perturbation strategies to train Generalised EBMs with Energy Discrepancy holds promise for future work.\n\n> For comparison with score-based methods, [2] shows a much better performance in FID on Cifar10. Could the author give some discussion about these results as the author has claimed that the proposed energy discrepancy has advantages of score matching?\n> \n\n**ANSWER:** The results in [2] are possible because the score-based method in [2] trains a diffusion model, i.e. the work uses an annealing scheme to learn the scores of data at various noise scales. This achieves two things: Firstly, the annealing scheme makes score matching aware of other modes in the distribution and the scores in low-density areas are estimated accurately. Secondly, the approach learns a sampler as part of the model, which enables a high sample quality. However, score-based diffusion models do not learn an energy-based model or a density. Energy-based models, on the other hand, learn the data generating density but are hard to sample from, resulting in lower FID scores compared to diffusion models.\n\nTraining energy-based models with score matching is challenging because the estimated scores are only accurate in the vicinity of modes of the data distribution. [6] elaborates these difficulties of score matching and how they influenced the development of score-based diffusion models ([2, 6]). Energy Discrepancy alleviates some of these difficulties by implicitly incorporating data information at different noise scales since the Gaussian Energy Discrepancy is equivalent to a multi-noise-scale score matching loss.\n\n[1] Xie, Jianwen, et al. \"A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference.\" arXiv preprint arXiv:2301.09300 (2023).\n\n[2] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" arXiv preprint arXiv:2011.13456 (2020)\n\n[3] Pang, Bo et al. \"Learning Latent Space Energy-Based Prior Model.\" NeurIPS, 2020\n\n[4] Nijkamp, Erik, et al. \"MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMC\" ICLR, 2022.\n\n[5] Arbel, Michael, Liang Zhou, and Arthur Gretton: Generalized energy based models, ICLR 2021.\n\n[6] Song, Yang and Ermon, Stefano: Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS 2019"}, {"review_id": "7p1U6wyBk5", "reviewer": "Reviewer_37sQ", "summary": "Energy discrepancy is presented as a new loss for the training of EBM. \nED interpolates between the losses of score matching and maximum-likelihood estimation.\nEfficacy of ED on a latent variable energy-based model is demonstrated to tackle manifold hypothesis as an important challenge in the adoption of likelihood-based training.\nExtensive numerical experiments are done to show ED's superiority over contrast divergence and score matching.", "questions": "How important is the w-stabilization term for the numerical experiments?  If w is small(close to 0), are numerical experiments results still hold?", "limitations": "No negative social impact is seen.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Theoretical derivation is rigorous.\n2. Numerical experiments are solid.\n3. Paper is in-general well written.\n4. The proposed algorithm is easy to implement.\n", "weaknesses": "No obvious weakness to me. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive and helpful comments. Here is our answer regarding the w-stabilisation:\n\n> How important is the w-stabilization term for the numerical experiments? If w is small(close to 0), are numerical experiment results still hold?\n> \n\n**ANSWER:** We found that w-stabilisation is important to make Energy Discrepancy scalable. Initially, Energy Discrepancy was developed without this stabilisation and worked well on two-dimensional data when $M= 256$ contrastive samples were used. With the w-stabilisation, however, we were able to reduce the number of contrastive samples to $M=1$ in two-dimensional settings and learn high-dimensional distributions as in our image modelling experiments using $M=16$. A study on the effect of $w$ and $M$ can be found in Appendix D.5. in Figure 21.\n\nIn general, we found that our experimental results are robust to the choice of $w$ and the parameter requires no fine tuning. The choice becomes less consequential the larger $M$ is chosen. Large values for $w$ encourage smoother energy landscapes. Small values of $w$ may underestimate the variance of the distribution if $M$ is chosen too small, but typically stabilise the training sufficiently to produce good results. Since $w$ adds no computational complexity and corrects the approximation bias of the contrastive potential, we found that it is reasonable to choose $w=1$, which appears to work well in all settings we investigated."}]

Author rebuttal:
We thank all reviewers for their constructive and extensive comments that help us to improve this work.

We would first like to summarise the paper according to the reviewers:

- Our work proposes a new practical, easy to implement, and fast training technique for energy-based models that does not require MCMC sampling. Reviews agree that this approach is novel:
    
    > **BMfu:** “Energy Discrepancy (ED) is a novel loss for training enerygy-based models.”; **BJCK:** “The idea is novel to me and experimentally improves over existing approaches.”; **NEwe:** “This paper proposes a new training criteria, which does not rely on possibly ill-conditioned score functions and time-consuming MCMC samples.”; **M9yD:** “I think this is very good paper that proposes a novel optimization method in enery-based models.”; **37sQ:** “The proposed algorithm is easy to implement.”
    > 
    
    Furthermore, the reviewers agree that the contribution is significant:
    
    > **BMfu:** “Although the ED loss function work for only low-dimensional data distributions, this work lays the basis for alleviating the issues of maximum likelihood methods and score-based methods for training energy-based models.”; **M9yD:** “The experiments show that the energy discrepancy could achieve better performance than score matching and contrastive divergence in image generation.”
    > 
- Our work introduces theoretical guarantees demonstrating the validity of our approach. Furthermore, we draw connections to score matching and maximum likelihood estimation. The reviews reflect the soundness of our approach as follows:
    
    > **NEwe:** “The authors also build connections between the proposed method and score-matching estimates or MLEs.”; **37sQ:** “ED interpolates between the losses of score matching and maximum-likelihood estimation. Theoretical derivation is rigorous.”
    > 

We now summarise the concerns that were raised most frequently by reviewers and explain how we addressed them in our rebuttal:

- **Experiments on discrete spaces**
    
    Reviewer ****BJCK**** is interested in additional experiments on discrete spaces. We added three new experiments to the attached pdf regarding different discrete settings: Using an Energy Discrepancy based on a Bernoulli perturbation, we learn the connectivity matrix of an Ising model, two dimensional densities that are mapped into $\\{0, 1\\}^{32}$ via grey codes, as well as various image data sets with binary pixel values (MNIST, Polyglot, Silhouettes). Our results are competitive at a low computational cost.
    
    Reviewer ****NEwe**** is interested in experiments on data space, directly. We would like to refer to the new experiments on discrete data (Figure 1, 2, 3 on the attached PDF) to demonstrate that Energy Discrepancy is capable of learning high-dimensional distributions on pixel space, directly. However, we also discussed why we think that for contrastive learning methods like ours, latent variables or other types of hybrid models are necessary to model many image data sets due to the manifold hypothesis.
    
- **Concerns regarding the w-stabilisation:**
    
    Reviewer ****BJCK**** is concerned that the w-stabilisation only applies to the Gaussian case and that more motivation is needed in the main text. To respond to this concern, we are going to add a new paragraph to the main paper that motivates the w-stabilisation and how it applies to **any** type of perturbation. The main idea of the stabilisation is to control the variance of a log-sum-exp operation in the sample approximated loss functional. To support our case that this stabilisation applies to other perturbations, we have added ablation studies (see Figure 4 in the attached PDF) comparing empirical results with and without $w$ in the case of a Bernoulli perturbation.
    
    Reviewer ****37sQ**** is interested if Energy Discrepancy works if the stabilisation term goes to zero. Here, we would like to refer to the appendix of our paper. Figure 21 shows a comparison of learned energies for various choices of $w$ and $M$. One can see that Energy Discrepancy works for all values of $w$ if the number of perturbed samples $M$ is large. Introducing even a very small stabilisation $w$ reduces the number of required samples drastically.


Meta-review: This paper proposes a new loss function for learning energy-based models that can be evaluated without computing scores or running long MCMC chains. Experiments show that the proposed method is promising. The reviewers all recognize the contribution as novel and substantial. All reviewers vote in favor of accepting this work and indicate that their concerns have been sufficiently addressed during author discussion. I think the proposed method is an interesting addition to the generative modeling toolbox and I am happy to support accepting this work.
Decision: Accept (poster)