Title: Energy Discrepancies: A Score-Independent Loss for Energy-Based Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Energy-based models

statistical discrepancy

latent-variable model

density estimation
Abstract: Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that energy discrepancy approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum energy discrepancy estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.

Energy Discrepancies: A Score-Independent Loss for Energy-Based Models

Abstract

Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that energy discrepancy approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum energy discrepancy estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.

1 Introduction

Energy-Based Models (EBMs) are a class of parametric unnormalised probabilistic models of the general form p ebm ∝ exp( -U ) originally inspired by statistical physics. EBMs can be flexibly modelled through a wide range of neural network functions which, in principle, permit the modelling of any positive probability density. Through sampling and inference on the learned energy function, the EBM can then be used as a generative model or in numerous other downstream tasks such as improving robustness in classification or anomaly detection (Grathwohl et al., 2020), simulation-based inference (Glaser et al., 2022), or learning neural set functions (Ou et al., 2022).

Despite their flexibility, EBMs are limited in machine learning applications by the difficulty of their training. The normalisation constant of EBMs, also known as the partition function, is typically intractable making standard techniques such as Maximum Likelihood Estimation (MLE) infeasible. For this reason, EBMs are commonly trained with an approximate maximum likelihood method called Contrastive Divergence (CD) (Hinton, 2002) which approximates the gradient of the log-likelihood using short runs of a Markov chain Monte Carlo (MCMC) method. However, contrastive divergence with short run MCMC leads to malformed estimators of the energy function (Nijkamp et al., 2020a), even for relatively simple restricted Boltzmann-machines (Carreira-Perpiñán & Hinton, 2005). This can, in part, be attributed to the fact that contrastive divergence is not the gradient of any fixed

objective function (Sutskever & Tieleman, 2010), which severely limits the theoretical understanding of CD and motivated various adjustments of the algorithm (Du et al., 2021; Yair & Michaeli, 2021).

Score-based methods such as Score Matching (SM) (Hyvärinen & Dayan, 2005; Vincent, 2011; Song et al., 2020) and Kernel Stein Discrepancy (KSD) estimation (Liu et al., 2016; Chwialkowski et al., 2016; Gorham & Mackey, 2017; Barp et al., 2019) are a family of competing approaches which offer tractable loss functions and are, by construction, independent of the normalising constant of the distribution. However, such methods suffer from nearsightedness as they fail to resolve global features in the distribution without vast amounts of data. In particular, both SM and KSD estimators are unable to capture the mixture weights of two well-separated Gaussians (Song & Ermon, 2019; Zhang et al., 2022; Liu et al., 2023).

We propose a new loss functional for energy-based models called Energy Discrepancy (ED) which compares the data distribution and the energy-based model via two contrasting energy contributions. By definition, energy discrepancy only depends on the energy function and is independent of the scores or MCMC samples from the energy-based model. In our theoretical section, we show that this leads to a loss functional that can be defined on general measure spaces without Euclidean structure and demonstrate its close connection to score matching and maximum likelihood estimation in the Euclidean case. In our practical section, we focus on a simple implementation of energy discrepancy on Euclidean space which requires less evaluations of the energy-based model than the parameter update of contrastive divergence or score-matching. We demonstrate that the Euclidean energy-discrepancy alleviates the problem of nearsightedness of score matching and approximates maximum-likelihood estimation with better theoretical guarantees than contrastive divergence.

On high-dimensional image data, energy-based models face the additional challenge that under the manifold hypothesis (Bengio et al., 2013), the data distribution is not a positive probability density and does, strictly speaking, not permit a representation of the form of an energy-based model. Energy discrepancy is particularly sensitive to singular data supports and requires the transformation of the data distribution to a positive density. We approach this problem by training latent energy-based priors (Pang et al., 2020) which employ a lower-dimensional latent representation in which the data distribution is positive.

Our contributions are the following: 1) We present energy discrepancy, a new estimation loss for the training of energy-based models that can be computed without MCMC or spatial gradients of the energy function. 2) We show that, as a loss function, ED interpolates between the losses of score matching and maximum-likelihood estimation and overcomes the nearsightedness of score-based methods. 3) We introduce a novel variance reduction trick called w -stabilisation that drastically reduces the computational cost of approximating energy discrepancy stably.

2 Training of Energy-Based Models

In the following, let p data ( x ) be an unknown data distribution which we are trying to estimate from independently distributed data { x i } ∼ p data . Energy-based models are parametric distributions of the form

for which we want to find the scalar energy function E θ such that p θ ≈ p data . Typically, energy-based models are trained with contrastive divergence which estimates the gradient of the log-likelihood

using Markov chain Monte Carlo methods to approximate the expectation for p θ . For computational efficiency, the Markov chain is only run for a small number of steps. As a result, contrastive divergence does not learn the maximum-likelihood estimator and can produce malformed estimates of the energy function (Nijkamp et al., 2020a).

Alternatively, the discrepancy between data distribution and energy-based model can be measured by comparing their score functions ∇ x log p data and ∇ x log p θ which, by definition, are independent of the normalising constant. The comparison of the scores is achieved with the Fisher divergence . After applying an integration-by-parts and discarding constants with respect to θ , this leads to the score-matching loss (Hyvärinen & Dayan, 2005)

As this only requires the computation of expectations with respect to p data , the requirement that the data distribution attains a density can be relaxed, yielding a loss function for p θ which can be readily approximated. Score-based methods are nearsighted as the score function only contributes local information to the loss. In a mixture of well-separated distributions, the score matching loss decomposes into a sum of objective functions that only see the local mode and are not capable of resolving the weights of the mixture components (Song & Ermon, 2019; Zhang et al., 2022).

3 Energy Discrepancies

To illustrate the idea behind the proposed objective function, we start by motivating energy discrepancy from the perspective of explicit score matching. In the following, we will denote the EBM as p ebm ∝ exp( -U ) , where U is the energy function that is learned. The nearsightedness of score matching arises due to the presence of large regions of low probability which are separating the modes of the data distribution. To increase the probability mass in these regions, we follow (Zhang et al., 2020) and perturb p data and p ebm through a convolution with a Gaussian kernel γ s ( y -x ) ∝ exp( -∥ y -x ∥ 2 / 2 s ) , i.e.

0.0

0.2

0.4

0.6

0.8

1.0

Mixture weight

0.00

0.02

0.04

0.06

0.08

0.10

Loss (SM / MLE/ ED)

SM

MLE

EDt=2

EDt=4

EDt=8

EDt=32

The resulting perturbed divergence ˜ SM( p data , U ) := SM( p s , U s ) retains its unique optimum at exp( -U ∗ ) ∝ p data (Zhang et al., 2020) but alleviates the nearsightedness as the data distribution is more spread out. The perturbation with γ s simultaneously makes the two distributions more similar which comes at a potential loss of discriminatory power. We mitigate this by integrating the score matching objectives over s over an interval of noise-scales [0 , t ] . It turns out that this integral can be evaluated as the difference of two contrasting energy-contributions:

The proof is given in Appendix A.3. The contrasting expression on the right-hand is now independent of the score and normalisation of the EBM. We argue that such constructed objective functions are useful losses for energy-based modelling.

3.1 A contrastive approach to learning the energy

We lift the idea of learning the energy-based distribution through the contrast of two energycontributions to a general estimation loss called Energy Discrepancy . We will show that energy discrepancy can be defined independently of an underlying perturbative process and is well-posed even on non-Euclidean measure-spaces:

Definition 1 (Energy Discrepancy) . Let p data be a positive density on a measure space ( X , d x ) 1 and let q ( ·| x ) be a conditional probability density. Define the contrastive potential induced by q as

We define the energy discrepancy between p data and U induced by q as

In this paper, we shall largely focus on the case where the data is Euclidean, i.e. X = R d , and the base-distribution d x is the standard Lebesgue measure. However, this framework also admits X being discrete spaces like spaces of graphs, or continuous spaces with non-trivial base measures such as manifolds. Specifically, the validity of our approach is characterised by the following non-parametric estimation result:

Theorem 1. Let p data be a positive probability density on ( X , d x ), and let q ( ·| x ) be a conditional probability density. Under mild assumptions on q and the set of admissible energy functions U , energy discrepancy ED q is functionally convex in U and has, up to additive constants, a unique global minimiser U ∗ = arg min ED q ( p data , U ) with exp( -U ∗ ) ∝ p data .

The assumption on the conditional density q describes that y ∼ q ( ·| x ) involves some loss of information. The assumption that p data has full support turns out to be critical when scaling energydiscrepancy to high-dimensional data. The proof of Theorem 1 is given in Appendix A.1.

3.2 Choices for the conditional distribution q

Def. 1 offers a wide range of possible choices for the perturbation distribution q . In Appendix D.3 we discuss a possible choice in the discrete space { 0 , 1 } d . For the remainder of this paper, however, we will focus on Euclidean data x ∈ R d and hope that the generality of our result inspires future work. Our requirements on q are that simulating from the conditional distribution y ∼ q ( y | x ) and computing the convolution that defines U q are numerically tractable. On continuous spaces, a natural candidate for q is the transition density of a diffusion process which arises as solution to a stochastic differential equations of the form d x t = a ( x t )d t +d w t with drift a and standard Brownian motion w t (see Øksendal, 2003). The conditional density q t ( ·| x ) represents the probability density of the perturbed particle x t that was initialised at x 0 = x . The resulting transition density then satisfies both of our requirements by employing the Feynman-Kac formula as we line out in Appendix B.2. Although this approach makes the choice of q flexible, the following interpolation result stresses that not much is lost when choosing a Gaussian transition density γ t ∝ exp( -∥ x -y ∥ 2 / 2 t ) :

Theorem 2. Let q t be the transition density of the diffusion process d x t = a ( x t )d t + d w t , let p ebm ∝ exp( -U ) be the energy-based distribution and p t the data-distribution convolved with q t .

The energy discrepancy is given by a multi-noise scale score matching loss

If a = 0 , i.e. q t is the Gaussian transition density γ t , the energy discrepancy converges to the loss of maximum likelihood estimation a linear rate in time

where c ( t ) is independent of U and W 2 denotes the Wasserstein distance 2 .

Theorem 2 has two main messages: All diffusion-based energy discrepancies behave like a multi-noise scale score matching loss, which is independent of the drift a . In fact, we show in Appendix A.2 that for a linear drift α x t , the induced energy discrepancy is always equivalent to the energy discrepancy based on a Gaussian perturbation. Furthermore, estimation with a Gaussian-based energy discrepancy approximates maximum likelihood estimation for large t , thus enjoying its attractive asymptotic properties provided ED γ t ( p data , U ) can be approximated with low variance. We demonstrate the result in a mixture model in Figure 1 and Appendix D.1 and give a proof of above theorem in Appendices A.3 and A.4.

Connection to Contrastive Divergence. Due to the generality of our result we can also make a direct connection between energy discrepancy and contrastive divergence. To this end, suppose that for θ fixed, q satisfies the detailed balance relation q ( y | x ) exp( -E θ ( x )) = q ( x | y ) exp( -E θ ( y )) . In this case, energy discrepancy becomes the loss function

which, after taking gradients in θ yields the contrastive divergence update 3 . See Appendix A.5 for details. The non-parametric estimation result from Theorem 1 holds true for the contrastive objective in (6). This means that each step of contrastive divergence optimises an objective function with minimum at p data ≈ p θ . However, the objective function is adjusted in each step of the algorithm.

4 Training Energy-Based Models with Energy Discrepancy

In sight of Theorem 2, we will discuss how to approximate ED from samples for Euclidean data { x i } ⊂ R d and a Gaussian conditional distribution γ t ( y -x ) ∝ exp( -∥ y -x ∥ 2 / 2 t ) . First, the outer expectations on the right-hand of (5) can be computed as plug-in estimators by simulating the Gaussian perturbation x i t = x i + √ t ξ for ξ ∼ N (0 , I ) and averaging { U ( x i ) } and { U γ t ( x i t ) } . The critical step is then finding a low-variance estimate of the contrastive potential U γ t itself.

Due to the symmetry of the Gaussian transition density, we can interpret γ t as the law of a Gaussian random variable with mean x i t and variance t , i.e. the conditioned random variable x i t + √ t ξ ′ | x i t for ξ ′ ∼ N (0 , I ) has the density γ t ( x i t -x ) . Hence, we can write U γ t as the expectation

for i = 1 , 2 , . . . , N . The conditioning expresses that we keep x i t fixed when taking the expectation with respect to ξ ′ . It is then possible to calculate the expectation by sampling ξ ′ i,j ∼ N (0 , I ) and calculating the mean as for the outer expectations. However, we find that this approximation is not sufficient as it is biased due to the logarithm and prone to numerical instabilities because of missing bounds on the value of U γ t ( x i t ) . To stabilise training, we augment the Monte Carlo estimate of U γ t ( x i t ) by an additional term w/M exp( -U ( x i )) which we call

3

2

1

0

1

2

3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

estimated energy

w = 0.0

w = 0.1

w = 1.0

w = 4.0

x

w -stabilisation. This results in the following approximation of the contrastive potential:

The w -stabilisation dampens the contributions of contrastive samples x i t + √ t ξ ′ i,j whose energy is higher than the energy of the data point x i . This provides a deterministic upper bound for the approximate contrastive potential in (4) and reduces the variance of the estimation. We find that the w -stabilisation drastically reduces the number of samples M needed for the stable training of deep energy-based models. We illustrate the effect of the stabilisation in Figures 2 and 24 and discuss our reasoning in more details in Appendix B.1. The full loss is now formed for U := E θ with ξ i ∼ N (0 , I ) , ξ ′ i,j ∼ N (0 , I ) and tunable hyperparameters t, M, w as

The loss is evaluated using the numerically stabilised logsumexp function. The justification of the approximation is given by the following theorem:

Theorem 3. Assume that x ↦→ exp( -E θ ( x )) is uniformly bounded. Then, for every ε > 0 there exist N and M ( N ) such that ∣ ∣ L t,M ( N ) ,w ( θ ) -ED γ t ( p data , E θ ) ∣ ∣ < ε almost surely.

We give the proof in Appendix B.1. This result forms the basis for proofs of asymptotic consistency of our estimators which we leave for future work. It is noteworthy that an analogous implementation of energy discrepancy for other perturbation kernels and other spaces is possible. For example, we construct a similar loss for discrete spaces using a Bernoulli perturbation in Appendix D.3. Similarly to the Gaussian case, the w -stabilisation provides a useful variance reduction for the training with Bernoulli-based energy discrepancy.

4.1 Training Energy-Based Models under the Manifold Hypothesis

LeCun (2022) suggests that maximum-likelihood-based training of energy-based models can lead to the formation of undesirable canyon shaped energy-functions. Indeed, energy discrepancy yields an

ED

SM

ED

SM

ED

SM

Data

Data

Data

energy with low values on the data-support and rapidly diverging values outside of it when being used on image data, directly. Such learned energies fail to represent the distribution between data-points and are not suitable for inference or image generation. We attribute this phenomenon to the manifold hypothesis, which states that the data concentrates in the vicinity of a low-dimensional manifold (Bengio et al., 2013). In this case, the data distribution is not a positive density and can not be written as an energy-based model as log p data is not well-defined. Additionally, Gaussian perturbations of a data point ˜ x := x + ξ are orthogonal to the data manifold with high-probability and the negative samples in the contrastive term are not informative.

To resolve this problem, we will work with a lower-dimensional latent representation of the data distribution in which positivity can be ensured. We follow Pang et al. (2020) and define a variational decoder network p ϕ ( x | z ) and an energy-based tilting of a Gaussian prior p θ ( z ) = exp( -E θ ( z )) p 0 ( z ) on latent space, resulting in the so-called latent energy-based prior models (LEBMs) p ϕ,θ ( x ) ∝ ∫ p ϕ ( x | z ) p θ ( z )d z . To obtain the latent representation of data we sample from the posterior p ϕ,θ ( z | x ) ∝ p ϕ ( x | z ) exp( -E θ ( z )) p 0 ( z ) using a Langevin sampler. For the training of model, the contrastive divergence algorithm is replaced with energy discrepancy (for details see appendix C). LEBMs provide an interesting benchmark to compare energy discrepancy with contrastive divergence in high dimensions. However, other ways to tackle the manifold problem are elements of ongoing research.

5 Empirical Studies

To support our theoretical discussion, we evaluate the performance of energy discrepancy on lowdimensional datasets as well as for the training of latent EBMs on high-dimensional image data sets. In our discussion, we emphasise the comparison with contrastive divergence and score matching as the dominant methodologies in EBM training.

5.1 Density Estimation

We first demonstrate the effectiveness of energy discrepancy on several two-dimensional distributions. Figure 3 displays the estimated unnormalised densities as well as samples that were synthesised with Langevin dynamics for energy discrepancy (ED), score matching (SM) and contrastive divergence (CD). More experimental results and details are given in Appendix D.4. Our results confirm the aforementioned nearsightedness of score matching which does not learn the uniform weight distribution of the Gaussian mixture. For CD, it can be seen that CD consistently produces flattened energy landscapes which can be attributed to the short-run MCMC (Nijkamp et al., 2020a, see) not having converged. Consequently, the synthe-

0

10k

20k

30k

40k

50k

Iterations

0

1

2

3

4

5

MSE of log-density

ED

SM

CD

sised samples of energies learned with CD can lie outside of the data-support. In contrast, ED is able to model multi-modal distributions faithfully and learns sharp edges in the data support as in the chessboard data set. We quantify our results in Figure 4 which shows the mean squared error of the estimated log-density of 25-Gaussians over the number of training iterations. The partition function is estimated using importance sampling log Z ≈ logsumexp( -E ( x i ) -log p data ( x i )) -log N , where x i is sampled from the data distribution p data ( x ) and N =5 , 000 . It shows that ED outperforms SM and CD with faster convergence, lower mean square error, and better stability.

812811657168480

3

S8

5

18

3

6

21

3

2

80

D

8

67

3

7

9

2831

6

3

5.2 Image Modelling

In this experiment, our methods are evaluated by training a latent EBM on three image datasets: SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), and CelebA (Liu et al., 2015). The effectiveness of energy discrepancy is diagnosed through image generation, image reconstruction from their latent representation, and the faithfulness of the learned latent representation. The model architectures, training details, and the choices of hyper-parameters can be found in Appendix C.3.

Image Generation and Reconstruction. We benchmark latent EBM priors trained with energy discrepancy (ED-LEBM) and score matching (SM-LEBM) against various baselines for latent variable models which are included in Table 1. Note that the original work on latent EBMs (Pang et al., 2020) uses contrastive divergence (CD-LEBM) (see appendix C for details).

For a well-learned model, the latent EBM should produce realistic samples and faithful reconstructions. The reconstruction error is measured via the mean square error while the image generation is measured with the FID (Heusel et al., 2017) which are both reported in Table 1. We observe that ED can improve the contrastive divergence benchmark on SVHN and CelebA while the results on CIFAR-10 could not be improved. However, we emphasise that ED only requires M (here, M = 16 ) evaluations of the energy function per data point which is significantly less than CD and SM that both require the calculation of a high-dimensional spatial gradient. Besides the quantitative metrics, we present qualitative results of the generated samples in Figure 5. It can be seen that our model generates diverse high-quality images. The qualitative results of the reconstruction are shown in Figure 6, for which we use the test set of CelebA 64 × 64 . The right column shows the original image x to be reconstructed. The left column shows the reconstruction based on a latent variable initialised from the base distribution z 0 ∼ p 0 ( z ) , and the middle column displays the image reconstructed from z k ∼ p ϕ,θ ( z | x ) which is sampled via Langevin dynamics. One can see that our model can successfully reconstruct the test images, verifying the validity of the latent

XK

X

prior learned with energy discrepancy. In addition, we showcase the scalability of our approach by

(a) Smiling

(b) Male

999

(c) Eyeglasses

(d) Blond Hair

applying it successfully to high-resolution images (CelebA 128 × 128 ). More results can be found in Appendix D.5.

Image Interpolation and Manipulation. We use a latent-variable to model the effective lowdimensional structure in the data set. To probe how well the latent space describes the geometry and meaning of the data-manifold we analyse the structure of the latent space through interpolation and attribute manipulation. For the interpolation between two samples we linearly interpolate between their latent representations which were sampled from the posterior distribution. The results in Figure 7 demonstrate that the latent space has learned the data-manifold well and almost all intermediate samples appear as realistic faces. Further results are given in Figures 19 and 20. In addition, we can utilize the labels in the CelebA dataset to modify the attributes of an image through the manipulation technique proposed in (Kingma & Dhariwal, 2018). Specifically, each image in the dataset is associated with a binary label that indicates the presence or absence of attributes such as smiling, male, eyeglasses. For each manipulated attribute, we compute the average latent vectors z pos of images with and z neg of images without the attribute in the training set. Then, we use the difference z pos -z neg as the direction for manipulating the attribute of an image. The results presented in Figures 8 and 22 confirm the meaningfulness of the latent space learned by energy discrepancy.

5.3 Anomaly Detection

In a well-learned energy-based model the likelihood should be higher for in-distribution examples and lower for out-of-distribution examples. Based on this principle, we conduct anomaly detection experiments to compare our method with other baselines. Specifically, given a test sample x , we first sample z from the posterior p θ ( z | x ) by Langevin dynamics, and then compute the unnormalised log-density p θ ( x , z ) as the decision function. Following the protocol in (Zenati et al., 2018), we designate each digit class in the MNIST dataset as an anomaly and leave the rest as training data. The area under the precision-recall curve (AUPRC) is used to evaluate different methods. As shown in Table 2, our model consistently outperforms the baseline methods, demonstrating the advantages of training latent EBMs with energy discrepancy.

6 Related Work

Training Energy-based models. While energy-based models have been around for some time (Hinton, 2002), the training of energy-based models remains challenging. For a summary on existing methods for the training of energy-based models see Song & Kingma (2021) and LeCun et al. (2006). Contrastive divergence is still the most used option for training energy-based models. Recent extensions have improved the scalability of the basic algorithm, making it possible to train EBMs on high-dimensional data (Ngiam et al., 2011; Dai et al., 2015; Xie et al., 2016; Nijkamp et al., 2019; Du & Mordatch, 2019). Despite these advances, it has been noted that contrastive divergence is not the gradient of any fixed loss-function (Sutskever & Tieleman, 2010) and can yield energy functions that do not adequately reflect the data distribution (Nijkamp et al., 2020a). This has motivated improvements to the standard methodology (Du et al., 2021) by approximating overlooked entropy terms or by improving the convergence of MCMC sampling with an underlying diffusion model (Gao et al., 2021).

Score-based methods (Hyvärinen & Dayan, 2005; Vincent, 2011; Song et al., 2020) are typically implemented by directly learning the score function instead of the energy function. Song & Ermon (2019); Song et al. (2021b) point out the importance to introduce multiple noise levels at which the score is learned. Li et al. (2019) adopt the strategy of learning an energy-based model using multi-scale denoising score matching.

We learn a latent EBM prior (Pang et al., 2020) on high-dimensional data to address situations where the data lives on a lower dimensional submanifold. This methodology has been improved with a multi-stage approach (Xiao & Han, 2022) with score-independent noise contrastive estimation (Gutmann & Hyvärinen, 2010).

Related loss functions. Energy discrepancies can be derived from Kullback Leibler-contraction divergences (Lyu, 2011) which we also discuss briefly in Appendix A.6. Diffusion recovery likelihood (Gao et al., 2021) technically optimises the same loss as us but the EBM is trained with contrastive divergence. Luo et al. (2023) make similar observations to us while estimating the KL-contraction by leveraging the time evolution of the energy function along a diffusion process. Energy discrepancy also shares similarities with other contrastive loss functions in machine learning. The w -stabilised energy-discrepancy loss with w,M = 1 is equivalent to conditional noise contrastive estimation (Ceylan & Gutmann, 2018) with Gaussian noise. For M > 1 , the stabilised ED loss shares great structural similarity to certain contrastive learning losses such as InfoNCE (van den Oord et al., 2018). We hope that such observations can lead to interpretations of the w -stabilisation.

Theoretical connections between likelihood and score-based functionals. The connection between likelihood methods and score-based methods generalises the de Bruijn identity (Stam, 1959; Lyu, 2009) which explains that score matching appears frequently as the limit of EBM training methods. Similar connections have been mentioned and exploited by Song et al. (2021a) to choose a weighting scheme for a combination of score matching losses.

Generative modelling for manifold-valued data. The manifold hypothesis was, for example, described in Bengio et al. (2013). Prior work to us shows how score-based diffusion models detect this manifold (Pidstrigach, 2022) and give reasons why CD is more robust to this issue than other contrastive methods (Yair & Michaeli, 2021). Arbel et al. (2021) learn an energy-based model as a tilt of a GAN-based prior that models the data-manifold. We believe that a combination of above results with energy discrepancy could enable training EBMs with energy discrepancy on data space.

7 Discussion and Outlook

We demonstrate that energy discrepancy provides a new tool for the fast training of energy-based models without MCMC or scores. We show for Euclidean data that ED interpolates between score matching and maximum likelihood estimation, thus alleviating problems of nearsightedness of score matching without additional annealing strategies as in score-based diffusion models. Based on our theoretical analysis, we show that training EBMs using energy discrepancy yields more accurate estimates of the energy function for two-dimensional data than explicit score matching and contrastive divergence. In this task, it is robust to the hyperparameters used, making energy discrepancy a useful tool for energy-based modelling with little tuning required. We further establish that energy discrepancy achieves comparable results to contrastive divergence at a lower computational cost

when learning a lower-dimensional energy-based prior for high-dimensional data. This shows that energy discrepancy is scalable to high dimensions.

Limitations: Energy-based models make the assumption that the data-distribution has a positive density which is violated for most high-dimensional data sets due to the manifold hypothesis. Compared to contrastive divergence or diffusion-based strategies, energy discrepancy is especially sensitive to such singularities in the data set. Currently, this limits energy discrepancy to settings in which a lower-dimensional representation with non-singular support can be learned or constructed efficiently and accurately.

Outlook: This work can be extended in various ways. Firstly, we want to explore other choices for the conditional distribution q . In particular, the effect of different types of noise such as Laplace noise or anisotropic Gaussian noise are open questions. Furthermore, energy discrepancy is a wellsuited objective function for discrete data for appropriately chosen perturbations. We present a preliminary study of energy discrepancy on discrete spaces in the workshop paper Schröder et al. (2023). Secondly, we believe that improvements to our methodology can be made by learning the energy function of image-data on pixel space directly, as in this domain specific architectures such as U-nets (Ronneberger et al., 2015; Song & Ermon, 2019) can be used in the modelling. However, this requires further work in learning the data-manifold during training so that early saturation can be prevented. Finally, the partial differential equations arising in Euclidean energy discrepancies promise exciting insights into the connections between energy-based learning, stochastic differential equations, and optimal control which we want to investigate further.

References

Arbel, Michael, Zhou, Liang, and Gretton, Arthur. Generalized energy based models. In International Conference on Learning Representations , 2021.

Barp, Alessandro, Briol, Francois-Xavier, Duncan, Andrew, Girolami, Mark, and Mackey, Lester. Minimum stein discrepancy estimators. Advances in Neural Information Processing Systems , 32, 2019.

Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(8):1798-1828, 2013.

Carreira-Perpiñán, Miguel Á. and Hinton, Geoffrey. On contrastive divergence learning. In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics , volume R5 of Proceedings of Machine Learning Research , pp. 33-40. PMLR, 06-08 Jan 2005.

Ceylan, Ciwan and Gutmann, Michael U. Conditional noise-contrastive estimation of unnormalised models. In Dy, Jennifer and Krause, Andreas (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 726-734. PMLR, 10-15 Jul 2018.

Chwialkowski, Kacper, Strathmann, Heiko, and Gretton, Arthur. A kernel test of goodness of fit. In International conference on machine learning , pp. 2606-2615. PMLR, 2016.

Dai, Bin and Wipf, David. Diagnosing and enhancing vae models. In International Conference on Learning Representations , 2018.

Dai, Hanjun, Singh, Rishabh, Dai, Bo, Sutton, Charles, and Schuurmans, Dale. Learning discrete energy-based models via auxiliary-variable local exploration. Advances in Neural Information Processing Systems , 33:10443-10455, 2020.

Dai, Jifeng, Lu, Yang, and Wu, Ying Nian. Generative modeling of convolutional neural networks. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Du, Yilun and Mordatch, Igor. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems , 32, 2019.

Du, Yilun, Li, Shuang, Tenenbaum, Joshua, and Mordatch, Igor. Improved contrastive divergence training of energy-based models. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 2837-2848. PMLR, 18-24 Jul 2021.

Friedman, Avner. Stochastic differential equations and applications . Courier Corporation, 2012.

Gao, Ruiqi, Song, Yang, Poole, Ben, Wu, Ying Nian, and Kingma, Diederik P. Learning energybased models by diffusion recovery likelihood. In 9th International Conference on Learning Representations, ICLR , 2021.

Gelman, Andrew and Rubin, Donald B. Inference from iterative simulation using multiple sequences. Statistical science , pp. 457-472, 1992.

Ghosh, Partha, Sajjadi, Mehdi SM, Vergari, Antonio, Black, Michael, and Scholkopf, Bernhard. From variational to deterministic autoencoders. In International Conference on Learning Representations , 2019.

Glaser, Pierre, Arbel, Michael, Doucet, Arnaud, and Gretton, Arthur. Maximum likelihood learning of energy-based models for simulation-based inference. arXiv preprint arXiv:2210.14756 , 2022.

Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.

Gorham, Jackson and Mackey, Lester. Measuring sample quality with kernels. In International Conference on Machine Learning , pp. 1292-1301. PMLR, 2017.

Grathwohl, Will, Wang, Kuan-Chieh, Jacobsen, Jörn-Henrik, Duvenaud, David, Norouzi, Mohammad, and Swersky, Kevin. Your classifier is secretly an energy based model and you should treat it like one. In 8th International Conference on Learning Representations, ICLR , 2020.

Grathwohl, Will, Swersky, Kevin, Hashemi, Milad, Duvenaud, David, and Maddison, Chris. Oops i took a gradient: Scalable sampling for discrete distributions. In International Conference on Machine Learning , pp. 3831-3841. PMLR, 2021.

Gutmann, Michael and Hyvärinen, Aapo. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.

Han, Tian, Lu, Yang, Zhu, Song-Chun, and Wu, Ying Nian. Alternating back-propagation for generator network. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Heusel, Martin, Ramsauer, Hubert, Unterthiner, Thomas, Nessler, Bernhard, and Hochreiter, Sepp. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Hinton, Geoffrey E. Training products of experts by minimizing contrastive divergence. Neural Comput. , 14(8):1771-1800, aug 2002.

Hyvärinen, Aapo and Dayan, Peter. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Kingma, Diederik P and Ba, Jimmy. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015 , 2015.

Kingma, Diederik P and Welling, Max. Auto-encoding variational bayes. 2nd International Conference on Learning Representations, ICLR , 2014.

Kingma, Durk P and Dhariwal, Prafulla. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems , 31, 2018.

Krizhevsky, Alex, Hinton, Geoffrey, et al. Learning multiple layers of features from tiny images. 2009.

Kumar, Rithesh, Ozair, Sherjil, Goyal, Anirudh, Courville, Aaron, and Bengio, Yoshua. Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.08508 , 2019.

LeCun, Yann. From machine learning to autonomous intelligence lecture 2, 2022. URL https: //leshouches2022.github.io/SLIDES/lecun-20220720-leshouches-02.pdf .

LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, and Huang, Fujie. A tutorial on energybased learning. Predicting structured data , 1(0), 2006.

Li, Zengyi, Chen, Yubei, and Sommer, Friedrich T. Learning energy-based models in highdimensional spaces with multiscale denoising-score matching. Entropy , 2019.

Liu, Qiang, Lee, Jason, and Jordan, Michael. A kernelized stein discrepancy for goodness-of-fit tests. In Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pp. 276-284. PMLR, 2016.

Liu, Xing, Duncan, Andrew B, and Gandy, Axel. Using perturbation to improve goodness-of-fit tests based on kernelized stein discrepancy. International Conference on Machine Learning ICML , 40, 2023.

Liu, Ziwei, Luo, Ping, Wang, Xiaogang, and Tang, Xiaoou. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision , pp. 3730-3738, 2015.

Luo, Weijian, Jiang, Hao, Hu, Tianyang, Sun, Jiacheng, Li, Zhenguo, and Zhang, Zhihua. Training energy-based models with diffusion contrastive divergences, 2023.

Lyu, Siwei. Interpretation and generalization of score matching. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , UAI '09, pp. 359-366, Arlington, Virginia, USA, 2009. AUAI Press. ISBN 9780974903958.

Lyu, Siwei. Unifying non-maximum likelihood learning objectives with minimum kl contraction. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011.

Majerek, Dariusz, Nowak, Wioletta, and Ziba, Wies. Conditional strong law of large number. International Journal of Pure and Applied Mathematics , 20, 01 2005.

Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011.

Ngiam, Jiquan, Chen, Zhenghao, Koh, Pang Wei, and Ng, A. Learning deep energy models. In International Conference on Machine Learning , 2011.

Nijkamp, Erik, Hill, Mitch, Zhu, Song-Chun, and Wu, Ying Nian. Learning non-convergent nonpersistent short-run mcmc toward energy-based model. In Advances in Neural Information Processing Systems , volume 32, 2019.

Nijkamp, Erik, Hill, Mitch, Han, Tian, Zhu, Song-Chun, and Wu, Ying Nian. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence , 2020a.

Nijkamp, Erik, Pang, Bo, Han, Tian, Zhou, Linqi, Zhu, Song-Chun, and Wu, Ying Nian. Learning multi-layer latent variable model via variational optimization of short run mcmc for approximate inference. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16 , pp. 361-378. Springer, 2020b.

Ou, Zijing, Xu, Tingyang, Su, Qinliang, Li, Yingzhen, Zhao, Peilin, and Bian, Yatao. Learning neural set functions under the optimal subset oracle. Advances in Neural Information Processing Systems , 35:35021-35034, 2022.

Pang, Bo, Han, Tian, Nijkamp, Erik, Zhu, Song-Chun, and Wu, Ying Nian. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems , 33:21994-22008, 2020.

Peyré, Gabriel and Cuturi, Marco. Computational optimal transport. Foundations and Trends in Machine Learning , 11 (5-6):355-602, 2019.

Pidstrigach, Jakiw. Score-based generative models detect manifolds. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems , volume 35, pp. 35852-35865. Curran Associates, Inc., 2022.

Raginsky, Maxim and Sason, Igal. Concentration of measure inequalities in information theory, communications, and coding. Foundations and Trends® in Communications and Information Theory , 10(1-2):1-246, 2013. ISSN 1567-2190. doi: 10.1561/0100000064.

Ronneberger, Olaf, Fischer, Philipp, and Brox, Thomas. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention - MICCAI , 9351:234-241, 2015.

Schröder, Tobias, Ou, Zijing, Li, Yingzhen, and Duncan, Andrew B. Training discrete ebms with energy discrepancy. Sampling and Optimization in Discrete Spaces workshop, ICML , 2023.

Song, Yang and Ermon, Stefano. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019.

Song, Yang and Kingma, Diederik P. How to Train Your Energy-Based Models. arXiv e-prints , 2021.

Song, Yang, Garg, Sahaj, Shi, Jiaxin, and Ermon, Stefano. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence , pp. 574-584. PMLR, 2020.

Song, Yang, Durkan, Conor, Murray, Iain, and Ermon, Stefano. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems , 34:1415-1428, 2021a.

Song, Yang, Sohl-Dickstein, Jascha, Kingma, Diederik P., Kumar, Abhishek, Ermon, Stefano, and Poole, Ben. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR , 2021b.

Stam, A.J. Some inequalities satisfied by the quantities of information of fisher and shannon. Information and Control , 2(2):101-112, 1959. ISSN 0019-9958.

Sutskever, Ilya and Tieleman, Tijmen. On the convergence properties of contrastive divergence. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pp. 789-795. JMLR Workshop and Conference Proceedings, 2010.

van den Oord, Aäron, Li, Yazhe, and Vinyals, Oriol. Representation learning with contrastive predictive coding. arXiv preprint 1807.03748 , 2018.

Vincent, Pascal. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Xiao, Zhisheng and Han, Tian. Adaptive multi-stage density ratio estimation for learning latent space energy-based model. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems , volume 35, pp. 21590-21601. Curran Associates, Inc., 2022.

Xie, Jianwen, Lu, Yang, Zhu, Song-Chun, and Wu, Yingnian. A theory of generative convnet. In International Conference on Machine Learning , pp. 2635-2644. PMLR, 2016.

Yair, Omer and Michaeli, Tomer. Contrastive divergence learning is a time reversal adversarial game. In International Conference on Learning Representations , 2021.

Zenati, Houssam, Foo, Chuan Sheng, Lecouat, Bruno, Manek, Gaurav, and Chandrasekhar, Vijay Ramaseshan. Efficient gan-based anomaly detection. arXiv preprint arXiv:1802.06222 , 2018.

Zhang, Mingtian, Hayes, Peter, Bird, Thomas, Habib, Raza, and Barber, David. Spread divergence. In III, Hal Daumé and Singh, Aarti (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 11106-11116. PMLR, 2020.

Zhang, Mingtian, Key, Oscar, Hayes, Peter, Barber, David, Paige, Brooks, and Briol, François-Xavier. Towards healing the blindness of score matching. NeurIPS 2022 Workshop on Score-Based Methods , 2022.

Øksendal, Bernt. Stochastic Differential Equations : An Introduction with Applications . Universitext. Springer, Berlin, Heidelberg, sixth edition. edition, 2003. ISBN 9783642143946.

Appendix for 'Energy Discrepancies: A Score-Independent Loss for Energy-Based Models'

Contents

A Proofs and Derivations

In this section we are going to prove Theorem 1 and Theorem 2. Furthermore, we are going to generalise Theorem 2 to arbitrary diffusion processes and discuss the connections of energy discrepancy to other training losses for energy-based models.

A.1 Proof of the Non-Parametric Estimation Theorem 1

In this subsection we give a formal proof for the uniqueness of minima of ED q ( p data , U ) as a functional in the energy function U . We first reiterate the theorem as stated in the paper:

Theorem 1. Let p data be a positive probability density on ( X , d x ), and let q ( ·| x ) be a conditional probability density. Under mild assumptions on q and the set of admissible energy functions U , energy discrepancy ED q is functionally convex in U and has, up to additive constants, a unique global minimiser U ∗ = arg min ED q ( p data , U ) with exp( -U ∗ ) ∝ p data .

For this theorem we need to specify additional assumptions on the conditional distribution q and on the optimisation domain to guarantee uniqueness. Firstly, we require the energy-based distribution to be normalisable which implies that exp( -U ) ∈ L 1 ( X , d x ) . For the existence and uniqueness of minimisers we have to constrain the space of energy functions since ED q ( p data , U ) = ED q ( p data , U + c )

for any constant c ∈ R . Hence, we restrict the optimisation domain to functions U that satisfy min x ∈X U ( x ) = 0 . To specify the condition on q we define an equivalence relation on X :

Definition 2 ( q -equivalence) . For a pair of samples x 1 , x 2 ∈ X we say that x 1 and x 2 are q -neighbours if there exists a y ∈ Y such that x 1 , x 2 ∈ supp( q ( y |· )) . We denote the set of q -neighbours of x as N q ( x ) . We then define the q -equivalence relation as:

It is easy to see that q -equivalence defines an equivalence relation, i.e. it is symmetric, reflexive, and transitive. We summarise the assumptions for the non-parametric approximation theorem as follows:

Assumption 1. We define the optimisation domain

We then make the following assumptions on q and G :

All elements of X are q -equivalent, i.e. for every x ∈ X it holds that [ x ] q = X .

There exists a U ∗ ∈ G such that exp( -U ∗ ) ∝ p data .

Under Assumption 1, ED q ( p data , U ) has a unique global minimiser U ∗ = -log p data + c in G . We prove this by computing the first and second variation of ED q . Note that G may not be a vector space in general since in some cases 0 / ∈ G . Furthermore, we did not specify a norm on G . We will omit this technical issue in this discussion. We start from the following lemmata and complete the proof of Theorem 1 in Corollary 1.

Lemma 1. Let h ∈ G be arbitrary. The first variation of ED q is given by

Proof. We define the short-hand notation U ϵ := U + ϵh . The energy discrepancy at U ε reads

For the first functional derivative, we only need to calculate

Plugging this expression into ED q ( p data , U ϵ ) and setting ϵ = 0 yields the first variation of ED q .

Lemma 2. The second variation of ED q is given by

Proof. For the second order term, we have based on equation 8 and the quotient rule for derivatives:

We obtain the desired result by interchanging the outer expectations with the derivatives in ϵ .

Corollary 1. Let c = min x ∈X ( -log p data ( x )) . For U ∗ = -log( p data ) -c ∈ G it holds under Assumption 1 that

Furthermore, U ∗ is the unique global minimiser of ED q ( p data , · ) in G .

Proof. By definition, the variance is non-negative, i.e. for every h ∈ G :

Consequently, the energy discrepancy is convex and an extremal point of ED q ( p data , · ) is a global minimiser. We are left to show that the minimiser is obtained at U ∗ and unique. First of all, we have:

By applying the outer expectations we obtain

where we used that the marginal distributions ∫ p data ( x ) q ( y | x )d x cancel out and the conditional probability density integrates to one. This implies

for all h ∈ G . We now show that the second variation is also positive definite, i.e.

where p data ( z | y ) = q ( y | z ) p data ( z ) ∫ q ( y | z ′ ) p data ( z ′ )d z ′ . Assume that there exists a test function h ∈ G such that E p data ( x ) E q ( y | x ) Var p data ( z | y ) [ h ( z )] = 0 . Since this is the expectation of a non-negative random variable, it follows for all y ∈ Y that Var p data ( z | y ) [ h ( z )] = 0 . Consequently, h ( z ) has to be constant p data ( ·| y ) -almost surely and there exists a measurable function g : Y → R such that h ( z ) = g ( y ) for all z ∈ supp( p ( ·| y )) . Next, assume that z 1 and z 2 are q -neighbours. By definition, there exists a y ∈ Y such that z 1 , z 2 ∈ supp( q ( y |· )) . Since the data density is positive, the support of q ( y |· ) equals the support of p data ( ·| y ) . Consequently, it holds that h ( z 1 ) = g ( y ) = h ( z 2 ) . It now follows from the definition of q -equivalence that h has to be constant almost surely on all q -equivalence classes [ x ] q , since any two elements in an equivalence class can be connected by a chain of q -neighbours. By Assumption 1, [ x ] q = X , and thus h is constant on X . However, there are no non-trivial constant test function in G . Hence, the second variation has to be positive definite for all h ∈ G . Consequently, U ∗ is the unique global minimiser of ED q which completes the statement in Theorem 1.

A.2 Equivalence of Energy Discrepancy for Brownian Motion and Ornstein-Uhlenbeck Processes

In this subsection we show that an energy discrepancy based on an Ornstein-Uhlenbeck process is equivalent to the energy discrepancy based on a time-changed Brownian motion.

Proposition 1. Let q t be the transition density for the Ornstein-Uhlenbeck process d x t = α x t d t + √ β d w t with standard Brownian motion w t , and let γ t ( y | x ) ∝ exp( -∥ y -x ∥ 2 / 2 t ) be the Gaussian transition density of Brownian motion. Then,

Proof. At time t , the Ornstein-Uhlenbeck process has distribution

with σ α ( t ) = √ β 2 α ( e 2 αt -1) and σ 0 ( t ) = √ βt . The Ornstein-Uhlenbeck process is variance exploding for α ≥ 0 and variance preserving for α < 0 . Based on (9). the transition density of x t is given as

Hence, we obtain via the change of variables ξ ′ := ( y -e αt x ) /σ α ( t ) ∼ N (0 , I ) for the contrastive potential

We now evaluate the contrastive potential at the forward process y = x t which yields

where we used that e -αt σ α ( t ) = σ -α ( t ) in the second equality and the change of variables x = w σ 2 -α ( t ) -ξ ′ in the third equality. Hence, the energy discrepancy for the Ornstein-Uhlenbeck process is equivalent to the energy discrepancy for Brownian motion with time parameter

Notice that for the variance-exploding process with α > 0 the contrasting particles have a finite horizon since σ -α ( t ) t →∞ - - - → √ β 2 α < ∞ . Hence, the maximum-likelihood limit in Theorem 2 is only achieved for the variance preserving process with α < 0 and for the critical case of Brownian motion with α = 0 .

A.3 Interpolation between Score-Matching and Maximum-Likelihood Estimation

Wefirst prove the result as stated in the Gaussian case. We then show how the result can be generalised to arbitrary diffusions by using Ito calculus.

Gaussian case Denote the Gaussian density as

and define the convolved distributions p t := γ t ∗ p data and exp( -U t ) := γ t ∗ exp( -U ) .

Proposition 2. The energy discrepancy is the multi noise-scale score-matching loss

Proof. It is known that γ t is the solution of the heat equation:

Consequently, both, p t and exp( -U t ) satisfy the heat-equation because the integral commutes with the differential operators. Based on the heat-equation we can derive the following non-linear partial differential equation for the contrastive potential U t :

Since ∂ t e -U t = -e -U t ∂ t U t , we get after cancellation of the exponentials:

The integral notation of the contrastive term in energy discrepancy takes the form

We now take a derivative of the energy discrepancy and find

where we used integration by parts twice in the final equation to shift the differential operator from p t to U t . Now, plugging in the differential equation for U t we find

Finally, we obtain energy discrepancy by integrating above expression:

This gives the desired integral representation in Proposition 2.

Proposition 3. Let γ t be the Gaussian transition density and p ebm ∝ exp( -U ) the energy-based distribution. The energy discrepancy converges to a cross entropy loss at a linear rate in time

where c ( t ) is a renormalising constant independent of U .

For the proof we employ the following lemma of Yihong Wu which was given in Raginsky & Sason (2013).

Lemma 3. Let γ t be the Gaussian transition density of a standard Brownian motion. Let µ, ν be probability distributions and denote µ t := γ t ∗ µ and ν t := γ t ∗ ν . The following information-transport inequality holds:

Proof. Let π be a probability density of ( x , x ′ ) with marginal distributions µ ( x ) and ν ( x ′ ) (also called a coupling in optimal transport). We have by the decomposition of Kullback-Leibler divergences

Hence, we find by rearranging the inequality

The right hand side is the Kullback-Leibler divergence between Gaussians, so

Since the coupling was arbitrary, we can minimise over all couplings π of µ and ν which results in the Wasserstein-distance

where Π( µ, ν ) denotes the set of all joint distributions with marginals µ and ν

The proof of Proposition 3 then follows:

Proof. Let µ = p data and ν = p ebm , denote the convolved distributions as p t, data and p t, ebm . Notice that for any x , y ∈ R d

since both models have the same normalising constant. We then have for arbitrary x

with U independent entropy term c ( t ) := E p t, data ( y ) [log p t, data ( y )] . Since x was chosen arbitrarily, we can integrate with respect to p data ( x ) and find

.

A.4 Representing Energy Discrepancy as Multi-Scale SM for General Diffusion Processes

We now prove the connection between energy discrepancy and multi-noise scale score matching in a general context. For all following results we will assume that x t is some stochastic diffusion process which satisfies the SDE d x t = a ( x t )d t + b ( x t )d w t and assume that x 0 ∼ p data . Let q t denote the associated transition probability density. To make the exposition cleaner we write U t := U q t .

The main idea will be the following observation:

Proposition 4. The diffusion-based energy discrepancy is given by the expectation of the Ito integral

Proof. The stochastic integral with respect to the differential d U s ( x s ) is defined to satisfy the following generalisation of the fundamental theorem of calculus:

We obtain the desired result by taking expectations on both sides in ( x s ) 0 ≤ s ≤ t .

Notice that the law of the random variable x s is fixed by the initial distribution of the diffusion x 0 ∼ p data . These distributions are implied when taking the expectation. We will now explore this connection further. For this we make some basic assumptions which allow us to connect stochastic differential equations with partial differential equations.

Assumption 2. Consider the stochastic differential equation d x t = a ( x t )d t + b ( x t )d w t for drift a : R d → R d and b : R d → R k . Further, define the diffusion matrix Σ( x ) = b ( x ) b ( x ) T ∈ R d × d . We make the following assumptions:

There exists a µ > 0 such that for all ξ , x ∈ R d ⟨ ξ , Σ( x ) ξ ⟩ ≥ µ ∥ ξ ∥ 2

Σ and a are bounded and uniformly Lipschitz-continuous in x on every compact subset of R d

Σ is uniformly Hölder-continuous in x

Theorem 4 (Fokker-Planck equation) . Under Assumption 2, x t has a transition density function given by

Furthermore, q t satisfies the Fokker-Planck partial differential equation

For a reference, see (Friedman, 2012, Theorem 5.4)

The Fokker-Planck equation yields the following important differential equation for the contrastive potential U t :

Proposition 5. Consider the stochastic differential equation d x t = a ( x t )d t + b ( x t )d w t for drift a : R d → R d and b : R d → R k , and diffusion matrix Σ( x ) = b ( x ) b ( x ) T ∈ R d × d that satisfies assumptions 2. Let q t be the associated transition density and define the contrastive potential U t ( y ) := -log ∫ q t ( y | x ) exp( -U ( x ))d x . Furthermore, we define the scalar field

and the linear operator

Then, the contrastive potential satisfies the non-linear partial differential equation

Proof. We commute the linear operator of the Fokker-Planck equation to see that e -U t satisfies the Fokker-Planck equation in Theorem 4, too, i.e.

We now expand the term corresponding to the drift term:

Similarly, we treat the diffusion term:

Finally, the time derivative simply becomes ∂ t e -U t ( y ) = -∂ t U t ( y ) e -U t ( y ) . We can now collect all terms independent of U and identify

as well as the linear operator term

Finally, we have

This gives us the partial differential equation

Cancelling all exponentials from both sides of the equation yields the desired result.

Theorem 5. The energy discrepancy takes the form of a generalised multi-noise scale score matching loss:

Proof. For this proof we return to the stochastic process U s ( x s ) from Proposition 4. By Ito's formula, U s ( x s ) satisfies the stochastic differential equation

Under the additional integrability condition that E ∫ t 0 ∥ b T ( x s ) ∇ U s ( x s ) ∥ 2 d s < ∞ , the stochastic integral with respect to Brownian motion d w s has expectation zero. Furthermore, we can replace ∂ s U s ( x s ) with our previously obtained non-linear partial differential equation

Due to opposing signs, the drift a cancels, i.e.

Consequently, we obtain the final energy discrepancy expression using Proposition 4

with U -independent constant 0 c ( a, Σ)( x s )d s . This completes the proof.

As a corollary we obtain the proof of the first statement in Theorem 2: Assume that q t is defined through the stochastic differential equation d x t = a ( x t )d t +d w t . In this case, Σ = I and ∑ d i,j =1 ∂ x j (Σ ij ( x s ) ∂ x i U ( x s )) = ∆ U ( x s ) . Consequently, we obtain from Theorem 5 the score matching representation of ED q t in Theorem 2. In the special case that Σ = bb T is independent of x we obtain an integrated sliced score-matching loss (Song et al., 2020).

A.5 Connections of Energy Discrepancy with Contrastive Divergence

The contrastive divergence update can be derived from an energy discrepancy when, for E θ fixed, q satisfies the detailed balance relation

To see this, we calculate the contrastive potential induced by q : We have

Consequently, the energy discrepancy induced by q is given by

Updating θ based on a sample approximation of this loss leads to the contrastive divergence update

Three things are important to notice:

Implicitly, the distribution q depends on E θ and needs to adjusted in each step of the algorithm

For fixed q , ED q ( p data , E θ ) satisfies Theorem 1. This means that each step of contrastive divergence optimises a loss with minimiser E ∗ θ = -log p data + c . However, q needs to be adjusted in each step as otherwise the contrastive potential is not given by the energy function E θ itself.

This result highlights the importance to use Metropolis-Hastings adjusted Langevin-samplers to implement CD to ensure that the implied q distribution satisfies the detailed balance relation. This matches the observations found by Yair & Michaeli (2021).

A.6 Derivation of Energy Discrepancy from KL Contractions

AKullback-Leibler contraction is the divergence function KL( p data ∥ p ebm ) -KL( Qp data ∥ Qp ebm ) (Lyu, 2011) for the convolution operator Qp ( y ) = ∫ q ( y | x ) p ( x )d x . The linearity of the convolution operator retains the normalisation of the measure, i.e. for the energy-based distribution p ebm we have

The KL divergences then become with U q := -log Q exp( -U ( x ))

Since the normalisation cancels when subtracting the two terms we find

where c is a constant that contains the U -independent entropies of p data and Qp data .

B Aspects of Training Energy-Based Models with Energy Discrepancy

In this section, we discuss the w -stabilisation in depth. Furthermore, we give a proof for Theorem 3 and show energy discrepancy can be approximated for other diffusion processes via the Feynman-Kac formula.

B.1 Conceptual Understanding of the w -Stabilisation

The w -stabilisation is a useful stabilisation for all types of perturbation q . For example, we use the same stabilisation in the discrete setting (see Appendix D.3). To provide a better intuition for the stabilisation, we will investigate it more closely for the Gaussian perturbation.

The critical step for using energy discrepancy in practice is a stable approximation of the contrastive potential. For the Gaussian-based energy discrepancy, we can write the contrastive potential as U t ( y ) = -log E [exp( -U ( y + √ t ξ ′ ))] with ξ ′ ∼ N (0 , I ) and y ∈ R d . A naive approximation of the expectation with a Monte-Carlo estimator, however, is biased because of Jensen's inequality, i.e. for ξ ′ , ξ ′ j ∼ N (0 , I ) we have

Our first observation is that the appearing bias can be quantified to leading order. For this, define v t ( y ) := E [exp( -U ( y + √ t ξ ′ ))] and ̂ v t ( y ) := 1 M ∑ M j =1 exp( -U ( y + √ t ξ ′ j )) . We use the Taylor-expansion of log(1 + u ) ≈ u -1 / 2 u 2 +h . o . t . which gives

The linear term in the Taylor-expansion does not contribute because E [ ̂ v t ( y )] = v t ( y ) . In the final equation we used that Var( ̂ v t ( y )) = Var[exp( -U ( y + √ t ξ ′ ))] /M because all ξ ′ j are independent. The Taylor expansion shows that the dominating contribution to the bias is the variance of the approximated convolution integral.

Our second observation is that this occurring bias can become infinite for malformed energy functions. For this reason, the optimiser may start to increase the bias instead of minimising our target loss. To illustrate how a high-variance estimator of the contrastive potential can be divergent, consider the energy function

The energy function does not strictly adhere to our conditions that the energy based model should be normalisable. Our argument still holds when exp( -U ) is changed to be normalisable. In theory, the contrastive potential at 0 is upper bounded because

because exp( -U ( x )) converges to an indicator function on { x ≤ 0 } as b →∞ . The Monte Carlo estimator of the contrastive potential, on the other hand, has upper bound

which can be seen by applying standard inequalities for the logsumexp function 4 . Hence, as long as there exists a j such that ξ j ≤ 0 , the estimated contrastive potential does not diverge. If, however, ξ ′ j > 0 for every j = 1 , . . . , M , then

Consequently, the approximate contrastive potential may attain diverging values at discontinuities in the energy function. Indeed, this phenomenon is observed for w = 0 in Figure 2. Here, the learned energy becomes discontinuous at the edge of the support and the energy discrepancy loss diverges during training. In low dimensions, this problem can be alleviated by using variance reduction techniques such as antithetic variables or by using large enough values of M during the training. The stabilising effect of M is observed in our ablation studies in Figure 24. In high-dimensional settings, however, such variance reduction techniques are infeasible.

The idea of the w -stabilisation is that the value of the energy at non-perturbed data points U ( x 0 ) is guaranteed to stay controlled since it is minimised in the optimisation of ED. Hence, the diverging contrasting potential can be controlled by including U ( x 0 ) in the summation in the logsumexp operation which acts as a soft-min over all contrasting energy contributions. Indeed, this augmentation provides a deterministic upper bound to the approximated contrastive potential:

Additionally, the w -stabilisation introduces a negative bias to the approximated contrastive potential. Hence, if tuned correctly, it counteracts the bias introduced by the Jensen-gap of the logarithm.

To gain additional intuition on the effect of w , notice that by the same bounds as before,

for every data point x . This tells us that, roughly speaking, a perturbed data point with U ( x 0 ) -U ( x t + √ t ξ ′ ) > log( w ) should have a small contribution to the loss and the optimisation converges if the data distribution is learned or when the bound is violated at all perturbed data points. Thus, log( w ) describes a weak notion of a margin between positive and negative energy contributions. Consequently, large values for w ∈ (0 , 1) tend to lead to flatter learned energies, while smaller values lead to steeper learned energies. This intuition is confirmed by Figures 2 and 24.

Asymptotic consistency of sample approximation of ED We give a proof for Theorem 3 which states that our approximation of energy discrepancy is justified. To make the exposition easier to understand, we first show how the energy discrepancy is transformed into a conditional expectation. Recall the probabilistic representation of the contrastive potential Section 4. Using E θ ( x ) = log(exp( E θ ( x ))) we obtain the following rewritten form of energy discrepancy:

The conditioning means that the expectation is not taken with respect to y or x and ξ in the inner expectation. The conditioning is important to understand how the law of large numbers is to be applied. We now come to the proof that our approximation is consistent with the definition of energy discrepancy:

Theorem 3. Assume that x ↦→ exp( -E θ ( x )) is uniformly bounded. Then, for every ε > 0 there exist N and M ( N ) such that ∣ ∣ L t,M ( N ) ,w ( θ ) -ED γ t ( p data , E θ ) ∣ ∣ < ε almost surely.

Proof. First, consider independent random variables x ∼ p data , ξ ∼ N (0 , I ) , and ξ ′ j i.i.d ∼ N (0 , I ) . Using the triangle inequality, we can upper bound the difference | ED γ t ( p data , E θ ) -L t,M,w ( θ ) | by upper bounding the following two terms, individually:

The first term can be bounded by a sequence ε N a.s. - - → 0 due to the normal strong law of large numbers. The second term can be estimated by applying the following conditional version of the strong law of large numbers (Majerek et al., 2005, Theorem 4.2):

Next, we have that the deterministic sequence w/M → 0 . Thus, adding the regularistion w/M does not change the limit in M . Furthermore, since the logarithm is continuous, the limit also holds after applying the logarithm. Finally, the estimate translates to the sum by another application of the triangle inequality. We define

For each i = 1 , 2 , . . . , N there exists a sequence ε i,M a.s. - - → 0 such that

Hence, for each ε > 0 there exists an N ∈ N and an M ( N ) ∈ N such that | ED γ t ( p data , E θ ) -L t,M ( N ) ,w ( θ ) | < ε almost surely.

B.2 Approximation of Energy Discrepancy based on general Ito Diffusions

Energy discrepancies are useful objectives for energy-based modelling when the contrastive potential can be approximated easily and stabily. In most cases this requires us to write the contrastive potential as an expectation which can be computed using Monte Carlo methods. We show how such a probabilistic representation can be achieved for a much larger class of stochastic processes via application of the Feynman-Kac formula. We first highlight the difficulty. Consider the integral ∫ f ( y ) q t ( y | x ) p data ( x )d x d y . Since the expectation is taken in y , the integral can be represented as an expectation of the forward process associated with q t , i.e.

where x i t are simulated processes initialised at x i 0 = x i ∼ p data . Next, consider the integral

This integral is more difficult to approximate because the function g is evaluated at the starting point of the diffusion x t but weighted by it's transition probability density. To compute such integrals without sampling from g we use the Feynman-Kac formula, see e.g. Øksendal (2003):

Theorem 6 (Feynman-Kac) . Let g ∈ C 2 0 ( R d ) and c ∈ C ( R d ) . Assume that v ∈ C 1 , 2 ( R ≥ 0 , R d ) is bounded on K × R d with K compact and satisfies

Then, v has the probabilistic representation

where ( y t ) t ≥ 0 is a diffusion process with infinitesimal generator A .

We will establish that v ( t, y ) satisfies a partial differential equation of the above form which yields a probabilistic representation of the contrastive potential. We know that v ( t, y ) satisfies the FokkerPlanck equation (10). By applying the product rule to each term in the Fokker-Planck equation we find

By comparing with Theorem 6, we identify the infinitesimal generator

Hence we associate the forward diffusion process d x t = a ( x t )d t + b ( x t )d w t with it's backwards process with infinitesimal generator A

with Σ( y ) = b ( y ) b T ( y ) . This yields the probabilistic representation of v ( t, y ) in terms of the backward process y t :

Hence, we also obtain a probabilistic representation for the contrastive potential by choosing g ( x ) := exp( -U ( x )) . This finally gives

Unlike the contrasting term in contrastive divergence, this expression can indeed be calculated by simulating stochastic processes that are entirely independent of U . For this we simulate from the forward process starting at x which yields ˜ x t , where the tilde denotes that this simulation may not be exact. We then simulate M copies of the reverse process and keep all values at intermediate steps, i.e. (˜ y j t 0 = ˜ x t , ˜ y j t 1 , . . . , ˜ y j t K = t ) for j = 1 , . . . , M . Finally we evaluate the contrastive potential as

The simulation method for the stochastic process and for the integration ∫ t 0 c ( y s )d s may be altered in this approximation. At this stage, it is unclear what practical implications the weighting term ∫ t 0 c ( y s )d s has. Notice that the process y t is initialised at the final simulated position of the forward process ˜ x t . Furthermore, the bias correction with the w -stabilisation or an alternative method should still be relevant for stable training of energy-based models.

C Latent Space Energy-Based Prior Models

In this section, we first briefly review the latent space energy-based prior models (LEBMs) and its variants: CD-LEBM, SM-LEBM, and ED-LEBM. We then proceed to the experimental details.

C.1 A Brief Review of LEBMs

Latent space energy-based prior models (Pang et al., 2020) seek to model latent variable models p ϕ,θ ( x ) = ∫ p ϕ ( x | z ) p θ ( z )d z with an EBM prior p θ ( z ) = exp( -E θ ( z )) p 0 ( z ) Z θ , where p 0 ( z ) is a base distribution which we choose as standard Gaussian (Pang et al., 2020). LEBMs often perform better than latent variable models with a fixed Gaussian prior like VAEs since the EBM prior is more informative and expressive (Pang et al., 2020, Appendix C). However, training LEBMs is more expensive compared to latent variable models with fixed Gaussian prior because of the cost of training for energy-based models. This motivates to explore various training strategies for EBMs such as contrastive divergence, score matching, and the proposed energy discrepancy, where we find that energy discrepancy is the most efficient in terms of computational complexity.

The parameter update for the LEBM can be derived from maximum-likelihood estimation of p ϕ,θ ( x ) . Using the identity E p ϕ,θ ( z | x ) [ ∇ ϕ,θ log p ϕ,θ ( z | x )] =0 , the gradient of the log-likelihood of a data point x is given by

The posterior p ϕ,θ ( z | x ) prescribes the latent representation of the data point x . Consequently, in each parameter update, samples are generated from the posterior distribution p ϕ,θ ( z | x ) via running Langevin dynamics and are treated as data on latent space. The generator is then updated via

Similarly, the maximum-likelihood update for the EBM parameters θ is given by ∇ θ log p ϕ,θ ( x ) = E p ϕ,θ ( z | x ) [ ∇ θ log p θ ( z )] . As with any EBM, this gradient can not be used, directly, since this would require a tractable normalisation constant Z θ . To make this update tractable, we replace the gradient of the log-likelihood with contrastive divergence, score matching, and energy discrepancy as outlined below.

Algorithm 4 Learning latent space energy-based prior models

1: repeat

2: Sample training data points { x i } N i =1 ∼ p data ( x )

3: For each x i , sample the corresponding latent variable z i ∼ p ϕ,θ ( z | x i ) via z i k +1 = z i k + ϵ 2 ∇ z log p ϕ,θ ( z | x i ) + √ ϵ ω k , ω k ∼ N (0 , I ) , z i 0 ∼ p 0 ( z )

by maximizing log-likelihood

5: Update parameter α by running Algorithms 1, 2, or 3 θ ← θ -η θ ∇ θ L CD, SM, or ED ( θ )

4: Update parameter ϕ ϕ ← ϕ + η ϕ ∇ ϕ 1 N ∑ N i =1 log p ϕ ( z i | x i )

6: until convergence of parameters ( ϕ, θ )

CD-LEBM (Pang et al., 2020). The contrastive divergence update is obtained as per usual by expressing the gradient of the log likelihood in terms of the energy function

∇ θ log p ϕ,θ ( x ) = E p ϕ,θ ( z | x ) [ ∇ θ log p θ ( z )] = E p θ ( z ) [ ∇ θ E θ ( z )] -E p ϕ,θ ( z | x ) [ ∇ θ E θ ( z )] . Therefore, the EBM prior can be learned by minimising

Note that optimizing CD-LEBM is computationally expensive, as training the EBM prior requires simulating Langevin dynamics to sample z from p ϕ,θ ( z | x ) to generate positive samples and p θ ( z ) to generate negative samples.

SM-LEBM. The second solution is to minimize the Fisher divergence between the posterior and prior, which has the following form

This is equivalent to score matching (Hyvärinen & Dayan, 2005) when p ϕ,θ ( z | x ) is treated as parameter independent data distribution. We refer to this approach as score-matching LEBM, in which the EBM prior is learned by minimising

where the parameters of p ϕ,θ ( z | x ) are suppressed in the update. Note that score matching generally requires computing the Hessian of the log density as in but in score-matching LEBM, we have ∇ z log p ( z | x ) = ∇ z log p ( x | z ) + ∇ z log p ( z ) .

ED-LEBM. Finally, the EBM prior can be learned by minimising the energy discrepancy between the posterior and the EBM prior with ˜ E θ ( z ):= E θ ( z ) -log p 0 ( z ) , which can be estimated as follows

with z i ∼ p ϕ,θ ( z | x i ) . Note that energy discrepancy does not require simulating MCMC sampling on the EBM prior and calculating the score of the log density, which is computationally friendly for large-scale training. It is critical to include the base distribution p 0 ( z ) in the energy function ˜ E θ . We summarize the training process of the EBM prior using CD-, SM-, and ED-LEBM in Algorithms 1, 2, and 3, with the training procedure of LEBM given in Algorithm 4.

(d) EBM prior

C.2 Langevin Sampling, Reconstruction, and Generation

To sample from the EBM prior p θ ( z ) and posterior p ϕ,θ ( z | x ) we employ a standard unadjusted Langevin sampling routine, i.e. we repeat for k = 0 , 1 , . . . , K

where z 0 ∼ p 0 ( z ) and the distribution p ( z ) is replaced by the prior or posterior densities, respectively.

The generator is modelled as the Gaussian p ϕ ( x | z ) = N ( µ ϕ ( z ) , σ 2 I ) . In reconstruction of x , we sample from the posterior z x ∼ p ϕ,θ ( z | x ) and compute the reconstruction as ˆ x = µ ϕ ( z x ) . In data generation, we sample from the EBM prior z gen ∼ p θ ( z ) and compute the generated synthetic data point as x gen = µ ϕ ( z gen ) .

C.3 Experimental Details of LEBMs

Datasets. We use the following datasets in image modelling: SVHN (Netzer et al., 2011), CIFAR10 (Krizhevsky et al., 2009), and CelebA (Liu et al., 2015). SVHN is of resolution 32 × 32 , and containts 73 , 257 training images and 26 , 032 test images. CIFAR-10 consists of 50 , 000 training images and 10 , 000 test images with a resolution of 32 × 32 . For CelebA, which contains 162 , 770 training images and 19 , 962 test images, we follow the pre-processing step in (Pang et al., 2020), taking 40 , 000 examples of CelebA as training data and resizing it to 64 × 64 . In anomaly detection, we follow the setting in (Zenati et al., 2018) and the dataset can be found in their published code 5 .

Model Architectures. We adopt the same network architecture used in CD-LEBM (Pang et al., 2020), with the details depicted in Table 3, where convT( n ) indicates a transposed convolutional operation with n output channels. We use Leaky ReLU as activation functions and the slope is set to be 0 . 2 and 0 . 1 in the generator and EBM prior, respectively.

Details of Training and Inference. Here, we provide a detailed description of the hyperparameters setup for ED-LEBM. Following (Pang et al., 2020), we utilise Xavier normal (Glorot & Bengio, 2010)

to initialise the parameters. For the posterior sampling during training, we use the Langevin sampler with step size of 0 . 1 and run it for 20 steps for SVHN and CelebA, and 40 steps on CIFAR-10. We set t = 0 . 25 , M = 16 , w = 1 throughout the experiments. The proposed models are trained for 200 epochs using the Adam optimizer (Kingma & Ba, 2015) with a fixed learning 0 . 0001 for the generator and 0 . 00005 for the EBM prior. We choose the largest batch size from { 128 , 256 , 512 } such that it can be trained on a single NVIDIA-GeForce-RTX-2080-Ti GPU. In test time, we observed that slightly increasing the number of Langevin sampler steps can improve reconstruction performance. Therefore, we choose 100 steps with a step size of 0 . 1 for posterior sampling. Based on the insights gained from the MCMC diagnostic presented in Figure 21, we choose 500 steps with a step size of 0 . 2 to ensure convergence of the Langevin dynamics when sampling from the EBM prior.

Evaluation Metrics. In image modelling, we use FID and MSE to quantitatively evaluate the quality of the generated samples and reconstructed images. On all datasets the FID is computed based on 50 , 000 samples and the MSE is computed on the test set. Following (Zenati et al., 2018; Pang et al., 2020), we report the performance using AUPRC in anomaly detection and results are averaged over last 10 epochs to account for variance.

D Additional Experimental Results

In this section we present additional experimental results to highlight the effect of various parameters of our approach.

D.1 Experimental Setup for Figure 1 (Healing the nearsightedness of score-matching)

Amajor problem of score-based methods is their nearsightedness, which refers to their inability to capture global properties of a distribution with disjoint supports such as the mixture weights of two well-separated modes (Zhang et al., 2022). In sight of Theorem 2, energy discrepancy should alleviate this problem as it implicitly compares the scores of both distributions at multiple noise-scales. Following Zhang et al. (2022), we investigate this by computing energy discrepancy as a function of the mixture weight ρ for the mixture of two Gaussians g 1 := N ( -5 , 1) and g 2 := N (5 , 1) , i.e.,

where the true data has the mixture weight ρ = 0 . 2 . We compare energy discrepancy L t,M =32 ,w =1 ( ρ ) ≈ ED( p ρ =0 . 2 , log p ρ ) with the objective of maximum likelihood estimation MLE( ρ ) := E p ρ =0 . 2 ( x ) [ -log p ρ ( x )] and the score matching objective which here is given by the Fisher divergence SM( ρ ) := 1 2 E p ρ =0 . 2 ( x ) [ ∥∇ x log p ρ =0 . 2 ( x ) -∇ x p ρ ( x ) ∥ 2 2 ] . The losses as functions of ρ are shown in Figure 1. We find that energy discrepancy is convex as a function of the mixture weight and approximates the negative log-likelihood as t increases. Consequently, energy discrepancy can capture the mixture weight well for sufficiently large values of t . SM, on the other hand, is a constant function and is blind to the value of the mixture weight.

To further investigate the impact of t and M on the efficiency of energy discrepancy, we minimise the energy discrepancy loss L t,M =32 ,w =1 ( ρ ) as a function of the scalar parameter ρ for various choices of M and t . We compute the mean-square error of 50 independent estimated mixture weights for choice of t and M . As shown in Figure 10, the estimation performance approaches that of the maximum likelihood estimator as t increases, which verifies the statement in Theorem 2. Moreover, if the number of samples M used to estimate the contrastive potential is increased, the estimation performance can be further increased towards the mean-square error of the maximum-likelihood estimator.

3

2

1

0

1

2

3

x

0

10

20

30

40

50

estimated energy

3

2

1

0

1

2

3

x

0

10

20

30

40

50

estimated energy

3

2

1

0

1

2

3

x

0

10

20

30

40

50

estimated energy

0

10

20

30

40

50

epochs

600

500

400

300

200

100

0

loss (ED with w = 0)

D.2 Experimental Setup for Figure 2 (Understanding the w -stablisation)

To probe our interpretation of the w -stablisation, we train a neural-network to learn the energy function using 4 , 096 data points of a one-dimensional standard Gaussian p data ( x ) ∝ exp( -x 2 / 2) . The neural network uses an input layer, a hidden linear layer of width two R 2 → R 2 , and a scalar output layer R 2 → R with a Sigmoid Linear Unit activation between the layers. This neural network has sufficient capacity to model the Gaussian data as well as degenerate energy functions that illustrate potential pitfalls of energy discrepancy for w = 0 . The energy discrepancy is set up with hyperparameters M = 4 , t = 1 , and w ∈ { 0 , 0 . 05 , 0 . 25 , 2 . } and is trained for 50 epochs with Adam. Our results are shown in Figure 2 which confirms the relevance of the w -stablisation to obtain a stable optimisation of energy discrepancy. We remark here that the degenerate case w = 0 is not strictly reproducible. Different types of lacking smoothness of the energy-function at the edge of the support lead to diverging loss values. We chose a result that illustrates the best the theoretical exposition of the w -stablisation in Appendix B.1 and refer to Figure 11 to reflect other malformed estimated energies as well as an example of a diverging loss history.

D.3 Energy Discrepancy on the Discrete Space { 0 , 1 } d

Energy discrepancies are, in principle, well-defined on discrete spaces. To illustrate this point, we describe the energy-discrepancy loss for { 0 , 1 } d valued data such as images with binary pixel values, in which case the discrete energy-discrepancy is straight forward to implement. We will replace the Gaussian transition density with a Bernoulli distribution. For ε ∈ (0 , 1) , let ξ ∼ Bernoulli( ε ) d . Then the transition y = x + ξ mod(2) is symmetric and induces a symmetric transition density q ( y -x ) . Because of the symmetry, the energy discrepancy can be implemented in the same way as in the continuous case, i.e.

with ξ i , ξ ′ i,j ∼ Bernoulli and where ⊕ is the addition modulo 2. To test the effectiveness of energy discrepancy with Bernoulli perturbation, we test energy discrepancy on various discrete settings. In Figure 12, we estimate the symmetric connectivity matrix of Ising models following the exposition in Grathwohl et al. (2021). We then learn the densities of two-dimensional synthetic data sets that were mapped to the space { 0 , 1 } 32 via grey codes in Figure 13, following Dai et al. (2020). Finally, we estimate the densities of image data with binary pixel values for the static and dynamic MNIST dataset, the omniglot dataset, and the caltech silhouettes dataset. As shown in Figure 14, energy discrepancy performs competitively to contrastive divergence. We propose other potential discrete perturbations in Schröder et al. (2023).

了

T

ED

SM

CD

ED

SM

CD

ED

SM

CD

Data

Data

Data

ED

SM

ED

SM

CD

Data

Data

Data

D.4 Additional Density Estimation Results

Here, we provide additional details and results on the density estimation experiments.

Details of Training and Inference. Our choice for the energy-net for density estimation is a 4 -layer feed-forward neural network with 128 hidden units and softplus activation function. In the context of energy discrepancy, we select t = 1 , M = 4 , and w = 1 as hyperparameters. For the contrastive divergence approach, we utilise CD-1, in which the gradient of the log-likelihood in Equation (1) is estimated by employing a Langevin sampler with a single step and a step size of 0 . 1 . For score matching, we train EBMs using the explicit score matching in (2), where the Laplacian of the score is explicitly computed. We train the model using the Adam optimizer with a learning rate of 0 . 001 and iterations of 50 , 000 . After training, synthesised samples are drawn by simulating Langevin dynamics with 100 steps and a step size of 0 . 1 .

Additional Experimental Results. The additional results depicted in Figure 15 demonstrate the strong performance of energy discrepancy on various toy datasets, consistently yielding accurate energy landscapes. In contrast, contrastive divergence consistently produces flattened energy landscapes. Despite the success of score matching in these toy examples, score matching struggles to effectively learn distributions with disjoint support which can be seen in the results in Figure 3.

Comparison with Denoising Score

Matching We further compare energy discrepancy with denoising score matching (DSM) (Vincent, 2011). Specifically, we set w = 1 , M = 4 and experiment with various t . As shown in Figure 16, DSM fails to work when the noise scale is too large or too small. This is because DSM is a biased estimator which is optimised for p θ ∗ ( y ) = ∫ γ t ( y -x ) p data ( x )d x . In contrast, energy discrepancy is

ED:√t=10.0

ED:√t=1.0

ED:Vt=0.1

ED:Vt=0.05

ED:Vt=0.01

Estimated

density

DSM:vt=10.0

DSM:Vt=1.0

DSM:Vt=0.1

DSM:vt=0.05

DSM:Vt=0.01

Estimated

density

more robust to the choice of t since energy discrepancy considers all noise scales up to t simulta-

neously and has an unique optimum p θ ∗ ( x ) = p data ( x ) . However, in the case that t is large and M is small, estimation with energy discrepancy deteriorates due to high variance of the estimated loss function. This provides an explanation for the superior performance of energy discrepancy at √ t = 1 compared to √ t = 10 . Further ablation studies are presented in Figure 23.

D.5 Additional Image Modelling Results

Additional Image Generation and Reconstruction Results. Figures 17 and 18 show additional examples of image generation on CelebA 128 × 128 and image reconstruction on CelebA 64 × 64 . The images are computed through the sampling process outlined in Appendix C.2.

Additional Image Interpolation and Manipulation Results. Figures 19, 20 and 22 show additional results of image interpolation and manipulation on CelebA 64 × 64 . Note that there are two types of interpolations: posterior interpolation and prior interpolation. For posterior interpolation, we consider two real images x 1 and x 2 from the dataset and perform linear interpolation among their corresponding latent variables z 1 ∼ p ϕ,θ ( z | x 1 ) and z 2 ∼ p ϕ,θ ( z | x 2 ) . For prior interpolation, we apply linear interpolation between z 1 ∼ p θ ( z ) and z 2 ∼ p θ ( z ) .

Long-run MCMC Diagnostics. Figure 21 depicts several convergence diagnostics for long-run MCMC on the EBM prior, where we simulate Langevin dynamics with a large number of steps ( 2 , 000 ). Firstly, the energy profiles converge at approximately 250 steps, as demonstrated in Figure 21a, and the quality of the synthesized samples improves as the number of steps increases. Secondly, we compute the Gelman-Rubin statistic ˆ R (Gelman & Rubin, 1992) using 64 chains. The histograms of ˆ R over 5 , 000 × 64 chains are shown in Figure 21b, with a mean of 1 . 08 < 1 . 20 , indicating that the Langevin dynamics have approximately converged. Thirdly, we present auto-correlation results in Figure 21c using 5 , 000 chains, where the mean is depicted as a line and the standard deviation as bands. The auto-correlation decreases to zero within 200 steps, which is consistent with the Gelman-Rubin statistic that assesses convergence across multiple chains.

D.6 Qualitative Results on the Effect of t , M , and w

The hyperparameters t, M, w play important roles in energy discrepancy. Here, we provide some qualitative results to understand their effects. According to Theorem 2, t controls the nearsightedness of energy discrepancy. For small t , energy discrepancy behaves like score matching 1 t ED γ t ( p data , U ) = 1 t ∫ t 0 SM( p s , U s )d s ≈ SM( p data , U ) and is expected to be unable to resolve local mixture weights. This assertion can be confirmed by qualitative results depicted in Figure 23, which show that when t = 0 . 0025 , energy discrepancy fails to identify the weights of components in the 25-Gaussians and pinwheel datasets. For large t , energy discrepancy inherits favourable properties

of the maximum likelihood estimator. While large values of t consequently mitigate problems of nearsightedness, it is worth noting that energy discrepancy may encounter issues with high variance when t become excessively large. In such situations, it is necessary to consider increasing the value of M to reduce the variance.

We also investigate the effect of w in Figure 24. As pointed out by the analysis in Appendix B.1, w serves as a stabilises training of energy based models with energy discrepancy. Based on our experimental observations, when w = 0 and M is small ( e.g., M ≤ 128 in the 25-Gaussians dataset and M ≤ 32 in the pinwheel dataset), energy discrepancy exhibits rapid divergence within 100 optimisation steps and fails to converge in the end. If, however, w is increased, e.g. to 1 , energy discrepancy shows stable convergence even with M = 1 . This property is highly appealing as it significantly reduces the computational complexity. Additionally, we find in Figure 2 that larger w tends to result in a flatter estimated energy landscapes which aligns with our intuition gained in Appendix B.1.

7

B

5

7

16

S

5

10:0

5

1050

10

36

10

456

4

28

58

28

2

5

17

5

17

Right: reconstructed images by sampling from the posterior.

555

0

250

500

750

1000

1250

1500

1750

2000

Iterations

600

500

400

300

200

100

0

Energy Profile

(a) Energy Profile

1.000

1.025

1.050

1.075

1.100

1.125

1.150

1.175

1.200

R

0

20

40

60

80

100

(b) Gelman-Rubin

0

200

400

600

800

1000

Lag

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Correlation

(c) Auto-correlation

T

(a) Smiling

(b) Male

(c) Eyeglasses

(d) Blond Hair

M=1

M=2

M=4

M=8

M=16

M=32

M=64

M=128

M=256

M=512

Vt = 0.05

.00

=

Vt = 50.0

M=1

M=2

M=4

M=8

M=16

M=32

M=64

M=128

M=256

M=512

Vt = 0.05

5555555555

1.00

5555555555

√t=

Vt =50.0

SSSSS

M=1

M=2

M=4

M=8

M=16

M=32

M=64

M=128

M=256

M=512

W=0.05

W=1.00

w=50.0

M=1

M=2

M=4

M=8

M=16

M=32

M=64

M=128

M=256

M=512

SSSS

W=0.05

W=1.00

5555555555

W=50.0

9555555555

Peer reviews:
comments: SUMMARY:
Energy discrepancy is presented as a new loss for the training of EBM. 
ED interpolates between the losses of score matching and maximum-likelihood estimation.
Efficacy of ED on a latent variable energy-based model is demonstrated to tackle manifold hypothesis as an important challenge in the adoption of likelihood-based training.
Extensive numerical experiments are done to show ED's superiority over contrast divergence and score matching.

STRENGTHS:
1. Theoretical derivation is rigorous.
2. Numerical experiments are solid.
3. Paper is in-general well written.
4. The proposed algorithm is easy to implement.

WEAKNESSES:
No obvious weakness to me.

QUESTIONS:
How important is the w-stabilization term for the numerical experiments?  If w is small(close to 0), are numerical experiments results still hold?

LIMITATIONS:
No negative social impact is seen.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper has proposed a new loss funciton, i.e., Energy Discrepancy to train energy-based models without computaiton of MCMC. The proposed loss function could be directly derived from the energy function without relying on MCMC samples.\n\nSTRENGTHS:\nThe proposed energy discrepancy could be directly computed from the energy function and alleviate the problem of nearsightedness of score matching and approximates maximum-likelihood estimation. The experiments show that the energy discrepancy could achieve better performance than score matching and contrastive divergence in image generation.\n\nWEAKNESSES:\nI think this is very good paper that proposes a novel optimization method in enery-based models. I only have several questions about the experiment part. Pleare refer to the question section.\n\nQUESTIONS:\n1. Model in [1] utlizes MCMC-based maximum likelihood algorithm to train a normalizing flow model for image generation. Could the proposed energy discrepancy used in this model?\n2. According to Table 1, the performance of CD-LEBM and the proposed ED-LEBM are quite closed. The gap is much smaller than the experiments in density estimation.\n3. Are there any comparison with GAN-based models?\n4. For comparison with score-based methods, [2] shows a much better performance in FID on Cifar10. Could the author give some discussion about these results as the author has claimed that the proposed energy discrepancy has advantages of score matching?\n\nReference:\n\n[1] Xie, Jianwen, et al. \"A Tale of Two Latent Flows: Learning Latent Space Normalizing Flow with Short-run Langevin Flow for Approximate Inference.\" arXiv preprint arXiv:2301.09300 (2023).\n\n[2] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" arXiv preprint arXiv:2011.13456 (2020).\n\nLIMITATIONS:\nPlease refer to the question section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new loss function for training energy-based models, called energy discrepancy (ED). ED does not rely on score functions and MCMC samples. Instead, it is defined as the difference between the energy function of data and some conditional samples. They prove that optimizing this objective function yields the appropriate energy function. Then, they build connections between ED, score matching and contrastive divergence. Finally, this paper focuses on training the latent energy-based prior model, which is a VAE with EBM prior.\n\nFor experiments, the authors first showcase the density estimation on several 2D pdfs. ED outperforms SM and CD on pdf estimation and sampling. Then, they train latent EBMs on SVHN, CIFAR-10 and CelebA. ED also outperforms SM and CD on image reconstruction/generating and out-of-distribution detection.\n\nSTRENGTHS:\n1. Training EBMs is an important problem in generative modeling. This paper proposes a new training criteria, which does not rely on possibly ill-conditioned score functions and time-consuming MCMC samples.\n\n2. The authors also build connections between the proposed method and score-matching estimates or MLEs.\n\nWEAKNESSES:\n1. The authors do not train EBMs directly on the data space. Instead, they train a VAE with a EBM prior. Since CD or SM can lead to competitive EBMs, I am wondering if there is any empirical result on training EBMs directly on the data space using the proposed loss?\n\n2. Since the proposed loss function is similar to the conditional NCE, I think one promising baseline could be training the same model using CNCE loss.\n\n3. The authors said ED provides fast training (in Line 338). Is there any comparison about the running time?\n\nQUESTIONS:\n1. In the Equation between Line 187 and 188, is there $E_\\theta(x^i)$ missing behind $w / M$?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new loss function for training energy-based models, which they dub \"energy discrepancy.\"  The aim is to provide a viable alternative to contrastive divergence and score matching based methods that suffer from near-sightedness -- these approaches lack global information and can have difficulties fitting well-separated Gaussians.  The energy discrepancy method seeks to overcome this difficulty by perturbing the distributions to increase the mass of the low probability regions separating the peaks of the distribution.  This is done at different noise levels and then integrated.  The proposed loss is theoretically justified and then validated experimentally in both synthetic and real-world settings.\n\nSTRENGTHS:\nThe approach is well-motivated for the most part and the presentation is mostly clear.  The idea is novel to me and experimentally improves over existing approaches.\n\nWEAKNESSES:\nI'm not certain that the approach is potentially as practical as it appears.  My main concern is that the w-stabilisation procedure seems to be doing a lot of heavy lifting and that is tailored to the Gaussian case.  Given other discussion in this work, I don't think this is a huge problem.  However, the case for acceptance would be bolstered if a broader set of applications were considered.\n\nQUESTIONS:\n-  There is a bit of magic in the w-stabilisation procedure.  I read the explanation in Appendix B, but I feel that more explanation is needed in the main text.  In particular, does this approach generalize beyond the Gaussian case?  This makes me think that the proposed approach is really quite specific (at least if you want to be able to do it in practice).    Does this need to be included as a limitation?\n\n-  Is it possible to include some discrete setting experiments as well?  \n\nMinor typos:\n\n- \"normalisation of EBMs, also known as partition function\" sounds funny.  Maybe \"normalisation constant\"?\n- The sentence in lines 30-33 doesn't really make sense.\n- \"i.e.\" and \"e.g.\" must always be followed by a comma.\n- \"line out in\" -> \"outline in\"?\n- Lots of \"which\" usage that I think requires a comma.\n\nLIMITATIONS:\nYes, but see above comments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider the problem of training energy-based models. Due to the limitations of two current approaches known as approximate maximum likelihood methods (which might lead to malformed estimators of the energy function) and score-based methods (which fail to resolve global features in the distribution without big data), the authors propose a novel loss function for this task named Energy Discrepancy (ED) to overcome those issues with rigorous theoretical guarantee. Moreover, they also conduct several experiments to demonstrate the efficacy of the ED loss function in learning low-dimensional data distributions compared to two previous methods. However, the authors point out that their approach does not work for high-dimensional data due to some manifold hypothesis.\n\nSTRENGTHS:\n1. Originality: Energy Discrepancy (ED) is a novel loss for training enerygy-based models.\n\n2. Quality: Theoretical results are solid and associated with rigorous proof, but I do not spend time double-checking all of them. Additionally, a lot of experiments are carried out to empirically justify the effectiveness of the proposed loss function ED. This significantly strengthens the contributions of the paper. \n\n3. Clarity: The paper is well written and organized, which makes it easy to follow.\n\n4. Significance: Although the ED loss function work for only low-dimensional data distributions, this work lays the basis for alleviating the issues of maximum likelihood methods and score-based methods for training energy-based models.\n\nWEAKNESSES:\n1. Clarity:  There are some places that the authors introduce results without necessary intuitions. For instance, in line 96, they directly suggest using Gaussian kernels without explanation.\n\nQUESTIONS:\n1. In equation (2), why is the score-matching loss defined between $p_{\\text{data}}$ and $E_{\\theta}$ rather than between $p_{\\text{data}}$ and $p_{\\theta}$?\n\n2. In the beginning of section 3, can we use other kernels rather than Gaussian kernels to perturb $p_{\\text{data}}$ and $p_{\\theta}$? \n\n3. In equation (5), is the energy discrepancy $ED_q$ a proper metric?\n\n4. In Theorem 2, the authors should either briefly introduce the definition of Wasserstein distance or cite relevant papers to that distance so that readers from different communities can understand.\n\nLIMITATIONS:\nThe limitations are discussed in Section 7 of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}