SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tobias Schröder<sup>1*</sup> Zijing Ou<sup>1†</sup> Jen Ning Lim<sup>2</sup>, Yingzhen Li<sup>1</sup> Sebastian J. Vollmer<sup>3</sup> Andrew B. Duncan<sup>1,4</sup>  
<sup>1</sup>Imperial College London 2University of Warwick 3DFKI and RPTU Kaiserslautern 4The Alan Turing Institute {t.schroeder21, z.ou22, yingzhen.li, a.duncan}@imperial.ac.uk jen-ning.lim@warwick.ac.uk sebastian.vollmer@dfki.de
Energy-based models are a simple yet powerful class of probabilistic models, but their widespread adoption has been limited by the computational burden of training them. We propose a novel loss function called Energy Discrepancy (ED) which does not rely on the computation of scores or expensive Markov chain Monte Carlo. We show that energy discrepancy approaches the explicit score matching and negative log-likelihood loss under different limits, effectively interpolating between both. Consequently, minimum energy discrepancy estimation overcomes the problem of nearsightedness encountered in score-based estimation methods, while also enjoying theoretical guarantees. Through numerical experiments, we demonstrate that ED learns low-dimensional data distributions faster and more accurately than explicit score matching or contrastive divergence. For high-dimensional image data, we describe how the manifold hypothesis puts limitations on our approach and demonstrate the effectiveness of energy discrepancy by training the energy-based model as a prior of a variational decoder model.
Energy-Based Models (EBMs) are a class of parametric unnormalised probabilistic models of the general form  $p_{\mathrm{ebm}} \propto \exp(-U)$  originally inspired by statistical physics. EBMs can be flexibly modelled through a wide range of neural network functions which, in principle, permit the modelling of any positive probability density. Through sampling and inference on the learned energy function, the EBM can then be used as a generative model or in numerous other downstream tasks such as improving robustness in classification or anomaly detection (Grathwohl et al., 2020), simulation-based inference (Glaser et al., 2022), or learning neural set functions (Ou et al., 2022).
Despite their flexibility, EBMs are limited in machine learning applications by the difficulty of their training. The normalisation constant of EBMs, also known as the partition function, is typically intractable making standard techniques such as Maximum Likelihood Estimation (MLE) infeasible. For this reason, EBMs are commonly trained with an approximate maximum likelihood method called Contrastive Divergence (CD) (Hinton, 2002) which approximates the gradient of the log-likelihood using short runs of a Markov chain Monte Carlo (MCMC) method. However, contrastive divergence with short run MCMC leads to malformed estimators of the energy function (Nijkamp et al., 2020a), even for relatively simple restricted Boltzmann-machines (Carreira-Perpina & Hinton, 2005). This can, in part, be attributed to the fact that contrastive divergence is not the gradient of any fixed
*Correspondence to: Tobias Schröder, t.schroeder21@imperial.ac.uk
†Code: https://github.com/J-zin/energy-discrepancy
objective function (Sutskever & Tieleman, 2010), which severely limits the theoretical understanding of CD and motivated various adjustments of the algorithm (Du et al., 2021; Yair & Michaeli, 2021).
Score-based methods such as Score Matching (SM) (Hyvärinen & Dayan, 2005; Vincent, 2011; Song et al., 2020) and Kernel Stein Discrepancy (KSD) estimation (Liu et al., 2016; Chwialkowski et al., 2016; Gorham & Mackey, 2017; Barp et al., 2019) are a family of competing approaches which offer tractable loss functions and are, by construction, independent of the normalising constant of the distribution. However, such methods suffer from nearsightedness as they fail to resolve global features in the distribution without vast amounts of data. In particular, both SM and KSD estimators are unable to capture the mixture weights of two well-separated Gaussians (Song & Ermon, 2019; Zhang et al., 2022; Liu et al., 2023).
We propose a new loss functional for energy-based models called Energy Discrepancy (ED) which compares the data distribution and the energy-based model via two contrasting energy contributions. By definition, energy discrepancy only depends on the energy function and is independent of the scores or MCMC samples from the energy-based model. In our theoretical section, we show that this leads to a loss functional that can be defined on general measure spaces without Euclidean structure and demonstrate its close connection to score matching and maximum likelihood estimation in the Euclidean case. In our practical section, we focus on a simple implementation of energy discrepancy on Euclidean space which requires less evaluations of the energy-based model than the parameter update of contrastive divergence or score-matching. We demonstrate that the Euclidean energy-discrepancy alleviates the problem of nearsightedness of score matching and approximates maximum-likelihood estimation with better theoretical guarantees than contrastive divergence.
On high-dimensional image data, energy-based models face the additional challenge that under the manifold hypothesis (Bengio et al., 2013), the data distribution is not a positive probability density and does, strictly speaking, not permit a representation of the form of an energy-based model. Energy discrepancy is particularly sensitive to singular data supports and requires the transformation of the data distribution to a positive density. We approach this problem by training latent energy-based priors (Pang et al., 2020) which employ a lower-dimensional latent representation in which the data distribution is positive.
Our contributions are the following: 1) We present energy discrepancy, a new estimation loss for the training of energy-based models that can be computed without MCMC or spatial gradients of the energy function. 2) We show that, as a loss function, ED interpolates between the losses of score matching and maximum-likelihood estimation and overcomes the nearsightedness of score-based methods. 3) We introduce a novel variance reduction trick called  $w$ -stabilisation that drastically reduces the computational cost of approximating energy discrepancy stably.
2 Training of Energy-Based Models
In the following, let  $p_{\mathrm{data}}(\mathbf{x})$  be an unknown data distribution which we are trying to estimate from independently distributed data  $\{\mathbf{x}^i\} \sim p_{\mathrm{data}}$ . Energy-based models are parametric distributions of the form
$$
p _ {\theta} (\mathbf {x}) \propto \exp (- E _ {\theta} (\mathbf {x}))
$$
for which we want to find the scalar energy function  $E_{\theta}$  such that  $p_{\theta} \approx p_{\mathrm{data}}$ . Typically, energy-based models are trained with contrastive divergence which estimates the gradient of the log-likelihood
$$
\nabla_ {\theta} \log p _ {\theta} (\mathbf {x}) = \mathbb {E} _ {p _ {\theta} (\mathbf {y})} [ \nabla_ {\theta} E _ {\theta} (\mathbf {y}) ] - \nabla_ {\theta} E _ {\theta} (\mathbf {x}). \tag {1}
$$
using Markov chain Monte Carlo methods to approximate the expectation for  $p_{\theta}$ . For computational efficiency, the Markov chain is only run for a small number of steps. As a result, contrastive divergence does not learn the maximum-likelihood estimator and can produce malformed estimates of the energy function (Nijkamp et al., 2020a).
Alternatively, the discrepancy between data distribution and energy-based model can be measured by comparing their score functions  $\nabla_{\mathbf{x}}\log p_{\mathrm{data}}$  and  $\nabla_{\mathbf{x}}\log p_{\theta}$  which, by definition, are independent of the normalising constant. The comparison of the scores is achieved with the Fisher divergence. After applying an integration-by-parts and discarding constants with respect to  $\theta$ , this leads to the score-matching loss (Hyvarinen & Dayan, 2005)
$$
\operatorname {S M} \left(p _ {\text {d a t a}}, E _ {\theta}\right) := \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \left[ - \Delta_ {\mathbf {x}} E _ {\theta} (\mathbf {x}) + \frac {1}{2} \| \nabla_ {\mathbf {x}} E _ {\theta} (\mathbf {x}) \| ^ {2} \right]. \tag {2}
$$
As this only requires the computation of expectations with respect to  $p_{\mathrm{data}}$ , the requirement that the data distribution attains a density can be relaxed, yielding a loss function for  $p_{\theta}$  which can be readily approximated. Score-based methods are nearsighted as the score function only contributes local information to the loss. In a mixture of well-separated distributions, the score matching loss decomposes into a sum of objective functions that only see the local mode and are not capable of resolving the weights of the mixture components (Song & Ermon, 2019; Zhang et al., 2022).
3 Energy Discrepancies
To illustrate the idea behind the proposed objective function, we start by motivating energy discrepancy from the perspective of explicit score matching. In the following, we will denote the EBM as  $p_{\mathrm{ebm}} \propto \exp(-U)$ , where  $U$  is the energy function that is learned. The nearsightedness of score matching arises due to the presence of large regions of low probability which are separating the modes of the data distribution. To increase the probability mass in these regions, we follow (Zhang et al., 2020) and perturb  $p_{\mathrm{data}}$  and  $p_{\mathrm{ebm}}$  through a convolution with a Gaussian kernel  $\gamma_s(\mathbf{y} - \mathbf{x}) \propto \exp(-\|\mathbf{y} - \mathbf{x}\|^2 / 2s)$ , i.e.
$$
p _ {s} (\mathbf {y}): = \int \gamma_ {s} (\mathbf {y} - \mathbf {x}) p _ {\mathrm {d a t a}} (\mathbf {x}) \mathrm {d} \mathbf {x},
$$
$$
\exp (- U _ {s} (\mathbf {y})) := \int \gamma_ {s} (\mathbf {y} - \mathbf {x}) \exp (- U (\mathbf {x})) \mathrm {d} \mathbf {x}.
$$
Figure 1: Loss of energy discrepancy, score matching, and maximum likelihood estimation on the task of estimating the weight in a mixture of two Gaussians. For details, see Appendix D.1.
The resulting perturbed divergence  $\widetilde{\mathrm{SM}}(p_{\mathrm{data}}, U) \coloneqq \mathrm{SM}(p_s, U_s)$  retains its unique optimum at  $\exp(-U^*) \propto p_{\mathrm{data}}$  (Zhang et al., 2020) but alleviates the nearsightedness as the data distribution is more spread out. The perturbation with  $\gamma_s$  simultaneously makes the two distributions more similar which comes at a potential loss of discriminatory power. We mitigate this by integrating the score matching objectives over  $s$  over an interval of noise-scales  $[0, t]$ . It turns out that this integral can be evaluated as the difference of two contrasting energy-contributions:
$$
\int_ {0} ^ {t} \operatorname {S M} \left(p _ {s}, U _ {s}\right) \mathrm {d} s = \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} [ U (\mathbf {x}) ] - \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {t} \left(\mathbf {x} _ {t} - \mathbf {x}\right)} \left[ U _ {t} \left(\mathbf {x} _ {t}\right) \right]. \tag {3}
$$
The proof is given in Appendix A.3. The contrasting expression on the right-hand is now independent of the score and normalisation of the EBM. We argue that such constructed objective functions are useful losses for energy-based modelling.
3.1 A contrastive approach to learning the energy
We lift the idea of learning the energy-based distribution through the contrast of two energy contributions to a general estimation loss called Energy Discrepancy. We will show that energy discrepancy can be defined independently of an underlying perturbative process and is well-posed even on non-Euclidean measure-spaces:
Definition 1 (Energy Discrepancy). Let  $p_{\mathrm{data}}$  be a positive density on a measure space  $(\mathcal{X}, \mathrm{d}\mathbf{x})^1$  and let  $q(\cdot|\mathbf{x})$  be a conditional probability density. Define the contrastive potential induced by  $q$  as
$$
U _ {q} (\mathbf {y}) := - \log \int q (\mathbf {y} | \mathbf {x}) \exp (- U (\mathbf {x})) \mathrm {d} \mathbf {x}. \tag {4}
$$
We define the energy discrepancy between  $p_{\mathrm{data}}$  and  $U$  induced by  $q$  as
$$
\operatorname {E D} _ {q} \left(p _ {\text {d a t a}}, U\right) := \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} [ U (\mathbf {x}) ] - \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {q (\mathbf {y} | \mathbf {x})} [ U _ {q} (\mathbf {y}) ]. \tag {5}
$$
In this paper, we shall largely focus on the case where the data is Euclidean, i.e.  $\mathcal{X} = \mathbb{R}^d$ , and the base-distribution  $\mathrm{dx}$  is the standard Lebesgue measure. However, this framework also admits  $\mathcal{X}$  being discrete spaces like spaces of graphs, or continuous spaces with non-trivial base measures such as manifolds. Specifically, the validity of our approach is characterised by the following non-parametric estimation result:
<sup>1</sup>The integrals should be interpreted as Lebesgue integrals, i.e., if  $\mathcal{X}$  is discrete, the integral will be a sum.
Theorem 1. Let  $p_{\mathrm{data}}$  be a positive probability density on  $(\mathcal{X}, \mathrm{d}\mathbf{x})$ , and let  $q(\cdot|\mathbf{x})$  be a conditional probability density. Under mild assumptions on  $q$  and the set of admissible energy functions  $U$ , energy discrepancy  $\mathrm{ED}_q$  is functionally convex in  $U$  and has, up to additive constants, a unique global minimiser  $U^* = \arg \min \mathrm{ED}_q(p_{\mathrm{data}}, U)$  with  $\exp(-U^*) \propto p_{\mathrm{data}}$ .
The assumption on the conditional density  $q$  describes that  $\mathbf{y} \sim q(\cdot|\mathbf{x})$  involves some loss of information. The assumption that  $p_{\mathrm{data}}$  has full support turns out to be critical when scaling energy-discrepancy to high-dimensional data. The proof of Theorem 1 is given in Appendix A.1.
3.2 Choices for the conditional distribution  $q$
Def. 1 offers a wide range of possible choices for the perturbation distribution  $q$ . In Appendix D.3 we discuss a possible choice in the discrete space  $\{0,1\}^d$ . For the remainder of this paper, however, we will focus on Euclidean data  $\mathbf{x} \in \mathbb{R}^d$  and hope that the generality of our result inspires future work. Our requirements on  $q$  are that simulating from the conditional distribution  $\mathbf{y} \sim q(\mathbf{y}|\mathbf{x})$  and computing the convolution that defines  $U_q$  are numerically tractable. On continuous spaces, a natural candidate for  $q$  is the transition density of a diffusion process which arises as solution to a stochastic differential equations of the form  $\mathrm{d}\mathbf{x}_t = \mathbf{a}(\mathbf{x}_t)\mathrm{d}t + \mathrm{d}\mathbf{w}_t$  with drift  $\mathbf{a}$  and standard Brownian motion  $\mathbf{w}_t$  (see Øksendal, 2003). The conditional density  $q_t(\cdot |\mathbf{x})$  represents the probability density of the perturbed particle  $\mathbf{x}_t$  that was initialised at  $\mathbf{x}_0 = \mathbf{x}$ . The resulting transition density then satisfies both of our requirements by employing the Feynman-Kac formula as we line out in Appendix B.2. Although this approach makes the choice of  $q$  flexible, the following interpolation result stresses that not much is lost when choosing a Gaussian transition density  $\gamma_t \propto \exp(-\|\mathbf{x}-\mathbf{y}\|^2/2t)$ :
Theorem 2. Let  $q_{t}$  be the transition density of the diffusion process  $\mathrm{d}\mathbf{x}_t = \mathbf{a}(\mathbf{x}_t)\mathrm{d}t + d\mathbf{w}_t$ , let  $p_{\mathrm{ebm}} \propto \exp (-U)$  be the energy-based distribution and  $p_t$  the data-distribution convolved with  $q_{t}$ .
1. The energy discrepancy is given by a multi-noise scale score matching loss
$$
\mathrm {E D} _ {q _ {t}} (p _ {\mathrm {d a t a}}, U) = \int_ {0} ^ {t} \mathbb {E} _ {p _ {s} (\mathbf {x} _ {s})} \left[ - \Delta U _ {q _ {s}} (\mathbf {x} _ {s}) + \frac {1}{2} \| \nabla U _ {q _ {s}} (\mathbf {x} _ {s}) \| ^ {2} \right] \mathrm {d} s + \mathrm {c o n s t .}
$$
2. If  $\mathbf{a} = 0$ , i.e.  $q_{t}$  is the Gaussian transition density  $\gamma_{t}$ , the energy discrepancy converges to the loss of maximum likelihood estimation a linear rate in time
$$
\left| \operatorname {E D} _ {\gamma_ {t}} \left(p _ {\text {d a t a}}, U\right) + \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \left[ \log p _ {\text {e b m}} (\mathbf {x}) \right] - c (t) \right| \leq \frac {1}{2 t} \mathbb {W} _ {2} ^ {2} \left(p _ {\text {d a t a}}, p _ {\text {e b m}}\right)
$$
where  $c(t)$  is independent of  $U$  and  $\mathbb{W}_2$  denotes the Wasserstein distance<sup>2</sup>.
Theorem 2 has two main messages: All diffusion-based energy discrepancies behave like a multi-noise scale score matching loss, which is independent of the drift  $\mathbf{a}$ . In fact, we show in Appendix A.2 that for a linear drift  $\alpha \mathbf{x}_t$ , the induced energy discrepancy is always equivalent to the energy discrepancy based on a Gaussian perturbation. Furthermore, estimation with a Gaussian-based energy discrepancy approximates maximum likelihood estimation for large  $t$ , thus enjoying its attractive asymptotic properties provided  $\mathrm{ED}_{\gamma_t}(p_{\mathrm{data}}, U)$  can be approximated with low variance. We demonstrate the result in a mixture model in Figure 1 and Appendix D.1 and give a proof of above theorem in Appendices A.3 and A.4.
Connection to Contrastive Divergence. Due to the generality of our result we can also make a direct connection between energy discrepancy and contrastive divergence. To this end, suppose that for  $\theta$  fixed,  $q$  satisfies the detailed balance relation  $q(\mathbf{y}|\mathbf{x})\exp (-E_{\theta}(\mathbf{x})) = q(\mathbf{x}|\mathbf{y})\exp (-E_{\theta}(\mathbf{y}))$ . In this case, energy discrepancy becomes the loss function
$$
\operatorname {E D} _ {q} \left(p _ {\text {d a t a}}, E _ {\theta}\right) = \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \left[ E _ {\theta} (\mathbf {x}) \right] - \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {q (\mathbf {y} | \mathbf {x})} \left[ E _ {\theta} (\mathbf {y}) \right] \tag {6}
$$
which, after taking gradients in  $\theta$  yields the contrastive divergence update<sup>3</sup>. See Appendix A.5 for details. The non-parametric estimation result from Theorem 1 holds true for the contrastive objective in (6). This means that each step of contrastive divergence optimises an objective function with minimum at  $p_{\mathrm{data}} \approx p_{\theta}$ . However, the objective function is adjusted in each step of the algorithm.
2For a reference on Wasserstein distances, see Peyre & Cuturi (2019)
3The implicit dependence of  $q$  on  $\theta$  is ignored when taking the gradient. Notice that CD is not the gradient of any fixed objective function (Sutskever & Tieleman, 2010).
4 Training Energy-Based Models with Energy Discrepancy
In sight of Theorem 2, we will discuss how to approximate ED from samples for Euclidean data  $\{\mathbf{x}^i\} \subset \mathbb{R}^d$  and a Gaussian conditional distribution  $\gamma_t(\mathbf{y} - \mathbf{x}) \propto \exp(-\|\mathbf{y} - \mathbf{x}\|^2 / 2t)$ . First, the outer expectations on the right-hand of (5) can be computed as plug-in estimators by simulating the Gaussian perturbation  $\mathbf{x}_t^i = \mathbf{x}^i + \sqrt{t}\pmb{\xi}$  for  $\pmb{\xi} \sim \mathcal{N}(0, \mathbf{I})$  and averaging  $\{U(\mathbf{x}^i)\}$  and  $\{U_{\gamma_t}(\mathbf{x}_t^i)\}$ . The critical step is then finding a low-variance estimate of the contrastive potential  $U_{\gamma_t}$  itself.
Due to the symmetry of the Gaussian transition density, we can interpret  $\gamma_{t}$  as the law of a Gaussian random variable with mean  $\mathbf{x}_t^i$  and variance  $t$ , i.e. the conditioned random variable  $\mathbf{x}_t^i + \sqrt{t} \pmb{\xi}' | \mathbf{x}_t^i$  for  $\pmb{\xi}' \sim \mathcal{N}(0, \mathbf{I})$  has the density  $\gamma_{t}(\mathbf{x}_{t}^{i} - \mathbf{x})$ . Hence, we can write  $U_{\gamma_t}$  as the expectation
$$
U _ {\gamma_ {t}} (\mathbf {x} _ {t} ^ {i}) = - \log \mathbb {E} _ {\gamma_ {1} (\pmb {\xi} ^ {\prime})} [ \exp (- U (\mathbf {x} _ {t} ^ {i} + \sqrt {t} \pmb {\xi} ^ {\prime})) | \mathbf {x} _ {t} ^ {i}) ]
$$
for  $i = 1,2,\dots ,N$ . The conditioning expresses that we keep  $\mathbf{x}_t^i$  fixed when taking the expectation with respect to  $\xi^\prime$ . It is then possible to calculate the expectation by sampling  $\xi^{\prime i,j}\sim \mathcal{N}(0,\mathbf{I})$  and calculating the mean as for the outer expectations. However, we find that this approximation is not sufficient as it is biased due to the logarithm and prone to numerical instabilities because of missing bounds on the value of  $U_{\gamma_t}(\mathbf{x}_t^i)$ . To stabilise training, we augment the Monte Carlo estimate of  $U_{\gamma_t}(\mathbf{x}_t^i)$  by an additional term  $w / M\exp (-U(\mathbf{x}^i))$  which we call
Figure 2: Estimated energy functions for Gaussian data with various choices of  $w$ . Increasing  $w$  leads to flatter energy-landscapes, and training becomes unstable if  $w = 0$ . For details, see Appendices B.1 and D.2.
$w$ -stabilisation. This results in the following approximation of the contrastive potential:
$$
U _ {\gamma_ {t}} \left(\mathbf {x} _ {t} ^ {i}\right) \approx - \log \left(\frac {w}{M} \exp (- U \left(\mathbf {x} ^ {i}\right)) + \frac {1}{M} \sum_ {j = 1} ^ {M} \exp (- U \left(\mathbf {x} _ {t} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {\prime i, j}\right))\right) \quad \boldsymbol {\xi} ^ {\prime i, j} \sim \mathcal {N} (0, \mathbf {I})
$$
The  $w$ -stabilisation dampens the contributions of contrastive samples  $\mathbf{x}_t^i + \sqrt{t}\pmb{\xi}^{\prime i,j}$  whose energy is higher than the energy of the data point  $\mathbf{x}^i$ . This provides a deterministic upper bound for the approximate contrastive potential in (4) and reduces the variance of the estimation. We find that the  $w$ -stabilisation drastically reduces the number of samples  $M$  needed for the stable training of deep energy-based models. We illustrate the effect of the stabilisation in Figures 2 and 24 and discuss our reasoning in more details in Appendix B.1. The full loss is now formed for  $U \coloneqq E_{\theta}$  with  $\pmb{\xi}^i \sim \mathcal{N}(0, \mathbf{I})$ ,  $\pmb{\xi}^{\prime i,j} \sim \mathcal{N}(0, \mathbf{I})$  and tunable hyperparameters  $t, M, w$  as
$$
\mathcal {L} _ {t, M, w} (\theta) := \frac {1}{N} \sum_ {i = 1} ^ {N} \log \left(\frac {w}{M} + \frac {1}{M} \sum_ {j = 1} ^ {M} \exp \left(E _ {\theta} \left(\mathbf {x} ^ {i}\right) - E _ {\theta} \left(\mathbf {x} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {\prime i, j}\right)\right)\right).
$$
The loss is evaluated using the numerically stabilised logsumexp function. The justification of the approximation is given by the following theorem:
Theorem 3. Assume that  $\mathbf{x} \mapsto \exp(-E_{\theta}(\mathbf{x}))$  is uniformly bounded. Then, for every  $\varepsilon > 0$  there exist  $N$  and  $M(N)$  such that  $\left|\mathcal{L}_{t,M(N),w}(\theta) - \mathrm{ED}_{\gamma_t}(p_{\mathrm{data}},E_\theta)\right| < \varepsilon$  almost surely.
We give the proof in Appendix B.1. This result forms the basis for proofs of asymptotic consistency of our estimators which we leave for future work. It is noteworthy that an analogous implementation of energy discrepancy for other perturbation kernels and other spaces is possible. For example, we construct a similar loss for discrete spaces using a Bernoulli perturbation in Appendix D.3. Similarly to the Gaussian case, the  $w$ -stabilisation provides a useful variance reduction for the training with Bernoulli-based energy discrepancy.
4.1 Training Energy-Based Models under the Manifold Hypothesis
LeCun (2022) suggests that maximum-likelihood-based training of energy-based models can lead to the formation of undesirable canyon shaped energy-functions. Indeed, energy discrepancy yields an
Figure 3: Comparison of energy discrepancy, score matching and contrastive divergence on density estimation. The 1st and 2nd rows are the estimated density and synthesised samples, respectively.