SlowGuess/NeurIPS_2023_OCR huggingface dataset:
energy with low values on the data-support and rapidly diverging values outside of it when being used on image data, directly. Such learned energies fail to represent the distribution between data-points and are not suitable for inference or image generation. We attribute this phenomenon to the manifold hypothesis, which states that the data concentrates in the vicinity of a low-dimensional manifold (Bengio et al., 2013). In this case, the data distribution is not a positive density and can not be written as an energy-based model as  $\log p_{\mathrm{data}}$  is not well-defined. Additionally, Gaussian perturbations of a data point  $\tilde{\mathbf{x}} \coloneqq \mathbf{x} + \boldsymbol{\xi}$  are orthogonal to the data manifold with high-probability and the negative samples in the contrastive term are not informative.
To resolve this problem, we will work with a lower-dimensional latent representation of the data distribution in which positivity can be ensured. We follow Pang et al. (2020) and define a variational decoder network  $p_{\phi}(\mathbf{x}|\mathbf{z})$  and an energy-based tilting of a Gaussian prior  $p_{\theta}(\mathbf{z}) = \exp(-E_{\theta}(\mathbf{z}))p_{0}(\mathbf{z})$  on latent space, resulting in the so-called latent energy-based prior models (LEBMs)  $p_{\phi,\theta}(\mathbf{x}) \propto \int p_{\phi}(\mathbf{x}|\mathbf{z})p_{\theta}(\mathbf{z})\mathrm{d}\mathbf{z}$ . To obtain the latent representation of data we sample from the posterior  $p_{\phi,\theta}(\mathbf{z}|\mathbf{x}) \propto p_{\phi}(\mathbf{x}|\mathbf{z})\exp(-E_{\theta}(\mathbf{z}))p_{0}(\mathbf{z})$  using a Langevin sampler. For the training of model, the contrastive divergence algorithm is replaced with energy discrepancy (for details see appendix C). LEBMs provide an interesting benchmark to compare energy discrepancy with contrastive divergence in high dimensions. However, other ways to tackle the manifold problem are elements of ongoing research.
5 Empirical Studies
To support our theoretical discussion, we evaluate the performance of energy discrepancy on low-dimensional datasets as well as for the training of latent EBMs on high-dimensional image data sets. In our discussion, we emphasise the comparison with contrastive divergence and score matching as the dominant methodologies in EBM training.
5.1 Density Estimation
We first demonstrate the effectiveness of energy discrepancy on several two-dimensional distributions. Figure 3 displays the estimated unnormalised densities as well as samples that were synthesised with Langevin dynamics for energy discrepancy (ED), score matching (SM) and contrastive divergence (CD). More experimental results and details are given in Appendix D.4. Our results confirm the aforementioned nearsight-edness of score matching which does not learn the uniform weight distribution of the Gaussian mixture. For CD, it can be seen that CD consistently produces flattened energy landscapes which can be attributed to the short-run MCMC (Nijkamp et al., 2020a, see) not having converged. Consequently, the synthe
sised samples of energies learned with CD can lie outside of the data-support. In contrast, ED is able to model multi-modal distributions faithfully and learns sharp edges in the data support as in the chessboard data set. We quantify our results in Figure 4 which shows the mean squared error of the estimated log-density of 25-Gaussians over the number of training iterations. The partition function is estimated using importance sampling  $\log Z\approx \mathrm{logsumexp}(-E(\mathbf{x}_i) - \log p_{\mathrm{data}}(\mathbf{x}_i)) - \log N$ , where  $\mathbf{x}_i$  is sampled from the data distribution  $p_{\mathrm{data}}(\mathbf{x})$  and  $N = 5,000$ . It shows that ED outperforms SM and CD with faster convergence, lower mean square error, and better stability.
Figure 4: Density estimation accuracy in the 25-Gaussians dataset.
(a) SVHN  $(32\times 32)$
(b) CIFAR-10  $(32\times 32)$
(c) CelebA  $(64\times 64)$
Figure 5: Generated examples by the ED-LEBM trained on SVHN, CIFAR-10, and CelebA.
Table 1: Comparison of  $\operatorname{MSE}\left( \downarrow \right)$  and  $\operatorname{FID}\left( \downarrow \right)$  on the SVHN, CIFAR-10, and CelebA datasets.
SVHN	CIFAR-10	CelebA
MSE	FID	MSE	FID	MSE	FID
VAE (Kingma & Welling, 2014)	0.019	46.78	0.057	106.37	0.021	65.75
2s-VAE (Dai & Wipf, 2018)	0.019	42.81	0.056	72.90	0.021	44.40
RAE (Ghosh et al., 2019)	0.014	40.02	0.027	74.16	0.018	40.95
SRI (Nijkamp et al., 2020b)	0.018	44.86	0.020	-	-	61.03
SRI (L=5) (Nijkamp et al., 2020b)	0.011	35.32	-	-	0.015	47.95
CD-LEBM (Pang et al., 2020)	0.008	29.44	0.020	70.15	0.013	37.87
SM-LEBM	0.010	34.44	0.026	77.82	0.014	41.21
ED-LEBM (ours)	0.006	28.10	0.023	73.58	0.009	36.73
5.2 Image Modelling
In this experiment, our methods are evaluated by training a latent EBM on three image datasets: SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), and CelebA (Liu et al., 2015). The effectiveness of energy discrepancy is diagnosed through image generation, image reconstruction from their latent representation, and the faithfulness of the learned latent representation. The model architectures, training details, and the choices of hyper-parameters can be found in Appendix C.3.
Image Generation and Reconstruction. We benchmark latent EBM priors trained with energy discrepancy (ED-LEBM) and score matching (SM-LEBM) against various baselines for latent variable models which are included in Table 1. Note that the original work on latent EBMs (Pang et al., 2020) uses contrastive divergence (CD-LEBM) (see appendix C for details).
For a well-learned model, the latent EBM should produce realistic samples and faithful reconstructions. The reconstruction error is measured via the mean square error while the image generation is measured with the FID (Heusel et al., 2017) which are both reported in Table 1. We observe that ED can improve the contrastive divergence benchmark on SVHN and CelebA while the results on CIFAR-10 could not be improved. However, we emphasise that ED only requires  $M$  (here,  $M = 16$ ) evaluations of the energy function per data point which is significantly less than CD and SM that both require the calculation of a high-dimensional spatial gradient. Besides the quantitative metrics, we present qualitative results of the generated samples in Figure 5. It can be seen that our model generates diverse high-quality images. The qualitative results of the reconstruction are shown in Figure 6, for which we use the test set of CelebA  $64 \times 64$ . The right column shows the original image  $\mathbf{x}$  to be reconstructed. The left column shows the reconstruction based on a latent variable initialised from the base distribution  $\mathbf{z}_0 \sim p_0(\mathbf{z})$ , and the middle column displays the image reconstructed from  $\mathbf{z}_k \sim p_{\phi,\theta}(\mathbf{z}|\mathbf{x})$  which is sampled via Langevin dynamics. One can see that our model can successfully reconstruct the test images, verifying the validity of the latent
prior learned with energy discrepancy. In addition, we showcase the scalability of our approach by
Figure 6: Image reconstruction results on CelebA  $64 \times 64$ .
Figure 7: Image interpolation results on CelebA  $64 \times 64$ . The first and last columns show the observed images, while the middle columns display the interpolation results using the inferred latent vectors.
(a) Smiling
(c) Eyeglasses
(d) Blond Hair
Figure 8: Attribute manipulation by interpolating the latent variable along an attribute vector. The middle image corresponds to the original image, and each subfigure represents a different attribute.
Table 2: Comparison of AUPRC(↑) for unsupervised anomaly detection on MNIST dataset.
Heldout Digit	1	4	5	7	9
VAE (Kingma & Welling, 2014)	0.063	0.337	0.325	0.148	0.104
ABP (Han et al., 2017)	0.095 ± 0.03	0.138 ± 0.04	0.147 ± 0.03	0.138 ± 0.02	0.102 ± 0.03
MEG (Kumar et al., 2019)	0.281 ± 0.04	0.401 ± 0.06	0.402 ± 0.06	0.290 ± 0.04	0.342 ± 0.03
BiGAN-σ (Zenati et al., 2018)	0.287 ± 0.02	0.443 ± 0.03	0.514 ± 0.03	0.347 ± 0.02	0.307 ± 0.03
CD-LEBM (Pang et al., 2020)	0.336 ± 0.01	0.630 ± 0.02	0.619 ± 0.01	0.463 ± 0.01	0.413 ± 0.01
SM-LEBM	0.285 ± 0.01	0.663 ± 0.01	0.610 ± 0.01	0.471 ± 0.01	0.422 ± 0.01
ED-LEBM (ours)	0.342 ± 0.01	0.740 ± 0.01	0.708 ± 0.01	0.501 ± 0.02	0.444 ± 0.01
applying it successfully to high-resolution images (CelebA  $128 \times 128$ ). More results can be found in Appendix D.5.
Image Interpolation and Manipulation. We use a latent-variable to model the effective low-dimensional structure in the data set. To probe how well the latent space describes the geometry and meaning of the data-manifold we analyse the structure of the latent space through interpolation and attribute manipulation. For the interpolation between two samples we linearly interpolate between their latent representations which were sampled from the posterior distribution. The results in Figure 7 demonstrate that the latent space has learned the data-manifold well and almost all intermediate samples appear as realistic faces. Further results are given in Figures 19 and 20. In addition, we can utilize the labels in the CelebA dataset to modify the attributes of an image through the manipulation technique proposed in (Kingma & Dhariwal, 2018). Specifically, each image in the dataset is associated with a binary label that indicates the presence or absence of attributes such as smiling, male, eyeglasses. For each manipulated attribute, we compute the average latent vectors  $\mathbf{z}_{\mathrm{pos}}$  of images with and  $\mathbf{z}_{\mathrm{neg}}$  of images without the attribute in the training set. Then, we use the difference  $\mathbf{z}_{\mathrm{pos}} - \mathbf{z}_{\mathrm{neg}}$  as the direction for manipulating the attribute of an image. The results presented in Figures 8 and 22 confirm the meaningfulness of the latent space learned by energy discrepancy.
5.3 Anomaly Detection
In a well-learned energy-based model the likelihood should be higher for in-distribution examples and lower for out-of-distribution examples. Based on this principle, we conduct anomaly detection experiments to compare our method with other baselines. Specifically, given a test sample  $\mathbf{x}$ , we first sample  $\mathbf{z}$  from the posterior  $p_{\theta}(\mathbf{z}|\mathbf{x})$  by Langevin dynamics, and then compute the unnormalised log-density  $p_{\theta}(\mathbf{x},\mathbf{z})$  as the decision function. Following the protocol in (Zenati et al., 2018), we designate each digit class in the MNIST dataset as an anomaly and leave the rest as training data. The area under the precision-recall curve (AUPRC) is used to evaluate different methods. As shown in Table 2, our model consistently outperforms the baseline methods, demonstrating the advantages of training latent EBMs with energy discrepancy.
Training Energy-based models. While energy-based models have been around for some time (Hinton, 2002), the training of energy-based models remains challenging. For a summary on existing methods for the training of energy-based models see Song & Kingma (2021) and LeCun et al. (2006). Contrastive divergence is still the most used option for training energy-based models. Recent extensions have improved the scalability of the basic algorithm, making it possible to train EBMs on high-dimensional data (Ngiam et al., 2011; Dai et al., 2015; Xie et al., 2016; Nijkamp et al., 2019; Du & Mordatch, 2019). Despite these advances, it has been noted that contrastive divergence is not the gradient of any fixed loss-function (Sutskever & Tieleman, 2010) and can yield energy functions that do not adequately reflect the data distribution (Nijkamp et al., 2020a). This has motivated improvements to the standard methodology (Du et al., 2021) by approximating overlooked entropy terms or by improving the convergence of MCMC sampling with an underlying diffusion model (Gao et al., 2021).
Score-based methods (Hyvärinen & Dayan, 2005; Vincent, 2011; Song et al., 2020) are typically implemented by directly learning the score function instead of the energy function. Song & Ermon (2019); Song et al. (2021b) point out the importance to introduce multiple noise levels at which the score is learned. Li et al. (2019) adopt the strategy of learning an energy-based model using multi-scale denoising score matching.
We learn a latent EBM prior (Pang et al., 2020) on high-dimensional data to address situations where the data lives on a lower dimensional submanifold. This methodology has been improved with a multi-stage approach (Xiao & Han, 2022) with score-independent noise contrastive estimation (Gutmann & Hyvarinen, 2010).
Related loss functions. Energy discrepancies can be derived from Kullback Leibler-contraction divergences (Lyu, 2011) which we also discuss briefly in Appendix A.6. Diffusion recovery likelihood (Gao et al., 2021) technically optimises the same loss as us but the EBM is trained with contrastive divergence. Luo et al. (2023) make similar observations to us while estimating the KL-contraction by leveraging the time evolution of the energy function along a diffusion process. Energy discrepancy also shares similarities with other contrastive loss functions in machine learning. The  $w$ -stabilised energy-discrepancy loss with  $w$ ,  $M = 1$  is equivalent to conditional noise contrastive estimation (Ceylan & Gutmann, 2018) with Gaussian noise. For  $M > 1$ , the stabilised ED loss shares great structural similarity to certain contrastive learning losses such as InfoNCE (van den Oord et al., 2018). We hope that such observations can lead to interpretations of the  $w$ -stabilisation.
Theoretical connections between likelihood and score-based functionals. The connection between likelihood methods and score-based methods generalises the de Bruijn identity (Stam, 1959; Lyu, 2009) which explains that score matching appears frequently as the limit of EBM training methods. Similar connections have been mentioned and exploited by Song et al. (2021a) to choose a weighting scheme for a combination of score matching losses.
Generative modelling for manifold-valued data. The manifold hypothesis was, for example, described in Bengio et al. (2013). Prior work to us shows how score-based diffusion models detect this manifold (Pidstrigach, 2022) and give reasons why CD is more robust to this issue than other contrastive methods (Yair & Michaeli, 2021). Arbel et al. (2021) learn an energy-based model as a tilt of a GAN-based prior that models the data-manifold. We believe that a combination of above results with energy discrepancy could enable training EBMs with energy discrepancy on data space.
7 Discussion and Outlook
We demonstrate that energy discrepancy provides a new tool for the fast training of energy-based models without MCMC or scores. We show for Euclidean data that ED interpolates between score matching and maximum likelihood estimation, thus alleviating problems of nearsightedness of score matching without additional annealing strategies as in score-based diffusion models. Based on our theoretical analysis, we show that training EBMs using energy discrepancy yields more accurate estimates of the energy function for two-dimensional data than explicit score matching and contrastive divergence. In this task, it is robust to the hyperparameters used, making energy discrepancy a useful tool for energy-based modelling with little tuning required. We further establish that energy discrepancy achieves comparable results to contrastive divergence at a lower computational cost
when learning a lower-dimensional energy-based prior for high-dimensional data. This shows that energy discrepancy is scalable to high dimensions.
Limitations: Energy-based models make the assumption that the data-distribution has a positive density which is violated for most high-dimensional data sets due to the manifold hypothesis. Compared to contrastive divergence or diffusion-based strategies, energy discrepancy is especially sensitive to such singularities in the data set. Currently, this limits energy discrepancy to settings in which a lower-dimensional representation with non-singular support can be learned or constructed efficiently and accurately.
Outlook: This work can be extended in various ways. Firstly, we want to explore other choices for the conditional distribution  $q$ . In particular, the effect of different types of noise such as Laplace noise or anisotropic Gaussian noise are open questions. Furthermore, energy discrepancy is a well-suited objective function for discrete data for appropriately chosen perturbations. We present a preliminary study of energy discrepancy on discrete spaces in the workshop paper Schröder et al. (2023). Secondly, we believe that improvements to our methodology can be made by learning the energy function of image-data on pixel space directly, as in this domain specific architectures such as U-nets (Ronneberger et al., 2015; Song & Ermon, 2019) can be used in the modelling. However, this requires further work in learning the data-manifold during training so that early saturation can be prevented. Finally, the partial differential equations arising in Euclidean energy discrepancies promise exciting insights into the connections between energy-based learning, stochastic differential equations, and optimal control which we want to investigate further.
TS and ABD are grateful to G.A. Pavliotis for insightful discussions throughout the project. TS was supported by the EPSRC-DTP scholarship partially funded by the Department of Mathematics, Imperial College London. ZO was supported by the Lee Family Scholarship. JNL was supported by the Feuer International Scholarship in Artificial Intelligence. ABD was supported by Wave 1 of The UKRI Strategic Priorities Fund under the EPSRC Grant EP/T001569/1 and EPSRC Grant EP/W006022/1, particularly the "Ecosystems of Digital Twins" theme within those grants and The Alan Turing Institute. We thank the anonymous reviewer for their comments.
Arbel, Michael, Zhou, Liang, and Gretton, Arthur. Generalized energy based models. In International Conference on Learning Representations, 2021.
Barp, Alessandro, Briol, Francois-Xavier, Duncan, Andrew, Girolami, Mark, and Mackey, Lester. Minimumstein discrepancy estimators. Advances in Neural Information Processing Systems, 32, 2019.
Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1828, 2013.
Carreira-Perpínán, Miguel Á. and Hinton, Geoffrey. On contrastive divergence learning. In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, volume R5 of Proceedings of Machine Learning Research, pp. 33–40. PMLR, 06–08 Jan 2005.
Ceylan, Ciwan and Gutmann, Michael U. Conditional noise-contrastive estimation of unnormalised models. In Dy, Jennifer and Krause, Andreas (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 726-734. PMLR, 10-15 Jul 2018.
Chwialkowski, Kacper, Strathmann, Heiko, and Gretton, Arthur. A kernel test of goodness of fit. In International conference on machine learning, pp. 2606-2615. PMLR, 2016.
Dai, Bin and Wipf, David. Diagnosing and enhancing vae models. In International Conference on Learning Representations, 2018.
Xie, Jianwen, Lu, Yang, Zhu, Song-Chun, and Wu, Yingnian. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644. PMLR, 2016.
Yair, Omer and Michaeli, Tomer. Contrastive divergence learning is a time reversal adversarial game. In International Conference on Learning Representations, 2021.
Zenati, Houssam, Foo, Chuan Sheng, Lecouat, Bruno, Manek, Gaurav, and Chandrasekhar, Vijay Ramaseshan. Efficient gan-based anomaly detection. arXiv preprint arXiv:1802.06222, 2018.
Zhang, Mingtian, Hayes, Peter, Bird, Thomas, Habib, Raza, and Barber, David. Spread divergence. In III, Hal Daumé and Singh, Aarti (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11106-11116. PMLR, 2020.
Zhang, Mingtian, Key, Oscar, Hayes, Peter, Barber, David, Paige, Brooks, and Briol, François-Xavier. Towards healing the blindness of score matching. NeurIPS 2022 Workshop on Score-Based Methods, 2022.
Øksendal, Bernt. Stochastic Differential Equations : An Introduction with Applications. Universitext. Springer, Berlin, Heidelberg, sixth edition. edition, 2003. ISBN 9783642143946.
Appendix for "Energy Discrepancies: A Score-Independent Loss for Energy-Based Models"
A Proofs and Derivations 15
A.1 Proof of the Non-Parametric Estimation Theorem 1 15
A.2 Equivalence of Energy Discrepancy for Brownian Motion and Ornstein-Uhlenbeck Processes 17
A.3 Interpolation between Score-Matching and Maximum-Likelihood Estimation 18
A.4 Representing Energy Discrepancy as Multi-Scale SM for General Diffusion Processes 21
A.5 Connections of Energy Discrepancy with Contrastive Divergence 23
A.6 Derivation of Energy Discrepancy from KL Contractions 24
B Aspects of Training Energy-Based Models with Energy Discrepancy 24
B.1 Conceptual Understanding of the  $w$ -Stabilisation 24
B.2 Approximation of Energy Discrepancy based on general Ito Diffusions 27
C Latent Space Energy-Based Prior Models 28
C.1 A Brief Review of LEBMs 28
C.2 Langevin Sampling, Reconstruction, and Generation 30
C.3 Experimental Details of LEBMs 30
D Additional Experimental Results 31
D.1 Experimental Setup for Figure 1 (Healing the nearsightedness of score-matching) 31
D.2 Experimental Setup for Figure 2 (Understanding the  $w$ -stabilisation) 32
D.3 Energy Discrepancy on the Discrete Space  $\{0,1\}^d$  32
D.4 Additional Density Estimation Results 33
D.5 Additional Image Modelling Results 34
D.6 Qualitative Results on the Effect of  $t,M$  ,and  $\pmb{w}$  34
A Proofs and Derivations
In this section we are going to prove Theorem 1 and Theorem 2. Furthermore, we are going to generalise Theorem 2 to arbitrary diffusion processes and discuss the connections of energy discrepancy to other training losses for energy-based models.
A.1 Proof of the Non-Parametric Estimation Theorem 1
In this subsection we give a formal proof for the uniqueness of minima of  $\mathrm{ED}_q(p_{\mathrm{data}},U)$  as a functional in the energy function  $U$ . We first reiterate the theorem as stated in the paper:
For this theorem we need to specify additional assumptions on the conditional distribution  $q$  and on the optimisation domain to guarantee uniqueness. Firstly, we require the energy-based distribution to be normalisable which implies that  $\exp(-U) \in L^{1}(\mathcal{X},\mathrm{d}\mathbf{x})$ . For the existence and uniqueness of minimisers we have to constrain the space of energy functions since  $\mathrm{ED}_q(p_{\mathrm{data}},U) = \mathrm{ED}_q(p_{\mathrm{data}},U + c)$
for any constant  $c \in \mathbb{R}$ . Hence, we restrict the optimisation domain to functions  $U$  that satisfy  $\min_{\mathbf{x} \in \mathcal{X}} U(\mathbf{x}) = 0$ . To specify the condition on  $q$  we define an equivalence relation on  $\mathcal{X}$ :
Definition 2 (q-equivalence). For a pair of samples  $\mathbf{x}_1, \mathbf{x}_2 \in \mathcal{X}$  we say that  $\mathbf{x}_1$  and  $\mathbf{x}_2$  are  $q$ -neighbours if there exists a  $\mathbf{y} \in \mathcal{Y}$  such that  $\mathbf{x}_1, \mathbf{x}_2 \in \mathrm{supp}(q(\mathbf{y}|\cdot))$ . We denote the set of  $q$ -neighbours of  $\mathbf{x}$  as  $\mathcal{N}_q(\mathbf{x})$ . We then define the  $q$ -equivalence relation as:
$$
\mathbf {x} _ {1} \sim_ {q} \mathbf {x} _ {2} \Leftrightarrow t h e r e e x i s t \left(\mathbf {z} _ {1: R}\right) w i t h \mathbf {z} _ {1} = \mathbf {x} _ {1}, \quad \mathbf {z} _ {R} = \mathbf {x} _ {2}, a n d \quad \mathbf {z} _ {r + 1} \in \mathcal {N} _ {q} (\mathbf {z} _ {r})
$$
It is easy to see that  $q$ -equivalence defines an equivalence relation, i.e. it is symmetric, reflexive, and transitive. We summarise the assumptions for the non-parametric approximation theorem as follows:
Assumption 1. We define the optimisation domain