SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We use the following datasets in image modelling: SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), and CelebA (Liu et al., 2015). SVHN is of resolution  $32 \times 32$ , and contains 73, 257 training images and 26, 032 test images. CIFAR-10 consists of 50, 000 training images and 10, 000 test images with a resolution of  $32 \times 32$ . For CelebA, which contains 162, 770 training images and 19, 962 test images, we follow the pre-processing step in (Pang et al., 2020), taking 40, 000 examples of CelebA as training data and resizing it to  $64 \times 64$ . In anomaly detection, we follow the setting in (Zenati et al., 2018) and the dataset can be found in their published code<sup>5</sup>.
Model Architectures. We adopt the same network architecture used in CD-LEBM (Pang et al., 2020), with the details depicted in Table 3, where  $\mathrm{convT}(n)$  indicates a transposed convolutional operation with  $n$  output channels. We use Leaky ReLU as activation functions and the slope is set to be 0.2 and 0.1 in the generator and EBM prior, respectively.
Details of Training and Inference. Here, we provide a detailed description of the hyperparameters setup for ED-LEBM. Following (Pang et al., 2020), we utilise Xavier normal (Glorot & Bengio, 2010)
$^{5}$ https://github.com/houssamzenati/Efficient-GAN-Anomaly-Detection
to initialise the parameters. For the posterior sampling during training, we use the Langevin sampler with step size of 0.1 and run it for 20 steps for SVHN and CelebA, and 40 steps on CIFAR-10. We set  $t = 0.25$ ,  $M = 16$ ,  $w = 1$  throughout the experiments. The proposed models are trained for 200 epochs using the Adam optimizer (Kingma & Ba, 2015) with a fixed learning 0.0001 for the generator and 0.00005 for the EBM prior. We choose the largest batch size from {128, 256, 512} such that it can be trained on a single NVIDIA- GeForce-RTX-2080-Ti GPU. In test time, we observed that slightly increasing the number of Langevin sampler steps can improve reconstruction performance. Therefore, we choose 100 steps with a step size of 0.1 for posterior sampling. Based on the insights gained from the MCMC diagnostic presented in Figure 21, we choose 500 steps with a step size of 0.2 to ensure convergence of the Langevin dynamics when sampling from the EBM prior.
Evaluation Metrics. In image modelling, we use FID and MSE to quantitatively evaluate the quality of the generated samples and reconstructed images. On all datasets the FID is computed based on 50,000 samples and the MSE is computed on the test set. Following (Zenati et al., 2018; Pang et al., 2020), we report the performance using AUPRC in anomaly detection and results are averaged over last 10 epochs to account for variance.
In this section we present additional experimental results to highlight the effect of various parameters of our approach.
D.1 Experimental Setup for Figure 1 (Healing the nearsightedness of score-matching)
A major problem of score-based methods is their nearsight-edness, which refers to their inability to capture global properties of a distribution with disjoint supports such as the mixture weights of two well-separated modes (Zhang et al., 2022). In sight of Theorem 2, energy discrepancy should alleviate this problem as it implicitly compares the scores of both distributions at multiple noise-scales. Following Zhang et al. (2022), we investigate this by computing energy discrepancy as a function of the mixture weight  $\rho$  for the mixture of two Gaussians  $g_{1} := \mathcal{N}(-5,1)$  and  $g_{2} := \mathcal{N}(5,1)$ , i.e.,
Figure 10: Study of the influence of  $t$  and  $M$  on estimating mixing weights.
$$
p _ {\rho} (x) = \rho g _ {1} (x) + (1 - \rho) g _ {2} (x).
$$
where the true data has the mixture weight  $\rho = 0.2$ . We compare energy discrepancy  $\mathcal{L}_{t,M=32,w=1}(\rho) \approx \mathrm{ED}(p_{\rho=0.2}, \log p_{\rho})$  with the objective of maximum likelihood estimation  $\mathrm{MLE}(\rho) := \mathbb{E}_{p_{\rho=0.2}(x)}[-\log p_{\rho}(x)]$  and the score matching objective which here is given by the Fisher divergence  $\mathrm{SM}(\rho) := \frac{1}{2}\mathbb{E}_{p_{\rho=0.2}(x)}[\|\nabla_x\log p_{\rho=0.2}(x) - \nabla_xp_{\rho}(x)\|_2^2]$ . The losses as functions of  $\rho$  are shown in Figure 1. We find that energy discrepancy is convex as a function of the mixture weight and approximates the negative log-likelihood as  $t$  increases. Consequently, energy discrepancy can capture the mixture weight well for sufficiently large values of  $t$ . SM, on the other hand, is a constant function and is blind to the value of the mixture weight.
To further investigate the impact of  $t$  and  $M$  on the efficiency of energy discrepancy, we minimise the energy discrepancy loss  $\mathcal{L}_{t,M=32,w=1}(\rho)$  as a function of the scalar parameter  $\rho$  for various choices of  $M$  and  $t$ . We compute the mean-square error of 50 independent estimated mixture weights for choice of  $t$  and  $M$ . As shown in Figure 10, the estimation performance approaches that of the maximum likelihood estimator as  $t$  increases, which verifies the statement in Theorem 2. Moreover, if the number of samples  $M$  used to estimate the contrastive potential is increased, the estimation performance can be further increased towards the mean-square error of the maximum-likelihood estimator.
Figure 11: Potential outcomes for the estimated energy and loss history when ED does not converge with  $w = 0$ .
D.2 Experimental Setup for Figure 2 (Understanding the  $w$ -stabilisation)
To probe our interpretation of the  $w$ -stabilisation, we train a neural-network to learn the energy function using 4,096 data points of a one-dimensional standard Gaussian  $p_{\mathrm{data}}(x) \propto \exp(-x^2/2)$ . The neural network uses an input layer, a hidden linear layer of width two  $\mathbb{R}^2 \to \mathbb{R}^2$ , and a scalar output layer  $\mathbb{R}^2 \to \mathbb{R}$  with a Sigmoid Linear Unit activation between the layers. This neural network has sufficient capacity to model the Gaussian data as well as degenerate energy functions that illustrate potential pitfalls of energy discrepancy for  $w = 0$ . The energy discrepancy is set up with hyperparameters  $M = 4$ ,  $t = 1$ , and  $w \in \{0, 0.05, 0.25, 2\}$  and is trained for 50 epochs with Adam. Our results are shown in Figure 2 which confirms the relevance of the  $w$ -stabilisation to obtain a stable optimisation of energy discrepancy. We remark here that the degenerate case  $w = 0$  is not strictly reproducible. Different types of lacking smoothness of the energy-function at the edge of the support lead to diverging loss values. We chose a result that illustrates the best the theoretical exposition of the  $w$ -stabilisation in Appendix B.1 and refer to Figure 11 to reflect other malformed estimated energies as well as an example of a diverging loss history.
D.3 Energy Discrepancy on the Discrete Space  $\{0,1\}^d$
Energy discrepancies are, in principle, well-defined on discrete spaces. To illustrate this point, we describe the energy-discrepancy loss for  $\{0,1\}^d$  valued data such as images with binary pixel values, in which case the discrete energy-discrepancy is straight forward to implement. We will replace the Gaussian transition density with a Bernoulli distribution. For  $\varepsilon \in (0,1)$ , let  $\xi \sim \mathrm{Bernoulli}(\varepsilon)^d$ . Then the transition  $\mathbf{y} = \mathbf{x} + \boldsymbol{\xi} \bmod(2)$  is symmetric and induces a symmetric transition density  $q(\mathbf{y} - \mathbf{x})$ . Because of the symmetry, the energy discrepancy can be implemented in the same way as in the continuous case, i.e.
$$
\mathcal {L} (\theta) := \frac {1}{N} \sum_ {i = 1} ^ {N} \left(w + \sum_ {j = 1} ^ {M} \exp \left(E _ {\theta} \left(\mathbf {x} _ {i}\right) - E _ {\theta} \left(\mathbf {x} ^ {i} \oplus \boldsymbol {\xi} ^ {i} \oplus \boldsymbol {\xi} ^ {\prime i, j}\right)\right)\right)
$$
with  $\xi^i, \xi'^{i,j} \sim$  Bernoulli and where  $\oplus$  is the addition modulo 2. To test the effectiveness of energy discrepancy with Bernoulli perturbation, we test energy discrepancy on various discrete settings. In Figure 12, we estimate the symmetric connectivity matrix of Ising models following the exposition in Grathwohl et al. (2021). We then learn the densities of two-dimensional synthetic data sets that were mapped to the space  $\{0,1\}^{32}$  via grey codes in Figure 13, following Dai et al. (2020). Finally, we estimate the densities of image data with binary pixel values for the static and dynamic MNIST dataset, the omniglot dataset, and the caltech silhouettes dataset. As shown in Figure 14, energy discrepancy performs competitively to contrastive divergence. We propose other potential discrete perturbations in Schröder et al. (2023).
Figure 12: Results of learning Ising models. Left: ground truth; Right: learned pattern.
Figure 13: Visualisation of the energy function learned on 2D plan. Here we train a discrete EBM on 32-dim grey codes which are one-to-one mapping from  $\mathbb{R}^2$  to  $\{0,1\}^{32}$ .
Figure 14: Generated samples on discrete image modelling. Here we train a discrete EBM using the Bernoulli transition. Left to right: Static MNIST, Dynamic MNIST, Omniglot, Caltech Silhouettes.
Figure 15: Additional results on density estimation.
D.4 Additional Density Estimation Results
Here, we provide additional details and results on the density estimation experiments.
Details of Training and Inference. Our choice for the energy-net for density estimation is a 4-layer feed-forward neural network with 128 hidden units and softmax activation function. In the context of energy discrepancy, we select  $t = 1$ ,  $M = 4$ , and  $w = 1$  as hyperparameters. For the contrastive divergence approach, we utilise CD-1, in which the gradient of the log-likelihood in Equation (1) is estimated by employing a Langevin sampler with a single step and a step size of 0.1. For score matching, we train EBMs using the explicit score matching in (2), where the Laplacian of the score is explicitly computed. We train the model using the Adam optimizer with a learning rate of 0.001 and iterations of 50,000. After training, synthesised samples are drawn by simulating Langevin dynamics with 100 steps and a step size of 0.1.
Additional Experimental Results. The additional results depicted in Figure 15 demonstrate the strong performance of energy discrepancy on various toy datasets, consistently yielding accurate energy landscapes. In contrast, contrastive divergence consistently produces flattened energy landscapes. Despite the success of score matching in these toy examples, score matching struggles to effectively learn distributions with disjoint support which can be seen in the results in Figure 3.
Comparison with Denoising Score
Matching We further compare energy discrepancy with denoising score matching (DSM) (Vincent, 2011). Specifically, we set  $w = 1$ ,  $M = 4$  and experiment with various  $t$ . As shown in Figure 16, DSM fails to work when the noise scale is too large or too small. This is because DSM is a biased estimator which is optimised for  $p_{\theta^*}(\mathbf{y}) = \int \gamma_t(\mathbf{y} - \mathbf{x}) p_{\mathrm{data}}(\mathbf{x}) \, \mathrm{d}\mathbf{x}$ . In contrast, energy discrepancy is
Figure 16: Comparing energy discrepancy (ED) with denoising score matching (DSM) with different noise scales.
more robust to the choice of  $t$  since energy discrepancy considers all noise scales up to  $t$  simulta-
Figure 17: Generated images on CelebA  $128 \times 128$ .
neously and has an unique optimum  $p_{\theta^*}(\mathbf{x}) = p_{\mathrm{data}}(\mathbf{x})$ . However, in the case that  $t$  is large and  $M$  is small, estimation with energy discrepancy deteriorates due to high variance of the estimated loss function. This provides an explanation for the superior performance of energy discrepancy at  $\sqrt{t} = 1$  compared to  $\sqrt{t} = 10$ . Further ablation studies are presented in Figure 23.
D.5 Additional Image Modelling Results
Additional Image Generation and Reconstruction Results. Figures 17 and 18 show additional examples of image generation on CelebA  $128 \times 128$  and image reconstruction on CelebA  $64 \times 64$ . The images are computed through the sampling process outlined in Appendix C.2.
Additional Image Interpolation and Manipulation Results. Figures 19, 20 and 22 show additional results of image interpolation and manipulation on CelebA  $64 \times 64$ . Note that there are two types of interpolations: posterior interpolation and prior interpolation. For posterior interpolation, we consider two real images  $\mathbf{x}_1$  and  $\mathbf{x}_2$  from the dataset and perform linear interpolation among their corresponding latent variables  $\mathbf{z}_1 \sim p_{\phi, \theta}(\mathbf{z}|\mathbf{x}_1)$  and  $\mathbf{z}_2 \sim p_{\phi, \theta}(\mathbf{z}|\mathbf{x}_2)$ . For prior interpolation, we apply linear interpolation between  $\mathbf{z}_1 \sim p_{\theta}(\mathbf{z})$  and  $\mathbf{z}_2 \sim p_{\theta}(\mathbf{z})$ .
Long-run MCMC Diagnostics. Figure 21 depicts several convergence diagnostics for long-run MCMC on the EBM prior, where we simulate Langevin dynamics with a large number of steps (2,000). Firstly, the energy profiles converge at approximately 250 steps, as demonstrated in Figure 21a, and the quality of the synthesized samples improves as the number of steps increases. Secondly, we compute the Gelman-Rubin statistic  $\hat{R}$  (Gelman & Rubin, 1992) using 64 chains. The histograms of  $\hat{R}$  over  $5,000 \times 64$  chains are shown in Figure 21b, with a mean of  $1.08 < 1.20$ , indicating that the Langevin dynamics have approximately converged. Thirdly, we present auto-correlation results in Figure 21c using 5,000 chains, where the mean is depicted as a line and the standard deviation as bands. The auto-correlation decreases to zero within 200 steps, which is consistent with the Gelman-Rubin statistic that assesses convergence across multiple chains.
D.6 Qualitative Results on the Effect of  $t$ ,  $M$ , and  $w$
The hyperparameters  $t, M, w$  play important roles in energy discrepancy. Here, we provide some qualitative results to understand their effects. According to Theorem 2,  $t$  controls the nearsight-ness of energy discrepancy. For small  $t$ , energy discrepancy behaves like score matching  $\frac{1}{t}\mathrm{ED}_{\gamma_t}(p_{\mathrm{data}},U) = \frac{1}{t}\int_0^t\mathrm{SM}(p_s,U_s)\mathrm{d}s \approx \mathrm{SM}(p_{\mathrm{data}},U)$  and is expected to be unable to resolve local mixture weights. This assertion can be confirmed by qualitative results depicted in Figure 23, which show that when  $t = 0.0025$ , energy discrepancy fails to identify the weights of components in the 25-Gaussians and pinwheel datasets. For large  $t$ , energy discrepancy inherits favourable properties
of the maximum likelihood estimator. While large values of  $t$  consequently mitigate problems of nearsightedness, it is worth noting that energy discrepancy may encounter issues with high variance when  $t$  become excessively large. In such situations, it is necessary to consider increasing the value of  $M$  to reduce the variance.
We also investigate the effect of  $w$  in Figure 24. As pointed out by the analysis in Appendix B.1,  $w$  serves as a stabilises training of energy based models with energy discrepancy. Based on our experimental observations, when  $w = 0$  and  $M$  is small (e.g.,  $M \leq 128$  in the 25-Gaussians dataset and  $M \leq 32$  in the pinwheel dataset), energy discrepancy exhibits rapid divergence within 100 optimisation steps and fails to converge in the end. If, however,  $w$  is increased, e.g. to 1, energy discrepancy shows stable convergence even with  $M = 1$ . This property is highly appealing as it significantly reduces the computational complexity. Additionally, we find in Figure 2 that larger  $w$  tends to result in a flatter estimated energy landscapes which aligns with our intuition gained in Appendix B.1.
Figure 18: Qualitative results of reconstruction on test images. Left: real image from the dataset. Right: reconstructed images by sampling from the posterior.
Figure 19: Linear interpolation results in posterior latent space between real images.
Figure 20: Linear interpolation results in prior latent space between generated images.
(a) Energy Profile
(b) Gelman-Rubin
(c) Auto-correlation
Figure 21: Diagnostics for the mixing of MCMC chains with 2,000 steps on CelebA  $64 \times 64$ . Top: Trajectory in the data space. Bottom: (a) Energy profile over time; (b) Histograms of Gelman-Rubin statistic of multiple chains; (c) Auto-correlation of a single chain over time lags.
(b) Male
(c) Eyeglasses
(d) Blond Hair
Figure 22: Attribute manipulation results on CelebA  $64 \times 64$ . Each row is made by interpolating the latent variable along an attribute vector, with the middle image being the original image.
Figure 23: Density estimation on 25-Gaussians and pinwheel with different  $t$ ,  $M$  and  $w = 1$ .
Figure 24: Density estimation 25-Gaussians and pinwheel with different  $w$ ,  $M$  and  $t = 1$ .