SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for every data point  $\mathbf{x}$ . This tells us that, roughly speaking, a perturbed data point with  $U(\mathbf{x}_0) - U(\mathbf{x}_t + \sqrt{t}\pmb{\xi}') > \log (w)$  should have a small contribution to the loss and the optimisation converges if the data distribution is learned or when the bound is violated at all perturbed data points. Thus,  $\log (w)$  describes a weak notion of a margin between positive and negative energy contributions. Consequently, large values for  $w\in (0,1)$  tend to lead to flatter learned energies, while smaller values lead to steeper learned energies. This intuition is confirmed by Figures 2 and 24.
It holds that  $\min (u_1,u_2,\dots ,u_M) - \log (M)\leq -\mathrm{LSE}(-u_1, - u_2,\dots , - u_M)\leq \min (u_1,u_2,\dots ,u_M)$
Asymptotic consistency of sample approximation of ED We give a proof for Theorem 3 which states that our approximation of energy discrepancy is justified. To make the exposition easier to understand, we first show how the energy discrepancy is transformed into a conditional expectation. Recall the probabilistic representation of the contrastive potential Section 4. Using  $E_{\theta}(\mathbf{x}) = \log (\exp (E_{\theta}(\mathbf{x})))$  we obtain the following rewritten form of energy discrepancy:
$$
\begin{array}{l} \operatorname {E D} _ {\gamma_ {t}} \left(p _ {\text {d a t a}}, E _ {\theta}\right) = \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \left[ E _ {\theta} (\mathbf {x}) \right] + \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {t} (\mathbf {y} - \mathbf {x})} \left[ \log \mathbb {E} _ {\gamma_ {1} \left(\boldsymbol {\xi} ^ {\prime}\right)} \left[ \exp \left(- E _ {\theta} (\mathbf {y} + \sqrt {t} \boldsymbol {\xi} ^ {\prime}) | \mathbf {y} \right] \right] \right. \\ = \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} [ E _ {\theta} (\mathbf {x}) ] + \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {1} (\pmb {\xi})} \left[ \log \mathbb {E} _ {\gamma_ {1} (\pmb {\xi} ^ {\prime})} \left[ \exp (- E _ {\theta} (\mathbf {x} + \sqrt {t} \pmb {\xi} + \sqrt {t} \pmb {\xi} ^ {\prime}) | \mathbf {x}, \pmb {\xi} \right] \right] \\ = \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {1} (\boldsymbol {\xi})} \left[ \right. \log \left(\exp \left(E _ {\theta} (\mathbf {x})\right)\right) + \log \mathbb {E} _ {\gamma_ {1} \left(\boldsymbol {\xi} ^ {\prime}\right)} \left[ \right. \exp \left( \right.- E _ {\theta} (\mathbf {x} + \sqrt {t} \boldsymbol {\xi} + \sqrt {t} \boldsymbol {\xi} ^ {\prime}) | \mathbf {x}, \boldsymbol {\xi} \left. \right]\left. \right] \\ = \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {1} (\boldsymbol {\xi})} \left[ \log \left(\mathbb {E} _ {\gamma_ {1} \left(\boldsymbol {\xi} ^ {\prime}\right)} \left[ \exp \left(E _ {\theta} (\mathbf {x}) - E _ {\theta} (\mathbf {x} + \sqrt {t} \boldsymbol {\xi} + \sqrt {t} \boldsymbol {\xi} ^ {\prime}) | \mathbf {x}, \boldsymbol {\xi} \right]\right)\right) \right. \\ \end{array}
$$
The conditioning means that the expectation is not taken with respect to  $\mathbf{y}$  or  $\mathbf{x}$  and  $\pmb{\xi}$  in the inner expectation. The conditioning is important to understand how the law of large numbers is to be applied. We now come to the proof that our approximation is consistent with the definition of energy discrepancy:
Proof. First, consider independent random variables  $\mathbf{x} \sim p_{\mathrm{data}}$ ,  $\boldsymbol{\xi} \sim \mathcal{N}(0,\mathbf{I})$ , and  $\xi^{\prime j} \stackrel{i.i.d}{\sim} \mathcal{N}(0,\mathbf{I})$ . Using the triangle inequality, we can upper bound the difference  $|\mathrm{ED}_{\gamma_t}(p_{\mathrm{data}},E_\theta) - \mathcal{L}_{t,M,w}(\theta)|$  by upper bounding the following two terms, individually:
$$
\begin{array}{l} \left| \mathrm {E D} _ {\gamma_ {t}} \left(p _ {\text {d a t a}}, E _ {\theta}\right) - \frac {1}{N} \sum_ {i = 1} ^ {N} \log \mathbb {E} \left[ \exp \left(E _ {\theta} \left(\mathbf {x} ^ {i}\right) - E _ {\theta} \left(\mathbf {x} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {\prime j}\right) \mid \mathbf {x} ^ {i}, \boldsymbol {\xi} ^ {i} \right] \right. \right. \\ + \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \log \mathbb {E} \left[ \exp \left(E _ {\theta} \left(\mathbf {x} ^ {i}\right) - E _ {\theta} \left(\mathbf {x} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {\prime j}\right) \mid \mathbf {x} ^ {i}, \boldsymbol {\xi} ^ {i} \right] - \mathcal {L} _ {t, M, w} (\theta) \right. \right. \\ \end{array}
$$
The first term can be bounded by a sequence  $\varepsilon_N \xrightarrow{a.s.} 0$  due to the normal strong law of large numbers. The second term can be estimated by applying the following conditional version of the strong law of large numbers (Majerek et al., 2005, Theorem 4.2):
$$
\frac {1}{M} \sum_ {j = 1} ^ {M} \exp \left(E _ {\theta} (\mathbf {x}) - E _ {\theta} (\mathbf {x} + \sqrt {t} \boldsymbol {\xi} + \sqrt {t} \boldsymbol {\xi} ^ {\prime j})\right) \xrightarrow {a . s .} \mathbb {E} \left[ \exp (E _ {\theta} (\mathbf {x}) - E _ {\theta} (\mathbf {x} + \sqrt {t} \boldsymbol {\xi} + \sqrt {t} \boldsymbol {\xi} ^ {\prime}) | \mathbf {x}, \boldsymbol {\xi} \right]
$$
Next, we have that the deterministic sequence  $w / M \to 0$ . Thus, adding the regularization  $w / M$  does not change the limit in  $M$ . Furthermore, since the logarithm is continuous, the limit also holds after applying the logarithm. Finally, the estimate translates to the sum by another application of the triangle inequality. We define
$$
\Delta e _ {\theta} (\mathbf {x}, \boldsymbol {\xi}, \boldsymbol {\xi} ^ {\prime}) := \exp \left(E _ {\theta} (\mathbf {x}) - E _ {\theta} \left(\mathbf {x} + \sqrt {t} \boldsymbol {\xi} + \sqrt {t} \boldsymbol {\xi} ^ {\prime}\right)\right)
$$
For each  $i = 1,2,\ldots ,N$  there exists a sequence  $\varepsilon_{i,M}\xrightarrow{a.s.}0$  such that
$$
\begin{array}{l} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \log \mathbb {E} \left[ \Delta e _ {\theta} \left(\mathbf {x} ^ {i}, \boldsymbol {\xi} ^ {i}, \boldsymbol {\xi} ^ {\prime}\right) \mid \mathbf {x} ^ {i}, \boldsymbol {\xi} ^ {i} \right] - \mathcal {L} _ {t, M, w} (\theta) \right| \\ \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \left| \log \mathbb {E} \left[ \Delta e _ {\theta} \left(\mathbf {x} ^ {i}, \boldsymbol {\xi} ^ {i}, \boldsymbol {\xi} ^ {\prime}\right) \mid \mathbf {x} ^ {i}, \boldsymbol {\xi} ^ {i} \right] - \log \frac {1}{M} \sum_ {j = 1} ^ {M} \Delta e _ {\theta} \left(\mathbf {x} ^ {i}, \boldsymbol {\xi} ^ {i}, \boldsymbol {\xi} ^ {\prime j}\right) \right| \\ <   \frac {1}{N} \sum_ {i = 1} ^ {N} \varepsilon_ {i, M} \leq \max  (\varepsilon_ {1, M}, \dots , \varepsilon_ {N, M}). \\ \end{array}
$$
Hence, for each  $\varepsilon > 0$  there exists an  $N \in \mathbb{N}$  and an  $M(N) \in \mathbb{N}$  such that  $|\mathrm{ED}_{\gamma_t}(p_{\mathrm{data}}, E_\theta) - \mathcal{L}_{t,M(N),w}(\theta)| < \varepsilon$  almost surely.
B.2 Approximation of Energy Discrepancy based on general Ito Diffusions
Energy discrepancies are useful objectives for energy-based modelling when the contrastive potential can be approximated easily and stably. In most cases this requires us to write the contrastive potential as an expectation which can be computed using Monte Carlo methods. We show how such a probabilistic representation can be achieved for a much larger class of stochastic processes via application of the Feynman-Kac formula. We first highlight the difficulty. Consider the integral  $\int f(\mathbf{y})q_t(\mathbf{y}|\mathbf{x})p_{\mathrm{data}}(\mathbf{x})\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y}$ . Since the expectation is taken in  $\mathbf{y}$ , the integral can be represented as an expectation of the forward process associated with  $q_{t}$ , i.e.
$$
\int f (\mathbf {y}) q _ {t} (\mathbf {y} | \mathbf {x}) p _ {\mathrm {d a t a}} (\mathbf {x}) \mathrm {d} \mathbf {x} \mathrm {d} \mathbf {y} = \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x} _ {0})} [ f (\mathbf {x} _ {t}) ] \approx \frac {1}{N} \sum_ {i = 1} ^ {N} f (\mathbf {x} _ {t} ^ {i})
$$
where  $\mathbf{x}_t^i$  are simulated processes initialised at  $\mathbf{x}_0^i = \mathbf{x}^i\sim p_{\mathrm{data}}$ . Next, consider the integral
$$
v (t, \mathbf {y}) := \int q _ {t} (\mathbf {y} | \mathbf {x}) g (\mathbf {x}) d \mathbf {x}.
$$
This integral is more difficult to approximate because the function  $g$  is evaluated at the starting point of the diffusion  $\mathbf{x}_t$  but weighted by its transition probability density. To compute such integrals without sampling from  $g$  we use the Feynman-Kac formula, see e.g. Øksendal (2003):
Theorem 6 (Feynman-Kac). Let  $g \in \mathcal{C}_0^2(\mathbb{R}^d)$  and  $c \in \mathcal{C}(\mathbb{R}^d)$ . Assume that  $v \in \mathcal{C}^{1,2}(\mathbb{R}_{\geq 0},\mathbb{R}^d)$  is bounded on  $K \times \mathbb{R}^d$  with  $K$  compact and satisfies
$$
\left\{ \begin{array}{c c} \partial_ {t} v (t, \mathbf {y}) = \mathcal {A} v (t, \mathbf {y}) + c (\mathbf {y}) v (t, \mathbf {y}) & \text {f o r a l l} t > 0, \mathbf {y} \in \mathbb {R} ^ {d} \\ v (0, \mathbf {y}) = g (\mathbf {y}) & \text {f o r a l l} \mathbf {y} \in \mathbb {R} ^ {d} \end{array} . \right. \tag {12}
$$
Then,  $v$  has the probabilistic representation
$$
v (t, \mathbf {y}) = \mathbb {E} _ {\mathbf {y}} \left[ \exp \left(\int_ {0} ^ {t} c (\mathbf {y} _ {s}) \mathrm {d} s\right) g (\mathbf {y} _ {t}) \right]
$$
where  $(\mathbf{y}_t)_{t\geq 0}$  is a diffusion process with infinitesimal generator  $\mathcal{A}$ .
We will establish that  $v(t, \mathbf{y})$  satisfies a partial differential equation of the above form which yields a probabilistic representation of the contrastive potential. We know that  $v(t, \mathbf{y})$  satisfies the Fokker-Planck equation (10). By applying the product rule to each term in the Fokker-Planck equation we find
$$
\begin{array}{l} \partial_ {t} v (t, \mathbf {y}) = \left(\overbrace {\sum_ {i = 1} ^ {d} \left(- a _ {i} (\mathbf {y}) + \sum_ {j = 1} ^ {d} \partial_ {\mathbf {y} _ {j}} \Sigma_ {i j} (\mathbf {y})\right)} ^ {\alpha (\mathbf {y})} \partial_ {\mathbf {y} _ {i}} + \frac {1}{2} \sum_ {i, j} ^ {d} \Sigma_ {i j} (\mathbf {y}) \partial_ {\mathbf {y} _ {i}} \partial_ {\mathbf {y} _ {j}}\right) v (t, \mathbf {y}) \\ + \underbrace {\left(\frac {1}{2} \sum_ {i , j = 1} ^ {d} \partial_ {\mathbf {y} _ {i}} \partial_ {\mathbf {y} _ {j}} \Sigma_ {i j} (\mathbf {y}) - \sum_ {i = 1} ^ {d} \partial_ {\mathbf {y} _ {i}} a _ {i} (\mathbf {y})\right)} _ {c (\mathbf {y})} v (t, \mathbf {y}) \\ \end{array}
$$
$$
v (0, \mathbf {y}) = g (\mathbf {y}).
$$
By comparing with Theorem 6, we identify the infinitesimal generator
$$
\mathcal {A} = \sum_ {i = 1} ^ {d} \alpha_ {i} (\mathbf {y}) \frac {\partial}{\partial_ {\mathbf {y} _ {i}}} + \frac {1}{2} \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} \Sigma_ {i j} (\mathbf {y}) \frac {\partial^ {2}}{\partial_ {\mathbf {y} _ {i}} \partial_ {\mathbf {y} _ {j}}}
$$
Hence we associate the forward diffusion process  $\mathrm{d}\mathbf{x}_t = a(\mathbf{x}_t)\mathrm{d}t + b(\mathbf{x}_t)\mathrm{d}\mathbf{w}_t$  with it's backwards process with infinitesimal generator  $\mathcal{A}$
$$
\mathrm {d} \mathbf {y} _ {t} = \alpha (\mathbf {y} _ {t}) \mathrm {d} t + b (\mathbf {y} _ {t}) \mathrm {d} \mathbf {w} _ {t} ^ {\prime}
$$
with  $\Sigma (\mathbf{y}) = b(\mathbf{y})b^{T}(\mathbf{y})$ . This yields the probabilistic representation of  $v(t,\mathbf{y})$  in terms of the backward process  $\mathbf{y}_t$ :
$$
v (t, \mathbf {y}) = \int q _ {t} (\mathbf {y} | \mathbf {x}) g (\mathbf {x}) \mathrm {d} \mathbf {x} = \mathbb {E} _ {\mathbf {y}} \left[ \exp \left(\int_ {0} ^ {t} c (\mathbf {y} _ {s}) \mathrm {d} s\right) g (\mathbf {y} _ {t}) \right]
$$
Hence, we also obtain a probabilistic representation for the contrastive potential by choosing  $g(\mathbf{x}) \coloneqq \exp(-U(\mathbf{x}))$ . This finally gives
$$
U _ {t} (\mathbf {y}) = - \log \mathbb {E} _ {\mathbf {y}} \left[ \exp \left(\int_ {0} ^ {t} c (\mathbf {y} _ {s}) \mathrm {d} s - U (\mathbf {y} _ {t})\right) \right].
$$
Unlike the contrasting term in contrastive divergence, this expression can indeed be calculated by simulating stochastic processes that are entirely independent of  $U$ . For this we simulate from the forward process starting at  $\mathbf{x}$  which yields  $\tilde{\mathbf{x}}_t$ , where the tilde denotes that this simulation may not be exact. We then simulate  $M$  copies of the reverse process and keep all values at intermediate steps, i.e.  $(\tilde{\mathbf{y}}_{t_0}^j = \tilde{\mathbf{x}}_t, \tilde{\mathbf{y}}_{t_1}^j, \dots, \tilde{\mathbf{y}}_{t_{K = t}}^j)$  for  $j = 1, \dots, M$ . Finally we evaluate the contrastive potential as
$$
U _ {t} (\mathbf {x} _ {t}) \approx - \log \frac {1}{M} \sum_ {j = 1} ^ {M} \exp \left(\left(\sum_ {k = 1} ^ {K} c \left(\tilde {\mathbf {y}} _ {t _ {k}} ^ {j}\right) \left(t _ {k} - t _ {k - 1}\right)\right) - U \left(\tilde {\mathbf {y}} _ {t} ^ {j}\right)\right)
$$
The simulation method for the stochastic process and for the integration  $\int_0^t c(\mathbf{y}_s)\mathrm{d}s$  may be altered in this approximation. At this stage, it is unclear what practical implications the weighting term  $\int_0^t c(\mathbf{y}_s)\mathrm{d}s$  has. Notice that the process  $\mathbf{y}_t$  is initialised at the final simulated position of the forward process  $\tilde{\mathbf{x}}_t$ . Furthermore, the bias correction with the  $w$ -stabilisation or an alternative method should still be relevant for stable training of energy-based models.
C Latent Space Energy-Based Prior Models
In this section, we first briefly review the latent space energy-based prior models (LEBMs) and its variants: CD-LEBM, SM-LEBM, and ED-LEBM. We then proceed to the experimental details.
C.1 A Brief Review of LEBMs
Latent space energy-based prior models (Pang et al., 2020) seek to model latent variable models  $p_{\phi, \theta}(\mathbf{x}) = \int p_{\phi}(\mathbf{x}|\mathbf{z})p_{\theta}(\mathbf{z})\mathrm{d}\mathbf{z}$  with an EBM prior  $p_{\theta}(\mathbf{z}) = \frac{\exp(-E_{\theta}(\mathbf{z}))p_{0}(\mathbf{z})}{Z_{\theta}}$ , where  $p_{0}(\mathbf{z})$  is a base distribution which we choose as standard Gaussian (Pang et al., 2020). LEBMs often perform better than latent variable models with a fixed Gaussian prior like VAEs since the EBM prior is more informative and expressive (Pang et al., 2020, Appendix C). However, training LEBMs is more expensive compared to latent variable models with fixed Gaussian prior because of the cost of training for energy-based models. This motivates to explore various training strategies for EBMs such as contrastive divergence, score matching, and the proposed energy discrepancy, where we find that energy discrepancy is the most efficient in terms of computational complexity.
The parameter update for the LEBM can be derived from maximum-likelihood estimation of  $p_{\phi, \theta}(\mathbf{x})$ . Using the identity  $\mathbb{E}_{p_{\phi, \theta}(\mathbf{z}|\mathbf{x})}[\nabla_{\phi, \theta}\log p_{\phi, \theta}(\mathbf{z}|\mathbf{x})] = 0$ , the gradient of the log-likelihood of a data point  $\mathbf{x}$  is given by
$$
\nabla_ {\phi , \theta} \log p _ {\phi , \theta} (\mathbf {x}) = \mathbb {E} _ {p _ {\phi , \theta} (\mathbf {z} | \mathbf {x})} [ \nabla_ {\phi , \theta} \log p _ {\phi , \theta} (\mathbf {z}, \mathbf {x}) ] = \mathbb {E} _ {p _ {\phi , \theta} (\mathbf {z} | \mathbf {x})} [ \nabla_ {\phi} \log p _ {\phi} (\mathbf {x} | \mathbf {z}) + \nabla_ {\theta} \log p _ {\theta} (\mathbf {z}) ].
$$
The posterior  $p_{\phi, \theta}(\mathbf{z}|\mathbf{x})$  prescribes the latent representation of the data point  $\mathbf{x}$ . Consequently, in each parameter update, samples are generated from the posterior distribution  $p_{\phi, \theta}(\mathbf{z}|\mathbf{x})$  via running Langevin dynamics and are treated as data on latent space. The generator is then updated via
$$
\nabla_ {\phi} \log p _ {\phi , \theta} (\mathbf {x}) = \mathbb {E} _ {p _ {\phi , \theta} (\mathbf {z} | \mathbf {x})} [ \nabla_ {\phi} \log p _ {\phi} (\mathbf {x} | \mathbf {z}) ],.
$$
Similarly, the maximum-likelihood update for the EBM parameters  $\theta$  is given by  $\nabla_{\theta}\log p_{\phi ,\theta}(\mathbf{x}) = \mathbb{E}_{p_{\phi ,\theta}(\mathbf{z}|\mathbf{x})}[\nabla_{\theta}\log p_{\theta}(\mathbf{z})]$ . As with any EBM, this gradient can not be used, directly, since this would require a tractable normalisation constant  $Z_{\theta}$ . To make this update tractable, we replace the gradient of the log-likelihood with contrastive divergence, score matching, and energy discrepancy as outlined below.
Algorithm 1 CD-LEBM	Algorithm 2 SM-LEBM	Algorithm 3 ED-LEBM
1: sample from posterior and prior z+~p(z|x); z-~pθ(z)	1: sample from posterior z~p(z|x)	1: sample from posterior z~p(z|x)
2: evaluate the energy difference dθ←Eθ(z+)−Eθ(z-)	2: evaluate the score difference dθ←∇z log pθ(z)−∇z log p(z|x)	2: evaluate the energy difference dθ←1/M∑j=1M eEθ(z)−Eθ(z+√tξ+√tξij)
3: Update parameter θ using (13) θ←θ−ηθ∇θdθ	3: Update parameter θ using (14) θ←θ−ηθ∇θ1/2||dθ||2	3: Update parameter θ using (15) θ←θ−ηθ∇θ log(w/M+dθ)
Figure 9: The training procedure for the EBM prior. We use one training sample only to illustrate.
Algorithm 4 Learning latent space energy-based prior models
1: repeat
2: Sample training data points {xi}i=1N ~ pdata(x)
3: For each xi, sample the corresponding latent variable zi ~ pφ,θ(z|xi) via zi(k+1) = zi(k) + ε/2 ∇z log pφ,θ(z|xi) + √εωk, ωk ~ N(0,I), zi0 ~ p0(z)
4: Update parameter φ by maximizing log-likelihood φ ← φ + ηφ∇φ 1/N ∑i=1N log pφ(zi|xi)
5: Update parameter α by running Algorithms 1, 2, or 3 θ ← θ - ηθ∇θLCD, SM, or ED(θ)
6: until convergence of parameters (φ,θ)
CD-LEBM (Pang et al., 2020). The contrastive divergence update is obtained as per usual by expressing the gradient of the log likelihood in terms of the energy function
$$
\nabla_ {\theta} \log p _ {\phi , \theta} (\mathbf {x}) = \mathbb {E} _ {p _ {\phi , \theta} (\mathbf {z} | \mathbf {x})} [ \nabla_ {\theta} \log p _ {\theta} (\mathbf {z}) ] = \mathbb {E} _ {p _ {\theta} (\mathbf {z})} [ \nabla_ {\theta} E _ {\theta} (\mathbf {z}) ] - \mathbb {E} _ {p _ {\phi , \theta} (\mathbf {z} | \mathbf {x})} [ \nabla_ {\theta} E _ {\theta} (\mathbf {z}) ].
$$
Therefore, the EBM prior can be learned by minimising
$$
\mathcal {L} _ {\mathrm {C D}} (\theta) := \frac {1}{N} \sum_ {i = 1} ^ {N} E _ {\theta} \left(\mathbf {z} _ {+} ^ {i}\right) - E _ {\theta} \left(\mathbf {z} _ {-} ^ {i}\right), \quad \mathbf {z} _ {+} ^ {i} \sim p _ {\phi , \theta} (\mathbf {z} | \mathbf {x} ^ {i}), \mathbf {z} _ {-} ^ {i} \sim p _ {\theta} (\mathbf {z}). \tag {13}
$$
Note that optimizing CD-LEBM is computationally expensive, as training the EBM prior requires simulating Langevin dynamics to sample  $\mathbf{z}$  from  $p_{\phi,\theta}(\mathbf{z}|\mathbf{x})$  to generate positive samples and  $p_{\theta}(\mathbf{z})$  to generate negative samples.
SM-LEBM. The second solution is to minimize the Fisher divergence between the posterior and prior, which has the following form
$$
\frac {1}{2} \mathbb {E} _ {p _ {\phi , \theta} (\mathbf {z} | \mathbf {x})} [ \| \nabla_ {\mathbf {z}} \log p _ {\theta} (\mathbf {z}) - \nabla_ {\mathbf {z}} \log p _ {\phi , \theta} (\mathbf {z} | \mathbf {x}) \| _ {2} ^ {2} ].
$$
This is equivalent to score matching (Hyvarinen & Dayan, 2005) when  $p_{\phi,\theta}(\mathbf{z}|\mathbf{x})$  is treated as parameter independent data distribution. We refer to this approach as score-matching LEBM, in which the EBM prior is learned by minimising
$$
\mathcal {L} _ {\mathrm {S M}} (\theta) := \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{2} \| \nabla_ {\mathbf {z}} \log p _ {\theta} \left(\mathbf {z} ^ {i}\right) - \nabla_ {\mathbf {z}} \log p \left(\mathbf {z} ^ {i} \mid \mathbf {x}\right) \| _ {2} ^ {2}, \quad \mathbf {z} ^ {i} \sim p _ {\phi , \theta} (\mathbf {z} \mid \mathbf {x} ^ {i}). \tag {14}
$$
where the parameters of  $p_{\phi, \theta}(\mathbf{z}|\mathbf{x})$  are suppressed in the update. Note that score matching generally requires computing the Hessian of the log density as in but in score-matching LEBM, we have  $\nabla_{\mathbf{z}}\log p(\mathbf{z}|\mathbf{x}) = \nabla_{\mathbf{z}}\log p(\mathbf{x}|\mathbf{z}) + \nabla_{\mathbf{z}}\log p(\mathbf{z})$ .
ED-LEBM. Finally, the EBM prior can be learned by minimising the energy discrepancy between the posterior and the EBM prior with  $\tilde{E}_{\theta}(\mathbf{z})\coloneqq E_{\theta}(\mathbf{z}) - \log p_0(\mathbf{z})$ , which can be estimated as follows
$$
\mathcal {L} _ {\mathrm {E D}} (\theta) := \frac {1}{N} \sum_ {i = 1} ^ {N} \log \left(\frac {w}{M} + \frac {1}{M} \sum_ {j = 1} ^ {M} \exp \left(\tilde {E} _ {\theta} \left(\mathbf {z} ^ {i}\right) - \tilde {E} _ {\theta} \left(\mathbf {z} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {i} + \sqrt {t} \boldsymbol {\xi} ^ {\prime i, j}\right)\right)\right) \tag {15}
$$
with  $\mathbf{z}^i\sim p_{\phi ,\theta}(\mathbf{z}|\mathbf{x}^i)$ . Note that energy discrepancy does not require simulating MCMC sampling on the EBM prior and calculating the score of the log density, which is computationally friendly for large-scale training. It is critical to include the base distribution  $p_0(\mathbf{z})$  in the energy function  $\tilde{E}_{\theta}$ . We summarize the training process of the EBM prior using CD-, SM-, and ED-LEBM in Algorithms 1, 2, and 3, with the training procedure of LEBM given in Algorithm 4.
Table 3: Model architectures of LEBMs on various datasets.
(a) Generator for SVHN 32 × 32, ngf = 64	(b) Generator for CIFAR-10 32 × 32, ngf = 128
Layers	In-Out Size	Stride	Layers	In-Out Size	Stride
Input: x	1x1x100	-	Input: x	1x1x128	-
4x4 convT(ngf x 8), LReLU	4x4x(ngf x 8)	1	8x8 convT(ngf x 8), LReLU	8x8x(ngf x 8)	1
4x4 convT(ngf x 4), LReLU	8x8x(ngf x 4)	2	4x4 convT(ngf x 4), LReLU	16x16x(ngf x 4)	2
4x4 convT(ngf x 2), LReLU	16x16x(ngf x 2)	2	4x4 convT(ngf x 2), LReLU	32x32x(ngf x 2)	2
4x4 convT(3), Tanh	32x32x3	2	3x3 convT(3), Tanh	32x32x3	1
(c) Generator for CelebA 64 × 64, ngf = 128	(d) Generator for MNIST 28 × 28, ngf = 16
Layers	In-Out Size	Stride	Layers	In-Out Size	Stride
Input: x	1x1x100	-	Input: x	16	-
4x4 convT(ngf x 8), LReLU	4x4x(ngf x 8)	1	4x4 convT(ngf x 8), LReLU	4x4x(ngf x 8)	1
4x4 convT(ngf x 4), LReLU	8x8x(ngf x 4)	2	3x3 convT(ngf x 4), LReLU	7x7x(ngf x 4)	2
4x4 convT(ngf x 2), LReLU	16x16x(ngf x 2)	2	4x4 convT(ngf x 2), LReLU	14x14x(ngf x 2)	2
4x4 convT(ngf x 1), LReLU	32x32x(ngf x 1)	2	4x4 convT(1), Tanh	28x28x1	2
4x4 convT(3), Tanh	64x64x3	2
(d) EBM prior
Layers	In-Out Size
Input: z	16/100/128
Linear, LReLU	200
Linear, LReLU	200
Linear	1
C.2 Langevin Sampling, Reconstruction, and Generation
To sample from the EBM prior  $p_{\theta}(\mathbf{z})$  and posterior  $p_{\phi, \theta}(\mathbf{z}|\mathbf{x})$  we employ a standard unadjusted Langevin sampling routine, i.e. we repeat for  $k = 0,1,\ldots,K$
$$
\mathbf {z} _ {k + 1} ^ {i} = \mathbf {z} _ {k} ^ {i} + \frac {\epsilon}{2} \nabla_ {\mathbf {z}} \log p (\mathbf {z}) + \sqrt {\epsilon} \boldsymbol {\omega} _ {k}, \quad \boldsymbol {\omega} _ {k} \sim \mathcal {N} (0, \mathbf {I})
$$
where  $\mathbf{z}_0\sim p_0(\mathbf{z})$  and the distribution  $p(\mathbf{z})$  is replaced by the prior or posterior densities, respectively.
The generator is modelled as the Gaussian  $p_{\phi}(\mathbf{x}|\mathbf{z}) = \mathcal{N}(\mu_{\phi}(z),\sigma^{2}\mathbf{I})$ . In reconstruction of  $\mathbf{x}$ , we sample from the posterior  $\mathbf{z}_{\mathbf{x}}\sim p_{\phi ,\theta}(\mathbf{z}|\mathbf{x})$  and compute the reconstruction as  $\hat{\mathbf{x}} = \mu_{\phi}(\mathbf{z}_{\mathbf{x}})$ . In data generation, we sample from the EBM prior  $\mathbf{z}_{\mathrm{gen}}\sim p_{\theta}(\mathbf{z})$  and compute the generated synthetic data point as  $\mathbf{x}_{\mathrm{gen}} = \mu_{\phi}(\mathbf{z}_{\mathrm{gen}})$ .
C.3 Experimental Details of LEBMs