SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {G} := \left\{U: \mathcal {X} \mapsto \mathbb {R} \text {s u c h t h a t} \exp (- U) \in L ^ {1} (\mathcal {X}, \mathrm {d x}), U \in L ^ {1} (p _ {\text {d a t a}}), \text {a n d} \min  _ {\mathbf {x} \in \mathcal {X}} U (x) = 0 \right\}
$$
We then make the following assumptions on  $q$  and  $\mathcal{G}$ :
1. All elements of  $\mathcal{X}$  are  $q$ -equivalent, i.e. for every  $\mathbf{x} \in \mathcal{X}$  it holds that  $[\mathbf{x}]_q = \mathcal{X}$ .
2. There exists a  $U^{*}\in \mathcal{G}$  such that  $\exp (-U^{*})\propto p_{\mathrm{data}}$
Under Assumption 1,  $\mathrm{ED}_q(p_{\mathrm{data}},U)$  has a unique global minimiser  $U^{*} = -\log p_{\mathrm{data}} + c$  in  $\mathcal{G}$ . We prove this by computing the first and second variation of  $\mathrm{ED}_q$ . Note that  $\mathcal{G}$  may not be a vector space in general since in some cases  $0\notin \mathcal{G}$ . Furthermore, we did not specify a norm on  $\mathcal{G}$ . We will omit this technical issue in this discussion. We start from the following lemmata and complete the proof of Theorem 1 in Corollary 1.
Lemma 1. Let  $h \in \mathcal{G}$  be arbitrary. The first variation of  $\mathrm{ED}_q$  is given by
$$
\frac {\mathrm {d}}{\mathrm {d} \epsilon} \operatorname {E D} _ {q} \left(p _ {\text {data}}, U + \epsilon h\right) \Bigg | _ {\epsilon = 0} = \mathbb {E} _ {p _ {\text {data}} (\mathbf {x})} [ h (\mathbf {x}) ] - \mathbb {E} _ {p _ {\text {data}} (\mathbf {x})} \mathbb {E} _ {q (\mathbf {y} | \mathbf {x})} \mathbb {E} _ {p _ {U} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ] \tag {7}
$$
where  $p_U(\mathbf{z}|\mathbf{y}) = \frac{q(\mathbf{y}|\mathbf{z})\exp(-U(\mathbf{z}))}{\int q(\mathbf{y}|\mathbf{z}')\exp(-U(\mathbf{z}'))\mathrm{d}\mathbf{z}'}.$
Proof. We define the short-hand notation  $U_{\epsilon} \coloneqq U + \epsilon h$ . The energy discrepancy at  $U_{\varepsilon}$  reads
$$
\mathrm {E D} _ {q} (p _ {\mathrm {d a t a}}, U _ {\epsilon}) = \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} [ U _ {\epsilon} (\mathbf {x}) ] + \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} \mathbb {E} _ {q (\mathbf {y} | \mathbf {x})} \left[ \log \int q (\mathbf {y} | \mathbf {z}) \exp (- U _ {\epsilon} (\mathbf {z})) \mathrm {d} \mathbf {z} \right].
$$
For the first functional derivative, we only need to calculate
$$
\frac {\mathrm {d}}{\mathrm {d} \epsilon} \log \int q (\mathbf {y} | \mathbf {z}) \exp (- U _ {\epsilon} (\mathbf {z})) \mathrm {d} \mathbf {z} = \int \frac {- q (\mathbf {y} | \mathbf {z}) h (\mathbf {z}) \exp (- U _ {\epsilon} (\mathbf {z}))}{\int q (\mathbf {y} | \mathbf {z} ^ {\prime}) \exp (- U _ {\epsilon} (\mathbf {z} ^ {\prime})) \mathrm {d} \mathbf {z} ^ {\prime}} \mathrm {d} \mathbf {z} = - \mathbb {E} _ {p _ {U _ {\epsilon}} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ]. \tag {8}
$$
Plugging this expression into  $\mathrm{ED}_q(p_{\mathrm{data}},U_\epsilon)$  and setting  $\epsilon = 0$  yields the first variation of  $\mathrm{ED}_q$ .
Lemma 2. The second variation of  $\mathrm{ED}_q$  is given by
$$
\frac {\mathrm {d} ^ {2}}{\mathrm {d} \epsilon^ {2}} \mathrm {E D} _ {q} (p _ {\mathrm {d a t a}}, U + \epsilon h) \Bigg | _ {\epsilon = 0} = \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} \mathbb {E} _ {q (\mathbf {y} | \mathbf {x})} \operatorname {V a r} _ {p _ {U} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ].
$$
Proof. For the second order term, we have based on equation 8 and the quotient rule for derivatives:
$$
\begin{array}{l} \frac {\mathrm {d} ^ {2}}{\mathrm {d} \epsilon^ {2}} \log \int q (\mathbf {y} | \mathbf {z}) \exp (- U _ {\epsilon} (\mathbf {z})) \mathrm {d} \mathbf {z} \\ = \frac {\int q (\mathbf {y} | \mathbf {z}) \exp (U _ {\epsilon} (\mathbf {z})) h ^ {2} (\mathbf {z}) \mathrm {d} \mathbf {z} \int q (\mathbf {y} | \mathbf {z} ^ {\prime}) \exp (- U _ {\epsilon} (\mathbf {z} ^ {\prime})) \mathrm {d} \mathbf {z} ^ {\prime}}{\left(\int q (\mathbf {y} | \mathbf {z} ^ {\prime}) \exp (- U _ {\epsilon} (\mathbf {z} ^ {\prime})) \mathrm {d} \mathbf {z} ^ {\prime}\right) ^ {2}} \\ - \frac {\int q (\mathbf {y} | \mathbf {z}) \exp (U _ {\epsilon} (\mathbf {z})) h (\mathbf {z}) d \mathbf {z} \int q (\mathbf {y} | \mathbf {z} ^ {\prime}) \exp (- U _ {\epsilon} (\mathbf {z} ^ {\prime})) h (\mathbf {z} ^ {\prime}) d \mathbf {z} ^ {\prime}}{\left(\int q (\mathbf {y} | \mathbf {z} ^ {\prime}) \exp (- U _ {\epsilon} (\mathbf {z} ^ {\prime})) d \mathbf {z} ^ {\prime}\right) ^ {2}} \\ = \mathbb {E} _ {p _ {U _ {\epsilon}} (\mathbf {z} | \mathbf {y})} [ h ^ {2} (\mathbf {z}) ] - \mathbb {E} _ {p _ {U _ {\epsilon}} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ] ^ {2} = \operatorname {V a r} _ {p _ {U _ {\epsilon}} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ]. \\ \end{array}
$$
We obtain the desired result by interchanging the outer expectations with the derivatives in  $\epsilon$ .
Corollary 1. Let  $c = \min_{\mathbf{x} \in \mathcal{X}}(-\log p_{\mathrm{data}}(\mathbf{x}))$ . For  $U^{*} = -\log (p_{\mathrm{data}}) - c \in \mathcal{G}$  it holds under Assumption 1 that
$$
\begin{array}{l} \frac{\mathrm{d}}{\mathrm{d}\epsilon}\mathrm{ED}_{q}(p_{\mathrm{data}},U^{*} + \epsilon h)\bigg|_{\epsilon = 0} = 0 \\ \frac{\mathrm{d}^{2}}{\mathrm{d}\epsilon^{2}}\mathrm{ED}_{q}(p_{\mathrm{data}},U^{*} + \epsilon h)\bigg|_{\epsilon = 0} > 0\qquad \text{for all}\qquad h\in \mathcal{G}  . \\ \end{array}
$$
Furthermore,  $U^{*}$  is the unique global minimiser of  $\mathrm{ED}_q(p_{\mathrm{data}},\cdot)$  in  $\mathcal{G}$ .
Proof. By definition, the variance is non-negative, i.e. for every  $h \in \mathcal{G}$ :
$$
\frac {\mathrm {d} ^ {2}}{\mathrm {d} \epsilon^ {2}} \mathrm {E D} _ {q} (p _ {\text {d a t a}}, U + \epsilon h) \bigg | _ {\epsilon = 0} = \operatorname {V a r} _ {p _ {U} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ] \geq 0.
$$
Consequently, the energy discrepancy is convex and an extremal point of  $\mathrm{ED}_q(p_{\mathrm{data}},\cdot)$  is a global minimiser. We are left to show that the minimiser is obtained at  $U^{*}$  and unique. First of all, we have:
$$
\mathbb {E} _ {p _ {U ^ {*}} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ] = \int \frac {q (\mathbf {y} | \mathbf {z}) \exp (- U ^ {*} (\mathbf {z}))}{\int q (\mathbf {y} | \mathbf {z} ^ {\prime}) \exp (- U ^ {*} (\mathbf {z} ^ {\prime})) \mathrm {d} \mathbf {z} ^ {\prime}} h (\mathbf {z}) \mathrm {d} \mathbf {z} = \int \frac {q (\mathbf {y} | \mathbf {z}) p _ {\mathrm {d a t a}} (\mathbf {z})}{\int q (\mathbf {y} | \mathbf {z} ^ {\prime}) p _ {\mathrm {d a t a}} (\mathbf {z} ^ {\prime}) \mathrm {d} \mathbf {z} ^ {\prime}} h (\mathbf {z}) \mathrm {d} \mathbf {z}.
$$
By applying the outer expectations we obtain
$$
\begin{array}{l} \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} \mathbb {E} _ {q (\mathbf {y} | \mathbf {x})} \mathbb {E} _ {p _ {U ^ {*}} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ] = \int \int p _ {\mathrm {d a t a}} (\mathbf {x}) q (\mathbf {y} | \mathbf {x}) \mathrm {d} \mathbf {x} \int \frac {q (\mathbf {y} | \mathbf {z}) p _ {\mathrm {d a t a}} (\mathbf {z})}{\int q (\mathbf {y} | \mathbf {z} ^ {\prime}) p _ {\mathrm {d a t a}} (\mathbf {z} ^ {\prime}) \mathrm {d} \mathbf {z} ^ {\prime}} h (\mathbf {z}) \mathrm {d} \mathbf {y} \mathrm {d} \mathbf {z} \\ = \int \int q (\mathbf {y} | \mathbf {x}) p _ {\mathrm {d a t a}} (\mathbf {z}) h (\mathbf {z}) \mathrm {d} \mathbf {y} \mathrm {d} \mathbf {z} = \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {z})} [ h (\mathbf {z}) ], \\ \end{array}
$$
where we used that the marginal distributions  $\int p_{\mathrm{data}}(\mathbf{x})q(\mathbf{y}|\mathbf{x})\mathrm{d}\mathbf{x}$  cancel out and the conditional probability density integrates to one. This implies
$$
\frac{\mathrm{d}}{\mathrm{d}\epsilon}\mathrm{ED}_{q}(p_{\mathrm{data}},U^{*} + \epsilon h)\bigg|_{\epsilon = 0} = \mathbb{E}_{p_{\mathrm{data}}(\mathbf{z})}[h(\mathbf{z})] - \mathbb{E}_{p_{\mathrm{data}}(\mathbf{z})}[h(\mathbf{z})] = 0.
$$
for all  $h \in \mathcal{G}$ . We now show that the second variation is also positive definite, i.e.
$$
\frac {\mathrm {d} ^ {2}}{\mathrm {d} \epsilon^ {2}} \mathrm {E D} _ {q} (p _ {\mathrm {d a t a}}, U ^ {*} + \epsilon h) \bigg | _ {\epsilon = 0} = \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} \mathbb {E} _ {q (\mathbf {y} | \mathbf {x})} \operatorname {V a r} _ {p _ {\mathrm {d a t a}} (\mathbf {z} | \mathbf {y})} [ h (\mathbf {z}) ] > 0.
$$
where  $p_{\mathrm{data}}(\mathbf{z}|\mathbf{y}) = \frac{q(\mathbf{y}|\mathbf{z})p_{\mathrm{data}}(\mathbf{z})}{\int q(\mathbf{y}|\mathbf{z}')p_{\mathrm{data}}(\mathbf{z}')\mathrm{d}\mathbf{z}'}$ . Assume that there exists a test function  $h \in \mathcal{G}$  such that  $\mathbb{E}_{p_{\mathrm{data}}(\mathbf{x})}\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\operatorname{Var}_{p_{\mathrm{data}}(\mathbf{z}|\mathbf{y})}[h(\mathbf{z})] = 0$ . Since this is the expectation of a non-negative random variable, it follows for all  $\mathbf{y} \in \mathcal{V}$  that  $\operatorname{Var}_{p_{\mathrm{data}}(\mathbf{z}|\mathbf{y})}[h(\mathbf{z})] = 0$ . Consequently,  $h(\mathbf{z})$  has to be constant  $p_{\mathrm{data}}(\cdot |\mathbf{y})$ -almost surely and there exists a measurable function  $g: \mathcal{V} \to \mathbb{R}$  such that  $h(\mathbf{z}) = g(\mathbf{y})$  for all  $\mathbf{z} \in \operatorname{supp}(p(\cdot |\mathbf{y}))$ . Next, assume that  $\mathbf{z}_1$  and  $\mathbf{z}_2$  are  $q$ -neighbours. By definition, there exists a  $\mathbf{y} \in \mathcal{V}$  such that  $\mathbf{z}_1, \mathbf{z}_2 \in \operatorname{supp}(q(\mathbf{y}|\cdot))$ . Since the data density is positive, the support of  $q(\mathbf{y}|\cdot)$  equals the support of  $p_{\mathrm{data}}(\cdot |\mathbf{y})$ . Consequently, it holds that  $h(\mathbf{z}_1) = g(\mathbf{y}) = h(\mathbf{z}_2)$ . It now follows from the definition of  $q$ -equivalence that  $h$  has to be constant almost surely on all  $q$ -equivalence classes  $[\mathbf{x}]_q$ , since any two elements in an equivalence class can be connected by a chain of  $q$ -neighbours. By Assumption 1,  $[\mathbf{x}]_q = \mathcal{X}$ , and thus  $h$  is constant on  $\mathcal{X}$ . However, there are no non-trivial constant test function in  $\mathcal{G}$ . Hence, the second variation has to be positive definite for all  $h \in \mathcal{G}$ . Consequently,  $U^*$  is the unique global minimiser of  $\operatorname{ED}_q$  which completes the statement in Theorem 1.
A.2 Equivalence of Energy Discrepancy for Brownian Motion and Ornstein-Uhlenbeck Processes
In this subsection we show that an energy discrepancy based on an Ornstein-Uhlenbeck process is equivalent to the energy discrepancy based on a time-changed Brownian motion.
Proposition 1. Let  $q_{t}$  be the transition density for the Ornstein-Uhlenbeck process  $\mathrm{d}\mathbf{x}_t = \alpha \mathbf{x}_t\mathrm{d}t + \sqrt{\beta}\mathrm{d}\mathbf{w}_t$  with standard Brownian motion  $\mathbf{w}_t$ , and let  $\gamma_t(\mathbf{y}|\mathbf{x}) \propto \exp (-\| \mathbf{y} - \mathbf{x}\|^2 /2t)$  be the Gaussian transition density of Brownian motion. Then,
$$
\mathrm {E D} _ {q _ {t}} (p _ {\mathrm {d a t a}}, U) = \mathrm {E D} _ {\gamma_ {\sigma_ {- \alpha} ^ {2} (t)}} (p _ {\mathrm {d a t a}}, U) - \alpha t
$$
where  $\sigma_{\alpha}(t) = \sqrt{\frac{\beta}{2\alpha} (e^{2\alpha t} - 1)}$  and  $\sigma_0(t) = \sqrt{\beta t}$ .
Proof. At time  $t$ , the Ornstein-Uhlenbeck process has distribution
$$
\mathbf {x} _ {t} \stackrel {{d}} {{=}} e ^ {\alpha t} \mathbf {x} _ {0} + \sigma_ {\alpha} (t) \boldsymbol {\xi}, \quad \boldsymbol {\xi} \sim \mathcal {N} (0, \mathbf {I}) \tag {9}
$$
with  $\sigma_{\alpha}(t) = \sqrt{\frac{\beta}{2\alpha} (e^{2\alpha t} - 1)}$  and  $\sigma_0(t) = \sqrt{\beta t}$ . The Ornstein-Uhlenbeck process is variance exploding for  $\alpha \geq 0$  and variance preserving for  $\alpha < 0$ . Based on (9), the transition density of  $\mathbf{x}_t$  is given as
$$
q _ {t} (\mathbf {y} | \mathbf {x}) = \frac {1}{\sqrt {2 \pi \sigma_ {\alpha} (t) ^ {d}}} \exp \left(- \frac {\| \mathbf {y} - e ^ {\alpha t} \mathbf {x} \| ^ {2}}{2 \sigma_ {\alpha} ^ {2} (t)}\right)
$$
Hence, we obtain via the change of variables  $\xi^{\prime} := (\mathbf{y} - e^{\alpha t}\mathbf{x}) / \sigma_{\alpha}(t) \sim \mathcal{N}(0,\mathbf{I})$  for the contrastive potential
$$
\begin{array}{l} U _ {t} (\mathbf {y}) = - \log \int q _ {t} (\mathbf {y} | \mathbf {x}) \exp (- U (\mathbf {x})) \mathrm {d} \mathbf {x} \\ = - \log \int \gamma_ {1} (\boldsymbol {\xi} ^ {\prime}) \exp \left(- U \left(e ^ {- \alpha t} (\mathbf {y} - \sigma_ {\alpha} (t) \boldsymbol {\xi} ^ {\prime})\right)\right) d \boldsymbol {\xi} ^ {\prime} - \alpha t. \\ \end{array}
$$
We now evaluate the contrastive potential at the forward process  $\mathbf{y} = \mathbf{x}_t$  which yields
$$
\begin{array}{l} U _ {t} (\mathbf {x} _ {t}) = - \log \int \gamma_ {1} (\pmb {\xi} ^ {\prime}) \exp \left(- U \left(e ^ {- \alpha t} (e ^ {\alpha t} \mathbf {x} _ {0} + \sigma_ {\alpha} (t) \pmb {\xi} - \sigma_ {\alpha} (t) \pmb {\xi} ^ {\prime})\right)\right) d \pmb {\xi} ^ {\prime} - \alpha t \\ = - \log \int \gamma_ {1} (\pmb {\xi} ^ {\prime}) \exp \left(- U \left(\mathbf {x} _ {0} + \sigma_ {- \alpha} (t) \pmb {\xi} - \sigma_ {- \alpha} (t) \pmb {\xi} ^ {\prime}\right)\right) \mathrm {d} \pmb {\xi} ^ {\prime} - \alpha t \\ = - \log \int \gamma_ {\sigma_ {- \alpha} ^ {2} (t)} \left(\mathbf {w} _ {\sigma_ {- \alpha} ^ {2} (t)} - \mathbf {x}\right) \exp (- U (\mathbf {x}) \mathrm {d} \mathbf {x} - \alpha t \\ \end{array}
$$
where we used that  $e^{-\alpha t}\sigma_{\alpha}(t) = \sigma_{-\alpha}(t)$  in the second equality and the change of variables  $\mathbf{x} = \mathbf{w}_{\sigma_{-\alpha}^2 (t)} - \xi^\prime$  in the third equality. Hence, the energy discrepancy for the Ornstein-Uhlenbeck process is equivalent to the energy discrepancy for Brownian motion with time parameter
$$
\sigma_ {- \alpha} (t) = \sqrt {\frac {\beta}{2 \alpha} (1 - e ^ {- 2 \alpha t})}
$$
Notice that for the variance-exploding process with  $\alpha > 0$  the contrasting particles have a finite horizon since  $\sigma_{-\alpha}(t) \xrightarrow{t \to \infty} \sqrt{\frac{\beta}{2\alpha}} < \infty$ . Hence, the maximum-likelihood limit in Theorem 2 is only achieved for the variance preserving process with  $\alpha < 0$  and for the critical case of Brownian motion with  $\alpha = 0$ .
A.3 Interpolation between Score-Matching and Maximum-Likelihood Estimation
We first prove the result as stated in the Gaussian case. We then show how the result can be generalised to arbitrary diffusions by using Ito calculus.
Gaussian case Denote the Gaussian density as
$$
\gamma_ {t} (\mathbf {y} - \mathbf {x}) := \frac {1}{\sqrt {2 \pi t} ^ {d}} \exp \left(- \frac {\| \mathbf {y} - \mathbf {x} \| ^ {2}}{2 t}\right).
$$
and define the convolved distributions  $p_t \coloneqq \gamma_t * p_{\mathrm{data}}$  and  $\exp(-U_t) \coloneqq \gamma_t * \exp(-U)$ .
Proposition 2. The energy discrepancy is the multi noise-scale score-matching loss
$$
\mathrm {E D} _ {\gamma_ {t}} (p _ {\mathrm {d a t a}}, U) = \int_ {0} ^ {t} \mathbb {E} _ {p _ {s} (\mathbf {x} _ {s})} \left[ - \Delta U _ {s} (\mathbf {x} _ {s}) + \frac {1}{2} \| \nabla U _ {s} (\mathbf {x} _ {s}) \| ^ {2} \right] \mathrm {d} s
$$
Proof. It is known that  $\gamma_{t}$  is the solution of the heat equation:
$$
\partial_ {t} \gamma_ {t} (\mathbf {y} - \mathbf {x}) = \frac {1}{2} \Delta_ {y} \gamma_ {t} (\mathbf {y} - \mathbf {x}).
$$
Consequently, both,  $p_t$  and  $\exp(-U_t)$  satisfy the heat-equation because the integral commutes with the differential operators. Based on the heat-equation we can derive the following non-linear partial differential equation for the contrastive potential  $U_t$ :
$$
\begin{array}{l} \partial_ {t} e ^ {- U _ {t} (\mathbf {y})} = \int \partial_ {t} \gamma_ {t} (\mathbf {y} - \mathbf {x}) e ^ {- U (\mathbf {x})} d \mathbf {x} \\ = \frac {1}{2} \int \Delta_ {\mathbf {y}} \gamma_ {t} (\mathbf {y} - \mathbf {x}) e ^ {- U (\mathbf {x})} d \mathbf {x} \\ = \frac {1}{2} \Delta_ {\mathbf {y}} e ^ {- U _ {t} (\mathbf {y})} \\ = - \frac {1}{2} \nabla_ {\mathbf {y}} \cdot \left(\left(\nabla_ {\mathbf {y}} U _ {t} (\mathbf {y})\right) e ^ {- U _ {t} (\mathbf {y})}\right) \\ = \left(\frac {1}{2} \| \nabla_ {\mathbf {y}} U _ {t} (\mathbf {y}) \| ^ {2} - \frac {1}{2} \Delta_ {\mathbf {y}} U _ {t} (\mathbf {y})\right) e ^ {- U _ {t} (\mathbf {y})} \\ \end{array}
$$
Since  $\partial_t e^{-U_t} = -e^{-U_t}\partial_t U_t$ , we get after cancellation of the exponentials:
$$
\partial_ {t} U _ {t} (\mathbf {y}) = \frac {1}{2} \Delta_ {y} U _ {t} (\mathbf {y}) - \frac {1}{2} \| \nabla_ {\mathbf {y}} U _ {t} (\mathbf {y}) \| ^ {2}
$$
The integral notation of the contrastive term in energy discrepancy takes the form
$$
\mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {t} (\mathbf {y} - \mathbf {x})} [ U _ {t} (\mathbf {y}) ] = \int U _ {t} (\mathbf {y}) p _ {t} (\mathbf {y}) d \mathbf {y}.
$$
We now take a derivative of the energy discrepancy and find
$$
\begin{array}{l} \partial_ {t} \mathrm {E D} _ {\gamma_ {t}} (p _ {\text {d a t a}}, U) = - \partial_ {t} \int U _ {t} (\mathbf {y}) p _ {t} (\mathbf {y}) \mathrm {d} \mathbf {y} \\ = - \int (\partial_ {t} U _ {t} (\mathbf {y})) p _ {t} (\mathbf {y}) d \mathbf {y} - \int U _ {t} (\mathbf {y}) \partial_ {t} p _ {t} (\mathbf {y}) d \mathbf {y} \\ = - \int \left(\partial_ {t} U _ {t} (\mathbf {y})\right) p _ {t} (\mathbf {y}) d \mathbf {y} - \int U _ {t} (\mathbf {y}) \frac {1}{2} \Delta_ {\mathbf {y}} p _ {t} (\mathbf {y}) d \mathbf {y} \\ = - \int (\partial_ {t} U _ {t} (\mathbf {y})) p _ {t} (\mathbf {y}) d \mathbf {y} - \int \frac {1}{2} (\Delta_ {\mathbf {y}} U _ {t} (\mathbf {y})) p _ {t} (\mathbf {y}) d \mathbf {y} \\ \end{array}
$$
where we used integration by parts twice in the final equation to shift the differential operator from  $p_t$  to  $U_t$ . Now, plugging in the differential equation for  $U_t$  we find
$$
\begin{array}{l} \partial_ {t} \mathrm {E D} _ {\gamma_ {t}} (p _ {\mathrm {d a t a}}, U) = \int \left(- \frac {1}{2} \Delta_ {\mathbf {y}} U _ {t} (\mathbf {y}) + \frac {1}{2} \| \nabla_ {\mathbf {y}} U _ {t} (\mathbf {y}) \| ^ {2} - \frac {1}{2} \Delta_ {\mathbf {y}} U _ {t} (\mathbf {y})\right) p _ {t} (\mathbf {y}) \mathrm {d} \mathbf {y} \\ = \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {t} (\mathbf {y} - \mathbf {x})} \left[ - \Delta_ {\mathbf {y}} U _ {t} (\mathbf {y}) + \frac {1}{2} \| \nabla_ {\mathbf {y}} U _ {t} (\mathbf {y}) \| ^ {2} \right] \\ \end{array}
$$
Finally, we obtain energy discrepancy by integrating above expression:
$$
\begin{array}{l} \mathrm {E D} _ {\gamma_ {t}} \left(p _ {\text {d a t a}}, U\right) = \int_ {0} ^ {t} \partial_ {s} \mathrm {E D} _ {\gamma_ {s}} \left(p _ {\text {d a t a}}, U\right) \mathrm {d} s \\ = \int_ {0} ^ {t} \mathbb {E} _ {p _ {\text {d a t a}} (\mathbf {x})} \mathbb {E} _ {\gamma_ {s} (\mathbf {y} - \mathbf {x})} \left[ - \Delta_ {\mathbf {y}} U _ {s} (\mathbf {y}) + \frac {1}{2} \| \nabla_ {\mathbf {y}} U _ {s} (\mathbf {y}) \| ^ {2} \right] d s \\ \end{array}
$$
This gives the desired integral representation in Proposition 2.
Proposition 3. Let  $\gamma_{t}$  be the Gaussian transition density and  $p_{\mathrm{ebm}}\propto \exp (-U)$  the energy-based distribution. The energy discrepancy converges to a cross entropy loss at a linear rate in time
$$
\left| \mathrm {E D} _ {\gamma_ {t}} (p _ {\mathrm {d a t a}}, U) + \mathbb {E} _ {p _ {\mathrm {d a t a}} (\mathbf {x})} [ \log p _ {\mathrm {e b m}} (\mathbf {x}) ] - c (t) \right| \leq \frac {1}{2 t} \mathbb {W} _ {2} ^ {2} (p _ {\mathrm {d a t a}}, p _ {\mathrm {e b m}})
$$
where  $c(t)$  is a renormalising constant independent of  $U$ .
For the proof we employ the following lemma of Yihong Wu which was given in Raginsky & Sason (2013).
Lemma 3. Let  $\gamma_{t}$  be the Gaussian transition density of a standard Brownian motion. Let  $\mu, \nu$  be probability distributions and denote  $\mu_{t} \coloneqq \gamma_{t} * \mu$  and  $\nu_{t} \coloneqq \gamma_{t} * \nu$ . The following information-transport inequality holds:
$$
\mathrm {K L} (\mu_ {t} \parallel \nu_ {t}) \leq \frac {1}{2 t} \mathbb {W} _ {2} ^ {2} (\mu , \nu)
$$
Proof. Let  $\pi$  be a probability density of  $(\mathbf{x},\mathbf{x}^{\prime})$  with marginal distributions  $\mu (\mathbf{x})$  and  $\nu (\mathbf{x}^{\prime})$  (also called a coupling in optimal transport). We have by the decomposition of Kullback-Leibler divergences
$$
\begin{array}{l} \int \operatorname {K L} \left(\gamma_ {t} (\cdot - \mathbf {x}) \| \gamma_ {t} (\cdot - \mathbf {x} ^ {\prime})) \pi (\mathbf {x}, \mathbf {x} ^ {\prime}) \mathrm {d} \mathbf {x} \mathrm {d} \mathbf {x} ^ {\prime} - \operatorname {K L} \left(\mu_ {t} \| \nu_ {t}\right) \right. \\ = \int \operatorname {K L} \left(\frac {\gamma_ {t} (\mathbf {y} - \mathbf {x}) \pi \left(\mathbf {x} , \mathbf {x} ^ {\prime}\right)}{\mu_ {t} (\mathbf {y})} \| \frac {\gamma_ {t} (\mathbf {y} - \mathbf {x}) \pi \left(\mathbf {x} , \mathbf {x} ^ {\prime}\right)}{\nu_ {t} (\mathbf {y})}\right) \mu_ {t} (\mathbf {y}) \mathrm {d} \mathbf {y} \geq 0 \\ \end{array}
$$
Hence, we find by rearranging the inequality
$$
\operatorname {K L} \left(\mu_ {t} \| \nu_ {t}\right) \leq \int \operatorname {K L} \left(\gamma_ {t} (\cdot - \mathbf {x}) \| \gamma_ {t} (\cdot - \mathbf {x} ^ {\prime})\right) \pi \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) \mathrm {d} \mathbf {x} \mathrm {d} \mathbf {x} ^ {\prime}
$$
The right hand side is the Kullback-Leibler divergence between Gaussians, so
$$
\operatorname {K L} \left(\gamma_ {t} (\cdot - \mathbf {x}) \| \gamma_ {t} (\cdot - \mathbf {x} ^ {\prime})\right) = \frac {1}{2 t} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| ^ {2}
$$
Since the coupling was arbitrary, we can minimise over all couplings  $\pi$  of  $\mu$  and  $\nu$  which results in the Wasserstein-distance
$$
\mathrm {K L} \left(\mu_ {t} \| \nu_ {t}\right) \leq \min  _ {\pi \in \Pi (\mu , \nu)} \int \mathrm {K L} \left(\gamma_ {t} (\cdot - \mathbf {x}) \| \gamma_ {t} (\cdot - \mathbf {x} ^ {\prime})\right) \pi (\mathbf {x}, \mathbf {x} ^ {\prime}) \mathrm {d} \mathbf {x} \mathrm {d} \mathbf {x} ^ {\prime} = \frac {1}{2 t} \mathbb {W} _ {2} ^ {2} (\mu , \nu)
$$
where  $\Pi (\mu ,\nu)$  denotes the set of all joint distributions with marginals  $\mu$  and  $\nu$
The proof of Proposition 3 then follows:
Proof. Let  $\mu = p_{\mathrm{data}}$  and  $\nu = p_{\mathrm{ebm}}$ , denote the convolved distributions as  $p_{t,\mathrm{data}}$  and  $p_{t,\mathrm{ebm}}$ . Notice that for any  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$
$$
\log \frac {p _ {t , \mathrm {e b m}} (\mathbf {y})}{p _ {\mathrm {e b m}} (\mathbf {x})} = U (\mathbf {x}) - U _ {t} (\mathbf {y})
$$
since both models have the same normalising constant. We then have for arbitrary  $\mathbf{x}$
$$
\begin{array}{l} \operatorname {K L} \left(p _ {t, \text {d a t a}} \| p _ {t, \text {e b m}}\right) = \int \left(\log p _ {t, \text {d a t a}} (\mathbf {y}) - \log p _ {t, \text {e b m}} (\mathbf {y})\right) p _ {t, \text {d a t a}} (\mathbf {y}) \mathrm {d} \mathbf {y} \\ = \int \left(\log p _ {t, \mathrm {d a t a}} (\mathbf {y}) - \log \frac {p _ {t , \mathrm {e b m}} (\mathbf {y})}{p _ {\mathrm {e b m}} (\mathbf {x})} - \log p _ {\mathrm {e b m}} (\mathbf {x})\right) p _ {t, \mathrm {d a t a}} (\mathbf {y}) \mathrm {d} \mathbf {y} \\ = \int \left(\log p _ {t, \text {d a t a}} (\mathbf {y}) + U _ {t} (\mathbf {y}) - U (\mathbf {x})\right) p _ {t, \text {d a t a}} (\mathbf {y}) d \mathbf {y} - \log p _ {\mathrm {e b m}} (\mathbf {x}) \\ = c (t) + \int U _ {t} (\mathbf {y}) p _ {t, \mathrm {d a t a}} (\mathbf {y}) \mathrm {d} \mathbf {y} - U (\mathbf {x}) - \log p _ {\mathrm {e b m}} (\mathbf {x}) \\ \end{array}
$$
with  $U$  independent entropy term  $c(t) \coloneqq \mathbb{E}_{p_{t,\mathrm{data}}(\mathbf{y})}[\log p_{t,\mathrm{data}}(\mathbf{y})]$ . Since  $\mathbf{x}$  was chosen arbitrarily, we can integrate with respect to  $p_{\mathrm{data}}(\mathbf{x})$  and find