SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the previous section, we assume for a flexible neural network parameterized  $f_{\theta}$ , the energy-based model  $\tilde{q}_{\theta}(\tilde{x}) = \exp(-f(\tilde{x})) / Z(\theta)$  trained by Equation 4 can recover the target noisy data distribution  $\tilde{q}_{\theta^{*}} = \tilde{p}_{d}$  so there exists an underlying model  $q$  such that  $\tilde{q}_{\theta^{*}} = q * k$  and  $q = p_{d}$ . This assumption is commonly used in the literature on score-based methods. For example, in the score-based diffusion models literature [35, 14, 2], for any data  $x \in \mathbb{R}^D$ , the score function  $\nabla_{\tilde{x}} \log \tilde{q}_{\theta}(\tilde{x})$  is usually parameterized by a neural network  $\mathrm{NN}_{\theta}(\cdot): \mathbb{R}^D \to \mathbb{R}^D$ . However, this parameterization cannot guarantee  $\mathrm{NN}_{\theta}(\tilde{x})$  is a conservative vector field, or in other words, there doesn't exist a distribution  $\tilde{q}_{\theta}(\tilde{x})$  such that  $\nabla_{\tilde{x}} \tilde{q}_{\theta}(\tilde{x}) = \nabla_{\tilde{x}} \log \tilde{q}_{\theta}(\tilde{x})$  and  $\nabla_{\tilde{x}}^2 \log \tilde{q}(\tilde{x})$  is symmetric [32, 33]. Therefore, perfect score estimation  $\nabla_{\tilde{x}} \log \tilde{p}_d(\tilde{x}) = \nabla_{\tilde{x}} \log \tilde{q}_{\theta}(\tilde{x})$  is implicitly assumed to allow an EBM interpretation.
However, the underlying clean model doesn't always exist for imperfect model  $\tilde{q}_{\theta} \neq \tilde{p}_d$ . We here provide the sufficient and necessary conditions which guarantee the existence of the underlying clean model.
Theorem A.1 (Necessary and Sufficient conditions for the existence of the underlying clean model.). For a model  $\tilde{q}_{\theta}$  with the convolutional noise distribution  $k(\epsilon) = \mathcal{N}(0,\sigma^2 I)$ , there exists an underlying model  $q$  such that  $q*k = \tilde{q}$  if and only if  $\mathcal{F}(\tilde{q}_{\theta}) / \mathcal{F}(k)$  is positive semi-definite. Additionally, the underlying distribution  $q$  can be written as
$$
q = \mathcal {F} ^ {- 1} \left(\mathcal {F} \left(\tilde {q} _ {\theta}\right) / \mathcal {F} (k)\right), \tag {22}
$$
where  $\mathcal{F}^{-1}$  is the inverse Fourier transform. This theorem is a straightforward corollary of Bochner's Theorem 8. However, for the energy model  $\tilde{q}_{\theta}(\tilde{x}) \propto \exp(-f_{\theta}(\tilde{x}))$ , it's difficult to design a functioning family of  $f$  that satisfies the positive semi-definite condition and have the tractable score function at the same time 9. We thus leave the design of better energy function parameterizations as a promising future direction.
7A continuous function  $f:\mathbb{R}^d\to \mathbb{C}$  is positive semi-definite if for all  $n\in \mathbb{N}$ , all sets of pairwise distinct centers  $X = \{x_{1},\ldots ,x_{N}\} \in \mathbb{R}^{d}$  and all  $\alpha \in \mathbb{C}^N$ $\sum_{i = 1}^{N}\sum_{j = 1}^{N}\alpha_{i}\overline{\alpha_{j}} f(x_{i} - x_{j})\geq 0$ , see [44, Definition 6.1]
Bochner's Theorem [44, Theorem 6.6]: A continuous function  $f: \mathbb{R}^d \to \mathbb{C}$  is positive semi-definite if and only if it is the Fourier transform of a finite non-negative Borel measure on  $\mathbb{R}^d$ .
For example, one can define a noisy energy-based model  $\tilde{q}_{\theta} = \exp (-f_{\theta}(\tilde{x})) / Z(\theta)$  with  $-f_{\theta}(\tilde{x}) = \log \int (-g_{\theta}(x) - 1 / \sigma^2 ||\tilde{x} -x||_2^2)\mathrm{d}x$  , which always allows an underlying clean energy-based model  $q_{\theta}(x) =$ $\exp (-g_{\theta}(x)) / Z(\theta)$  such that  $\tilde{q}_{\theta}(\tilde{x}) = q_{\theta}(x)*k$  with  $k(\epsilon) = \mathcal{N}(0,\sigma^2 I)$  . However, the score function  $\nabla_{\tilde{x}}\log \tilde{q} (\tilde{x}) = -\nabla_{\tilde{x}}f_{\theta}(\tilde{x})$  is intractable in this case.
A.3 Proof of Theorem 2.2
Derivation of the Mean Identity
We let  $\tilde{q}_{\theta}(\tilde{x}) = \int k(\tilde{x} |x)q_{\theta}(x)\mathrm{d}\tilde{x}$ , where  $k(\tilde{x} |x) = \mathcal{N}(x,\sigma^2 I)$ , we have
$$
\begin{array}{l} \nabla_ {\tilde {x}} \log \tilde {q} _ {\theta} (\tilde {x}) = \frac {\nabla_ {\tilde {x}} \tilde {q} _ {\theta} (\tilde {x})}{\tilde {q} _ {\theta} (\tilde {x})} = \frac {\int \nabla_ {\tilde {x}} k (\tilde {x} | x) q _ {\theta} (x) \mathrm {d} x}{\tilde {q} _ {\theta} (\tilde {x})} \\ = - \frac {1}{\sigma^ {2}} \int \left((\tilde {x} - x) \frac {k (\tilde {x} | x) q _ {\theta} (x)}{\tilde {q} _ {\theta} (\tilde {x})}\right) d x \\ \Longrightarrow \sigma^ {2} \nabla_ {\tilde {x}} \log \tilde {q} _ {\theta} (\tilde {x}) + \tilde {x} = \int x \frac {k (\tilde {x} | x) q _ {\theta} (x)}{\tilde {q} _ {\theta} (\tilde {x})} d x = \langle x \rangle_ {q _ {\theta} (x | \tilde {x})} \\ \end{array}
$$
where we define the model denoising posterior using Bayes rule  $q_{\theta}(x|\tilde{x}) \equiv k(\tilde{x} |x)q_{\theta}(x) / \tilde{q}_{\theta}(\tilde{x})$ . The second equality is due to the following Gaussian distribution property
$$
\nabla_ {\tilde {x}} k (\tilde {x} | x) = \frac {1}{\sqrt {2 \pi \sigma^ {2}}} \nabla_ {\tilde {x}} e ^ {\frac {- (\tilde {x} - x) ^ {2}}{2 \sigma^ {2}}} = - \frac {\tilde {x} - x}{\sigma^ {2}} \frac {1}{\sqrt {2 \pi \sigma^ {2}}} e ^ {\frac {- (\tilde {x} - x) ^ {2}}{2 \sigma^ {2}}} = - \frac {\tilde {x} - x}{\sigma^ {2}} k (\tilde {x} | x). \tag {23}
$$
Derivations of the Analytical Full Covariance Identity
We have derived the mean identity
$$
\mu_ {q} (\tilde {x}) \equiv \left\langle x \right\rangle_ {q _ {\theta} (x | \tilde {x})} = \sigma^ {2} \nabla_ {\tilde {x}} \log \tilde {q} _ {\theta} (\tilde {x}) + \tilde {x}. \tag {24}
$$
Taking the gradient over  $x$  in both side and scaling with  $\sigma^2$ , we have
$$
\sigma^ {2} \nabla_ {\tilde {x}} \mu_ {q} (\tilde {x}) = \sigma^ {4} \nabla_ {\tilde {x}} ^ {2} \log \tilde {q} _ {\theta} (\tilde {x}) + \sigma^ {2} I. \tag {25}
$$
We can also expand the hessian of the log  $\tilde{q}_{\theta}(\tilde{x})$  ..
$$
\begin{array}{l} \nabla_ {\tilde {x}} ^ {2} \log \tilde {q} _ {\theta} (\tilde {x}) = - \frac {1}{\sigma^ {2}} \int \nabla_ {\tilde {x}} \left((\tilde {x} - x) \frac {k (\tilde {x} | x) q _ {\theta} (x)}{\tilde {q} _ {\theta} (\tilde {x})}\right) d x \\ = - \frac {1}{\sigma^ {2}} \int \frac {k (\tilde {x} | x) q _ {\theta} (x)}{\tilde {q} _ {\theta} (\tilde {x})} d x + \frac {1}{\sigma^ {2}} \int (\tilde {x} - x) \frac {\nabla_ {\tilde {x}} k (\tilde {x} | x) \tilde {q} _ {\theta} (\tilde {x}) q _ {\theta} (x) - \nabla_ {\tilde {x}} \tilde {q} _ {\theta} (\tilde {x}) k (\tilde {x} | x) q _ {\theta} (x)}{\tilde {q} _ {\theta} ^ {2} (\tilde {x})} d x \\ \Longrightarrow \sigma^ {2} \nabla_ {\tilde {x}} ^ {2} \log \tilde {q} _ {\theta} (\tilde {x}) + 1 = \int (\tilde {x} - x) \frac {\nabla_ {\tilde {x}} k (\tilde {x} | x) q _ {\theta} (x) - \nabla_ {\tilde {x}} \log \tilde {q} _ {\theta} (\tilde {x}) k (\tilde {x} | x) q _ {\theta} (x)}{\tilde {q} _ {\theta} (\tilde {x})} d x \\ = \int (\tilde {x} - x) \frac {- \frac {1}{\sigma^ {2}} (\tilde {x} - x) k (\tilde {x} | x) q _ {\theta} (x) + \frac {1}{\sigma^ {2}} (\tilde {x} - \langle x \rangle_ {q _ {\theta} (x | \tilde {x})}) k (\tilde {x} | x) q _ {\theta} (x)}{\tilde {q} _ {\theta} (\tilde {x})} d x \\ \Rightarrow \sigma^ {4} \nabla_ {\tilde {x}} ^ {2} \log \tilde {q} _ {\theta} (\tilde {x}) + \sigma^ {2} I = \int \left(- (\tilde {x} - x) ^ {2} + (\tilde {x} - x) (\tilde {x} - \langle x \rangle_ {q _ {\theta} (x | \tilde {x})})\right) q _ {\theta} (x | \tilde {x}) d x \\ = \left\langle x ^ {2} \right\rangle_ {q _ {\theta} (x | \bar {x})} - \left\langle x \right\rangle_ {q _ {\theta} (x | \bar {x})} ^ {2} \equiv \Sigma_ {q} (\tilde {x}) \\ \end{array}
$$
Therefore, we obtain the analytical full covariance identity.
$$
\Sigma_ {q} (\tilde {x}) = \sigma^ {2} \nabla_ {\tilde {x}} \mu_ {q} (\tilde {x}). \tag {26}
$$
A.4 Proof of Theorem 2.3
Lemma A.2 (KL to Gaussian [2]). Let  $p(x)$  be a distribution with mean  $\mu_p$  and covariance  $\Sigma_p$  and  $q(x) = \mathcal{N}(\mu_q, \Sigma_q)$ , denote the differential entropy as  $\mathrm{H}(p) \equiv -\int p(x) \log p(x) \, \mathrm{d}x$ , we have
$$
\operatorname {K L} (p \| q) = \operatorname {K L} \left(\mathcal {N} \left(\mu_ {p}, \Sigma_ {p}\right) \| q\right) + \mathrm {H} \left(\mathcal {N} \left(\mu_ {p}, \Sigma_ {p}\right)\right) - \mathrm {H} (p) \tag {27}
$$
The proof can be found in [2] Lemma 2.
We can then prove Theorem 2.3. Since  $p(\tilde{x} | x)p_d(x) = p(x|\tilde{x})\tilde{p}_d(\tilde{x})$ , where  $\tilde{p}_d(\tilde{x}) = \int p_d(x)p(\tilde{x} |x)\mathrm{d}x$ , we have
$$
\mathrm {K L} (p (\tilde {x} | x) p _ {d} (x) \| q (x | \tilde {x}) \tilde {p} _ {d} (\tilde {x})) = \langle \mathrm {K L} (p (x | \tilde {x}) \| q (x | \tilde {x})) \rangle_ {\tilde {p} (\tilde {x})} \tag {28}
$$
Assume Gaussian distribution  $q(x|\tilde{x}) = \mathcal{N}(\mu_q(\tilde{x}),\Sigma_q(\tilde{x}))$  and denote the mean and covariance of the true posterior are  $\mu_p(\tilde{x})$  and  $\Sigma_p(\tilde{x})$ , then the optimal  $q^*$  is
$$
\begin{array}{l} q ^ {*} = \arg \min  _ {q} \mathrm {K L} (p (\tilde {x} | x) p _ {d} (x) \| q (x | \tilde {x}) \tilde {p} _ {d} (\tilde {x})) (29) \\ = \arg \min  _ {q} \left\langle \mathrm {K L} (p (x | \tilde {x}) | | q (x | \tilde {x})) \right\rangle_ {\tilde {p} (\tilde {x})} (30) \\ = \arg \min  _ {q} \left\langle \mathrm {K L} \left(\mathcal {N} \left(\mu_ {p}, \Sigma_ {p}\right) \mid \mid q (x | \tilde {x})\right) + \mathrm {H} \left(\mathcal {N} \left(\mu_ {p}, \Sigma_ {p}\right)\right) - \mathrm {H} \left(p (x | \tilde {x})\right) \right\rangle_ {\tilde {p} (\tilde {x})} (31) \\ = \arg \min  _ {q} \left\langle \mathrm {K L} \left(\mathcal {N} \left(\mu_ {p}, \Sigma_ {p}\right) \mid \mid q (x | \tilde {x})\right) \right\rangle_ {\tilde {p} (\tilde {x})} + c o n s t.. (32) \\ \end{array}
$$
Therefore, the optimal  $q(x|\tilde{x}) = \mathcal{N}(\mu_q(\tilde{x}),\Sigma_q(\tilde{x}))$  under the joint KL has the mean and covariance  $\mu_q^* (\tilde{x}) = \mu_p(\tilde{x}),\Sigma_q^* (\tilde{x})) = \Sigma_p(\tilde{x}).$
B Connection to Analytical DDPM
Paper [2] considers the constrained variational family  $q_{\theta}(x|\tilde{x}) = \mathcal{N}(\mu_{\theta}(\tilde{x}),\sigma_q^2 I)$  and derive the optimal  $\sigma_q^*$  as
$$
\sigma_ {q} ^ {* 2} = \arg \min  _ {\sigma_ {q}} \mathrm {K L} (p (\tilde {x} | x) p _ {d} (x) \| q _ {\theta} (x | y)) \tilde {p} _ {d} (\tilde {x})) = \frac {1}{d} \left\langle \operatorname {T r} \left(\operatorname {C o v} _ {q (x | \tilde {x})} [ x ]\right) \right\rangle_ {\tilde {p} _ {d} (\tilde {x})}, \tag {33}
$$
$$
\sigma_ {q} ^ {* 2} = \sigma^ {2} - \frac {\sigma^ {4}}{d} \left\langle \| s _ {q _ {\theta}} (\tilde {x}) \| _ {2} ^ {2} \right\rangle_ {\tilde {p} _ {d} (\tilde {x})}. \tag {34}
$$
To make a deep connection, we can also plug our analytical full covariance (Equation 10) into Equation 16
$$
\begin{array}{l} \sigma_ {q} ^ {* 2} = \sigma^ {2} + \frac {\sigma^ {4}}{d} \mathrm {T r} \left\langle \nabla_ {x} ^ {2} \log q _ {\theta} (\tilde {x}) \right\rangle_ {\tilde {p} _ {d} (\tilde {x})} \\ = \sigma^ {2} - \frac {\sigma^ {4}}{d} \operatorname {T r} \left\langle s _ {q _ {\theta}} (\tilde {x}) s _ {q _ {\theta}} (\tilde {x}) ^ {T} \right\rangle_ {\tilde {p} _ {d} (\tilde {x})} = \sigma^ {2} - \frac {\sigma^ {4}}{d} \left\langle \| s _ {q _ {\theta}} (\tilde {x}) \| _ {2} ^ {2} \right\rangle_ {\tilde {p} _ {d} (\tilde {x})}, \tag {35} \\ \end{array}
$$
which recovers Equation 17, where the first equality is due to the well-known Fisher information identity [10].
All the experiments conducted in this paper are run on one single NVIDIA GTX 3090.
C.1 Effect of the Single Noise Choice on MNIST
Figure 11 shows the samples generated by our method with the EBM trained with difference  $\sigma \in \{0.3,0.5,0.8\}$  in the noise distribution  $p(\tilde{x} |x)$ , we can find the image quality also heavily depends on the choice of the noise scale and  $\sigma = 0.5$  achieves the best visual quality, we then use this hyper-parameter in the subsequent comparisons.
C.2 Multi-level Noise Details
For full details on the architecture and noise schedule used in the multi-level noise experiments in Section 5, we refer to Appendix B of [36]. For our multi-level Gibbs sampling procedure, we used 3 Gibbs steps at each noise level and 3 Rademacher samples for each diagonal Hessian computation. Following [36], we used a total of 232 noise levels, distributed according to their proposed geometric schedule, and applied a final denoising step in which the mean of the clean distribution conditioned on the final output of the sampling procedure is returned (the final output of the sampling procedure is a sample from the noised distribution from the noise distribution at the smallest noise level). This denoising step was previously found to improve FID scores [17] significantly.
(a)  $\sigma = 0.3$
(b)  $\sigma = 0.5$
(c)  $\sigma = 0.8$
Figure 11: Sample comparisons with different  $\sigma$  value.
Figure 12: Mode Collapse visualization of 25 Markov chains, we plot the samples every 20 Gibbs steps, we can find less modes are covered if we run the Gibbs sampling for a longer time.