SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} v _ {T} = - \frac {1}{2} \int_ {0} ^ {T} (1 + (h _ {t} ^ {\leftarrow}) ^ {2}) \Phi_ {t, T} ^ {(h ^ {\leftarrow})} (\nabla \cdot (p _ {t} ^ {\leftarrow} \mathcal {E} _ {t} ^ {\leftarrow})) \mathrm {d} t \\ \approx - \frac {1}{2} \sum_ {j} (1 + (h _ {t _ {j}} ^ {\leftarrow}) ^ {2}) \Phi_ {t _ {j}, T} ^ {(h ^ {\leftarrow})} \Big (\pmb {\nabla} \cdot \big (p _ {t _ {j}} ^ {\leftarrow} \phi_ {j} \big) \Big) \Delta t, \\ \end{array}
$$
and recall that the leading order term (10) is simply
$$
L (h ^ {\leftarrow}) = \frac {1}{2} \int_ {\mathbb {R} ^ {d}} \frac {v _ {T} ^ {2} (x)}{p _ {0} (x)} \mathrm {d} x.
$$
Therefore, after plugging the decomposition of  $\mathcal{E}_t^{\leftarrow}$  into  $L(h^{\leftarrow})$ , we have
$$
L (h ^ {\leftarrow}) = \frac {(1 + h ^ {2}) ^ {2}}{8} (\Delta t) ^ {2} \sum_ {j, k} \int_ {\mathbb {R} ^ {d}} \frac {\Phi_ {t _ {j} , T} ^ {(h ^ {\leftarrow})} \left(\nabla \cdot \left(p _ {t _ {j}} ^ {\leftarrow} \phi_ {j}\right)\right) \Phi_ {t _ {k} , T} ^ {(h ^ {\leftarrow})} \left(\nabla \cdot \left(p _ {t _ {k}} ^ {\leftarrow} \phi_ {k}\right)\right)}{p _ {0}}.
$$
We had chosen  $h_t^{\leftarrow} = \mathsf{h}$  for all  $t \in [0, T]$  for simplicity. Either by Cauchy-Schwartz inequality, or by assumptions on the independence of each  $\phi_j$  as discussed above, the main feature is the following term
$$
\frac {(1 + \mathsf {h} ^ {2}) ^ {2}}{8} \Delta t \int_ {\mathbb {R} ^ {d}} \frac {1}{p _ {0}} \left[ \Phi_ {t _ {j}, T} ^ {(h \leftarrow)} \left(\nabla \cdot \left(p _ {t _ {j}} ^ {\leftarrow} \phi_ {j}\right)\right) \right] ^ {2}.
$$
Therefore, it makes sense to study how this quantity scales for each  $j$ , and for two asymptotic regions  $\mathsf{h} = 0$  and  $\mathsf{h} \to \infty$ . For each fixed  $j$ , we can easily observe that the above quantity arises from choosing the error function as a pulse-shape error, i.e.,  $\phi_{k} = 0$  if  $k \neq j$  (with certain normalization rescaling).
F Proof of Prop. 3.6 and the discussion of the large diffusion limit
F.1 Proof of Prop. 3.6
Let us pick an  $a = \mathfrak{h}^{-\beta}\ll 1$  . We split  $v_{T}$  (10) into two parts:
$$
\begin{array}{l} v _ {T} = - \frac {1 + \mathsf {h} ^ {2}}{2} \int_ {0} ^ {T} \Phi_ {t, T} ^ {(h ^ {\leftarrow})} \big (\underbrace {\nabla \cdot (p _ {t} ^ {\leftarrow} \mathcal {E} _ {t} ^ {\leftarrow})} _ {=: \Gamma_ {t} ^ {\leftarrow}} \big)   \mathrm {d} t \\ = - \frac {1 + \mathsf {h} ^ {2}}{2} \int_ {T - a} ^ {T} \Phi_ {t, T} ^ {(h \leftarrow)} \left(\Gamma_ {t} ^ {\leftarrow}\right) \mathrm {d} t - \frac {1 + \mathsf {h} ^ {2}}{2} \int_ {0} ^ {T - a} \Phi_ {t, T} ^ {(h \leftarrow)} \left(\Gamma_ {t} ^ {\leftarrow}\right) \mathrm {d} t. \\ \end{array}
$$
Upper bound
By Cauchy-Schwartz inequality  $((x + y)^2 = x^2 + y^2 + 2xy \leq (1 + \alpha^2)x^2 + (1 + \alpha^{-2})y^2$  for any  $x, y$  and  $\alpha > 0$ ),
$$
\begin{array}{l} L (h) = \frac {1}{2} \int \frac {v _ {T} ^ {2}}{p _ {0}} \\ \leq \left(1 + \alpha^2\right) \underbrace {\frac {\left(1 + h ^ {2}\right) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {\left(\int_ {T - a} ^ {T} \Phi_ {t , T} ^ {(h ^ {\leftarrow})} (\Gamma_ {t} ^ {\leftarrow}) \mathrm {d} t\right) ^ {2}}{p _ {0}}} _ {=: \mathcal {T} _ {1}} \\ + (1 + \alpha^ {- 2}) \underbrace {\frac {\left(1 + h ^ {2}\right) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {\left(\int_ {0} ^ {T - a} \Phi_ {t , T} ^ {(h \leftarrow)} (\Gamma_ {t} ^ {\leftarrow}) \mathrm {d} t\right) ^ {2}}{p _ {0}}} _ {=: \mathcal {T} _ {2}}. \\ \end{array}
$$
For the second term  $\mathcal{T}_2$
$$
\mathcal {T} _ {2} = \frac {(1 + h ^ {2}) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {\left(\int_ {0} ^ {T - a} \Phi_ {t , T} ^ {(h ^ {\leftarrow})} (\Gamma_ {t} ^ {\leftarrow}) \mathrm {d} t\right) ^ {2}}{p _ {0}}
$$
$$
\begin{array}{l} \stackrel {\text {C a u c h y i n e q .}} {\leq} \frac {(1 + \mathsf {h} ^ {2}) ^ {2} (T - a)}{8} \int_ {0} ^ {T - a} \left(\int_ {\mathbb {R} ^ {d}} \frac {\Phi_ {t , T} ^ {(h \leftarrow)} (\Gamma_ {t} ^ {\leftarrow}) ^ {2}}{p _ {0}}\right) d t \\ \stackrel {(2 9)} {\leq} \frac {(1 + h ^ {2}) ^ {2} T}{2} \int_ {0} ^ {T - a} \left(\int_ {\mathbb {R} ^ {d}} \frac {\Phi_ {t , T} ^ {(h ^ {\leftarrow})} (\Gamma_ {t} ^ {\leftarrow}) ^ {2}}{\rho_ {0}}\right) d t \\ \stackrel {(3 3)} {\leq} \frac {(1 + \mathsf {h} ^ {2}) ^ {2} T}{2} \int_ {0} ^ {T - a} \left(\int_ {\mathbb {R} ^ {d}} \frac {\Gamma_ {t} ^ {\leftarrow 2}}{\rho_ {t} ^ {\leftarrow}}\right) \exp \left(- \int_ {t} ^ {T} \left((\mathsf {h} ^ {2} - C _ {r} ^ {\leftarrow , (2)}) \kappa_ {r} ^ {\leftarrow} - C _ {r} ^ {\leftarrow , (1)}\right) \mathrm {d} r\right) \mathrm {d} t \\ \leq \frac {(1 + h ^ {2}) ^ {2} T}{2} \sup  _ {t \in [ 0, T ]} \left(\int_ {\mathbb {R} ^ {d}} \frac {\Gamma_ {t} ^ {\leftarrow - 2}}{\rho_ {t} ^ {\leftarrow}}\right) \exp \left(\int_ {0} ^ {T} C _ {r} ^ {\leftarrow , (2)} \kappa_ {r} ^ {\leftarrow} + C _ {r} ^ {\leftarrow , (1)} d r\right) \int_ {0} ^ {T - a} e ^ {- h ^ {2} \gamma (T - t)} d t \\ \leq \frac {(1 + h ^ {2}) T}{2 h ^ {2}} \sup  _ {t \in [ 0, T ]} \left(\int_ {\mathbb {R} ^ {d}} \frac {\Gamma_ {t} ^ {\leftarrow 2}}{\rho_ {t} ^ {\leftarrow}}\right) \exp \left(\int_ {0} ^ {T} C _ {r} ^ {\leftarrow , (2)} \kappa_ {r} ^ {\leftarrow} + C _ {r} ^ {\leftarrow , (1)} d r\right) \frac {(1 + h ^ {2}) e ^ {- a h ^ {2} \gamma}}{\gamma}, \\ \end{array}
$$
which decays exponentially fast as  $h \to \infty$  as long as  $h^2 \gg a^{-1} \gg 1$ . To get the second line above, we used Cauchy-Schwarz inequality and Fubini's theorem. Recall the definition of  $\gamma \in \mathbb{R}^{+}$  in the statement of Prop. 3.6.
Overall, when  $a = \mathsf{h}^{-\beta}\ll 1$
$$
L (h) \leq \left(1 + \alpha^ {- 2}\right) C \frac {\left(1 + h ^ {2}\right) \exp \left(- h ^ {2 - \beta} \gamma\right)}{\gamma} + \left(1 + \alpha^ {2}\right) \mathcal {T} _ {1},
$$
$$
\begin{array}{l} C = \frac {(1 + h ^ {2})}{2 h ^ {2}} T \sup  _ {t \in [ 0, T ]} \left(\int_ {\mathbb {R} ^ {d}} \frac {\Gamma_ {t} ^ {- 2}}{\rho_ {t} ^ {-}}\right) \exp \left(\int_ {0} ^ {T} C _ {r} ^ {\leftarrow , (2)} \kappa_ {r} ^ {\leftarrow} + C _ {r} ^ {\leftarrow , (1)} d r\right) \\ \lesssim \frac {T}{2} \sup  _ {t \in [ 0, T ]} \left(\int_ {\mathbb {R} ^ {d}} \frac {\Gamma_ {t} ^ {\leftarrow 2}}{p _ {t} ^ {\leftarrow}}\right) \exp \left(\int_ {0} ^ {T} C _ {r} ^ {\leftarrow , (2)} m _ {r} ^ {\leftarrow} + (\lim  _ {\mathfrak {h} \to \infty} C _ {r} ^ {\leftarrow , (1)}) \mathrm {d} r\right). \\ \end{array}
$$
Please refer to Lem. E.3 for  $\lim_{h\to \infty}C_r^{\leftarrow ,(1)}$
The lower bound can be proved in the same way: by Cauchy-Schwarz inequality again,
$$
\begin{array}{l} L (\mathfrak {h}) \geq (1 - \alpha^ {2}) \mathcal {T} _ {1} + (1 - \alpha^ {- 2}) \mathcal {T} _ {2} \\ \geq (1 - \alpha^ {2}) \mathcal {T} _ {1} - (\alpha^ {- 2} - 1) C \frac {(1 + h ^ {2}) \exp (- h ^ {2 - \beta} \gamma)}{\gamma}. \\ \end{array}
$$
The remaining task is to estimate  $\mathcal{T}_1$ .
Asymptotic limit of  $\mathcal{T}_1$
We only provide an asymptotic result below. As  $a \ll 1$
$$
\begin{array}{l} \mathcal {T} _ {1} := \frac {(1 + h ^ {2}) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {\left(\int_ {T - a} ^ {T} \Phi_ {t , T} ^ {(h ^ {\leftarrow})} (\Gamma_ {t} ^ {\leftarrow}) \mathrm {d} t\right) ^ {2}}{p _ {0}} \\ \sim \frac {(1 + h ^ {2}) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {1}{p _ {0}} \left(\left(\int_ {T - a} ^ {T} \Phi_ {t, T} ^ {(h ^ {\leftarrow})} \mathrm {d} t\right) (\Gamma_ {T} ^ {\leftarrow})\right) ^ {2} \\ \sim \frac {(1 + h ^ {2}) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {1}{p _ {0}} \left(\int_ {T - a} ^ {T} \exp \left(\frac {h ^ {2}}{2} (T - t) \mathcal {K} _ {T} ^ {\leftarrow}\right) \Gamma_ {T} ^ {\leftarrow} d t\right) ^ {2} \\ \sim \frac {(1 + h ^ {2}) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {1}{\rho_ {0}} \left(\int_ {T - a} ^ {T} \exp \left(\frac {h ^ {2}}{2} (T - t) \mathcal {K} _ {T} ^ {\leftarrow}\right) \Gamma_ {T} ^ {\leftarrow} d t\right) ^ {2}, \\ \end{array}
$$
where we used  $\Gamma_{t}^{\leftarrow}\approx \Gamma_{T}^{\leftarrow}$ , when  $t\approx T$  in the second line; we used  $\frac{\mathcal{L}_t^{(h\leftarrow)}}{\mathrm{h}^2 / 2}\approx \frac{\mathcal{L}_T^{(h\leftarrow)}}{\mathrm{h}^2 / 2}$  when  $t\approx T$  in the third line;  $\mathcal{L}_T^{(h\leftarrow)} = \mathfrak{h}^2 /2\mathcal{K}_T^{\leftarrow}$  (40) herein. In the last line, we used the fact that  $\rho_0\sim p_0$  when  $\mathsf{h}\to \infty$
By Lem. D.2,
$$
\begin{array}{l} \mathcal {T} _ {1} \sim \frac {(1 + \mathrm {h} ^ {2}) ^ {2}}{8} \sum_ {k = 1} ^ {\infty} \left(\frac {1 - e ^ {- a \frac {\mathrm {h} ^ {2}}{2} \lambda_ {k} ^ {(\mathrm {h})}}}{\frac {\mathrm {h} ^ {2}}{2} \lambda_ {k} ^ {(\mathrm {h})}}\right) ^ {2} \left(\alpha_ {k} ^ {(\mathrm {h})}\right) ^ {2} \\ \sim \frac {1}{2} \sum_ {k = 1} ^ {\infty} \left(\frac {\alpha_ {k} ^ {(h)}}{\lambda_ {k} ^ {(h)}}\right) ^ {2} \quad (\text {b y} a = h ^ {- \beta}, h \gg 1), \\ \end{array}
$$
where  $(\lambda_k^{(\mathsf{h})},\phi_k^{(\mathsf{h})})$  are eigen pairs of  $\kappa_T^{\leftarrow}$  and  $\Gamma_T^{\leftarrow} = \sum_{k = 1}^{\infty}\alpha_k^{(\mathsf{h})}\phi_k^{(\mathsf{h})}$ . When  $\mathsf{h}\to \infty$ , we know that the eigenvalues of  $\kappa_T^{\leftarrow}$ , which depends on  $\mathsf{h}$  and has the form
$$
\mathcal {K} _ {T} ^ {\leftarrow} (\mu) (x) = \triangle \mu (x) + \nabla \cdot (\nabla U _ {0} (x) \mu (x)) + \mathsf {h} ^ {- 2} \nabla \cdot \Big (\big (\nabla U _ {0} (x) - x \big) \mu \Big),
$$
should converge to  $\kappa^{\infty}$ , defined as
$$
\mathcal {K} ^ {\infty} (\mu) := \triangle \mu + \nabla \cdot (\nabla U _ {0} \mu).
$$
Therefore, in the limit,
$$
\lim  _ {\mathsf {h} \to \infty} \mathcal {T} _ {1} = \frac {1}{2} \sum_ {k = 1} ^ {\infty} \left(\alpha_ {k} ^ {(\infty)} / \lambda_ {k} ^ {(\infty)}\right) ^ {2},
$$
where  $(\lambda_k^{(\infty)},\phi_k^{(\infty)})$  are eigen pairs of  $\kappa^{\infty}$  and  $\Gamma_T^{\leftarrow} = \sum_{k = 1}^{\infty}\alpha_k^{(\infty)}\phi_k^{(\infty)}$
Upper bound of  $\mathcal{T}_1$
By the upper bound in Lem. D.2 (together with  $a = \mathsf{h}^{-\beta}$  and  $\mathsf{h} \gg 1$ ) or by applying  $\lambda_k^{(\mathsf{h})} \geq \kappa_0$  directly to the asymptotic of  $\mathcal{T}_1$ , when  $\mathsf{h} \gg 1$ ,
$$
\mathcal {T} _ {1} \lesssim \frac {1}{2 \kappa_ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \frac {\Gamma_ {T} ^ {\leftarrow 2}}{\rho_ {0}} = \frac {1}{2 \kappa_ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \frac {(\boldsymbol {\nabla} \cdot (p _ {T} ^ {\leftarrow} \mathcal {E} _ {T} ^ {\leftarrow})) ^ {2}}{\rho_ {0}} \sim \frac {1}{2 m _ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \frac {(\boldsymbol {\nabla} \cdot (p _ {0} \mathcal {E} _ {T} ^ {\leftarrow})) ^ {2}}{p _ {0}}.
$$
The term  $\mathcal{T}$  in Prop. 3.6 is simply the limit of  $\mathcal{T}_1$ .
F.2 Remark on the large diffusion limit
Score function parameterization in constrained score models
We would like to remark on the parameterization of score function. It is common in literature to directly parameterize  $\mathfrak{S}_t$  (4) via some neural network. However, in general, this practice can
rarely guarantee that  $\mathfrak{S}_t$  (which is supposed to approximate  $\nabla \log p_t$ ) has the gradient form as well [23, 25]. If we instead parameterize  $\log p_t(x) \approx \mathrm{N}_t(x; \theta)$ , where  $\mathrm{N}(\cdot; \theta)$  is some (scalar-valued) neural network with parameter  $\theta$ , we can simply use the following ansatz in training
$$
\mathfrak {S} _ {t} (x) = \nabla \mathrm {N} _ {t} (x; \theta).
$$
Consequently, the error of score function estimate also has a gradient form:
$$
\begin{array}{l} \epsilon \mathcal {E} _ {t} ^ {\leftarrow} (x) \stackrel {(6)} {=} \mathfrak {S} _ {t} ^ {\leftarrow} (x) - \nabla \log p _ {t} ^ {\leftarrow} (x) \\ = \nabla \mathrm {N} _ {t} ^ {\leftarrow} (x; \theta) - \nabla \log p _ {t} ^ {\leftarrow} (x) \\ = \nabla \underbrace {\left(\mathrm {N} _ {t} ^ {\leftarrow} (x ; \theta) - \log p _ {t} ^ {\leftarrow} (x)\right)} _ {=: \varphi}. \\ \end{array}
$$
This validates the discussion in § 3.6. Apart from the apparent benefit in preserving the gradient form, this ansatz can help us identify some interesting structure for the upper bound of  $\lim_{\mathfrak{h}\to \infty}L(\mathfrak{h})$  in (18), discussed in § 3.6.
More discussion on the upper bound
Because the score error  $\mathcal{E}_T^{\leftarrow}$  comes from approximating  $\nabla \log p_0$ , it is reasonable to consider the case  $\varphi = \log p_0$ , the upper bound (18), or equivalently (19), becomes
$$
\mathcal {T} \lesssim \frac {1}{2 m _ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \left(\left| \nabla U _ {0} \right| ^ {2} - \triangle U _ {0}\right) ^ {2} e ^ {- U _ {0}}. \tag {41}
$$
This quantity heuristically characterizes how difficult the probability distribution  $p_0$  can be learn by score-based diffusion models with large diffusion coefficient.
It is of interest to further explore how to utilize the above upper bound to improve the training of score function, or provide theoretical understanding about what types of probability distributions are easy/difficult to learn via score-based SDEs. We shall leave an extensive study of these important and interesting questions for future work. Below, we shall provide a simple example.
Example:  $d$ -dimensional Gaussian
When  $p_0 = e^{-U_0} = \mathcal{N}(\mu, \sigma_0^2)$ , the quantity  $m_0$  (14) is  $m_0 = \frac{1}{\sigma_0^2}$ . Then this upper bound (18) is
$$
\frac {\sigma_ {0} ^ {4}}{2} \int_ {\mathbb {R} ^ {d}} \left(\left| \frac {x - \mu}{\sigma_ {0} ^ {2}} \right| ^ {2} - \frac {d}{\sigma_ {0} ^ {2}}\right) ^ {2} e ^ {- U _ {0}} = d.
$$
This is compatible with the insight that it is more difficult to learn a high dimensional probability distribution.
G Numerical experiment: 1D Gaussian case
We demonstrate and validate main findings via a simple 1D Gaussian. Suppose  $f_{t}(x) = -\frac{1}{2} x$ ,  $g_{t} = 1$ , and the exact data distribution is a Gaussian  $p_0 = \mathcal{N}(0,\sigma_0^2)$ . The error is quantified by  $\mathrm{KL}\big(p_0||\widetilde{q}_T\big)$ . We investigate this error as a function of the magnitude of  $h^{\leftarrow}$  (which is chosen as a constant function  $h_t^{\leftarrow} = \mathsf{h}$  for any  $t \in [0,T]$ ).
G.1 Explicit formulas for 1D Gaussian case
From solving the SDE for the forward process, we know that
$$
p _ {t} = \mathcal {N} \left(0, \sigma_ {t} ^ {2}\right), \quad \sigma_ {t} ^ {2} = \sigma_ {0} ^ {2} e ^ {- t} + 1 - e ^ {- t}, \quad \nabla \log p _ {t} (x) = - \frac {x}{\sigma_ {t} ^ {2}}. \tag {42}
$$
The backward dynamics (7) on  $t\in [0,T]$  is
$$
\left\{ \begin{array}{l} \mathrm {d} \widetilde {Y} _ {t} = \Big (\frac {1}{2} \widetilde {Y} _ {t} + \frac {1 + h _ {t} ^ {\leftarrow 2}}{2} \big (- \frac {\widetilde {Y} _ {t}}{\sigma_ {T - t} ^ {2}} + \epsilon \mathcal {E} _ {t} ^ {\leftarrow} (\widetilde {Y} _ {t}) \big) \Big) \mathrm {d} t + h _ {t} ^ {\leftarrow} \mathrm {d} W _ {t}, \\ \mathrm {l a w} (\widetilde {Y} _ {0}) = p _ {T}. \end{array} \right.
$$
In this example, there is only one source of error, which is  $\mathcal{E}^{\leftarrow}$ , as we know explicitly the distribution  $p_T$  and we can choose the time step small enough such that numerical discretization error is negligible. Due to the structure of the score function, it is reasonable to consider the following ansatz
$$
\mathcal {E} _ {t} ^ {\leftarrow} (x) = \alpha_ {t} x. \tag {43}
$$
This example has explicit formulas:  $\widetilde{Y}_T$  is a Gaussian distribution with  $\mathbb{E}[\widetilde{Y}_T] = 0$  and
$$
\operatorname {V a r} \left(\widetilde {Y} _ {T}\right) = G _ {T} ^ {- 2} \operatorname {V a r} \left(\widetilde {Y} _ {0}\right) + \int_ {0} ^ {T} G _ {T} ^ {- 2} G _ {t} ^ {2} h _ {t} ^ {\leftarrow 2} \mathrm {d} t,
$$
$$
G _ {t} := \exp \left(- \int_ {0} ^ {t} \frac {1}{2} + \frac {1 + h _ {s} ^ {\leftarrow 2}}{2} \left(- \frac {1}{\sigma_ {T - s} ^ {2}} + \epsilon \alpha_ {s}\right) \mathrm {d} s\right).
$$
The sample generation error quantified by KL divergence also has an explicit formula:
$$
\operatorname {K L} \left(p _ {0} | | \widetilde {q} _ {T}\right) = \frac {1}{2} \log \left(\frac {\operatorname {V a r} \left(\widetilde {Y} _ {T}\right)}{\sigma_ {0} ^ {2}}\right) + \frac {\sigma_ {0} ^ {2}}{2 \operatorname {V a r} \left(\widetilde {Y} _ {T}\right)} - \frac {1}{2}.
$$
G.2 Experiment 1: Fix error magnitude  $\epsilon$  and consider various error types
We choose  $T = 2$ ,  $h_t^{\leftarrow} = \mathfrak{h}$  for all  $t \in [0, T]$ , and the error function is chosen as
$$
\epsilon = 0. 0 2, \quad \mathcal {E} _ {t} ^ {\leftarrow} (x) = \nabla \log p _ {t} ^ {\leftarrow} (x) \times \left\{ \begin{array}{c l} 1 & \text {c a s e 1 ;} \\ - 1 & \text {c a s e 2 ;} \\ \frac {1 + \sin (2 \pi t / T)}{2} & \text {c a s e 3 ;} \\ \mathbb {I} _ {t <   0. 9 5 T} & \text {c a s e 4 ;} \\ \mathbb {I} _ {t > 0. 9 9 T} & \text {c a s e 5 .} \end{array} \right. \tag {44}
$$
For these choices, we only perturb the true score function by a bounded prefactor. We can observe that the error  $\mathrm{KL}(p_0||\widetilde{q}_T)$  is a complicated function of  $h$  in general in Fig. 6. When we only perturb the score function during the initial period of the generative process (case 4), we can clearly observe that the sampling error decays exponentially fast with respect to  $h^2$ , which numerically validates Prop. 3.4. When we only perturb the score function near the end of the generative process (case 5), increasing the diffusion coefficient  $h$  will actually increase the error, which is predicted by Prop. 3.5. For a general error (case 1, 2, 3), overall we can still expect that increasing diffusion coefficient  $h$  will generally suppress the error when  $\sigma_0$  is small.
Table 3: Approximated value of  $L(\mathsf{h})$  when  $\mathrm{h}^2\approx 20$ . We can observe that in each column (referring to a specific error type) the value is almost independent of the  $\sigma_0$  in particular when  $\sigma_0 < 1$ .
σ0	Case 1	Case 2	Case 3
0.2	0.2567	0.3032	0.0658
0.4	0.2569	0.3028	0.0636
0.6	0.2570	0.3018	0.0597
0.8	0.2569	0.3023	0.0544
1.5	0.2570	0.3017	0.0321
2.0	0.2564	0.3022	0.0198
3.0	0.2566	0.3020	0.0121
G.3 Experiment 2: consider  $L(h)$  for various error types
We further numerically approximate  $L(\mathsf{h}) \equiv L(\mathsf{h}, \mathcal{E}^{\leftarrow}, p_0)$  by linear regression and study how  $\sigma_0$  (i.e., the data distribution  $p_0$ ), error type  $\mathcal{E}^{\leftarrow}$ , and diffusion coefficient  $\mathsf{h}$  affect the leading order term  $L(\mathsf{h}, \mathcal{E}^{\leftarrow}, p_0)$ . Recall that we use  $L(\mathsf{h})$  as a simplified notation when  $\mathcal{E}^{\leftarrow}$  and  $p_0$  are clear from context; see § 3. As a remark, to approximate  $L(\mathsf{h})$ , we used the leading-order approximation that  $\mathrm{KL}(p_0||\widetilde{q}_T) = L(\mathsf{h})\epsilon^2 + \mathcal{O}(\epsilon^3)$ : we choose a few  $\epsilon$  values and use linear regression to estimate  $L(\mathsf{h})$ .
In Fig. 7, we can clearly observe that  $L(\mathfrak{h})$  converges to a constant extremely fast when  $\mathfrak{h}$  increases for cases  $\sigma_0 < 1$ . The value of  $\mathcal{E}_T^{\leftarrow}$  for the case 3 is only  $1/2$  of that for case 1 and case 2. By Prop. 3.6, we know that  $\lim_{\mathfrak{h} \to \infty} L(\mathfrak{h})$  for the case 3 should be approximated  $1/4$  of that for cases 1 and 2. This is also (approximately) numerically observed in Table 3.
Figure 6: We show  $\log_{10}\left(\mathrm{KL}\big(p_0||q_T\big)\right)$  as a function of  $\mathsf{h}^2$  for the 1D Gaussian model.  $T = 2$  different  $\sigma_0$  and error functions in (44) are considered.
H More details about numerical experiments in  $\S 4$
In this section, we discuss datasets, network architectures, evaluation metrics, numerical schemes (exponential integrator), and the default weight in denoising score matching.
H.1 Datasets
1D 2-mode Gaussian mixture:  $p_0(x) = \sum_{i=1}^{2} 0.5 \mathcal{N}(x; (-1.0)^i, 0.01)$ .
- 2D 4-mode Gaussian mixture:  $p_0(x) = \sum_{i,j=1}^{2} 0.25\mathcal{N}\big(x;((-1.0)^i, (-1.0)^j), 0.05^2 I_2\big)$ .
- Swiss roll: Swiss roll generates samples by  $(x, y) = (t \sin(t), t \cos(t))$  with  $t$  drawn from the uniform distribution  $\mathcal{U}\left(\frac{3\pi}{2}, \frac{9\pi}{2}\right)$ .
Figure 7: We show  $\log_{10}\left(L(\mathsf{h})\right)$  as a function of  $\mathfrak{h}^2$  for the 1D Gaussian model.  $T = 2$ , different  $\sigma_0$  and error functions in (44) are considered.
- MNIST: MNIST [34] contains 60,000  $28 \times 28$  gray-scale images with hand-written digits.
- CIFAR-10: CIFAR-10 [22] contains 60,000  $32 \times 32$  RGB images with ten categories.
H.2 Network architectures and other parameters
For experiments on 1D/2D Gaussian mixtures, the exact scores can be obtained analytically if we use VP-SDE. We set  $T = 4$  and  $g_{t} = 1$  for  $t \in [0,T]$ . For the time discretization when solving the reverse SDE with Euler-Maruyama method, we apply 40,000 steps and 80,000 steps for 1D and 2D Gaussian mixtures, respectively.
For experiments on Swiss roll, we apply a three-layer neural network for score matching, where the width of each layer is set as 50, 50, and 2 and we apply ReLU as the nonlinear activation for two hidden layers. We set  $T = 1$  and  $g_{t} = \sqrt{0.1(1 - t) + 20t}$  for  $t \in [0,T]$ . The learning rate is set as 0.01 and decays by 0.5 every 8,000 steps. The batch size is set as 400. We train the neural network for 20,000 steps. For the time discretization when solving the reverse SDE with Euler-Maruyama method, we apply 20,000 steps.
For experiments on MNIST, we apply the net architecture in [17] for score matching, where we use two resolution blocks in U-net and set the multipliers of channels to be one and two. We set  $T = 1.4$  and  $g_{t} = \sqrt{0.1(1 - t) + 20t}$  for  $t \in [0,T]$ . The number of iteration is 20,000, the batch size is set as 64. We solve the reverse SDE with exponential integrator; see also Appx. H.4.
For experiments on CIFAR-10, we apply the DDPM++ cont. in [30] as the net architecture, and use their pretrained checkpoint in for score estimation. We use the same setting as [30], i.e.,  $T = 1$  and  $g_{t} = \sqrt{0.1(1 - t) + 20t}$  for  $t \in [0,T]$ . For the time discretization when sampling with Euler-Maruyama method, we apply 100, 200, 500, 1000, 2000, 3000 and 4000 steps.
H.3 Evaluation metrics
For experiments of 1D/2D Gaussian mixtures and Swiss roll, we apply approximated divergences for evaluating the performances. Specifically, we discretize the space into 100 bins in each dimension, then obtain the empirical densities of 10,000 true samples and 10,000 generated samples, and use Jensen-Shannon divergence, Kullback-Leibler divergence and Wasserstein distance between both empirical densities as the metrics for evaluation.
H.4 Exponential integrator
We shall explain the exponential integrator for (2) with  $t \in [0, T]$ , i.e., the following equation
$$
\begin{array}{l} \mathrm {d} Y _ {t} = - f _ {t} ^ {\leftarrow} \left(Y _ {t}\right) \mathrm {d} t + \frac {\left(g _ {t} ^ {\leftarrow}\right) ^ {2} + \left(h _ {t} ^ {\leftarrow}\right) ^ {2}}{2} \nabla \log p _ {t} ^ {\leftarrow} \left(Y _ {t}\right) \mathrm {d} t + h _ {t} ^ {\leftarrow} \mathrm {d} W _ {t} \\ = \frac {1}{2} g _ {t} ^ {\leftarrow 2} Y _ {t} \mathrm {d} t + \frac {(g _ {t} ^ {\leftarrow}) ^ {2} + (h _ {t} ^ {\leftarrow}) ^ {2}}{2} \nabla \log p _ {t} ^ {\leftarrow} (Y _ {t}) \mathrm {d} t + h _ {t} ^ {\leftarrow} \mathrm {d} W _ {t}. \\ \end{array}
$$
Then for any time  $t \in [t_k, t_{k+1}]$ , and given  $\hat{Y}_{t_k}$ , we approximate the above dynamics by
$$
\mathrm {d} \hat {Y} _ {t} \approx \frac {1}{2} g _ {t} ^ {\leftarrow 2} \hat {Y} _ {t} \mathrm {d} t + \frac {\left(g _ {t} ^ {\leftarrow}\right) ^ {2} + \left(h _ {t} ^ {\leftarrow}\right) ^ {2}}{2} \nabla \log p _ {t _ {k}} ^ {\leftarrow} (\hat {Y} _ {t _ {k}}) \mathrm {d} t + h _ {t} ^ {\leftarrow} \mathrm {d} W _ {t}.
$$
This dynamics is a linear SDE and we can solve it exactly