SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L \left(h ^ {\leftarrow}\right) = \frac {1}{2} \int_ {\mathbb {R} ^ {d}} \frac {v _ {T} ^ {2} (x)}{p _ {0} (x)} \mathrm {d} x, \quad v _ {T} = - \frac {1}{2} \int_ {0} ^ {T} \left(1 + h _ {t} ^ {\leftarrow 2}\right) \Phi_ {t, T} ^ {\left(h ^ {\leftarrow}\right)} \left(\boldsymbol {\nabla} \cdot \left(p _ {t} ^ {\leftarrow} \mathcal {E} _ {t} ^ {\leftarrow}\right)\right) \mathrm {d} t. \tag {10}
$$
3.3 The role of  $h^{\leftarrow}$  in the Fokker-Planck operator  $\mathcal{L}_t^{(h^{\leftarrow})}$
Let the potential  $U_{t} \coloneqq -\log p_{t}$ , and by the notation of time-reversal,  $U_{t}^{\leftarrow} \equiv U_{T - t}$ . When  $h^{\leftarrow} > 0$  we can rewrite
$$
\mathcal {L} _ {t} ^ {\left(h ^ {\leftarrow}\right)} (\mu) = h _ {t} ^ {\leftarrow 2} / 2 \Big (\triangle \mu + \nabla \cdot \left(\nabla V _ {t} ^ {\leftarrow} \mu\right) \Big), \quad V _ {t} ^ {\leftarrow} (x) := \left(1 + ^ {1} / h _ {t} ^ {\leftarrow 2}\right) U _ {t} ^ {\leftarrow} (x) - \frac {\left| x \right| ^ {2}}{2 h _ {t} ^ {\leftarrow 2}}. \tag {11}
$$
We now introduce a probability distribution induced by the potential  $V_{t}^{\leftarrow}$ :
$$
\rho_ {t} ^ {\leftarrow} \propto \exp \left(- V _ {t} ^ {\leftarrow}\right). \tag {12}
$$
By convection, we also have  $V_{t} = V_{T - t}^{\leftarrow}$  and  $\rho_{t} = \rho_{T - t}^{\leftarrow}$ . Note that  $V_{t}^{\leftarrow}$  depends on  $h^{\leftarrow}$  and when  $h^{\leftarrow} \rightarrow \infty$ , we have  $V_{t}^{\leftarrow,\infty} = U_{t}^{\leftarrow}$ . The role of  $h^{\leftarrow}$  in  $\mathcal{L}_t^{(h^\leftarrow)}$  now can be viewed as the time re-scaling and the effect of  $\mathcal{L}_t^{(h^\leftarrow)}$  at a local time  $t$  can be viewed as evolving the Fokker-Planck equation associated with the overdamped Langevin dynamics in the time-dependent potential  $V_{t}^{\leftarrow}$  for  $\mathcal{O}\bigl (h_t^{\leftarrow 2} / 2\bigr)$  amount of time. When  $h^{\leftarrow} \rightarrow \infty$ ,  $\mathcal{L}_t^{(h^\leftarrow)}$  can be roughly viewed as constructing an "almost quasi-static" thermodynamics [6] bridging the initial  $p_T$  and the (quasi-)equilibrium  $p_0 = e^{-U_T\leftarrow}$ : for any distribution  $\mu_t^{\leftarrow}$  (probably far away from  $\rho_t^{\leftarrow}$ ), within a short time period  $\Delta t$  slightly larger than  $\mathcal{O}\bigl (1 / h_t^{\leftarrow 2}\bigr)$ , we have  $\mu_{t + \Delta t}^{\leftarrow}\approx \rho_{t + \Delta t}^{\leftarrow}$ , provided that  $s\mapsto \mu_s^{\leftarrow}$  evolves according to  $\mathcal{L}_s^{(h\leftarrow)}$ ; see Appx. C.2. This key finding will guide our analysis of the solution operator  $\Phi_{t,T}^{(h\leftarrow)}$ .
3.4 Score function is perturbed by a pulse
From (10), we know that  $v_{T}$  combines the averaged effect of  $\Phi_{t,T}^{(h^{\leftarrow})}(\nabla \cdot (p_{t}^{\leftarrow}\mathcal{E}_{t}^{\leftarrow}))$  for various  $t$ . As a first result, we consider  $\mathcal{E}_t^\leftarrow (x) = E(x)\delta_{t - s}$  for a fixed time instance  $s\in [0,T)$ , where  $\delta_{t - s}$  is the Dirac function. In this case,  $v_{T}$  no longer involves time integration and we have
$$
v _ {T} = - \frac {\left(1 + h _ {s} ^ {\leftarrow 2}\right)}{2} \Phi_ {s, T} ^ {\left(h ^ {\leftarrow}\right)} \left(\boldsymbol {\nabla} \cdot \left(p _ {s} ^ {\leftarrow} E\right)\right),
$$
$$
L \left(h ^ {\leftarrow}\right) = \frac {\left(1 + h _ {s} ^ {- 2}\right) ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {\left(\Phi_ {s , T} ^ {\left(h ^ {\leftarrow}\right)} \left(\boldsymbol {\nabla} \cdot \left(p _ {s} ^ {\leftarrow} E\right)\right)\right) ^ {2}}{p _ {0}}. \tag {13}
$$
To proceed, we need to make additional assumptions:
Assumption 3.3.
(1) For any  $t \in [0, T]$ ,  $U_t = -\log p_t$  is assumed to be strongly convex and the Hessian of  $U_t$  is bounded by two positive numbers  $m_t$  and  $M_t$  as below
$$
m _ {t} \boldsymbol {I} _ {d} \preceq \nabla^ {2} U _ {t} (x) \preceq M _ {t} \boldsymbol {I} _ {d}, \quad \forall x \in \mathbb {R} ^ {d}. \tag {14}
$$
Moreover, assume that
$$
m _ {t} \geq 1, \quad t \in [ 0, T ], \quad a n d \quad m _ {0} > 1. \tag {15}
$$
(2) For all  $t \in [0, T]$ , we choose  $h_t^{\leftarrow} = \mathfrak{h}$  as constant.
$$
\kappa_ {t} ^ {\leftarrow} := \left(1 + h ^ {- 2}\right) m _ {t} ^ {\leftarrow} - h ^ {- 2} \equiv \left(1 + h ^ {- 2}\right) m _ {T - t} - h ^ {- 2}, \tag {16}
$$
which characterizes the Hessian lower bound of  $V_{t}^{\leftarrow}$  (11). Note  $\kappa_{t}^{\leftarrow} \approx m_{t}^{\leftarrow}$  when  $h \gg 1$ . We would like to explain the reason behind the above assumptions, in particular, their practical relevance. Part (1) Strong convexity: this is a common assumption for Langevin sampling analysis [11, 14]. As the role of  $\mathcal{L}_t^{(h\leftarrow)}$  is essentially simulating a Langevin dynamics with time-dependent potential, it is reasonable to use this assumption as a starting point. Moreover, the algorithmic improvement in gDDIM [10] is highly inspired by a form with assuming the data distribution as a Gaussian; Fréchet inception distance (FID) [15], a widely used metric to evaluate the quality of generative model, essentially treats the data (in the feature space) as Gaussians. Therefore, we believe that this assumption can still capture some main features of realistic datasets. Part (2)  $m_t \geq 1$  for any  $t \in [0,T]$ : The second assumption  $m_t \geq 1$  means that  $p_t$  is more localized (smaller variance) than the standard Gaussian (unit variance), which is compatible with Assumption 3.1 (3). It can also ensure that  $V_{t}$  is strongly convex with positive Hessian lower bounds, i.e.,  $\kappa_{t}^{\leftarrow} \geq 1$ , for any  $t \in [0,T]$  and  $h \in (0,\infty)$ .
Proposition 3.4. Under Assumptions 3.1 and 3.3, suppose that  $\mathcal{E}_t^{\leftarrow}(x) = E(x)\delta_{t - s}$  for some fixed  $s\in [0,T)$ . If  $\mathsf{h}\geq \mathsf{h}_{lb}:= \max \left\{1 / 2,\mathsf{h}_0(1 / 2),\sqrt{\max\{0, - \frac{c_U}{\ln(2)},\sup_{t\in[s,T]}C_t^{\leftarrow,(2)}\}}\right\}$ , we have the upper bound of  $L(h)$  in (13):
$$
L (\mathfrak {h}) \leq C _ {\mathfrak {h}} (1 + \mathfrak {h} ^ {2}) ^ {2} \exp \left(- \int_ {s} ^ {T} \left(\mathfrak {h} ^ {2} - C _ {r} ^ {\leftarrow , (2)}\right) \kappa_ {r} ^ {\leftarrow} - C _ {r} ^ {\leftarrow , (1)} \mathrm {d} r\right), \tag {17}
$$
where  $C_{\mathsf{h}} = \frac{1}{2}\int \left(\nabla \cdot (p_s^{\leftarrow}E)\right)^2 /\rho_s^{\leftarrow},C_t^{\leftarrow ,(2)} = \frac{20 + 30M_t^{\leftarrow 2}}{m_t^{\leftarrow 2}}$ ; see Appx. E.1 for details about  $C_t^{\leftarrow ,(1)}$  and  $\mathsf{h}_0(1 / 2);c_U\in \mathbb{R}$  comes from Assumption 3.1. Moreover,  $\lim_{\mathsf{h}\to \infty}C_{\mathsf{h}},\lim_{\mathsf{h}\to \infty}C_{t}^{\leftarrow ,(1)}$  exist.
See Appx. E.1 for proofs. We remark that the above bound focuses on capturing the scaling with respect to  $\mathsf{h}^2$  but may not be tight for other parameters. It remains interesting to see how we can improve the above upper bound. The main conclusion is that: if the error function  $\mathcal{E}_t^{\leftarrow}$  is a pulse at time  $t = s$ , then for a large h,  $L(\mathsf{h})$  will decay to zero exponentially fast with respect to h. For 1D Gaussian case, we can clearly observe such an exponential decay in Fig. 2a, where we pick  $\mathcal{E}_t^{\leftarrow} = \mathbb{I}_{t\leq 0.95T}\nabla \log p_t^{\leftarrow}$ . The intuition behind this exponential suppressed prefactor is that for large  $h$ ,  $\Phi_{s,T}^{(h\leftarrow)}$  can be viewed as an almost quasi-static thermodynamics dragging any positive measure towards  $\rho_T^{\leftarrow} \approx p_0$ , as mentioned above in § 3.3. As  $\nu = \nabla \cdot (p_s^{\leftarrow}E)$  has measure zero, we can split it into positive and negative parts:  $\nu = \nu^{+} - \nu^{-}$  (assume  $\int \nu^{\pm} = 1$  WLOG). Each term  $\Phi_{s,T}^{(h\leftarrow)}(\nu^{+}) \approx \rho_T^{\leftarrow} \approx \Phi_{s,T}^{(h\leftarrow)}(\nu^{-})$ , which explains that  $\Phi_{s,T}^{(h\leftarrow)}(\nu) \approx 0$  for large h.
(a)  $\mathcal{E}_t^{\leftarrow} = \mathbb{I}_{t\leq 0.95T}\nabla \log p_t^{\leftarrow}$
Figure 2: 1D Gaussian  $p_0 = \mathcal{N}(0, \sigma_0^2)$  (with different  $\sigma_0$  smaller than one) and  $T = 2$ . Panel (a) validates the exponential decay of  $L(\mathsf{h})$  when the score function has no error near  $t \approx T$ , similar to Prop. 3.4. Panel (b) validates Prop. 3.5 that the ODE model  $(h = 0)$  outperforms the SDE model when there is a large score error at  $t \approx T$ . Panel (c) validates Prop. 3.6 that  $\lim_{\mathsf{h} \to \infty} L(\mathsf{h})$  exists.
(b)  $\mathcal{E}_t^{\leftarrow} = \mathbb{I}_{t\geq 0.995T}\nabla \log p_t^{\leftarrow}$
(c)  $\mathcal{E}_t^{\leftarrow} = \nabla \log p_t^{\leftarrow}$
3.5 Score function is only perturbed near the end of the generative process
Proposition 3.5. Under Assumptions 3.1 and 3.3, suppose that  $\mathcal{E}_t^{\leftarrow}(x) = \mathbb{I}_{t\in [T - a,T]}E(x)$  where  $a\ll 1$ . Then when  $a\ll 1$  and  $\mathsf{h}\gg 1$ , asymptotically,
$$
L (0) \sim \frac {a ^ {2}}{8} \int_ {\mathbb {R} ^ {d}} \frac {(\boldsymbol {\nabla} \cdot (p _ {0} E)) ^ {2}}{p _ {0}}, \quad L (\mathfrak {h}) \lesssim \frac {\left(1 - e ^ {- a \frac {\mathfrak {h} ^ {2}}{2} \kappa_ {0}}\right) ^ {2}}{2 \kappa_ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \frac {(\boldsymbol {\nabla} \cdot (p _ {0} E)) ^ {2}}{p _ {0}},
$$
with  $\kappa_0 = (1 + 1 / \mathfrak{h}^2)m_0 - 1 / \mathfrak{h}^2$  as in (16). The upper bound of  $L(\mathfrak{h})$  is tight asymptotically.
We remark that we made no assumption on how  $ah^2$  scales. If  $h \gg 1$ ,  $L(h) / L(0) \lesssim 4\left(1 - e^{-ah^2 \kappa_0 / 2}\right)^2 / a^2 \kappa_0^2$ . Case (I): If  $ah^2 \gg 1$ , then  $L(h) / L(0) \lesssim 4 / a^2 \kappa_0^2$ , which is large as  $a \ll 1$ . Case (II): If  $ah^2 \ll 1$ , then  $L(h) / L(0) \lesssim h^4$ , which is still large. In either case,  $L(h) / L(0)$  is expected to be large for a general  $E$  and the ODE model is preferred in this case. The intuition is that there is almost no time for the operator  $\mathcal{L}_t^{(h\gets)}$  to suppress the error  $E$ , so the prefactor  $1 + (h_t^{\leftarrow})^2$  in  $v_{T}$  (10) dominates (which is the key difference compared with Prop. 3.4). The proof is postponed to Appx. E.3. In Fig. 2b, we consider 1D Gaussian and only perturb the score function at the end of the generative process ( $\mathcal{O}_t^{\leftarrow} = \mathbb{I}_{t \geq 0.995T} \nabla \log p_t^{\leftarrow}$ ); clearly, the SDE-based models have comparatively larger error.
3.6 General error in score function
We can generalize Prop. 3.4 and Prop. 3.5 to a general error function  $\mathcal{E}^{\leftarrow}$ , and observe that  $L(\mathsf{h})$  will converge to a constant exponentially fast as  $\mathsf{h} \to \infty$ .
Proposition 3.6. Under Assumptions 3.1 and 3.3, we consider a general error function  $(t,x)\mapsto \mathcal{E}_t^{\leftarrow}(x)$ . Let  $\gamma = \inf_{\mathrm{h}\geq \mathrm{h}_{lb}}\inf_{t\in [0,T]}\kappa_t^{\leftarrow}$ . For any  $\alpha \in (0,1)$  and  $\beta \in (0,2)$ , when  $h\gg 1$
$$
L (\mathfrak {h}) \lesssim (1 + \alpha^ {2}) \mathcal {T} + (1 + \alpha^ {- 2}) C \gamma^ {- 1} (1 + \mathfrak {h} ^ {2}) \exp (- \mathfrak {h} ^ {2 - \beta} \gamma),
$$
$$
L (\mathfrak {h}) \gtrsim (1 - \alpha^ {2}) \mathcal {T} - (\alpha^ {- 2} - 1) C \gamma^ {- 1} (1 + \mathfrak {h} ^ {2}) \exp (- \mathfrak {h} ^ {2 - \beta} \gamma),
$$
where  $C$  is given in Appx. F.1 and  $\mathcal{T}$  (only depending on  $p_0$  and  $\mathcal{E}_T^{\leftarrow}$ ) is upper bounded by
$$
0 \leq \mathcal {T} \lesssim \frac {1}{2 m _ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \frac {\left(\boldsymbol {\nabla} \cdot \left(p _ {0} \mathcal {E} _ {T} ^ {\leftarrow}\right)\right) ^ {2}}{p _ {0}} \equiv \frac {1}{2 m _ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \left(\nabla \log p _ {0} \cdot \mathcal {E} _ {T} ^ {\leftarrow} + \boldsymbol {\nabla} \cdot \mathcal {E} _ {T} ^ {\leftarrow}\right) ^ {2} p _ {0}. \tag {18}
$$
In the limit  $\mathsf{h} \to \infty$ ,  $(1 - \alpha^2) \mathcal{T} \lesssim L(\mathsf{h}) \lesssim (1 + \alpha^2) \mathcal{T}$ , where  $\alpha \in (0, 1)$  is arbitrary. Hence, the tail behavior is that  $L(\mathsf{h})$  converges to  $\mathcal{T}$  exponentially fast as  $\mathsf{h} \to \infty$ . If we assume that  $\mathcal{E}_T^{\leftarrow} = \nabla \log p_0$ ,  $p_0 = \mathcal{N}(0, \sigma_0^2 I_d)$  in  $d$ -dimension, then the above upper bound is simply  $\mathcal{T} \lesssim d$ , which is independent of  $\sigma_0$  (see Appx. F.2). For 1D Gaussian in Fig. 2c, we can indeed observe that  $\lim_{\mathsf{h} \to \infty} L(\mathsf{h})$  exists, and is bounded above by  $d = 1$ ; see Appx. G for more types of error functions.
The above upper bound has an interesting tight connection to the generator of (overdamped) Langevin dynamics with drift  $-\nabla U_0 \equiv \nabla p_0$ . If we adopt constrained score models [23, 25], namely, parameterizing  $\log p_t$  instead of the score function  $\nabla \log p_t$  during training, the error  $\mathcal{E}_T^{\leftarrow} = \nabla \varphi$  for some scalar-valued function  $\varphi$ . Then the above upper bound becomes
$$
\frac {1}{2 m _ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} (\triangle \varphi - \nabla U _ {0} \cdot \nabla \varphi) ^ {2} e ^ {- U _ {0}} = \frac {1}{2 m _ {0} ^ {2}} \int_ {\mathbb {R} ^ {d}} \left(\mathcal {L} ^ {*} \varphi\right) ^ {2} e ^ {- U _ {0}}, \tag {19}
$$
where  $\mathcal{L}^*(\varphi) \coloneqq \triangle \varphi - \nabla U_0 \cdot \nabla \varphi$  whose adjoint operator  $\mathcal{L}(\mu) = \nabla \cdot (\nabla U_0 \mu) + \triangle \mu$  is the Fokker-Planck generator of the following Langevin dynamics  $\mathrm{d}X_t = -\nabla U_0(X_t) \, \mathrm{d}t + \sqrt{2} \, \mathrm{d}W_t$  where  $W_t$  is the standard Brownian motion. We remark that this formula (19) is general for constrained score models [23, 25]; see also F.2 for elaborations. An interesting open question is whether and how we can take the above upper bound into consideration when designing the loss function.
3.7 An application: exploring the effect of training weight
The above theoretical discussions suggest that diffusion models with large diffusion coefficients are more negatively affected by score error near data's side, whereas the ODE model is more negatively affected by the score error near the noise end. This leads us to conjecture that if we can control the training (e.g., by optimizing the training weight  $\omega_{t}$ ), so that the score error distribution near the noise end is reduced and meanwhile the score-matching loss is not significantly impacted, then it will very likely improve the ODE models. We report preliminary numerical experiments to support this idea in Appx. I, whereas a comprehensive study will be left as future works.
We present experiments on 2D Gaussian mixture model, Swiss roll, CIFAR10 to support our theoretical results: when numerical discretization error is not dominating, the sampling quality increases as  $h$  increases, a reminiscent of Prop. 3.4. Experimental details are postponed to Appx. H, as well as more numerical results (e.g., results about 1D Gaussian mixture and MNIST). Results by adopting various weight functions, a technique arising from theoretical predictions, are postponed to Appx. I. Source codes are available at https://github.com/yucaoyc/OptimalDiffusion.
Example 1: 2D 4-mode Gaussian mixture. We verify the theoretical results on 2D Gaussian mixture with a specified score error. In Fig. 3, a clear trend is that a higher  $h$  produces generated distributions that better match the marginal densities of  $p_0$ , and it is numerically verified by the purple line of Fig. 4a. In Fig. 4, with multiple settings of  $\mathcal{E}_t^{\leftarrow}$  and  $\epsilon$ , we observe a consistent phenomenon that as  $h$  increases, the KL divergence of true data and generated data converges exponentially fast, thus validating Prop. 3.6. It worths noticing that in all three settings of  $\mathcal{E}_t^{\leftarrow}$ , by simply adopting a larger diffusion coefficient  $h$  in (3), we can obtain better generative models without any extra training costs.
Figure 3: Visualization of marginal densities of 2D 4-mode Gaussian mixture, where  $\mathcal{E}_t^{\leftarrow} = \nabla \log p_t^{\leftarrow}$  and  $\epsilon = 0.2$ . The top row shows the marginal distribution of the first coordinate and the bottom row for the second coordinate. The red lines are the exact marginal distributions of  $p_0$  and the histograms (blue) visualize the empirical densities of generated samples.
(a)  $\mathcal{E}_t^{\leftarrow} = \nabla \log p_t^{\leftarrow}$
(b)  $\mathcal{E}_t^{\leftarrow} = \frac{1 + \sin(2\pi t / T)}{2}\nabla \log p_t^{\leftarrow}$
(c)  $\mathcal{E}_t^{\leftarrow} = \mathbb{I}_{t < 0.95T}\nabla \log p_t^{\leftarrow}$
Figure 4: Numerical results of 2D 4-mode Gaussian mixture. The above panels show that the KL divergences between the true distribution and the generated samples overall decay as  $h$  increases for various types of error perturbation of score functions.
Example 2: Swiss roll. We consider Swiss roll, a more complex distribution without exact score function available. We train the score function with the denoising score-matching objective [33] (Appx. H). The first plot in Fig. 5a shows the difference between true data and generated data measured by Wasserstein distance, which decays to zero exponentially fast, verifying Prop. 3.6. In the second plot of Fig. 5a, we tested various  $h$  and time steps for the generative process. The ODE model  $(h = 0)$  does not improve, as the number of time discretization steps increases, but near the continuous-time limit, all SDE cases  $(h > 0)$  are better than the ODE model. It suggests that our conclusions here is limited to the continuous-time setting. The exploration of time discretization errors will be future works. The generated data results in Fig. 5b demonstrate that with increasing  $h$ , the sample quality is improved; see Appx. H.6 for more results.
(a) Improved model performance with larger h
Figure 5: Numerical results of Swiss roll. Panel (a) shows the decay of Wasserstein distance between the true distribution and the generated samples with increasing  $h$  and 20,000 time-discretization steps, and the decay of Wasserstein distance with the increasing number of time-discretization steps and different  $h$ . Panel (b) shows generated samples with different  $h$ .
(b) Visualization results for selective h
Example 3: CIFAR-10. When using a large amount of parameters for score matching in practice, we observe that SDEs appear to perform better than ODEs as discretization error decreases, a result similarly observed on Swiss roll. This implicates the practical applications on generating samples of better quality under a given (pre-trained) score-matching model.
Table 1: CIFAR-10: We evaluate FIDs with different discretization steps and  $\mathrm{h}^2/\mathrm{g}^2$  on a pre-trained checkpoint entitled "vp/cifar10_ddpmpp連續" in [30]. We do not use any correctors and evaluate FIDs on  $10^4$  samples, thus the results for  $\mathrm{h}^2/\mathrm{g}^2 = 0, 1$  are different from the reported values.
Discretization steps	100	200	500	1000	2000	3000	4000
h2/g2=0	22.43	8.12	7.20	6.89	6.98	7.27	7.33
h2/g2=1	31.72	15.72	7.23	6.70	6.71	6.95	7.08
h2/g2=2	52.77	26.68	10.78	6.99	6.70	6.69	6.98
h2/g2=4	92.83	46.11	20.47	10.17	7.20	7.09	7.01
5 Summary and outlook
In this work, we study the effect of the diffusion coefficient on the quality of overall sample generation in the generative process. Theoretically, we provide understandings of scenarios in which the ODE-based model and the SDE-based model is superior than the other; see Prop. 3.4 and Prop. 3.5. Numerically, these results are validated via toy examples as well as benchmark datasets.
There are many interesting directions for continuing works. (1) As we focus on the asymptotic case, a time-dependent  $h_t^{\leftarrow}$  with large magnitude (i.e.,  $h_t^{\leftarrow} \gg 1$  for all  $t$ ) is essentially no different from a constant  $h_t^{\leftarrow} \equiv \mathsf{h}$  with  $\mathsf{h} \gg 1$ . Whether it is possible to use time-dependent  $t \mapsto h_t^{\leftarrow}$  to combine the advantages of ODE and large diffusion cases in dealing with different types of error of score functions in the non-asymptotic region is still an open question. (2) Can we design a practical criterion that directly learn the optimal magnitude of the noise-level function  $h_t^{\leftarrow}$  by looking at the score-training error distribution? Can we develop certain theoretical understanding of the empirical results in [19]? (3) How can we find a stable and accurate numerical scheme to deal with the fast diffusion case? (4) How can we generalize the above theoretical results by removing the convexity assumption, and including the low-dimensional feature of datasets into the theory [8, 5, 18]?
Concerning the broad impact, though we don't foresee any negative social impact of this work, the potential improvement of generative model might relate to creation of "deep fakes".
YC is sponsored by Shanghai Pujiang Program, and acknowledges the support from City University of Hong Kong during his visit. JC and YL are supported by the NSFC Major Research Plan - Interpretable and General-purpose Next-generation Artificial Intelligence (92370205). XZ is supported by General Research Funds from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. 11308121, 11318522) and the NSFC/RGC Joint Research Scheme [RGC Project No. N-CityU102/20 and NSFC Project No. 12061160462].
Supplementary Material for "Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations"
A Notation Convention.
Table 2: Summary of important quantities in this paper
Quantity	Notation	Notes
Forward process	Xt	t=0: data distribution
Backward/generative process	Yt	t=0: noise distribution
Backward process with inexact score	Ŷt
Distribution of forward process	pt	pt:=law(Xt)
Distribution of backward process	qt	law(Yt):=qt≡pT-t≡pt
Distribution of approximated backward process	qtd	qtd:=law(Ŷt), qtd=qt when error ε=0
Error function of the score	εS←	0≤ε≪1 and S←=O(1)
The exact potential	Ut	pt:=e-UT
Modified potential	Vt←	see (11)
Normalizing constants	ZV:=∫e-V	Vis arbitrary
B Discussion and proof for  $\S 2$
B.1 Proof of (3)
We re-state the conclusion in (3) in the following lemma:
Lemma B.1. For any function  $h_t^{\leftarrow}$ , if one chooses  $A^{\leftarrow}$  as in (3), then we have  $q_t(x) = p_{T-t}(x)$  for any  $t \in [0, T]$  and  $x \in \mathbb{R}^d$ .
Proof. We can easily write down the Fokker-Planck equation of (1)
$$
\partial_ {t} p _ {t} (x) = - \boldsymbol {\nabla} \cdot \left(f _ {t} (x) p _ {t} (x)\right) + \frac {g _ {t} ^ {2}}{2} \triangle p _ {t} (x).
$$
Since we need to ensure  $q_{t} = p_{T - t}$ , we require
$$
\begin{array}{l} \partial_ {t} q _ {t} (x) = \nabla \cdot \left(f _ {t} ^ {\leftarrow} (x) q _ {t} (x)\right) - \frac {g _ {t} ^ {\leftarrow 2}}{2} \triangle q _ {t} (x) \\ = \nabla \cdot \left(f _ {t} ^ {\leftarrow} (x) q _ {t} (x) - \frac {g _ {t} ^ {\leftarrow 2} + h _ {t} ^ {\leftarrow 2}}{2} \nabla q _ {t} (x)\right) + \frac {h _ {t} ^ {\leftarrow 2}}{2} \triangle q _ {t} (x) \\ = \nabla \cdot \left(\left(f _ {t} ^ {\leftarrow} (x) - \frac {g _ {t} ^ {\leftarrow 2} + h _ {t} ^ {\leftarrow 2}}{2} \nabla \log q _ {t} (x)\right) q _ {t} (x)\right) + \frac {h _ {t} ^ {\leftarrow 2}}{2} \triangleq q _ {t} (x) \left. \right. \\ = \nabla \cdot \left(\left(f _ {t} ^ {\leftarrow} (x) - \frac {g _ {t} ^ {\leftarrow 2} + h _ {t} ^ {\leftarrow 2}}{2} \nabla \log p _ {t} ^ {\leftarrow} (x)\right) q _ {t} (x)\right) + \frac {h _ {t} ^ {\leftarrow 2}}{2} \triangle q _ {t} (x) \left. \right. \\ = - \boldsymbol {\nabla} \cdot \left(A _ {t} ^ {\leftarrow} (x) q _ {t} (x)\right) + \frac {h _ {t} ^ {\leftarrow 2}}{2} \triangle q _ {t} (x), \\ \end{array}
$$
by noting that  $A_{t}^{\leftarrow}(x)$  is specified in (3). This equation is exactly the Fokker-Planck equation of (2).
B.2 The role of  $g_{t}$