SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu Cao *
Jingrun Chen†
Yixin Luo
Xiang Zhou
The diffusion model has shown remarkable success in computer vision, but it remains unclear whether the ODE-based probability flow or the SDE-based diffusion model is more superior and under what circumstances. Comparing the two is challenging due to dependencies on data distributions, score training, and other numerical issues. In this paper, we study the problem mathematically for two limiting scenarios: the zero diffusion (ODE) case and the large diffusion case. We first introduce a pulse-shape error to perturb the score function and analyze error accumulation of sampling quality, followed by a thorough analysis for generalization to arbitrary error. Our findings indicate that when the perturbation occurs at the end of the generative process, the ODE model outperforms the SDE model with a large diffusion coefficient. However, when the perturbation occurs earlier, the SDE model outperforms the ODE model, and we demonstrate that the error of sample generation due to such a pulse-shape perturbation is exponentially suppressed as the diffusion term's magnitude increases to infinity. Numerical validation of this phenomenon is provided using Gaussian, Gaussian mixture, and Swiss roll distribution, as well as realistic datasets like MNIST and CIFAR-10.
Diffusion models have achieved remarkable success in various artificial intelligence context generation tasks, particularly in computer vision [13]. This technique is rapidly evolving with industrial-level products like DALL-E series. The diffusion model was first proposed and studied by Sohl-Dickstein et al. [26] in 2015. Later, Song and Ermon [29] proposed score matching with Langevin dynamics (SMLD) and Ho et al. [16] further explored the Denoising Diffusion Probabilistic Models (DDPM). Both formalisms can be interpreted as time-discretization of stochastic differential equations (SDEs) [30]. Since the publication of these seminal works, many techniques have been proposed to improve the efficiency and accuracy of diffusion models, such as DDIM [28], Analytic-DPM [4], gDDIM [36], EDM [19], and consistency model [31], among others.
The score-based diffusion model involves two steps [30, 17]. Firstly, one estimates the score function, which is the gradient of the logarithm of the probability density function, in the form of a neural network. This step uses trajectories of an Ornstein-Uhlenbeck (OU) process starting with given data
$^{*}$ Institute of Natural Sciences and School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China, yucao@sjtu.edu.cn
†University of Science and Technology of China, Hefei 230026, China; Suzhou Institute of Advanced Research, University of Science and Technology of China, Suzhou 215123, China, jingrunchen@ustc.edu.cn
$^{\ddagger}$ University of Science and Technology of China, Hefei 230026, China; Suzhou Institute of Advanced Research, University of Science and Technology of China, Suzhou 215123, China, seeing@mail.ustc.edu.cn
$^{\S}$ School of Data Science and Department of Mathematics, City University of Hong Kong, Kowloon, Hong Kong SAR, xizhou@cityu.edu.hk
samples. This process of injecting noise into structured data is usually referred to as the inference process. Secondly, new samples are generated by simulating a time-reversed SDE, with a drift term depending on the learned score function from the first step. This is known as the generative process.
In general, there are two diffusion coefficients  $g$  in the inference process and  $h$  in the generative process; see § 2. Regardless of the choice of  $h$  and  $g$ , it is always possible to design an SDE in the generative process that matches the forward inference process in the weak sense, i.e., the probability density functions match for both processes. We highlight that this function  $h$  (unlike  $g$ ) does not only appear in the diffusion term, but also enters the drift term in the generative SDE. The choice of  $g$  is equivalent to time re-scaling (see Appx. B.2), while the choice of  $h$  is an important topic in practice. Two common choices of  $h$  are Probability Flow  $h = 0$  [9, 32], which refers to as an ODE, and an SDE-based diffusion model with  $h = g$  [16, 29, 30]. When the score training is accurate, the choice of this function  $h$  does not affect the sample generation quality in the continuous-time setting.
In practice, numerical error is inevitable during training the score function. Recent theoretical works [10, 7] have shown that the sample generation quality are affected by three aspects: (1) the truncation of simulation time to a finite  $T$ ; (2) the inexact score function; (3) the time-discretization error. The first error is not significantly since the forward OU process converges to the equilibrium Gaussian measure exponentially fast in  $T$ . The third error can be reduced systematically by more efficient numerical schemes [21], such as exponential integrator proposed in [35, 36]. The inexact training of score function has a few important but subtle consequences. Recent works [10, 7] analyzed the convergence rate of diffusion models, provided that the score training error is sufficient small. However, once the score training is not accurate, the nice equivalence of the generated distribution free of the generative diffusion coefficient  $h$  no longer holds as in the idealized situation of exact score function. This raises a key question of our interest about how the choice of  $h$  can affect the sample quality in the face of the inexact score training error. Qualitatively, there are two distinctive cases:  $h = 0$  or  $h$  is large. An important question to ask is: in the presence of non-negligible score training errors, which  $h$  will produce better sampling quality? Is it the probability flow  $(h = 0)$  or the SDE? More quantitatively, what magnitude of  $h$  is optimal?
Related works The impact of  $h$  on the generative process seems not yet fully investigated in recent literature, as most experiments used the default choice of this parameter. However, some authors have reported related empirical observations. For example, Song et al. [30] empirically observed that the choice of  $h = g$  produces better sample generation quality than the ODE case ( $h = 0$ ) with real datasets. On the other hand, Denoising Diffusion Implicit Models (DDIM) in [28] includes both deterministic and stochastic samplers and points out that the probability flow ( $h = 0$ ) can produce better samples with improved numerical schemes for the generative process. [36] generalized the DDIM and tried to explain the advantages of a deterministic sampling scheme over the stochastic one for fast sampling. Moreover, Karras et. al., [19] had empirically searched for optimal coefficients which had shown to bring practical advantages. None of these empirical results delivered comprehensive investigations on the influence of the diffusion coefficient, and a consistent and affirmative answer to our question still awaits. Recently, there has been rapid progress in theoretical works on error analysis for diffusion models, as seen in [10, 7] and references therein. However, these analyses usually assume specific settings of  $h$ , such as  $h = g$ . Furthermore, it seems that directly analyzing upper bounds based on these error estimations cannot provide adequate information about choosing the optimal  $h$ ; see Appx. B.4. Albergo et al. proposed a unified framework known as stochastic interpolants and slightly discussed the optimal choice between the probability flow and diffusion models [1, Sec. 2.4]. It is interesting to see how our theoretical analysis below can generalize to their promising unified settings [1].
Our approach To investigate the effect of the diffusion coefficient  $h$  on sampling quality, we adopt the continuous-time framework, which precludes time discretization errors. We measure sample quality by the KL divergence between the data distribution  $p_0$  and the distribution of the generative SDE at the terminal time  $T$ . Given the assumption that the score function carries numerical errors, we consider  $h$  as a controller and aim to minimize the KL divergence with respect to  $h$ . While the optimization problem is straightforward to set up, it is challenging to draw valuable theoretical insights in a general setting of approximate score functions. Therefore, we choose the asymptotic approach, assuming that the error from the training score is reasonably small with a magnitude of  $\epsilon$ . Under this assumption, the leading-order term of the KL divergence takes the form
error of sample generation in KL divergence  $= L(h)\epsilon^{2} + \mathcal{O}\big(\epsilon^{3}\big)$
This  $\epsilon^2$  order is known in [7, 10], but the dependence of this Gateaux differential  $L(h)$  on  $h$  and other factors has yet to be understood at all. Our contribution is to analyze how  $L(h)$  behaves as  $h$  varies; in particular, by considering the constant  $h$  in two limiting situations:  $h = 0$  and  $h \gg 1$ .
Main Contributions We summarize main contributions below:
- We prove that when the error in score function approximation is a time-localized function only at the beginning of the inference step (i.e., at the end of the generative process), the ODE case  $(h = 0)$  outperforms the SDE case  $(h \to \infty)$ ; see Prop. 3.5. If this (time-localized) error occurs in the middle, then the SDE case has an exponentially smaller error than the ODE case  $(h = 0)$ , as  $h \to \infty$  (see Prop. 3.4). See Appx. E.4 for reasons behind the time-localized choice.
- For a general score training error, we prove that as  $h \to \infty$ , the leading-order term  $L(h)$  above converges exponentially fast to a constant, which only depends on the distribution  $p_0$  and the score training error at the end of the generative process; see Prop. 3.6. The conclusion about the optimal  $h$  depends on how the score training error is distributed over the time horizon  $[0, T]$ .
Numerically, we validate the above phenomenon for 1D Gaussian, 2D Gaussian mixture, and Swiss roll distribution, as well as realistic datasets like MNIST and CIFAR-10. Due to the tight connection between the distribution of score training error and  $h$ , our results may suggest backwardly modifying loss functions during training to adapt to a particular diffusion coefficient of interest. This is a topic of independent interest and we report some preliminary experiments in Appx. I to validate potential applications of our theoretical analysis. A comprehensive investigation will be left as future works.
Notation convention The time duration  $T > 0$  is a fixed parameter. For any time-dependent function  $(t,x)\mapsto f_t(x)$ , where  $x\in \mathbb{R}^d$  and  $t\in [0,T]$ , we denote  $f_{t}^{\leftarrow}(x)\equiv f_{T - t}(x)$ . Sometimes we directly use the "arrowed" variables  $f_{t}^{\leftarrow}(x)$  to highlight the direction of time is from reference noise to the data distribution (i.e., the generative direction) even without referring to  $f$  first. The notation  $f\lesssim g$  means that  $f\leq cg$  where  $c\to 1$  in a certain limit, i.e.,  $\lim \sup f / g\leq 1$ ;  $f\sim g$  means  $\lim (f / g) = 1$ . The asymptotic parameter will be explained below explicitly. When two matrices  $A\spreceq B$ , it means  $B - A$  is positive semidefinite.  $I_{d}$  is the  $d$ -dimensional identity matrix;  $\mathbb{I}_A$  means an indicator function of a set  $A$ ; Id is the identity operator. For a random variable  $X$ ,  $\operatorname {law}(X)$  means the distribution of  $X$ . Some important quantities are summarized in Appx. A.
Score-based generative models Suppose we have a collection of data from an unknown distribution with density  $p_0$ , we can inject noise into data via the following SDEs:
$$
\mathrm {d} X _ {t} = f _ {t} \left(X _ {t}\right) \mathrm {d} t + g _ {t} \mathrm {d} W _ {t}, \quad \operatorname {l a w} \left(X _ {0}\right) = p _ {0}, \tag {1}
$$
where the drift coefficient  $f_{(\cdot)}(\cdot): \mathbb{R}^d \times \mathbb{R} \to \mathbb{R}^d$  is a time-dependent vector field, and the diffusion coefficient  $g_{(\cdot)}: \mathbb{R} \to \mathbb{R}$  is a scalar-valued function (for simplicity). A widely used example is variance-preserving SDE (VP-SDE) with  $f_t(x) = -\frac{1}{2} g_t^2 x$  and  $g > 0$  is typically chosen as a non-decreasing function in literature [30]. Without loss of generality, we can assume  $g_t = 1$  since for any non-zero  $g$ , its effect is simply to re-scale the time; see Appx. B.2.
Denote the probability density of  $X_{t}$  as  $p_t$  and the score function is defined as  $\nabla \log p_t$ . One main innovation in diffusion models is to find a "backward" SDE  $Y_{t}$  such that  $Y_{t}$  drives the state with distribution  $p_T$  back to  $p_0$ . We adopt the arrow of time in this backward direction now and write  $Y_{t}$  as
$$
\mathrm {d} Y _ {t} = A _ {t} ^ {\leftarrow} \left(Y _ {t}\right) \mathrm {d} t + h _ {t} ^ {\leftarrow} \mathrm {d} W _ {t}, \quad \operatorname {l a w} \left(Y _ {0}\right) = p _ {T}, \tag {2}
$$
where  $h_t^{\leftarrow}$  is an arbitrary real-valued function of time. The distribution of  $Y_{t}$  is denoted as  $q_{t}$ . Provided that the score function is available, we can select  $A^{\leftarrow}$  such that  $q_{t}$  is the same as  $p_{T - t}$ , in particular,  $q_{T} = p_{0}$ . It is not hard to derive that we can ensure  $q_{t} \equiv p_{T - t}$  if we choose
$$
A _ {t} ^ {\leftarrow} (x) = - f _ {t} ^ {\leftarrow} (x) + \frac {\left(g _ {t} ^ {\leftarrow}\right) ^ {2} + \left(h _ {t} ^ {\leftarrow}\right) ^ {2}}{2} \nabla \log p _ {t} ^ {\leftarrow} (x). \tag {3}
$$
A self-contained proof is provided in Appx. B.1. When  $h^{\leftarrow} = g^{\leftarrow}$ , it refers to the backward SDE used in [30]; when  $h^{\leftarrow} = 0$ , it refers to the probability flow therein. More general interpolation of diffusion and flow can be found in, e.g., [1].
Training of score functions The above score function  $(t,x)\mapsto \nabla \log p_t^{\leftarrow}(x)$  needs to be trained from data. Denoising score matching [33] refers to the following score-matching loss (SML) function to train the score whose parameterized architecture is denoted as  $\mathfrak{S}$ :
$$
\min  _ {\mathfrak {S}} \int_ {0} ^ {T} \omega_ {t} \mathbb {E} _ {X _ {0} \sim p _ {0}} \mathbb {E} _ {X _ {t} \sim p _ {t | 0} (\cdot | X _ {0})} \left[ \left\| \mathfrak {S} _ {t} (X _ {t}) - \nabla \log p _ {t | 0} (X _ {t} | X _ {0}) \right\| ^ {2} \right] \mathrm {d} t, \tag {4}
$$
where  $p_{t|0}(x_t|x_0)$  is the transition probability of the state  $x_0$  at time 0 towards the state  $x_t$  at time  $t$  for the forward process (1). The function  $\omega_t \geq 0$  is a weight function. The default choice in many literature is that  $g_t = \sqrt{\beta_0 + (\beta_1 - \beta_0)t}$ ,  $0 < \beta_0 < \beta_1$  are parameters, and one chooses the weight function as follows:
$$
\text {D e f a u l t w e i g h t :} \quad \omega_ {t} = \varpi_ {t} ^ {2}, \quad \varpi_ {t} = \sqrt {1 - e ^ {- \frac {1}{2} t ^ {2} \left(\beta_ {1} - \beta_ {0}\right) - t \beta_ {0}}}. \tag {5}
$$
The quantity  $\varpi_{t}$  has the meaning as the standard deviation of  $X_{t}$  conditioned on a fixed  $X_0$  in the forward process. See § 3.7 and Appx. I for more weight functions.
Source of errors There is usually intrinsic error due to an inexact score function. It is not negligible in many scenarios, e.g., there is only a finite amount of samples of  $p_0$  available or only a small neural network architecture is achievable. However, it is reasonable to assume that this non-negligible error is reasonably small, and we decompose the trained score function  $\mathfrak{S}_t^{\leftarrow}$  into
$$
\mathfrak {S} _ {t} ^ {\leftarrow} (x) = \nabla \log p _ {t} ^ {\leftarrow} (x) + \epsilon \mathcal {E} _ {t} ^ {\leftarrow} (x), \tag {6}
$$
where  $\epsilon$  is small,  $\mathcal{E}_t^{\leftarrow}$  is assumed to be  $\mathcal{O}(1)$  and the total error is  $\epsilon \mathcal{E}_t^{\leftarrow}$ . The generative process used in practice has to use law  $(Y_0) = \mathcal{N}(0, I_d)$  instead since  $p_T$  is intractable in (2). By choosing  $T$  large enough so that  $p_T \approx \mathcal{N}(0, I_d)$ , we can neglect this error due to the finite  $T$ . Besides, we also need some numerical schemes to simulate this generative SDE, which also leads into discretization errors. In summary, there are three sources of errors (1)  $p_T \neq \mathcal{N}(0, I_d)$ : this is the source of errors in the initial distribution of the generative process; (2)  $\mathcal{E}^{\leftarrow} \neq 0$ : error from imperfect score function from training; (3) numerical discretization of the generative process. The third error can be systemically eliminated by choosing a high-order scheme [21] or an extremely small time step. It has been observed that by choosing a more accurate numerical scheme, e.g., exponential integrator, one can reduce the computational costs [35, 36]. As for the first error, if one chooses the OU process for (1),  $p_T$  converges to  $\mathcal{N}(0, I_d)$  exponentially fast and thus  $T = \mathcal{O}(\ln(\delta))$  where  $\delta$  is the error between  $p_0$  and the distribution of generated samples. Therefore, the choice of  $T$  is, in practice, not hard to manage. More details about these three error sources can be found in, e.g., [7, 10] or Appx. B.3.
Figure 1: Schematic illustration of the main message: the distribution of the score error  $\mathcal{E}_t^{\leftarrow}$  w.r.t. time also matters, in addition to the score-matching loss (4). Asymptotically, the score error can be viewed as "additive" and the error from two time regions (blue and green) might decay or magnify as the magnitude of diffusion coefficient  $h^{\leftarrow}$  increases (see the right picture). The yellow region is a transitional region whether the effect of h is not easy to decide.
3 Asymptotic analysis of terminal errors
We use the KL divergence between the data distribution  $p_0$  and the distribution of generated samples to quantify the performance of generative model, which depends on the error of score function  $\epsilon \mathcal{E}^{\leftarrow}$ ,
the diffusion coefficient  $h^{\leftarrow}$ , and data distribution  $p_0$ . To extract the main feature, we first let  $\epsilon \rightarrow 0$  and estimate
$$
\text {s a m p l e} = L \left(h ^ {\leftarrow}, \mathcal {E} ^ {\leftarrow}, p _ {0}\right) \epsilon^ {2} + \mathcal {O} \left(\epsilon^ {3}\right).
$$
Whenever  $p_0$  and  $\mathcal{E}^{\leftarrow}$  are obvious from the context, we simply write  $L(h^{\leftarrow}) \equiv L(h^{\leftarrow}, \mathcal{E}^{\leftarrow}, p_0)$ . Next, we formulate the main problem setup and assumptions in § 3.1. The expression of  $L$  is shown in Prop. 3.2. Then we let  $h_t^{\leftarrow} \equiv \mathsf{h}$  be independent of time, and study how the leading order function  $L$  depends on  $\mathsf{h}$  in various settings of the error function  $\mathcal{E}_t^{\leftarrow}$ . Firstly, we consider  $\mathcal{E}_t^{\leftarrow}(x) = \delta_{t - s} E(x)$  as a time-localized function and two limiting scenarios:  $h^{\leftarrow} = \mathsf{h}$  where  $\mathsf{h} = 0$  (ODE case) and  $\mathsf{h} \to \infty$  (SDE case with large diffusion). When  $\mathcal{E}_t^{\leftarrow}(x) = \delta_{t - s} E(x)$  is a time-localized function at the end of the generative process (i.e.,  $s$  is close to  $T$ ), the ODE case will outperform the SDE case (see Prop. 3.5); otherwise, the SDE case has an exponentially smaller error than the ODE case as  $\mathsf{h} \to \infty$  (see Prop. 3.4). Secondly, by combing Prop. 3.4 and Prop. 3.5, the tail behavior of  $\mathsf{h} \mapsto L(\mathsf{h})$  for a general  $\mathcal{E}^{\leftarrow}$  is described in Prop. 3.6. The reasons behind considering this pulse-shape  $\mathcal{E}_t^{\leftarrow}$  will be discussed in Appx. E.4.
3.1 Set-up and assumptions
We fix the time  $T$  and consider the following SDE for the generative process in  $t \in [0, T]$ ,
$$
\mathrm {d} \widetilde {Y} _ {t} = \left(- f _ {t} ^ {\leftarrow} (\widetilde {Y} _ {t}) + \frac {\left(g _ {t} ^ {\leftarrow}\right) ^ {2} + \left(h _ {t} ^ {\leftarrow}\right) ^ {2}}{2} \mathfrak {S} _ {t} ^ {\leftarrow} (\widetilde {Y} _ {t})\right) \mathrm {d} t + h _ {t} ^ {\leftarrow} \mathrm {d} W _ {t}, \quad \operatorname {l a w} (\widetilde {Y} _ {0}) = p _ {T}, \tag {7}
$$
which can be regarded as a perturbed equation (2) of  $Y_{t}$ . Denote the distribution of  $\widetilde{Y}_{t}$  as  $\widetilde{q}_t$ . Note that  $\widetilde{q}_T$  depends on both  $h^{\leftarrow}$  and  $\epsilon \mathcal{E}_t^{\leftarrow}$  (hidden inside  $\mathfrak{S}_t^\leftarrow \equiv \nabla \log p_t^\leftarrow +\epsilon \mathcal{E}_t^{\leftarrow}$ ); however, when  $\epsilon = 0$ , by (3),  $\widetilde{q}_T \equiv q_T \equiv p_0$  for any  $h^{\leftarrow}$ . We quantify the sample generation quality via
$$
\operatorname {K L} \left(p _ {0} | | \widetilde {q} _ {T}\right) = \int p _ {0} \log \left(p _ {0} / \widetilde {q} _ {T}\right).
$$
Due to the presence of  $\epsilon \mathcal{E}^{\leftarrow}$  with non-zero  $\epsilon$ , in general  $\mathrm{KL}\big(p_0||\widetilde{q}_T\rangle > 0$ .
Assumption 3.1. Throughout this section, we assume that:
(1) For the forward process, we assume  $f_{t}(x) = -\frac{1}{2} x$ ,  $g_{t} = 1$  without loss of generality.
(2) The data distribution has the density  $p_0$ .
(3) There exists  $c_U \in \mathbb{R}$  such that  $U_0(x) - |x|^2 / 2 \geq c_U$ , for any  $x \in \mathbb{R}^d$ , where  $U_0 := -\log p_0$ .
Recall that a generic  $g_{t}$  is equivalent to the time re-scaling (Appx. B.2). So this choice of  $g_{t} = 1$  refers to the generic choice in VP-SDE [30]. The second assumption is also mild; in practice, if  $p_0$  is a delta distribution, a common practice is that one tries to learn the mollified version  $p_{\sigma}(x) := \int_{\mathbb{R}^d} \frac{1}{(2\pi\sigma^2)^{d/2}} \exp \left( -(x - y)^2 / 2\sigma^2 \right) p_0(y) \, \mathrm{d}y$  instead, as in the GAN [27] and early-stop techniques [7, 10]. The third assumption is not restricted, for example,  $U_0(x) = |x|^2$  and  $c_U = 0$ . As many realistic datasets are almost compactly supported, we expect that  $\rho_0 = e^{-U_0}$  decays faster than a Gaussian, namely,  $\rho_0(x) \leq \widetilde{C} e^{-|x|^2/2}$  for some  $\widetilde{C} > 0$ , which reduces to the third one.
3.2 Asymptotic expansion of the KL divergence with respect to  $\epsilon$
We introduce a time-dependent operator
$$
\mathcal {L} _ {t} ^ {(h \leftarrow)} (\mu) (x) := - \boldsymbol {\nabla} \cdot \left(\left(\frac {1}{2} x + \frac {1 + h _ {t} ^ {\leftarrow 2}}{2} \nabla \log p _ {t} ^ {\leftarrow} (x)\right) \mu (x)\right) + \frac {h _ {t} ^ {\leftarrow 2}}{2} \Delta \mu (x), \tag {8}
$$
which is the generator in the Fokker-Planck equation of  $q_{t}$  for (2). Define an operator  $\Phi_{s,t}^{(h\leftarrow)}$  as follows: given any function  $\mu$ , define  $\Phi_{s,t}^{(h\leftarrow)}(\mu)$  to be the solution at time  $t$  of the following Fokker-Planck equation with  $r \in [s,t]$ :  $\partial_r\theta_r = \mathcal{L}_r^{(h\leftarrow)}(\theta_r)$ , and with initial condition  $\theta_s = \mu$ . We define  $\Phi_{s,t}^{(h\leftarrow)}(\mu) \coloneqq \theta_t$ . Properties of this operator are collected in Appx. C.1.
We use the notation  $\widetilde{q}^{\epsilon}$  to refer to  $\widetilde{q}$  since we need to calculate the its derivative for the small parameter  $\epsilon$ . The dependence of  $\widetilde{q}$  on  $h^{\leftarrow}$  is suppressed for short notations. We have the following asymptotic result with the proof given in Appx. C.3.
Proposition 3.2. Define  $v_{T} \coloneqq \partial_{\epsilon} \widetilde{q}_{T}^{\epsilon}|_{\epsilon = 0}$  as the first-order perturbation of  $\widetilde{q}_{T}^{\epsilon}$ . We have
$$
K L \left(p _ {0} \| \widetilde {q} _ {T} ^ {\epsilon}\right) = L \left(h ^ {\leftarrow}\right) \epsilon^ {2} + \mathcal {O} \left(\epsilon^ {3}\right), \tag {9}
$$