SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Eunbi Yoon $^{1*}$ , Keehun Park $^{2}$ , Sungwoong Kim $^{3\dagger}$ , Sungbin Lim $^{1,4,5\dagger}$
<sup>2</sup>Artificial Intelligence Graduate School, UNIST
$^{3}$ Department of Artificial Intelligence, Korea University
$^{4}$ LG AI Research
SNU-LG AI Research Center
yon6286@gmail.com; sungbin@korea.ac.kr
Investigating the optimal stochastic process beyond Gaussian for noise injection in a score-based generative model remains an open question. Brownian motion is a light-tailed process with continuous paths, which leads to a slow convergence rate for the Number of Function Evaluation (NFE). Recent studies have shown that diffusion models suffer from mode-collapse issues on imbalanced data. In order to overcome the limitations of Brownian motion, we introduce a novel score-based generative model referred to as Lévy-Itô Model (LIM). This model utilizes isotropic  $\alpha$ -stable Lévy processes. We first derive an exact reverse-time stochastic differential equation driven by the Lévy process, then develop the corresponding fractional denoising score matching. LIM takes advantage of the heavy-tailed properties of the Lévy process. Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models across various image datasets such as CIFAR10, CelebA, and imbalanced dataset CIFAR10LT. Comparing our results to those of DDPM with 3.21 Fréchet Inception Distance (FID) and 0.6437 Recall on the CelebA dataset, we achieve 1.58 FID and 0.7006 Recall using the same architecture. LIM shows the best performance in NFE 500 with  $2\times$  faster total wall-clock time than the baseline.
Score-based diffusion models train time-dependent score models given a diffusion process and generate samples from the corresponding reverse process [39]. The feasibility of score-based diffusion models is based on the time-reversal formula of the Stochastic Differential Equations (SDEs) driven by the Brownian motion [2]. The exact form of the density function of the Brownian motion enables efficient sampling and easy theoretical handling compared to other types of noise. The score-based diffusion model is easy to train because the score function is approximated through Denoising Score-Matching (DSM). Additionally, the solution of reverse SDE can be simulated through the numerical SDE solver. This enables effective training and utilization of the model in various applications. One limitation of score-based diffusion models is that the paths produced by the SDEs need numerous iterations to achieve a data distribution, leading to slower sampling. To solve these problems, various methods have been proposed to reduce the convergence rate while maintaining to generate high fidelity samples [22]. Another effort has also been made to use distillation to reduce the number of steps [33]. In addition, it has been pointed out that Diffusion models show significant degradation in terms of fidelity and diversity when dealing with imbalanced datasets [30]. Thus, there is a need
*This work is done at UNIST.
Figure 1: Forward and reverse processes of Diffusion model and LIM. The formula indicates there exists a reverse-time stochastic differential equation (SDEs) corresponding to SDEs driven by isotropic  $\alpha$ -stable Lévy processes and can be exactly expressed by the fractional score function  $S_t^{(\alpha)}(\mathbf{x})$ .
Figure 2: Trajectories of forward and reverse SDEs driven by (a) Brownian process; (b) isotropic  $\alpha$ -stable Lévy process. Red boxes indicate large jumps generated by the heavy-tail property of  $\alpha$ -stable distribution. (c) Comparison of Maximum Mean Discrepancy (MMD) between the Diffusion Model (DM) [39] and the proposed model on the mixture of Gaussians.
to perform further research and develop score-based generative models that use alternative noises. Recently, there have been attempts to use new noises such as gamma distribution [24] or generalized Gaussian distribution [8]. These models have shown possibilities in expanding the range of noise processes. However, they did not propose a time-reversal formula, which is an important property for the feasibility of score-based generative models. As a result, the time-reversal formula has not yet been fully proven for alternative noises.
In this paper, we propose a novel score-based generative model, Lévy-Itô Model (LIM), which utilizes isotropic  $\alpha$ -stable Lévy processes as noise injection. This is the first attempt to use the isotropic  $\alpha$ -stable Lévy process in a score-based generative model and prove the exact time-reversal formula of SDEs driven by Lévy process. It is noted that this is also the first continuous-time model using the heavy-tailed process, which allows for the derivation of a probability fractional ODE (See more details in Appendix C). We derive a fractional score function to match the drift term of the generator of time-reversal SDEs driven by Lévy process and propose fractional Denoising Score Matching, which is a more general version of DSM [40] to approximate the fractional score function.
Looking at the reverse path of LIM in Figure 2, we observe large jumps occurring near the noise space. It causes the trajectory to move quickly towards the sample space, which is shown in Figure 1. As LIM has sufficient ability to move from the sample space to the noise space in spite of small noise coefficient due to large jump noises, there is a higher likelihood of reaching multiple modes more precisely. These characteristics can be observed in Figure 3, which is based on the simplest form of an imbalanced dataset. LIM's superiority can be observed not only from the ratio aspect but also from FID and MMD, which are calculated directly without using any embeddings for the sets of true samples and generated samples. LIM shows that the distribution  $p_{\theta}$  of its generated samples is similar to the ground truth distribution  $p_{\mathrm{data}}$ .
Figure 3: Mode estimation for the synthetic data. The dataset consists of two mixtures of Gaussian distributions, with a ground truth ratio of 10:1. The blue clusters in subplots (a) is the ground truth and each (b) and (c) represents the distribution of generated samples in 2D by the diffusion model and LIM. In subplot (b), it can be seen that for the Diffusion model, the generated samples have a ratio of 5.6:1, which is quite different from the ratio of ground truth in subplot (a). However, in subplot (c), for LIM, the generated samples have a ratio of 11.1:1, which is relatively close to the ground truth. This similarity can be observed not only from the ratio aspect but also from FID (Fréchet Inception Distance) and Maximum Mean Discrepancy (MMD). The diffusion model shows FID of  $8.312 \pm 0.904$  and MMD of  $0.02 \pm 0.002$ . On the other hand, the LIM achieves FID of  $0.663 \pm 0.376$  and MMD of  $0.026 \pm 0.003$ .
We draw inspiration from the observation that the time-reversal formula for SDEs is known not only for diffusion processes but also for SDEs driven by Lévy processes containing jumps [7]. In addition, the invariant measure of the Euler scheme has been shown to converge to the invariant measure of the solution of SDEs driven by Lévy process [6]. Moreover, there is an example of using Lévy processes in Markov Chain Monte Carlo (MCMC), where the corresponding research demonstrated a faster convergence rate for NFE both theoretically and experimentally [37]. However, it cannot be directly extended to SDEs driven by multivariate stable processes and use an approximation to calculate the score function of a data distribution, which degrades the potential performance. These observations motivate us to consider using Lévy processes in score-based generative models.
Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to diffusion models across various image datasets such as CIFAR10, CelebA, and imbalanced dataset CIFAR10LT. We realize that the heavy-tail property of the Lévy process allows us to overcome the limitations of Brownian motion with respect to mode estimation, sample diversity, and convergence rate for NFE. Through the above experiments, we emphasize we have further expanded the scope of score-based generative models.
The contributions of our paper can be summarized as follow:
- We propose a novel score-based generative model, Lévy-Itô Model (LIM), which utilizes isotropic  $\alpha$ -stable Lévy processes as noise injection.
- We derive an exact reverse-time stochastic differential equation driven by the Lévy process and develop the corresponding fractional Denoising Score Matching.
- We validate that LIM allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models across various image datasets and shows more robustness against imbalanced data.
Diffusion models have shown improvements in performance through various approaches such as incorporating guidance [18], introducing new architectures [29], and proposing novel training methods [13]. Additionally, various attempts to reduce convergence rates have been proposed, such as using ODE solvers [23], or Fourier Neural Operator [42]. Despite these advancements, diffusion models still face inherent limitations such as slow convergence rates and the mode-collapse issue in imbalanced datasets [30]. There have been attempts to use new noises in score-based generative models. The
denoising generative model with gamma distribution is one such example [24], which shows faster convergence for NFE than score-based diffusion models. [8] proposes using heavy-tailed noises in score-based generative models by extending the noise injection to generalized Gaussian distributions. The authors state that the heavy-tailed DSM is more robust against imbalance tasks than score-based diffusion models. They commonly use a variant of the Denoising Diffusion Probabilistic Model (DDPM) structure to apply alternative noises. These models have shown the potential to expand the range of noise processes, however, their performance in terms of the Fréchet Inception Distance (FID) is not as high as that of score-based diffusion models.
Since these distributions follow a Lévy process [11, 41], the drift term for the time-reversal formula can be computed according to [7]. For isotropic  $\alpha$ -stable Lévy processes, which follow symmetric and stable distributions, an exact drift term can be computed from the time-reversal formula. Training can be stable and accurate without the needs for integration. Furthermore, with the knowledge of the exact drift term, a reverse sampling formula ensuring convergence can be derived. In contrast, the drift terms used in Heavy-tailed DSM and Denoising Diffusion Gamma Models appear as an integral, making it difficult to derive an exact form. Even if integration is used during training or approximations are applied to the drift term, accumulating errors lead to inaccurate and unstable training. Moreover, when using an approximation for the drift term to make training practical, there's no assurance that the distance between the distribution of actual data,  $p_{\mathrm{data}}$  and the distribution obtained from reverse sampling,  $p_{\theta}$  aligns within a certain range. For Heavy-tailed DSM, despite proposing a modified score function corresponding to the Generalized Gaussian distribution and providing a training method, there is no theoretical foundation proving that the modified score function converges to  $p_{\mathrm{data}}$  through reverse sampling. Similarly, Denoising Diffusion Gamma Models also fail to propose an exact drift term and present limitations by not providing theoretical evidence for the convergence of reverse sampling using the proposed score function. As a result, the FID performances of these two papers are significantly lower than that of DDPM [15].
3.1 Isotropic alpha-stable distributions
Let  $\alpha \in (0,2]$  be a characteristic exponent that determines the decay rate, and  $\gamma$  be a scale parameter. For  $d\in \mathbb{N}$ ,  $\mathcal{S}\alpha \mathcal{S}^d (\gamma)$  denotes the  $d$ -dimensional isotropic  $\alpha$ -stable distribution. If a random variable  $X\in \mathbb{R}^d$  follows  $\mathcal{S}\alpha \mathcal{S}^d (\gamma)$ , then the characteristic function of  $X$  is  $\mathbb{E}[e^{i\langle \mathbf{u},X\rangle}] = e^{-\gamma^{\alpha}||\mathbf{u}||^{\alpha}}$  and has heavy-tail properties with tail index  $\alpha$ , i.e.,  $\mathbb{P}(||X|| > r)\sim r^{-\alpha}$ . A closed-form of density
Figure 4: Comparison of samples from Gaussian, non-isotropic, and isotropic  $\alpha$ -stable distribution.
function for  $S\alpha S^d(\gamma)$  is unknown, except for  $\alpha = 1$ ,  $\alpha = 1.5$  and  $\alpha = 2$  cases<sup>3</sup>. Note that the behavior of isotropic  $\alpha$ -stable distribution is different from non-isotropic  $\alpha$ -stable in multi-dimension since the components of isotropic  $\alpha$ -stable are not independent unless  $\alpha = 2$ . Contrary to [43] which can not be extended to SDEs that are driven by  $\alpha$ -stable Lévy processes, we utilize isotropic  $\alpha$ -stable processes for driving noise in SDEs. See Figure 4 for the comparison between sample distributions of Gaussian, non-isotropic  $\alpha$ -stable, and isotropic  $\alpha$ -stable random variable.
3.2 Lévy processes
An  $\mathbb{R}^d$ -valued stochastic process  $(L_t)_{t \geq 0}$  with  $L_0 = 0$  is called Lévy process if (i)  $L_t$  has independent increments, (ii)  $L_t$  has stationary increments, and (iii)  $L_t$  has stochastically continuous sample paths. Thus, Lévy process may have at most countably many discontinuous points, also known as jumps [3]. If for all  $s < t$ ,  $(L_t - L_s) \stackrel{d}{=} L_{t-s}$  follows  $S\alpha S^d((t - s)^{1 / \alpha})$ , where  $\stackrel{d}{=}$  means that the two processes have the same law, then  $L_t$  is referred to as a  $d$ -dimensional isotropic  $\alpha$ -stable Lévy process, denoted by  $L_t^\alpha$ . Notably,  $L_t^\alpha$  is a prototypical pure jump process. The  $\alpha$ -stable distribution has a heavy-tailed property, meaning that the frequency of large jumps increases as  $\alpha$  decreases. According to the
3When  $\alpha = 2$  , it holds  $\mathcal{S}\alpha \mathcal{S}^d (\gamma) = \mathcal{N}^d (0,\sqrt{2}\gamma \mathbf{I})$  where  $\mathcal{N}$  is a Gaussian distribution and  $I\in \mathbb{R}^{d\times d}$  is the identity operator
Levy-Itô decomposition [21], we can split the sample paths of Lévy process into drift parts, Brownian process parts, and jump parts. The focus of this paper is on rotationally invariant pure-jump Lévy processes, whose Lévy measure is  $\nu(\mathrm{d}\mathbf{x}) \sim ||\mathbf{x}||^{-d - \alpha}\mathrm{d}\mathbf{x}$ .
4 Lévy-Itô Model
4.1 Time-Revesal of SDEs driven by Lévy processes
In this section, we provide a time-reversal theory for SDEs driven by Lévy processes as follows:
$$
d \vec {X} _ {t} = b (t, \vec {X} _ {t -}) d t + \sigma_ {L} (t) d L _ {t} ^ {\alpha}, \quad t \in [ 0, T ] \tag {1}
$$
where  $\vec{X}_{t-}$  denotes the left limit of  $\vec{X}$  at time  $t$  which implies the càdlàg (right conti. with left limits) solution. We define fractional score function  $S_{t}^{(\alpha)}$  by,
$$
S _ {t} ^ {(\alpha)} (\mathbf {x}) := \frac {\Delta^ {\frac {\alpha - 2}{2}} \nabla p _ {t} (\mathbf {x})}{p _ {t} (\mathbf {x})}, \quad \mathbf {x} \in \mathbb {R} ^ {d}, t \in [ 0, 1 ] \tag {2}
$$
where  $\Delta^{\frac{\gamma}{2}}$  denote the fractional Laplacian of order  $\frac{\gamma}{2}$  for  $\gamma \in (-1,2)$  [27]. Precisely,  $\Delta^{\frac{\gamma}{2}}f(\mathbf{x})\coloneqq$ $\mathcal{F}^{-1}\{\| \mathbf{u}\| ^{\gamma}\mathcal{F}\{f\} (\mathbf{u})\}$  where  $\mathcal{F}$  and  $\mathcal{F}^{-1}$  are the Fourier and the inverse Fourier transform, respectively. The time-reversal of  $\vec{X}_t$  is defined by  $\vec{X}_t\coloneqq \vec{X}_{t - }\vec{X}_{T - }$  where  $t$  runs backward from  $T$  to 0. Note that  $\vec{X}_t$  is a Lévy process with respect to natural filtration when time flows forwards [17], which is the càglàd (left conti. with right limits). Our first main result is the derivation of the reverse-time SDE corresponding to (1).
Theorem 4.1 (Time-Reversal of SDE). Let  $1 < \alpha \leq 2$ . Suppose  $b(t, \mathbf{x})$  is locally bounded and the marginal distribution of  $(X_{t})_{t \in [0, T]}$  has a density function  $p_{t}(\mathbf{x})$ .
$$
d \bar {X} _ {t} = \left(b (t, \bar {X} _ {t +}) - \alpha \cdot \sigma_ {L} ^ {\alpha} (t) S _ {t} ^ {(\alpha)} (\bar {X} _ {t +})\right) \bar {d t} + \sigma_ {L} (t) d \bar {L} _ {t} ^ {\alpha} + d \bar {Z} _ {t}, \tag {3}
$$
where  $\bar{d} t$  is an infinitesimal negative timestep,  $\bar{L}_t^\alpha$  is the backward version of the isotropic  $\alpha$ -stable Lévy process, and  $\bar{Z}_t$  is the backward version of a Lévy-type stochastic integral  $Z_t$  such that  $\mathbb{E}[Z_t] = 0$  with finite variation.
See more detail in Theorem B.1 for the definition of  $Z_{t}$ . Now we propose a score-based generative model with Lévy process  $L_{t}^{\alpha}$ , refers to Lévy-Ito Model (LIM). In [39], the VP-SDE formula is defined as  $d\vec{X}_t = -\frac{\beta(t)}{2}\vec{X}_tdt + (\beta(t))^{\frac{1}{2}}dB_t$  and  $\beta(t)$  is set to a linear function  $\beta(t) = (\beta_1 - \beta_0)t + \beta_0$ . On the other hand, we extended [39]'s formula to the SDE driven by isotropic  $\alpha$ -stable Lévy process,  $d\vec{X}_t = -\frac{\beta(t)}{\alpha}\vec{X}_tdt + (\beta(t))^{\frac{1}{\alpha}}dL_t^{\alpha}$ , where we simulate the noise term in (3) using only  $L_{t}^{\alpha}$ . This choice is motivated by the fact that, in contrast to  $\sigma_L(t)dL_t^{\alpha}$ , which exhibits infinite variation,  $dZ_t$  has a mean of 0 and finite variation, rendering it comparatively negligible. The specific  $\beta(t)$  in our paper is derived from the cosine schedule proposed in [25] and tailored to LIM as  $\beta(t) = -\alpha\frac{d\log(\cos(\frac{\pi t}{2}))}{dt}$  for setting a forward SDE with  $b(t,\mathbf{x}) = -\frac{\beta(t)}{\alpha}\mathbf{x}$  and  $\sigma_L(t) = \beta^{\frac{1}{\alpha}}(t)$  in (1). Then the solution  $\vec{X}_t$  has the same law with  $a(t)\vec{X}_0 + \gamma(t)\epsilon$  where  $\epsilon \sim S\alpha S^d(1)$ ,  $a(t) = \exp(-\int_0^t\frac{\beta(s)}{\alpha}ds)$ , and  $\gamma(t) = (1 - a^\alpha(t))^{\frac{1}{\alpha}}$ . As  $b(t,\mathbf{x}) = -\frac{\beta(t)}{\alpha}\mathbf{x}$  satisfies the condition of Theorem 4.1, we can obtain the reverse-time SDE of the beta-schedule version,
$$
d \bar {X} _ {t} = \Big (- \frac {\beta (t)}{\alpha} \bar {X} _ {t +} - \alpha \cdot \beta (t) \cdot S _ {t} ^ {(\alpha)} (\bar {X} _ {t +}) \Big) \bar {d t} + (\beta (t)) ^ {\frac {1}{\alpha}} d \bar {L} _ {t} ^ {\alpha}. \tag {4}
$$
For  $t < s$ , the solution of equation (4) can be represented as an integral, utilizing the semi-linear structure of the reverse SDE,
$$
\mathbf {x} _ {t} = \frac {a (t)}{a (s)} \mathbf {x} _ {s} - \alpha^ {2} \cdot a (t) \int_ {s} ^ {t} \frac {d}{d u} \Big (\frac {1}{a (u)} \Big) S _ {u} ^ {(\alpha)} (\mathbf {x} _ {u}) d u + \int_ {s} ^ {t} (\beta (u)) ^ {\frac {1}{\alpha}} \frac {a (t)}{a (u)} d \bar {L} _ {t} ^ {\alpha}. \qquad (5)
$$
We can get faster sampling method by using an approximation for the second term  $\int_s^t\frac{\beta(u)}{a(u)} S_u^{(\alpha)}(\mathbf{x}_u)du$  This term can be approximated as  $(\int_s^t\frac{\beta(u)}{a(u)} du)\cdot S_s^{(\alpha)}(\mathbf{x}_s)$  and it is possible to calculate  $\int_s^t\beta (u)\frac{a(t)}{a(u)}$  the scale parameter  $\gamma$  of which follows  $\gamma^{\alpha} = \left|\int_s^t\frac{d}{du}\left(e^{-\int_u^t\beta (k)}\right)\right|$
Corollary 4.1 (Variant of Euler-Maruyama with dynamic time increment). Suppose the fractional score function in the SDE given by equation (4) satisfies the conditions stated in Theorem E.2, and  $a(t), \gamma(t)$  are bounded. Then, there exists a Markov chain  $(\mathbf{x}_t)$  that follows:
$$
\mathbf {x} _ {t} = \frac {a (t)}{a (s)} \mathbf {x} _ {s} + \alpha^ {2} \Big (\frac {a (t)}{a (s)} - 1 \Big) S _ {s} ^ {(\alpha)} (\mathbf {x} _ {s}) + \left(\left(\frac {a (t)}{a (s)}\right) ^ {\alpha} - 1\right) ^ {\frac {1}{\alpha}} \boldsymbol {\epsilon}. \tag {6}
$$
Here,  $\epsilon \sim S\alpha S^d(1)$  for  $s > t$ , and  $\Delta t = s - t \ll 1$ . As a result of the conditions being satisfied, the Wasserstein-1 distance between the invariant measures of the solution of Equation (4) and  $(\mathbf{x}_t)$  is bounded by  $(\Delta t)^{\frac{1}{\alpha}}$  [6]. (See more detail in Appendix  $E$ )
The paper [6] outlines a way to find bounds on the Wasserstein-1 distance by determining bounds for the approximation of the drift term,  $J_{1}$ , and the approximation of the stochastic term,  $J_{2}$ . In the equation, only the bound for the approximation of the drift term,  $J_{1}$ , is utilized. Furthermore, since the law of a weak solution is same to the law of the strong solution, the Wasserstein-1 distance between the invariant measures for the strong solution and the approximation can be used for the that of the weak solution and the approximation as well. In addition, we can also find a probability ODE corresponding to (1).
Theorem 4.2 (Probability Fractional ODE). Let  $b(t, \mathbf{x}) : \mathbb{R}^d \to \mathbb{R}$  and  $\sigma_L(t) : \mathbb{R} \to \mathbb{R}$  be functions that satisfy the Lipschitz condition as stated in [35]. For the SDE following (1), the solution  $(X_t)_{t \in [0,T]}$  of the SDE satisfies the following ODE:
$$
d \vec {X} _ {t} \stackrel {\rightarrow} {=} \left(b (t, \vec {X} _ {t -}) - \sigma_ {L} ^ {\alpha} (t) S _ {t} ^ {(\alpha)} (\vec {X} _ {t -})\right) d t. \tag {7}
$$
Due to its semilinear structure, the solution to (7) can be represented as an integral. This is shown in Lemma D.2. The Euler method can then be used to find the solution.
Corollary 4.2 (Deterministic ODE sampling). If the drift term in (7) is Lipschitz continuous and the solution  $\mathbf{x}_t$  has a bounded second derivative, then a sequence  $(\mathbf{x}_t)$  can be obtained using the Euler-scheme:
$$
\mathbf {x} _ {t} = \frac {a (t)}{a (s)} \mathbf {x} _ {s} + \alpha \cdot \left(\frac {a (t)}{a (s)} - 1\right) \cdot S _ {s} ^ {(\alpha)} (\mathbf {x} _ {s}) \tag {8}
$$
where  $s > t$ . When the step size is  $\Delta t$ , the global truncation error is bounded by  $O(\Delta t)$  [5].
4.2 Estimating Fractional Score Function
Let  $q_{\alpha}(\mathbf{x})$  be the density of  $S\alpha S^d(1)$ . A weak solution of the beta-schedule version is  $\vec{X}_t = a(t)\vec{X}_0 + \gamma(t)\epsilon$  where  $\epsilon \sim S\alpha S^d(1)$  with the following transition density:
$$
p _ {t} (\mathbf {x} _ {t} | \mathbf {x} _ {0}) = \frac {1}{\gamma^ {d} (t)} q _ {\alpha} \left(\frac {\mathbf {x} _ {t} - a (t) \mathbf {x} _ {0}}{\gamma (t)}\right). \tag {9}
$$
We train the fractional score model  $\mathbf{s}_t(\mathbf{x}_t;\theta)$  which estimates  $S_{t}^{(\alpha)}(\mathbf{x}_{t})$  at the optimal point. The method of approximating the fractional score function  $S_{t}^{(\alpha)}(\mathbf{x}_{t})$  can be equivalent to estimating the fractional score function  $S_{t}^{(\alpha)}(\mathbf{x}_{t}|\mathbf{x}_{0})$ , which replaces the fractional score function of term  $p_t(\mathbf{x}_t)$  in (2) with that of the transition density (9).
Theorem 4.3 (Fractional Denoising Score Matching). For a parameter  $\theta$ , two loss functions  $L_{1}(\theta)$  and  $L_{2}(\theta)$  are given as follows:
$$
L _ {1} (\theta) = \mathbb {E} _ {t, \mathbf {x} _ {t} \sim p _ {t} \left(\mathbf {x} _ {t}\right)} \| \mathbf {s} _ {t} \left(\mathbf {x} _ {t}; \theta\right) - S _ {t} ^ {(\alpha)} \left(\mathbf {x} _ {t}\right) \| _ {2} ^ {2}. \tag {10}
$$
$$
L _ {2} (\theta) = \mathbb {E} _ {t, (\mathbf {x} _ {0}, \mathbf {x} _ {t}) \sim p _ {t} (\mathbf {x} _ {0}, \mathbf {x} _ {t})} \| \mathbf {s} _ {t} (\mathbf {x} _ {t}; \theta) - S _ {t} ^ {(\alpha)} (\mathbf {x} _ {t} | \mathbf {x} _ {0}) \| _ {2} ^ {2}. \tag {11}
$$
Then, there exists a constant  $C$  satisfying  $L_{1}(\theta) = L_{2}(\theta) + C$  so that the optimization of two loss functions with respect to  $\theta$  is equivalent.