Score-based Generative Models with Lévy Processes
Authors: Eunbi Yoon
Keehun Park
Sungwoong Kim
Sungbin Lim
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Generative Model
Score-based Method
Lévy processes

Abstract: Investigating the optimal stochastic process beyond Gaussian for noise injection in a score-based generative model remains an open question. Brownian motion is a light-tailed process with continuous paths, which leads to a slow convergence rate for the Number of Function Evaluation (NFE). Recent studies have shown that diffusion models suffer from mode-collapse issues on imbalanced data.
In order to overcome the limitations of Brownian motion, we introduce a novel score-based generative model referred to as Lévy-Itō Model (LIM). This model utilizes isotropic $\alpha$-stable Lévy processes. We first derive an exact reverse-time stochastic differential equation driven by the Lévy process and develop the corresponding fractional denoising score matching. The proposed generative model takes advantage of the heavy-tailed properties of the Lévy process. Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models across various image datasets such as CIFAR10, CelebA, and imbalanced dataset CIFAR10LT. Comparing our results to those of DDPM with 3.21 Fréchet Inception Distance (FID) and 0.6437 Recall on the CelebA dataset, we achieve 1.58 FID and 0.7006 Recall using the same architecture. LIM shows the best performance in NFE 500 with $2\times$ faster total wall-clock time than the baseline.

Score-based Generative Models with Lévy Processes
Eunbi Yoon Keehun Park², Sungwoong Kim?t, Sungbin Lim L4.5t
Department of Statistics, Korea University
Artificial Intelligence Graduate School, UNIST
3 Department of Artificial Intelligence, Korea University
'LG AI Research
SNU-LGG Research Center
yon62860gmail com: sungbin@korea.c ac kr
Abstract
Investigating the optimal stochastic process beyond Gaussian for noise injection in score-based generative model remains an open question. Brownian motion is light-tailed process with continuous paths, which leads to a slow convergence rate for the Number of Function Evaluation (NFE). Recent studies have shown that diffusion models suffer from mode-collapss issues on imbalanced data. In order to overcome the limitations of Brownian motion, introduce novel score- based generative model referred to as Lévy-Itō Model (LIM).. This model utilizes isotropic o-stable Lévy processes. We first derive an exact reverse-time stochastic differential equation driven by the Lévy process, then develop the corresponding fractional denoising score matching. LIM takes advantage of the heavy-tailed properties of the Lévy process. Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models acToss various image datasets such as CIFARIO. CelebA, and imbalanced dataset CIFARIOLT. Comparing our results to those DDPM with 3.21 Fréchet Inception Distance (FID) and 0.6437 Recall on the CelebA dataset, we achieve 1.58 FID and 0. 0.7006 Recall using the same architecture. LIM shows the best performance in NFE 500 with 2x faster total wall-clock time than the baseline.
1 Introduction
Score-based diffusion models train dime-dependent score models given a diffusion process and generate samples from the corresponding reverse process [39]. The feasibility of score-based based diffusion models is based the time-reversal formula of the Stochastic Differential Equations (SDEs) driven by the Brownian motion [2]. The exact form the density function of the Brownian motion enables efficient sampling and easy theoretical handling compared to other types of noise. The score-based diffusion model easy to train because the score function is approximated through Denoising Score- Matching (DSM). Additionally, the solution of reverse SDE can be simulated through the numerical SDE solver. This enables effective training and utilization of the model in various applications. One limitation of score-based diffusion models is that the paths produced by the SDEs need numerous iterations achieve data distribution, leading slower sampling. To solve these problems, various methods have been proposed to reduce the convergence rate while maintaining to generate high fidelity samples [22]. Another effort has also been made to use distillation to reduce the number of steps [33]. addition, has been pointed out that Diffusion models show significant degradation in terms of fidelity and diversity when dealing with imbalanced datasets [30]. Thus, there is a need
"This work done UNIST.
1 CCorresponding Author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

Brownian
(a = 2)
Lévy (a < 2)
Fonward SDE dx,= b(1, _, oz(0
Xo (data)
Xy (noise)
Reverse SDE dx,- (b(1.)) - a (x( dr a.(0)
Figure 1: Forward and reverse processes Diffusion model and LIM. The formula indicates there exists reverse-time stochastic differential equation (SDEs) corresponding to SDEs driven by isotropic
o-stable Lévy processes and can be exactly expressed by the fractional score function s(a) (x).
Forward SDE
E
Reverse SDE
[c]
[b]
Figure 2: Trajectories of forward and reverse SDEs driven by (a) Brownian process; (b) isotropic a- stable Lévy process. Red boxes indicate large jumps generated by the heavy-tail property of a-stable distribution. (c) Comparison of Maximum Mean Discrepancy (MMD) between the Diffusion Model (DM) [39] and the proposed model on the mixture of Gaussians.
perform further research and develop score-based generative models that use alternative noises. Recently, there have been attempts to use new noises such as gamma distribution [24] or generalized Gaussian distribution [8]. These models have shown possibilities in expanding the range of noise processes. However, they did not propose a time-reversal formula, which an important property for the feasibility of score-based generative models. As result, the time-reversal formula has not yet been fully proven for alternative noises.
this paper, we propose a novel score-based generative model, Lévy-Itô Model (LIM), which utilizes isotropic a-stable Lévy processes as noise injection. This is the first attempt to use the isotropic a-stable Lévy process in a acore-based generative model and prove the exact time-reversal formula of SDEs driven by Lévy process. It is noted that this is also the first continuous-time model using the heavy-tailed process, which allows for the derivation of probability fractional ODE (See more details in Appendix C). We derive a fractional score function to match the drift term of the generator of time-reversal SDEs driven by Lévy process and propose fractional Denoising Score Matching, which a more general version of DSM [40] to approximate the fractional score function.
Looking at the reverse path of LIM in Figure 2, we observe large jumps occurring near the noise space. It causes the trajectory move quickly towards the sample space, which is shown in Figure As LIM has sufficient ability to move from the sample space to the noise space spite small noise coefficient due to large jump noises, there is higher likelihood of reaching multiple modes more precisely. These characteristics can be observed in Figure which is based on the simplest form of an imbalanced dataset. L.M's superiority can he observed not only from the ratio aspect but also from FID and MMD, which are calculated directly without using any embeddings for the sets of true samples and generated samples. LIM shows that the distribution pe its generated samples similar to the ground truth distribution Pdata-

---

Ground Truth
(b) Diffusion Model
Lávy Itó Modal
10
30-
-5
-10
-20
Figure 3: Mode estimation for the synthetic data. The dataset consists of two mixtures of Gaussian distributions, with a ground truth ratio of 10:1. The blue clusters subplots (a) is the ground truth and each (b) and (c) represents the distribution of generated samples in 2D by the diffusion model and LIM. In subplot (b), can be seen that for the Diffusion model, the generated samples have a ratio of 5.6:1, which is quite different from the ratio of ground truth in subplot (a). However, in subplot (c), for LIM, the generated samples have ratio of 11.1:1, which is relatively close to the ground truth. This similarity can be observed not only from the ratio aspect but also from FID (Fréchet Inception Distance) and Maximum Mean Discrepancy (MMD). The diffusion model shows FID of 8.312 + 0.904 and MMD of 0.02 0.002. On the other hand, the LIM achieves FID 0.663 0.376 and MMD of 0.026 + 0.003.
We draw inspiration from the observation that the time-reversal formula for SDEs is known not only for diffusion processes but also for SDEs driven by Lévy processes containing jumps [7]. In addition, the invariant measure the Euler scheme been shown to converge to the invariant measure of the solution of SDEs driven by Lévy process [6]. Moreover, there is an example of using Lévy processes in Markov Chain Monte Carlo (MCMC), where the corresponding research demonstrated a faster convergence rate for NEE both theoretically and experimentally [37]. However, it cannot be directly extended to SDEs driven by multivariate stable processes and use approximation to calculate the score function a data distribution, which degrades the potential performance. These observations motivate us to consider using Lévy processes in score-based generative models.
Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to diff'usion models across various image datasets such as CIFAR 10, CelebA, and imbalanced dataset CIFAR IOLT. We realize that the heavy-tail property of the Lévy process allows to overcome the limitations of Brownian motion with respect mode estimation, sample diversity, and convergence rate for NFE. Through the above experiments, we emphasize we have further expanded the scope of score-based generative models.
The contributions of our paper can be summarized as follow:
We propose a novel score- basred generative model, Lévy-Itô Model (LIM), which utilizes isotropic a-stable Lévy processes as noise injection.
We derive exact reverse-time stochastic differential equation driven by the Lévy process and develop the corresponding fractional Denoising Score Matching.
validate that L.M allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models across various image datasets and shows more robustness against imbalanced data.
2 Related works
Diffusion models have shown improvements in performance through various approaches such as incorporating guidance [18], introducing new architectures [29]. and proposing novel training methods [13]. Additionally, various attempts to reduce convergence rates have been proposed, such as using ODE solvers [23], Fourier Neural Operator [42]. Despite these advancements, diffusion models still face inherent limitations such as slow convergence rates and the mode-collapse issue in imbalanced datasets [30]. There have been attempts to new noises in score-based generative models. The

---

denoising generative model with gamma distribution is one such example [24], which shows faster convergence for NFE than score-based diffusion models. [8] proposes using heavy-tailed noises in score-based generative models by extending the noise injection to generalized Gaussian distributions. The authors state that the heavy-tailed DSM is more robust against imbalance tasks than score-based diffusion models. They commonly use a variant of the Denoising Diffusion Probabilistic Model (DDPM) structure to apply alternative noises. These models have shown the potential to expand the range of noise processes. however, their performance in terms of the Fréchet Inception Distance (FID) is not as high as that of score-based diffusion models.
Since these distributions follow a Lévy process [11, 41], the drift term for the time-reversal formula can be computed according [7]. For isotropic a-stable Lévy processes, which follow symmetric and stable distributions, an exact drift term can be computed from the time-reversal formula. Training can be stable and accurate without the needs for integration. Furthermore, with the knowledge of the exact drift term, a reverse sampling formula ensuring convergence can be derived. In contrast, the drift terms used in Heavy-tailed DSM and Denoising Diffusion Gamma Models appear difficult to derive an exact form. Even integration is used during training or approximations are applied to the drift term, accumulating eггоrs lead to inaccurate and unstable training. Moreover, when using an approximation for the drift term to make training practical. there's no assurance that the distance between the distribution of actual data, Pasu and the distribution obtained from reverse sampling, Pe aligns within a certain range. For Heavy-tailed DSM, despite proposing modified score function corresponding to the Generalized Gaussian distribution and providing a training method, there is no theoretical foundation proving that the modified score function converges to Pdata through reverse sampling. Similarly, Denoising Diffusion Gamma Models also fail to propose an exact drift term and present limitations by not providing theoretical evidence for the convergence of reverse sampling using the proposed score function. As result, the FID performances of these two papers are significantly lower than that of DDPM [15].
integral, making it
3 Background
3.1 Isotropic alpha-stable distributions
Gaussian
non-isotrepic gw].5
Iotrosic
Let a (0,2] be a characteristic exponent that determines the decay rate, and y be a scale parameter. For d E N. (7))(7) denotes the d-dimensional isotropic a-stable distri- bution. If random variable X E IRd fol- lows (a))( then the characteristic func- tion of X is B[e+(u.X]] = e->l |=]"" ull" and has heavy-tail properties with tail index a, i.e.,
-70
Figure 4: Comparison of samples from Gaussian, non-isotropic, and isotropic c-stable distribution.
[[||X|| > r) ~ ~rr A closed-form of density function for (aS)(1) is unknown, except for = = 1.5 and a = 2 cases". Note that the behavior of isotropic a-stable distribution different from non-isotropic a-stable in multi-dimension since the components of isotropic a-stable are not independent unless a = 2. Contrary to [43] which can not be extended to SDEs that are driven by a-stable Lévy processes, we utilize isotropic a-stable processes for driving noise in SDEs. See Figure 4 for the comparison between sample distributions of Gaussian, non-isotropic a-stable, and isotropic a-stable random variable.
3.2 Lévy processes
An IR stochastic process (L))r2 with Lo = 0 is called Lévy process if (i) L, has independent increments, (ii) Le has stationary increments, and (iii) L, has stochastically continuous sample paths. Thus, Lévy process may have at most countably many discontinuous points, also known as jumps [3].
If for all s < t, (Le L.) d L.-, follows SaSA((t s)2/a), where d means that the two processes have the same law, then L, is referred to as d-dimensional isotropic stable Lévy process, denoted by Lę Notably, LE a prototypical pure jump process. The a-stable distribution has heavy-tailed property, meaning that the frequency of large jumps increases as a decreases. According to the
Wheen 2, it holds SaS5())) (O, V2y1) where N is a Gaussian distribution and I E Rxxd is the identity operator

---

Lévy-It decomposition [21], we can split the sample paths of Lévy process into drift parts, Brownian process parts, and jump parts. The focus of this paper on rotationally invariant pure-jump Lévy processes, whose Lévy measure is v(dx) ~ ||x|| ||x||-d-axx.
4 Lévy-Itō Model
4.1 Teme-Revesal of SDEs driven by Lévy processes
this section, we provide a time-reversal theory for SDEs driven by Lévy processes as follows:
dx, = b(t, (.tt. (t)dLE E [0.
(I)
where X, denotes the left limit of X at time which implies the càdlåg (right conti. with left limits) solution. We define fractional score function s(a) by,
A 2-2 Vx((x),
(x)
x EER
Rd . [0.1]
(2)
(x(
where A denote the fractional Laplacian of order 3/2 for E (-1,2) [27]. Precisely, A fxx) : F [[]][[[[[[]] where F and F-1 are the Fourier and the inverse Fourier transform, respec- tively. The time-reversal of X is defined by X, X,_ - -x-- where t runs backward from T to O. Note that X, is Lévy process with respect to natural filtration when time flows forwards [17], which is the càglåd (left conti. with right limits). Our first main result is the derivation of the reverse-time
SDE corresponding to (1).
Theorem 4.1 (Time--Rversal SDE). Let 1 < S 2. Suppose b(t, x) is locally bounded and the marginal distribution of (X,)) has a density function (x()).
dX. = (bt,, - a. - - (X(+) dt (t)dd + dZ,,
(3)
where dt is an infinitesimal negative timestep, Lg is the backward version of the isotropic a-stable Lévy process, and Ze the backward version of Lévy-type stochastic integral Ze such that [[ = 0 with finite variation.
See more detail in Theorem B.1 for the definition of Z₁ Now we propose score-based generative model with Lévy process LØ, refers to Lévy-Itö Model (LIM). In [39], the VP-SDE formula is defined
as dx, = *(t) x + (B(t)) 1 dB; and (tt) is set to a linear function (t)) = (31 36 + fo. On the other hand, we extended [39]'s s formula the SDE driven by isotropic a-stable Lévy process,
2
dX, 20 dt + (3()) à dL;, where we simulate the noise term in (3) using only L₁ This choice is motivated by the fact that, in contrast to a (t)dL,, which exhibits infinite variation, dZ, has a mean of 0 and finite variation, rendering it comparatively negligible. The specific Btt) in our paper dlog(cooe $ derived from the cosine schedule proposed in [25] and tailored to LIM as 3(t)= = -a dt for setting forward SDE with b(t, x)
9(4) x and (() 3- (1) in (1). Then the solution X, has the same law with Xo + 7()) where
SaS a(t) exp(- f 3(+) ds), and ((t) = (1 (t)) As b(t, = ((t) x satisfies the condition of Theorem 4.1, we can obtain the ceverse-time SDE of the beta-schedule version,
ax. =
(-
8(0)
3(1) S." -(1)-s(0))))))) (Xx))) (8())) aLc.
(4)
For t < s, the solution of equation (4) can be represented as an integral, utilizing the semi-linear structure of the reverse SDE,
X, a² aat)) S. d (11) a(s)
a(t)
) (x))) S (B(W))) a(t) dig.
(5)
du La(u)
a(u)

---

We can get faster sampling method by using an approximation for the second term f" g(w) s((x) (x,)du.
atw)
This term can be approximated (0= B(w) du) -/ (xx) and it is possible to calculate 1 ((u) a(t) a(u)
a(a)²
the scale parameter 7 of which follows 7 = J du ((=)))()))
Corollary 4.1 (Variant of Euler- Malluymm with dynamic time increment). Suppose the fractional score function in the SDE given by equation (4) satisfies the conditions stated in Theorem E.2, and a(t), (1) are bounded. Then, there exists Markov chain (xe) that follows:
a(t) a(s)
X + a² a(t)
1) () (x.) +
( [(()) (((s)
((19)-11)
1
(6)
a(s)
Here, (1)S for & > 1, and At = = - & As a result of the conditions being satisfied, the Wosserstein-1 distance between the invariant measures of the solution of Equation (4) and (xx) is bounded by (At)* [6]. (See more detail in Appendix E)
The paper [6] outlines a way to find bounds on the Wasserstein-1 distance by determining bounds for the approximation of the drift term, J1. and the approximation of the stochastic term, Jz. In the equation, only the bound for the approximation of the drift term, J1, is utilized. Furthermore, since the law of weak solution is same to the law of the strong solution, the Wasserstein-1 distance between the invariant measures for the strong solution and the approximation can be used for the that of the weak solution and the approximation well. In addition, we can also find probability ODE corresponding to (1).
Theorem 4.2 (Probability Fractional ODE). Let b(t,x): B + R and (z R - R be func- tions that satisfy the Lipschitz condition as stated [35]. For the SDE following (1), the solution (X of the SDE satisfies the following ODE:
dx, d
de.
Due to its semilinear structure, the solution to (7) can be represented as an integral. This is shown in Lemma D.2. The Euler method can then be used to find the solution.
Corollary 4.2 (Deterministic ODE sampling). If the drift term in (7) is Lipschitz continuous and the solution has a bounded second derivative, then a sequence (xx) can be obtained using the Euler-scheme:
X, x. + a (a())-)) -1) (x.)
a(t)
(8)
a(s)
where 8 > f. When the step size is At, the global truncation error is bounded by O(At) [5].
4.2 Estimating Fractional Score Function
Let (x) be the density of (o)5 A weak solution of the beta-schedule version is X²
X + rt)e where ~ S1) with the following transition density:
1
a(t)xo
px(x((20) (d(() dex
7(t)
We train the fractional score model si (xxxi) 0) which estimates s(a) (xx) at the optimal point. The method of approximating the fractional score function s(") (xx) can be equivalent to estimating the
fractional score function S!a) (x=|xo), which replaces the fractional score function of term Pr Pr(x.) in
(2) with that of the transition density (9).
Theorem 4.3 (Fractional Denoising Score Matching). For a parameter 0 two loss functions L (0) and L2 ()) are given as follows:
(0) = E, (ls 0) - s((0) (x:))1
(10)
L2(0) (x..0 [x<((x;; 8) s(a) (x+ xx)
(11)
Then, there exists a constant satisfying L (0) = (2) + C so that the optimization of two loss functions with respect to 0 is equivalent.

---

The drift term of the reverse-time SDEs can be accurately determined by training Si (x,;; 0) to match (/() (x, Xn), stated in Theorem 4.3. Moreover, if have the jump-type Lévy process 1.f, calculate ((a) (xx(Xo) Xo) explicitly as a linear expression.
Theorem 4.4. Let < a < 2. If the transition density function pr (xx (x)) follows (9), then the fractional score function of pe (xx(X) can be represented as linear form such that
5/ (x)(xo)
(x+(xx)
-0xxx a(t)xo)
(12)
(x(
7a-1(1)
7(t)
Theorem 4.4 implies that the computational cost of training a fractional score model s, (x,;)) similar to DSM [40].
4.3 Loss function
We set beta schedule cosine beta schedule ((t) [25] following ((t)
I+f tan
L I+
for small number 0 < 8 <<' Then, the drift term a(t) cost +)
and the diffusion term
cost T 3)
(() (1 a(t)*) a. We use the U-net architecture as in DDPM [15] for training and apply the loss (11) to train s, (x,; 8). For e ~ Sa)4(1) and 2y ~ Pdata X = a(t)x, + 7t)ee, (0, 1) denotes a uniform distribution. Then, the loss with the relative weight (t) is defined as
(()) (0,11 Eag Be-Sas"(1) ||> (4)s,
(13)
The fractional score model (x;; 0) learns the law of the solution X, for each because the beta- schedule version solution used during training the weak solution of the given forward SDEs.
5 Experiments
We employ continuous time step throughout our experiments and set NFE to a fixed value of 500 in all cases. Given that Lévy processes exhibit higher occurrence of extreme values compared to Gaussian noise, we opt for the smooth L1 Loss instead of the L2 Loss to ensure stable training. We use the modified VP-SDE to fit the Lévy process and quadratic timestep during reverse sampling.
5.1 Mode Estimation
We conduct the class imbalance experiment for mode estimation with two kinds of datasets. First customized imbalanced CIFARIO with ratio given of 1:2:3:4:5, We choose t-distributed stochastic neighbor embedding (I-SNE), which reduces complex data of high dimen- sions two dimensions. In table 1 and Fiure 5, the mode estimation of LIM much closer to training data than the diffusion model. Also, the corresponding FID, Recall, and MMD of LIM show improved per- formances. The second dataset is CIFAR IOLT includ- ing 10 classes, which follow heavy-tailed distribution
IT'
Figure 5: t-SNE and evaluation in the imbalanced CIFAR1O. LIM (bluc), DM (red), and training data (green).
Table shows that as we move from class O with a high ratio to class with a low ratio, the overall performance tends to decrease. However, LIM shows better performances compared to the diffusion model with respect to FID and Recall. These results demonstrate that LIM more robust for mode estimation in imbalanced datasets, and alleviates the mode-collapse issue.
5.1.1 Sample Diversity
"The ratio is given by 5000:2997: 1000:2997 11002999777991117777558822223399

---

Metric
Model
airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
DM
13.92
8.87
14.58
15.44
16.48
13.19
11.05
8.38
FIDJ
LIM
13.92
8.24
15.02
16.07
10.05
18.24
13.23
9.77
9.18
7.76
Recally
DM
0.6646
0.6238
0.6968
0.6076
0.6936
0.6792
0.6816
0.6734
0.6342
0.6098
LIM
0.6912
0.6516
0.7138
0.6272
0.6976
0.7212
0,7082
0,7040
0.6472
0.6366
Table 2: Results of conditional generation on CIFAR10(32 x 32).
Metric
Model
airplane
automobile
bird
deer
dog
frog
horse
ship
truck
FID
DM
22.00
12.69
30.55
37.14
22.08
49.31
38.67
44.82
50.97
36,08
LIM
15,394
9,5131
19.928
25.66
15,90
33,5393
29.46
24.88
30.34
34,302
0.5200
0.5828
0.6046
Recallf
LIM
0.6768
0.6118
0.6778
0.5696
0.6248
0.6578
0.5686
0.6100
0.5906
0.6084
Table 3: Results of conditional generation on CIFAR10LT 32).
Metric
1.M
DM
We choose Recall as evaluation metric for sample di- versity. In Table 5, we can see that LIM exceeds other diffusion models in Recall. Moreover, even in the condi- tional generation, shows higher Recall for each class, proving that the generated samples are more diverse than those of the diffusion model (Table 2).
FID
21.07
62.62
Recallf
0.5549
0.5002
MMD
0.00416
0.01396
Table Imbalanced CIFARIO results.
Imputation The imputation refers to the task restoring the masked regions an image. To measure the diversity of samples, we use Learned Perceptual Image Patch Similarity (LPIPS) as metric, where higher LPIPS indicates greater diversity and a lower value implies more similarity between samples. For each original image I, we generate 10 imputed images { | and calculate the average of LPIPS for the 10 samples. As shown in Figure 6, the diffusion model tends to generate similar eyes and mouths in the masked region, while LIM produces a much wider range diverse images and shows higher LPIPS.
origin image
2.0
(DM)
(LIM)
mesked image
imputation result
Figure 6: Imputaion images on CelebA-HQ dataset. Diffusion model(DM) generates only similar eyes and lips, while LIM generates a much more diverse range of shape and color of eyes and lips. LPIPS of DM is 0.0132, but LIM 0.0197.
5.2 Convergence rate for NFE
We compare LIM and diffusion models CIFAR10 (32 32) and CelebA (64 x 64) dataset in terms of convergence rate for NFE. Figure shows that FID score LIM drops at earlier NFE than the baselines SDE-bbsed sampling. Moreover, it shows a competitive convergence rate for NFE compared to ODE-based models. These experimental results demonstrate that LIM is capable of converging to the sample space faster than the baselines. See Appendix G.6 to figure out the wall clock time per NEE on various datasets.

---

NCSN++ (clee))
DDPM Analytic-DDIM LIM
Analytic.DDIM
10:0
FID score
LIM
FID score
10.0
5.0
2.0
200
1000
NFE
(a) CIFAR10
(b) CelebA 64x64
Figure Convergence rate for NFE measured by FID score on CIFARIO and CelebA.
5.3
Sample Quality
Table 5 displays the performance of LIM in comparison previous approaches for unconditional image generation on CIFARIO (32 x 32) and CelebA (64 x 64). measure FID scores and Recall using reverse sampling and generate 50k samples for all datasets. For CelebA, we randomly select 50k samples from the training dataset with 5 iterations. We evaluate NLL using uniformly dequantized test dataset and calculate the log-likelihood through the average of iterations using the ODE solver [39]. Since it is difficult numerically calculate the density function of the isotropic a-stable distribution higher dimensions (d > 100) [26]. we approximate the density function with non-isotropic o-stable distributions to evaluate the NLL of LIM. For CIFARIO 10, we train 2 backbone
models: DDPM [15] and (dee++(deep) [39]. See Appendix G for more detail.
The performance of LIM can differ with different a. In- deed, this seems intuitively natural since images with resolution less prefer large jumps, while images with high resolution and multiple modalities can benefit explo- ration from large jumps. Table provides comparison of LIM's performance across different a values. Although the overall tendency of declining FID with lower values remains consistent, appears that differences in FID it- self arises when using different architectures
a
CIFARIO
CelebA
1.2
5.15
2.99
1.5
2.86
1.57
1.8
2.44
3.21
Table 4: a selection on CIFARIO
(32x32) and CelebA (64x64).
CIFARIO (32x32). In spite of using half of NFE used [39]. LIM shows competitive FID score, Recall, and NLL. Especially CelebA (64 x64), LIM achieves FID score of 1.57 and Recall of 0.7007. It is an impressive performance since we do use any other guidance or improved model architecture. We also conducted performance comparisons between diffusion model [39] and LIM using the ADM architecture [9] on ImageNet 64)x64). Diffusion model [39] achieves FID of 14.23, Precision of 0.6711, and Recall 0.6932. In constrast, LIM achieves a FID of 12.97, Precision 0.6782, and Recall 0.6782, For comparing the performance high-resolution dataset, we chose the DDPM architecture [15], and trained LIM and diffusion model [39] on the CelebA- -H dataset (256 x 256). This indicates that when it comes Cel LIM attains an FID score of 7.76 with 500 NFE, whereas the diffusion model achieves a higher FID of 1 11.87 with 1000 NFE based DDPM architecture [15].
5.4 Deterministic ODE sampling
NFE
20
50
100
200
have the option utilize DDIM inference for our model. When we pretrained model on CelebA (64x64), Table 6 displays the FID scores acquired by DDIM and LIM-DDIM for various NFEs. The results demonstrate that LIM-DDIM outperforms DDIM terms of performance.
DDIM
6.64
5.23
4.78
LIM-DDIM
6.73
4.80
3.95
Table 6: DDIM and LIM-DDIM with 1.5 on CelebA (64x64).

---

Figure 8: Generated samples on CelebA-HQ 256 256, LSUN-Church 256 256, and L.UU-
Bedroom 256 x 256.
Model
Noise
NFE
FID.
Recall
(NLL(bits/dim))
CIFARIO (32 x 32)
DDPM [15]
Gaussian
1000
3.17
3.70
DDPM cont. (VP) [39]
Gaussian
1000
3.24
0.6784
3.21
NCSN+ cont. (deep, VP) [39]
Gaussian
1000
2.44
0.6860
3.13
LIM-DDPM cont. (Ours)
Lévy (a=1.8)
500
3.37
0.6874
3.08
LIM-NCSN+ cont. (deep, Ours)
Lévy (0=1.8)
500
2.44
0.6960
2.97
CelebA (64 x 64)
DDPM [15]
Gaussian
1000
3.50
DDPM cont. (VP) [39]
Gaussian
1000
3.21
0.6437
2.39
LIM-DDPM cont. (Ours)
Lévy (a=1.5)
500
1.57
0.7007
2.23
Table 5: Unconditional generation on CIFARIO (32 32) and CelebA (64x64), iNLL is approximated by using non-isotropic a-stable distribution prior density.
6 Limitation
Many diffusion models have been conducted to find the best model architectures such as NCSN+ [39] and ADM [9]. In contrast, Lévy process differs significantly from Brownian motion. Therefore, it is uncertain whether using the same model architecture that has been used previously is the best approach for LIM. Although, this study, to compare the pure effect of the Lévy process, there is a need to find model architecture that suitable for the Lévy process in order to improve performance further.
the same structure of the existing diffusion model
7
Conclusion
The aim of this study is expand the variety of noise distributions utilized in score-based generative models by introducing an exact rime-reversal formula for SDEs with Lévy processes. We propose a new score- score-based generative model, Lévy-Itó Model (LIM). The effectiveness of this model has been verified by empirical results, which show that LIM performs well with various image dataset. result, this study provides a solution and demonstrates the potential for utilizing wider range of non- Gaussaa Markov processes in score-based generative models.
10

---

Acknowledgments and Disclosure of Funding
This work supported by LG AT Research. This work was also supported by Institute of Infor- mation & communications Technology Planning Evaluation(ITPP grant funded by Korean governmentMMSIT)NG 2022-0-00612, Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AI: No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University), National Research Foundation of Korean(NRF) funded by the Korean ICIC100925)) and grant from Korea University (K2304791). This work also was supported by Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) and Gwangju Metropolitan City. The authors especially thank Prof. Nicolas Privault for his interest in our paper and valuable discussions that helped us correct errors in our main result. The first author would like to thank Woo-Joo Kim for insightful discussion and anonymous reviewers for their constructive discussion and editorial efforts.
(ivermmen((SSTT)2222222

---

References
[1] M. Abramowitz and A. Stegun. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Dover Publications, 1964.
[2] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, pages 313-326, 1982. URL https: /twp///wwwwaaa///// www.ecedireecc.......... wtt.ccooccccieeee science/ atticlee/phs$/0304144999900
[3] David Applebaum. Lévy processes and stochastic calculus. Cambridge university press, 2009.
[4] Björ Böttcher. Feller evolution systems: Generators and approximation. Stochastics and Dynamics, 14(03):1350025, 2014.
[5] J. C. Butcher. Numerical Differential Equation Methods. John Wiley and Sons, Ltd, 2016. URL ht https://oniineeibrarr
com/ http//////111111/22///1111111111 ch2.
[6] Peng Chen, Changsong Deng, Rene Schilling, and Lihu Xu. Approximation of the invariant measure of stable sdes by an euler-maruyama scheme, 2022. URL ctttps//arxiv..eet
abs/2205. 221///2222........
[7] Giovanni Conforti and Christian Léonard. Time reversal of markov processes with jumps under finite entropy condition. Stochastic Processes and their Applications, 2021. doi:
10.48550/ARXIV.21000 109488 URL httph:tt///at /arrii////aivvv//a/ http:///waxxx...aaaaaaaa/e/ argg/bs///11 10948.
[8] Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching, 2021.
[9] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis, 2021.
[10] John R Dormand and Peter Prince. A family embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26 1980.
[11] A. Dytso, Bustin, and et al. Analytical properties of generalized gaussian distributions.. 2018.
[12] Gerald Folland. Real Analysis: Modern Techniques and Their Applications. John Wiley and Sons, 2013.
[13] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy, 2023.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Güiter Klambauer, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to nash equilibrium. arXiv preprint 1rXiv:1706.0883 08500, 12(1), 2017.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 6840-6851, Curran Associates, Inc., 2020. URL https: htttps:/pproceedings wwweoiinnnnnnniiss.n.eee. cccpapprr/222/ffi10 4 Paper. pdf.
[16] Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in 1076, 1989.
and Computation, 18(3):1059-
[17] Jean Jacod and Philip Protter. Time Reversal on Levy Processes. The Annals of Probability, 16 (2):620 641, 1988. doi: 10. 10.1214/aop/| 17699 URL https://doi. org/ /00 1214/aop/ 1176991776.
[18] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models, 2023.
[19] Thomas G. Kurtz. Equivalence of stochastic equations and martingale problems. Stochastic Analysis 2010, 2011.

---

[20] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models, Advances in Neural Information Processing Systems, 32, 2019.
[21] Paul Lévy. Sur les intégrales dont les éléments sont des variables aléatoires indépendantes. Annali della Scuola Normale Superiore di Pisa-Classe di Scienze, 3(3-4):337-366, 1934.
[22] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao, Pseudo numerical methods for diffusion models manifolds, 2022.
[23] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. 2022.
[24]
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models, 2021.
[25] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021.
URL https:t/t/////t //apps/////xvvv.///// http://wwwww..gg..aaa//// wwtg//////////eeoe /bsp/a/....999999
[26] John Nolan. Multivariate elliptically contoured stable distributions: Theory and estimation.
Computational Statistics, 28, 10 2013. doi: 10.1007/s00180 80-003-0396-1111
[27] Manuel Ortigueira, Taous-Meriem Laleg-Kirati, and J A Tenreiro Machado. Riesz potential versus fractional laplacian. Journal of Statistical Mechanics: Theory and Experiment, 2014.
[28] Antonis Papapantoleon. An introduction to lévy processes with applications in finance, 2008.
[29] William Peebles and Saining Xic. Scalable diffusion models with transformers, 2023.
[30] Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang. Baas-balancing diff'usion models, 2023.
[31] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241 Springer, 2015.
[32] Ludger Rueschendorf, Alexander Schnurr, and Viktor Wolf. Comparison of time- inhomogeneous markov processes, 2015. URL https: http///axxvvv/// wwtps/.rrg/bbb//baeeeee 02925.
[33] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.
[34] Tim Salimans and Durk P Kingma. Weight normalization: simple reparameterization
accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
[35] Schertzer, M. Larchevê que, J. Duan, V. V. Yanovsky, and S. Lovejoy. Fractional fokker-planck equation for nonlinear stochastic differential equations driven by non non-gaussian lévy stable noises. Journal of Mathematical Physics, 42(1):200-212, jan 2001. doi: 10. 10.1063/1.1118734,
[36] Jinghai Shao. Weak convergence of muler-maruyama's approximation for sdes under integrability condition, 2018. URL https: httpp:://aaxiv.orrt h/tp///ba///a 1108..8080000 07250.
[37] Umut Şimşekli. Fractional langevin monte carlo: Exploring lévy driven stochastic differential equations for markov chain monte carlo. In International Conference on Machine Learning, pages 3200-3209. PMLR, 2017.
[38] John Skilling. Maximum Entropy and Bayesian Methods: Cambridge, England, 1988, volume 36.
Springer Science & Business Media, 2013.
[39] Yang Song, Jascha Sohk-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.

---

[40] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23:1661-1674 2011.
[41] Yingjian Wang and Lawrence Carin. Levy measure decompositions for the beta and gamma processes, 2012.
[42]
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. 2023.
[43] Umut Şimşekli. Fractional langevin monte carlo: Exploring lévy driven stochastic differential equations for markov chain monte carlo, 2017. URL httttt://aariii org/ aaa///////// /ttb://10..000 03649.


Reviews:
[{"review_id": "CvTi2rSyS9", "reviewer": "Reviewer_xP7H", "summary": "This paper presents a new score-based generative model called Lévy-Ito ̄ Model (LIM) that tackles the challenges of slow convergence rate of Number of Function Evaluation (NFE) and mode collapse in diffusion models when applied to imbalanced data. This model leverages isotropic-stable Lévy processes. Initially, the authors derive a precise stochastic differential equation in reverse-time, driven by the Lévy process, and subsequently establish the corresponding fractional denoising score matching technique. The proposed generative model harnesses the advantageous heavy-tailed characteristics of the Lévy process. The experimental findings demonstrate that LIM achieves faster and more diverse sampling, while maintaining exceptional fidelity when compared to existing diffusion models across a range of image datasets.", "questions": "Please refer to Weaknesses", "limitations": "Yes, the authors adequately discussed the limitations of this paper. However, potential negative societal impacts are not discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper investigates a novel optimal non-Gaussian stochastic process(isotropic alpha-stable Lévy processes) for injecting noise, and prove the exact time-reversal formula of SDEs driven by Lévy process.\n2. Based on isotropic alpha-stable Lévy processes， this paper proposes a novel score-based diffusion model called Lévy-Ito ̄ Model (LIM).\n3. Compared to existing diffusion models, LIM offers faster and more diverse sampling capabilities while maintaining high fidelity across a range of image datasets.", "weaknesses": "1. In P2-L31, the authors make claims about the slow convergence rate and mode-collapse issues of previous score-based diffusion models without providing sufficient explanation or evidence to support these claims. \n2. In Figure 3, the authors doesn't explain the significance and differences of the two blue clusters in each of the three different subplots. And how is the FID calculated?\n3. The related work section provides limited information and lacks a detailed investigation of the recent developments in the last two years.\n4. In P5-L156, there are grammatical errors, such as repetitive words and incorrect prepositions. These should be corrected for clarity and readability.\n5. Indeed, the experiments seem somewhat limited as they only validate the results on low-resolution datasets such as CIFAR-10 (32x32) and CalebA (64x64). \n6. In Appendix C, on P23, Equation 112 is identified as incorrect. The authors should revise the equation or provide the correct version to ensure the accuracy of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback on our paper. We have done our best to answer your keen questions.\n\n> Weakness 1\n> \n\n<Slow convergence>\n\nThe reason for the slow convergence of Diffusion models is that the Brownian motion follows a light-tailed distribution and has a continuous path. Various methods have been proposed to reduce the convergence rate while maintaining a fast sampling process and high fidelity [Liu et al. 2022]. Another efforts have also been made to use distillation to reduce the number of steps [Salimans et al., 2022].\n\n<Mode-collapse issue>\n\nIt has been pointed out that Diffusion models show significant degradation in terms of fidelity and diversity when dealing with imbalanced datasets where the number of samples per class varies [Qin et al., 2023] with mode collapse issue for tail classes.\n\n> Weakness 2\n> \n\nTwo mixtures of Gaussian distribution is the simplest form of an imbalanced dataset. In subplot (b), it can be seen that for the Diffusion model, the generated samples have a ratio of 5.6:1. However for LIM, the generated samples have a ratio of 11.1:1, which is relatively close to the ground truth. This similarity can be observed not only from the ratio aspect but also from FID and MMD. The FID and MMD values for each model are summarized in the table below.\n\n|  | Diffusion models | LIM |\n| --- | --- | --- |\n| FID$ (\\downarrow )$ | 8.312 $\\pm$ 0.904 | $\\textbf{0.663 $\\pm$ 0.376}$ |\n| MMD ($\\downarrow $) | 0.026 $\\pm$ 0.003 | $\\textbf{0.02 $\\pm$ 0.002}$ |\n\nFID and MMD are calculated directly without using any embeddings for the sets of true samples and generated samples. LIM demonstrates that the distribution $p_{\\theta}$ of its generated samples is similar to the ground truth distribution $p_{\\text{data}}$. We will incorporate these insights and make the necessary revisions accordingly.\n\n> Weakness 3\n> \n\nDiffusion models have shown advancements in performance through various approaches such as incorporating guidance [Kim et al., 2023][Song et al., 2021], introducing new architectures [Peebles et al., 2023], and proposing novel training methods [Hang et al., 2023]. Additionally, various attempts to reduce convergence rates have been proposed, such as using ODE solvers or [Lu et al., 2022], Fourier neural Operator [Zheng et al., 2022], and distillation [Salimans et al., 2022]. Despite these advancements, diffusion models still face inherent limitations such as slow convergence rates and the mode-collapse issue in imbalanced datasets [Qin et al., 2023]. \n\nThere are a few papers that have explored the use of noises other than Brownian motion, such as Denoising Diffusion Gamma models [Nachmani et al., 2021] and Heavy-tailed DSM [Deasy et al., 2021]. Both approaches share the common aspect of utilizing DDPM formulas and heavy-tailed distributions. Heavy-tailed DSM employs a Generalized Gaussian distribution and claims to have strengths for imbalanced tasks. Denoising Diffusion Gamma models use a Gamma distribution for noise injection and highlight the advantage of faster convergence rates for sampling. However, they face challenges in terms of performance compared to standard Diffusion models. We will add this information and proceed to modify the related work section accordingly.\n\n> Weakness 4\n> \n\nWe appreciate your feedback regarding the grammatical errors present throughout the paper. We will ensure to correct all of them and address the factors that could potentially compromise the readability of the text.\n\n> Weakness 5\n> \n\nTo compare the performance on a high-resolution dataset, we chose the DDPM architecture [Ho et al., 2020], and trained LIM and diffusion models [Song et al., 2020] on the CelebA-HQ dataset (256x256). We measured and compared the FID score for each model. \n\n| Model  | FID |\n| --- | --- |\n| Diffusion model [Song et al] (NFE = 1000) | 11.87 |\n| LIM (NFE = 500) | $\\textbf{7.76}$ |\n\nIt shows that LIM outperforms Diffusion models for CelebA-HQ. However, since this experiment was limited to the DDPM architecture and conducted only on CelebA-HQ, we plan to conduct additional experiments in the next paper revision.\n\n> Weakness 6\n> \n\nThank you for pointing out the inconsistencies in our proofs. We will correct the error in Equation 112 and carefully revise all other formulas to make sure everything is correct. \n\n---\n\n[Deasy et al., 2021] Heavy-tailed denoising score matching\n\n[Hang et al., 2023] Efficient Diffusion Training via Min-SNR Weighting Strategy (2023) \n\n[Ho et al., 2020] Denoising Diffusion Probabilistic Models (NeurIPS 2020)\n\n[Kim et al., 2023] Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models (ICML 2023) \n\n[Liu et al. 2022] Pseudo numerical methods for diffusion models on manifolds (2022)\n\n[Lu et al., 2022] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (NeurIPS 2022) \n\n[Nachmani et al., 2021] Denoising Diffusion Gamma Models (2021)\n\n[Qin et al., 2023] Class-Balancing Diffusion Models (CVPR 2023)\n\n[Salimans et al., 2022] Progressive distillation for fast sampling of diffusion models (ICLR 2022)\n\n[Song et al., 2020] Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2020)\n\n[Song et al., 2021] Denoising diffusion implicit models (ICLR 2021) \n\n[Peebles et al., 2023] Scalable Diffusion Models with Transformers (2023)\n\n[*Zheng* et al, 2022] Fast Sampling of Diffusion Models via Operator Learning (NeurIPS 2022)"}, {"review_id": "BOCOFCA1PO", "reviewer": "Reviewer_9rZc", "summary": "Prior score-based/diffusion generative models have been defined by Brownian motion. This paper proposes a method of replacing the continuous Gaussian processes with different processes dependent on the characteristic exponent value. The heavy tail property defined by the Levy process allows for higher chance of making larger steps, thus inducing more diverse and faithful sample of the true distribution of the model than its Weiner process counterpart. \n\nThe author shows that the model converges faster due to the Levy process having the capability of making large jumps during the forward and reverse process. The resulting model displays promising qualitative and quantitative results in datasets such as CIFAR10 and CelebA and synthetic datasets.", "questions": "- Seen in Figure 3, why are diffusion models bad at faithfully learning the true data distribution? It would be nice if there were more insight on the lackluster performance of diffusion models based on the Weiner process interpretation. Theoretically, it seems like it should perform just as well as the Levy process, could it be due to the number of steps taken by the ODE solver?\n- Can the score function learned from the Levy noising process be compatible with other models such as using this learned score function for DDIM inferencing?\n-  Why doesn't the heavy tail distribution create instability to the diffusion model path? Line 53-54 states that the variation becomes smaller as it approaches the sample space but wouldn't this also have adverse affect for finer pixel detailing due to the heavy tail? (I have limited knowledge of Levy process, so I may just have misunderstood). (I am willing to raise up my score)", "limitations": "Adequately addresses all limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Overall, the paper is well written/organized with clear explanation on the theoretical foundation.\n- Experiments are very well thought out and very useful in showing the advantages of utilizing a Levy stable distribution.\n- Image metrics show extremely promising results (i.e., FID score).", "weaknesses": "- Table 5 in the appendix shows promising FID scores but are heavily dependent on the optimal alpha value. It would be nice if we could get all alpha FID results for Table 1 and not just the optimal alpha value FID score. This would help us understand whether Levy process performs generally better (i.e., when alpha is between 1 and 2) than the baseline diffusion model (i.e., alpha equal 2) or if this seems more like a sensitive hyperparameter for best results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1\n> \n\nThank you for your valuable feedback. We conducted additional experiments regarding $\\alpha$-selection for CIFAR10, and CelebA. Below are the FID results based on different values of $\\alpha$ for CIFAR10 and CelebA:\n\n| $\\alpha$ | CIFAR10 (32x32) | CelebA (64x64) |\n| --- | --- | --- |\n| 1.2 | 5.15 | 2.99 |\n| 1.5 | 2.86 | 1.57 |\n| 1.8 | 2.44 | 2.85 |\n| Diffusion Model [Song et al, 2020] | 2.44  | 3.21 |\n\nThe proposed method's performance may differ with different $\\alpha$, which means that the best $\\alpha$ can differ from data or resolution. Indeed, this seems intuitively natural since images with low resolution less prefer large jumps, while images with high resolution and multiple modalities can benefit exploration from large jumps. Within similar resolution image datasets, there should be a reasonable range of $\\alpha$, and here it is found to be between 1.5 and 1.8.\n\nTable 5 uses the DDPM architecture [Ho et al., 2020], whereas above table employs the NCSN++ deep architecture [Song et al., 2020]. Although the overall tendency of declining FID with lower $\\alpha$ values remains consistent, it appears that differences in FID itself arises when using different architectures on CIFAR10.\n\n> Question 1\n> \n\nThank you for your insightful comments. \n\nThe Wiener process follows a light-tailed distribution and has a continuous path, resulting in slow convergence rates. Consequently, smaller step sizes are necessary to reach the correct sample, requiring more steps in total. Diffusion models can converge theoretically to the data distribution $p_{\\text{data}}$ when a sufficiently large number of steps are taken. However, during reverse sampling, instead of using the score function directly, a score model is used to approximate it. Theoretically, the quality of the score model and NFE determine the quality of the generated samples. However diffusion models use the score model for reverse sampling, it struggles with accurate mode estimation due to limited exploration range of noises compared to LIM during training the score model. You can observe this tendency in Figure 3.\n\nSuccessful mode estimation can be achieved by training the score models to capture $p(c)$ for each class $c$ of $p_{\\text{data}}$. To do this, the (fractional) score function of the perturbed distribution $p_t(\\mathbf{x})$ should be learned. The ability of the forward process to explore the sample space effectively determines the score model's capacity for mode estimation as the ratio between the true distribution $p_{\\text{data}}(c)$ and the model's predicted distribution $p_{\\theta}(c)$ for each mode is proportional to $\\frac{p_{\\theta}(c)}{p_{\\text{data}}(c)} \\propto \\frac{\\int_{\\mathbf{x}\\in\\mathbb{R}^d} p_{\\theta}(\\mathbf{x},c) d\\mathbf{x}}{\\int_{\\mathbf{x}\\in\\mathbb{R}^d}p_{\\text{data}}(\\mathbf{x},c)d\\mathbf{x}}$ [Qin et al., 2023].\n\nIn a balanced dataset, where  $p_{\\theta}(\\mathbf{x},c)$ is learned similarly for each $c$, the convergence of $\\frac{p_{\\theta}(c)}{p_{\\text{data}}(c)}$ to 1 is easier to obtain. However, in cases of imbalanced datasets, the exploration range of the noised data due to the properties of Brownian motion limits the frequency of learning for the minor class $c$. In contrast, the heavy-tailed and discontinuous Lévy process has a wider exploration range than Brownian motion, particularly benefiting the accurate learning of $p_{\\theta}(\\mathbf{x},c)$ for the minor class (Figure 4).\n\n> Question 2\n> \n\nThank you for the insightful question. We can indeed apply DDIM inference to our model. This is possible because we have derived the probability ODE represented by the fractional score function (Theorem C.1). By applying the Euler method to (Theorem C.1), we arrive at a sampling formula with a structure similar to the fractional score function form of DDIM inference (Corollary E.2). When employing a pre-trained model on the CelebA dataset, the obtained FID scores by LIM-DDIM according to different NFEs are presented in the below table.\n\n| NFE (FID) | 20 | 50 | 100 | 200 | 1000 |\n| --- | --- | --- | --- | --- | --- |\n| LIM-DDIM ($\\alpha =1.5$) | 6.73 | 4.80 | $\\textbf{3.95}$ | - | - |\n| DDIM | 6.64 | 5.23 | - | 4.78 | 4.88 |\n\n> Question 3\n> \n\nThe extended version of the Forward process for the given $\\alpha$ in the general VP-SDE framework is given as follows:\n\n$dX_t = -\\frac{\\beta(t)}{\\alpha}X_tdt + (\\beta(t))^{\\frac{1}{\\alpha}}dL^{\\alpha}_t$ -(1)\n\nHere, the weak solution $X_t$ of (1) is given as $X_t = a(t)X_0 +\\gamma(t)\\epsilon$, where $\\epsilon \\sim \\mathcal{S}\\alpha\\mathcal{S}(1)$.\n\nFor this, $a(t)$ is set to the cosine schedule $a(t) = \\cos(\\frac{\\pi}{2}t)$ independent of $\\alpha$. According to the VP-SDE structure, $\\tilde\\gamma_{\\alpha}(t)$ becomes $\\tilde\\gamma_{\\alpha}(t) = (1-a^{\\alpha}(t))^{\\frac{1}{\\alpha}}$. Therefore, when $\\alpha_1<\\alpha_2$, it holds that $\\tilde\\gamma_{\\alpha_1}(t) \\le \\tilde\\gamma_{\\alpha_2}(t)$. In other words, LIM has sufficient ability to move from the sample space to the noise space in spite of small noise coefficient $\\tilde\\gamma_\\alpha(t)=(1-a^{\\alpha}(t))^{\\frac{1}{\\alpha}}$ due to large jump noises. \n\nIf we fix $\\gamma(t)$ for each $\\alpha$ and adjust $\\tilde a_{\\alpha}(t)$ to correspondingly follow the VP-SDE structure in the forward process such that $\\tilde a_{\\alpha}(t)=(1-\\gamma^{\\alpha}(t))^{\\frac{1}{\\alpha}}$, then for $\\alpha_1<\\alpha_2$, it holds that $\\tilde a_{\\alpha_1}(t) \\le \\tilde a_{\\alpha_2}(t)$.\n\nTherefore, as $\\alpha$ decreases, a gradual degradation effect occurs for the mean corresponding to the forward process $X_t$ with $a(t)X_0$*.* Regardless of whether we keep $a(t)$ fixed or fix $\\gamma(t)$ for various $\\alpha$ values, LIM can reach the noise space.\n\n---\n\n[Ho et al., 2020] Denoising Diffusion Probabilistic Models (NeurIPS 2020)\n\n[Qin et al., 2023] Class-Balancing Diffusion Models (CVPR 2023)\n\n[Song et al., 2020] Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2020)"}, {"review_id": "98KLqP8elW", "reviewer": "Reviewer_QCLF", "summary": "The paper introduces the Levy-Ito Model, a novel score-based generative model (SBGM) that utilizes the isotropic $\\alpha$-Levy process as perturbation noise. The authors highlight that their proposed method is the first continuous-time SBGM to incorporate a heavy-tailed process. They aim to leverage the advantages of heavy-tailed processes observed in various cases. For instance, such processes enhance convergence speed in Markov chain Monte Carlo (MCMC) methods like the Langevin algorithm. Additionally, heavy-tailed processes offer faster mode-hopping behavior compared to Gaussians.\n\nFirst, the authors establish the exact time-reversed stochastic differential equations (SDEs) with Levy process perturbation noise. The authors emphasize that unlike the Wiener process, whose sample paths should be continuous, the sample paths of the Levy process may be discontinuous (jumps) at multiple locations. Consequently, common differential equation techniques may not be applicable. Addressing several technical challenges associated with this SDE formulation, the paper introduces the \"fractional score function,\" replacing the conventional score function in the drift term of the reverse process.\n\nFurthermore, the paper proposes fractional denoising score matching (fractional DSM) to approximate the fractional score in the reverse process. This approach may serve as a more general version of DSM.\n\nAdditionally, the authors derive a probability flow formulation for the reverse SDE, similar to conventional SBGMs. This demonstrates that the proposed method can leverage other fast sampling techniques developed for SBGMs, such as advanced integrators.\n\nFinally, the paper presents several experiments to showcase the proposed methods' effectiveness. For instance, mode estimations and sample diversities are evaluated to compare the proposed method to the previous Wiener process-based approaches.", "questions": "- It needs to be clarified what evidence supports the claim that conventional SBGM exhibits mode collapsing, as mentioned in lines 28-32 of the paper.\n- In lines 213-214, it is stated that noise scheduling was performed using the cosine function. However, later in the paper, it is mentioned that VP (presumably referring to a different noise scheduling method) was used. This inconsistency needs to be clarified about the accurate statement.\n- Section 5.2 should precede Section 5.1 to explain better the arguments the paper aims to address. By presenting Section 5.2 before Section 5.1, the authors can provide a clearer context and explanation of the differences they intend to discuss. This reordering would enhance the coherence and logical flow of the paper.\n- The positioning of the images in Figure 5 and the corresponding columns in the table appears to be reversed, leading to confusion. The inconsistency between the image positions and table columns can create difficulties understanding and interpreting the data.\n- There is a mismatch between the order in Tables 2 & 3 and their actual sequence in the document. This inconsistency can cause confusion and make it challenging for readers to locate and reference the correct tables.\n- Regarding the differences presented in Tables 2 & 3, it is unclear whether they can be considered statistically significant. 1. Further statistical tests or discussions of the observed variations are necessary to determine the significance of the findings.", "limitations": "Please refer to the comments provided in the weaknesses section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. The authors have succeeded in providing a fresh perspective on the topic, shedding new light on the subject matter and offering valuable contributions to the machine learning communities.", "weaknesses": "In light of the overall quality of the paper, the experimental parts would benefit from further refinement.\n\nFirst of all, the discussion on convergence rate can be improved. For example, regarding the results in Figure 7, the convergence rate may be influenced by various aspects, such as network architectures, noise scheduling, or the quality of the trained models. In this aspect, analysis on toy datasets would be more beneficial to provide better evidence.\n\nSecondly, there is room for improvement in the experiments related to Figure 3. It is important to note that FID utilizes classifiers trained on ImageNet datasets. This raises questions about the suitability of FID in analyzing the results for mixtures of Gaussians. I believe a distance metric like MMD may be more appropriate for this purpose.\n\nLastly, there should be more in-depth discussions about the experiments on image generation benchmark datasets. In Table 1, the transition from Wiener to Levy process in the CIFAR-10 results shows minimal improvement in \"recall.\" However, considering that other values exhibit more significant variations due to network architecture, it becomes necessary to determine the significance of this difference. It is also worth exploring whether the observed differences could be attributed to variations in noise scheduling. Similar patterns also emerge in the results for the CelebA dataset, warranting further analysis and discussion.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weaknesses\n> \n\nWe sincerely appreciate your detailed suggestions for possible improvements. As you suggested, we will include detailed experimental results on how the convergence rate varies based on 1) architecture and 2) noise scheduling in the next paper revision. While we aimed to investigate the impact of 1) and 2) using the toy dataset, the limited dimensionality of the dataset hindered us from obtaining significant differences. Therefore, due to time and resource limitations, we regret that we could not provide detailed experimental results with real datasets in this rebuttal. However, we would be more than happy to provide supplementary explanations regarding Figure 3.\n\nThe FID in Figure 3 is calculated directly from FID formula without utilizing the embeddings of a specific model. Additional measurements for MMD are also conducted and MMD are measured directly from the samples. Below is a table summarizing the values of FID and MMD measured using Diffusion models and LIM:\n\n|  |  Diffusion models | LIM |\n| --- | --- | --- |\n| FID ($\\downarrow $) | 8.312 $\\pm$ 0.904 | $\\textbf{0.663 $\\pm$ 0.376}$ |\n| MMD ($\\downarrow $)  | 0.026 $\\pm$ 0.003 | $\\textbf{0.02 $\\pm$ 0.002}$ |\n\nIt can be observed that the distribution of generated samples $p_{\\theta}$ from LIM is more closer to the ground truth distribution $p_{\\text{data}}$.  We will incorporate these details during the paper revision period.\n\n> Question 1\n> \n\n<Mode-collapse issue>\n\nMode-collapse issue has been pointed out that Diffusion models show significant degradation in terms of fidelity and diversity when dealing with imbalanced datasets where the number of samples per class varies [Qin et al., 2023]. Especially, this issue intensifies for tail classes.\n\n> Question 2\n> \n\nWe appreciate your feedback on the terminology that might cause confusion in our paper.\n\nIn [Song et al., 2020], the VP-SDE formula is defined as:\n\n$dX_t = -\\frac{\\beta(t)}{2}X_t dt +(\\beta(t))^{\\frac{1}{2}}dB_t$\n\nand in that paper, $\\beta(t)$ is specifically set to a linear function $\\beta(t) = (\\beta_1-\\beta_0)t+\\beta_0$. On the other hand, we extended [Song et al., 2020]'s formula to an isotropic $\\alpha$-stable Lévy process as:\n\n$dX_t = -\\frac{\\beta(t)}{\\alpha}X_t dt +(\\beta(t))^{\\frac{1}{\\alpha}}dL^{\\alpha}_t$\n\nThe specific $\\beta(t)$ in our paper is derived from the cosine schedule proposed in [Nichol et al., 2021] and tailored to LIM as $\\beta(t) = -\\alpha\\frac{d \\log(\\cos(\\frac{\\pi t}{2}))}{dt}$. We will make sure to provide a clearer version in the revised manuscript.\n\n> Question 3\n> \n\nWe sincerely appreciate for your thorough review on our paper. Highlighting motivation task for LIM before introducing sample quality, indeed appears far more reasonable. We believe that your editorial efforts provide us guidance toward a better direction in which we are able to make improvements in terms of both the readability and persuasiveness of the paper. In revision period, we will incorporate the feedback you provided and proceed to revise the paper.\n\n> Question 4\n> \n\nAs you mentioned, the locations of the image and table in Figure 5 have been leading confusion. Therefore, we have rearranged their locations to make Figure 5 more understandable. Additionally, we will provide additional detailed explanations for Figure 5 to further enhance the quality of the paper. We sincerely appreciate for your valuable insights.\n\n> Question 5\n> \n\nThank you for your valuable feedback. We will reposition Table 2 and Table 3 as you recommended to improve readability and clarity.\n\n> Question 6\n> \n\nWe sincerely appreciate your keen insights and comments on our paper. We have done our best to answer your valuable questions. It seems that there might have been a lack of experiments to demonstrate LIM's superiority in mode estimation aspects for multi-modal datasets, which is especially evident in Tables 2 and 3. Unfortunately, due to limited resource and time, we were unable to incorporate the feedback into the current rebuttal with actual results. Again we greatly apologize for this limitation.\n\nHowever, in order to validate the hypothesis that LIM can generate high-fidelity and diverse samples for imbalanced datasets, we are committed to conducting further experiments as soon as possible. We will conduct additional experiments and measure additional metrics to supplement the experimental results, and addressing your valuable feedback.\n\n---\n\n[Chen et al., 2022] Approximation of the invariant measure of stable SDEs by an euler–maruyama scheme (2022)\n\n[Nichol et al., 2021] Improved denoising diffusion probabilistic models, 2021\n\n[Qin et al., 2023] Class-Balancing Diffusion Models (CVPR 2023)\n\n[Song et al., 2020] Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2020)"}, {"review_id": "q3diwTIUjh", "reviewer": "Reviewer_Z4Ey", "summary": "Score-based generative models (SBGMs) generally employ Brownian motion, also known as the Wiener process, for noise injection. However, using Brownian motion in SBGMs often leads to issues such as mode collapse or slow sampling. To address these problems, the authors propose SBGMs with an isotropic α-stable Levy distribution named Levy-Ito Model (LIM). The α-stable Levy distribution exhibits a heavy-tail property, enabling the Levy-Ito Model to achieve improved mode estimation, sample diversity, and faster convergence in terms of neural function evaluations (NFEs) compared to SBGMs employing Brownian motion. For the first time, this paper proves the time-reversal formula of stochastic differential equations (SDEs) with Levy processes. It also establishes the sampling equation and presents a numerical solver known as v-Euler-Maruyama for LIM. Furthermore, the paper introduces fractional denoising score matching, which elucidates the training process of LIM. Empirically, the proposed method shows superior mode estimation, diversity in image generation and imputation, and faster convergence of LIM compared to previous SBGMs.", "questions": "The primary advantages of LIM appear to arise from employing heavy-tailed noise injections, an approach already investigated in prior works. What are the specific benefits of utilizing an isotropic α-stable distribution compared to other potential heavy-tailed distributions?\n\nTraining and sampling from the ImageNet dataset with SBGMs are challenging due to its vast diversity. Considering LIM's improvement in generation diversity compared to the previous DMs, I expect that LIM will meaningfully outperform DMs on ImageNet. Although the paper lacks experiments on ImageNet, future research in this area would be intriguing. \n\nSome minor typos:\n- line 156: the the Wasserstein-1 -> the Wasserstein-1\n- line 158: v-Euler-Maruyamais -> v-Euler-Maruyama is\n- It$\\bar{o}$ -> It$\\hat{o}$\n", "limitations": " ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In contrast to the previous works, this paper presents a valuable contribution for non-Gaussian distributions where a rigorous proof of the exact time-reversal formula for stochastic differential equations (SDEs) is provided. The provided derivation establishes a solid foundation for the proposed method.\n\nThe paper provides clear and comprehensive proofs and demonstrations of the reverse SDE formula, sampling equation, numerical solver (v-Euler-Maruyama), and fractional denoising score matching. These findings effectively illustrate the construction, training, and utilization of the Levy-Ito Model (LIM) in the context of sampling.\n\nFigure 2 visually compares Brownian motion and the Levy process, clearly highlighting the advantageous characteristics of the Levy process, including its notable large jumps.\nSeveral toy examples effectively illustrate the superior capabilities of the proposed method for mode estimation compared to previous Diffusion Models (DMs).\n\nThe paper is well-written and easily readable.", "weaknesses": "Enhanced diversity is the primary improvement of the Levy-Ito Model over previous Diffusion Models (DMs). Conducting more experiments on mode estimation and sample diversity would further strengthen the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable and keen insights to enhance the completeness of our paper.\n\n> Question 1\n> \n\nLIM, Heavy-tailed DSM [Deasy et al., 2021], and Denoising Diffusion Gamma Models [Nachmani et al., 2021] all share the advantage of a faster convergence rate for sampling compared to DDPM [Ho et al., 2020]. Additionally, both LIM and Heavy-tailed DSM excel in mode estimation for imbalanced datasets, as mentioned [Deasy et al., 2021]. However, LIM outperforms significantly other models due to 1) its ability to find the exact drift term in the time-reversal formula, 2) precise and stable training, and 3) guaranteed convergence for reverse sampling.\n\nSince all three distributions follow a Lévy process [Dytso et al., 2018][Wang et al., 2012], the drift term for the time-reversal formula can be computed according to [Conforti et al., 2021]. For Isotropic $\\alpha$-stable Lévy processes, which follow symmetric and stable distributions, an exact drift term can be computed from the time-reversal formula (Theorem 4.1). Moreover, training can be stable and accurate without the need for integration, as described by (Theorem 4.3). Furthermore, with the knowledge of the exact drift term, a reverse sampling formula ensuring convergence can be derived (Corollary E.1). \n\nIn contrast, the drift terms used in Heavy-tailed DSM and Denoising Diffusion Gamma Models appear as an integral, making it difficult to derive an exact form. Even if integration is used during training or approximations are applied to the drift term, accumulating errors lead to inaccurate and unstable training. Moreover, when using an approximation for the drift term to make training practical, there's no assurance that the distance between the distribution $p_{\\text{data}}$ of actual data and the distribution $p_{\\theta}$ obtained from reverse sampling aligns within a certain range. \n\nIn fact, for Heavy-tailed DSM, despite proposing a modified score function corresponding to the Generalized Gaussian distribution and providing a training method, there is no theoretical foundation proving that the modified score function converges to $p_{\\text{data}}$ through reverse sampling. Similarly, Denoising Diffusion Gamma Models also fails to propose an exact drift term and present limitations by not providing theoretical evidence for the convergence of reverse sampling using the proposed score function. In fact, the FID performance of these two papers is significantly lower than that of the existing DDPM performance.\n\nWe will incorporate the clear distinctions from the previously mentioned existing heavy-tailed noise methods into the paper revision.\n\n> Question 2\n> \n\nUtilizing ImageNet, which has a larger number of classes compared to CIFAR10, to showcase the diversity of LIM through experimental results seems to be a convincing and effective approach. We conducted performance comparisons between Diffusion model [Song et al, 2020] and LIM using the ADM architecture [Dhariwal et al., 2021] on ImageNet 64x64. The results are summarized in the following table:\n\n| Model | FID ($\\downarrow)$ | Precision ($\\uparrow$) | Recall $(\\uparrow)$ |\n| --- | --- | --- | --- |\n| Diffusion model [Song et al, 2020] | 14.23 | 0.6711 | 0.6932 |\n| LIM ($\\alpha=1.8$) | $\\textbf{12.97}$ | $\\textbf{0.6782}$ | $\\textbf{0.6937}$ |\n\nThe experimental results confirm that LIM performs better than the Diffusion model [Song et al., 2020] in terms of FID, precision, and recall. However, the difference in recall is slight compared to what was expected. This aspect might be due to only using $\\alpha=1.8$, and we can expect more improvement of LIM on ImageNet by searching better alpha values\n\nIn the revised paper, we will further compare LIM's performance with heavy architectures like NCSN++ [Song et al., 2020] or DiT [Peebles et al., 2021], using the high-resolution dataset ImageNet 256 with varying $\\alpha$.\n\n> Question 3\n> \n\nWe sincerely appreciate your thorough review of our paper and your detailed feedback on areas that need improvement. All the typos you mentioned have been corrected within our paper, and we will carefully inspect for any other potential errors to enhance the overall quality and completeness of the paper. Thank you once again.\n\n---\n\n[Conforti et al., 2021] Time reversal of markov processes with jumps under a finite entropy condition (Stochastic Processes and their Applications, 2021)\n\n[Deasy et al., 2021] Heavy-tailed denoising score matching\n\n[Dhariwal et al., 2021] Diffusion Models Beat GANs on Image Synthesis\n\n[Dytso et al., 2018] Analytical properties of generalized Gaussian distributions\n\n[Ho et al., 2020] Denoising Diffusion Probabilistic Models (NeurIPS 2020)\n\n[Nachmani et al., 2021] DENOISING DIFFUSION GAMMA MODELS\n\n[Peebles et al., 2021] Scalable Diffusion Models with Transformers\n\n[Song et al., 2020] Score-Based Generative Modeling through Stochastic Differential Equations\n\n[Wang et al., 2012] Lévy Measure Decompositions for the Beta and Gamma Processes"}]

Author rebuttal:

Meta-review: All reviewers appreciated the paper and the authors successfully answered the questions raised in the reviews
Decision: Accept (spotlight)