Title: Score-based Generative Models with Lévy Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative Model

Score-based Method

Lévy processes
Abstract: Investigating the optimal stochastic process beyond Gaussian for noise injection in a score-based generative model remains an open question. Brownian motion is a light-tailed process with continuous paths, which leads to a slow convergence rate for the Number of Function Evaluation (NFE). Recent studies have shown that diffusion models suffer from mode-collapse issues on imbalanced data.
In order to overcome the limitations of Brownian motion, we introduce a novel score-based generative model referred to as Lévy-Itō Model (LIM). This model utilizes isotropic $\alpha$-stable Lévy processes. We first derive an exact reverse-time stochastic differential equation driven by the Lévy process and develop the corresponding fractional denoising score matching. The proposed generative model takes advantage of the heavy-tailed properties of the Lévy process. Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models across various image datasets such as CIFAR10, CelebA, and imbalanced dataset CIFAR10LT. Comparing our results to those of DDPM with 3.21 Fréchet Inception Distance (FID) and 0.6437 Recall on the CelebA dataset, we achieve 1.58 FID and 0.7006 Recall using the same architecture. LIM shows the best performance in NFE 500 with $2\times$ faster total wall-clock time than the baseline.

Score-based Generative Models with Lévy Processes

Abstract

Investigating the optimal stochastic process beyond Gaussian for noise injection in a score-based generative model remains an open question. Brownian motion is a light-tailed process with continuous paths, which leads to a slow convergence rate for the Number of Function Evaluation (NFE). Recent studies have shown that diffusion models suffer from mode-collapse issues on imbalanced data. In order to overcome the limitations of Brownian motion, we introduce a novel scorebased generative model referred to as Lévy-Itô Model (LIM). This model utilizes isotropic α -stable Lévy processes. We first derive an exact reverse-time stochastic differential equation driven by the Lévy process, then develop the corresponding fractional denoising score matching. LIM takes advantage of the heavy-tailed properties of the Lévy process. Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models across various image datasets such as CIFAR10, CelebA, and imbalanced dataset CIFAR10LT. Comparing our results to those of DDPM with 3.21 Fréchet Inception Distance (FID) and 0.6437 Recall on the CelebA dataset, we achieve 1.58 FID and 0.7006 Recall using the same architecture. LIM shows the best performance in NFE 500 with 2 × faster total wall-clock time than the baseline.

1 Introduction

Score-based diffusion models train time-dependent score models given a diffusion process and generate samples from the corresponding reverse process [39]. The feasibility of score-based diffusion models is based on the time-reversal formula of the Stochastic Differential Equations (SDEs) driven by the Brownian motion [2]. The exact form of the density function of the Brownian motion enables efficient sampling and easy theoretical handling compared to other types of noise. The score-based diffusion model is easy to train because the score function is approximated through Denoising ScoreMatching (DSM). Additionally, the solution of reverse SDE can be simulated through the numerical SDE solver. This enables effective training and utilization of the model in various applications. One limitation of score-based diffusion models is that the paths produced by the SDEs need numerous iterations to achieve a data distribution, leading to slower sampling. To solve these problems, various methods have been proposed to reduce the convergence rate while maintaining to generate high fidelity samples [22]. Another effort has also been made to use distillation to reduce the number of steps [33]. In addition, it has been pointed out that Diffusion models show significant degradation in terms of fidelity and diversity when dealing with imbalanced datasets [30]. Thus, there is a need

(a = 2)

Lévy

(α< 2)

dx, = b(t, X,-)dt + oL(t)dL

ForwardSDE

Xo (data)

(ssiou) μx

dx,=

ReverseSDE

b(t,x) - α · o(t)S(a)(x))dt + o(t)dL

(a)

(b)

(c)

Forward SDE

Reverse SDE

XT

DM(SDE)

LIM-1.5

0.375

NFE

to perform further research and develop score-based generative models that use alternative noises. Recently, there have been attempts to use new noises such as gamma distribution [24] or generalized Gaussian distribution [8]. These models have shown possibilities in expanding the range of noise processes. However, they did not propose a time-reversal formula, which is an important property for the feasibility of score-based generative models. As a result, the time-reversal formula has not yet been fully proven for alternative noises.

In this paper, we propose a novel score-based generative model, Lévy-Itô Model (LIM), which utilizes isotropic α -stable Lévy processes as noise injection. This is the first attempt to use the isotropic α -stable Lévy process in a score-based generative model and prove the exact time-reversal formula of SDEs driven by Lévy process. It is noted that this is also the first continuous-time model using the heavy-tailed process, which allows for the derivation of a probability fractional ODE (See more details in Appendix C). We derive a fractional score function to match the drift term of the generator of time-reversal SDEs driven by Lévy process and propose fractional Denoising Score Matching , which is a more general version of DSM [40] to approximate the fractional score function.

Looking at the reverse path of LIM in Figure 2, we observe large jumps occurring near the noise space. It causes the trajectory to move quickly towards the sample space, which is shown in Figure 1. As LIM has sufficient ability to move from the sample space to the noise space in spite of small noise coefficient due to large jump noises, there is a higher likelihood of reaching multiple modes more precisely. These characteristics can be observed in Figure 3, which is based on the simplest form of an imbalanced dataset. LIM's superiority can be observed not only from the ratio aspect but also from FID and MMD, which are calculated directly without using any embeddings for the sets of true samples and generated samples. LIM shows that the distribution p θ of its generated samples is similar to the ground truth distribution p data .

(b) Diffusion Model

15

15

15

10

10

10

5

5

5

0

0

0

-5

-5

-5

10

-10

-10

15

15

-15

-20

15

-10

-5

0

5

10

15

15

-10

-5

0

5

10

15

15

-10

-5

0

5

10

15

We draw inspiration from the observation that the time-reversal formula for SDEs is known not only for diffusion processes but also for SDEs driven by Lévy processes containing jumps [7]. In addition, the invariant measure of the Euler scheme has been shown to converge to the invariant measure of the solution of SDEs driven by Lévy process [6]. Moreover, there is an example of using Lévy processes in Markov Chain Monte Carlo (MCMC), where the corresponding research demonstrated a faster convergence rate for NFE both theoretically and experimentally [37]. However, it cannot be directly extended to SDEs driven by multivariate stable processes and use an approximation to calculate the score function of a data distribution, which degrades the potential performance. These observations motivate us to consider using Lévy processes in score-based generative models.

Our experimental results show LIM allows for faster and more diverse sampling while maintaining high fidelity compared to diffusion models across various image datasets such as CIFAR10, CelebA, and imbalanced dataset CIFAR10LT. We realize that the heavy-tail property of the Lévy process allows us to overcome the limitations of Brownian motion with respect to mode estimation, sample diversity, and convergence rate for NFE. Through the above experiments, we emphasize we have further expanded the scope of score-based generative models.

The contributions of our paper can be summarized as follow:

We propose a novel score-based generative model, Lévy-Itô Model (LIM), which utilizes isotropic α -stable Lévy processes as noise injection.

We derive an exact reverse-time stochastic differential equation driven by the Lévy process and develop the corresponding fractional Denoising Score Matching.

We validate that LIM allows for faster and more diverse sampling while maintaining high fidelity compared to existing diffusion models across various image datasets and shows more robustness against imbalanced data.

2 Related works

Diffusion models have shown improvements in performance through various approaches such as incorporating guidance [18], introducing new architectures [29], and proposing novel training methods [13]. Additionally, various attempts to reduce convergence rates have been proposed, such as using ODEsolvers [23], or Fourier Neural Operator [42]. Despite these advancements, diffusion models still face inherent limitations such as slow convergence rates and the mode-collapse issue in imbalanced datasets [30]. There have been attempts to use new noises in score-based generative models. The

denoising generative model with gamma distribution is one such example [24], which shows faster convergence for NFE than score-based diffusion models. [8] proposes using heavy-tailed noises in score-based generative models by extending the noise injection to generalized Gaussian distributions. The authors state that the heavy-tailed DSM is more robust against imbalance tasks than score-based diffusion models. They commonly use a variant of the Denoising Diffusion Probabilistic Model (DDPM) structure to apply alternative noises. These models have shown the potential to expand the range of noise processes, however, their performance in terms of the Fréchet Inception Distance (FID) is not as high as that of score-based diffusion models.

Since these distributions follow a Lévy process [11, 41], the drift term for the time-reversal formula can be computed according to [7]. For isotropic α -stable Lévy processes, which follow symmetric and stable distributions, an exact drift term can be computed from the time-reversal formula. Training can be stable and accurate without the needs for integration. Furthermore, with the knowledge of the exact drift term, a reverse sampling formula ensuring convergence can be derived. In contrast, the drift terms used in Heavy-tailed DSM and Denoising Diffusion Gamma Models appear as an integral, making it difficult to derive an exact form. Even if integration is used during training or approximations are applied to the drift term, accumulating errors lead to inaccurate and unstable training. Moreover, when using an approximation for the drift term to make training practical, there's no assurance that the distance between the distribution of actual data, p data and the distribution obtained from reverse sampling, p θ aligns within a certain range. For Heavy-tailed DSM, despite proposing a modified score function corresponding to the Generalized Gaussian distribution and providing a training method, there is no theoretical foundation proving that the modified score function converges to p data through reverse sampling. Similarly, Denoising Diffusion Gamma Models also fail to propose an exact drift term and present limitations by not providing theoretical evidence for the convergence of reverse sampling using the proposed score function. As a result, the FID performances of these two papers are significantly lower than that of DDPM [15].

3 Background

3.1 Isotropic alpha-stable distributions

Let α ∈ (0 , 2] be a characteristic exponent that determines the decay rate, and γ be a scale parameter. For d ∈ N , S α S d ( γ ) denotes the d -dimensional isotropic α -stable distribution . If a random variable X ∈ R d follows S α S d ( γ ) , then the characteristic function of X is E [ e i 〈 u ,X 〉 ] = e -γ α || u || α and has heavy-tail properties with tail index α , i.e., P ( || X || > r ) ∼ r -α . A closed-form of density

function for S α S d ( γ ) is unknown, except for α = 1 , α = 1 . 5 and α = 2 cases 3 . Note that the behavior of isotropic α -stable distribution is different from non-isotropic α -stable in multi-dimension since the components of isotropic α -stable are not independent unless α = 2 . Contrary to [43] which can not be extended to SDEs that are driven by α -stable Lévy processes, we utilize isotropic α -stable processes for driving noise in SDEs. See Figure 4 for the comparison between sample distributions of Gaussian, non-isotropic α -stable, and isotropic α -stable random variable.

3.2 Lévy processes

An R d -valued stochastic process ( L t ) t ≥ 0 with L 0 = 0 is called Lévy process if (i) L t has independent increments, (ii) L t has stationary increments, and (iii) L t has stochastically continuous sample paths. Thus, Lévy process may have at most countably many discontinuous points, also known as jumps [3]. If for all s < t , ( L t -L s ) d = L t -s follows S α S d (( t -s ) 1 / α ) , where d = means that the two processes have the same law, then L t is referred to as a d -dimensional isotropic α -stable Lévy process, denoted by L α t . Notably, L α t is a prototypical pure jump process. The α -stable distribution has a heavy-tailed property, meaning that the frequency of large jumps increases as α decreases. According to the

Lévy-Itô decomposition [21], we can split the sample paths of Lévy process into drift parts, Brownian process parts, and jump parts. The focus of this paper is on rotationally invariant pure-jump Lévy processes, whose Lévy measure is ν ( d x ) ∼ || x || -d -α d x .

4 Lévy-Itô Model

4.1 Time-Revesal of SDEs driven by Lévy processes

In this section, we provide a time-reversal theory for SDEs driven by Lévy processes as follows:

where → X t -denotes the left limit of → X at time t which implies the càdlàg (right conti. with left limits) solution. We define fractional score function S ( α ) t by,

where ∆ γ 2 denote the fractional Laplacian of order γ 2 for γ ∈ ( -1 , 2) [27]. Precisely, ∆ γ 2 f ( x ) := F -1 {|| u || γ F{ f } ( u ) } where F and F -1 are the Fourier and the inverse Fourier transform, respectively. The time-reversal of → X t is defined by ← X t := → X t --→ X T -where t runs backward from T to 0 . Note that ← X t is a Lévy process with respect to natural filtration when time flows forwards [17], which is the càglàd (left conti. with right limits). Our first main result is the derivation of the reverse-time SDE corresponding to (1).

Theorem 4.1 (Time-Reversal of SDE) . Let 1 < α ≤ 2 . Suppose b ( t, x ) is locally bounded and the marginal distribution of ( X t ) t ∈ [0 ,T ] has a density function p t ( x ) .

where ¯ dt is an infinitesimal negative timestep, ¯ L α t is the backward version of the isotropic α -stable Lévy process, and ¯ Z t is the backward version of a Lévy-type stochastic integral Z t such that E [ Z t ] = 0 with finite variation.

See more detail in Theorem B.1 for the definition of Z t . Now we propose a score-based generative model with Lévy process L α t , refers to L évyI t¯ o M odel (LIM). In [39], the VP-SDE formula is defined as d → X t = -β ( t ) 2 → X t dt +( β ( t )) 1 2 dB t and β ( t ) is set to a linear function β ( t ) = ( β 1 -β 0 ) t + β 0 . On the other hand, we extended [39]'s formula to the SDE driven by isotropic α -stable Lévy process, d → X t = -β ( t ) α → X t dt + ( β ( t )) 1 α dL α t , where we simulate the noise term in (3) using only L α t . This choice is motivated by the fact that, in contrast to σ L ( t ) dL α t , which exhibits infinite variation, dZ t has a mean of 0 and finite variation, rendering it comparatively negligible. The specific β ( t ) in our paper is derived from the cosine schedule proposed in [25] and tailored to LIM as β ( t ) = -α d log(cos( π t 2 )) dt for setting a forward SDE with b ( t, x ) = -β ( t ) α x and σ L ( t ) = β 1 α ( t ) in (1). Then the solution → X t has the same law with a ( t ) → X 0 + γ ( t ) /epsilon1 where /epsilon1 ∼ S α S d (1) , a ( t ) = exp( -∫ t 0 β ( s ) α ds ) , and γ ( t ) = (1 -a α ( t )) 1 α . As b ( t, x ) = -β ( t ) α x satisfies the condition of Theorem 4.1, we can obtain the reverse-time SDE of the beta-schedule version,

For t < s , the solution of equation (4) can be represented as an integral, utilizing the semi-linear structure of the reverse SDE,

Wecan get faster sampling method by using an approximation for the second term ∫ t s β ( u ) a ( u ) S ( α ) u ( x u ) du . This term can be approximated as ( ∫ t s β ( u ) a ( u ) du ) · S ( α ) s ( x s ) and it is possible to calculate ∫ t s β ( u ) a ( t ) a ( u ) , the scale parameter γ of which follows γ α = ∣ ∣ ∫ t s d du ( e -∫ t u β ( k ) )∣ ∣ .

∣ ∣ Corollary 4.1 (Variant of Euler-Maruyama with dynamic time increment) . Suppose the fractional score function in the SDE given by equation (4) satisfies the conditions stated in Theorem E.2, and a ( t ) , γ ( t ) are bounded. Then, there exists a Markov chain ( x t ) that follows:

Here, /epsilon1 ∼ S α S d (1) for s > t , and ∆ t = s -t /lessmuch 1 . As a result of the conditions being satisfied, the Wasserstein-1 distance between the invariant measures of the solution of Equation (4) and ( x t ) is bounded by ( ∆ t ) 1 α [6]. (See more detail in Appendix E)

The paper [6] outlines a way to find bounds on the Wasserstein-1 distance by determining bounds for the approximation of the drift term, J 1 , and the approximation of the stochastic term, J 2 . In the equation, only the bound for the approximation of the drift term, J 1 , is utilized. Furthermore, since the law of a weak solution is same to the law of the strong solution, the Wasserstein-1 distance between the invariant measures for the strong solution and the approximation can be used for the that of the weak solution and the approximation as well. In addition, we can also find a probability ODE corresponding to (1).

Theorem 4.2 (Probability Fractional ODE) . Let b ( t, x ) : R d → R and σ L ( t ) : R → R be functions that satisfy the Lipschitz condition as stated in [35]. For the SDE following (1), the solution ( X t ) t ∈ [0 ,T ] of the SDE satisfies the following ODE:

Due to its semilinear structure, the solution to (7) can be represented as an integral. This is shown in Lemma D.2. The Euler method can then be used to find the solution.

Corollary 4.2 (Deterministic ODE sampling) . If the drift term in (7) is Lipschitz continuous and the solution x t has a bounded second derivative, then a sequence ( x t ) can be obtained using the Euler-scheme:

where s > t . When the step size is ∆ t , the global truncation error is bounded by O ( ∆ t ) [5].

4.2 Estimating Fractional Score Function

Let q α ( x ) be the density of S α S d (1) . A weak solution of the beta-schedule version is → X t = a ( t ) → X 0 + γ ( t ) /epsilon1 where /epsilon1 ∼ S α S d (1) with the following transition density:

We train the fractional score model s t ( x t ; θ ) which estimates S ( α ) t ( x t ) at the optimal point. The method of approximating the fractional score function S ( α ) t ( x t ) can be equivalent to estimating the fractional score function S ( α ) t ( x t | x 0 ) , which replaces the fractional score function of term p t ( x t ) in (2) with that of the transition density (9).

Theorem 4.3 (Fractional Denoising Score Matching) . For a parameter θ , two loss functions L 1 ( θ ) and L 2 ( θ ) are given as follows:

Then, there exists a constant C satisfying L 1 ( θ ) = L 2 ( θ ) + C so that the optimization of two loss functions with respect to θ is equivalent.

The drift term of the reverse-time SDEs can be accurately determined by training s t ( x t ; θ ) to match S ( α ) t ( x t | x 0 ) , as stated in Theorem 4.3. Moreover, if we have the jump-type Lévy process L α t , we can calculate S ( α ) t ( x t | x 0 ) explicitly as a linear expression.

Theorem 4.4. Let 1 < α < 2 . If the transition density function p t ( x t | x 0 ) follows (9), then the fractional score function of p t ( x t | x 0 ) can be represented as a linear form such that

Theorem 4.4 implies that the computational cost of training a fractional score model s t ( x t ; θ ) is similar to DSM [40].

4.3 Loss function

We set beta schedule as cosine beta schedule β ( t ) [25] , following β ( t ) = α 1+ s π 2 tan ( t + s 1+ s π 2 ) for small number 0 < s /lessmuch 1 . Then, the drift term a ( t ) = cos( t + s 1+ s π 2 ) cos( s 1+ s π 2 ) and the diffusion term γ ( t ) = (1 -a ( t ) α ) 1 α . We use the U-net architecture as in DDPM [15] for training and apply the loss (11) to train s t ( x t ; θ ) . For /epsilon1 ∼ S α S d (1) and x 0 ∼ p data , x t = a ( t ) x 0 + γ ( t ) /epsilon1 , U (0 , 1) denotes a uniform distribution. Then, the loss with the relative weight γ ( t ) is defined as

The fractional score model s t ( x t ; θ ) learns the law of the solution X t for each t because the betaschedule version solution used during training is the weak solution of the given forward SDEs.

5 Experiments

We employ continuous time step throughout our experiments and set NFE to a fixed value of 500 in all cases. Given that Lévy processes exhibit a higher occurrence of extreme values compared to Gaussian noise, we opt for the smooth L 1 Loss instead of the L 2 Loss to ensure stable training. We use the modified VP-SDE to fit the Lévy process and quadratic timestep during reverse sampling.

5.1 Mode Estimation

We conduct the class imbalance experiment for mode estimation with two kinds of datasets. First is customized imbalanced CIFAR10 with a ratio given of 1:2:3:4:5. We choose t-distributed stochastic neighbor embedding (t-SNE), which reduces complex data of high dimensions to two dimensions. In table 1 and Fiure 5, the mode estimation of LIM is much closer to training data than the diffusion model. Also, the corresponding FID, Recall, and MMD of LIM show improved performances. The second dataset is CIFAR10LT including 10 classes, which follow heavy-tailed distribution

4 . Table 3 shows that as we move from class 0 with a high ratio to class 9 with a low ratio, the overall performance tends to decrease. However, LIM shows better performances compared to the diffusion model with respect to FID and Recall. These results demonstrate that LIM is more robust for mode estimation in imbalanced datasets, and alleviates the mode-collapse issue.

5.1.1 Sample Diversity

We choose Recall as an evaluation metric for sample diversity. In Table 5, we can see that LIM exceeds other diffusion models in Recall. Moreover, even in the conditional generation, it shows higher Recall for each class, proving that the generated samples are more diverse than those of the diffusion model (Table 2).

↓

Imputation The imputation refers to the task of restoring the masked regions in an image. To measure the diversity of samples, we use Learned Perceptual Image Patch Similarity (LPIPS) as a metric, where a higher LPIPS indicates greater diversity and a lower value implies more similarity between samples. For each original image x , we generate 10 imputed images { ˆ x i } 10 i =1 and calculate the average of LPIPS for the 10 samples. As shown in Figure 6, the diffusion model tends to generate similar eyes and mouths in the masked region, while LIM produces a much wider range of diverse images and shows higher LPIPS.

origin image masked image

α =2.0

(DM)

α=1.5

(LIM)

imputation result

5.2 Convergence rate for NFE

We compare LIM and diffusion models on CIFAR10 ( 32 × 32 ) and CelebA ( 64 × 64 ) dataset in terms of convergence rate for NFE. Figure 7 shows that FID score of LIM drops at earlier NFE than the baselines in SDE-based sampling. Moreover, it shows a competitive convergence rate for NFE compared to ODE-based models. These experimental results demonstrate that LIM is capable of converging to the sample space faster than the baselines. See Appendix G.6 to figure out the wall clock time per NFE on various datasets.

NCSN++(deep)

30.0

DDPM

60.0

1

DDPM

DDIM

Analytic-DDIM

Analytic-DDIM

DDIM

10.0

PNDM

score

LIM

score

LIM

S

FID

10.0

FID

5.0

4.0

5.0

4.0 -

3.0

3.0

2.0

2.0

1.0

1.0

20

50

200

500

1000

2050

200

500

1000

NFE

NFE

(a) CIFAR10

(b) CelebA 64x64

5.3 Sample Quality

Table 5 displays the performance of LIM in comparison to previous approaches for unconditional image generation on CIFAR10 ( 32 × 32 ) and CelebA ( 64 × 64 ). We measure FID scores and Recall using reverse sampling and generate 50k samples for all datasets. For CelebA, we randomly select 50k samples from the training dataset with 5 iterations. We evaluate NLL using a uniformly dequantized test dataset and calculate the log-likelihood through the average of 5 iterations using the ODE solver [39]. Since it is difficult to numerically calculate the density function of the isotropic α -stable distribution in higher dimensions ( d > 100 ) [26], we approximate the density function with non-isotropic α -stable distributions to evaluate the NLL of LIM. For CIFAR10, we train 2 backbone models: DDPM [15] and NCSN++(deep) [39]. See Appendix G for more detail.

The performance of LIM can differ with different α . Indeed, this seems intuitively natural since images with low resolution less prefer large jumps, while images with high resolution and multiple modalities can benefit exploration from large jumps. Table 4 provides a comparison of LIM's performance across different α values. Although the overall tendency of declining FID with lower values remains consistent, it appears that differences in FID it-

self arises when using different architectures on CIFAR10 (32 × 32). In spite of using half of NFE used in [39], LIM shows competitive FID score, Recall, and NLL. Especially in CelebA (64 × 64), LIM achieves FID score of 1.57 and Recall of 0.7007. It is an impressive performance since we do not use any other guidance or improved model architecture. We also conducted performance comparisons between diffusion model [39] and LIM using the ADM architecture [9] on ImageNet (64 × 64). Diffusion model [39] achieves a FID of 14.23, Precision of 0.6711, and Recall 0.6932. In constrast, LIM achieves a FID of 12.97, Precision of 0.6782, and Recall of 0.6782. For comparing the performance on a high-resolution dataset, we chose the DDPM architecture [15], and trained LIM and diffusion model [39] on the CelebA-HQ dataset (256 × 256). This indicates that when it comes to CelebA-HQ, LIM attains an FID score of 7.76 with 500 NFE, whereas the diffusion model achieves a higher FID of 11.87 with 1000 NFE based on DDPM architecture [15].

5.4 Deterministic ODE sampling

We have the option to utilize DDIM inference for our model. When we use a pretrained model on CelebA (64x64), Table 6 displays the FID scores acquired by DDIM and LIM-DDIM for various NFEs. The results demonstrate that LIM-DDIM outperforms DDIM in terms of performance.

6 Limitation

Many diffusion models have been conducted to find the best model architectures such as NCSN++ [39] and ADM [9]. In contrast, Lévy process differs significantly from Brownian motion. Therefore, it is uncertain whether using the same model architecture that has been used previously is the best approach for LIM. Although, in this study, we use the same structure of the existing diffusion model to compare the pure effect of the Lévy process, there is a need to find a model architecture that is suitable for the Lévy process in order to improve performance further.

7 Conclusion

The aim of this study is to expand the variety of noise distributions utilized in score-based generative models by introducing an exact time-reversal formula for SDEs with Lévy processes. We propose a new score-based generative model, Lévy-Itô Model (LIM). The effectiveness of this model has been verified by empirical results, which show that LIM performs well with various image dataset. As a result, this study provides a solution and demonstrates the potential for utilizing a wider range of non-Gaussian Markov processes in score-based generative models.

Acknowledgments and Disclosure of Funding

This work was supported by LG AI Research. This work was also supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korean government(MSIT)(No. 2022-0-00612, Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AI; No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University), National Research Foundation of Korean(NRF) funded by the Korean government(MSIT)(2021R1C1C1009256), and a grant from Korea University (K2304791). This work also was supported by Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) and Gwangju Metropolitan City. The authors especially thank Prof. Nicolas Privault for his interest in our paper and valuable discussions that helped us correct errors in our main result. The first author would like to thank Woo-Joo Kim for insightful discussion and anonymous reviewers for their constructive discussion and editorial efforts.

References

M. Abramowitz and I. A. Stegun. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables . Dover Publications, 1964.

Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , pages 313-326, 1982. URL https://www.sciencedirect.com/science/ article/pii/0304414982900515 .

David Applebaum. Lévy processes and stochastic calculus . Cambridge university press, 2009.

Björn Böttcher. Feller evolution systems: Generators and approximation. Stochastics and Dynamics , 14(03):1350025, 2014.

J. C. Butcher. Numerical Differential Equation Methods . John Wiley and Sons, Ltd, 2016. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/9781119121534.ch2 .

Peng Chen, Changsong Deng, Rene Schilling, and Lihu Xu. Approximation of the invariant measure of stable sdes by an euler-maruyama scheme, 2022. URL https://arxiv.org/ abs/2205.01342 .

Giovanni Conforti and Christian Léonard. Time reversal of markov processes with jumps under a finite entropy condition. Stochastic Processes and their Applications , 2021. doi: 10.48550/ARXIV.2104.10948. URL https://arxiv.org/abs/2104.10948 .

Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching, 2021.

Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.

John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics , 6(1):19-26, 1980.

A. Dytso, Bustin, and et al. Analytical properties of generalized gaussian distributions., 2018.

Gerald B. Folland. Real Analysis: Modern Techniques and Their Applications . John Wiley and Sons, 2013.

Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy, 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. arXiv preprint arXiv:1706.08500 , 12(1), 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf .

Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation , 18(3):10591076, 1989.

Jean Jacod and Philip Protter. Time Reversal on Levy Processes. The Annals of Probability , 16 (2):620 - 641, 1988. doi: 10.1214/aop/1176991776. URL https://doi.org/10.1214/aop/ 1176991776 .

Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models, 2023.

Thomas G. Kurtz. Equivalence of stochastic equations and martingale problems. Stochastic Analysis 2010 , 2011.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems , 32, 2019.

Paul Lévy. Sur les intégrales dont les éléments sont des variables aléatoires indépendantes. Annali della Scuola Normale Superiore di Pisa-Classe di Scienze , 3(3-4):337-366, 1934.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds, 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022.

Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models, 2021.

Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021. URL https://arxiv.org/abs/2102.09672 .

John Nolan. Multivariate elliptically contoured stable distributions: Theory and estimation. Computational Statistics , 28, 10 2013. doi: 10.1007/s00180-013-0396-7.

Manuel D Ortigueira, Taous-Meriem Laleg-Kirati, and J A Tenreiro Machado. Riesz potential versus fractional laplacian. Journal of Statistical Mechanics: Theory and Experiment , 2014.

Antonis Papapantoleon. An introduction to lévy processes with applications in finance, 2008.

William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023.

Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang. Class-balancing diffusion models, 2023.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

Ludger Rueschendorf, Alexander Schnurr, and Viktor Wolf. Comparison of timeinhomogeneous markov processes, 2015. URL https://arxiv.org/abs/1505.02925 .

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.

Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems , 29, 2016.

D. Schertzer, M. Larchevê que, J. Duan, V. V. Yanovsky, and S. Lovejoy. Fractional fokker-planck equation for nonlinear stochastic differential equations driven by non-gaussian lévy stable noises. Journal of Mathematical Physics , 42(1):200-212, jan 2001. doi: 10.1063/1.1318734.

Jinghai Shao. Weak convergence of euler-maruyama's approximation for sdes under integrability condition, 2018. URL https://arxiv.org/abs/1808.07250 .

Umut ¸ Sim¸ sekli. Fractional langevin monte carlo: Exploring lévy driven stochastic differential equations for markov chain monte carlo. In International Conference on Machine Learning , pages 3200-3209. PMLR, 2017.

John Skilling. Maximum Entropy and Bayesian Methods: Cambridge, England, 1988 , volume 36. Springer Science & Business Media, 2013.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2020.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 23:1661-1674, 2011.

Yingjian Wang and Lawrence Carin. Levy measure decompositions for the beta and gamma processes, 2012.

Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning, 2023.

Umut ¸ Sim¸ sekli. Fractional langevin monte carlo: Exploring lévy driven stochastic differential equations for markov chain monte carlo, 2017. URL https://arxiv.org/abs/1706.03649 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nScore-based generative models (SBGMs) generally employ Brownian motion, also known as the Wiener process, for noise injection. However, using Brownian motion in SBGMs often leads to issues such as mode collapse or slow sampling. To address these problems, the authors propose SBGMs with an isotropic α-stable Levy distribution named Levy-Ito Model (LIM). The α-stable Levy distribution exhibits a heavy-tail property, enabling the Levy-Ito Model to achieve improved mode estimation, sample diversity, and faster convergence in terms of neural function evaluations (NFEs) compared to SBGMs employing Brownian motion. For the first time, this paper proves the time-reversal formula of stochastic differential equations (SDEs) with Levy processes. It also establishes the sampling equation and presents a numerical solver known as v-Euler-Maruyama for LIM. Furthermore, the paper introduces fractional denoising score matching, which elucidates the training process of LIM. Empirically, the proposed method shows superior mode estimation, diversity in image generation and imputation, and faster convergence of LIM compared to previous SBGMs.\n\nSTRENGTHS:\nIn contrast to the previous works, this paper presents a valuable contribution for non-Gaussian distributions where a rigorous proof of the exact time-reversal formula for stochastic differential equations (SDEs) is provided. The provided derivation establishes a solid foundation for the proposed method.\n\nThe paper provides clear and comprehensive proofs and demonstrations of the reverse SDE formula, sampling equation, numerical solver (v-Euler-Maruyama), and fractional denoising score matching. These findings effectively illustrate the construction, training, and utilization of the Levy-Ito Model (LIM) in the context of sampling.\n\nFigure 2 visually compares Brownian motion and the Levy process, clearly highlighting the advantageous characteristics of the Levy process, including its notable large jumps.\nSeveral toy examples effectively illustrate the superior capabilities of the proposed method for mode estimation compared to previous Diffusion Models (DMs).\n\nThe paper is well-written and easily readable.\n\nWEAKNESSES:\nEnhanced diversity is the primary improvement of the Levy-Ito Model over previous Diffusion Models (DMs). Conducting more experiments on mode estimation and sample diversity would further strengthen the proposed method.\n\nQUESTIONS:\nThe primary advantages of LIM appear to arise from employing heavy-tailed noise injections, an approach already investigated in prior works. What are the specific benefits of utilizing an isotropic α-stable distribution compared to other potential heavy-tailed distributions?\n\nTraining and sampling from the ImageNet dataset with SBGMs are challenging due to its vast diversity. Considering LIM's improvement in generation diversity compared to the previous DMs, I expect that LIM will meaningfully outperform DMs on ImageNet. Although the paper lacks experiments on ImageNet, future research in this area would be intriguing. \n\nSome minor typos:\n- line 156: the the Wasserstein-1 -> the Wasserstein-1\n- line 158: v-Euler-Maruyamais -> v-Euler-Maruyama is\n- It$\\bar{o}$ -> It$\\hat{o}$", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Levy-Ito Model, a novel score-based generative model (SBGM) that utilizes the isotropic $\\alpha$-Levy process as perturbation noise. The authors highlight that their proposed method is the first continuous-time SBGM to incorporate a heavy-tailed process. They aim to leverage the advantages of heavy-tailed processes observed in various cases. For instance, such processes enhance convergence speed in Markov chain Monte Carlo (MCMC) methods like the Langevin algorithm. Additionally, heavy-tailed processes offer faster mode-hopping behavior compared to Gaussians.\n\nFirst, the authors establish the exact time-reversed stochastic differential equations (SDEs) with Levy process perturbation noise. The authors emphasize that unlike the Wiener process, whose sample paths should be continuous, the sample paths of the Levy process may be discontinuous (jumps) at multiple locations. Consequently, common differential equation techniques may not be applicable. Addressing several technical challenges associated with this SDE formulation, the paper introduces the \"fractional score function,\" replacing the conventional score function in the drift term of the reverse process.\n\nFurthermore, the paper proposes fractional denoising score matching (fractional DSM) to approximate the fractional score in the reverse process. This approach may serve as a more general version of DSM.\n\nAdditionally, the authors derive a probability flow formulation for the reverse SDE, similar to conventional SBGMs. This demonstrates that the proposed method can leverage other fast sampling techniques developed for SBGMs, such as advanced integrators.\n\nFinally, the paper presents several experiments to showcase the proposed methods' effectiveness. For instance, mode estimations and sample diversities are evaluated to compare the proposed method to the previous Wiener process-based approaches.\n\nSTRENGTHS:\nOverall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. The authors have succeeded in providing a fresh perspective on the topic, shedding new light on the subject matter and offering valuable contributions to the machine learning communities.\n\nWEAKNESSES:\nIn light of the overall quality of the paper, the experimental parts would benefit from further refinement.\n\nFirst of all, the discussion on convergence rate can be improved. For example, regarding the results in Figure 7, the convergence rate may be influenced by various aspects, such as network architectures, noise scheduling, or the quality of the trained models. In this aspect, analysis on toy datasets would be more beneficial to provide better evidence.\n\nSecondly, there is room for improvement in the experiments related to Figure 3. It is important to note that FID utilizes classifiers trained on ImageNet datasets. This raises questions about the suitability of FID in analyzing the results for mixtures of Gaussians. I believe a distance metric like MMD may be more appropriate for this purpose.\n\nLastly, there should be more in-depth discussions about the experiments on image generation benchmark datasets. In Table 1, the transition from Wiener to Levy process in the CIFAR-10 results shows minimal improvement in \"recall.\" However, considering that other values exhibit more significant variations due to network architecture, it becomes necessary to determine the significance of this difference. It is also worth exploring whether the observed differences could be attributed to variations in noise scheduling. Similar patterns also emerge in the results for the CelebA dataset, warranting further analysis and discussion.\n\nQUESTIONS:\n- It needs to be clarified what evidence supports the claim that conventional SBGM exhibits mode collapsing, as mentioned in lines 28-32 of the paper.\n- In lines 213-214, it is stated that noise scheduling was performed using the cosine function. However, later in the paper, it is mentioned that VP (presumably referring to a different noise scheduling method) was used. This inconsistency needs to be clarified about the accurate statement.\n- Section 5.2 should precede Section 5.1 to explain better the arguments the paper aims to address. By presenting Section 5.2 before Section 5.1, the authors can provide a clearer context and explanation of the differences they intend to discuss. This reordering would enhance the coherence and logical flow of the paper.\n- The positioning of the images in Figure 5 and the corresponding columns in the table appears to be reversed, leading to confusion. The inconsistency between the image positions and table columns can create difficulties understanding and interpreting the data.\n- There is a mismatch between the order in Tables 2 & 3 and their actual sequence in the document. This inconsistency can cause confusion and make it challenging for readers to locate and reference the correct tables.\n- Regarding the differences presented in Tables 2 & 3, it is unclear whether they can be considered statistically significant. 1. Further statistical tests or discussions of the observed variations are necessary to determine the significance of the findings.\n\nLIMITATIONS:\nPlease refer to the comments provided in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrior score-based/diffusion generative models have been defined by Brownian motion. This paper proposes a method of replacing the continuous Gaussian processes with different processes dependent on the characteristic exponent value. The heavy tail property defined by the Levy process allows for higher chance of making larger steps, thus inducing more diverse and faithful sample of the true distribution of the model than its Weiner process counterpart. \n\nThe author shows that the model converges faster due to the Levy process having the capability of making large jumps during the forward and reverse process. The resulting model displays promising qualitative and quantitative results in datasets such as CIFAR10 and CelebA and synthetic datasets.\n\nSTRENGTHS:\n- Overall, the paper is well written/organized with clear explanation on the theoretical foundation.\n- Experiments are very well thought out and very useful in showing the advantages of utilizing a Levy stable distribution.\n- Image metrics show extremely promising results (i.e., FID score).\n\nWEAKNESSES:\n- Table 5 in the appendix shows promising FID scores but are heavily dependent on the optimal alpha value. It would be nice if we could get all alpha FID results for Table 1 and not just the optimal alpha value FID score. This would help us understand whether Levy process performs generally better (i.e., when alpha is between 1 and 2) than the baseline diffusion model (i.e., alpha equal 2) or if this seems more like a sensitive hyperparameter for best results.\n\nQUESTIONS:\n- Seen in Figure 3, why are diffusion models bad at faithfully learning the true data distribution? It would be nice if there were more insight on the lackluster performance of diffusion models based on the Weiner process interpretation. Theoretically, it seems like it should perform just as well as the Levy process, could it be due to the number of steps taken by the ODE solver?\n- Can the score function learned from the Levy noising process be compatible with other models such as using this learned score function for DDIM inferencing?\n-  Why doesn't the heavy tail distribution create instability to the diffusion model path? Line 53-54 states that the variation becomes smaller as it approaches the sample space but wouldn't this also have adverse affect for finer pixel detailing due to the heavy tail? (I have limited knowledge of Levy process, so I may just have misunderstood). (I am willing to raise up my score)\n\nLIMITATIONS:\nAdequately addresses all limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new score-based generative model called Lévy-Ito ̄ Model (LIM) that tackles the challenges of slow convergence rate of Number of Function Evaluation (NFE) and mode collapse in diffusion models when applied to imbalanced data. This model leverages isotropic-stable Lévy processes. Initially, the authors derive a precise stochastic differential equation in reverse-time, driven by the Lévy process, and subsequently establish the corresponding fractional denoising score matching technique. The proposed generative model harnesses the advantageous heavy-tailed characteristics of the Lévy process. The experimental findings demonstrate that LIM achieves faster and more diverse sampling, while maintaining exceptional fidelity when compared to existing diffusion models across a range of image datasets.\n\nSTRENGTHS:\n1. This paper investigates a novel optimal non-Gaussian stochastic process(isotropic alpha-stable Lévy processes) for injecting noise, and prove the exact time-reversal formula of SDEs driven by Lévy process.\n2. Based on isotropic alpha-stable Lévy processes， this paper proposes a novel score-based diffusion model called Lévy-Ito ̄ Model (LIM).\n3. Compared to existing diffusion models, LIM offers faster and more diverse sampling capabilities while maintaining high fidelity across a range of image datasets.\n\nWEAKNESSES:\n1. In P2-L31, the authors make claims about the slow convergence rate and mode-collapse issues of previous score-based diffusion models without providing sufficient explanation or evidence to support these claims. \n2. In Figure 3, the authors doesn't explain the significance and differences of the two blue clusters in each of the three different subplots. And how is the FID calculated?\n3. The related work section provides limited information and lacks a detailed investigation of the recent developments in the last two years.\n4. In P5-L156, there are grammatical errors, such as repetitive words and incorrect prepositions. These should be corrected for clarity and readability.\n5. Indeed, the experiments seem somewhat limited as they only validate the results on low-resolution datasets such as CIFAR-10 (32x32) and CalebA (64x64). \n6. In Appendix C, on P23, Equation 112 is identified as incorrect. The authors should revise the equation or provide the correct version to ensure the accuracy of the paper.\n\nQUESTIONS:\nPlease refer to Weaknesses\n\nLIMITATIONS:\nYes, the authors adequately discussed the limitations of this paper. However, potential negative societal impacts are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-19"}