SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The drift term of the reverse-time SDEs can be accurately determined by training  $\mathbf{s}_t(\mathbf{x}_t;\theta)$  to match  $S_{t}^{(\alpha)}(\mathbf{x}_{t}|\mathbf{x}_{0})$ , as stated in Theorem 4.3. Moreover, if we have the jump-type Lévy process  $L_{t}^{\alpha}$ , we can calculate  $S_{t}^{(\alpha)}(\mathbf{x}_{t}|\mathbf{x}_{0})$  explicitly as a linear expression.
Theorem 4.4. Let  $1 < \alpha < 2$ . If the transition density function  $p_t(\mathbf{x}_t|\mathbf{x}_0)$  follows (9), then the fractional score function of  $p_t(\mathbf{x}_t|\mathbf{x}_0)$  can be represented as a linear form such that
$$
S _ {t} ^ {(\alpha)} (\mathbf {x} _ {t} | \mathbf {x} _ {0}) = \frac {\Delta^ {\frac {\alpha - 2}{2}} \nabla p _ {t} (\mathbf {x} _ {t} | \mathbf {x} _ {0})}{p _ {t} (\mathbf {x} _ {t} | \mathbf {x} _ {0})} = - \frac {1}{\gamma^ {\alpha - 1} (t)} \cdot \frac {1}{\alpha} \cdot \left(\frac {\mathbf {x} _ {t} - a (t) \mathbf {x} _ {0}}{\gamma (t)}\right). \tag {12}
$$
Theorem 4.4 implies that the computational cost of training a fractional score model  $\mathbf{s}_t(\mathbf{x}_t;\theta)$  is similar to DSM [40].
4.3 Loss function
We set beta schedule as cosine beta schedule  $\beta(t)$  [25], following  $\beta(t) = \frac{\alpha}{1 + s} \frac{\pi}{2} \tan \left( \frac{t + s}{1 + s} \frac{\pi}{2} \right)$  for small number  $0 < s \ll 1$ . Then, the drift term  $a(t) = \frac{\cos \left( \frac{t + s}{1 + s} \frac{\pi}{2} \right)}{\cos \left( \frac{s}{1 + s} \frac{\pi}{2} \right)}$  and the diffusion term  $\gamma(t) = (1 - a(t)^{\alpha})^{\frac{1}{\alpha}}$ . We use the U-net architecture as in DDPM [15] for training and apply the loss (11) to train  $\mathbf{s}_t(\mathbf{x}_t; \theta)$ . For  $\epsilon \sim S_{\alpha} S^d(1)$  and  $x_0 \sim p_{\mathrm{data}}$ ,  $\mathbf{x}_t = a(t) \mathbf{x}_0 + \gamma(t) \epsilon$ ,  $U(0,1)$  denotes a uniform distribution. Then, the loss with the relative weight  $\gamma(t)$  is defined as
$$
L (\theta ; \gamma (t)) = \mathbb {E} _ {t \sim U (0, 1)} \mathbb {E} _ {x _ {0} \sim p _ {\mathrm {d a t a}}} \mathbb {E} _ {\epsilon \sim \mathcal {S} \alpha \mathcal {S} ^ {d} (1)} \left\| \gamma^ {\alpha - 1} (t) \mathbf {s} _ {t} (\mathbf {x} _ {t}; \theta) + \frac {\epsilon}{\alpha} \right\| _ {2} ^ {2}. \tag {13}
$$
The fractional score model  $\mathbf{s}_t(\mathbf{x}_t;\theta)$  learns the law of the solution  $X_{t}$  for each  $t$  because the betaschedule version solution used during training is the weak solution of the given forward SDEs.
We employ continuous time step throughout our experiments and set NFE to a fixed value of 500 in all cases. Given that Lévy processes exhibit a higher occurrence of extreme values compared to Gaussian noise, we opt for the smooth  $L_{1}$  Loss instead of the  $L_{2}$  Loss to ensure stable training. We use the modified VP-SDE to fit the Lévy process and quadratic timestep during reverse sampling.
5.1 Mode Estimation
We conduct the class imbalance experiment for mode estimation with two kinds of datasets. First is customized imbalanced CIFAR10 with a ratio given of 1:2:3:4:5. We choose t-distributed stochastic neighbor embedding (t-SNE), which reduces complex data of high dimensions to two dimensions. In table 1 and Figure 5, the mode estimation of LIM is much closer to training data than the diffusion model. Also, the corresponding FID, Recall, and MMD of LIM show improved performances. The second dataset is CIFAR10LT including 10 classes, which follow heavy-tailed distribution
Figure 5: t-SNE and evaluation in the imbalanced CIFAR10. LIM (blue), DM (red), and training data (green).
5.1.1 Sample Diversity
4. Table 3 shows that as we move from class 0 with a high ratio to class 9 with a low ratio, the overall performance tends to decrease. However, LIM shows better performances compared to the diffusion model with respect to FID and Recall. These results demonstrate that LIM is more robust for mode estimation in imbalanced datasets, and alleviates the mode-collapse issue.
4The ratio is given by 5000:2997:1796:1077:645:387:232:139:83:50
Metric	Model	airplane	automobile	bird	cat	deer	dog	frog	horse	ship	truck
FID↓	DM	13.92	8.87	14.58	15.44	10.35	16.48	13.19	11.05	9.50	8.38
LIM	13.92	8.24	15.02	16.07	10.05	18.24	13.23	9.77	9.18	7.76
Recall↑	DM	0.6646	0.6238	0.6968	0.6076	0.6936	0.6792	0.6816	0.6734	0.6342	0.6098
LIM	0.6912	0.6516	0.7138	0.6272	0.6976	0.7212	0.7082	0.7040	0.6472	0.6366
Table 2: Results of conditional generation on CIFAR10(32 × 32).
Metric	Model	airplane	automobile	bird	cat	deer	dog	frog	horse	ship	truck
FID↓	DM	22.00	12.69	30.55	37.14	22.08	49.31	38.67	44.82	50.97	36.08
LIM	15.594	9.5131	19.928	25.66	15.90	33.5393	29.46	24.88	30.34	34.302
Recall↑	DM	0.6412	0.5716	0.6500	0.5200	0.6240	0.5828	0.5796	0.6246	0.5890	0.6046
LIM	0.6768	0.6118	0.6778	0.5696	0.6248	0.6578	0.5686	0.6100	0.5906	0.6084
We choose Recall as an evaluation metric for sample diversity. In Table 5, we can see that LIM exceeds other diffusion models in Recall. Moreover, even in the conditional generation, it shows higher Recall for each class, proving that the generated samples are more diverse than those of the diffusion model (Table 2).
Table 3: Results of conditional generation on CIFAR10LT(32 × 32).
Metric	LIM	DM
FID↓	21.07	62.62
Recall↑	0.5549	0.5002
MMD↓	0.00416	0.01396
Table 1: Imbalanced CIFAR10 results.
Imputation The imputation refers to the task of restoring the masked regions in an image. To measure the diversity of samples, we use Learned Perceptual Image Patch Similarity (LPIPS) as a metric, where a higher LPIPS indicates greater diversity and a lower value implies more similarity between samples. For each original image  $x$ , we generate 10 imputed images  $\{\hat{x}_i\}_{i=1}^{10}$  and calculate the average of LPIPS for the 10 samples. As shown in Figure 6, the diffusion model tends to generate similar eyes and mouths in the masked region, while LIM produces a much wider range of diverse images and shows higher LPIPS.
Figure 6: Imputation images on CelebA-HQ dataset. Diffusion model(DM) generates only similar eyes and lips, while LIM generates a much more diverse range of shape and color of eyes and lips. LPIPS $\uparrow$  of DM is 0.0132, but LIM is 0.0197.
5.2 Convergence rate for NFE
We compare LIM and diffusion models on CIFAR10  $(32 \times 32)$  and CelebA  $(64 \times 64)$  dataset in terms of convergence rate for NFE. Figure 7 shows that FID score of LIM drops at earlier NFE than the baselines in SDE-based sampling. Moreover, it shows a competitive convergence rate for NFE compared to ODE-based models. These experimental results demonstrate that LIM is capable of converging to the sample space faster than the baselines. See Appendix G.6 to figure out the wall clock time per NFE on various datasets.
(b) CelebA 64x64
Figure 7: Convergence rate for NFE measured by FID score on CIFAR10 and CelebA.
5.3 Sample Quality
Table 5 displays the performance of LIM in comparison to previous approaches for unconditional image generation on CIFAR10  $(32 \times 32)$  and CelebA  $(64 \times 64)$ . We measure FID scores and Recall using reverse sampling and generate  $50k$  samples for all datasets. For CelebA, we randomly select  $50k$  samples from the training dataset with 5 iterations. We evaluate NLL using a uniformly dequantized test dataset and calculate the log-likelihood through the average of 5 iterations using the ODE solver [39]. Since it is difficult to numerically calculate the density function of the isotropic  $\alpha$ -stable distribution in higher dimensions  $(d > 100)$  [26], we approximate the density function with non-isotropic  $\alpha$ -stable distributions to evaluate the NLL of LIM. For CIFAR10, we train 2 backbone models: DDPM [15] and NCSN++(deep) [39]. See Appendix G for more detail.
The performance of LIM can differ with different  $\alpha$ . Indeed, this seems intuitively natural since images with low resolution less prefer large jumps, while images with high resolution and multiple modalities can benefit exploration from large jumps. Table 4 provides a comparison of LIM's performance across different  $\alpha$  values. Although the overall tendency of declining FID with lower values remains consistent, it appears that differences in FID it-
α	CIFAR10	CelebA
1.2	5.15	2.99
1.5	2.86	1.57
1.8	2.44	3.21
self arises when using different architectures on CIFAR10  $(32\times 32)$ . In spite of using half of NFE used in [39], LIM shows competitive FID score, Recall, and NLL. Especially in CelebA  $(64\times 64)$ , LIM achieves FID score of 1.57 and Recall of 0.7007. It is an impressive performance since we do not use any other guidance or improved model architecture. We also conducted performance comparisons between diffusion model [39] and LIM using the ADM architecture [9] on ImageNet  $(64\times 64)$ . Diffusion model [39] achieves a FID of 14.23, Precision of 0.6711, and Recall 0.6932. In constraint, LIM achieves a FID of 12.97, Precision of 0.6782, and Recall of 0.6782. For comparing the performance on a high-resolution dataset, we chose the DDPM architecture [15], and trained LIM and diffusion model [39] on the CelebA-HQ dataset  $(256\times 256)$ . This indicates that when it comes to CelebA-HQ, LIM attains an FID score of 7.76 with 500 NFE, whereas the diffusion model achieves a higher FID of 11.87 with 1000 NFE based on DDPM architecture [15].
5.4 Deterministic ODE sampling
We have the option to utilize DDIM inference for our model. When we use a pretrained model on CelebA (64x64), Table 6 displays the FID scores acquired by DDIM and LIM-DDIM for various NFEs. The results demonstrate that LIM-DDIM outperforms DDIM in terms of performance.
Table 4:  $\alpha$  selection on CIFAR10  $(32\times 32)$  and CelebA  $(64\times 64)$ .
NFE	20	50	100	200
DDIM	6.64	5.23	-	4.78
LIM-DDIM	6.73	4.80	3.95	-
Table 6: DDIM and LIM-DDIM with  $\alpha  = {1.5}$  on CelebA (64×64).
Figure 8: Generated samples on CelebA-HQ  $256 \times 256$ , LSUN-Church  $256 \times 256$ , and LSUN-Bedroom  $256 \times 256$ .
Model	Noise	NFE	FID↓	Recall↑	†NLL(bits/dim)↓
CIFAR10 (32 × 32)
DDPM [15]	Gaussian	1000	3.17	-	≤ 3.70
DDPM cont. (VP) [39]	Gaussian	1000	3.24	0.6784	3.21
NCSN++ cont. (deep, VP) [39]	Gaussian	1000	2.44	0.6860	3.13
LIM-DDPM cont. (Ours)	Lévy (α=1.8)	500	3.37	0.6874	3.08
LIM-NCSN++ cont. (deep, Ours)	Lévy (α=1.8)	500	2.44	0.6960	2.97
CelebA (64 × 64)
DDPM [15]	Gaussian	1000	3.50	-	-
DDPM cont. (VP) [39]	Gaussian	1000	3.21	0.6437	2.39
LIM-DDPM cont. (Ours)	Lévy (α=1.5)	500	1.57	0.7007	2.23
Table 5: Unconditional generation on CIFAR10  $(32\times 32)$  and CelebA  $(64\times 64)$ . †NLL is approximated by using non-isotropic  $\alpha$ -stable distribution as a prior density.
Many diffusion models have been conducted to find the best model architectures such as NCSN++ [39] and ADM [9]. In contrast, Lévy process differs significantly from Brownian motion. Therefore, it is uncertain whether using the same model architecture that has been used previously is the best approach for LIM. Although, in this study, we use the same structure of the existing diffusion model to compare the pure effect of the Lévy process, there is a need to find a model architecture that is suitable for the Lévy process in order to improve performance further.
The aim of this study is to expand the variety of noise distributions utilized in score-based generative models by introducing an exact time-reversal formula for SDEs with Lévy processes. We propose a new score-based generative model, Lévy-Itô Model (LIM). The effectiveness of this model has been verified by empirical results, which show that LIM performs well with various image datasets. As a result, this study provides a solution and demonstrates the potential for utilizing a wider range of non-Gaussian Markov processes in score-based generative models.
This work was supported by LG AI Research. This work was also supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korean government(MSIT)(No. 2022-0-00612, Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AI; No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University), National Research Foundation of Korean(NRF) funded by the Korean government(MSIT)(2021R1C1C1009256), and a grant from Korea University (K2304791). This work also was supported by Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) and Gwangju Metropolitan City. The authors especially thank Prof. Nicolas Privault for his interest in our paper and valuable discussions that helped us correct errors in our main result. The first author would like to thank Woo-Joo Kim for insightful discussion and anonymous reviewers for their constructive discussion and editorial efforts.
[1] M. Abramowitz and I. A. Stegun. Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Dover Publications, 1964.
[2] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, pages 313-326, 1982. URL https://www.sciencedirect.com/science/article/pii/0304414982900515.
[3] David Applebaum. Lévy processes and stochastic calculus. Cambridge university press, 2009.
[4] Björn Böttcher. Feller evolution systems: Generators and approximation. Stochastics and Dynamics, 14(03):1350025, 2014.
[5] J. C. Butcher. Numerical Differential Equation Methods. John Wiley and Sons, Ltd, 2016. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/9781119121534.ch2.
[6] Peng Chen, Changsong Deng, Rene Schilling, and Lihu Xu. Approximation of the invariant measure of stable sdes by an euler-maruyama scheme, 2022. URL https://arxiv.org/abs/2205.01342.
[7] Giovanni Conforti and Christian Léonard. Time reversal of markov processes with jumps under a finite entropy condition. Stochastic Processes and their Applications, 2021. doi: 10.48550/ARXIV.2104.10948. URL https://arxiv.org/abs/2104.10948.
[8] Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching, 2021.
[9] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
[10] John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
[11] A. Dytso, Bustin, and et al. Analytical properties of generalized gaussian distributions., 2018.
[12] Gerald B. Folland. Real Analysis: Modern Techniques and Their Applications. John Wiley and Sons, 2013.
[13] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy, 2023.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. arXiv preprint arXiv:1706.08500, 12(1), 2017.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf.
[16] Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3):1059-1076, 1989.
[17] Jean Jacod and Philip Protter. Time Reversal on Levy Processes. The Annals of Probability, 16 (2):620 - 641, 1988. doi: 10.1214/aop/1176991776. URL https://doi.org/10.1214/aop/1176991776.
[18] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models, 2023.
[19] Thomas G. Kurtz. Equivalence of stochastic equations and martingale problems. Stochastic Analysis 2010, 2011.
[40] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23:1661-1674, 2011.
[41] Yingjian Wang and Lawrence Carin. Levy measure decompositions for the beta and gamma processes, 2012.
[42] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning, 2023.
[43] Umut Şimşekli. Fractional Langevin monte carlo: Exploring lévy driven stochastic differential equations for markov chain monte carlo, 2017. URL https://arxiv.org/abs/1706.03649.