Improved Sampling Algorithms for Lévy-Itô Diffusion Models

Authors: Vadim Popov, Assel Yermekova, Tasnima Sadekova, Artem Khrapov, Mikhail Sergeevich Kudinov
Primary area: generative models
Keywords: generative modeling, diffusion models, Lévy-Itô models, α-stable Lévy processes, stochastic differential equations

Abstract:
Lévy-Itô denoising diffusion models relying on isotropic α-stable noise instead of Gaussian distribution have recently been shown to improve performance of conventional diffusion models in image generation on imbalanced datasets while performing comparably in the standard settings. However, the stochastic algorithm of sampling from such models consists in solving the stochastic differential equation describing only an approximate inverse of the process of adding α-stable noise to data which may lead to suboptimal performance. In this paper, we derive a parametric family of stochastic differential equations whose solutions have the same marginal densities as those of the forward diffusion and show that the appropriate choice of the parameter values can improve quality of the generated images when the number of reverse diffusion steps is small. Also, we demonstrate that Lévy-Itô diffusion models are applicable to diverse domains and show that a well-trained text-to-speech Lévy-Itô model may have advantages over standard diffusion models on highly imbalanced datasets.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1729817507309

thread_type:
official_review

<thread_object>
creator: reviewer_fdtf

<content>
summary:
A family of inverse dynamics is derived for diffusion models that use isotropic (\alpha)-stable noise instead of Gaussian noise. Such models have been proposed recently with a deterministic inverse dynamics and a stochastic one. The deterministic dynamics is guaranteed to retrieve the exact marginal distributions. Unfortunately, the stochastic dynamics yields only approximate solutions.  This paper addresses this limitation. The algorithm is expected to be effective for data generation on imbalanced datasets. Its practical usefulness is experimentally examined in applications to image generation and text-to-speech tasks.

strengths:
A parametric family of inverse dynamics is derived for diffusion models that use isotropic (\alpha)-stable noise instead of Gaussian noise. Unlike the previously proposed one, this family is guaranteed to exactly retrieve the target marginal distributions for any parameter setting in ideal cases.

weaknesses:
Writing in section 3.3 is sloppy. Since "some intractable data-dependent process Z_t" is not explained sufficiently, it is difficult to grasp the relation between the previously proposed algorithm and the current one.

questions:
The relation between the algorithm by Yoon et al (2023) and the current one is unclear. Please explain more details on how equations (4) and (11) are related.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730012281538

thread_type:
official_review

<thread_object>
creator: reviewer_9gsh

<content>
summary:
The paper offers a novel method for sampling using a Lévy-Itô diffusion model based on a new formulation of its corresponding SDE. The authors identify an issue in existing methods for the reverse SDE caused by neglecting one of the terms, and offer a solution. The paper includes experiments which compare the generation results of the proposed solutions to alternatives, as well as a demonstration of its use in addressing skewed training datasets in the text-to-speech field.

strengths:
- The paper analyzes Lévy processed and their use for diffusion models, a field which is under-explored.
- The proposition in the paper is straightforward and mathematically justified.
- The experiments in the paper provide evidence of the benefits of the new method.

weaknesses:
- The presentation of the paper could be improved, both at the sentence and at the paragraph level.
- The figures and naming scheme makes it hard to follow the result and illustrations in the paper. (specifically in Fig. (4), Fig (1), Tab. (5))
- The takeaway from the experiment shown in Tab. (4) is unclear. While Tab. (3) shows a small advantage for the proposed method, the results in Tab. (4) do not reflect that.
- The evidence for an advantage in underrepresented data could be made stronger (using more examples, specifically in the image domain).

If some of my weaknesses are properly addressed I am inclined to raise my score.

questions:
- I believe it would be easier to follow the comparisons by using a consistent naming scheme, instead of the current SDE(<number>)
- How do extremely small NFEs (5-15) effect the sampling quality of the proposed method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730161252975

thread_type:
official_review

<thread_object>
creator: reviewer_ef2f

<content>
summary:
Levy diffusion models perform better (especially on rare classes) when it’s trained on imbalanced datasets. However, the previous reverse Levy process doesn’t have same marginal probabilities at each noise level due to the omitted intractable term, resulting in non-exact sampling. 
The authors propose a novel parametric family of SDE whose solutions have the same marginal densities as the forward levy diffusion process, leading to exact solutions. 
Empirical experiments demonstrate that the proposed reverse process has superior sample quality with small number of sampling steps, and also performs good in terms of sample diversity.

strengths:
- The paper is well-organized, and the technique derivations are solid. Also most of the arguments are well supported by experiments. 
- Instead of developing a reverse-time process of the forward Levy-Ito SDE, the authors propose parametric reverse-time SDEs that have the same marginal probability densities as the forward process, which gives exact sampling. 
- It seems like compare to the baseline SDE, the proposed one is good with multiple numerical solvers. For example, in Table 1 and 2, SDE(11) performs good with both solvers, while baseline SDE approximate(9) has a different performance with different solvers.

weaknesses:
- Though the improvement of FID scores given small number of sampling steps (e.g. N=20) is huge, the improvement on imbalanced CIFAR10 is marginal. Also, it could be nice to provide some FID scores of samples from Gaussian diffusion models, which shows a clear improvement in imbalanced dataset. 
- In the speech synthesis experiment, the authors compare the proposed model only with the Gaussian-based diffusion model. However, it remains unclear whether this model is still SOTA when compared to the baseline Lévy-Itô diffusion model.

questions:
- In table 3, I wonder how many times the authors run the experiments? It seems that for both FID and coverage metrics, the proposed SDE has very close performance to the baseline SDE. 
- In the speech synthesis experiment, could you provide a comparison to the baseline Lévy-Itô diffusion model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668904937

thread_type:
official_review

<thread_object>
creator: reviewer_7oel

<content>
summary:
This paper introduces a paremetric family of SDEs allowing different sampling scheme from a single pretrained Levy Ito diffusion model. It brings the toolbox for Levy Ito diffusion models (LIM) closer to that of standard "Gaussian" diffusion models (DM). Empirically, the authors find that the new paremetric family of SDEs has benefit in term of generated samples quality and diversity at low number of functional evaluations, which lowers the computational cost of sampling from Levy Ito diffusion models. Finally, they train a text-to-speech diffusion model on an imbalanced dataset and evalutate the benefits of LIMs compared to DM.

strengths:
* The parametric family of SDEs derived by the author is new and offer more flexibility for LIM sampling.
* The authors give a rigorous derivation of the proposed SDEs
* The paper clearly tries to show how their theoretical results compare with existing litterature on LIMs. It explains clearly what is the limitation of Yoon et al (approximation of the reverse process).
* Promising results for text to speech models
* The paper is well written overall

weaknesses:
* The experiments about text to speech models are interesting and promising but they are not tied to the main results of the paper (theoretical and empirical contributions to LIM sampling). The baseline DM against which authors compare is also new and has the modifications outlined in lines 809 to 825 in the Appendix.
* The simple toy example explains well why processes with zero mean and finite variations cannot be omitted completely in low NFE schemes. However, in my opinion, the paper does not explain clearly what steps/changes in the parametric family of SDEs prevents any such (intractable) process to come into play. I understand from line 286 that the reverse time SDE is no longer a model of the forward time SDE and that trajectories are different. Is that what allows you to have "better" SDEs?
* (Not a real weakness per se) The new paremetric family of SDEs does not improve the capabilities of LIM in term of imbalance "correction". As examplified by table 4.

questions:
* Please explain weakness 2.
* For the text to speech model, even on the main mode (female speakers), LIMs outerperform Gaussian DMs. Why is that? Wouldn't we expect Gaussians DM to perform better on the main mode of the distribution? Does fixing the "bias" in the text encoder (line 809 to 825) unfairly disadvantage Gaussians DM even on the main mode? What would the results have been with a standard encoder (or alternatively do you have other baselines for text to speech imbalanced modeling that would shed light on the benefits of LIM on both the main mode and the tail mode)? You are also using an ODE sampler, which, in your image generation experiment, favors the less frequent class. 
* It seems that the common accepted reason for which LIMs are better for imbalanced datasets is that Lévy Ito processes, with their heavy tails and jump possibilities, better cover less probable isolated modes of the data distribution. In your opinion, why does this phenomenon subsist when using ODE sampling? In your imbalanced CIFAR experiment, classes 8 and 9 seem favored by the ODE.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731986134659

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comment and updates summary

comment:
We would like to thank all the reviewers for their valuable comments. We have updated our paper according to some of their suggestions, and here's a brief summary of these updates:

1) We have added a reference to the formula in the paper [1] where the intractable process $Z_{t}$ is defined through the expression of its characteristic exponent as well as a brief comment on it following the suggestion of the Reviewer fdtf who marked this process as unclear and not properly explained.
2) In the section devoted to experimental results we have made the notation more clear and easy to follow and the results easier to compare by replacing ODE(10), SDE(9) and SDE(11) with ODE, SDE-A (approximate SDE) and SDE-E (exact SDE, our method) correspondingly as suggested by the Reviewer 9Gsh. These notation changes also affected Tables 1-4 and Figure 4.
3) Addressing the weakness pointed out by the Reviewer 9Gsh, we have added more discussion on the results from Table 4.
4) We have added FID score obtained by common diffusion model on imbalanced CIFAR10 to serve as a reference.
5) Addressing concerns of the Reviewer EF2f who pointed out that the improvement on imbalanced CIFAR10 is very little, we performed 5 runs of generation on this dataset instead of 1 to obtain more reliable results and updated Tables 3 and 4. They did not change significantly, the proposed method SDE-E still outperforms SDE-A (0.3 FID and 1.5% coverage). In Table 4 coverage for less represented classes (with larger class ids) changed to more extent after 4 additional experiment runs than that for more represented classes because 
coverage is calculated as a percentage of images within a class having at least one generated image as their nearest neighbour, and the number of images in less represented classes is very little (50 for the smallest class with id=9) meaning that variance of the results across different runs is larger.

[1] Score-based Generative Models with Levy Processes, Yoon et al.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a parametric family of SDEs for Lévy-Itô diffusion models to improve the sampling accuracy and efficiency. The proposed method is verified through experiments on imbalanced datasets for different tasks. 

Strengths:
1. The proposed SDE formulation is novel and mathematically rigorous, addressing specific limitations of prior Lévy-Itô diffusion models.

2. The method has good flexibility and can be applied to different tasks such as image generation and speech synthesis.

3. The method shows consistent improvements in sample quality, and robustness in imbalanced datasets.



Weaknesses:

1. The gains in imbalanced datasets are not very big. Nevertheless, they are consistent and meaningful.

2. The comparisons with SOTA methods could be more extensive.

Overall, the paper presents a interesting theoretical contribution with empirical success in  different applications. While some concerns regarding the gains and  baseline comparisons remain, considering the overall quality and novelty of the work, I would like to recommend acceptance.