Heavy-Tailed Diffusion with Denoising Levy Probabilistic Models
Authors: Dario Shariatian
Umut Simsekli
Alain Oliviero Durmus
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion
generative model
deep learning
machine learning
heavy-tail
Primary area: generative models
TL;DR: The paper introduces the Denoising Lévy Probabilistic Model (DLPM), which replaces Gaussian noise in denoising diffusion probabilistic models (DDPM) with heavy-tailed α-stable noise to improve performance, showing better tail coverage

Abstract: Investigating noise distributions beyond Gaussian in diffusion generative models remains an open challenge. The Gaussian case has been a large success experimentally and theoretically, admitting a unified stochastic differential equation (SDE) framework, encompassing score-based and denoising formulations. Recent studies have investigated the potential of \emph{heavy-tailed} noise distributions to mitigate mode collapse and effectively manage datasets exhibiting class imbalance, heavy tails, or prominent outliers. 
Very recently, Yoon et al.\ (NeurIPS 2023), presented the Levy-Ito model (LIM), directly extending the SDE-based framework to a class of heavy-tailed SDEs, where the injected noise followed an $\alpha$-stable distribution -- a rich class of heavy-tailed distributions. 
Despite its theoretical elegance and performance improvements, LIM relies on highly involved mathematical techniques, which may limit its accessibility and hinder its broader adoption and further development. 
In this study, we take a step back, and instead of starting from the SDE formulation, we extend the denoising diffusion probabilistic model (DDPM) by directly replacing the Gaussian noise with $\alpha$-stable noise. 
By using only elementary proof techniques, we show that the proposed approach, \emph{denoising L\'{e}vy probabilistic model} (DLPM) algorithmically boils down to running vanilla DDPM with minor modifications, hence allowing the use of existing implementations with minimal changes. 
Remarkably, as opposed to the Gaussian case, DLPM and LIM yield different training algorithms and different backward processes, leading to distinct sampling algorithms. 
This fundamental difference translates favorably for the performance of DLPM in various aspects: our experiments show that DLPM achieves better coverage of the tails of the data distribution, better generation of unbalanced datasets, and improved computation times requiring significantly smaller number of backward steps.

Published as a conference paper at ICLR 2025
HEAVY-TAILED DIFFUSION WITH DENOISING LÉVY PROBABILISTIC MODELS
Dario Shariatian', Umut Simsekli', Alain Durmus? 'INRIA Department of Computer Science, PSL Research University, Paris, France focole Polytechnique CMAP, IP Paris, Palaiseau, France ddario... shariatian, umut a simsekli @inria. fr alain. dww.uu@poyyeeeeiiieeeeee edu
ABSTRACT
Exploring noise distributions beyond Gaussian in diffusion models remains an open challenge. While Gaussian-based models succeed within unified SDE framework, recent studies suggest that heavy-tailed noise distributions, like a- stable distributions, may better handle mode collapse and effectively manage datasets exhibiting class imbalance, heavy tails, or prominent outliers. Recently, Yoon et al. (NeurIPS 2023), presented the Lévy-Itô model (LIM), directly extend- ing the SDE-based framework to a class of heavy-tailed SDEs, where the injected noise followed a-stable distribution, a rich class of heavy-tailed distributions. However, the LIM framework relies on highly involved mathematical techniques with limited flexibility, potentially hindering broader adoption and further devel- opment. In this study, instead of starting from the SDE formulation, we extend the denoising diffusion probabilistic model (DDPM) by replacing the Gaussian noise with a-stable noise. By using only elementary proof techniques, the proposed approach, Denoising Lévy Probabilistic Model (DLPM), boils down to vanilla DDPM with minor modifications. As opposed to the Gaussian case, DLPM and LIM yield different training algorithms and different backward processes, leading distinct sampling algorithms. These fundamental differences translate favorably for DLPM as compared to LIM: our experiments show improvements in coverage of data distribution tails, better robustness to unbalanced datasets, and improved computation times requiring smaller number of backward steps.
INTRODUCTION
The evolution of generative models has introduced several approaches, with diffusion models emerg- ing one of the most prominent. These models transform data distribution into Gaussian distri- bution via forward noising process and then learn to reverse it. The foundational work on denoising in this context was presented by Sohl-Dickstein et al. (2015), where the goal is to reverse Markov chain that progressively adds Gaussian noise to the data. This framework culminated in denoising diffusion probabilistic models (DDPM) by Ho al. (2020), which demonstrated state-of-the-art performance in image generation, while drawing connections to score matching techniques (Song & Ermon (2020)). A unified theoretical framework, based on stochastic differential equations (SDEs), further integrated score matching and the denoising framework (Song et al. (2021)). Various gener- ative models build up on this framework, improving its performance (Dhariwal & Nichol (2021a); Karras et al. (2022)).
Despite their success, diffusion models exhibit limitations, such as requiring large number of steps (Ho et al. (2020)) and empirically struggling with imbalanced datasets (Zhang et al. (2024); Yoon et al. (2023)), often leading to mode collapse (Dhariwal & Nichol (2021a); Deasy et al. (2022)). Tackling heavy-tailed datasets also presents challenge for diffusion models, the finite-time dif- fusion processes generate finite variance data distributions, which are ill-suited for modeling heavy- tailed data. Such datasets, like financial datasets, could benefit from extensions beyond the Gaussian setting, as demonstrated in Borak et al. (2005). Moreover, although many techniques exist to im- prove quality the expense of diversity (Dhariwal & Nichol (2021b); Song et al. (2023)), by only

---

Published as a conference paper at ICLR 2025
using Gaussian noise, the methods typically require highly nontrivial setups in order to obtain im- proved diversity while maintaining a reasonable quality (e.g., FID score) (Sadat et al. (2024); Nobis et al. (2024)).
Several approaches have been explored to address the limitations of diffusion models, particularly through the use of Gau-ians,,,, heavy-tailed noise distributions. The motivation behind this is that heavy-tailed distributions, which can take on larger values, may reduce the number of sampling steps and better capture multimodal data distributions by identifying isolated modes through large noise injections (Yoon et al. (2023)). These distributions are also capable of modeling extreme events or rare occurrences in the tails, making them suitable for tasks such as audio generation (Chen et al. (2020); Kong et al. (2021)). where rare but important variations in amplitude or pitch (e.g., prosodies) can enhance sample quality and diversity. An early attempt at using heavy-tailed noise distributions was made by Nachmani et al. (2021), who replaced Gaussian noise with Gamma- distributed noise, reducing the number of diffusion steps and improving data diversity. Similarly, Deasy et al. (2022) employed generalized Gaussian distributions in wore-matching formulation to improve robustness on unbalanced datasets. However, despite these promising directions, the performance boosts are limited, since both approaches fail to provide time-reversal formula, either using annealed Langevin dynamics when the score is available, or using heuristics based on the Gaussian approach.
Very recently, partially inspired by Simsekli (2017); Huang et al. (2020b), who employ haavy-tailed SDEs in Monte Carlo sampling for challenging distributions, Yoon al. (2023) extended the SDE framework by replacing the light-tailed Brownian motion with a heavy-tailed driving process, in- troducing the Lévy-Itô model (LIM), improving performance on image data, particularly for unbal- anced datasets, offering gains in metrics like FID and diversity. The tail index a controls the degree of tail heaviness, enabling tunable performance based on the characteristics of the data. When the noising process reduces to the Brownian motion light-tailed), however, whenever a < 2, the process becomes heavy-tailed with infinite variance (notably, heavier-tailed than the distributions explored by Nachmani et al. (2021), Deasy et al. (2022)).
While injecting heavy-tailed, infinite-varians variance noise might seem natural when the data distribution itself is also heavy-tailed, Yoon et al. (2023) further illustrated that the heavy-tailed noise can also be beneficial when sampling from compactly supported data distributions (e.g.. generating images), es- pecially in the presence of class imbalances (i.e., the large 'jumps' introduced by the heavy tails can help finding weakly represented modes). Indeed, perhaps being counter-intuine that a heavy-tailed process can indeed converge to a light-tailed distribution with appropriate care Simsekli 2); Simsekli et al. (2020); Huang et al. (2020a), making them suitable for sampling from broad range of data distributions.
it has been shown
Motivation. While LIM demonstrates promising results, the technical complexity of the time- reversed SDE presents significant challenges. The Lévy process with a < 2 has discontinuous paths and no variance, preventing the use of standard analysis tools. The proof techniques rely on fractional calculus and estimations for nneudo-differeniil operators, which might not be accessible for the broader community of diffusion-based generative models. While the theory is elegant, we argue that the highly technical nature of LIM, originated due to the use of continuous-time processes, might hinder its development. For instance, it is highly non-trivis non-trivial to use arbitrary noise schedules.
Moreover, the loss function used in LIM presents some shortcomings: the theory requires squared le loss, assuming the loss remains finite for the considered neural network. However, this may not always hold, as the noise term is heavy-tailed and admits no variance, potentially leading to infinite loss values. As a result, LIM experiments must revert to an l loss for stable training, suggesting that the original loss function may indeed be unworkable. Additionally, LIM is constrained to isotropic noise, further limiting its flexibility.
To overcome these issues, we propose simpler yet effective alternative for incorporating heavy- tailed distributions, focusing on a discrete-time framework that leverages more elementary mathe- matical tools while maintaining performance improvements over Gaussian- models.
2

---

Published as a conference paper at ICLR 2025
Contributions. As opposed to LIM which extended the SaE-b frame- work, here, we take step back and di- rectly work on the discrete-time DDPM process and replace the Gaussian noise with a-stable noise. More precisely, we propose the following Markov process noising process:
unified
DDPM
Score-hased SDF
o-stable noise
a-stable noise
DLPM (This study)
LIM (Yoon et al. (2023))
not unified
Figure 1: Illustration of available methods.
X; = - 04 (a)
(I)
where (.0) follows multivariate o-stable distribution. When 2, the process recovers the standard Gaussian DDPM, but for a < 2, it introduces heavy-tailed noise with infinite variance. A comparison between DLPM and LIM is provided in Figure 1. Our contributions are follows.
Simplified mathematical framework. Leveraging a property of stable distributions (cf. Theo- rem 1), we decompose (d)) into product of one-dimensional random variable and Gaussian vector. This transformation reduces the forward process to a Gaussian one with modified scaling, allowing us approximate the reverse process using only elementary tools. We call the resulting generative model Denoising Lévy Probabilistic Model (DLPM).
Extension to deterministic sampling. Building on the DDIM framework (Song et al. (2020)), introduce deterministic sampler for DLPM, termed DLIM, which further reduces the number of sampling iterations, boosting efficiency.
Compatibility with existing methods. DLPM maintains compatibility with existing DDPM im- plementations, requiring only minor modifications, making it practical and flexible alternative to LIM. This simplicity extends for instance to noise schedules, which are difficult to manipulate the continuous-time LIM framework.
Distinct Algorithms from LIM. Unlike the Gaussian case, where DDPM and the bose-bbsed SDE formulation are two sides of the same coin (Song al. (2021)), we show that DLPM and LIM result in distinct training algorithms and backward processes. Importantly, as these heavy-tailed distributions admit no variance, we cannot simply use the square lg loss, problem which DLPM carefully addresses.
Improved performances. Thanks to our conditionally Gaussian representation strategy, our net- works are only modeling conditional densities given the heavy-tailed variables. Hence the network not learning heavy-tailed distribution directly, but light-tailed conditional distribution, intuitively an easier task. These differences work in favor of DLPM across several performance aspects, partic- ularly where heavy-tailed noise injections are already known offer advantages. Our experiments show that DLPM provides (i) better coverage of the tails eration of unbalanced datasets, and (iii) faster computation times, requiring fewer backward steps.
data distribution, (ii) improved gen-
2 BACKGROUND ON Q-STABLE DISTRIBUTIONS
The family of a-stable distributions appears as the limiting distribution in the generalized cen- tral limit theorem (Gnedenko & Kolmogorov (1968)). the one dimensional case, an a-stable distributed random variable X is defined through its characteristic function (Samorodnitsky et al. (1996)): for E
E [cix]= exp(inu - low| ipp sgn())) where taan(Ta/2) (2/7T) log loul otherwise
if #
Here, ( a IR is the location parameter (ii) (0,2) is the tail index (iii) > 0 the scale parameter (iv) 3 E [-1, determines the right- or left-skewness, and sgn is the sign function. We denote the a-stable distribution by S1,3(14, o)..
In the case where a < 1 and = 1, the support of the distribution becomes the positive real line (i.e.. the random variable is positive), hence we call this distribution 'positive stable'. On the other hand, in the case where 3 = 0, the distribution Sa,0(H, or) is symmetric around μ4, and denoted by Sa(,,,) Furthermore, in the case 2, the distribution reduces Gaussian (, a)
3

---

Published as a conference paper at ICLR 2025
(i,, )0²), hence it is light-tailed. However, whenever < 2, Sa ()) has heavy tails, i.e, the decay rate of its tail distribution satisfies P([X] > r) ~r as r + oo (see (Nolan, 2020, Theorem 1.2)). This implies that [[[X|]] < so if and only if a <.
As opposed to Gaussians, there are multiple ways of extending the a-stable distributions to the mul- tivariate setting. In this paper, will be interested in two major cases: (i) the isotropic (also called rotationally invariant)' and (ii) the non-isotropic with independent components. These distributions also defined through their respective characteristic functions. The random variable X E Rd is isotropic a-stable if its characteristic function is given by: for all u E R Elexp(iu" X)] exp(iμ" u o" [=]l"), where μ E F is the location parameter and old plays the role of covari- ance m We denote it by X ~ St (p, ola). Similarly, X follows the non-isotropic a-stable distribution S. (p. ola), if for any E Rd, Elexp(iu" X)] exp(iµu
σa E [u,]%). While both of these distributions share similar characteristics, such as having power-law tails with the same exponent, the components of the isotropic case are dependent, which results in significant differ- ence compared to the non non-isotropic case, which has independent coordinates. When 2 both options coincide with multivariate Gaussian.
The following property of stable distributions will form the backbone of our algorithm.
Theorem (See (Samorodnitsky et al., 1996, Equation 2.5.3)). Let a < 2, and let ~ St (n, ota.
Then, X _ μ + A1/2G, where - denotes equality distribution, A Sa/2,1 (0,c)) is a one- and G ~ (o)
dimensional positive stable random variable with == coos//a
This theorem shows that zero-mean, unit-scale isotropic stable random-vector can be equivalently written the product of one dimensional positive stable random variable and standard Gaussian random vector. This fundamental property will have a significant impact in terms of incorporating o-stable noise in DDPMs in simple way as, conditioned Gaussian. We conclude this section by noting that a similar decomposition for the non-isotropic case: if X S% (n, aTa,, then = +u+o a A 1/22 @ G, where @ is the component-wing multiplication and A 1/2 {A! id Ji=1 E Rd is vector with i.i.d. components with A; (O,
A, the distribution of X is just
3 DENOISING LÉVY PROBABILISTIC MODELS
In this section, we develop our algorithm by following a similar route to the development on DDPM: we identify the forward and backward processes associated with (1) and construct a variational approximation for the backward chain for sampling. Here, focus isotropic a-stable noise; however, adaptation to non-isotropic a-stable noise is straightforward, as all our derivations rely on Theorem 1, hence it is omitted.
3.1 MARKOVIAN FORWARD PROCESS
Recall that DLPM is based on the following recursion (restatement of (1)):
Xo ~ P+ and X, xx= + +are
where ps is the data distribution and (c(a))T independent and distributed S (0, Ta)- Thanks to the 'stability' property of a-stable distributions, ie., the sum of two a-stable random variables is still a-stable (see Appendix B for details), we can explicitly characterize the con- ditional distribution of X, given Xo. Setting for any E {1....,Th TT. N1-1t := I 2t and we show in Proposition that:
01-44 (710/11111/
X, 4 No + 01-tEE
where & ~ S (0,0 Ta). Similarly to DDPM, the noising schedule parameters ((1, ot) and the horizon T are set so that the final distribution of Xx is approximately equal St (0, Ia),
'The noise distribution used LIM the isotropic a-stable distribution. Note that our framework allows for different types of a-stable distributions by following a single mathematical recipe. in this isotropic case the components are not independent even though the covariance matrix is diagonal.
4

---

Published as a conference paper at ICLR 2025
choosing either (i) M1-1 + 0 as increases, or (ii) M1-1 1 and 01-1t increasing with t. Following the terminology used in DDPM', we refer to schedule (i) as scale preserving and schedule (ii) as scale exploding.
3.2 GENERATIVE PROCESS
Once the forward process is run for large enough time-steps T, it is clear that Xy will be approxi- mately stable-distinn distributed. Hence, to go back to the data distribution pe, we need to time-revert the forward process so that the reversed process can take some o-stable noise and generate a sample from px, which is ultimate goal. More precisely, aim to find a backward process associated to
the Markov chain {X, i.e., Markov chain {X+) such that the two processes {Xr-t)'= and {X have the same distributions. Since, by (Nolan, 2010, Theorem 1.9), any non-degenerate o-stable distribution has smooth density with respect to the Lebesgue measure, the Markov chain
(2) admits a transition density denoted by by p
k(aa "fjt-1- In addition, X, admits a density as well, denoted
(a) and with transition Then, it can be easily verified that Markov process starting from PT" densities, for I E {O....,
"t-ije ( x p| ( (x,, (#e, 22-1) for any It-1, backward process associated with (Xe) where x denotes equality up to a normalization constant. As in the case of DDPM, this backward transition densities are unfortunately intractable, hence, we will develop a variational scheme for their approximation.
1}. To (a)
3.2.1 APPROXIMATION OF THE BACKWARD TRANSITION DENSITIES IN DDPM
To ease the introduction of our approach, let first recall the strategy taken in DDPM, which approximates the backward kernels by relying on a variational approximation for Te (2u:r) = 2 and Io.T : (Zo. ..T E [T+(11 More precisely, the goal is to find the 'closest' distribution to To (a) }), indexed by parameter 0 taking values in some parameter space θ (typically taken neural network).
Pr " ((T) Il (.00) (x==1/x0), where
0:7" family of distributions (
The variational family is assumed to have the same decomposition as T. (a) 0:T b (Zo.T), thus such (+() (1=1)((--111zr) where 77 is chosen be the density of (f ydt) as an approximation of p Then, B is obtained by minimizing the following objec- tive function (Ho al., 2020, Equation 5):
that ((.())
(D) with LD EJKL(K ( (1x,,, X.) X
(3)
t=
k-110 denotes the conditional density of X,-1 given Xo and X,. As 2 in this case, k.(0) k is Gaussian (Ho et al., 2020, Equation 6,7),
where KL denotes the Kullback-Leibler divergence and (a)
motivating the choice of Gaussian densities T (z,- 1 as elements of the variational family at hand, since one obtains a closed-form formula for the KT, terms, i.e.,
$ 1 (Z 12x) = &a 1 (2)))
where (I, + (a Σ) is the density of the d-dimensional Gaussian distribution with mean and covariance matrix Σ. and m Σイ- are functions of x; parameterized by 0. This approach (a) relies the fact that k is analytically tractable. Unfortunately, when < 2, it is not the case
anymore. We now expose our methodology address this limitation.
3.2.2 A DATA AUGMENTATION APPROACH
To obtain a tractable objective function for learning variational approximation of the backward transition densities, we rely on a data augmentation approach, which a classical MCMC technique
In the DDPM literature, these noising schedules are referred to as variance preserving, or variance explod- ing. We the 'scalc' instead of the variance here, since the variance does not exist when a < 2.
5

---

Published as a conference paper at ICLR 2025
(see Brooks al. (2011), Chapter 10). Consider the Markov chain Yo P.. and for {l,...,T),
Y 7e + o,A,
(5)
where {GeTT= and {At) independent random variables, distributed according to Gt (N Ia), and A, 10, with CΛ cosa/"(Ta/4 From Theorem 1, {Yo) is Markov chain that admits the same distribution {X+}]=o- As result, conditioned on [ and Yo.
[Ki]z= is Markov chain with Gaussian transition densities:
( 0, (1h.r (ou a1:r) II Da(ue 1. 1(
Again, we can explicitly characterize the conditional distribution of Y, given Yo, (A.)T=1 Setting for any t E {1...., T,, Ei-1(a = 1 1/2
0k/ x we show in Proposition 5 that:
Y, = + (A.)(A
(6)
where Et (Q, We propose approximating the backward process associated (Y,) given {At)=-1 adapting the DDPM approach. This time, for the backward process, we use the conditional density of 1 given Yo, Y, and AI:r!
T(a) 1:710.a (31:T 180. 18. a1r)) == (o) (UT) II k 1-1,,, (1 a1:e),
where k (()) (--1j0,,,a Nt. yo, a1:)) is now the tractable density of Gaussian distribution:
Proposition The density of the backward process associated to {Y:lT- given Yo, {A,)T- de- noted by k(()) 1 (1yt, Vo, ), is the density of Gaussian distribution ( (m; Ez-1), with mean m and variance Σn-1 equal
1
1 Vo. a1:e)
(ve
vo)) {a = ((-
7
where Ei is in (6), r, = 7o -1---- and e, (y1,, (y+- 71-1300/0011
See Appendix C.3 for the derivations required for Proposition For our generative model, we recon- sider the family of Gaussian variational approximation introduced in (4), modified to account for an
iid. sequence {At) ' forr(woo (Vo.T) == 7 (UT) I Ti_ t.a. (#t-1 1 a1 14 ( (a.r))da1:
where 4/ ( (AIT) = II (n(()),)) and up(a) denotes the density of Sa/2,1 (0, CA), and 7 Th t,a (Ve- 1l5,, ana) = 1 (Vt, anz), 1 (Vt, a1:{)
3.3 VARIATIONAL INFERENCE OBJECTIVE
We consider the following loss function:
Ll E Σ (0,
where
LLL (0, A = E KL [ (-18, Yo, Ald) |l 42 T- (.Y., Aut
and r 0, ((a) 1-1,,1.4 denotes the conditional density of Y given Yo, Y, and Aut. In order to ensure that the expectations with respect to Air are finite, we need to choose r < a when 21, For simplicity, in the rest of the paper, we will use r=
A comparison with DDPM's loss function DD (cf. (3)) immediately illustrates that, thanks to The- orem 1, L is almost identical to LD up to taking expectations with respect to one-dimensional random variables and the taking square-root of the summands'. We show in Appendix C.4.3 that, alike DDPM, our loss is obtained from a KL minimization principle, serving as an upper bound to:
E (KL
(0) (-IALT) T (-((()))))))
WWe note that, with the choices of and 1, exactly recover DDPM.

---

Published as a conference paper at ICLR 2025
when (0, 1]. This additionally shows how any zero of the loss corresponds perfect generative model, while maintaining a similar objective function. The crucial property of (7) is that, since both ((a) ---1jt,0,a and ( are Gaussian (thanks to the conditioning), the KL term admits closed-form analytical formula, as in the case of DDPM.
From a practical perspective, (7) suggests that we can use the same software architecture as for DDPM, with slight modification compute the outer expectation, which can be simply estimated by a Monte Carlo, or median-of-meat procedure (Lugosi & Mendelson (2019)). In order to obtain final denoising training loss, we provide three design choices, for which the full details are given in Appendix C.4. They are similar to what is classically done in diffusion models (Ho et al. (2020); Nichol & Dhariwal (2021); Karras et al. (2022)):
DI. We set fixed variance Σ Σn for the reverse process.
D2. We reparameterize the model to predict the value of et (3 yo) with a network et setting
m (1.) = "te
(Y, 106-01--ll ( (Ye, A1:))
Morcover, drop the dependency of El on {Ar)==1 making Ei only function of Y, This enables re-using classical diffusion models network architectures.
D3. Assuming DI, D2, obtain ())
A1æ) - Ec -Y,, Y] where Ex, and (V yo) = (ye - >1- (01100//1--/ 01-42. We then fix At, = 1.
E AA |1e
X,a =
Proposition (Simplified denoising loss). With the design choices DI, D2, D3, we obtain the simplified denoising objective function:
GSSmmple (0) DB
cYY,, Yo)l A1..) 2/2
where the model E is designed to fit the noise (Y,,, Yo) = (Y, - (o) added at time-step and "/1-,, 01-st are as given Proposition 4. Thus the model is not learning heavy-tailed tailed distribution directly, but the light-tailed conditional distribution.
See Appendix C.4.2 for the derivations required for Proposition 2. Finally, in Appendix C.5, we show that on some conditions, satisfied under design choices D1, D2, D3, the expectation of each term in LL can rewritten as an expectation with respect to only one univariate random variable (as opposed to / variables, i.e., AI:)), reducing the additional computational burden of accommodating heavy tails. As we will estimate the expectations by Monte Carlo averaging, reducing the number of random variables in the expectation is equally important to reduce the error in the estimation. The resulting loss is given in Proposition 9.
3.4 DETERMINISTIC SAMPLING WITH DLIM
Using the techniques in denoising diffusion implicit models (DDIM, Song al. (2020)), we can recover a deterministic sampling scheme. We call this algorithmic extension Denoising Lévy Implicit Models (DLIM), which details are given in Appendix D. We recapitulate the resulting sampling and training algorithms in Appendix A. In this context and in the case of the non-isotropic Cauchy distribution (a = 1), it is possible to bypass the data augmentation technique, since Mosed-form KL divergence between two such distributions exists. Full details about these Cauchy DLIM are given in Appendix D.4, but we leave their experimental exploration to future work.
3.5
COMPARING DLPM TO LIM
The objective function (7) is slightly different from the one obtained in the continuous setting of LIM by Yoon et al. (2023). The training equations are very similar, and can be reformulated to involve denoising loss (see Appendix 1. and (29)):
h, 9 + E (Il²9 (X,) - e (x,, As a refresher, in the case DDPM, one sets p = 2, 17 = 2. In the case of DLPM, our discussion leads to the choice p 2, (see (8)). In the case of LIM, the theory relies squared

---

Published as a conference paper at ICLR 2025
lg loss, setting p = 2 2 in order to properly derive the loss and effectively approximate the true score of the data, at various noisescales. One must therefore make the assumption that L is not infinite for each parameter 9 considered, which may not hold since e, Xx), is a-stable distributed and admits no variance. In the experiments for LIM, one is forced to revert to an & loss, by setting 1) 1 1, obtain stable training, potentially indicating that the loss is infinite.
Additionally, DLPM and LIM yield different backward processes, which in turn lead to distinct sam- pling algorithms cf. Table 4 in the Appendix. Finally, the LIM framework can only accommodate isotropic noise. We refer to Appendix E. for a detailed comparison between DLPM and LIM.
4 EXPERIMENTS
After the groundwork of the previous sections, we design experiments to demonstrate the practical strengths of our DLPM approach as compared to LIM,, apart from its technical simplicity. We recall that setting a 2 simply reverts LIM to classical diffusion, and DLPM reverts to DDPM, appart from the square-root in the loss function. As specified in (Yoon et al. (2023), Appendix G), LIM relies on gradient and noise clipping, which introduces extra hyperparameters that must be fine- tuned for each dataset. In the experiments we use these clipping parameters only when specified. The experimental details relative to this section are available in Appendix G.
As our loss function (7) involves an expectation with respect to AITT, we propose estimating it by using the median-of-nen
estimator, which is known to have better performance for heavy-tailed distributions (I (Lugosi & Mendelson (2019)). For an integer M, this approach requires sampling M² take
many Arr terms, then split them into M groups of size M. To approximate the expectation, the sample mean of each group, and finally take the median of the computed M sample means. In our experiments, we explore M (approximating the expectation with only one sample), denoted simply by DLPM, and M 5, denoted by DLPM,. Similarly, we denote DLIM and DLIM; for the corresponding deterministic sampling schemes.
Finally, we consider the range 1.5 & & 2.0. In our experiments on images, we make use of the dataset CIFARIO 1 LT (long tail), that has been introduced in Yoon et al. (2023) as an unbalanced modification of the CIFAR10 dataset.
4.1 DATA COVERAGE AND MODE COLLAPSE IN TWO-DIMENSIONAL DATA
Before progressing to higher dimensional problems, we start with easily controlled and visualized two-dimensional datasets, in order to validate the competitiveness of our method in the contexts where heavy-tailed diffusions are of interest. In particular, we consider heavy-tailed and unbalanced multi-modal datasets. See Appendix G.1 for details about the experimental setup these settings.
Enhancing data coverage: capturing the tail of the dis- tribution. We set = 2 and generate data points dis- tributed S,, 0.05 Ia). with = 1.7. Our aim is to test the ability of each method to cover the dataset cor- rectly; the main challenge As we can visually observe in Figure 2, the backward dif- fusion process in the Gaussian case cannot produce truly heavy-tailed data, mainly stemming from the fact that its variance always finite. As expected, we see improve- ments when using noise with < 2.
Figure 2: DLPM with a 1.7 and 2.. The lighter-tailed process fails to capture the distribution's tail.
correctly capture the tails.
DLPM, a=1.7
DLPM, a=2.0
To quantify this behaviour, we utilize the mean square logarithmic error (MSLE) metric, which compares the relative error between the higher quantiles (i.e., the quan- tiles corresponding to the tail region that has probability 1- Q) of the true and the generated data distribution (see Appendix G.3 for detailed definitions). We observe in Table how, a gets smaller, one gets better tail approximation. Furthermore, DL.MM consistently outperforms LIM, indicating that the generation process benefits from the heavy-tailed denoising formulation, rather than the continuous-tin tiime one, in this setting.
Original Generated data
Original data Generafed data
8

---

Published as a conference paper at ICLR 2025
Method
1.5
1.6
1.7
1.8
1.9
2.0
DLPM
0.160 + 0.128
0.081 0.078
0.071 # 0.028
0.099 0.044
0.132 = 0.101
0.798
0.601
DDPM
0.528
0.400
1.0e-1
LIM
0.743 +: 0.290
0.497
0.311
0.267 a: 0.077
0.653
0.413
2,444 + 1.067
1.239
0.240
1.0e-08
8.6e-06
1.3e-10
0.8
7.9c-09
5.0e-3
Table I: MSLE5-0.95 & averaged over 20 runs. Figures below scores corresponds to p-values from Welch's t-test (assuming unequal variances), comparing the mean of DLPM with the given method.
Enhancing data coverage: addressing mode collapse. To assess the ro- bustmess of DLPM to mode collapse, we consider an unbalanced mixture of nine Gaussian distributions. We set their standard deviation to 0.05 and ar- range them in a grid-like pattern with equal spacing, in the square [-1, Their mixture weights range from .01 3 S. We the l2fe score (i.e. the harmonic mean of precision and recall, see Appendix G.3) to assess, in a single summary statistic, the quality and diversity in the generated data. As shown in Table 2, we are able to achieve improved scores by choosing a tail index 2 with DLPM. This not necessarily the case for LIM, which is consistently outperformed. Finally, DLPM; shows its strengths with better performance over all the range a, though at the cost of times the run time.
Figure 3: Gaussian grid
Method
0= 1.5
1.6
a= 1.7
1.8
1.9
2.0
DLPM
0,923 = 0.005
0.933 0.028
0,923
0.024
0.907
0.034
0.862 0.028
DLPMs
0.944 0.013
0.943
0.021
0.943 + 0.010
0.941
0.014
0.928
0.016
9.0e-3
1.6e-05
7.4c-2
9.0c-4
3.9e-3
LIM
0.842 # 0.039
0.850 + 0.046
0,868 + 0.034
0.874 # 0.030
0.884 = 0.017
0.874 # 0.027
1.7e-14
1.3e-09
5.7c-11
3.9c-09
1.9e-3
9.6e-2
DDPM
0,867 0.029
5.0e-1
Table 2: FT 1 score, averaged over 30 runs. Figures below scores corresponds to p-values from
Welch's 's t-test (assuming unequal variances), comparing the mean of DLPM with the given method.
4.2 EXPERIMENTS ON IMAGE DATA
To fairly illustrate the differences between LIM and DLPM, we use the same improved DDPM neu- ral network architecture, as designed in Nichol & Dhariwal (2021). The specific configuration for each dataset is carefully described in Appendix G.2. Our experiments designed compare deter- ministic and stochastic generation methods under varying conditions. As a visual check, examples of generated images are listed in Appendix G.4.
Convergence speed. Consistent with existing literature Song et al. (2020), our findings as shown in Figure confirm that deterministic generation outperforms its stochastic counterpart significantly, especially when fewer than 100 diffusion steps used, on both MNIST and CIFAR1O LT. As the number of diffusion steps increases, both of these sampling methods produce similar results. This observation highlights that the advantages of the diffusion process do not only stem from increased randomness at sampling time. These heavy-tailed processes may define more appropriate vector field on which the noise is transported back to the original data distribution, which would lead to improved model performance (see Karras et al. (2022) for similar discussions on DDPM vs DDIM).
The previous observations are quantitatively supported in Table 3, where we present results for both deterministic and stochastic sampling strategies. We compare both methods on stochastic generation high step count, compare their performance at their best regime, and on deterministic gener- ation at a small step count, to assess the tradeoff in computations/qualin As we can see, DLPMM surpasses LIM on both datasets. Moreover, these results show that LIM's performance deteriorates significantly when clipping is not used, raising questions about whether
offered by both methods.
'The exact mixture weights are {.01,-02, 02, 05, 15, 11, 2,.3).

---

Published as a conference paper at ICLR 2025
Table FIDJ, 1000 sampling steps for LIM and DLPM, 25 sampling steps for LIM-ODE and DLIM.
Figure 4: FID a 1.7
MNIST
MNIST
1.5
1.8
2.0
LIK-ODE
DDPM
3.43
LIM
14.37
11.54
11,18
13.75
11.69
10P
DLPM
w/ elipping
4.08
5.17
6.81
11.20
DLPMs
3.80
3.03
2,51
2.71
DLPM
5.39
2,94
2.93
3.24
3.63
DDIM
5.16
LIM-ODE
49.63
78.59
92,93
109.48
29.04
w/ clipping
45.72
68.15
85,09
113.20
102
10a
102
DLIM;
3.37
2.93
3,44
4.31
steps
DLIM
2.81
3.27
5.18
CIFAR10 LT
CIFARIO_LT
LIK-ODE
DDPM
19.05
LIM
75.38
35.15
31.14
21.68
21.56
DLPM
w/ clipping
16.13
16.21
17.67
19.24
DLPM
16.10
18.00
19.94
20.21
21.07
DDIM
23.44
LIM-ODE
42.07
91.64
105.95
407.79
32.00
w/ clipping
30.17
65,78
84.55
101.70
DLIM
20.69
20.77
21.96
22.79
23.99
steps
DLIM, steps
steps
steps
LIM, 5 steps
steps
steps
Figure 5: DLIM and LIM-ODE with small number of steps, the Gaussian grid of Figure 3.
framework of LIM is inherently well-suited heavy-tailed distributions. More interestingly, observe in Table 3 that DLPM consistently outperforms LIM and offers satisfying image quality at low number of steps, both for stochastic and deterministic sampling.
Generated images after 25 steps achieve FID score of 2.81 MNIST and of 20.69 CI- PAR10_LT, as compared to respectively 45.72 and 30.17 17 for LIM-ODE with clipping. On MNIST, with = 1.7, DLIM is able to match the sample quality of DLPM with 40 times less diffusion steps, further proving its efficacy. To visualize these behaviours, we display on Figure different generation with varying time horizon T. We able to approach the true data distribution more accurately.
how the backward process defined by DLIM is
Eventhough lower usually entails lower FID in this table, LIM-ODE shows than Gaussian diffusion at 25 reverse steps; since the image quality is still monotonically decreasing with a, except for 2, we can conjecture that the initial instability introduced by heavy-tails quickly counterbalanced by their benefits. DLPM; shows consistent improvement over baseline, more particularly for stochastic sampling. We provide results for non-isotropic generation, and additional metrics on image data in Appendix G.4 further supporting our claims.
performance
5 CONCLUSION
In this study, proposed DLPM and DLIM, as heavy-tailed generalizations of DDPM and DDIM. Contrary to similar methods, we believe our approach will be more accessible to the community, thanks to its elementary tools. The various experiments conducted suggest DLPM is more effective in leveraging the characteristics of heavy-tailed distributions, providing robust performance across heavy-tailed data, unbalanced datasets and requiring lower number of diffusion steps.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
A.D is funded by the European Union (ERC, Ocean, 101071601). D.S and U.S are partially funded by the European Union (ERC, Dynasty, 101039676). Views and opinions expressed are however those the author(s) only and do necessarily reflect those of the European Union or the Euro- pean Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
U.S is additionally funded by the French government under management of Agence Nationale de Recherche as part of the "France 2030" program, reference ANR-23-IACI-0008 (PR[AI]RIE- PSAI).
The authors are grateful to CLEPS infrastructure from the Inria of Paris for providing resources and support.
REFERENCES
Michaël Allouche, Stéphane Girard, and Emmanuel Gobet. EV-GAN:
Simulation of extreme events
with ReLU neural networks. Journal of Machine Learning Research, 23(150)):-39, 2022. URL
https:/ //hal sccence/haaaae h- 10221006366.3.
Szymon Borak, Wolfgang Karl Härdle, and Rafal Weron. Stable distributions. Statistical Tools for Finance and Insurance, 04 2005. doi:
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of Markov Chain Monte Carlo. CRC press, 2011.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Waveg- rad: Estimating gradients for waveform generation, 2020. URL httts:t//axxii.. wwwpxii.rrg//as.....
2009.000 00713.
Frédéric Chyzak and Frank Nielsen. A closed-form formula for the kullback-leibler divergence between cauchy distributions, 2019. URL https: http//aaxvvv.gggraaa//// http:/aab/999..11199
Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching, 2022.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021a.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis, 2021b. URL
https: /awxivv..vvv/vv///////...///.av. arg/abs///ul 05233.
B. V. Gnedenko and A. N. Kolmogorov. Limit distributions for sums of independent random vari- ables. Translated from the Russian, annotated, and revised by K. Chung. With appendices by J. L. Doob and P. L. Hsu. Revised edition. addison-Wesley Publishing Co., Reading, Mass.- London-Dov Mills., Ont., 1968.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
William Holt and Duy Nguyen. Essential aspects of hayesian data imputation. SSRN, 2023, jun 2023. doi:
htthtpp: /xx doi www.do.o....oo.ooo/oo orgg 2139/ ssrn 443414.
Lu-Jing Huang, Mateusz Majka, and Jian Wang. Approximation of heavy-tailed distributions via
stable-driven sdes, 2020a. URL https: rxxii.vvvooo www.:ii.ooor/aaaa. abs.///0..202222
Lu-Jing Huang, Mateusz B. Majka, and Jian Wang. Approximation of heavy distributions via stable-driven sdes, 2020b. URL https: :/ arrivvi
0221.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2021. URL http::/s //tp://////ooooo/////////00 09761.

---

Published as a conference paper at ICLR 2025
Gábor Lugosi and Shahar Mendelson. Mean estimation and regression under heavy-tailed distribu- tions: A survey. Foundations of Computational Mathematics, 19(5):1145-1190) 2019.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models, 2021.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021.
Gabriel Nobis, Maximilian Springenberg, Marco Aversa, Michael Detzel, Rembert Daems, Rod- erick Murray-Smith, Shinichi Nakajima, Sebastian Lapuschkin, Stefano Ermon, Tolga Birdal, Manfred Opper, Christoph Knochenhauer, Luis Oala, and Wojeiech Samek. Generative fractional diffusion models, 2024. URL peetts://arxae org/abs aataaabb/3100 316387668....
J. P. Nolan. Stable Distributions Models for Heavy Tailed Data. Birkhäuser, Boston, 2010. In progress, Chapter I online at academic2.american wat.emiee.aeerricccccuuupppopplo edu/-jjnoolan
John P Nolan. Univariate stable distributions. Springer Series in Operations Research and Financial
Engineering, 10:978-3, 2020.
Manuel D Ortigueira, Taous-Merien Laleg-Kirati, and J A Tenreiro Machado. Riesz potential ver- sus fractional laplacian. Journal of Statisticul Mechanicx: Theory and Experiment, 2014(9): P09032, sep 2014. doi: 10. 10010088/772-5548221110000 10.1088/ 1742- 5 68 2010/214//000 /09 /P09032.
14466//011/09//000033
htthttt::// /tx.//xxx////o/ doi. org/
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. Cads: Unleashing the diversity of diffusion models through https: /upps://arxiis /arxivv.vvvv. org/ aas//ab//// 2310 17347.
sampling, 2024. URL
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall, 2018.
Gennady Samorodnitsky, Murad S Taqqu, and RW Linde. Stable non-gaussian random processes: stochastic models with infinite variance. Bulletin of the London Mathematical Society, 28(134);
554-555, 1996.
Umut Simsekli. Fractional Langevin Monte carlo: Exploring Levy driven stochastic differen- tial equations for Markov chain Monte Carlo. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Pro- ceedings of Machine Learning Research, pp. 3200-3209. PMLR, 06-11 Aug 2017. URL https://proceedinng www.mmm.....m...mmmmmmmmmmm.eers
Umut Simsekli, Lingjiong Zhu, Yee Whye Teh, and Mert Gurbuzbalaban. Fractional underdamped langevin dynamics: Retargeting sgd with momentum under heavy-tailed gradient noise. In Inter- national conference on machine learning, pp. 8970-8980. PMLR, 2020.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. CoRR, abs/2010.02502, 2020. URL https: http:///axvvvvoiie abs///bs.......////// 02502.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020.
Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ily Sutskever. Consistency models, 2023. URL
peatps://arxiie orgg alss/ 2303 01469.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2023.
Pascal Vincent. A connection between matching and denoising autoencoders. Neural Comput., 23(7).166--1674, jul 2.1 ISSN 0899-7667. doi: 10.1162/NECO, a 00142. URL https: //doi. wow.ioorrg/10 00. 11 62, /EE//EECL_B a_00142.
12

---

Published as a conference paper at ICLR 2025
Ling Yang, Zhilong Zhang, Yang Song. Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuar Yang. Diffusion models: comprehensive survey of methods and applications, 2024.
Eun Bi Yoon, Keehun Park. Sungwoong Kim, and Sungbin Lim. Score-based gen- erative models with lévy processes. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 40694-40707. Curran Associates, Inc., 2023. URL
https:// igg neurips httpippperrfilleppppppreeeee
Tianjiao Zhang, Huangjie Zheng, Jiangchao Yao, Xiangfeng Wang, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed diffusion models with oriented calibration. The Tweifth Interna- fional Conference Learning Representations, 2024. URL https: //pppereveeeeeeeeeeeeeeeeeeeeeee ew net// forum? d=NN225xXXXX.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
The Appendix is organized as follows:
In Appendix A, we provide the pseudo-code for the training and sampling algorithms of DLPM and DLIM.
In Appendix B, we characterize the stability property of the o-stable distribution, and give explicit formulas for the distribution of the sum of two a-stable random variables.
In Appendix C, we provide the detailed theory and derivations of the DLPM framework. Working on the process {Xx] defined in (2), and its data augmentation counterpart {Yr)==q defined in (5), we start in Appendix C.2 by characterizing the distribution of X given Xo from given noise schedule {(Te: ox) ) as this defines the characteristic loca- tion Ni- and scale 07 i-r of the process at time t. Likewise we characterize the distribution Y, given Yo. (Ar) as this defines the characteristic location 71-x and variance 2 of the augmented process at time t.
In Appendix C.3, we focus on the Gaussian trick exploited by the process {Yr)== defined in (5). This leads us to the an explicit formula for the Gaussian density of the backward process conditioned on a sequence {At) of o-stable random variables.
In Appendix C.4, we further put this characterization into good use by obtaining a closed- form formula for the loss function. Follows discussion design choices for the model and the loss function at hand, which leads us to the simplified training loss (8) correspond- ing to the denoising loss for a-stable diffusion.
In Appendix C.4.3 we provide a more principled approach to derive the loss function at hand.
Finally, in Appendix C.5, we provide a faster sampling strategy for training DLPM, com- puting each loss term LtL, using only two heavy-tailed random variables per datapoint, instead é random variables.
In Appendix D, we adapt the setting for deterministic sampling in classical denoising diffu- sion DDIM (Song et al. (2020)) to a-stable framework. We naturally call this extension DLIM. Alike DDIM, is true that the same neural networks can be used for both the DLIM and DLPM generation procedures.
In Appendix E, compare in more details the two discrete and continuous frameworks DL.MM and LIM, underlining how they offer two distinct loss functions, training and sam- pling algorithms.
In Appendix F, we give proofs relative our technique for faster training, as introduced in Appendix C.5.
In Appendix G, we provide additional experimental details.
A ALGORITHMS FOR DLPM AND DLIM
In this section, explicitly provide the algorithms needed train and sample from the DLPM and DLIM generative methods.
A.1 TRAINING
The same models can be shared between DLPM and DLIM, as underlined in Appendix D.3. We introduce the values 01+t:: 71-+2 determined by the noise schedule, presented in Proposition 4:
1/a
II 7
Σ
71
T1++
oi
We define сд 2/ (Ta/4) in Theorem 1. We make the design choices D1, D2, D3 for our model. as described in Appendix C.4.2. Finally, we use the method for faster sampling as described in Appendix C.5, Proposition 9. The resulting training algorithm is given in Algorithm 2.
14

---

Published as a conference paper at ICLR 2025
Algorithm 2 DLPM training simplified loss
Require: model {E] 12
noise schedule o=))== data Yo
1: Sample | ~ 1,
2: Sample A, ~ S (2/2,1(0,00 CA)
3: Sample G. ~ Io,
4: Yt + 7-x 0 / G.
5: Lt-1 < (Y, A G
Golla
return 1-
A.2 SAMPLING
Given the noise schedule and sequence (ArTT- of a-stable random variables define:
2 (a(A1a) = 1- (A1:-1) Σ (A1a)
E (111(A1a)
11
V
1xxx
1=1
Proposition and Equation (12) from Proposition 5 for precise statements. We give our sampling algorithm for DLPM in Algorithm 3.
Algorithm 3 Stochastic sampling (DLPM)
Require: model {2+) {< noise schedule ((1,, (1H),
1: Sample Y7 ~ S(, Ia)
2: Sample [A, i.i.d., A₁ (0,22(0,ca)
3: fort to do
4: Compute Ei 11 P2 as in (9)
5: Sample G, ~ La)
Sif1 + 1-E1---11
6:
Yt
7:
+ Mt
e + Gr
8: end for
return Y
For DLIM, one can potentially provide Yr input, in order to use the support of the noise dis- tribution as a latent space, alike what is done by Song et al. (2020). We give our DLIM sampling algorithm in Algorithm 4.
Algorithm 4 Deterministic sampling (DLIM)
Require: model [T noise schedule oz) Sample Yr (0,
2: for + to do
Y,
01- 01-1-11 (Y()))
3:
Y-1+ 7
74
4: end for
return Yo
B ADDITIONAL REMARK ON a-STABLE DISTRIBUTIONS
Stable distributions are closed under convolution for fixed value of a. Since convolution is equiv- function, it follows that the product of two stable
alent to multiplication of the characteristic functions with the same a will yield another such characteristic function. We precisely characterize this stability property in the following proposition:
15

---

Published as a conference paper at ICLR 2025
Proposition 3 (See (Nolan, 2020, Proposition 1.3)). Let X1, X2 X2 be two random variables respec- rively distributed as X1 ~ Sa.e, (11.01) and X2 ~ Sa 02). with Pl, P2, 31, B2 E R and 01,02 O. Then, X X1 X2 is distributed Su,a(H, where:
31 + 320
== - 11
o = +
+ +0 09)2/a,
0 + 02
In particular, when X1, X2 are such that X1 (0, X2 S. (2, then X x (0, (of (o0) which is the key relation used in the later characterizations of the distribution of our forward process.
C THEORETICAL DERIVATIONS FOR DLPM
In this section, we provide the detailed theory and associated derivations of the DLPM framework.
C.I SETTING AND NOTATIONS
We will denote by db,(11p,, Σ) the density of ( E), where μ E Rd and E Rdxd We will denote by (p(a the density of (0,2,((, ca), where сд cos2/a (TTa4))
Forward process. We reintroduce the setting presented in Section 3, with the noising schedule being denoted by ox)) and the following forward process which DLPM is based:
X = 7 X + tote (a)
X Pt Pas
and
(10)
where p. is the data distribution and { )T are independent random variables distributed as St (0,
Data augmentation process.
We also introduce the associated data augmentation process:
Yo P. and Y, = - + +o,
(11)
where (G.)T=, and {A+) independent random variables distributed according G. (o Ia) and A, S./2, ca), with CΛ cosa/"(Ta/4)) From Theorem 1, (Y,)T= is Markov
chain that admits the same distribution as {X.) We will denote by (a) the distribution of Y..
and by s (a) (-1-) the transition density associated to the Markov chain (10).
Backward process. A backward process associated to the Markov chain (Y:TT= is Markov chain (Ý" T such that the two processes (Y -- and {r,)T= have the same distribution. For ease of presentation and following classical notations, will rather consider (Y YToTo where Y, f YTT-.. We will denote by % (-1+), the transition densities associated to the process { V.)To Since the true backward process is never available us, will focus approximation induced by a variational family. We consider the process { where Y is distributed as S(0, (-))) and the density of the distribution of Y
t-1 conditioned Y are given by T (-)-)) where E _ parameterizes neural network. We also denote by & 0. the joint distribution of { y and by T the marginal distribution of Y7.
Further notations. Finally, denote by pf (f(aat))) h//e-11 1. (+1. aI:), and T 11 (.1, and)
densities/transition densities associated to the processes (V,) ( given A, a for
1 < t < T. We we will also write of (-lyo, a1z)), 1,0, when further conditioning Yo.
k (-1 No, a1x), and T T2-1e.,0a (11, No a1:)
(a)
16

---

Published as a conference paper at ICLR 2025
C.2 FORWARD PROCESS
Let us now characterize the distribution of X given Xo. and Y, given Yo, (Ar]i- which are tractable thanks Proposition 3. These will come in handy, for instance when working the backward process in Appendix C.3.
Proposition 4 (Distribution of X2 given Xu). Let (Xili=o be the forward process as given in (10), and (1)= the noise schedule. Then the distribution of Xt given Xo is given for any t by
X n + 1
where & ~ (0, Ia). and n-xi, d1+t are given by:
1/a
Σ Je
n
m1
7h
01
The proof is an elementary induction based on Proposition 3.
Proposition 5 (Distribution of Y, given Yo, (A()T=) Let {Y:)== -o be the forward process as given in (11), ((n,o))) the noise schedule, and {At]r= the associated a a-tblle random variables, parameterizing the variance of the Gaussian noise increments. Then the distribution of Y, given Yo, {At]=1 is the Gaussian distribution with mean Yo and covariance matrix E1--ela i.e,
Yi 7 (AA)
where G, ~ ((,, and
71
We
)ar((ana)
Σ k=
Ni-t
v /akTR
(12)
The proof is elementary and omitted. It is worth mentioning the following recurrence, to speedup the computation the sequence (21=)((1:)))
E)-e(ana) = of + --
C.3 BACKWARD PROCESS
Consider the setting of the data augmentation approach given in (11). By the same arguments used in Section 3, it can be verified that a process starting from PT and with transition densities ; (Mt-1 18x)) Pt (a)) (1) (oll-//wwllll-- (w))) for any 3t-1, is backward process associated with (Y,T7 However, it raises two main problems. First (i), we cannot characterize the distribu- tion of [ (a) 1-11((111111 since we do not know the data distribution. Second (ii), in the case where
< 2, we do not have access to an explicit expression for To (a) 1- ( Bo).
Regarding (i), we have access to the distribution of Y, given Yo. so valid strategy consists in devising a method relying on characterizing the backward the process {Yi)7= given Yo. This is the classical strategy used in DDPM (Ho et al. (2020)), which is possible in the case 2 since T. (ax) 1-11t,0 (1t 3o) admits an analytical expression for any Yo: V:-1, Ut. thanks to the properties of the Gaussian distribution.
Regarding (ii, in the case where 2, make use of the trick introduced in Theorem 1, justifying the data augmentation approach. We will rather characterize the density of the Markov kernels associated to the backward of the process (Y:) given Y, and (At) This time, since manage Gaussian noise increments, we can fall back to the classical strategy, as we further develop in the following proposition.
Proposition 6 (Density of the backward process associated to {Y.TT given Yo. (A.)T2) Con- sider the setting of the data augmentation approach as given in (11). Let {(1, o be the noise k(o) schedule at hand. Let k (- Vo, a1x) be the density of the backward process associated to
17

---

Published as a conference paper at ICLR 2025
(Yr)i=o given Yo and {A.)i= Then k( [ (131, yo, yo,ara) ava) is the density of Gaussian distribu- tion (, 1x-1) with mean m and variance Σx-1 such that
1 (y: Te vo))
m (Uts No: an.)) = 7/
(13)
where
E1-1((11:) Σ
7- Vakok)
11
k=1
<
71-+330
01
?1 11
Pe(a1a)= =
Σn E1-r((ar1)
Eventhough F, involves multiple heavy-tailed random variables, is nonetheless bounded: < 1.
Proof. To determine k.(0) k (-1yt, yo. 01:t), we need work on the joint distribution of (Ye-1; Yt) conditioned Yo. {A, 1 which is Gaussian vector for which classical techniques will let us derive the distribution of y given Y. Before doing so we need to compute Pt. the covariance of Y--1 and Y, given Yo, {A,]==1 which do thanks to Proposition 5:
Pt = Cov(Y, Y Yo, AIT) = "eCoov(Y, Y,-1,,,,,111 yo, AIt) = Th
Denote by x/()) ki-1.110.a the density of (Y--1, Yi) conditioned on Yo, Arr. Denote by ka(+Iu, Σ) the density of a d-dimensional Gaussian distribution with mean μ and covariance E. From the results of Proposition 5, we can write
(())) a (2 BelBo. a1x) = tr ((*((r) (1-- V/ 7111100
(7 (F1-r-1(01--)))
PeTa 21 (1so(ar))1a))
Pela
Then the distribution of Y given Y,, Yo, Air is Gaussian distribution M(m, E.-1) (Holt & Nguyen, 2023, Theorem 3) with mean m, and variance Σ,- satisfying:
Pr
i 1o-1(Ve. (Yt, yo: 01:e) = 114--180 + (arr) (Vi 71-2300
1
) = E1 - Σnr((e1:))
By defining
71-42/0
(((a1:)) = 22 (
2
(yt,, Mo)
01-+t
Σ₁ (a1+t)
give the final expression for the mean m and variance ET- of the distribution of Y,-1 given Y,, Yo and [A,)
1 (y. -1-1-- -daaaaa((aaa)))a) 7/
m (Vi, Ho. a1:))
(14)
{_1(( = -1
Since (,() 1- - ( (1th
/1 and a,, nts E1 01
0, we have 0 < T, < 1.
18

---

Published as a conference paper at ICLR 2025
Case 2 As set 2, the random variables {A become detenministic, equal to 2. One can check that in this case, with the variance preserving schedule
/ 3r
7 var,
VB., 01 vI 0t.
then:
E 1 = 207-11 = =2(1 - a:)
Further noticing that 7/2 = /1-t// 71 11+10-1 1, one computes
r3 = 0f -
a4 )0//4-1
-
Or 01//0111
Ctt
so that one recovers the famous equations made popular the seminal DDPM paper (Ho et al., 2020, Equation 7):
m- Va+
01/
Y
cs (Y,, Yo)
(15)
Vai
V1
1 1- 1-ar/aa-1
Σ,11 (1 -(t-1)
1
with < (V,, Yo) = (Y VorY
Model for the reverse process. We propose approximating the backward process associated to {Yi) given {Ar)_11 adapting the DDPM approach. This time, for the backward process, we characterized the conditional density of Y,- given Yo. Y, and (A.T
K Fla) 1:7ju,a (11.T yo. air)) == PT (a)
(VT) II, ( (1t-1 yoo a1z)
where k/()) 100,1, a (-13,, yo. a1s) is the tractable density of a Gaussian distribution, as we have just proved in Proposition 6. We similarly reconsider the family of Gaussian variational approximation introduced in (4), modified to account for an i.i.d. sequence {A.)=
1 a1:)) = 12 a1.), E (1-,
(16)
where &u is the density of the multivariate Gaussian distribution, so that the overall model for the backward process is the following:
(or(1or)) = w (i II Vi-1 Ue. a1. a1d daj.
where 1/
(a) ((Ir) = II (ai), and pr is the density of 8% (0,01-ra)).
C.4 TRAINING Loss
In this section, we draw inspiration from DDPM (Ho et al. (2020)) to obtain a loss function admitting a floed-form formula. We further provide three design choices which lead to a simplified training loss, corresponding to the denoising loss for a-stable diffusion.
C.4.1 CLASSICAL LOSS FOR DDPM, a 2
We start by reviewing what is classically done for DDPM, i.e., the case a = 2. The variational approximation 7 0 decomposition x 0.7 iee. T approximation of p(a) Then, it is trained classical
E θ), for some parameter space O is designed admit the same
T (Tr) I' a 1/. (2t-1 (xz). where T is chosen
to be the density of St ( upper bound of the Kl. DD 0 + KT(p- ( between the and the generated distribution,
19

---

Published as a conference paper at ICLR 2025
which is form of evidence lower bound loss (Ho al., 2020, Equation 5). Thus resorts to optimize the following sum:
(0
< 2T + Σ 20)
(17)
t=2
where C' is constant that does not depend on 0, and
T E KT. (-X)) +2((10
(P( -E [og ( (Xoo
L (0) [L
k M (Xo Xt) || T [(-]je(+Xx))
where {X,]?-o is the process defined in (10), and / 1/0, is the conditional density of X,-1 given X,, We make the following classical remarks on the terms of this loss (Sohl-Dickstein et al. (2015), Yang et al. (2024)). The term LP does not depend on but only on the chosen time horizon for the forward process, that determines the final variance of the Gaussian distribution N(0, d1) It is neglected. The effect of optimizing the first term LP((B is negligible too.
More importantly, for the term L when using Gaussian variational approximations, i.e., as
T Ti-1je( (11 = &a ( E: 24 (z)))
where (z, Σ) + (2(2) m, Σ) is the d-dimensional density of the Gaussian distribution with mean m and covariance matrix Σ, mi S are some functions of x; parameterized by 6, it turns out that LP_ admits closed-form expression. For fixed variance Etg Ê,-1, with Σς-1 given in (15), one resorts to optimize convenient Lg loss function:
LP (0) ( (11, m (Ix))1
(18)
where >e, m, depend on the noise schedule (7t, 0e) and Ji, 20-
Unfortunately, as we mentioned in Section 3.2.2, this solution cannot be used as such to learn the backward transitions associated to {X.)²=o for a 2. The main issue that face stems from the fact that the density of o-stable distributions are in most cases unknown, in contrast to Gaussian distributions. As result, the conditional density #1-1, To. 3t x(0) ( .:) is unknown for a < 2, which prevents us to have an explicit expression for 0 + (0).
Moreover, the absence of second order moment for a-stable distributions challenges the most straightforward adaptation we can make to the previous loss considering the data augmentation setting. Indeed, to fit 8 to the data distribution, we aim to rely on Kullback-Leibler minimization, a.k.a. the maximum likelihood principle, and some associated upper bounds. A naive solution would consist considering the bounds obtained applying the Jensen inequality:
KT. T8)
< E KL ( (-1AArr)]]
and fall back the expression obtained in (17), only with conditioning on {At) However, as we see in (18), this expression would involve taking expectation of Ar, while it is distributed as 10/2,1 CA) and admits no first order moment. We are not aware of any bounds on KL (+(AA))))) that would lead to a meaningful optimization problem due to the heavy tailed nature of the distribution of (At)T=1-
C.4.2 LOSS FOR DLPM, FOR ANY
We now expose our methodology address this limitation. We keep the structure of the classical loss and aim at minimizing the error between the backward process ( I and its variational approximation ( T ( To do consider the same loss structure before, but take the square
20

---

Published as a conference paper at ICLR 2025
root of individual KL terms. See Appendix C.4.3 for principled approach leading similar loss. Thus we consider the following valid loss function:
I Σ (2 (0,
L:: :0+ 20(00 = E
(19)
where O,
= KL (((a ( (11,0, (-VV, Yo, A1a) T -1- (1:)) K
A1o
and k. (a)
1t-1 0,t,a denotes the conditional density of Y given Yo, Yt and {Ai)]= In order to ensure
that the expectations with respect to Aur are finite, we need to choose r < 2/2 when E (1, For
simplicity, in the rest of the paper, we will use r=
Proposition 7 (Training loss for DLPM). The loss LL admits a closed-form expression, such that one resorts to optimize the following loss for S.
(0, A1tt) = E
log
1111
Σ
+
i
m
A
2
11
-
where
1
me 1a,1((r, Yo, A1a) = - (Y, -iiiie(A Yo))
7t.
(A11(A = [T((A 11
Y1 n-1Yoo
(Yi, Yo)
01-1
E1-1(A1a) = Σ
n-+t A
km
[x(Aaa) =1 7 E1-1111 (
21-1(A1s))
where ii are the mean and variance of the backward transition kernels T6 t-1p We have omitted the arguments of the mean and variance functions for clarity.
Proof. Recall (Propasition 6) that the backward process Y conditioned on {ArTT Y,, Yo at time t is distributed as N Ex and, by design (Section 3.2.2), the backward transition kernels T "i-iat of each element of the variational family describe Gaussian transition kernel of mean af_ and variance Exf at time defined in 14. Since the KL term in Lt (0, Alz) corresponds to KL divergence between two Gaussian distributions, a closed-form formula is readily available. Here rewrite the equation with all the functions arguments written out explicitly:
1 (A1:))
$
(0, AIa)) = E
log
(20)
2
Σn-1 (A1+t)
2
Ê (A1:)) + -1(it-1(V,, (Y,, Yo, A1=) (Y,, A
+
A
Now we discuss further design choices for (0, Th (-1je,.- leading to simplified denois- ing loss, as is usually done in the literature. We denote them by D1, D2 and D3.
D1. We set fixed variance ΣΦ E, for the reverse process, but expect study on the effect of learning variance to yield similar results as in original DDPM (Ho et al. (2020)), and especially its improved version (Nichol & Dhariwal (2021)).
21

---

Published as a conference paper at ICLR 2025
D2. Following own experimental results and the usual recommendation for denoising diffu- sion models (see, c.g., Yang et al. (2024); Karras et al. (2022)), we reparameterize the output of the model to predict the value ( Yo) at time-step t rather than = (Vr, yo, a1.)). Since
1 (Y, ( Y
m (Y,, Yo, A1a)
we re-parameterize nf_ to be equal to
1 (Y, an-- A1:))) 7t
(V,, A1:)
with E? being the output of the model. Following experimental results (see the introduction of Appendix G), we drop the dependency of Ei on {Ai 1 which reasonable since approximates Yo) + e Yo). Thus Ex only depends on t, Y,. This choice achieves better performance in our experiments, allows further computational tricks (introduced in Appendix C.5), and enables one to re-use existing neural network architectures.
D3.
Assuming DI, D2, 1 becomes
L1_ = E [A 11, (V,, A1:) 6 (Yz. Yo)l
(21)
where Ae.a, = r (1 S and e2 (Y,, Y,,,H, Y)) = - - o The methodologi- cal knowledge of diffusion models motivates making specific choices for X, different from its defined value, resulting in classical technique for improving performances (c.g., see Karras et al. (2022); Ho al. (2020); Nichol & Dhariwal (2021); Yang et al. (2024)). We will stick to the classical choice of choosing At,au. 1, which experimentally works better and draws similarities to the sore-bbsed perspective (see Appendix E). Other choices and optimizations are left further work.
The proof of Proposition 2 follows immediately from these design choices, hence omitted.
C.4.3 PRINCIPLED APPROACH FOR DERIVING THE LOSS FUNCTION
In this section, we provide a more principled approach to derive the loss function for DLPM, as initially given in (19). We show the derivation for 1/2; however, the same derivation applies for any E (0,
Noting that Yo is independent of (ArT in (5), p. is the equal to &/ (-lair)) the conditional density of Yo given A, = ar for any 1 E {I,...,T) and therefore, we consider the valid loss function
(0( / Auerrvietaas (KL. (llar)) ( (Harr)))
1/2
While this function is still intractable, we can provide an upper bound which we can minimize. Indeed, using Jensen inequality twice, we bound this function by
(0)(0) = dai. (o) (1 {I dyo (vol (olai:))) 11% (solaiT) - log 0 (uolai))) 1/2
1/2
(worla))))) + Csta
< dan.r (aIT)
dyo.r o. o. (Vorrlar:)) log [oo o, (1.))
f da 1p (ai.)
(1)
1=0 (T, a1:)) + Csta Csty)
< f dai.r (a) IET= (7, (0..111))) 1/2 Cst + Cst
where we used Va+b < Va + vb when a b El- (Yo a1.T) )1
0 and L., =
(A,) {at) for O,
(0,00, E (a) (L ( (,YY,, Yo, . )1 ( T (1IY,
[ax
and Cst, f dyop. (10) logp, (wo ddoo and Cst ( Yo ( (o. (ar |T does not depend on 0 since 17 f is chosen S, (0, 0 - Regarding Lj. neglect this
22

---

Published as a conference paper at ICLR 2025
term, replacing the distribution $7 00 (.11, a1:r) by deterministic Dirac. One could alternatively employ the strategy of the discrete decoder for image data described by Ho al. (2020). We end up with the final loss function:
() = (air4iit (arT)) -T (0.
We can then provide an explicit expression for L0 a1:T) based on the result Proposition
C.5 REDUCING THE COMPUTATIONAL COST WITH FASTER SAMPLING AT EACH TIMESTEP
In this section, provide faster algorithm for training DLPM, computing each loss term L using only two heavy-tailed random variables per datapoint, instead of random variables.
Consider again the process (Y)) We replace the loss function (19) by an equivalent one:
Lime (0) E (0,
(22)
where 1. ~~ (12 7| = 2 0:) - 1). The standard technique for computing the loss consists the following loop:
1. Take batch of B datapoints (Yo)
2. For each datapoint Yg, draw random t, suggested by the alternative loss (22). Indeed, for single datapoint, (i) training on all timesteps rather than just one yields equal to in- ferior results, for much higher computational cost, (ii) it is beneficial for the model to proportionally spend more time learning specific time ranges (iii) thus the distribution of t can be optimized and is matter of ongoing methodological research, e.g., see Karras et al. (2022).
3. Draw sequences {Ai) of heavy-tailed random variables.
4. Compute the noised datapoints (Y;)
5. Compute the batch loss
B Σ i=1
()) = B
LL 10,1 Y, A1..)
such that (0)((0) x Lt. (0).
6. Do an optimization step.
Step 3 be expensive, since has to sample average O(T) d-dimensional heavy-tailed random variables to compute a single noised datapoint Y, from Y,- This is all the more inefficient as T can be quite large (indeed, image datasets
have T = 4000, see Appendix G).
One can guess that this abusive, especially since characterizing the distribution Y, given Yo only requires single heavy-tailed random variable:
Yi n- + 01-1A1/2...
where A ~ (0,2,1(,cc)) ca). G, La). As we formalize in the next proposition, it is indeed possible to bypass the sampling of whole sequence. Since we manipulate the joint distribution of (Y,, Y,-1, Y2) for the loss term LeL_ (0), we will actually need to sample two heavy-tailed random variables.
Proposition (Sampling two heavy-tailed n.v for each loss term). Suppose that the functions i E8-1 satisfy for any yı, a1:t!
E1-1-1 of o
(Ve, a1:)) = M y/ at
11
Df = Sh_
ye. 0r,
o opst-
23

---

Published as a conference paper at ICLR 2025
for some functions Mi-1, St-1 and where
2
Em-a(ana)
71 aR 7--s
k=1
as given in (12). Then each term B| (0, A1:e)) of the loss be computed sampling only two independent random variables A/. Ai distributed as Sa/2,1 ca):
E 10, A1.)) 1/2 E (T (0, (0,Ab, 1/2
where Ab.1 (Ag; A1).
and
1
1 Ab,1) 1
(0, (b,A),
E
log
(23)
2
E (A),1)
2
Σ (Ab,)) + 1 Zo: Ab,1)
Ab,1)1?
+
Ab.1
2S (Zr, Ab.1)
with (Ze)Tmo being stochastic process defined
Zo = You 24 = 71- + E.
where {Gt)T= is an i.idd. sequence distributed as (N, Ia), and
E/-((A)) = Ai (1 = 07 14 1
1(( 1 1: (Ab)
(%,(),
such that Ze d Y,, and:
my (2,, Zo, Ab.1) - (2e -00--0--i
01 ( (2,,, Zo))
7/6
Σ 1 (Ab.)) = [A - (A)
Zj 7
c 70)
01
In order to keep the notations similar for all > 2, in the case of Lesss we always set A3 = 0.
Proof. Remember the full equation for the loss, first given in Proposition (20):
1 (
(A1a) 1
(0, A1x) E
2 log
(A1=t) 2
E (AIT) + | (Y,, Yo, AI.T) nl aVe,(VV, A
A
2 (A1a)
Now, all the required variables and computations only depend on A,, Ei---1; this is the case for n6-1, Sp_ by hypothesis, and this is the case for m; Et-1 as one can see in (13), Rewriting the previous loss as
$h_1 Ze A,, E-
(0, A1a)
E
log
2
(4-1(A
2
8) (
2
Ac, 11-1-1(A1111))
| 1 (Y,, Yo, A - -MP
Ar,
ب 1|2
A E1-1-1
2 2r, As, -1
9-1-1
24

---

Published as a conference paper at ICLR 2025
it becomes clear how the expectation can be taken the joint distribution of
E1-1-1(Ara-1) o7-1-11 -
Yo. Yi-1, Yu, Ac.
A direct application of Lemma shows that this expectation can be taken on the joint distribution of the five random variables (Zo, Zr-1, Zz, Ai Ab). which only necessitates sampling two heavy- tailed random variables Ab. -4. Using the formulas for Zo. Ze-1 and Z₁ given Ai Ai as defined in Lemma we obtain the equivalent loss (23).
As we will prove in the next proposition, the conditions of Proposition are always satisfied under design choices DI, D2. Under design choice D3, we can also rewrite the simplified denoising loss given in Proposition
Proposition 9 (Sampling one heavy- r.v in the simplified loss). Assume the design choices DI, D2, D3 are satisfied. Then one can obtain the following simplified denoising objective function from the full objective function given in (23):
G.ll (A,) 1/2
gSimpleeess (0)
> E E (1e)
where (At)Tz is an i.idd sequence distributed as S.12211 (0, ca), and
= + σ1 - A1/2 G.
with (GrYT an i...d. sequence distributed as (0.
Proof. Let recall design choice D1:
E0 = (111)))))))
and design choice D2:
m (2,, A1a) = (7+ 0 (1
7
Since I only depends on Σi and E1 both 1 Σt- can expressed as functions of 2t, at and Ea=:)( Thus the assumptions of Proposition 8 are satisfied. Using the same notations, we can apply the same algebraic transformations as in (21), and by design choice D3, obtain:
ge (0)
E
E (1))
ea (2.. 20)1 Ab π1) 1/2
Finally, we apply Lemma to Σ/ to affirm that Σ d = of where Ag ~ Sa/2,1 (0, ca), and obtain
the final loss presented.
We stress that this denoising training loss is similar to that of LIM (Yoon et al., 2023, Theorem 4.3), but elevated to the necessary power to guarantee that the loss finite. See Appendix E. for more detailed discussion.
D DENOISING LÉVY IMPLICIT MODELS (DLIM)
Using the same techniques as in DDIM (Song et al. (2020)), we obtain a deterministic sampling process which naturally call Denoising Levy Implicit Models (DLIM). Alike the Gaussian treated in the original DDIM work, we will show that both DLPM and DLIM can share the same neural network.
25

---

Published as a conference paper at ICLR 2025
D.1 NON-MARKOVIAN FORWARD PROCESS
Let {pe]f=) be alternative noise schedule, proper to DLIM, that will ultimately tend to for deterministic generation. In the same way as in Section 3.2.2, we take data augmentation approach. We consider process (7+)=1 defined by Zo pt. where pa is the data distribution, Ze ~ S(n-r77, d1) and, for 1 < t &
Zt-1 - - 2o i---11 Pi //) Ex (Z,, Zo) + !
01 and {Ar) {G are independent random variables distributed according to A, 2,/2,1 (0, CA) and G. ~ (O,
where E, (Z,, 2o) (2--71-120)
Proposition 10. The distribution 74 given Zo is the same that Y, given Yo-
Proof. This is a simple proof by induction, where one can re-adapt the technique of (Song et al., 2020, Lemma B.1) with the property for addition of c-stable variable as introduced in Propo- sition 3. The case 1 = fied at timestep where 1 <
true by construction. Suppose now that the property is veri- T. Then, focusing the distribution of Zı-1 given Zo, e (Z,,, Zo) = (Z: - 011 is distributed as S. (0, Ig) by hypothesis, and thus by Proposi- tion 3 and since G) (0,
write
Z d 1 7o + 1
where 6 ~ S (0, Ia), which shows that indeed Z given Zo admits the same distribution as Y given Yo-
The design of this process makes the distribution of Z, given Zo match that of Y, given Yo, where [Y)) is the forward process of DLPM (5). The task of sampling from it is thus efficient and straightforward.
D.2 GENERATIVE PROCESS
We similarly reconsider the family of Gaussian variational approximation introduced in (4), which accounts for i.i.d. sequence {A, )
L our (Nor)) = / 4p
1 (ar=))r (Vr) II. t=1 &
(a)
Ti (16-1 We. d1.))dar:r
where (a) ( IT (baa) vp(a is the density of S,/22,1 (0, cA), and pr is the density of ((0,a)) We set
T 1
1|t,a (=t 1 u1:) Pa (zx- 1 [ plas)
with &a being the density of the multivariate Gaussian distribution: the variance is fixed, determined by the alternative noise schedule {p:]T= For deterministic sampling, one will ultimately choose
P. 0 for all t and sample the chain fe= follows:
74 ~ S(0, Ia)
Σ = (Z?) fort (T....,1)
As is available input, the model be fit approximate the value of cá(21, 20), and reparameterize ii as follows:
(2()) 24 21-0-11 + - - /a (
(24)
This is alike design choice D2 in Appendix C.4.2.
26

---

Published as a conference paper at ICLR 2025
D.3 LOSS FUNCTION AND EQUIVALENCE WITH DLPM
We denote by (1))) 11,10r, the density of - given 74, Zo and AIT. which is the density of the Gaussian distribution with 71-t-1 Zo + (of-1-1- / (7 Zo) and covariance pi AcTd- Since this distribution is now a given, we are inclined to use the loss function introduced in (7), which is:
(0)00) :- E
Lt (0, (lt))
where
2t- (1a) = E
KL (1(m) (11,00,,, Z. Zo. And) ) (- ALT)
Since for < 1 < T, (10)
1-1000,, and &7 i- are the densities of Gaussian distributions, we can analytically compute each term of the loss, as in (19):
(t, Ap ll 1 + (oi-1-1 P e Zo) (2;,
where (Z Zo) = (Z₁ - 1- 20) Since the variance of the elements of our variational family 1'7 O have been designed to match that of the backward process given Zp, A1:7, the ex- pression for the loss is readily in simpler format. Finally, using the reparameterization given in (24), The loss term L (0, A1T) becomes:
10, Alr) = N.A, Zo) E,
By comparing with the simpler DLPM loss (21) with design choices D1, D2, as introduced in Appendix C.4.2, we realize we obtained the same loss term, with different multiplicative factor Ni,,, instead of At, in (21). Finally, considering the alternative loss where X..., for all i, alike the design choice D3 in Appendix C.4.2, we fall back to the same simplified objective function obtained for DLPM:
where Xy,,, = -01-t- (of p.)
T
LiSmplee (0)
E E (1))
(Y, YoI1 A 1/2
=
D.4 CAUCHY DLIM
In the special case of non-isotropic Cauchy distribution (a = 1), it is possible to bypass the data augmentation machinery, since there exists a closed-fom formula for the KL divergence between two Cauchy distributions. Denote by Cauchy o) the mie-dimensionll Cauchy distribution tered at and scale a. Then (Chyzak & Nielsen, 2019, Theorem 1):
(11 22) + (a1 + 02)²²
(Cauhhchy() Cauchy(uz, log
(25)
40102
The forward process {2,)T= is defined with Zo ~ ps. where p. is the data distribution, Zr ~ S(1-7Z2, ol) and, for < TT
Z = 701- Zo + (of-1-1 1o 2o) + pte
where e, (2,, Zo) -2 11-1 and (e, ((a))T] _ Cauchy(O, 8T, where Cauchy(O,, Ia)
is d-dimensional Cauchy distribution with independent components, centered at 0, of unit scale. The distribution of Z, given Zo admits the same closed form expression given in Proposition 10.
We denote by A ( the density of Z,-1 given Zr, Zo-
Our generative process will be an element of parameterized family of distributions admitting Cauchy transitions:
7 (Bor)))) = (p(())) II
Th t=1
27

---

Published as a conference paper at ICLR 2025
where pr is the density of S(0, 01-rla), and
T t-1 (E(-1(2))) = C (2-111ff-12z)) pela)
where C(-1mf_1(21) pela) the density of the multivariate non-isotropic Cauchy distribution Cauchy(mij (z+), Ptla).
Instead of using the loss function L defined in (7), we derive the loss via the conventional evidence lower bound (ELBO) approach (see, e.g., Ho et al. (2020)). Omitting extremal terms, this yields:
gaunchy (0) = Σ
Ga (0)
where
t=2
Cancky (0) = E
KT. (a)
(1111 (- Zo) |I 71 (--1)e(112))
Using (25), we obtain a closed form formula for the final loss:
(mr,1-1 Zo) m.,,- (Z))) +11
Lunchyy (0)
log
4p
which could also serve as a template for another family of losses for heavy-tailed diffusion models. We leave these methodological explorations and possible extensions the isotropic Cauchy case for future work. Based on our experimental findings, we expect an isotropic implementation to significantly outperform a non-isotropic one.
We outline again that such simplifications are not available for DLPM, since we are not able to characterize the distribution of X,-1 given X,, Xg from the forward process (X+)T=--
E ADDITIONAL INFORMATION ON LEVY-ITO MODELS (LIM)
Here briefly recapitulate the work done by Yoon et al. (2023), introducing continuous diffusion models with a-stable heavy-tailed noise. Using notations closer to ours, we define the noising 1
schedule as any locally bounded continuous functions Y : (t. x) (t, X) and a (t) + at). We denote by Li the Levy process for which the increments between time < follow a symmetric
isotropic a-stable distribution (0, (1 =))a). In this setting, the forward process X,, with Xo ~ Ps, is written
dX2- = Xt, (t)dL;
(26)
where X,- denotes the left limit of X at time t. Similarly. X, is distributed St (n Ia) when using Euler steps. This defines the cadlag (right continuous with left limits) solution, which in the case of < 2 a.s admits discontinuous jumps. We then consider the following backward process Xx:
dx, = (-)(,xx)))))
- ao (1. (x++) dt + ottdLL + dZz
(27)
where Ze is the backward version of Levy-type stochastic integral Z, s.t E[Z]] 0 with finite variation, and S(()) is the fractional score function, defined to be
2 Vp((z)
((()) (x)
Pr(x))
where 1/2 denotes the fractional Laplacian of order n/22 (Ortigueira et al. (2014)). More precisely, 2 f(x) = [[] where F,F are the Fourier and inverse Fourier transforms.
The training loss is obtained using the classical technique of denoising score matching (Vincent (2011), 1)), where the following losses
L 0 + E| [|lso 1) (X)II, : 1
1++Elsa Elsa (X+, s. (x.1
(28)
are proven to be equivalent objective functions, with sy being the score approximation given by the model.
28

---

Published as a conference paper at ICLR 2025
E.1 COMPARING LIM AND DLPM
Let (X be the forward process introduced in (26). As stressed initially, the framework of manipulate, thus we do not characterize explicitly the distribution of X given Xo for an arbitrary noise schedule in the continuous case. Since the work done for LIM by Yoon et al. (2023) only provides the formulas for the preseerinn schedule, we stick to them in the following: we keep the notation 71-+2: 01-z for the continuous time regime equivalent of the scale preserving schedule we introduce in Appendix G, and they match on integer times t.
LIM is not straightforward
Considering an Euler scheme to obtain discretization for the forward and backward process, and using our own notations, both LIM and DLPM admit the same forward process ) Xo ~ p- and
= - + oeet ooee
where (ec J an iid sequence of random variable distributed as (0, We denote by (XP) Jt= the backward process associated to the Buler discretization of (27), where use neural
network sy to approximate the true score s(a), Since the true score of the data s(a) (xe|20) can be expressed as
s (xaloo)) a0
a-1 (1
where (D), = (x2s - --) 0 we write
2 (2z, z0) so so( ao
so that we rather work with Eg with the same intention that led us the design choices given in Appendix C.4.2.
Moreover, we denote by { { the backward process of DLPM, introduced in (16). As em- phasized in Table 4, the sampling strategies for LIM and DLPM differ fundamentally when # 2. This is also the case for the training procedure.
Stochastic sampling. The DLPM approach introduces the bounded random variable 0 < I' & 1, interacting with the mean and variance of the Gaussian conditional at hand. Three points: when = 2, r2 becomes deterministic and one recovers DDPM formulas. Second, I3 brings additional stochasticity in the sampling process. Third, it does so the interesting manner than it simultane- ously scales both (i) the magnitude of the noise added at time t 1 and (ii) the output of the noise model.
Deterministic sampling. In the case of the scale- scale-preserving schedule, these two equations do not describe the same sampling procedure.
Stochastic
Deterministic
X
a(l/n 1) 1
X!
1-a
Continuous (LIM)
+ (
1)1/ad
01+t
11-a
24
7t. o
7/
Yt
Y P.01-1ed VI
Y
01
Denoising (DLPM)
1
7t
7
Table 4: Distribution of XP_, Vi, (C)) are independent random variables distributed as
NO La). {41) independent random variables distributed St (0, eQ is the model at hand at time f, the formula for Σi is given (12), and I2 = E Eventhough T, involves two heavy-taile random variables, it is bounded: 0 S T, < (see Appendix C.3).
Training. Alike the Gaussian case (a 2), the score S(²) (2, (ar)) is linear expression of the noise term ez (T,, To), so the training equations are very similar, and can be reformulated to involve denoising loss:
C-1 : E (x,e Xo)13))
(29)
29

---

Published as a conference paper at ICLR 2025
In the case of DLPM, our discussion leads us to the choice p 2 and 1) 1 (see (8)).
In the case of LIM, the theory must rely on the choice = 2 and = 2 in order to obtain the denoising score matching loss equivalence (i.e., L, L' are equivalent in (28)). One must make the assumption that the losses L, L' are not infinite for some A which is not necessarily realistic because (, (X are heavy-tailed random variables involving a-stable noise, and as such admit no variance.
the case of LIM, in the experiments the parameters of the previous squared loss, in order to obtain more stable training, potentially indicating that indeed L, L' (28) might infinite.
and lj = are chosen, instead
TECHNICAL RESULTS
In this section, we give the proofs relative to our technique for faster training, as introduced in Appendix C.5. Lemma 1. Let Ab Ai bet two independent random variables distributed as S0/2,1 cA). Define Zo Yo. and
Z - 71-110 01 (Ai) 1/2 Gt.
Moreover, let 2 be equal to
7
(Z₁ 11 2o)) E (A,
7
where
(7(,) (4 (Ab) 1/2
A
AG07 1
Ze 70
6 (2, 70)
T1-22
Then the joint distribution of (oo -,-1, Ze, AG, Ab) matches the joint distribution of
(1(Ab.))
Yo. Y,-1, Yo, Ar, of
Proof. Consider the setting of Proposition 6. The distribution of Y given Y,, Yo, Air is charac-
terized by the values of E,- T;:
1 (Yr (Y),)))
m
7t
where
E1-((A1a) = o,A, + 1
Yo) = V, 7-40o0
01
Σ - 1 (A
rr(Ana)
D E (Al:))
Directly applying the result of Lemma 2, we can affirm that
(1=--((A1.1)) )
where AS ~ CO, In this conditions, the distribution of I' (AIx) is equal to that of ('((A) where
1207
(((A)) = o²A, + 7,
Since the distribution of Z, does not change we draw another independent A instead of A,, this ends the proof.
30

---

Published as a conference paper at ICLR 2025
Lemma (Sampling Σ1-4 with single heavy-tailed Lv). Consider the setting of the data augmen- tation approach Section 3.2.2, where in particular {At] are independent random variables distributed according to A, ~ 2/22,1 (,, CA)) with CΛ = co52/"(TT(/4) Consider the random variable E1-re (Ava), defined in (12):
2
E1-1((11)) = Σ
71 VAROK
71
k=
Then
- d
of
where A 82 (0,c)
Proof. By Proposition 5, Yt given Yo, A1 a random variable distributed Gaussian of variance Σ (A)))
Y m + (AireAA
where G2 is distributed as standard Gaussian. Remember that Y and X, admit the same distribu- tion, with X, 7o + & where é, is distributed as S. (0, La).
In the same spirit we can define a third sequence of random variables (Z0) with Zo = Xo, and
Z 70 + -1-IV A,G., where {A,)== are independent random variables distributed according to A ~ S 2/2,1 (0, CA). It then quite clear from Section that Ze and Yú admit the same distribution; in particular,
VE d 01 - VAGG'
From there, we use Lemma 3 to conclude that VEn - V Nr which ends the proof.
Lemma 3. Let A, A' be positive real random variables, let Z be a real continuous random variable with density pz- Suppose that AZ and A' Z admit the same distribution. Then A, A' admit the same distribution too.
Proof. Let be a measurable function. Then IB(h(A)) E(((AZ/Z)) E(b(A'Z/Z))
(()()))) This shows that A, A' have the same distribution.
G ADDITIONAL EXPERIMENTAL DETAILS
All experiments conducted using Py Torch. We use linear timesteps during training and sampling, and the ccale-preserving process being the only forward process readily provided by LIM. This entails choosing a sequence (33)) Jtm such that
7t = (1 3/ o, = (1-11-
resulting 11+t (1 ---1)))))) 1/a 1 and n- I 7t- With this choice, we obtain approximately Xr Ia) We choose {3, as the cosine schedule, as introduced by Nichol & Dhariwal (2021).
We do not give any the heavy-tailed random variables {A,) as input to the neural network ar- chitecture, as we have witnessed worse performance in every scenarios we tried: learned embedding added each model layer, concatenation model input, concatenation each layer, or feeding (og(A):)) instead of A to better manage large jumps. This corresponds to the design choice D2 Appendix C.4.2.
For image data generation with LIM, we use the same clipping hyper-parameters specified in Yoon et al. (2023).
All the training and experiments are conducted four NVIDIA RTX8000 GPU and four NVIDIA V100 GPU, where a single training run on MNIST or CIFARIO LT takes approximately I day per GPU, and requires about 4- 12GB of VRAM for the batch sizes use. Generating 5000 images with 1000 backward steps takes approximately 3-4 hours on one RTX8000 GPU,
we mention again that it traditionally called the variance preserving process
31

---

Published as a conference paper at ICLR 2025
G.1 2D DATA
We give more details about the mixture of Gaussian we consider in our experiment. It is designed in grid-like pattern in [-1, 1J2, as follows:
Σ w o(1;,0012)
where (w:) = (0.01, 0.1, 0.3, 0.2, 0.02, 0.15, 0.02, 0.15, 0.05), #4 = (21, with μe (i (mood3 3) 1, μ22 Li/3] 1, and o 0.05.
For our 2D datasets, we use 32000 datapoints for training, a batch size of 1024, and 25000 points for evaluation. We train each model for 10000 steps. Since we do not focus on the effect of diffusion steps, we set it to 100, where all methods have been observed to perform optimally.
The optimizer is Adam (Kingma & Ba (2017)) with learning rate 5e-3. We use a neural network consisting of four time-conditioned MLP blocks with skip connections, each of which consisting of two fully connected layers of width 64. The time passes through two fully connected layers of size 32x32, and is fed to each time conditioned block, where it passes through an additional 32x64 fully connected layer before being component-wise added to the middle layer.
We compute mean squared logarithmic error (MSLE) loss, designed to assess the fit to tails of distributions. Since it depends on the one-dimensional cumulative distribution function, we calculate it after projecting the data onto each dimension. In this simple setting, we keep the score computed on the first dimension.
We also compute the precision/recall metrics, as presented in Appendix G.3.
G.2 IMAGE DATA
We work on the MNIST and the CIFARIO LT dataset. CIFARIO LT consists of the CIFARIO images were artificial class unbalance has been introduced. The specific class counts we use [5000, 2997, 1796, 1077, 645, 387, 232, 139, 83, 50].
The optimizer is Adam (Kingma & Ba (2017)) with learning rate 1e-3 for MNIST and 2e-4 for CIFAR1O LT. We use the StepLR scheduler which scales the learning rate by y = .99 every N = 1000 steps for CIFAR10 LT and N 400 for MNIST.
To establish a fair comparison, LIM and DLPM use the same network model. We use a U-Net following the implementation of Nichol & Dhariwal (2021) available in https:: / thub. com/ openai/ / i mp improoveddde di fus i inn. We dimension the network as follows: we set the hidden layers [128, 256, 256, 256]. fix the number of residual blocks to each level, and add self- attention block at resolution 16x16, using 4 heads. We use an exponential moving average with a rate of 0.99 for MNIST and 0.9999 for CIFAR1O LT. We use the silu activation function at every layer. Diffusion time t is rescaled to (0, 1) and fed to the model through the Transformer sinusoidal position embedding (Vaswani et al. (2023)). We train MNIST for 120000 steps with batch size 256 with a time horizon T = 1000, and CIFAR_IT LTT for 400000 steps with batch size 100 with a time horizon T 4000.
We use the FID metric for assessing the quality of our generative models, computing this metric between 5000 using images and 5000 generated images.
G.3 METRICS FOR GENERATIVE MODELS
MSLE we use a mean squared logarithmic error (MSLE) metric tailored to measure the fit on the tails of the distribution at hand. Drawing inspiration from Allouche et al. (2022), we define the MSLE as the squared distance between the logarithm of the inverse cumulative distributions of the original and generated data. If F, f denote respectively the cumulative distribution function of the original data and the empirical cumulative distribution function of the generated data, then
MSLE(E) = [(²² (1og F F-1(p) - -log P p-(())² dp.
where & is chosen the be 0.95.
32

---

Published as a conference paper at ICLR 2025
Precision/recall These metrics introduced in the setting of generative models by Sajjadi et al. (2018), and assess the overlap sample distributions using local geometric structures. Precision measures how much the generated distribution is contained in the original data distribution (mea- suring quality), and recall measured how much of the original data distribution is covered by the generated distribution (diversity). We also consider the Fl score which define the harmonic mean of these two values:
precision recall FT
precision + recall
G.4 ADDITIONAL RESULTS
We provide more results MNIST and CIFARIO LT, with FID for non-isotropic noise, and with the F metric for other methods (with clipping enabled LIM and LIM-ODE). We also provide grid images in order to visually check the performance of DLPM.
MNIST
a= 1.5
a= 1.7
= 1.8
a = 1.9
a= 2.0
DLPM
44.17
14.06
5.74
3.62
DLIM
14.96
51.58
59.84
76.03
Table 5: FID4, 1000 sampling steps for DLPM 25 sampling steps for DLIM
DLIM DLPM
LIM
LIM-ODE DDPM
Q
1.7
0.884
0.887
0.857
0.869
1.8
0.874
0.881
0.821
0.875
1.9
0.877
0.878
0.700
0.808
2.0
0.820
0.871
0.694
0.772
0.881
Table MNIST, F."
DLIM
DLPM
LIM
LIM-ODE
DDPM
Q
1.7
0.676
0.675
0.679
0.677
1.8
0.669
0.680
0.677
0.673
1.9
0.667
0.669
0.661
0.669
2.0
0.664
0.667
0.660
0.665
0.666
Table 7: CIFARIO LT, FP
33

---

Published as a conference paper at ICLR 2025
5
0
4 8
9
6
8
3
6
8
8
3
1
P
4 8
5
9
8
8
4
9
6
2
0887749
8
3
101
G
60086151
4
5
0
4
8
|
4
4
/
9
Figure 6: MNIST, DLPM (a 1.7)
Figure 7: CIFARIO LT, DLPM (a 1.7)
34


Reviews:
[{"review_id": "GVIkzJExlb", "reviewer": "Reviewer_Eq2q", "summary": "DDPMs degrade images by adding Gaussian noise. This paper proposes substituting Gaussian noise with alpha-stable random variables and defines DLPMs, which are diffusion models with heavy-tailed noise. As the authors solely focus on the discrete-time setting, their exposition is simpler and easier to understand than a previous already existing extension of diffusion models to heavy-tailed noise.", "questions": "Proposition 9: If the network is initialized close to 0 and $\\hat{\\epsilon}^\\theta \\approx 0$, the training loss derived in proposition 9 becomes an expectation of $A_t$ an alpha-stable random variable with $\\alpha < 1$. Could the authors explain why this loss is not also infinite like in the LIM setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clear in its exposition and makes a good effort to clarify its difference with closely related prior work (Yoon et al). The derivations to make the training loss implementable are interesting.", "weaknesses": "1. The training loss is unstable, poorer approximations perform better: The authors overcome the bottleneck of not having analytical KL divergences available through conditioning. Under this conditioning, the denoising network still has to learn the conditional expectation of a heavy-tailed distribution contrary to what is said in the paper. In practice, in order to be implementable, the conditioning requires a Monte Carlo approximation of an expectation. In the experiments, approximations of the loss with a single sample perform better than approximations with 25 samples. This coupled with the authors' observation that using with alpha-stable distributions with $\\alpha < 0.5$ is \"too unstable\" to train underlines the fact that heavy-tailed distributions are not a practical tool for approximating distributions. \n2. Extension for extension's sake: The work to extend to alpha-stable noises is theoretically interesting but appears to be an extension purely done for extension's sake. Moreover, it follows prior work that has already extended SDE diffusion models to heavy-tailed noises. The authors can strengthen their work by showing why alpha-stable noises are an interesting extension through better experiments.  The authors could define better what it means to be heavy-tailed for bounded distributions like images. Are there real distributions (in the sense of natural signals) where heavy-tailed phenomena have been observed and could the authors show DLPMs are better suited there? For example in section 4.1 why isn't DDPM included in the comparisons? In section 4.2, the FIDs reported for DDPM appear to be a little too high. The FID on CIFAR10 for EDM, a simplified DDPM, (Karras et al 2022) is 1.97, so MNIST should not be harder. In my view, the current numbers, especially in Appendix G.4, do not exhibit a strong enough advantage for DLPMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QP3abtJr87", "reviewer": "Reviewer_o74J", "summary": "In the context of generating data from a heavy-tailed distributions, the authors introduce a new generative model termed the Denoising Levy Probabilistic Model (DLPM). Previous work by Yoon et al (2023) had introduced the Levy-Ito Model (LIM) which is an extension of the SDE from Score-Based Diffusion Models (SBDM) by Song et al (2021) to the case of heavy-tailed distributions, where the noise added at each step follows an $\\alpha$-stable distribution. On the other hand, the authors extend the Denoising Diffusion Probabilistic Model (DDPM) from Ho et al (2020) to the case of heavy-tailed distributions. It is well known that DDPM is equivalent to a particular discretization of the Variance Preserving SDE from Song et al (2021). However, when extended to heavy-tailed distributions, LIM and DLPM are not equivalent. In fact, the framework of DLPM is simpler and more principled than the LIM and allows for reusing previous code implementations of DDPM. A deterministic sampler counterpart to DLPM is introduced and showed to outperform the deterministic counterpart of LIM when few sampling steps are considered in experiments. More experiments are provided comparing the different models with mixed success.", "questions": "The fact that the neural network does not have the $\\alpha-$stable variable as input seems a bit magical. I may be missing something, but it seems that we build all this framework considering the $\\alpha-$stable noise but then it only appears implicitly through $Y_t$ and in the conditioning in the loss. I think the reader would benefit from a bit more explanation regarding where is the $\\alpha-$stable noise appearing even though the neural network does not model it.\n\nWhat is behind the empirically seemingly big difference between DDIM at $\\alpha=2$ and DDPM mentioned in Weaknesses item (2)?\n\nCould you provide the results for results for $\\alpha=1.5$ and $\\alpha=1.6$ available for the experiments with results depicted in Table 1 and Table 2? (c.f. the comment in Weaknesses item (3).)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The presentation is clear and the necessary background is succesfully introduced.  There is a good flow on building the DLPM as a natural extension of the DDPM.\n* The provided framework for DLPM has the nice property that it has DDPM as special case. More precisely, DLPM at each noising step adds a sample from a $\\alpha$-stable distribution. In the case of $\\alpha=2$, DLPM reduces to DDPM (except for a square root in the loss.) This is mathematically interesting and puts DDPM into a more general and flexible framework. \n* The authors do a good job comparing DLPM with LIM. In particular, DLPM uses more elementary tools and a more clear presentation than LIM. Yoon et al (2023) develop the theory for the LIM framework using an $\\ell_2$ loss while the experiments are done using an $\\ell_1$ loss. In contrast, the model for DLPM that is theoreticallly studied is the same as the one used for the experiments.\n* The reparameterization provided in assumption D2 ensures that the function that the neural network estimates does not depend explicitly on the $\\alpha-$stable distribution which means that previous implementations for DDPM can be reused.\n\nI find that their main contribution is to provide a better and simplified mathematical framework for diffusion when the noise added can have heavy-tails.", "weaknesses": "I do not find the claim that the authors make on the experiments as satisfactory as the theoretical counterpart. Especially in proposing a new framework (for a task that had an already existing framework, i.e., LIM) it is important to ask whether the new framework leads to better performance in a class of tasks. \n\n1) The experiment for CIFAR10_LT is not conclusive: we see in Table 3 that in terms of FID, LIM is no worse than DLPM and for most $\\alpha$ LIM actually outperforms DLPM. On the other hand, DLIM does outperform LIM-ODE measured by FID, but only when taking 25 sampling steps. The plot in Figure 4 shows that taking 100 steps or more LIM-ODE is comparable or outperforms DLIM. The only claim that could be drawn from here is that DLIM (the deterministic sampler for DLPM) is better when constrained to a few number of steps than LIM-ODE (the deterministic sampler for LIM.) Even this is not supported by further experimental data provided in the appendix, Table 6, where it is shown that LIM-ODE has better $F_1^\\text{pr}$ scores than DLIM for all $\\alpha.$ A more fair claim is that the performance of DLPM/DLIM is comparable to that of LIM/LIM-ODE.\n\n2) Also, in Table 4 we see that the FID for DLPM at $\\alpha=2$ is 21.07 while the FID for DDPM is 19.05. This is a big difference and seems to weaken the point that DLPM at $\\alpha=2$ is similar to DDPM. We know DDPM and DLPM at $\\alpha=2$ are different since one has a square root in the definition of the loss.\n\n3) It seems to me important to consider $\\alpha=1.5$ for the experiments whose results are shown in Table 1 and Table 2. Indeed, Yoon et al (2023) work in the interval $\\alpha \\in [1.5, 2]$ and we further see that performance for LIM is clearly increasing as $\\alpha$ decreases both for the results in Table 1 and Table 2.\n\n4) It is claimed in the paragraph above Table 2 that DLPM5 has a better performance than DLPM over all the range of $\\alpha$ for data coming from a Gaussian grid but a t-test shows that this difference is not statistically significant for $\\alpha=1.7$ and $\\alpha=1.8.$ Also, the results reported in Table 6 for $\\alpha=\\{1.7,1.8,1.9\\}$ do not seem to be statistically significant for supporting the claim that DLPM outperforms LIM, however, standard deviations are not provided.\n\n5) It also seems the experiments missed an important comparison with the ODE counterpart of DDPM, the DDIM from Song et al (2020). That is, even if DLIM outperforms LIM-ODE, it is still relevant to compare it with the non-heavy-tailed version of the ODE framework, which is DDIM.\n\nIf this concerns about the experiments are addressed, it is likely I will increase the score.\n\nSmaller comments\n* It is not clear why the results for the non-isotropic noise case are mentioned since they give consistently worse results. It may be better to just mention that non-isotropic noise does not provide better performance and push the results about it to the appendix. One advantage of DLPM mentioned is that it can handle the non-isotropic noise, but as said before, it does not seem to give better results.\n* line 383 it says 'appart' it should say 'apart.'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6JJwrxjxFI", "reviewer": "Reviewer_NXD4", "summary": "This work studies the diffusion-based models with a heavy-tailed noise and achieves great performance under the imbalanced, heavy-tailed data distribution. Different from the previous LIM method, which starts from a continuous SDE perspective, this work follows the DDPM framework and proposes DLPM and DLIM models. For the proposed methods, this work also shows a suitable training objective to achieve stable training for the $\\alpha$-stable distribution. The synthetic and real-world experiments also support their results.", "questions": "Q1: When considering CIFAT10_LT, LIM performs better than the DLPM method, which can not support the discussion of this work. Could the author discuss it in detail?\n\nQ2: As shown in the theoretical part, if $\\alpha=2$, the DLPM algorithm is equivalent to DDPM. However, the experiment results of DLPM ($\\alpha=2$) are worse than DDPM. Could the author discuss this experiment phenomenon in detail?\n\nQ3: Could the author discuss the reason for the slightly worse performance of $DLPM^{\\text{in}}$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The link between the $\\alpha$-stable distribution and Gaussian noise is helpful since we can adapt the DDPM framework by a simple transformation.\n2. The suitable training objective of DLPM bridges the gap between theory and application. On the contrary, the LIM needs to choose a $L_1$ loss to guarantee stable training.", "weaknesses": "It would be better to discuss the experiment results in detail. Please see the Question part to find the details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T1Ci406OrY", "reviewer": "Reviewer_zxJx", "summary": "This paper leverage a intrigue design, decomposing a heavy-tailed r.v. into a Gaussian and a standard alpha-stable r.v. , to propose a new heavy-tail diffusion model with minor modifications to the DDPM. Experiments showed its superiority.", "questions": "see Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The math derivation of the alpha-stable dm loss is rigorous.\n2. This way of building heavy-tail DM is easier to understand and implement, because DLPM only need sampling to the heavy-tail distribution, there is no need to combine the heavy-tail distribution, into the network.\n3. It is novel to make alpha-stable DM solvable via a decomposition.", "weaknesses": "1. I have a doubt on the convergence property, for t>0, the marginal distribution of this new dynamic should have infinite variance. Then really it would converge to the data distribution (with finite distribution) at t=0? Can you show me more theoretical convergence proof?\n2. The term 'Data Augmentation' typically means increase the size of the dataset by creating new data through modifications or transformations of the original data. I am quite confused why the technique in section 3.2.2 is called 'Data Augmentation'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)