Approximated Anomalous Diffusion: Gaussian Mixture Score-based Generative Models
Authors: Hengyuan Ma
Li Zhang
Xiatian Zhu
Jianfeng Feng
Venue: Submitted to ICLR 2023 (2023)

Abstract: Score-based generative models (SGMs) can generate high-quality samples via Langevin dynamics with a drift term and a diffusion term (Gaussian noise) iteratively calculated and added to a sample until convergence. In biological systems, it is observed that the neural population can conduct heavy-tailed L\'{e}vy dynamics for sampling-based probabilistic representation through neural fluctuations. Critically, unlike the existing sampling process of SGMs, L\'{e}vy dynamics can produce both large jumps and small roaming to explore the sampling space, resulting in better sampling results than Langevin dynamics with a lacking of large jumps. Motivated by this contrast, we explore a new class of SGMs with the sampling based on the L\'{e}vy dynamics. However, exact numerical simulation of the L\'{e}vy dynamics is significantly more challenging and intractable. We hence propose an approximation solution by leveraging Gaussian mixture noises during training to achieve the desired large jumps and small roaming properties. Theoretically, GM-SGMs conduct a probabilistic graphical model used by empirical Bayes for sampling, expanding the maximum a posteriori (MAP) estimation applied by conventional SGMs. Expensive experiments on the challenging image generation tasks show that our GM-SGMs exhibit superior sampling quality over prior art SGMs across various sampling iterations.

Under review as a conference paper at ICLR 2023
APPROXIMATED ANOMALOUS DIFFUSION: GAUSSIAN MIXTURE SCORE-BASED GENERATIVE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
basee-based generative models (SGMs) can generate high-quality samples via Langevin dynamics with drift term and diffusion term (Gaussian noise) itera- tively calculated and added to sample until convergence. In biological systems, it is observed that the neural population can conduct heavy-tailed Lévy dynam- probabilistic representation through neural fluctuations.
ics for sampling-ber Critically, unlike the existing sampling process of SGMs, heavy-tailed Lévy dy- namics can produce both large jumps and small roaming explore the sampling space, resulting in better sampling results than Langevin dynamics with lack- ing of large jumps. Motivated by this contrast, explore new class of SGMs with the sampling based on the Lévy dynamics. However, exact numerical sim- ulation of the Lévy dynamics is significantly more challenging and intractable. We hence propose an alternative solution by leveraging Gaussian mixture noises during training to mimic the desired large jumps and small roaming properties. Theoretically, GM-SGMs conduct probabilistic graphical model used by empir- ical Bayes for sampling, expanding the maximum posteriori (MAP) estimation applied by conventional SGMs. Expensive experiments on the challenging image generation tasks show that our GM-SGMs exhibit superior sampling quality over prior art SGMs across various sampling iterations.
INTRODUCTION
Score-based generative models (SGMs) (Song and Ermon 2019 2020 Song et al., 202 a: Dock- horn al. 2022 Karras et al., 2022) have recently demonstrated tremendous performance in data synthesis, especially ly high-quality images, along with easier model optimization (Song and Ermon 2019), richer generative diversity (Xiao et al. 2022), and solid theories (De Bortoli et al. 2021 During optimization, SGMs learn to fit score function by predicting the Gaussian noises added to a sample drawn from a target dataset. To generate a sample in the target distribution, the SGMs conduct Langevin dynamics constructed from the score function. This process reverses Brownian motion starting from the dataset distribution with i.i.d. Gaussian increments.
As special case of Monte Carlo Markov Chain methods, the Langevin dynamics has been widely applied for constructing sampling-hase based algorithms (Rey-Bellet and Spiliopoulos) 2015). However, increasing evidence from experimental observation suggests that in biological systems, the neural population implements sampling- based probabilistic representations through a heavy-tailed Lévy dynamics (He) 2014 Donoghue al. 2020 Townsend and Gong 2018 Muller et al. 2018). which instead reverses an anomalous diffusion process with heavy-tailed failed increments (Fig. Left)) The neural coding benefits from Lévy dynamics, since can implement large jumps that facilitates the process to escape from local minimal and explore the sampling space more thoroughly (Ye and Zhu 2018 Qi and Gong 2022). natural question arises: can we apply the Lévy dynamics instead of the Langevin dynamics for better sampling performance of SGMs?
Inspired by this insight, we explore a novel class of SGMs that reverse the anomalous diffusion for sampling. Nonetheless, exact numerical simulation of the Lévy dynamics (i.e., reversing the anoma- lous diffusion) is drastically more challenging and intractable, especially for high-dimensional data such images. To tackle this challenge, we consider Brownian motion with Gaussian mixture as an approximation of the anomalous diffusion. To tackle this challenge, we train the SGMs with Gaussian mixture noises enable both large jumps and small roaming during the sampling phase,

---

Under review as a conference paper at ICLR 2023
reminiscent of the Lévy dynamics. Concretely, construct novel variant of SGMs, namely Gaussian Mixture SGMs (GM-SGMs), that learn to denoise Gaussian mixture noises; doing so, model is enabled to reverse the Gaussian mixture Brownian motion by switching between large jump and small roaming during sampling, resembling the merits Lévy dynamics. Theo- retically, our GM-SGMs conduct the probabilistic graphical model (PGM) of empirical Bayes (EB) to reverse Gaussian mixture Brownian motion; Instead, conventional SGMs perform a PGM of maximum a posteriori (MAP) estimation reverse a Brownian motion. Empirically, extensive experiments on several challenging image generation tasks verify the ability of our GM-SGMs to automatically select large jump or small roaming during sampling and the promising ability of our GM-SGMs over state-of-the-art SGMs under different sampling budgets.
2
PRELIMINARY: SCORE-BASED GENERATIVE MODELS
Score matching was originally developed for non-normalized statistical learning (Hyvärinen and Dayan 2005). By observing i.i.d. samples of unknown (target) distribution p in dl dimensions, score matching directly approximates the score function s(x) == Vx log (x(x via a model sæ parameterized by for x E Rd. Score-based generative models (SGMs) aim to generate samples in the distribution p" via score matching through the following iterations
E7 So((,, 1) + 6, (= T=
XT ~ NO,, x1-
where e, is the step size and z, midd I). This process transforms Gaussian noise Xr towards a sample Xn obeying y². [q can be considered as the reverse a corrupting process where the noises gradually added to datum Xo
X ~ p" X = Xx + EiZt, t 0, O,..,,,
(2)
The first SGM, noise conditional score network (NCSN) (Song and Ermon 2019), is trained by fitting the score function s(x) via minimizing the weighted explicit score matching (ESM) objective
T
L(0; (o.))=))
Σ Mor)) v00,00) (..
| + n, s((s (((x+))) +n]]
where of is the noise variance at the time step I, and A(ax)) the weights for each time step f. Discard- ing the constant part independent from 0, the ESM can be rewritten as tractable denoising score matching (DSM) objective (Vincent 2011
T
L(0; (ot)T=)) = Max E Exap' m)(o)0.ff1)
|so(( + n.t)
Vxtn log (x + 1 x)]
2
= Σ
Mas |R a
so + 71
" of
2
1=1
where Po,(' (. (- x) = Ni x, I) Considering the sampling process as a stochastic differential
equation (SDE), [Song al. (2021b) further proposed an improved version NCSN++ that utilizes an existing numerical solver of SDEs to enhance the sampling quality.
3 GAUSSIAN MIXTURE SGMs
3.1 ANOMALOUS DIFFUSION PERSPECTIVE
In this section, introduce wider class of corrupting processes as well their reverses (i.e., the corresponding sampling process), followed by analyzing their fundamental properties. We first write Eq. 2 in continuous formula. Eq. is the discretized version of Brownian motion
Xo p" dx = dw,
(4)
where the stochastic increment is Gaussian and satisfies
Aw, Wi+ W. NO0 At)), At 0.
2

---

Under review as a conference paper at ICLR 2023
S=5(1.2.0.5)
MT.
$
AT.. -42"
000
5,5(1.6,0.5)
Nn (m..0.(11,22 (1))
5
0.2
A10, 1)
-6
-4
-2
6
-4
-2
Figure 1: (Left) The pdf of one-dimensional Lévy stable distribution with different a and standard Gaussian distribution. The Lévy stable distribution is more heavy-tailed than the Gaussian distribu- tion. (Right) Addition of two Gaussian distributions vs. their mixture. Best viewed in color.
The sampling process of SGMs (Eq.(I is the dicretized version of Langevin dynamics
dx } (t) Vx log (x)dt + c(t)dw,
(5)
where c(t) is a time-dependent positive scalar function, and represents reversing the time flow of the process. The Langevin dynamics (Eq. ( is the reverse process of the Brownian motion (Eq. (4)
Further, Eq. is an instance of more general class of stochastic process called stable Lévy process,
Xo p" dx dL",
where a is the Lévy index, and the increment AL; obeys a Lévy stable distribution
AL = Li- L SaS(x; Att).
The Lévy stable distribution SaS has the probabilistic distribution function (pdf)
1 nd expl ||ly|| cos(x y)dy.
SaS(x;a,))
Similar as the Langevin dynamics (Eq. Eq. has its reverse process called Lévy dynamics
<(1) D [p" (x) Vx logp"(x)) + ((t) a
dx
(x)
where (.) the Riesz fractional derivative of order a, and dT represents another stable Lévy process (Mandelbrot and Mandelbrot 1982). Theoretically, we can sample from p" through the Lévy dynamics (Eq. ( that reverses stable Lévy process (6).
When a = 2, 7. Sa.S' becomes the standard Gaussian distribution, and the stable Lévy process a < Sa.S heavy-tailed, and the stable Lévy process becomes anomalous diffusion (a.k.a,, super-diffusion Lévy flight) (Klages al. 2008 Metzler et al. (2014). To visually demonstrate the differences between Gaussian distribution and heavy-tailed Lévy stable distribution, depict their probability distribution functions (pdf) in one-dimensional case.
(Eq.66) becomes a Brownian motion. When
observed from Fig. (left) that compared to Gaussian distribution, the Lévy stable distribution is only more heavy-tailed but also more concentrated around zero center. In particular, the latter property prevents us from simply using a Gaussian distribution with high variance as substitution of the Lévy stable distribution.
We further inspect the stochastic behavior differences of Brownian motion and anomalous diffusion resulted from their different distributions the increment. As shown in Fig. 2 it is evident that the heavy-tailed property of Lévy stable distribution enables the sampling process (i.e. the Lévy dynamics ( to better explore the full space more efficiently (Ye and Zhu 2018). More specifi- cally, the Lévy dynamics can produce large jumps for allowing the sample point to traverse though low-probability regions. As result, it is less possible for the sample point to be trapped by local minimal. Also, it is easier for the sample point to switch between different modes, making it suit- able to deal with multi-mode distribution situations (Qi and Gong 2022). On the other hand, the higher concentration cal regions. Consequently, the anomalous diffusion involves at least two modes: (1) small roaming mode for careful searching local area, and (2) large jump mode for escaping from one
zero cases the Lévy dynamics to conduct small roaming for exploring lo-
3

---

Under review as a conference paper at ICLR 2023
Anomalous diffusion
Gaussian Brownian motion
Brownian motion
Lirge
S00AS SORTIRR
Figure 2: Visualization of (Leff) anomalous diffusion dLa (a 1.8), (Middle) Gaussian mixture Brownian motion dw1. 2023 for all the three simulations. Best viewed in color.
1 (3 = 2), and (Right) Brownian motion dw. We fixed the random seed at
new area for global exploring. It has been widely observed in the neural population of biological systems, where the neural fluctuation is used for probabilistic coding (He) 2014 Donoghue et al., 2020 Townsend and Gong, 2018 Muller al. 2018).
In contrast, the Gaussian distribution is relatively light-tailed. resulting in lacking of large jumps in the Langevin dynamics (Eq.(5). The sample point is thus easier to trapped local minimal. More importantly, for conventional SGMs, the light-tailed issue is not only caused by the stochastic increments Zt, but also intensified by the model So itself, which plays the main role to guide the sample point to the target distribution p'. To illustrate this, revisit the score function in the where the model sa(:, t) is trained to fit
conventional training objective
log Po, (x + 1/ x) -
" of
x x~p",
VIj ERR.
Ideally, the model sa(-,t) t) could learn the above map for all E Rd, since the range of a Gaussian distribution is the whole gd. In practice, however, often sa(- t) only learns this map well for those with relatively high probability being sampled from (00, I), whilst lacking experiences for the cow-probability others (corresponding to the large jumps). As result, the conventional training only allows SGMs learn to reverse a Brownian motion, with less possible large jumps and inferior capability of predicting the corresponding samples.
This insightful analysis above motivates to design a new class of SGMs capable of conducting large jumps similar as the anomalous diffusion for sampling. As the reverse capability of SGM largely shaped by the distribution of the training noises 77, the key is on reformulating this noise distribution for training.
3.2 GAUSSIAN MIXTURE
Although the anomalous diffusion comes with desired merits as discussed above, conducting gen- eral Lévy dynamics (Eq.(7) ) is computationally more challenging. First, the Riesz fractional deriva- tive D is difficult calculate (Çelik and Duman 2012). Second, the Lévy dynamics involves p" which is intractable for high-dimensional cases. Third, sampling from a Lévy stable distribu- tion (Mantegna 1994] is much more expensive than sampling from Gaussian distribution.
To bypass these obstacles, instead of struggling to approximate the Lévy dynamics, we resort to an alternative approach that mimics the functional properties (ie., ability to implement both large jumps and small roaming) of the anomalous diffusion. Formally, we exploit Gaussian mixture sampling
Nmnx (0, (1, (2, =
(x ()
1
where we mix (0, 7) with another Gaussian distribution with large variance (0,,332)) 1). Note, our mixture operation sums two pdfs up to a single mixed pdf. instead of taking the means of samples from two individual distributions respectively. We visualize their differences in Fig. (Right): (I) The latter has much less concentration near zero. (2) Whilst the former (ours) largely keeps the original concentration degree, and is also more heavy-tailed tailed than the latter.
Correspondingly, we define the Gaussian mixture Brownian motion as dw 18 where the increment
Aw Aw!
1,8
WHA
w
1.8
w 1.9
(8)
4

---

Under review as a conference paper at ICLR 2023
obeys the Gaussian mixture distribution Ninx (0, {Ati, (tA)). To verify that Gaussian mix- ture Brownian motion captures the behavior of the anomalous diffusion dL", we conduct a two- dimensional simulation shown in Fig- (Middle). We can that the large jumps property can be approximated well although not exactly for global space exploration, whilst preserving the small roaming trait in local regions.
3.3 TRAINING
As analyzed in Sec. 3.1 the distribution of Eq. determines what kinds of corrupting processes the SGM can reverse. To enable the model to reverse a Gaussian mixture Brownian motion, we propose Gaussian mixture SGMs (GM-SGM), which is trained minimize
T Σ (orr 1=1
50(x + 17,
(9)
where 8 2 is the only hyper-parameter. Compared to the conventional training objective ()q, the only difference is that the noise 7, now sampled from the Gaussian mixture (O, 1oj1, ()))() instead of ((0,of1) at each time step L. Minimizing our objective Eq. equivalent to alternatively minimize the original objective Eq.3) and the following objective
T Σ (0)
37
so Bn, t) + of
(10)
This provides new opportunities, where the scaled noises 3n added the sample x have much larger deviation than expected, foreing the model conduct a large jump in order to recover x p".
As there is no indication with the input about if the noise is scaled by 3 or not, the model must learn to decide on its own when to make the large jump, i.e., learning to switch between large jump mode and small roaming mode properly. After training, at each time step t, GM-SGM will automatically select one of possible denoising prior: the noise to be denoised at this step obeys (O, I) (O, of ())). This enables GM-SGM to vary the step size adaptively some degree with more robustness against outlier cases (e.g., satisfying ((r; O, l) << 1).
To reduce the stochasticity of training, we alternatively minimize two objectives Eq. 3 and E instead of uniformly sampling them. The training procedure is summarized in Alg
Algorithm GM-SGM training
Xi
Input:
The model So, the overall training iterations N, the parameter B.
for n to N do
2
Draw
(x), Uniform 11, T). 17
SGM (MAP)
N I)
n mod then 8= 1=
X:
X.
else
3=3
P(5)
B
end
Backpropagate on X(o.) |5((x + Bn,t) +0 |
GM-SGM (EB)
end for Output: 80
Figure 3: Schematic diagram of the sam- pling inference: SGMs GM-SGMs.
3.4 SAMPLING
During inference, we sample from a trained GM-SGM using exactly the same procedure as conven- tional SGMs (D). Thus, GM-SGMs do not introduce any extra cost during inference. This some inconsistent with our training noise increments 2; ~ Nmic (0, (I, 321)). We provide the theoretical insights and analysis follows.
5

---

Under review as a conference paper at ICLR 2023
4 THEORETICAL ANALYSIS
4.1 UNDERSTANDING GM-SGMS VIA PROBABILISTIC INFERENCE
GM-SGMs can be extended general cases, where we train the model via the Gaussian mixture
Nmix (x: O, p(3) N(x 0, (1)x:0,81)p((B)))) 1))(3)d3
where p(3) is a possibly continuous pdf of the scalar 3. In Sec. 3.2 the distribution of 3 is dis-
crete: p(3 = 1) = p(,3 = 3) = 0.5, corresponding to the small roaming and large jump modes respectively, In this general case, the corresponding objective is
T Σ
1 2
Bn of
Ex-p B
so(x 37.
3
(11)
To understand why the sampling process of GM-SGM can converge the target distribution, we analyze what the GM-SGM is really doing from the perspective of probabilistic inference.
Let us first revisit the conventional SGMs. For convenience, we denote - (x+ t) as Ea(x, t), an equivalent form of the SGM. Then &o is trained to infer the standardized noise & ttroouhh the probabilistic graphical model (PGM)
P(E Xx, x p(xe (g.))(()).
given X, X2 + of and t, where X p². AOO I). and P(E) .N(,.). This PGM is the same as that in a maximum posteriori estimation (MAP) formula. After estimating & by &a. at step 1, the model can push the sample point x, along the direction of - to approximate the target distribution. This approximation is reasonable due to following relationship.
Theorem 1. Given the corrupting process (a, > ass VL> 8)
Xo ~ p Xx = X + ochts &t mi.d NO,,
(12)
for any time points
S < t, we have
o² 0² (xt xu), (of - (0f-o?)(0 (o - of o²
p(x, X4, x,) = N(x
of o²
See the proof in A. of Appendix. As a result, if we set u = 0, it is clear that if x, moves along the direction - (x, xo)/o, with small enough step, it will be closer to x, for < t. During sampling, the SGM takes the step along this direction with scalar 3 controlling the step size at every time step and a random increment z, that represents the uncertainty in p(x, x,,
When applying the above analysis to GM-SGM, we find that this time Eo is trained to infer & through
p(E (), xx f p(x: 1 E, OP(E ())(3 xx, 1)d3, p(3 xx, xx p(xe B, (3(()
(13)
where p(3) is the distribution of | in Eq. (1). and p(E - 321). Therefore, GM- SGMs also infer a direction - to push the sample point X; closer to xs, s < t. The key difference that infers the direction - inferred through the PGM of maximum a posteriori estimation (MAP) formula with the SGMs, whilst through the PGM of empirical Bayes (EB) formula (BP and TA 200T) by our GM-SGM. As illustrated in Fig. 3 GM-SGM takes into consideration of probability that & is associated with different covariance matrices f2 corresponding to large jumps and small roaming.
With the above analysis, we can more clearly explain why we apply the Langevin dynamics as the sampling process of GM-SGM, with the increments in Gaussian distribution instead of Gaussian mixture. From the perspective of probabilistic inference, 2, represents the level of uncertainty of estimating & at time step t. Indeed, during training, we do not specify different levels of uncertainty of & for different 3 given the time t. This means the model infers at the same level of certainty at each time step t, independent of 3. Therefore, for GM-SGM we adopt the sampling process as the SGMs. Otherwise, when Gaussian mixture increments are used for sampling, the model performance would degrade due to inconsistent uncertainty introduced (verified in Sec. 5

---

Under review as a conference paper at ICLR 2023
4.2 RELATIONSHIP THE CONVENTIONAL SGMS
Following the relationships between our GM-SGMs and conventional SGMs as analyzed in Sec. further make additional interesting connections as follows.
First, GM-SGMs can be considered as a more robust variant of SGMs, Extra robustness comes from that the model is optimized to deal with large jumps 3m, NoO, 3 1) > (the outliers of (O o) at each step & (Eq. (10)).
Second, training GM-SGMs can be regarded an ensemble learning procedure of training SGMs. Specifically, for every 3, we train SGM (denoted as $3,5) by minimizing corresponding objective constructed by 3, followed by mixing these SGMs based on p(.).
Third, the sampling process GM-SGMs can be seen as reversing a mixture of multiple Brownian motions, where the noise term 2, amplified by different 3. These corrupting processes expand the hypothesis space of original SGMs which only contains single Brownian motion. The model sø adaptively selects
of them for denoising, i.e., switching from different modes (B for large jumps and 3 < 1 for small roaming). This will be verified in Fig. 4 As result, going beyond conventional SGMs with a need for designing step schedule for sampling, our GM-SGMs can vary the step sizes of the score function automatically.
EXPERIMENTS
We evaluate image generation tasks on CIFAR-I0 10 (Krizhevsky et al., 2009). CelebA (Liu et al. 2015) and LSUN (church and bedroom) Yu et al. 2015). We focus on comparing GM-SGMs with the prior art SGMs, namely NCSN+ (Song al. 202Tb). For fair comparison, we construct our proposed GM-SGM using the same U-Net architecture as NCSN++ We use the released check- points of NCSN+ for all the datasets except CelebA for which no checkpoints released and we use the released codes to train by ourselves. We first set 3 2 in Alg. [ and investigate the effect of different 3 later. We provide more detailed settings implementation in A.2 of Appendix.
ratio of variance
250
150
SGM
150
130
1.00
110
1:00
40
Figure 4: The norm ||s/|| of GM-SGMs and SGMs. (Left) The norms across several sampling processes. The ratio of (Middle) mean Tmean and (Right) variance Tear at different time steps.
5.1 SAMPLING STATISTICS
To verify that our GM-SGM indeed makes large jumps following the PGM of empirical Bayes in- ference, we track the values of the score function So (x,, t) evaluated during the sampling procedure and calculate the norm of the estimated noise vector &o (x,, t) for every time step (see Sec. 4.1 We denote < cGM-SGM (x,, f) and < (x,, f) as the noise vector produced by our GM-SGM and SGMs respectively. For relative comparison, we calculate the ratio of mean and the ratio of variance
[ E eM (X t)]]]
cM 1G.MMGGM (Xe: t]]
var
imean (t)
E
EGG Xx, ||]
Pvar
var
SGM 58
(x,, t)
where /reen/Tvvr represent the relative denoising step-sizeevariaatio taken by our GM-SGM over con- vnetional SGMs on average. We sample 512 images using GM-SGM and SGM trained on CelebA (64 64) (Luu et al. 2015) respectively and plot the results of sampling statistics in Fig. We draw several key observations: (1) With the Timean > across the whole sampling process, we validate that GM-SGM does take more large jumps. (2) F'vr starts the range of [50, 150] and surges
7

---

Under review as a conference paper at ICLR 2023
T-50
T-200
SEMM CM SON
T-50 T-200
T-20
T-200
SOM GM--SOM
Figure 5: Image samples generated by SGMs and GM-SGMs on CIFAR-10, CelebA with 64 x 64 resolution and LSUN (church and bedroom) with 256 256 resolutions.
rapidly afterwards followed by a quick dip. This implies that when the sample point is sufficiently informative (e.g., close to the target distribution), our GM-SGM would activately infer the noises based on different hypothesis of 8. In contrast, the norm variation by SGM is always much smaller. The underlying reason that, GM-SGM can take much more diverse sampling paths, expressed by Eq. [1. More specifically, at every step GM-SGM estimate with two possible selections the prior distribution of ş: No, I) or (0, B²1) at each Therefore, there are a total 2 denoising paths GM-SGM can choose from during sampling.
5.2 EVALUATION ON IMAGE SYNTHESIS QUALITY
Next, we evaluate the sampling quality. shown in Fig. our GM-SGMs can generate much better quality images than SGMs under the same iterations T. See A.7 of Appendix x for more visual comparison. For quantitative measurement, we adopt the popular Fréchet inception distance (FID) (Heusel et al. 2017). Lower FID is better. As shown in Table GM-SGM is consistently superior over SGM across different sampling budgets and generation tasks.
Table 1: FID score of generating object im- ages on CIFAR-10. Lower FID is better. SGM: NCSN+ Song et al. (202Ib)).
Table 2: FID score of generating human facial images on CelebA (64 x 64). Lower FID is better. SGM: NCSN++ (Song et al. (2021b)).
Iterations T
SGM
GM-SGM
Iterations T
SGM
GM-SGM
1000
2.50
2.25
500
4.01
3.75
500
3.54
3.53
400
6.56
3.98
400
4.35
4.00
333
14.53
4.10
333
6.24
4.55
200
257.64
4,51
250
12.70
6.18
100
435.59
5.99
200
29.39
8.76
80
437.98
6.20
100
306.91
37.63
66
436.91
6.26
50
456.49
29.41
20
439.32
27.81
5.3 GM-SGM PARAMETER ANALYSIS
We investigate the effect of 3 on sampling quality. The default setting is 3 = 2. In general, if B over small, our model advantage will be reduced due to reduced heavy-tail property. If 3 is over large, the denoising task would become too challenge to solve, hence harming the performance. To empirically validate this, we test GM-SGME by varying 3 in [1.5, 3] on CIFAR- 10 with 50 iterations. When / 1.5, the FID increases from 29.4 to 37.9. When 9 = 3, the FID goes to 105.4. However, both are still better than 456.5 by SGM. See Tab. 3 in A.3 of Appendix for more results. Also, we test other Gaussian mixture designs and find them less effective (see of Appendix).
8

---

Under review as a conference paper at ICLR 2023
5.4 FURTHER STUDY
We investigate more designs of GM-SGMs. First, we only use the objective ( for training. As shown in Fig. 7 of Appendix, the model fails to generate natural images even under many iterations. This is because this model only learns to implement large jumps. while ignoring small roaming which is also necessary for space exploring, as discussed Sec. [3.1 Second, instead of Gaussian increments as SGMs, we turn to use Gaussian mixture (0, {T, 3).)). We test this on
CelebA (64 64) under 333 iterations. We find the FID increases from 4.1 to 5.2 (still much better than 14.5 by SGMs). This verifies empirically our analysis in Sec. [.11 the probabilistic inference of our GM-SGM shares the same level of uncertainty for all | at each step t. We also investigate GM-SGMs using Gaussian mixture with general covariance matrices in A.5 of Appendix.
RELATED WORK
The first SGMs, the noise conditional score network (NCSN), is introduced (Song and Ermon 2019) Later on, Song and Ermon (2020) further improved the NCSN by scaling the noises and improving the stability with the moving average. By abstracting the previous SGMs into a unified framework based on the stochastic differential equation (SDE), Song et al. (2021b) proposed the NCSN++ for high-resolution image generation using numerical SDE solvers and several architectural en- hancements. Relying on Hamiltonian Monte Carlo methods (Neal et al.) [201I), vriically-damped Langevin diffusion (CLD) based SGMs introduced (Dockhorn et al. (022). Alternatively, [Jing et al. (022)) implemented the sampling process on a series of selected subspaces. Besides, Vahdat et al. (021) trained the SGMs in a latent space of variational autoencoder (Kingma and Welling, 2019). While|Karaal et al. (2022) focused refining the training Syper-parameters, sampling sched- ule and high-order numerical methods for better performance.
DDPMs (Ho al. 2020) are another class of iterative denoising based generation models. Their ori- gin is DickseDi et al. (013)) that proposed to destroy the data through a diffusion process, and learn to reverse this process via maximizing the variational bound. Although theoretically equivalent to SGMs (Vincent 2011). DDPMs adopt a variance preserving (VP) stochastic process for sampling. In contrast, SGMs consider a variance exploding (VE) stochastic process (i.e., the Langevin dynam- ies). There have been further improvements for DDPMs (Yang et al.) 2022). For instance, DDIMs (Song et al. (020) generate samples via a class of nor-Markovian diffusion processes. Nichol and
Dhariwal 2021) proposed to learn the noise schedules for better sampling quality. [Liu et al. (2021)
further improved DDIMs by implementing the sampling on manifold for better denoising. |ao
al. (2022b a attempted to optimize the discrete-time schedules for sampling speed-up.
Application of SGMs and DDPMs has been rapidly evolved, e.g., text-to-image generation (Nichol et al. | Saharia et al., 2022 Rombach et al. (022). Recently, SGMs were applied in creating,
editing, and recovering photo-realistic images (Meng et al. 2021 (Saharia et al,, 2021a b. Kawar
et al. 2022). or high fidelity audio streams (Kong et al. 2021 Chen et al. 20211)
Complementary to all the prior efforts, in this work we investigate the fundamental limitation of lacking large jumps in sampling with conventional SGMs and construct more robust variant char- acterized by superior exploration ability, along with solid theoretical justification.
7 CONCLUSION
Inspired by the heavy-tailed Lévy dynamics (the reverse of the anomalous diffusion) implemented by neural coding biological systems, we propose novel class of generative models, namely Gaus- sian mixture score-based generative models (GM-SGMs). In contrast to conventional SGMs based on Langevin dynamics, our models are featured with a relatively heavy-tailed sampling property and capability of more thoroughly exploring the sampling space. Specifically, GM-SGMs are trained by denoising Gaussian mixture noises with built-in automatic switches between large jumps and small roaming in the spirit of anomalous diffusion. Theoretically, GM-SGMs implement probabilistic graphical model (PGM) of an empirical Bayesian for generation, expanding the PGM of maximum a posteriori estimation used by the SGMs; This results in sampling process with automatic selection of the step schedule and more robustness. Empirically, we demonstrate that our GM-SGMs signifi- cantly outperform the conventional SGMs on diversity of challenging image generation tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In ICML, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICLR, 2022b.
Carlin BP and Louis TA. Bayes and empirical bayes methods for date analysis, 2001.
Cem Çelik and Melda Duman. Crank-nicolson method for the fractional diffusion equation with the riesz fractional derivative. Journal of computational physies, 2012.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In ICLR, 2021.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. NeurIPS, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In ICLR, 2022.
Thomas Donoghue, Matar Haller, Erik Peterson, Paroma Varma, Priyadarshiní Sehastian, Richard Gao, Torben Noto, Antonio H Lara, Joni D Wallis, Robert T Knight, et al. Parameterizing neural power spectra into periodic and aperiodic components. Nature neuroscience, 2020.
Biyu J He. Scale-free brain activity: past, present, and future. Trends in cognitive sciences, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score match- ing. JMLR, 2005.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion gener- ative models. arXiv preprint. 2022.
Alexia DolicoeurMMartincau Ke Li, Rémi Piché-Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint, 2022.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint, 2022.
Diederik Kingma and Max Welling. An introduction to variational autoencoders. arXiv preprint, 2019.
Rainer Klages, Günter Radons, and Igor Mihajlovič Sokolov. Anomalous transport. 2008.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwwae: A versatile diffusion model for audio synthesis. ICLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2021.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
10

---

Under review as a conference paper at ICLR 2023
Benoit B Mandelbrot and Benoit B Mandelbrot. The fractal geometry of nature. 1982.
Rosario Nunzio Mantegna. Fast, accurate algorithm for numerical simulation of levy stable stochas- tic processes. Physical Review E, 1994.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021.
Ralf Metzler, Jae- Jae-Hyung Jeon, Andrey G Cherstvy. and Eli Barkai. Anomalous diffusion models and their properties: non-stationarity, non-ergodicity, and ageing the centenary single particle tracking. Physical Chemistry Chemical Physics, 2014.
Lyle Muller, Frédéric Chavane, John Reynolds, and Terrence Sejnowski. Cortical travelling waves: mechanisms and computational principles. Nature Reviews Neuroscience, 2018.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models. arXiv preprint arXiv, 2021.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2011.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Yang Qi and Pulin Gong. Fractional neural sampling as a theory of spatiotemporal probabilistic computations in neural circuits. Nature communications, 2022.
Luc Rey-Bellet and Konstantinos Spiliopoulos. Irreversible langevin samplers and variance reduc- tion: large deviations approach. Nonlinearity, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint, 2021a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint, 2021b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NewrlPS, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-bar based generative models. In NeurIPS, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training score- based diffusion models. In NeurIPS, 2021a.

---

Under review as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Rory G Townsend and Pulin Gong. Detection and analysis of spatiotemporal patterns in brain actívity. PLoS computational biology, 2018.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Neural Information Processing Systems (NeurlPS). 2021.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 2011.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In ICLR, 2022.
Ling Yang, Zhilong Zhang, and Shenda Hong. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint, 2022.
Nanyang and Zhanxing Zhu. Stochastic fractional hamiltonian monte carlo. IJCAI, 2018
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint, 2015.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOF OF THM.
Proof. Using Bayes" theorem, have
p(x, Xx, Xu) x p(x: Xs; Xx)xxx)(xx, xx).
According to Eq. 12). we have
p(x, X,,Xa p(xx xx)
exp
1
1
|x, x 11
2 - o²))
2
o - 0²
and
1
p(xx| x.)
exp(
X3 X 1=
(
σ d
2 0
- 0
After taking logarithm at the both sides of the equation and ignoring a constant term, we have log p(x, Xx,
1
X4 x,
X x
2 20
2
20
of 0
x, (
0
( :
o..
X1
X
2 (oū
(o) 0
0
2
02
0
of 0
(
(0
1
X, (x
[xi
Xx))
2 [o) ( +
of o
o3
of
0
2 ( (o) o
X,
(x
?
(x x,))
which implies that p(x, matrix can be obtained
x,, x.) also obeys Gaussian distribution, and its mean and covariance
0
Ex, Ex,xx(x/xxxx.) xx.x)) [xx] = X,
2 0 (xt- x,),
(0 0²² (0 0 o cov xovx,~p(x//xxx [xo]
oi o
Hence, the estimated direction (x, = -(x, x guides the sample point x, closer to X,
for some 8 < 1. whose distribution is closer to p".
A.2 DETAILED SETTINGS
We train the GM-SGM on CIFAR-10 for 550k iterations at the batch size of 64 single NVIDIA RTX 3090 GPU to compare the public checkpoint of the SGM (NCSN++) trained on CIFAR-10. We train both GM-SGM and SGMs on CelebA (64 64) for 230k iterations the batch size of 64 on two NVIDIA RTX 3090 GPUs. We train the GM-SGM on LSUN (church) and LSUN (bedroom) at resolution of 256 x 256 for 250k iterations the batch size of 64 and compare the public checkpoint of the SGM (NCSN++) trained on LSUN (church) on 4 NVIDIA RTX A6000 optimizer and learning schedule as that in Song
GPUs. We apply the same hyper-parameter et al. (2021b). For sampling, we use the same generation process of continuous variance exploding SaNnsN+++Sonnga NCSN++ [Song et al. (202Ib) with both predictor and corrector for both SGMs and GM-SGM. For sampling. all the experiments apply the original linear schedule used in Song et al. (2021b) with both predictor and corrector (PC).
A.3 EXTRA RESULTS ON PARAMETER ANALYSIS
We report results how the parameter / influences the performance of GM-SGM in Tab. In particular, when = 1, we get the original SGM.
13

---

Under review as a conference paper at ICLR 2023
Table 3: The comparison of the performance of GM-SGMs under different Alg.
Metries:
FID score. Dataset: CIFAR-10.
Iterations
200
250
500
& (SGM)
456.49
29.39
12.70
3.54
1.5
37.91
9.06
7.07
3.33
3 2
29.41
8.76
6.18
3.53
105.40
11.24
8.08
3.64
A.4 COMPARISON TO OTHER METHODS
In this section, we compare our GM-SGMs with existing SGMs and DDPMs. First, we compare GM-SGMs with SGMs equipped with adaptive SDE solver (ASDE) (Jolicoeur- Marbbnneaa et al.) 2021]. As shown in Table our GM-SGMs outperform the competitors significantly. This suggests that our method has superior acceleration benefit. Also note GM-SGMs do not alter the sam- pling process, without adding extra per-step complexity during inference. In contrast, alternative acceleration methods usually increase per-step cost.
Table 4: FID of SGMs (Song et al. 2021b), GM-SGMs, and SGMs equipped with adaptive SDE solver (Jolicoeur-Martincau
2021). Dataset: CIFAR-I0.
adaptive SDE solver (VP) adaptive SDE solver (VE) SGM GM-SGM (ours)
Iterations T
49
50
50
50
FID
82.42
307.32
456.49
29.41
Next, we compare GM-SGMs with DDPMs using non-Gann non-Gaussian noises for training or sampling, where Gaussian noises are replaced by noises obeying Gamma distribution. As shown in Table 5 our GM-SGMs are superior over DDPMs, DDPM-based DDGM, and comparable to DDIM DDIM-based DDGM, despite that their base model DDPM and DDIM already have better low-iteration performance than our base model SGMs.
including DDGM (Nachmani et al. 202 I
Table 5: FID of SGMs (Song et al. 2021b), GM-SGMs, DDPM (Ho et al. 2020 DDIM Song al. 2020), and DDGM (based on DDPM and DDIM) (Nachmani al. 2021). Dataset: CelebA
(64 x 64).
Iterations T DDPM DDIM DDGM (DDPM-based) DDGM (DDIM- SGM GM-SGM (ours)
20
183.83
13.73
28.24
6.83
439.32
27.81
100
45.2
6.53
14.22
3.17
435.59
5,99
A.5
MORE GENERAL CASES
In Sec. 4.1 we consider the cases using Gaussian mixture N (x; 0. p(3)) for training. fact, this can be generalized to multi- multi-dimensional Gaussian mixture cases. Denoting a distribution of invertible matrix B as pBB), we can consider a more general training objective
T.
Byy of
Σ ((a1) Byp(B)
Ex-p' ~00,00))))) 2
so + By, 1) +
B
It is not obvious which kinds of p( (3) can train better GM-SGM. Here, we try p (B) motivated by the observation that for image generation, SGMs tend to generate images with too much high- frequency noise when the iterations number is low (see examples in Sec. 3.4 To suppress the generation for high- figeffryyyency components, we design p(B) matrix as follows
p(B I) 0.5, P( 2M) 0.5,
14

---

Under review as a conference paper at ICLR 2023
where M is filter operated on the height and width coordinates shrink the top-5% high- feehurryeency part of an image by 0.5. We obtain the following results: For CelebA (64 x 64), the FID of GM- SGM decreases to 3.77 400 iterations (better than the result of GM-SGM in the main paper); On CIFAR-10 10, the FID of GM-SGM decreases to 2.24 at 1 1000 iterations, better than SGMs (2.51). This implies that the optimal settings of Gaussian mixture be more complicated, which calls for further study in the future.
A.6 FAILURE EXAMPLES
In Fig. and Fig. we show the failure examples under improper parameter settings as mentioned in Sec.
Figure 6: Image samples generated by a GM- SGM trained CIFAR-10, where we apply the Gaussian mixture (00, (0.551, 23) in- stead of Gaussian mixture (0, {I, 221)) for training. Iterations: 2000.
Figure 7: Image samples generated by GM- SGM trained on CelebA (64 64), where apply NN 22 instead of Gaussian mixture (0, {I, 2)) for training. Iterations: 2000.
A.7 MORE VISUAL EXAMPLES
We first show examples generated by SGMs and GM-SGMs under several settings of iterations number T in Fig. 19
then we only show GM-SGMs with even lower iterations LSUN in Fig. 21 Under these low iterations, SGMs only generate high-frequecny noises similar as left of Fig. 8
15

---

Under review as a conference paper at ICLR 2023
Figure 8: Image samples generated by SGMs (left) and GM-SGMs (right). Dataset: CIFAR-10. Iterations: 50.
Figure 9: Image samples generated by SGMs (left) and GM-SGMs (right). Dataset: CIFAR-10. Iterations: 200.
16

---

Under review as a conference paper at ICLR 2023
Figure 10: Image samples generated by SGMs (left) and GM-SGMs (right). Dataset: CIFAR-10.
Iterations: 500.
Figure 11: Facial image samples generated by SGMs (left) and GM-SGMs (right). Dataset: CelebA
(64 x 64). Iterations: 20.
17

---

Under review as a conference paper at ICLR 2023
Figure 12: Facial image samples generated by SGMs (left) and GM-SGMs (right). Dataset: CelebA
(64 x 64). Iterations: 200.
Figure 13: Facial image samples generated by SGMs (left) and GM-SGMs (right). Dataset: CelebA
(64 x 64). Iterations: 400.
18

---

Under review as a conference paper at ICLR 2023
Figure 14: Church image samples generated by SGMs (left) and GM-SGMs (right). Dataset: LSUN (church) (256 x 256). Iterations: 100.
Figure 15: Church image samples generated by SGMs (left) and GM-SGMs (right). Dataset: LSUN (church) (256 x 256). Iterations: 200.
19

---

Under review as a conference paper at ICLR 2023
Figure 16: Church image samples generated by SGMs (left) and GM-SGMs (right). Dataset: LSUN (church) (256 x 256). Iterations: 400.
Figure 17: Bedroom image samples generated by SGMs (left) and GM-SGMs (right). Dataset:
LSUN (bedroom) (256 x 256). Iterations: 100.
20

---

Under review as a conference paper at ICLR 2023
Figure 18: Bedroom image samples generated by SGMs (left) and GM-SGMs (right). Dataset: LSUN (bedroom) (256 x 256). Iterations: 200.
Figure 19: Bedroom image samples generated by SGMs (left) and GM-SGMs (right). Dataset:
LSUN (bedroom) (256 x 256). Iterations: 400.
21

---

Under review as a conference paper at ICLR 2023
Figure 20: Church image samples generated by GM-SGMs with 50 iterations (left) and 40 iterations (right). Dataset: LSUN (church) (256 x 256). SGMs only produce unrecognizable noises (like left of Fig. I below 100 iterations.
tions (right). Dataset: LSUN (bedroom) (256 x 256). SGMs only produce unrecognizable noises Figure 21: Bedroom image samples generated by GM-SGMs with 50 iterations (left) and 40 itera-
(like left of Fig. IT] below 100 iterations.
22


Reviews:
[{"review_id": "ppbtFXJAOE", "reviewer": "Reviewer_9CCy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on heavy-tailed Levy dynamics can produce both large jumps and small roaming to explore the sampling space, resulting in better sampling results than Langevin dynamics with a lacking of large jumps, the authors explore a new class of score-based generative models (SGMs) with sampling based on the Levy dynamics. However, the exact numerical simulation of the Levy dynamics is significantly more challenging and intractable. The authors propose an approximation solution by leveraging Gaussian mixture noises during training to achieve the desired large jumps and small roaming properties.", "summary_of_the_review": "It seems to me that this work simply replaces Gaussian noise with Gaussian mixtures in SGMs. However, there are some questions that are not clear to me. 1) Why Gaussian mixtures can approximate heavy-tail distributions? This seems invalid to me. 2) Do we really need to replace Gaussian noise in the Langevin dynamics with others to improve sampling? As far as I am aware, the mainstream technique is to improve sampling techniques, e.g., adaptive bias force, parallel tempering, etc. ", "strength_and_weaknesses": "Strength: replacing Gaussian with Gaussian mixtures seems interesting, but I am not sure since I have never done any research in this area.\n\n\nWeakness: \n\n1. Why can we use Gaussian mixtures to approximate the Levy distribution? How accurate the approximation is? I wonder if there is any theoretical guarantee. Intuitive, Gaussian mixtures are still lighted-tailed. I would like to see some rigorous analysis of these.\n\n\n2.  It seems to me that by doing parallel tempering of Langevin dynamics and related techniques, we can also achieve both small roaming mode and large jump mode.\n\n3. The stationary distribution of Langevin dynamics is known. However, when replacing the Gaussian noise with the Levy distribution or even Gaussian mixtures, the stationary distribution will be changed completely, and actually, we do not know what is the stationary distribution anymore.\n\n\nQuestion to the author:\n\nAfter reading the paper, I am still not sure how to perform the reverse procedure when the Gaussian mixture is applied in the forward procedure. Can authors comment on this?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ou4FTigxcYp", "reviewer": "Reviewer_ULSW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors argue that the Lévy dynamics sampling method is preferable to sampling based on Langevin dynamics in score-based generative models (SGM). The reason is based on the hypothesis that since Lévy dynamics has a heavy-tailed distribution, it can produce both large jumps and small roaming in the search for the sampling space, and thus can provide better sampling results than Langevin dynamics, which is not expected to produce large jumps. On the other hand, since exact numerical simulation of Lévy dynamics is difficult, the authors propose to use Gaussian mixture noises as an approximate method. Specifically, the authors propose Gaussian mixture SGM (GM-SGM) as a new type of SGM that learns to denoise Gaussian mixture noises.", "summary_of_the_review": "The use of Lévy dynamics to create large jumps is a major contribution for achieving highly accurate image generation with a small number of steps. A discussion of the computation time would be helpful.", "strength_and_weaknesses": "Strength:\n1. Using Levy dynamics allows sample points to generate large jumps to pass through low-probability regions, making it less likely that they will remain at local minima.\n\n2. Because of the high density near the zero center in Lévy dynamics, there is an expectation that local regions can be searched efficiently.\n\n3. Experimental results show that a mixture of sampling from Gaussian distributions, which are computationally less expensive to approximate, instead of sampling from a Lévy distribution, still captures the desired behavior.\n\n4. GM-SGM can automatically select large jumps or small roaming when sampling.\n\n\nConcerns:\n1. How much computation time/cost is needed for inference at each step t compared to SGM?\n\n2. If GM-SGM can achieve high performance with a small number of iterations, can it achieve a better score with more iterations, or is the limit the current FID score (around 3.5)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MSaAtAyInI", "reviewer": "Reviewer_vUWR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work presented in this paper stems from the observation that sampling from a target distribution can benefit from heavy-tailed Levy dynamics, whereby the stochastic process is allowed to implement large jumps, thus avoiding local minima and allowing the process to explore the distribution space more efficiently.\n\nThus, the idea is to extend score-based generative models, a la Song et al.21b, such that the SDE dynamics describing the forward and backward diffusion processes can benefit from a Levy-like behavior, and allow large jumps in the corruption and denoising steps.\n\nHowever, the inversion of an alpha-stable Levy process is intractable, and the authors suggest approximating the Levy process by inducing a Gaussian mixture model on the Brownian motion term of diffusion of the vanilla score-based SDE model. The hope is that this approximation faithfully reproduces the properties of an alpha-stable Levy distribution. It is important to note that while training the proposed model requires the Gaussian mixture of noise variances, samping from it is instead done using the vanilla approach of simulating the reverse SDE with a single, time-varying Gaussian noise distribution.\n\nA series of experiments complement the methodological contribution of this work, whereby the authors compare their model to the vanilla score-based diffusion model, using three popular datasets for the task. In summary, the results indicate that the proposed method outperforms generative quality (in terms of FID score) of vanilla score-based models, at a fraction of the number of sampling steps.\n", "summary_of_the_review": "This work presents an interesting perspective, although it lacks a clear motivation. The attempt to solve the intractability of the proposed alpha-stable Levy diffusion process is not convincing in my opinion. The proposed approach achieves good empirical results, when compared to a simple baseline, but it has not been compared to alternative methods to accelerate sampling. There are many other weaknesses in the submitted paper, stemming from lack of rigor in the mathematical description of the model, as well as in the proposed approximation of the Levy diffusion process.\n\n\n=============================\n\nPost rebuttal comment\n\n=============================\n\nI've read the rebuttal, all reviews and their comments. I am willing to slightly raise my score, but I am not convinced this submission deserves to be accepted at this time. I hope the Authors will find comments useful to further improve their work.", "strength_and_weaknesses": "The strengths of the paper are:\n* Improving the efficiency of score-based generative models is an important topic of research, as it is well known that baselines are very costly both in terms of training and sampling\n\n* The experimental results show that the proposed method achieves superior quality of generated images when compared to a vanilla score-based generative model\n\n* This work attempts at porting the body of literature on sampling with alpha-stable Levy noise distributions to diffusion models\n\nThe weaknesses of the paper are:\n* There is no explicit motivation behind this work! In absence of it, my understanding is that the goal of this paper (based on the emphasis given on the results, and on the concluding remarks) is to accelerate sampling for diffusion models. I think that the authors should clearly state early on in their paper what is the problem they are trying to solve, and why current existing solutions are not sufficient.\n\n* Based on the assumption above, if the goal is sampling acceleration, then the experimental campaign can be improved, by comparing the proposed method to other methods that aim at accelerating sampling, or in general making diffusion models more efficient. For example, several works (for example, [1]) propose adaptive integration schemes to simulate the backward SDE of a score-based diffusion model. I suggest to compare the proposed method to comparable approaches in the literature, instead of using a vanilla baseline which in this case would be “easy to beat”.\n\n[1] https://arxiv.org/abs/2105.14080 “Gotta Go Fast When Generating Data with Score-Based Models”\n\n* The quality of mathematical notation and in general mathematical rigor can be improved. Starting from eq (1), the notation is borrowed from the work on Langevin dynamics from “Bayesian Learning via Stochastic Gradient Langevin Dynamics”, Welling and Teh. But then the subsequent notation comes from early work on score-based generative models, to end up (in sec 3) with the notation and formalism of SDEs. Now, the problem is that step size, noise terms, noise schedule, the diffusion term, etc… use a similar if not the same variable name.  This is very confusing, and should be cleaned up. \nI have nothing against presenting the background in reverse, starting from the discretized SDEs, to continuous time SDEs expressions of the diffusion process, but at least the notation should be coherent. Also, some more details (in the background section) could be useful, as for example stating why you focus on variance exploding SDEs, and not variance preserving.\n\n* I understand the definition of an alpha-stable Levy diffusion process is proposed as an original contribution, but I would kindly ask the authors to compare their proposal with similar work that question the use of Gaussian noise only, and proposed to use a different distribution, such as [2]\n\n[2] https://arxiv.org/abs/2110.05948, “Denoising Diffusion Gamma Models”\n\n* Due to the intractability of the reverse Levy process, which requires solving a fractional derivative (for which approximate algorithms exist) and, more importantly, requires access to the data distribution, which is exactly what we are aiming to sample from, the proposed approach is to “approximate” the Levy walk with a Gaussian mixture noise distribution. However, even if the simple 1-D example given in Fig.2 shows some similarities between the trajectories of the proposed approximation and a true Levy walk, in principle the approximation does not guarantee the same properties. There are no large jumps, and the basic principle of “escaping from a local minimum” of a Levy walk cannot be ensured.\n\n* Experiments. In general, experiments are appropriate, as comparing the method to vanilla score-based diffusion is a valid approach to make sure the proposed method works. Nevertheless, the weakness of the empirical evaluation is that it does not compare the proposed method to existing methods from the literature that aim at accelerating sampling. In my view, as there are no clear motivations in the context of score-based generative models, to model the diffusion process as an alpha-stable Levy diffusion, the main take home message I get from the experiments is that it produces better samples faster than the vanilla model. Then, it makes sense to ask how the proposed method fares when compared to, say, better SDE integrators.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jgimg7IjjS1", "reviewer": "Reviewer_HQ5G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a class of score-based generative models motivated by Levy dynamics to produce large jumps and small roaming to facilitate exploration. Since the exact numerical simulation of Levy dynamics is quite challenging and intractable, the authors proposed a vanilla mixture formulation to approximate Levy dynamics; the authors relate such an update with the probabilistic graphical model for illustration purposes. The authors also empirically verified that the exploration property has appealing properties in generative models.", "summary_of_the_review": "The idea of adopting Levy process to facilitate exploration is interesting; the Gaussian mixture approximation is acceptable; however, the method itself is not good enough in my opinion. As such, I tend to be conservative in my ratings.", "strength_and_weaknesses": "\n\nPros: the insight of exploring the mixture of Gaussian distributions to approximate heavy tail distributions is quite appealing. I like this idea and the local trap problem in generative tasks is always a critical concern. There are various popular methodologies to tackle this issue, but no one has studied the important Levy dynamics before. As such, it is worthwhile to study related algorithms and how such an idea performs in applications such as generative models.\n\nCons: \n\n1. My biggest concern is that the mixture formulation is not general enough; why it has to be 50% and 50%, just because it is easier to implement (n mod 2 ==0)? Can we achieve it in a better way, say probabilistically to achieve more flexible settings?\n\n2. the transition from Levy process to Gaussian mixture is a bit fast. For certain types of Levy process, what is the optimal Gaussian mixture under some formulations to approximate it?\n\n3. In section 4.2, the authors said the mixture update is more robust, but not enough proof can support that.\n\n4. The mixture formulation is first presented in section 3.2 and then becomes continuous in section 4.1, but then goes to discrete again, which looks confusing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject